Path information writing method and device, computer device, medium and program product

Abstract

The application relates to the technical field of distributed file systems, and discloses a path information disk landing method and device, computer equipment, a medium and a program product, which comprise the following steps: acquiring a path information list; traversing the path information list to obtain the node number of a traversed node and to-be-landed path information; determining the belonging object of the traversed node according to a preset object division parameter and the node number of the traversed node, and storing the node number of the traversed node and the to-be-landed path information into the corresponding belonging object; determining the node number of each belonging object; determining the data processing round number of a target belonging object and the processing node number of each data processing round according to the node number in the target belonging object and a preset number threshold; and performing a disk landing operation on the to-be-landed path information according to the data processing round number of the target belonging object and the processing node number of each data processing round. The application can shorten the time length of landing path information and reduce network pressure.

Description

Technical Field

[0001] This invention relates to the field of distributed file system technology, specifically to methods, apparatus, computer devices, media, and program products for writing path information to disk. Background Technology

[0002] In a distributed file system, path information (Backtrace) can typically be used to store the filename and directory structure corresponding to each node (Inode).

[0003] Whenever a new node is added or its path information is modified, the node needs to be recorded in the list of nodes whose path information needs to be updated. When the log is refreshed periodically or a refresh command is received, the node list can be scanned first to obtain the path information for each node. Then, the node and its corresponding path information can be recorded in the list of path information to be written to disk. Finally, each node in the path information list can be traversed, and at each node, the target function is called to write the path information for that node to disk.

[0004] Since each call to the target function actually invokes the interface of the Object Storage Device (OSD), meaning that storing the path information of each node requires a message exchange, and each message exchange involves serialization and deserialization processing, it wastes a considerable amount of time. Thus, when storing a large amount of path information, the frequent interactions put significant pressure on the network layer. Summary of the Invention

[0005] In view of this, the present invention provides a method, apparatus, computer equipment, medium and program product for storing path information on disk, so as to solve the problems of long storage time and high network pressure.

[0006] In a first aspect, the present invention provides a method for persisting path information to disk, the method being applied to a distributed file system, the method comprising:

[0007] Obtain a list of path information for disks to be dropped, wherein the list of path information includes multiple nodes and the path information for the disks to be dropped corresponding to each node;

[0008] For each node in the path information list, traverse it. For each node traversed, obtain the node number and the path information to be written to disk.

[0009] Based on the preset object partitioning parameters and the node number of the traversed node, determine the belonging object corresponding to the traversed node, and store the node number of the traversed node and the disk landing path information into the corresponding belonging object.

[0010] Determine the number of nodes included in each of the aforementioned belonging objects;

[0011] Based on the number of nodes included in the target object and a preset number threshold, determine the number of data processing rounds corresponding to the target object and the number of processing nodes corresponding to each data processing round, wherein the target object can be any object.

[0012] Based on the number of data processing rounds corresponding to the target object and the number of processing nodes corresponding to each data processing round, the disk placement operation is performed on the disk placement path information of each node included in the target object.

[0013] The method for storing path information on disk provided by this invention has the following advantages:

[0014] By assigning nodes in the path information list to corresponding belonging objects, and determining the number of data processing rounds for each belonging object based on the actual number of nodes it contains, the path information to be written to disk can be processed in multiple rounds. This way, the object storage device interface is called only once per data processing round, significantly reducing the number of calls compared to calling the interface once per path information entry. Furthermore, this reduces the number of interactions, greatly decreasing the pressure on the network layer and avoiding the time-consuming serialization and deserialization processes.

[0015] In one optional implementation, the preset object division parameter is a preset number of shifts;

[0016] The step of determining the belonging object corresponding to the traversed node based on the preset object partitioning parameters and the node number of the traversed node, and storing the node number of the traversed node and the disk placement path information into the corresponding belonging object, includes:

[0017] The node number of the traversed node is shifted to the right by the preset shift number, the empty left high-order bits after the right shift are filled with zeros, and the extra right low-order bits after the right shift are deleted to obtain the belonging object identifier corresponding to the traversed node.

[0018] Based on the attribution object identifier, determine whether there exists an attribution object corresponding to the attribution object identifier;

[0019] When it is determined that the belonging object exists, the node number of the traversed node and the path information to be placed on the disk are stored in the belonging object accordingly;

[0020] or,

[0021] When it is determined that the belonging object does not exist, the belonging object is generated based on the belonging object identifier;

[0022] The node number of the traversed node and the path information to be dropped onto the disk are stored in the corresponding belonging object.

[0023] Specifically, if the stored procedure is configured to automatically generate the owner object and its corresponding owner object identifier, and record the correspondence between the node number and the owner object identifier to ensure the accuracy of subsequent queries, then some storage space (memory and object storage device) is required to record the correspondence between the node number and the owner object identifier. This solution, whether in the stored procedure or in subsequent queries, can determine the owner object identifier using the node number and a preset shift value, thus saving storage space.

[0024] In one optional implementation, determining the number of data processing rounds corresponding to the target object and the number of processing nodes corresponding to each data processing round, based on the number of nodes included in the target object and a preset number threshold, includes:

[0025] Determine whether the number of nodes included in the target belonging object is greater than the preset number threshold;

[0026] When it is determined that the number of nodes included in the target object is greater than the preset number threshold, a modulo operation is performed based on the number of nodes included in the target object and the preset number threshold to obtain the number of data processing rounds corresponding to the target object and the number of processing nodes corresponding to each data processing round.

[0027] or,

[0028] When it is determined that the number of nodes included in the target object is less than or equal to the preset number threshold, the number of data processing rounds corresponding to the target object is determined to be one, and the number of nodes included in the target object is determined as the number of processing nodes in that data processing round.

[0029] Specifically, if there are too many path information entries in the belonging object, and the object storage device interface is still called once for each belonging object, it may cause excessive network pressure and data congestion. Therefore, this problem can be solved by limiting the amount of data processed in each data processing round by setting a preset quantity threshold.

[0030] In one optional implementation, the step of performing a disk write operation on the disk path information of each node included in the target object based on the number of data processing rounds corresponding to the target object and the number of processing nodes corresponding to each data processing round includes:

[0031] Based on the number of data processing rounds corresponding to the target object and the number of processing nodes corresponding to each data processing round, the nodes included in the target object are grouped to obtain multiple groups corresponding to the target object. The number of nodes corresponding to each group is less than or equal to the preset number threshold, and each group corresponds to one data processing round.

[0032] In the current data processing round, obtain the disk-reading path information of each node included in the target group, wherein the target group is the group corresponding to the current data processing round;

[0033] Perform disk placement operations on the disk placement path information of all nodes included in the target group.

[0034] Specifically, if there are too many path information entries in the belonging object, and the object storage device interface is still called once for each belonging object, it may cause excessive network pressure and data congestion. Therefore, this problem can be solved by limiting the amount of data processed in each data processing round by setting a preset quantity threshold.

[0035] In an optional implementation, before obtaining the list of path information for disk placement, the method further includes:

[0036] Get the list of nodes whose path information is to be dumped to disk;

[0037] Traverse each node in the node list to obtain the disk path information corresponding to each node;

[0038] The path information list is generated based on each node and the path information to be dropped onto the disk corresponding to each node.

[0039] In an optional implementation, the method further includes:

[0040] When a path information query instruction for a target node is obtained, the node number of the target node is extracted from the path information query instruction;

[0041] Based on the node number of the target node and the preset number of moves, the target belonging object identifier corresponding to the target node is obtained;

[0042] Based on the target belonging object identifier, determine the target belonging object corresponding to the target belonging object identifier;

[0043] Based on the node number of the target node, the path information of the target node is obtained from the target's home object.

[0044] Specifically, in related technologies, querying the path information of a target node requires traversing numerous path entries stored in memory or object storage devices to find the target node's path information. This solution, however, first locates the target's owning object and then queries that object for the target node's path information. Since the number of owning objects is relatively small compared to the number of path entries, and the path information contained within an owning object is also relatively limited, this solution significantly reduces the number of traversals required during the query process, thereby improving query efficiency.

[0045] Secondly, the present invention provides a path information disk persistence device, the device being applied to a distributed file system, the device comprising:

[0046] The acquisition module is used to acquire a list of path information to be dropped onto the disk, wherein the path information list includes multiple nodes and the path information to be dropped onto the disk corresponding to each node; the module iterates over each node in the path information list, and for each node traversed, it acquires the node number of the traversed node and the path information to be dropped onto the disk.

[0047] The determination module is used to determine the belonging object corresponding to the traversed node based on the preset object division parameters and the node number of the traversed node, and to store the node number of the traversed node and the disk landing path information into the corresponding belonging object; determine the number of nodes included in each belonging object; and determine the number of data processing rounds corresponding to the target belonging object and the number of processing nodes corresponding to each data processing round based on the number of nodes included in the target belonging object and the preset number threshold, wherein the target belonging object is any belonging object;

[0048] The disk placement module is used to perform disk placement operations on the disk placement path information of each node included in the target object, based on the number of data processing rounds corresponding to the target object and the number of processing nodes corresponding to each data processing round.

[0049] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the path information disk writing method of the first aspect or any corresponding embodiment described above.

[0050] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions, which are used to cause a computer to execute the path information disk writing method of the first aspect or any corresponding embodiment described above.

[0051] Fifthly, the present invention provides a computer program product, including computer instructions, which are used to cause a computer to execute the path information disk writing method of the first aspect or any corresponding embodiment described above. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of the present invention, the drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0053] Figure 1 This is a flowchart illustrating the method for storing path information on disk according to an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of path information according to an embodiment of the present invention;

[0055] Figure 3 This is a timing diagram of the data write-to-disk operation according to an embodiment of the present invention;

[0056] Figure 4 This is a flowchart illustrating another method for storing path information on disk according to an embodiment of the present invention;

[0057] Figure 5 This is a flowchart illustrating another method for storing path information on disk according to an embodiment of the present invention;

[0058] Figure 6 This is a flowchart illustrating the path information query method according to an embodiment of the present invention;

[0059] Figure 7 This is a structural block diagram of a path information disk placement device according to an embodiment of the present invention;

[0060] Figure 8 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] In a distributed file system, the Metadata Data Server (MDS) provides services for metadata operations, while the Object Storage Device (OSD) stores various types of data. Additionally, the distributed file system uses backtrace information to store the filename and directory structure corresponding to each inode. After adding a node or modifying its path information, the new or updated path information needs to be written to disk.

[0063] This invention provides a method for persisting path information to disk. By dividing the path information into objects and grouping them, network pressure can be reduced and the disk persistence time can be shortened.

[0064] According to an embodiment of the present invention, a method for storing path information on a disk is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0065] This embodiment provides a method for persisting path information to disk, applied to a distributed file system. The distributed file system can be configured on a computer cluster, which may include multiple computer nodes, each corresponding to a computer device, such as a server. This solution can be executed by the computer cluster. Figure 1 This is a flowchart of a method for storing path information on disk according to an embodiment of the present invention, including the following steps:

[0066] Step S101: Obtain the list of path information for disks to be dropped.

[0067] The path information list includes multiple nodes and the corresponding disk placement path information for each node. For example, the disk placement path information can be as follows: Figure 2As shown, "ino" represents the node number of the current node, "ancestors" represents path information, "dirino" represents the node number of the parent directory of the file or directory, "dname" represents the name of the parent directory of the file or directory, "dname" represents the version of the parent directory of the file or directory, and "share" is provided by the file system and is not visible to the user. Figure 2 We can determine that the path of the file corresponding to the current node is "file / dir1 / dir2 / file".

[0068] Specifically, when the distributed file system receives the first trigger command, it can first unlock the MDS lock control, and then obtain the corresponding list of paths to be written to disk (which can be called "backtrace_list"). The first trigger command can be a periodically triggered command or a command generated based on user operations.

[0069] In some optional implementations, before obtaining the list of path information to be disposed of, the method further includes:

[0070] Step a1: Obtain the list of nodes for the path information to be dumped to disk.

[0071] Step a2: Traverse each node in the node list to obtain the disk path information corresponding to each node.

[0072] Step a3: Generate a path information list based on each node and the path information to be dropped onto the disk corresponding to each node.

[0073] Specifically, whenever a distributed file system receives a creation or modification command, it can extract the inode number from the command and add it to the inode list (which can be called "dirty_inode_list") of the path information to be written to disk. For example, if it needs to... Figure 2 When “dir2” is changed to “dir3”, the generated modification command includes the node number “6597069796658”; if a new “fileX” needs to be created in “dir2”, the path information of “fileX” needs to be added, and the generated creation command includes the node number corresponding to “fileX”.

[0074] In this way, upon detecting the second trigger command, a list of nodes for the disk placement path information can be obtained first. Each node in this list is then traversed, and for each node reached, its updated information is retrieved by tracing upwards from that node, thus obtaining the disk placement path information for that node. Furthermore, the node number and disk placement path information of that node can be added to the path information list. The second trigger command can be a periodically triggered command or a command generated based on user actions.

[0075] Step S102: Traverse each node in the path information list. For each node traversed, obtain the node number and the path information to be written to disk.

[0076] Specifically, the distributed file system can traverse each node in the path information list, and obtain the node number and the path information to be written to disk for each node traversed.

[0077] Step S103: Based on the preset object partitioning parameters and the node number of the traversed node, determine the belonging object corresponding to the traversed node, and store the node number of the traversed node and the path information to be dropped onto the disk into the corresponding belonging object.

[0078] The preset object division parameter can be the preset number of belonging objects.

[0079] Specifically, the distributed file system can generate a number of ownership objects equal to the preset number of ownership objects. Based on this preset number of ownership objects, it divides all nodes in the path information list into multiple node groups, where the number of node groups equals the preset number of ownership objects. Then, the distributed file system can establish a correspondence between node groups and ownership objects. Further, based on the node number of the traversed node, it can determine the target node group to which the traversed node belongs, and then, based on the correspondence between node groups and ownership objects, it can determine the target ownership object corresponding to the target node group. Finally, the node number of the traversed node and the path information to be written to disk are stored in the target ownership object.

[0080] Step S104: Determine the number of nodes included in each belonging object.

[0081] Specifically, for each owned object, the distributed file system can count the nodes in that owned object to obtain the corresponding number of nodes in that owned object.

[0082] Step S105: Based on the number of nodes included in the target object and a preset number threshold, determine the number of data processing rounds corresponding to the target object and the number of processing nodes corresponding to each data processing round.

[0083] The target object can be any object.

[0084] Specifically, the distributed file system can determine the number of data processing rounds corresponding to the target object using the following formula: L1 = Ceil(S / M), where L1 is the number of data processing rounds, S is the number of nodes included in the target object, and M is a preset threshold. The distributed file system can use [S-(L1-1)*M] as the number of processing nodes in any data processing round, and the preset threshold as the number of processing nodes corresponding to each data processing round in other data processing rounds.

[0085] Step S106: Based on the number of data processing rounds corresponding to the target object and the number of processing nodes corresponding to each data processing round, perform disk write operations on the disk write path information of each node included in the target object.

[0086] Specifically, in each data processing round, the distributed file system obtains the same number of disk-writing path information as the number of processing nodes from the target object based on the number of processing nodes corresponding to that data processing round, and performs the disk-writing operation.

[0087] like Figure 3 As shown, the above operations are specifically executed by the MDS in the distributed file system. Steps S102 to S106 require the Finisher thread. Correspondingly, the specific processing in each data processing round by the MDS can be as follows: In each data processing round, the MDS can call the OSD's interface and transmit the disk-writing path information corresponding to that round to the OSD. The OSD can then perform a write operation based on the received disk-writing path information to complete the disk-writing operation. After completing each round of disk-writing operations, the OSD can send a reply message to the MDS. Upon receiving the reply message, the MDS begins the next data processing round.

[0088] In some optional implementations, the distributed file system can also obtain the storage space occupied by the path information to be written to disk for each node. Furthermore, it can calculate the storage space occupied by all path information corresponding to each data processing round. Additionally, the distributed file system can obtain the current network transmission rate and determine the corresponding target maximum data transfer volume based on that rate. In this way, the distributed file system can determine whether the storage space occupied by all path information corresponding to the current data processing round is less than or equal to the target maximum data transfer volume. If so, subsequent disk writing operations are performed. If not, the disk writing operation for the current data processing round can be paused, and the next round can begin, with all path information corresponding to the current data processing round processed last, and so on. This ensures efficient disk writing operations within the limits of network pressure, avoiding network congestion caused by excessive data volume.

[0089] The path information writing method provided in this embodiment divides the nodes in the path information list into corresponding belonging objects. For each belonging object, the number of data processing rounds is determined based on the actual number of nodes included in the belonging object. This allows for multiple rounds of writing the path information to be written to disk within each belonging object. In this way, the object storage device interface is called only once per data processing round, significantly reducing the number of calls compared to calling the interface once per path information written to disk. Furthermore, this reduces the number of interactions, greatly reducing the pressure on the network layer and avoiding the time-consuming problems of serialization and deserialization processing.

[0090] This embodiment provides a method for persisting path information to disk, applied to a distributed file system. The distributed file system can be configured on a computer cluster, which may include multiple computer nodes, each corresponding to a computer device, such as a server. This solution can be executed by the computer cluster. Figure 4 This is a flowchart of a method for storing path information on disk according to an embodiment of the present invention, including the following steps:

[0091] Step S401: Obtain the list of path information for disks to be dropped.

[0092] The path information list includes multiple nodes, as well as the path information to be dropped onto the disk corresponding to each node.

[0093] Step 402: Traverse each node in the path information list. For each node traversed, obtain the node number and the path information to be written to disk.

[0094] The specific processing of steps S401 to S402 is similar to that of steps S101 to S102, and will not be repeated here.

[0095] Step S403: Based on the preset object partitioning parameters and the node number of the traversed node, determine the belonging object corresponding to the traversed node, and store the node number of the traversed node and the path information to be dropped onto the disk into the corresponding belonging object.

[0096] Specifically, when the preset object division parameter is a preset number of shifts, step S303 includes:

[0097] Step S4031: Shift the node number of the traversed node to the right by a preset number of positions, fill the empty left high-order bits after the right shift with zeros, and delete the extra right low-order bits after the right shift to obtain the belonging object identifier corresponding to the traversed node.

[0098] The preset number of moves can be set according to actual business needs, representing the number of disk path information that can be stored in a given object.

[0099] For example, if the preset shift number is "16", and the node number of the traversed node is "30000000023", the resulting right shift by 16 bits will give the owner object identifier "000003000000", which is a hexadecimal number. Correspondingly, the owner object with the identifier "000003000000" can store 2... 16 Path information.

[0100] Step S4032: Determine whether there is a corresponding home object based on the home object identifier.

[0101] Step S4033: When it is determined that there is an owner object, the node number of the traversed node and the path information to be dropped to disk are stored in the owner object.

[0102] Step S4034: When it is determined that there is no belonging object, a belonging object is generated according to the belonging object identifier.

[0103] Step S4035: Store the node number of the traversed node and the path information to be dropped to disk in the corresponding belonging object.

[0104] Specifically, a distributed storage system can determine a list of owned objects (which can be called "obj_list") based on the obtained owned object identifiers. <obj_id,set <key>Does the ">" indicate whether there is a corresponding belonging object?

[0105] If so, the node number of the traversed node can be used as a subkey value (key) in the object to which it belongs, and the path information to be written to disk can be stored in the key corresponding to the subkey value.

[0106] If not, the owner object identifier can be added as a new key value to the owner object list, and the node number of the traversed node can be used as a subkey value in the newly created owner object. The path information to be dropped onto the disk can then be stored in the key corresponding to this subkey value.

[0107] For example, the final list of belonging objects can be as follows:

[0108] 000003000000--obj_id

[0109] 30000000023--key

[0110] 30000000708--key

[0111] 30000000abc--key ......

[0113] 000006000000--obj_id

[0114] 60000000023--key

[0115] 60000000503--key

[0116] 6000000060c--key ......

[0118] Among them, "000003000000" and "000006000000" are the identifiers of the objects to which the objects belong; "30000000023", "30000000708", and "30000000abc" are the node numbers of the nodes belonging to the object to which "000003000000" belongs; and "60000000023", "60000000503", and "6000000060c" are the node numbers of the nodes belonging to the object to which "000006000000" belongs.

[0119] Step S404: Determine the number of nodes included in each belonging object.

[0120] Step S405: Based on the number of nodes included in the target object and a preset number threshold, determine the number of data processing rounds corresponding to the target object and the number of processing nodes corresponding to each data processing round.

[0121] The target object can be any object.

[0122] Step S406: Based on the number of data processing rounds corresponding to the target object and the number of processing nodes corresponding to each data processing round, perform disk write-to-disk operation on the disk path information of each node included in the target object.

[0123] The specific processing of steps S404 to S406 is similar to that of steps S104 to S106, and will not be repeated here.

[0124] The path information storage method provided in this embodiment requires a portion of storage space (memory and object storage device) to record the mapping between node numbers and the corresponding owner object identifier during the storage process to ensure the accuracy of subsequent queries. However, this solution can determine the owner object identifier using the node number and a preset number of moves, both during the storage process and in subsequent queries, thus saving storage space.

[0125] This embodiment provides a method for persisting path information to disk, applied to a distributed file system. The distributed file system can be configured on a computer cluster, which may include multiple computer nodes, each corresponding to a computer device, such as a server. This solution can be executed by the computer cluster. Figure 5 This is a flowchart of a method for storing path information on disk according to an embodiment of the present invention, including the following steps:

[0126] Step S501: Obtain the list of path information for disks to be dropped.

[0127] The path information list includes multiple nodes, as well as the path information to be dropped onto the disk corresponding to each node.

[0128] Step S502: Traverse each node in the path information list. For each node traversed, obtain the node number and the path information to be written to disk.

[0129] Step S503: Based on the preset object partitioning parameters and the node number of the traversed node, determine the belonging object corresponding to the traversed node, and store the node number of the traversed node and the path information to be dropped onto the disk into the corresponding belonging object.

[0130] Step S504: Determine the number of nodes included in each belonging object.

[0131] The specific processing of steps S501 to S504 is similar to that of steps S101 to S104 (or steps S401 to S404), and will not be repeated here.

[0132] Step S505: Based on the number of nodes included in the target object and a preset number threshold, determine the number of data processing rounds corresponding to the target object and the number of processing nodes corresponding to each data processing round.

[0133] The target object can be any object.

[0134] Specifically, step S505 includes:

[0135] Step S5051: Determine whether the number of nodes included in the target object is greater than a preset threshold.

[0136] Step S5052: When it is determined that the number of nodes included in the target object is greater than the preset number threshold, a modulo operation is performed based on the number of nodes included in the target object and the preset number threshold to obtain the number of data processing rounds corresponding to the target object and the number of processing nodes corresponding to each data processing round.

[0137] Step S5053: When it is determined that the number of nodes included in the target object is less than or equal to a preset number threshold, the number of data processing rounds corresponding to the target object is determined to be one, and the number of nodes included in the target object is determined as the number of processing nodes in that data processing round.

[0138] The preset quantity threshold can be 500.

[0139] Specifically, the distributed file system can first determine whether the number of nodes included in the target object is greater than a preset threshold.

[0140] If so, it indicates that the number of pending disk paths in the target object is large. If the disk write operation is performed on all pending disk paths in the target object simultaneously, message congestion is highly likely to occur. Therefore, a modulo operation can be performed based on the number of nodes included in the target object and a preset threshold to obtain the target quotient and target remainder. The target quotient is then incremented by one to determine the number of data processing rounds for the target object. Specifically, the number of processing nodes for one data processing round is the target remainder, while the number of processing nodes for other data processing rounds is the preset threshold.

[0141] If the answer is no, it indicates that the number of pending disk write path information in the target attribution object meets the actual requirements. The data processing round count of the target attribution object can be determined as one, and the number of nodes included in the target attribution object can be determined as the number of processing nodes for this data processing round.

[0142] Step S506: Perform disk write operations on the pending disk write path information of each node included in the target attribution object according to the data processing round count corresponding to the target attribution object and the number of processing nodes corresponding to each data processing round.

[0143] Specifically, the above step S506 includes:

[0144] Step S5061: Group the nodes included in the target attribution object according to the data processing round count corresponding to the target attribution object and the number of processing nodes corresponding to each data processing round, to obtain multiple groups corresponding to the target attribution object.

[0145] Among them, the number of nodes corresponding to each group is less than or equal to a preset quantity threshold, and each group corresponds to a data processing round.

[0146] Specifically, the distributed file system can traverse the nodes included in the target attribution object in the storage order. When the preset quantity threshold of nodes is traversed, the traversed nodes are divided into a group, and then continue to traverse backward. After traversing the preset quantity threshold of nodes, a division operation is performed again, and so on. When the number of groups obtained by division is one less than the data processing round count, the traversal operation stops, and the remaining untraversed nodes are divided into a group. In this way, the grouping operation of the target attribution object can be completed to obtain multiple groups.

[0147] For example, there are 1534 pieces of pending disk write path information that need to be written to disk under the "000003000000" attribution object, and 644 pieces of pending disk write path information that need to be written to disk under the "000006000000" attribution object. The list class of the attribution objects after grouping can be as follows:

[0148] 00000, <batch_op0, <key1...key500>>>

[0149] 00000, <batch_op1, <key50's...key1000>>>

[0150] 00000, <batch_op2, <key1001..key1500>>>

[0151] 00000, <batch_op3, <key1501..key1534>>>

[0152] 00000, <batch_op0, <key1...key500>>>

[0153] 00000, <batch_op1, <key501..key644>>>

[0154] Among them, the object corresponding to "000003000000" has 4 data processing rounds, and the number of processing nodes corresponding to any data processing round from "batch_op0" to "batch_op2" is 500.

[0155] Step S5062: In the current data processing round, obtain the disk-down path information of each node included in the target group.

[0156] Among them, the target group is the group corresponding to the current data processing round.

[0157] For example, the object corresponding to "000003000000" has 4 data processing rounds. In the data processing round of "batch_op0", the disk-down path information corresponding to <key1...key500> can be obtained. In the data processing round of "batch_op1", the disk-down path information corresponding to <key501...key1000> can be obtained. In the data processing round of "batch_op2", the disk-down path information corresponding to <key1001...key1500> can be obtained. In the data processing round of "batch_op3", the disk-down path information corresponding to <key1501...key1534> can be obtained.

[0158] Step S5063: Perform a disk-down operation on the disk-down path information of all nodes included in the target group.

[0159] The specific processing of Step S5063 is similar to the specific processing of Step 106, which will not be elaborated here.

[0160] The path information persistence method provided in this embodiment addresses the issue in related technologies where one path information corresponds to one owner object, and each owner object requires two interactions (request and response). In the example given above, with a total of 2178 path information entries, persistence requires 2178*2 interactions. However, with this solution, there are 2 owner objects and 6 groups, requiring only 6*2 interactions. Clearly, the number of interactions is significantly reduced, as are the serialization and deserialization processes. This greatly shortens the persistence operation time and reduces network pressure, further improving the efficiency of persistence. Furthermore, if the number of path information entries in an owner object is too large, using one owner object to call the object storage device interface once may lead to excessive network pressure and data congestion. Therefore, limiting the amount of data processed in each data processing round by setting a preset threshold can solve this problem.

[0161] This embodiment provides a path information query method applied to a distributed file system. The distributed file system can be configured on a computer cluster, which may include multiple computer nodes, each corresponding to a computer device, such as a server. This solution can be executed by the computer cluster. Figure 6 This is a flowchart of a path information query method according to an embodiment of the present invention, including the following steps:

[0162] Step S601: When the path information query instruction for the target node is obtained, the node number of the target node is extracted from the path information query instruction.

[0163] Specifically, users can perform query operations on the client side. In this way, when the distributed file system obtains the path information query command for the target node, it can parse the path information query command and obtain the node number of the target node from it.

[0164] Step S602: Based on the node number of the target node and the preset number of moves, obtain the target belonging object identifier corresponding to the target node.

[0165] Specifically, the distributed file system shifts the node number of the target node to the right by a preset number of bits, fills the empty left high-order bits with zeros, and deletes the extra right low-order bits to obtain the target's owner object identifier.

[0166] Step S603: Determine the target belonging object corresponding to the target belonging object identifier based on the target belonging object identifier.

[0167] Specifically, the distributed file system can determine the target owner object from the list of owner objects based on the target owner object identifier.

[0168] Step S604: Obtain the path information of the target node from the target's owning object based on the node number of the target node.

[0169] Specifically, the distributed file system can determine the target node from the target's owning object based on the target node's node number, and then obtain the target node's path information and send it back to the client.

[0170] For example, the list of belonging objects can be as follows:

[0171] 000003000000--obj_id

[0172] 30000000023--key

[0173] 30000000708--key

[0174] 30000000abc--key ......

[0176] 000006000000--obj_id

[0177] 60000000023--key

[0178] 60000000603--key

[0179] 6000000060c--key ......

[0181] When the node number of the target node is "30000000023" and the preset number of moves is "16", the target belonging object identifier is "000003000000". The distributed file system can first determine the target belonging object based on "000003000000", and then obtain the path information of the target node in the target belonging object based on "30000000023".

[0182] The path information query method provided in this embodiment addresses a common problem in related technologies where querying the path information of a target node requires traversing numerous path entries stored in memory or object storage devices. This solution first locates the target's parent object and then queries that object for the target node's path information. Since the number of parent objects is relatively small compared to the number of path entries, and the path information contained within a single parent object is also relatively limited, this solution significantly reduces the number of traversals required during the query process, thereby improving query efficiency.

[0183] This embodiment also provides a path information disk-based device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware implementations, or a combination of software and hardware, are also possible and contemplated.

[0184] This embodiment provides a path information disk storage device, such as... Figure 7 As shown, it includes:

[0185] The acquisition module 701 is used to acquire a list of path information to be dropped onto the disk. The path information list includes multiple nodes and the path information to be dropped onto the disk corresponding to each node. Each node in the path information list is traversed, and the node number and path information to be dropped onto the disk are acquired for each node traversed.

[0186] The determination module 702 is used to determine the belonging object corresponding to the traversed node based on the preset object division parameters and the node number of the traversed node, and to store the node number of the traversed node and the path information to be dropped onto the disk into the corresponding belonging object; determine the number of nodes included in each belonging object; and determine the number of data processing rounds corresponding to the target belonging object and the number of processing nodes corresponding to each data processing round based on the number of nodes included in the target belonging object and the preset number threshold, wherein the target belonging object is any belonging object;

[0187] The disk placement module 703 is used to perform disk placement operations on the disk placement path information of each node included in the target object based on the number of data processing rounds corresponding to the target object and the number of processing nodes corresponding to each data processing round.

[0188] In some optional implementations, the preset object partitioning parameter is a preset number of shifts;

[0189] Determine module 702, used for:

[0190] Shift the node number of the traversed node to the right by a preset number of positions, fill the empty left high-order bits after the right shift with zeros, and delete the extra right low-order bits after the right shift to obtain the belonging object identifier corresponding to the traversed node.

[0191] Based on the attribution object identifier, determine whether there is an attribution object corresponding to the attribution object identifier;

[0192] When it is determined that an object belongs to the node, the node number of the traversed node and the path information to be dropped to the disk are stored in the object belonging to the node.

[0193] or,

[0194] When it is determined that no belonging object exists, a belonging object is generated based on the belonging object identifier;

[0195] The node number of each traversed node and the path information to be written to disk are stored in the corresponding belonging object.

[0196] In some alternative implementations, the determining module 702 is used for:

[0197] Determine whether the number of nodes included in the target object is greater than a preset threshold.

[0198] When the number of nodes included in the target object is greater than the preset number threshold, a modulo operation is performed based on the number of nodes included in the target object and the preset number threshold to obtain the number of data processing rounds corresponding to the target object and the number of processing nodes corresponding to each data processing round.

[0199] or,

[0200] When the number of nodes included in the target object is less than or equal to a preset threshold, the number of data processing rounds corresponding to the target object is set to one, and the number of nodes included in the target object is set as the number of processing nodes in that data processing round.

[0201] In some alternative implementations, the disk placement module 703 is used for:

[0202] Based on the number of data processing rounds corresponding to the target object and the number of processing nodes corresponding to each data processing round, the nodes included in the target object are grouped to obtain multiple groups corresponding to the target object. The number of nodes corresponding to each group is less than or equal to a preset number threshold, and each group corresponds to a data processing round.

[0203] In the current data processing round, obtain the disk path information of each node included in the target group, where the target group is the group corresponding to the current data processing round;

[0204] Perform disk write operations on the disk path information of all nodes included in the target group.

[0205] In some alternative implementations, the acquisition module 701 is further configured to:

[0206] Get the list of nodes whose path information is to be dumped to disk;

[0207] Iterate through each node in the node list to obtain the disk path information corresponding to each node;

[0208] A path information list is generated based on each node and the path information to be dropped onto the disk corresponding to each node.

[0209] In some alternative embodiments, the device further includes a query module 704 for:

[0210] When a path information query instruction for the target node is obtained, the node number of the target node is extracted from the path information query instruction.

[0211] Based on the node number of the target node and the preset number of moves, obtain the target belonging object identifier corresponding to the target node;

[0212] Based on the target object identifier, determine the target object corresponding to the target object identifier;

[0213] Based on the node number of the target node, obtain the path information of the target node from the target's owning object.

[0214] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0215] In this embodiment, the path information disk placement device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit), a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0216] This invention also provides a computer device having the above-described features. Figure 7 The path information is displayed on the disk by the device.

[0217] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 8 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 8 Take a processor 10 as an example.

[0218] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0219] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0220] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0221] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0222] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0223] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0224] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0225] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.< / key>

Claims

1. A method for storing path information on disk, characterized in that, The method is applied to a distributed file system, and the method includes: Obtain a list of path information for disks to be dropped, wherein the list of path information includes multiple nodes and the path information for the disks to be dropped corresponding to each node; For each node in the path information list, traverse it. For each node traversed, obtain the node number and the path information to be written to disk. Based on the preset object partitioning parameters and the node number of the traversed node, determine the belonging object corresponding to the traversed node, and store the node number of the traversed node and the disk landing path information into the corresponding belonging object. Determine the number of nodes included in each of the aforementioned belonging objects; Based on the number of nodes included in the target object and a preset number threshold, determine the number of data processing rounds corresponding to the target object and the number of processing nodes corresponding to each data processing round, wherein the target object can be any object. Based on the number of data processing rounds corresponding to the target object and the number of processing nodes corresponding to each data processing round, the disk-to-disk path information of each node included in the target object is disk-to-disk processed. Wherein, the preset object division parameter is a preset number of moves; the step of determining the belonging object corresponding to the traversed node based on the preset object division parameter and the node number of the traversed node, and storing the node number of the traversed node and the path information to be dropped onto the disk in the corresponding belonging object, includes: The node number of the traversed node is shifted to the right by the preset shift number, the empty left high-order bits after the right shift are filled with zeros, and the extra right low-order bits after the right shift are deleted to obtain the belonging object identifier corresponding to the traversed node. Based on the attribution object identifier, determine whether there exists an attribution object corresponding to the attribution object identifier; When it is determined that the belonging object exists, the node number of the traversed node and the path information to be placed on the disk are stored in the belonging object accordingly; or, When it is determined that the belonging object does not exist, the belonging object is generated based on the belonging object identifier; The node number of the traversed node and the path information to be dropped onto the disk are stored in the corresponding belonging object.

2. The method according to claim 1, characterized in that, The step of determining the number of data processing rounds corresponding to the target object and the number of processing nodes corresponding to each data processing round, based on the number of nodes included in the target object and a preset number threshold, includes: Determine whether the number of nodes included in the target belonging object is greater than the preset number threshold; When it is determined that the number of nodes included in the target object is greater than the preset number threshold, a modulo operation is performed based on the number of nodes included in the target object and the preset number threshold to obtain the number of data processing rounds corresponding to the target object and the number of processing nodes corresponding to each data processing round. or, When it is determined that the number of nodes included in the target object is less than or equal to the preset number threshold, the number of data processing rounds corresponding to the target object is determined to be one, and the number of nodes included in the target object is determined as the number of processing nodes in that data processing round.

3. The method according to claim 1 or 2, characterized in that, The step of performing disk write-to-disk operation on the disk path information of each node included in the target object, based on the number of data processing rounds corresponding to the target object and the number of processing nodes corresponding to each data processing round, includes: Based on the number of data processing rounds corresponding to the target object and the number of processing nodes corresponding to each data processing round, the nodes included in the target object are grouped to obtain multiple groups corresponding to the target object. The number of nodes corresponding to each group is less than or equal to the preset number threshold, and each group corresponds to one data processing round. In the current data processing round, obtain the disk-reading path information of each node included in the target group, wherein the target group is the group corresponding to the current data processing round; Perform disk placement operations on the disk placement path information of all nodes included in the target group.

4. The method according to claim 1 or 2, characterized in that, Before obtaining the list of path information to be disposed of, the method further includes: Get the list of nodes whose path information is to be dumped to disk; Traverse each node in the node list to obtain the disk path information corresponding to each node; The path information list is generated based on each node and the path information to be dropped onto the disk corresponding to each node.

5. The method according to claim 1, characterized in that, The method further includes: When a path information query instruction for a target node is obtained, the node number of the target node is extracted from the path information query instruction; Based on the node number of the target node and the preset number of moves, the target belonging object identifier corresponding to the target node is obtained; Based on the target belonging object identifier, determine the target belonging object corresponding to the target belonging object identifier; Based on the node number of the target node, the path information of the target node is obtained from the target's home object.

6. A path information disk placement device, characterized in that, The apparatus is used in a distributed file system, and the apparatus includes: The acquisition module is used to acquire a list of path information to be dropped onto the disk, wherein the path information list includes multiple nodes and the path information to be dropped onto the disk corresponding to each node; the module iterates over each node in the path information list, and for each node traversed, it acquires the node number of the traversed node and the path information to be dropped onto the disk. The determination module is used to determine the belonging object corresponding to the traversed node based on the preset object division parameters and the node number of the traversed node, and to store the node number of the traversed node and the disk landing path information into the corresponding belonging object; determine the number of nodes included in each belonging object; and determine the number of data processing rounds corresponding to the target belonging object and the number of processing nodes corresponding to each data processing round based on the number of nodes included in the target belonging object and the preset number threshold, wherein the target belonging object is any belonging object; The disk placement module is used to perform disk placement operations on the disk placement path information of each node included in the target object based on the number of data processing rounds corresponding to the target object and the number of processing nodes corresponding to each data processing round. Wherein, the preset object division parameter is a preset number of moves; the step of determining the belonging object corresponding to the traversed node based on the preset object division parameter and the node number of the traversed node, and storing the node number of the traversed node and the path information to be dropped onto the disk in the corresponding belonging object, includes: The node number of the traversed node is shifted to the right by the preset shift number, the empty left high-order bits after the right shift are filled with zeros, and the extra right low-order bits after the right shift are deleted to obtain the belonging object identifier corresponding to the traversed node. Based on the attribution object identifier, determine whether there exists an attribution object corresponding to the attribution object identifier; When it is determined that the belonging object exists, the node number of the traversed node and the path information to be placed on the disk are stored in the belonging object accordingly; or, When it is determined that the belonging object does not exist, the belonging object is generated based on the belonging object identifier; The node number of the traversed node and the path information to be dropped onto the disk are stored in the corresponding belonging object.

7. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the path information disk writing method according to any one of claims 1 to 5 by executing the computer instructions.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the path information disk writing method according to any one of claims 1 to 5.

9. A computer program product, characterized in that, Includes computer instructions, which are used to cause a computer to execute the path information disk writing method according to any one of claims 1 to 5.

Images