Flow control method and device, computer equipment and storage medium

Abstract

The invention discloses a flow control method and device, computer equipment and a storage medium. The method comprises the following steps: constructing an HDFS cluster network topology model; based on the HDFS cluster network topology model, determining a placement mode of data copies of data blocks in an HDFS cluster; and determining a change mode of the data copies of the data blocks in the HDFS cluster based on the placement mode of the data copies of the data blocks in the HDFS cluster, so as to control the flow among the M data centers of the HDFS cluster through the change mode of the data copies of the data blocks in the HDFS cluster. According to the method, the placement mode of the data copies of the data blocks is configured through the constructed HDFS cluster network topology model, so that the change mode of the data copies of the data blocks is determined based on the placement mode of the data copies of the data blocks, generation of cross-data center traffic is effectively controlled, and the fault tolerance rate and stability of an HDFS cluster are improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a flow control method, device, computer equipment and storage medium. Background Art

[0002] HDFS (Hadoop Distributed File System) is a highly fault-tolerant, high-throughput distributed file system designed for the storage of large-scale data sets. It is mainly used to store massive amounts of data. The HDFS cluster itself provides the important feature of rack awareness, the main purpose of which is to optimize data reading and storage, especially in a multi-rack environment.

[0003] Currently, the HDFS cluster optimizes HDFS cluster data reading and storage by understanding the physical location of data blocks and placing data copies of data blocks on multiple racks based on the data copies of data blocks and the rack distribution of nodes.

[0004] However, a method for implementing traffic control between data centers when data centers are introduced into the HDFS network topology structure (model) has not yet been proposed. Summary of the invention

[0005] The main purpose of this application is to provide a flow control method, device, computer equipment and storage medium, which solves the problem of achieving flow control between data centers when the data center is introduced into the HDFS network topology structure (model), effectively controls the generation of cross-data center flow, and improves the fault tolerance and stability of the HDFS cluster.

[0006] In order to achieve the above objectives, in a first aspect, the present application provides a flow control method, comprising:

[0007] Build the HDFS cluster network topology model;

[0008] Based on the HDFS cluster network topology model, determine the placement of data replicas of data blocks in the HDFS cluster;

[0009] Based on the placement method of the data copies of the data blocks in the HDFS cluster, determine the change method of the data copies of the data blocks in the HDFS cluster, so as to control the traffic between M data centers of the HDFS cluster through the change method of the data copies of the data blocks in the HDFS cluster, where M is an integer greater than 1.

[0010] In one embodiment, building a HDFS cluster network topology model includes:

[0011] Construct the root node of the HDFS cluster network topology model;

[0012] Connect the root node to each of the M data centers;

[0013] N racks are configured for each data center, and each data center is connected to P storage nodes through each rack in the N racks, where N and P are both integers greater than 1;

[0014] The network topology model consisting of the root node, M data centers, M*N racks, and M*N*P storage nodes is used as the HDFS cluster network topology model.

[0015] In one embodiment, based on the HDFS cluster network topology model, determining the placement of data replicas of data blocks in the HDFS cluster includes:

[0016] When there are three data copies of a data block in the HDFS cluster, obtain the first data copy, the second data copy, and the third data copy of the three data copies;

[0017] Based on the HDFS cluster network topology model, determine the storage node where the first data copy is placed;

[0018] Determine the storage node where the second data copy is placed based on the data center where the storage node where the first data copy is placed is located;

[0019] Based on the data center where the storage node where the first data copy is placed is located and the data center where the storage node where the second data copy is placed is located, determine the storage node where the third data copy is placed.

[0020] In one embodiment, based on the HDFS cluster network topology model, determining the storage node where the first data copy is placed includes:

[0021] Determine whether the client is in a target storage node, where the target storage node is the storage node where the client is located;

[0022] If the client is at the target storage node, the first data copy is preferentially placed at the target storage node;

[0023] If the client is not in the target storage node, the first data copy is placed in any storage node among the M*N*P storage nodes.

[0024] In one embodiment, determining a storage node for placing a second data copy based on a data center where a storage node for placing a first data copy is located includes:

[0025] A storage node in a data center different from the data center where the storage node where the first data copy is placed is selected from the M data centers as the storage node for placing the second data copy.

[0026] In one embodiment, determining a storage node for placing a third data copy based on a data center where a storage node for placing a first data copy is located and a data center where a storage node for placing a second data copy is located includes:

[0027] If the data center where the storage node for the first data copy is located is the same as the data center where the storage node for the second data copy is located, a storage node in a data center different from the data center where the storage node for the first data copy is located is selected from the M data centers as the storage node for the third data copy;

[0028] If the data center where the storage node for the first data copy is located is different from the data center where the storage node for the second data copy is located, a storage node in a different rack of the same data center as the data center where the storage node for the first data copy is located will be used as the storage node for the third data copy.

[0029] In one embodiment, the data copy is modified in at least one way including restoring the data copy;

[0030] Based on the placement method of the data copies of the data blocks in the HDFS cluster, determine the change method of the data copies of the data blocks in the HDFS cluster, including:

[0031] If the data copy stored in any storage node in any of the M data centers is lost;

[0032] A storage node is selected in any of the M data centers to restore the lost data copy.

[0033] In one embodiment, the modification of the data copy at least includes deletion of the data copy;

[0034] Based on the placement method of the data copies of the data blocks in the HDFS cluster, determine the change method of the data copies of the data blocks in the HDFS cluster, including:

[0035] If the data copies stored in any storage node in any of the M data centers are redundant;

[0036] Delete redundant copies of data.

[0037] In one embodiment, the data copy is changed in at least one of the following ways: moving the data copy;

[0038] Based on the placement method of the data copies of the data blocks in the HDFS cluster, determine the change method of the data copies of the data blocks in the HDFS cluster, including:

[0039] Calculate HDFS cluster parameters;

[0040] Classify the storage nodes in the HDFS cluster network topology model according to the HDFS cluster parameters and obtain the classification results;

[0041] According to the HDFS cluster parameters and classification results, a source node and a target node for data replica migration are selected in the same data center in the HDFS cluster network topology model to obtain a node pair;

[0042] The node pairs are made into moving tasks and distributed to the source nodes, so that the source nodes move the data copies to the target nodes.

[0043] In a second aspect, an embodiment of the present application provides a flow control device, comprising:

[0044] Model building module, used to build HDFS cluster network topology model;

[0045] The replica placement module is used to determine the placement method of data replicas of data blocks in the HDFS cluster based on the HDFS cluster network topology model;

[0046] The traffic control module is used to determine the change mode of the data copies of the data blocks in the HDFS cluster based on the placement mode of the data copies of the data blocks in the HDFS cluster, so as to control the traffic between M data centers of the HDFS cluster through the change mode of the data copies of the data blocks in the HDFS cluster, where M is an integer greater than 1.

[0047] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above methods when executing the computer program.

[0048] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0049] The embodiment of the present application provides a flow control method, device, computer equipment and storage medium, including: first constructing an HDFS cluster network topology model, then determining the placement mode of data copies of data blocks in the HDFS cluster based on the HDFS cluster network topology model, and then determining the change mode of data copies of data blocks in the HDFS cluster based on the placement mode of data copies of data blocks in the HDFS cluster, so as to control the flow between M data centers of the HDFS cluster through the change mode of data copies of data blocks in the HDFS cluster. The present application configures the placement mode of data copies of data blocks through the constructed HDFS cluster network topology model, thereby determining the change mode of data copies of data blocks based on the placement mode of data copies of data blocks, effectively controlling the generation of cross-data center flow and improving the fault tolerance and stability of the HDFS cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The drawings constituting a part of this application are used to provide a further understanding of this application, so that other features, purposes and advantages of this application become more obvious. The schematic embodiment drawings and their descriptions of this application are used to explain this application and do not constitute an improper limitation on this application. In the drawings:

[0051] Figure 1 This is an architecture diagram of a multi-data center provided in an embodiment of the present application;

[0052] Figure 2 It is a schematic diagram of a data replica placement method under a rack provided in an embodiment of the present application;

[0053] Figure 3 It is a flow chart of a flow control method provided in an embodiment of the present application;

[0054] Figure 4 It is a schematic diagram of an HDFS cluster network topology model provided in an embodiment of the present application;

[0055] Figure 5 It is a schematic diagram of a data replica placement method based on multiple data centers provided in an embodiment of the present application;

[0056] Figure 6 It is a schematic diagram of the recovery logic of data copies in multiple data centers provided by an embodiment of the present application;

[0057] Figure 7 It is a schematic diagram of a deletion logic of a data copy in a multi-data center provided by an embodiment of the present application;

[0058] Figure 8 is a schematic diagram of storage node classification provided by an embodiment of the present application;

[0059] Fig. 9is a schematic diagram of a configuration node pair provided in an embodiment of the present application;

[0060] Fig.10 is a schematic diagram of another configuration node pair provided in an embodiment of the present application;

[0061] Fig.11 is a schematic diagram of another configuration node pair provided in an embodiment of the present application;

[0062] Fig.12 is a schematic diagram of a flow control device provided in an embodiment of the present application;

[0063] Fig.13 It is a schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0065] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein.

[0066] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0067] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0068] It should be understood that in the present application, "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0069] It should be understood that in the present application, "plurality" refers to two or more than two. "And / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "Contains A, B and C", "Contains A, B, C" means that A, B, and C are all included, "Contains A, B or C" means that one of A, B, and C is included, and "Contains A, B and / or C" means that any one, any two, or any three of A, B, and C are included.

[0070] It should be understood that in this application, "B corresponding to A", "B corresponding to A", "A corresponds to B" or "B corresponds to A" means that B is associated with A and B can be determined based on A. Determining B based on A does not mean determining B based only on A, but B can also be determined based on A and / or other information. A and B match when the similarity between A and B is greater than or equal to a preset threshold.

[0071] Depending on the context, "if" as used herein may be interpreted as "when" or "when" or "in response to determining" or "in response to detecting."

[0072] The data involved in this application may be data authorized by the tester or fully authorized by all parties. The collection, dissemination, and use of the data shall comply with the relevant laws, regulations, and standards of the relevant countries and regions. The implementation methods / examples of this application may be combined with each other.

[0073] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0074] To facilitate understanding of this application, the key words in this application are explained as follows:

[0075] HDFS (Hadoop Distributed File System): A highly fault-tolerant, high-throughput distributed file system designed for the storage of large-scale data sets.

[0076] DN: DataNode, which is mainly responsible for the storage and management of actual data and is one of the core components of the HDFS architecture.

[0077] Rack: Rack refers to a standardized equipment installation framework in a data center, which is used to store servers, storage devices, network devices (such as switches, routers), and other computer hardware. From the perspective of HDFS, a rack is a physical facility that is higher than a normal storage node.

[0078] Data Center: Data Center, referred to as DC, is a centralized facility or location for storing, managing, processing, and distributing large amounts of data and information. From the perspective of HDFS, a data center is a physical facility based on the upper layer of the rack, usually containing computing, storage, network and other related equipment.

[0079] Data replicas: In an HDFS cluster, to ensure data fault tolerance, each data block is written into multiple identical replica blocks. By default, a data block has three data replica blocks.

[0080] Network topology: refers to the logical connection relationship between each DN node in the HDFS cluster. The basic concepts include racks, nodes, and paths.

[0081] Network traffic: refers to the amount of data transmitted in a network. It can be traffic in a wired or wireless network, covering the exchange of data packets between devices.

[0082] Balancer: refers to a utility in HDFS that is used to balance the distribution of data blocks across the cluster.

[0083] Next, the present application scheme will be described through specific embodiments in conjunction with the accompanying drawings.

[0084] HDFS (Hadoop Distributed File System) is a highly fault-tolerant, high-throughput distributed file system designed for the storage of large-scale data sets. It is mainly used to store massive amounts of data. The HDFS cluster itself provides the important feature of rack awareness, the main purpose of which is to optimize data reading and storage, especially in a multi-rack environment.

[0085] like Figure 1As shown in the figure, the multi-data center architecture diagram designed based on the HDFS rack-aware mechanism, where DataCenter1, DataCenter2, and DataCenter3 are all data centers, which are the largest storage units under the current physical architecture. Each DataCenter can only be bound to one rack, and the rack contains multiple storage nodes (DN). DN is the smallest storage unit, and the actual data will be written to the disk under the DN node.

[0086] In this architecture, by default, one data block corresponds to three data copies, and each DataCenter will be assigned at most two data copies. Since the three replica blocks are transmitted through the data pipeline, the previous DN will not continue to transmit to the next DN until the previous DN completes writing the data copy, so there will inevitably be traffic between data centers. In order to achieve data center-level fault tolerance, this part of traffic is unavoidable.

[0087] Further, such as Figure 2 As shown, when a data block corresponds to three data copies, one of the copy blocks will be allocated to one Rack, and the other two will be allocated to the same Rack.

[0088] In the data center, in the rack 1:1 architecture, although the rack awareness mechanism is used to write data replica blocks to multiple data centers to achieve data center-level fault tolerance, but in a single data center, since there can only be one rack, data replicas do not meet rack-level fault tolerance, and data center fault tolerance is incompatible with rack fault tolerance.

[0089] The existing multi-data center solution mainly relies on the rack awareness mechanism of HDFS itself. Therefore, there are the following problems under this architecture:

[0090] First, in the event of a DC1 failure, DC2 does not have rack-level disaster recovery capabilities. When DC1 fails, the HDFS cluster will be in a single data center and a single rack. This situation does not have rack-level disaster recovery capabilities. When three or more DNs fail, there is a risk of file loss.

[0091] Secondly, under the current default replica placement strategy, only one replica will be written to DC1 where the client is located, and the other two replicas will be in another DC2. If the DN of DC1 is down, since DC1 has no extra replicas, it has to read data from DC2, which causes cross-data center traffic.

[0092] Again, in the event of a DC1 failure, batch restarts cannot be performed when restarting DNs in DC2. After a DC1 failure, HDFS will be in a single data center (DC2) single rack architecture. To avoid missing files, when restarting DNs, the operation and maintenance staff can only restart two DNs at most at a time.

[0093] Finally, when using the Balance data balancing tool, there will be a large amount of network traffic across racks. Balance will prioritize balancing the capacity of DN nodes within the rack. In a single data center and single rack architecture, the Balance process will result in the situation where the same logical rack is actually transmitting data under multiple physical racks. This may cause a large amount of network bandwidth to be generated between racks.

[0094] Although the existing solutions have the disaster recovery capability of data at the data center level, they lack effective methods for controlling traffic between data centers. Data centers are often built in different places, and they communicate with each other through dedicated network lines, which are very expensive and have limited traffic. If traffic between data centers is not controlled, it will not only reduce the performance of the cluster, but in serious cases it will even explode the network bandwidth.

[0095] Therefore, this application proposes to introduce data centers into the HDFS network topology structure (model) to achieve traffic control between data centers.

[0096] See also Figure 3 , Figure 3 A flow chart of a flow control method provided in an embodiment of the present application. The method comprises the following steps:

[0097] Step S301: Build an HDFS cluster network topology model.

[0098] The present application constructs an HDFS cluster network topology model by first constructing the root node of the HDFS cluster network topology model, then connecting the root node to each of the M data centers, and then configuring N racks for each data center, and each data center is connected to P storage nodes through each of the N racks, where N and P are both integers greater than 1. The network topology model consisting of the root node, M data centers, M*N racks, and M*N*P storage nodes is used as the HDFS cluster network topology model.

[0099] Specifically, this application introduces the data center into the network topology structure. The data center is located at the top layer of the entire model, the rack is located in the middle layer, and the storage node DN is located at the bottom layer. In this way, each DN node has location information in the entire HDFS cluster, and the position relationship between DNs can also be calculated through the network topology structure.

[0100] like Figure 4 As shown in the figure, the " / " in the HDFS cluster network topology model represents the root node, D represents the data center, R represents the rack, and the bottom layer H represents the DN storage node. The HDFS cluster network topology model includes two data centers, namely D1 and D2. Data center D1 is configured with two racks, namely R1 and R2, and data center D2 is configured with two racks, namely R3 and R4. Rack R1 is configured with three storage nodes, namely H1, H2 and H3, rack R2 is configured with three storage nodes, namely H4, H5 and H6, rack R3 is configured with three storage nodes, namely H7, H8 and H9, and rack R4 is configured with three storage nodes, namely H10, H11 and H12.

[0101] From the above figure, we can see that the path of H1 is / D1 / R1 / H1, the parent node of H1 is R1, and the parent node of R1 is D1. These path information can be obtained by querying the configured information. With the path information of these storage nodes, the distance between any two storage node DNs can be calculated, for example:

[0102] distance( / D1 / R1 / H1, / D1 / R1 / H1)=0

[0103] distance( / D1 / R1 / H1, / D1 / R1 / H2)=2

[0104] distance( / D1 / R1 / H1, / D1 / R1 / H4)=4

[0105] distance( / D1 / R1 / H1, / D2 / R3 / H7)=6

[0106] This application is based on the open source HDFS distributed file system.

[0107] In the existing solutions, the data center layer is not specifically identified, and all storage node DNs are divided into racks, while racks are not divided additionally. Therefore, in this solution, data center nodes are added based on the existing rack network topology model, and a data center-level HDFS cluster network topology model is designed. The data center is used as a unique node, and racks will be divided into data centers. Through this model, the location of each DN node in the HDFS cluster can be identified and the data center can be identified, which provides a basis for the subsequent selection of DN nodes for data copies and the placement of data copies based on data centers.

[0108] Step S302: Based on the HDFS cluster network topology model, determine the placement method of data copies of data blocks in the HDFS cluster.

[0109] Based on the HDFS cluster network topology model, determine the placement method of data copies of data blocks in the HDFS cluster, including: when there are three data copies of the data block in the HDFS cluster, obtain the first data copy, the second data copy and the third data copy of the three data copies; based on the HDFS cluster network topology model, determine the storage node for placing the first data copy; based on the data center where the storage node for placing the first data copy is located, determine the storage node for placing the second data copy; based on the data center where the storage node for placing the first data copy is located and the data center where the storage node for placing the second data copy is located, determine the storage node for placing the third data copy.

[0110] Among them, based on the HDFS cluster network topology model, determining the storage node where the first data copy is placed includes: judging whether the client is in the target storage node, where the target storage node is the storage node where the client is located; if the client is in the target storage node, placing the first data copy in the target storage node first; if the client is not in the target storage node, placing the first data copy in any storage node among the M*N*P storage nodes.

[0111] Among them, based on the data center where the storage node where the first data copy is placed is located, determining the storage node where the second data copy is placed includes: selecting a storage node in a data center different from the data center where the storage node where the first data copy is placed from M data centers as the storage node for placing the second data copy.

[0112] Wherein, based on the data center where the storage node where the first data copy is placed and the data center where the storage node where the second data copy is placed, determining the storage node where the third data copy is placed includes: if the data center where the storage node where the first data copy is placed and the data center where the storage node where the second data copy is placed are the same data center, selecting a storage node in a data center different from the data center where the storage node where the first data copy is placed from M data centers as the storage node for placing the third data copy; if the data center where the storage node where the first data copy is placed and the data center where the storage node where the second data copy is placed are different data centers, using a storage node under a different rack in the same data center as the data center where the storage node where the first data copy is placed as the storage node for placing the third data copy.

[0113] like Figure 5As shown, DataCenter 1 and DataCenter 2, each DataCenter has two racks, and each rack has three storage node DNs.

[0114] Assume that the node where the client is located is storage node DN1. When selecting the storage node for the first data copy, the storage node DN1 where the client is located is preferred as the storage node for the first data copy. If the client is not in storage node DN1, the first data copy is placed in any of the remaining storage nodes.

[0115] After placing the first data copy, when placing the second data copy, the second data copy needs to be placed in a storage node in a different data center from the first data copy. If the first data copy is placed in storage node DN1, and storage node DN1 belongs to DataCenter1, then the second data copy needs to be placed in a storage node in DataCenter2. You can place it in Figure 5 The storage node DN8 in DataCenter2 is shown.

[0116] If there are no empty storage nodes in a different data center from the first data copy, the second data copy can be placed in any non-empty storage node.

[0117] After the first data copy and the second data copy are placed, the third data copy needs to be combined with whether the storage node where the first data copy is placed and the storage node where the second data copy is placed are in the same data center.

[0118] If the storage node where the first data copy is placed and the storage node where the second data copy is placed are in the same data center, the third data copy needs to be placed in a storage node in a data center different from this data center.

[0119] If the storage node where the first data copy is placed is not in the same data center as the storage node where the second data copy is placed, the third data copy must be placed in the same data center as the first data copy but in a different rack.

[0120] This application proposes a data center-aware data replica placement method, where the first replica is preferentially placed in the data center where the client is located, the second replica is preferentially selected in a node in a different data center from the first replica, the third replica is preferentially placed in the data center where the client is located, and the remaining replicas are randomly selected by default. In data block reading, the nodes where each data replica is located are sorted by the distance, and the node closest to the client is preferentially selected for data reading, which avoids cross-data center traffic and improves data reading efficiency.

[0121] Step S303: Based on the placement method of the data copies of the data blocks in the HDFS cluster, determine the change method of the data copies of the data blocks in the HDFS cluster, so as to control the traffic between the M data centers of the HDFS cluster through the change method of the data copies of the data blocks in the HDFS cluster.

[0122] Wherein, M is an integer greater than 1.

[0123] In the case where the data copy change method includes the recovery of the data copy, based on the placement method of the data copy of the data block in the HDFS cluster, the change method of the data copy of the data block in the HDFS cluster is determined, including: if the data copy stored in any storage node under any of the M data centers is lost; select a storage node under any of the M data centers to recover the lost data copy.

[0124] Among them, the actual scenarios of data replicas include DN failure, abnormal downtime caused by program killing, bad disk in DN, rack failure, data center downtime, etc. When facing this situation, the focus should be on controlling the recovery of missing replica blocks in multi-data center scenarios, that is, controlling the recovery of data replicas to reduce or avoid cross-data center traffic.

[0125] To avoid unnecessary cross-data center traffic, during the replica recovery process, it is preferred to restore data in the same data center as the lost data replica.

[0126] like Figure 6 In the recovery of data copies in multiple data centers shown in the figure, data copies 3 and b in DataCenter1 are both lost (i.e., the crossed 3 and b), then the storage node is selected in DataCenter1 to restore data copies 3 and b first. The restored data copies 3 and b are Figure 6 The unmarked 3 and b.

[0127] In the case where the data copy is changed by deleting the data copy, the data copy of the data block in the HDFS cluster is changed based on the placement of the data copy in the HDFS cluster, including: if the data copy stored in any storage node under any data center in the M data centers is redundant; the redundant data copy is deleted.

[0128] Among them, the redundant scenarios of data copies include DN DeCommission (node ​​decommissioning) and then going online again; the client or administrator reduces the expected number of copies of the file; the newly added copy is quickly recorded and lost in the system. When faced with this situation, you need to select the data copy for deletion, that is, the deletion of the data copy.

[0129] Since each rack can be configured with three storage nodes, correspondingly, the three storage nodes can store three data copies. If a rack corresponds to four data copies, it means that there is data copy redundancy, and one data copy needs to be deleted.

[0130] like Figure 7 As shown, the dark-colored data copies belong to the same rack, and if there is more than one data copy, data copy 4 is deleted. Similarly, the light-colored data copy c is deleted.

[0131] In the deletion of redundant replica blocks, data transmission is not involved and no cross-data center traffic is generated in this process. However, data center fault tolerance and rack fault tolerance need to be considered to improve the cluster fault tolerance rate.

[0132] In the case where the data copy is changed by moving the data copy, the data copy of the data block in the HDFS cluster is changed based on the placement of the data copy in the HDFS cluster, including: calculating the HDFS cluster parameters; classifying the storage nodes in the HDFS cluster network topology model according to the HDFS cluster parameters to obtain the classification results; selecting the source node and the target node for moving the data copy in the same data center in the HDFS cluster network topology model according to the HDFS cluster parameters and the classification results to obtain a node pair; making the node pair into a moving task and distributing it to the source node, so that the source node moves the data copy to the target node.

[0133] To move data copies, first, you need to calculate the HDFS cluster parameters. You can calculate the HDFS cluster parameters by looking up table data, such as the overall average utilization rate of the cluster, the utilization rate of each node, and how much data can be moved in or out of each node to avoid a balanced state.

[0134] After the HDFS cluster parameters are calculated, the storage nodes in the HDFS cluster network topology model can be classified according to the HDFS cluster parameters. The classification results are as follows: Figure 8 As shown, it includes overload nodes, nodes exceeding the average water level, nodes below the average water level and load nodes, wherein the average water levels of the nodes exceeding the average water level and the nodes below the average water level can be set according to specific circumstances and are not specifically limited here.

[0135] According to the HDFS cluster parameters and classification results, the source node Source and the target node Target for data copy migration are selected in the same data center in the HDFS cluster network topology model to obtain the node pair.<Source,Target> .

[0136] It should be noted that when selecting the source node Source and the target node Target, the priority of the same rack in the same data center and different racks in the same data center must be followed.

[0137] Exemplarily, the selection order of the source node Source and the target node Target is as follows:

[0138] 1. If Fig. 9 The overUtilized nodes to the underUtilized nodes of the same rack are shown.

[0139] 2. If Fig.10 The overUtilized nodes to the belowAvgUtilized nodes of the same rack are shown for the underUtilized nodes that were not selected in 1.

[0140] 3. If Fig.11 The aboveAvgUtilized nodes to the underUtilized nodes of the same rack are shown, for the underUtilized nodes not selected in 1.

[0141] 4. From the overUtilized nodes to the underUtilized nodes in other racks in the same data center.

[0142] 5. From the overUtilized nodes to the belowAvgUtilized nodes in other racks in the same data center, for the overUtilized nodes that were not selected in 4.

[0143] 6. From the aboveAvgUtilized nodes to the underUtilized nodes in other racks in the same data center, for the underUtilized nodes that were not selected in 4.

[0144] 7. Make the node pair into a moving task and distribute it to the source node, so that the source node moves the data copy to the target node, waits for the data to be moved successfully, and returns the end status of the iteration.

[0145] During the entire replica block movement process, the movement of the replica blocks is all within the same data center. Therefore, the entire process does not generate cross-data center traffic, and the replica blocks can be moved within the same data center or the same rack, reducing cross-data center traffic.

[0146] The embodiment of the present application provides a flow control method, including: first constructing an HDFS cluster network topology model, then determining the placement mode of data copies of data blocks in the HDFS cluster based on the HDFS cluster network topology model, and then determining the change mode of data copies of data blocks in the HDFS cluster based on the placement mode of data copies of data blocks in the HDFS cluster, so as to control the flow between M data centers of the HDFS cluster through the change mode of data copies of data blocks in the HDFS cluster. The present application configures the placement mode of data copies of data blocks through the constructed HDFS cluster network topology model, thereby determining the change mode of data copies of data blocks based on the placement mode of data copies of data blocks, effectively controlling the generation of cross-data center flow and improving the fault tolerance and stability of the HDFS cluster.

[0147] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0148] The following is an embodiment of the device of the present application. For details not described in detail, please refer to the corresponding method embodiment described above.

[0149] Fig.12 A schematic diagram of the structure of a flow control device provided by an embodiment of the present application is shown. For the sake of convenience, only the part related to the embodiment of the present application is shown. A flow control device includes a model building module 1201, a replica placement module 1202 and a flow control module 1203, which are as follows:

[0150] The model building module 1201 is used to build a HDFS cluster network topology model;

[0151] The replica placement module 1202 is used to determine the placement mode of the data replicas of the data blocks in the HDFS cluster based on the HDFS cluster network topology model;

[0152] The traffic control module 1203 is used to determine the change method of the data copies of the data blocks in the HDFS cluster based on the placement method of the data copies of the data blocks in the HDFS cluster, so as to control the traffic between M data centers of the HDFS cluster through the change method of the data copies of the data blocks in the HDFS cluster, where M is an integer greater than 1.

[0153] In one embodiment, the model building module 1201 is also used to build the root node of the HDFS cluster network topology model;

[0154] Connect the root node to each of the M data centers;

[0155] N racks are configured for each data center, and each data center is connected to P storage nodes through each rack in the N racks, where N and P are both integers greater than 1;

[0156] The network topology model consisting of the root node, M data centers, M*N racks, and M*N*P storage nodes is used as the HDFS cluster network topology model.

[0157] In an embodiment, the replica placement module 1202 is further configured to obtain a first data replica, a second data replica, and a third data replica of the three data replicas when there are three data replicas of the data block in the HDFS cluster;

[0158] Based on the HDFS cluster network topology model, determine the storage node where the first data copy is placed;

[0159] Determine the storage node where the second data copy is placed based on the data center where the storage node where the first data copy is placed is located;

[0160] Based on the data center where the storage node where the first data copy is placed is located and the data center where the storage node where the second data copy is placed is located, determine the storage node where the third data copy is placed.

[0161] In one embodiment, the replica placement module 1202 is further used to determine whether the client is in a target storage node, wherein the target storage node is the storage node where the client is located;

[0162] If the client is at the target storage node, the first data copy is preferentially placed at the target storage node;

[0163] If the client is not in the target storage node, the first data copy is placed in any storage node among the M*N*P storage nodes.

[0164] In one embodiment, the replica placement module 1202 is further configured to select, from the M data centers, a storage node in a data center different from the data center where the storage node where the first data replica is placed is located as the storage node for placing the second data replica.

[0165] In an embodiment, the replica placement module 1202 is further configured to select, from the M data centers, a storage node in a data center different from the data center where the storage node for placing the first data replica is located as the storage node for placing the third data replica if the data center where the storage node for placing the first data replica is located and the data center where the storage node for placing the second data replica is located are the same data center;

[0166] If the data center where the storage node for the first data copy is located is different from the data center where the storage node for the second data copy is located, a storage node in a different rack of the same data center as the data center where the storage node for the first data copy is located will be used as the storage node for the third data copy.

[0167] In one embodiment, the data copy is modified in at least one way including restoring the data copy;

[0168] The flow control module 1203 is also used for if a data copy stored in any storage node under any data center of the M data centers is lost;

[0169] A storage node is selected in any of the M data centers to restore the lost data copy.

[0170] In one embodiment, the modification of the data copy at least includes deletion of the data copy;

[0171] The flow control module 1203 is also used to ensure that the data copies stored in any storage node in any data center of the M data centers are redundant;

[0172] Delete redundant copies of data.

[0173] In one embodiment, the data copy is changed in at least one of the following ways: moving the data copy;

[0174] The flow control module 1203 is also used to calculate HDFS cluster parameters;

[0175] Classify the storage nodes in the HDFS cluster network topology model according to the HDFS cluster parameters and obtain the classification results;

[0176] According to the HDFS cluster parameters and classification results, a source node and a target node for data replica migration are selected in the same data center in the HDFS cluster network topology model to obtain a node pair;

[0177] The node pairs are made into moving tasks and distributed to the source nodes, so that the source nodes move the data copies to the target nodes.

[0178] The embodiment of the present application provides a flow control device, which can be used to: first construct an HDFS cluster network topology model, then determine the placement method of the data copy of the data block in the HDFS cluster based on the HDFS cluster network topology model, and then determine the change method of the data copy of the data block in the HDFS cluster based on the placement method of the data copy of the data block in the HDFS cluster, so as to control the flow between M data centers of the HDFS cluster through the change method of the data copy of the data block in the HDFS cluster. The present application configures the placement method of the data copy of the data block through the constructed HDFS cluster network topology model, thereby determining the change method of the data copy of the data block based on the placement method of the data copy of the data block, effectively controlling the generation of cross-data center flow and improving the fault tolerance and stability of the HDFS cluster.

[0179] This application Fig.13 A schematic diagram of a computer device is provided. Fig.13 As shown, the computer device 13 of this embodiment includes: a processor 1301, a memory 1302, and steps of the flow control method embodiment stored in the memory 1302 and executable on the processor 1301, such as Figure 3 Alternatively, when the processor 1301 executes the computer program 1303, the functions of each module / unit in the above-mentioned flow control device embodiments are realized, for example Fig.12 Functions of modules / units 1201 to 1203 are shown.

[0180] The present application also provides a readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the flow control method provided by the various embodiments described above.

[0181] Among them, the readable storage medium can be a computer storage medium or a communication medium. The communication medium includes any medium that facilitates the transmission of a computer program from one place to another. The computer storage medium can be any available medium that can be accessed by a general or special-purpose computer. For example, a readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application-specific integrated circuit (ASIC). In addition, the ASIC can be located in a user device. Of course, the processor and the readable storage medium can also exist in a communication device as discrete components. The readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0182] The present application also provides a program product, which includes an execution instruction, which is stored in a readable storage medium. At least one processor of the device can read the execution instruction from the readable storage medium, and at least one processor executes the execution instruction so that the device implements the flow control method provided by the various embodiments described above.

[0183] In the embodiments of the above-mentioned devices, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0184] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A flow control method, characterized in that: include: Build the HDFS cluster network topology model; Based on the HDFS cluster network topology model, determine the placement mode of data copies of data blocks in the HDFS cluster; Based on the placement method of the data copies of the data blocks in the HDFS cluster, determine the change method of the data copies of the data blocks in the HDFS cluster, so as to control the traffic between M data centers of the HDFS cluster by changing the data copies of the data blocks in the HDFS cluster, where M is an integer greater than 1.

2. The flow control method according to claim 1, characterized in that: The construction of the HDFS cluster network topology model includes: Constructing the root node of the HDFS cluster network topology model; Connecting the root node to each of the M data centers; N racks are configured for each data center, and each data center is connected to P storage nodes through each rack in the N racks, wherein N and P are both integers greater than 1; The network topology model composed of the root node, the M data centers, M*N racks and M*N*P storage nodes is used as the HDFS cluster network topology model.

3. The flow control method according to claim 2, characterized in that: The step of determining the placement of data replicas of data blocks in the HDFS cluster based on the HDFS cluster network topology model includes: When there are three data copies of the data block in the HDFS cluster, obtain a first data copy, a second data copy, and a third data copy of the three data copies; Based on the HDFS cluster network topology model, determine the storage node where the first data copy is placed; Determine a storage node for placing the second data copy based on the data center where the storage node for placing the first data copy is located; Based on the data center where the storage node where the first data copy is placed is located and the data center where the storage node where the second data copy is placed is located, determine the storage node where the third data copy is placed.

4. The flow control method according to claim 3, characterized in that: The step of determining the storage node for placing the first data copy based on the HDFS cluster network topology model includes: Determine whether the client is in a target storage node, wherein the target storage node is the storage node where the client is located; If the client is located at the target storage node, the first data copy is preferentially placed at the target storage node; If the client is not located at the target storage node, the first data copy is placed at any storage node among the M*N*P storage nodes.

5. The flow control method according to claim 3, characterized in that: The determining, based on the data center where the storage node where the first data copy is placed is located, the storage node where the second data copy is placed includes: A storage node in a data center different from the data center where the storage node where the first data copy is placed is selected from the M data centers as the storage node for placing the second data copy.

6. The flow control method according to claim 3, characterized in that: The determining, based on the data center where the storage node where the first data copy is placed is located and the data center where the storage node where the second data copy is placed is located, of the storage node where the third data copy is placed includes: If the data center where the storage node for placing the first data copy is located and the data center where the storage node for placing the second data copy is located are the same data center, a storage node in a data center different from the data center where the storage node for placing the first data copy is located is selected from the M data centers as the storage node for placing the third data copy; If the data center where the storage node for placing the first data copy is located and the data center where the storage node for placing the second data copy is located are different data centers, a storage node under a different rack in the same data center as the data center where the storage node for placing the first data copy is located will be used as the storage node for placing the third data copy.

7. The flow control method according to claim 1, characterized in that: The changing method of the data copy at least includes restoring the data copy; The step of determining a change mode of the data replica of the data block in the HDFS cluster based on the placement mode of the data replica of the data block in the HDFS cluster includes: If a data copy stored in any storage node in any of the M data centers is lost; A storage node is selected in any one of the M data centers to recover the lost data copy.

8. The flow control method according to claim 1, characterized in that: The data copy change method at least includes deleting the data copy; The step of determining a change mode of the data replica of the data block in the HDFS cluster based on the placement mode of the data replica of the data block in the HDFS cluster includes: If the data copies stored in any storage node in any of the M data centers are redundant; Delete redundant copies of data.

9. The flow control method according to claim 1, characterized in that: The changing method of the data copy at least includes moving the data copy; The step of determining a change mode of the data replica of the data block in the HDFS cluster based on the placement mode of the data replica of the data block in the HDFS cluster includes: Calculate HDFS cluster parameters; Classifying the storage nodes in the HDFS cluster network topology model according to the HDFS cluster parameters to obtain a classification result; According to the HDFS cluster parameters and the classification results, a source node and a target node for moving the data copy are selected in the same data center in the HDFS cluster network topology model to obtain a node pair; The node pair is made into a moving task and distributed to the source node, so that the source node moves the data copy to the target node.

10. A flow control device, characterized in that: include: Model building module, used to build HDFS cluster network topology model; A replica placement module, used to determine the placement mode of data replicas of data blocks in the HDFS cluster based on the HDFS cluster network topology model; A traffic control module is used to determine a change method of the data copies of the data blocks in the HDFS cluster based on the placement method of the data copies of the data blocks in the HDFS cluster, so as to control the traffic between M data centers of the HDFS cluster through the change method of the data copies of the data blocks in the HDFS cluster, where M is an integer greater than 1.

11. A computer device, characterized in that: comprising a memory, and one or more processors communicatively connected to the memory; The memory stores instructions that can be executed by the one or more processors, and the instructions are executed by the one or more processors to enable the one or more processors to implement the flow control method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The invention comprises a program or an instruction, and when the program or the instruction is executed on a computer, the flow control method according to any one of claims 1 to 9 is implemented.

Images