HDFS-based disk selection method, device, equipment, medium and product

Abstract

The application relates to a HDFS-based disk selection method and device, equipment, medium and product, in particular to the technical field of data processing. The application comprises the following steps: collecting the input-output operation time consumption of the disk on the data node; calculating the performance index value of the disk according to the input-output operation time consumption, wherein the performance index is used for describing the busy degree of the disk; and selecting the target disk according to the performance index value. In the application, the disk load is considered when the disk is selected, the performance index value of the disk is calculated, the suitable disk is selected based on the performance index value, the reasonable allocation of data is realized, the busy degrees of the disks are balanced, the data balance state of the data node is maintained, and the system bottleneck caused by the unbalanced data reading and writing is avoided.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a disk selection method, apparatus, device, medium and product based on HDFS. Background Technology

[0002] HDFS is a distributed file system that typically stores data across multiple disks on multiple DataNodes. HDFS provides two disk selection strategies: a round-robin volume selection policy and an available space volume selection policy.

[0003] Polling-based disk selection strategies select disks sequentially, and only start selecting from the beginning after all disks have been traversed, ensuring that every disk has a chance to be selected, thus avoiding situations where some disks are overused while others are idle. Disk selection strategies based on available space prioritize disks with more remaining space to store new data blocks, thereby mitigating data storage imbalance to some extent.

[0004] While round-robin volume storage ensures that every disk is used, uneven disk space distribution can occur in long-running clusters due to data deletion and hot-swapping. Using a selection strategy based on available space means that each newly written data entry will go to a newly added empty disk, while other disks remain idle during this period. This can create read / write bottlenecks on the new disk. Summary of the Invention

[0005] To address or at least partially address the aforementioned technical problems, this application provides a disk selection method, apparatus, device, medium, and product based on HDFS, which can select suitable disks and avoid system bottlenecks caused by uneven data read / write operations.

[0006] To achieve the above objectives, the technical solutions provided in this application are as follows:

[0007] In a first aspect, this application provides a disk selection method based on HDFS, comprising: collecting the input / output operation time of the disk on the data node; calculating the disk performance index value based on the input / output operation time, wherein the performance index is used to describe the disk's busyness; and selecting the target disk based on the performance index value.

[0008] As an optional implementation of this application, before selecting a target disk based on performance index values, the method further includes: obtaining the capacity and usage of the disk; calculating the remaining available space of the disk; determining whether the difference in remaining available space between any two disks is less than or equal to a preset threshold; and selecting a target disk based on performance index values, including: if the difference in remaining available space between any two disks is less than or equal to the preset threshold, then traversing all disks of the data node to query whether there is a first disk with remaining available space sufficient to store the amount of data; if so, then selecting the disk with the smallest performance index value from the first disk as the target disk.

[0009] As an optional implementation of this application, selecting a target disk based on performance index values ​​includes: if there are multiple disks with the smallest performance index values ​​in the first disk, then the disk with the smallest performance index value and the largest remaining available space is selected as the target disk.

[0010] As an optional implementation of this application, selecting a target disk based on performance index values ​​includes: if the difference in remaining available space between the disk with the largest remaining available space and the disk with the smallest remaining available space on the data node is greater than a preset threshold, then determining whether the amount of data to be stored is greater than the available space of the disk with the largest remaining available space in the low-availability disk group; if so, selecting the disk with the smallest performance index from other disks as the target disk, where the other disks are disks in the high-availability disk group that meet the size of the amount of data to be stored; wherein, the division between the low-availability disk group and the high-availability disk group is based on the sum of the smallest remaining available space and the preset threshold.

[0011] As an optional implementation of this application, after determining whether the amount of data to be stored is greater than the available space of the disk with the largest remaining available space in the low-availability disk group, the method further includes: if the amount of data to be stored is less than or equal to the available space of the disk with the largest remaining available space in the low-availability disk group, then determining whether the random probability is less than the selection probability of the high-availability volume group; if so, then selecting the disk with the smallest performance index from the high-availability disk group as the target disk; if not, then selecting the disk with the smallest performance index that meets the requirements of the amount of data to be stored from the low-availability disk group as the target disk.

[0012] Secondly, this application provides a disk selection device based on HDFS, the device comprising:

[0013] The acquisition module is used to collect the input / output operation time of the disk on the data node;

[0014] The calculation module is used to calculate the disk's performance metrics based on the input / output operation time. These performance metrics describe the disk's workload.

[0015] The selection module is used to select the target disk based on performance metric values.

[0016] As an optional implementation of this application, the selection module is further configured to: obtain the capacity and usage of the disk before selecting the target disk based on the performance index value; calculate the remaining available space of the disk; and determine whether the difference in the remaining available space between any two disks is less than or equal to a preset threshold.

[0017] The selection module is specifically used to: if the difference in remaining available space between any two disks is less than or equal to a preset threshold, then traverse all disks of the data node and query whether there is a first disk with remaining available space that meets the amount of data to be stored; if so, then select the disk with the smallest performance index value from the first disk as the target disk.

[0018] As an optional implementation of this application, the selection module is specifically used to: if there are multiple disks with the smallest performance index values ​​in the first disk, then the disk with the smallest performance index value and the largest remaining available space is selected as the target disk.

[0019] As an optional implementation of this application, the selection module is further configured to: if the difference in remaining available space between the disk with the largest remaining available space and the disk with the smallest remaining available space on the data node is greater than a preset threshold, then determine whether the amount of data to be stored is greater than the available space of the disk with the largest remaining available space in the low available space disk group; if so, select the disk with the smallest performance index from the other disks as the target disk, and the other disks are the disks in the high available space disk group that meet the amount of data to be stored.

[0020] As an optional implementation of this application, the selection module is further configured to: after determining whether the amount of data to be stored is greater than the available space of the disk with the largest remaining available space in the low-availability space disk group, further configured to: if the amount of data to be stored is less than or equal to the available space of the disk with the largest remaining available space in the low-availability space disk group, determine whether the random probability is less than the selection probability of the high-availability space volume group; if so, select the disk with the smallest performance index from the high-availability space disk group as the target disk; if not, select the disk with the smallest performance index that meets the requirements of the amount of data to be stored from the low-availability space disk group as the target disk.

[0021] Thirdly, this application provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the HDFS-based disk selection method as described in the first aspect or any of its optional embodiments.

[0022] Fourthly, this application provides a computer-readable storage medium, comprising: storing a computer program on the computer-readable storage medium, wherein when the computer program is executed by a processor, it implements the HDFS-based disk selection method as described in the first aspect or any of its optional embodiments.

[0023] Fifthly, this application provides a computer program product, comprising: the computer program product including a computer program that, when the computer program is run on a computer, causes the computer to implement the HDFS-based disk selection method as described in the first aspect or any of its optional embodiments.

[0024] The technical solution provided in this application has the following advantages compared with the prior art:

[0025] This disclosure provides a disk selection method based on HDFS. First, the input / output operation time of the disks on the data nodes is collected. Then, the disk performance metrics are calculated based on the input / output operation times, and the target disk is selected based on these performance metrics. Thus, this application considers disk load when selecting a disk. By calculating the disk performance metric values, it guides the data nodes in selecting suitable disks, achieving reasonable data allocation, balancing the workload of each disk, maintaining data balance across data nodes, and avoiding system bottlenecks caused by uneven data read / write operations. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram of a round-robin-based disk selection strategy;

[0029] Figure 2A This is a flowchart illustrating a disk selection method based on HDFS according to an embodiment of this application. Figure 1 ;

[0030] Figure 2B This is a schematic flowchart of a disk selection method based on HDFS according to an embodiment of this application;

[0031] Figure 3 A schematic diagram of the structure of a disk selection device based on HDFS provided in an embodiment of this application;

[0032] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0033] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the technical terms used in the description of the embodiments or the prior art will be briefly introduced below:

[0034] HDFS employs a master-slave architecture, consisting of a NameNode (NN) and DataNodes. The NameNode, as the master node, is primarily responsible for managing the HDFS namespace, resolving client requests, and controlling client access to HDFS. DataNodes, as slave nodes, are mainly used for physical storage, and their data volume depends on the cluster size.

[0035] The NameNode is the master node of HDFS. Its main functions are: to manage the HDFS namespace, cluster information, and data blocks; to maintain the entire HDFS file directory tree, the metadata of the file directories, and the list of data blocks corresponding to each file; to accept client operation requests; and to manage the mapping relationship between files and data blocks, and between data blocks and DataNodes.

[0036] DataNodes handle read and write requests from file system clients. Under the unified scheduling of the NameNode, they create, replicate, and delete data blocks. Each DataNode encapsulates a structure in memory: a data storage component (FsDataset), responsible for dynamically maintaining the set of all data blocks on its node. Because block files exist on various storage disks and directories, it also maintains storage directories (i.e., disks). Within a DataNode, each storage directory is encapsulated as a volume (FsVolume), simulating a "mount point" in a Linux system. Normally, an FsVolume can be considered an abstraction of the disk. Sometimes, access to the actual disk is required (e.g., retrieving available disk space), necessitating direct interaction between the abstraction and the actual disk, primarily achieved through Java calling respective Linux commands.

[0037] In HDFS, a block, or its replica data, is the basic storage unit that makes up a file. Files are connected in an ordered and structured manner using blocks. Replica data can be understood as the concrete storage representation of a block. HDFS configures the block size using `${dfs.Blocksize}`, with a default value of 128MB. When a new block is created, the replica location determines its storage location within the cluster, determined by both the location selection strategy and the storage strategy.

[0038] Currently, HDFS employs two disk selection strategies for writing data: a round-robin-based disk selection strategy and a disk selection strategy based on remaining available space. The disk selection strategy is specified by the configuration item `dfs.datanode.fsdataset.volume.choosing.policy`, with the default being the round-robin volume selection policy. Explanations of this configuration item are shown in Table 1 below.

[0039] Table 1

[0040]

[0041] The RoundRobinVolumeChoosingPolicy strategy is relatively simple: it uses a round-robin approach to select each volume sequentially. For example, ... Figure 1 As shown, there are two new replicas, Replica1 and Replica2, both of which need to be stored in the storage directory belonging to DISK. Using a round-robin disk selection strategy, Replica1 will be placed in the directory belonging to DISK1, and Replica2 will be placed in the directory belonging to DISK2. The processing of different types of replicas is similar.

[0042] This strategy is based on the concept of "round-robin," which selects disks sequentially in a certain order. Once the last disk is reached, the selection process restarts from the beginning, and so on, ensuring that every disk has a chance to be selected, thus achieving a balanced distribution of data across the disks. The design goal of this strategy is to address the problem of uneven disk space distribution, particularly when dealing with a large number of small files causing different disks on the same DataNode to have the same number of blocks but occupy different disk sizes. The round-robin strategy is implemented based on a simple principle: select disks sequentially, and after traversing all disks, start the selection process again from the beginning, ensuring that every disk has a chance to be selected, thereby avoiding situations where some disks are overused while others are idle. This strategy helps improve the overall efficiency and reliability of the storage system, especially when handling a large number of small files, as it better balances the load on different disks.

[0043] The Available Space Volume Choosing Policy takes into account the remaining available space on disks. This policy aims to optimize data distribution by prioritizing disks with more remaining space to store new data blocks, thus mitigating data imbalance to some extent. Specifically, when the remaining available space of all disks in the available disk list differs within a configured threshold (default 10GB), it indicates that the data volume across the disks is relatively even. In this case, the policy degenerates into a Round Robin Volume Choosing Policy, selecting each disk in turn in sequence.

[0044] Of the two disk selection strategies mentioned above, while the round-robin volume storage method ensures that every disk is used, uneven disk space distribution can occur in long-running clusters due to data deletion and disk hot-swapping. In long-running clusters using the round-robin strategy, DataNodes may sometimes fill their storage directories (disks / volumes) unevenly, resulting in some disks being full while others are underutilized. This can occur due to a large number of write and delete operations, or due to disk replacement. Furthermore, if we use a selection strategy based on available space, each newly written data will go to a newly added empty disk, leaving other disks idle during this period, which will create read / write bottlenecks on the new disk.

[0045] Furthermore, a DataNode can be configured with multiple storage directories to store data block files. These storage directories can be heterogeneous, designed to improve data block I / O throughput. Data blocks under each storage directory are managed by FsVolumelmpl objects, while the FsVolumeList class stores all FsVolumelmpl objects on the DataNode, providing disk-like services to FsDatasetlmpl. This design improves data block I / O throughput while also providing greater flexibility for data storage and management.

[0046] When implementing the AvailableSpaceVolumeChoosingPolicy, the allocation ratio of new data blocks can be controlled by adjusting the dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction configuration item to ensure that data is more evenly distributed across different disks. The default value for this configuration item is 0.75f, which is generally sufficient. However, this value can be adjusted based on actual circumstances and needs to optimize data storage distribution.

[0047] To address some or all of the technical problems existing in the aforementioned related technologies, embodiments of this application provide a disk selection method, apparatus, electronic device, medium, and product based on HDFS. The method first collects the input / output operation time of disks on data nodes, calculates the performance indicators of each disk based on the input / output operation time, and then selects the target disk based on the performance indicators. Thus, this application considers disk load when selecting disks, calculates disk performance indicator values, and guides data nodes to select suitable disks based on these performance indicator values, achieving reasonable data allocation, balancing the busy levels of each disk, maintaining the data balance of data nodes, and avoiding system bottlenecks caused by uneven data read / write operations.

[0048] To better understand the above-mentioned objectives, features, and advantages of this application, the solution of this application will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0049] Many specific details are set forth in the following description in order to provide a full understanding of this application, but this application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of this application, and not all embodiments.

[0050] This application provides a disk selection method based on HDFS, which can be implemented using an HDFS-based disk selection device or electronic device. The electronic device includes, but is not limited to, in-vehicle terminals, servers, personal computers, laptops, tablets, and smartphones. The operating system of the electronic device can include Android, Apple's iOS, Microsoft's Windows operating system, Linux (Unix), etc., and this application does not limit this. The electronic device can operate independently to implement this application, or it can connect to a network and implement this application through interaction with other computer devices on the network. The network where the electronic device is located includes, but is not limited to, the Internet, wide area networks (WANs), metropolitan area networks (MANs), local area networks (LANs), and virtual private networks (VPNs).

[0051] It should be noted that the scope of protection of the HDFS-based disk selection method described in this application is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this application is included within the scope of protection of this application.

[0052] like Figure 2A As shown, Figure 2AThis is a flowchart illustrating a disk selection method based on HDFS according to an embodiment of this application. Figure 1 This method can be executed by an HDFS-based disk selection device, which can be implemented in software and / or hardware and is typically integrated into an electronic device. For example... Figure 2A As shown, the method mainly includes the following steps S201 to S203:

[0053] S201, Time consumed by disk input / output operations on the data acquisition node.

[0054] In some implementations, disk information is read from the ` / proc / diskstats` file to collect the time spent on disk input / output operations on the data node, as shown in the `time spent doing I / Os(ms)` field in Table 2. In Linux, the `iostat` command can be used to view disk I / O information. The ` / proc / diskstats` file displays I / O statistics for block devices. Each line contains the following 14 fields.

[0055] Table 2

[0056] Number of columns meaning 1 majornumber Master device number 2 minormumber Secondary device number 3 devicename Equipment Name 4 Reads completed successfully Total number of reads successfully 5 readsmerged Merge read count 6 sectorsread Number of sectors read 7 timespentreading(ms) Reading time 8 writescompleted Total number of successful writes 9 writesmerged Number of merged writes 10 sectorswritten Number of write sectors 11 timespentwriting(ms) The time to write flowers 12 I / Os is currently in progress Number of I / O operations currently being processed 13 timespentdoingI / Os(ms) Input / output time (ms) 14 weightedtimespentdoingI / Os(ms) Weighted milliseconds of input / output operations

[0057] In some embodiments, the ioutil feature is enabled by adding configuration items as shown in Table 3 to the hdfs-site.xml configuration file. If the ioutil feature is not enabled, the RoundRobinVolumeChoosingPolicy strategy is used to select the target disk.

[0058] Table 3

[0059]

[0060] S202. Calculate the performance index values ​​of each disk based on the input / output operation time.

[0061] In some embodiments, the time difference between adjacent input / output operations is calculated, and the disk performance metric value ioutil is calculated in combination with the sampling time interval. This is shown in formula (1) below. The performance metric describes the disk's activity level; a higher performance metric value indicates a busier disk.

[0062]

[0063] In formula (1), Δt represents the time difference between adjacent input / output operations, and T is the sampling time interval. An exemplary sampling time interval is 1 second, which can be set through the configuration items shown in Table 4 below.

[0064] Table 4

[0065]

[0066] In some embodiments, after setting the storageLocations of DiskIOUtilManager when the data node starts, the DiskIOUtilManager thread is started. The StorageLocation class stores the URI and Storage Type (SSD / DISK / RAM) of a storage directory. The device name corresponding to the directory (e.g., / dev / sda, / dev / sdb) can be obtained through the directory path, and then a mapping is established between the device name and the IOStat object (a wrapper object storing ioutil values). After the DiskIOUtilManager thread starts, according to the default configuration IOUtilCollectInterval, it collects data every second and calculates the ioutil value. The calculated performance metric value is stored in a Map data structure, corresponding to the device name.

[0067] S203. Select the target disk based on the performance index values.

[0068] The target disk has the lowest performance index value. In some embodiments, during step S203, the storage space and used space of the disk are obtained, and the remaining available space of the disk is calculated. The available storage space minus the used space equals the remaining available space. It is determined whether the difference in remaining available space between any two disks is less than a preset threshold. If so, it indicates that the data between the disks is relatively balanced. Then, during step S203, all disks of the data node are traversed to query whether there is a first disk with remaining available space that meets the amount of data to be stored. If so, the disk with the lowest performance index value is selected from the first disks as the target disk. The preset threshold is set to 10GB by default. If there are multiple disks with the lowest performance index values, the disk with the largest remaining available space is selected as the target disk.

[0069] Optionally, after calculating the remaining available space of the disk, the disk with the largest remaining available space and the disk with the smallest remaining available space are determined; it is determined whether the difference in remaining available space between the disk with the largest remaining available space and the disk with the smallest remaining available space is less than or equal to a preset threshold. If so, it means that the difference in remaining available space between other disks must be less than the preset threshold, and then the disk with the smallest performance index value and which meets the amount of data to be stored is selected as the target disk.

[0070] In some embodiments, after determining whether the difference in remaining available space between the disk with the largest remaining available space and the disk with the smallest remaining available space is less than or equal to a preset threshold, if the difference in remaining available space is greater than the preset threshold, it indicates that the data distribution among the disks is relatively dispersed. Then, the amount of data to be stored is determined, and it is determined whether the amount of data to be stored is greater than the available space of the disk with the largest remaining available space in the low available space disk group. If so, the disk with the smallest performance index is selected from other disks as the target disk. The other disks are the disks in the high available space disk group that meet the amount of data to be stored.

[0071] The division between low-availability disk groups and high-availability disk groups is based on the sum of the minimum remaining available space and a preset threshold. Optionally, after calculating the remaining available space of the disks, a minimum value (minimum remaining available space) is determined, and the sum of the minimum remaining available space and the preset threshold is calculated. Based on this, two disk groups are formed: those less than or equal to the sum are low-availability disk groups, and those greater than the sum are high-availability disk groups. It should be noted that even if multiple minimum remaining available spaces are determined after calculating the remaining available space, only one of these is selected and summed with the preset threshold (10G). This sum is then used to divide the disks into high-availability disk groups with relatively high remaining available space and low-availability disk groups with relatively low remaining available space.

[0072] Optionally, after determining whether the difference in remaining available space between the disk with the largest remaining available space and the disk with the smallest remaining available space is less than or equal to a preset threshold, all disks of the data node are traversed to query whether there is a first disk with remaining available space that meets the amount of data to be stored. If so, the following steps can be executed: after dividing the disks into high-availability disk groups and low-availability disk groups, it is determined whether the remaining available space of the disk with the largest remaining available space in the low-availability disk group can meet the amount of data to be stored. If not, it means that the disk cannot accommodate the amount of data to be stored, and the disk with the smallest performance index that meets the amount of data to be stored is selected from the high-availability disk group as the target disk.

[0073] Understandably, if the difference between the maximum and minimum remaining available space exceeds a preset threshold, the data distribution across the disks of that data node is uneven. Furthermore, if it is determined that the disk with the largest remaining available space in the low-availability disk group cannot store the current replica data, then disks in other disk groups with slightly larger remaining available space (i.e., high-availability disk groups) are considered. Disks with sufficient remaining available space to store the required amount of data are preferred, and then the disk with the lowest performance index is selected as the target disk. This target disk has a large remaining space but low activity level, thus storing all data and distributing I / O pressure. Table 5 shows the configuration items involved in the embodiments of this application.

[0074] Table 5

[0075]

[0076] In some embodiments, after determining whether the amount of data to be stored is greater than the available space of the disk with the largest remaining available space in the low-availability disk group, if the amount of data to be stored is less than or equal to the available space of the disk with the largest remaining available space in the low-availability disk group, it indicates that the disk with the largest remaining available space in the low-availability disk group can accommodate the data to be stored. Then, it is further determined whether the random probability is less than the selection probability of the high-availability volume group. The calculation formula for the selection probability of the high-availability volume group is shown in the following formula (2):

[0077]

[0078] Where p is the probability of selecting a high-availability space volume group, representing the probability of selecting a high-availability space disk group; m is the number of disks in the high-availability space disk group; p0 represents the probability of selecting a high-availability space volume, which is the configuration value of the balanced-space-preference-fraction configuration item as shown in Table 5; n is the number of disks in the low-availability space disk group; q is the probability of selecting a low-availability space volume, q = 1 - p0.

[0079] If the random probability is less than the selection probability of the high-availability space volume group, then the disk with the lowest performance metric is selected as the target disk from the other disks in the high-availability space disk group. These other disks are those that meet the required data storage volume. This means that even if the disk with the largest remaining available space in the low-availability space disk group can accommodate the data to be stored, considering the calculated selection probability of the high-availability space volume group, it is more preferable to select the target disk from the high-availability space disk group when the random probability is less than the high-availability space volume group selection probability. This is because if the disk with the largest remaining available space in the low-availability space disk group can accommodate the data to be stored, it means that all disks in the high-availability space disk group are available to accommodate the data to be stored, thus the disk with the lowest performance metric is selected as the target disk from the high-availability space disk group. If the random probability is greater than or equal to the selection probability of the high-availability space volume group, then the disk that meets the required data storage volume and has the lowest performance metric is selected as the target disk from the low-availability space disk group. This means that if the disk with the largest remaining available space in the low-availability disk group can accommodate the data to be stored, the random probability is determined by combining the calculated high-availability volume group selection probability. If not, then a portion of the disks in the low-availability disk group that meet the amount of data to be stored are selected first, and then the disk with the smallest performance index is selected as the target disk.

[0080] In summary, this application provides a disk selection method based on HDFS. This method first collects the input / output operation time of the disks on the data nodes, calculates the disk performance metrics based on the input / output operation time, and then selects the target disk based on the performance metrics. Thus, this application considers disk load when selecting a disk, calculates the disk performance metric value, and guides the data nodes to select suitable disks based on the performance metric value, achieving reasonable data allocation, balancing the busy levels of each disk, maintaining the data balance of the data nodes, and avoiding system bottlenecks caused by uneven data read / write operations.

[0081] like Figure 2B As shown, Figure 2B The second flowchart illustrates a disk selection method based on HDFS provided in this application embodiment. The method includes the following steps:

[0082] The disk selection policy is set according to the data node parameter `dfs.donatanode.fsdataset.volume.choosing.policy`. If the disk selection policy is a round-robin volume selection policy (`RoundRobinVolumeChoosingPolicy`), the next disk is selected in a round-robin fashion until the current round of disk selection is completed.

[0083] If the disk selection strategy is based on AvailableSpaceVolumeChoosingPolicy, then it checks whether the difference in remaining available space between the disk with the largest remaining space and the disk with the smallest remaining space is less than or equal to a preset threshold. Specifically, it checks whether the difference between the maximum remaining available space (Max) and the minimum remaining available space (Min) is less than or equal to the preset threshold (balanced-space-threshold). The default preset threshold is 10GB. If yes, it checks whether ioutil is enabled. If ioutil is not enabled, it selects disks based on a round-robin strategy. If ioutil is enabled, it checks if there is a disk with remaining available space greater than or equal to the amount of data to be stored. If not, the current round of disk selection ends. If it exists, it checks if there is a disk with a minimum ioutil value among the disks with remaining available space greater than or equal to the amount of data to be stored. If a minimum ioutil value exists, this disk with the minimum ioutil value and the maximum remaining available space is selected as the target disk, completing the current round of disk selection. If no minimum ioutil value exists, it checks if multiple disks have the same minimum ioutil value. If so, the disk with the largest remaining available space is selected as the target disk, completing the current round of disk selection.

[0084] If the difference in remaining available space between the disk with the largest remaining space and the disk with the smallest remaining space is greater than a preset threshold, then it is determined whether the available space of the disk with the largest remaining available space in the low-availability disk group is sufficient to store the required amount of data. If not, it is determined whether to enable ioutil. If ioutil is not enabled, a disk is selected from the high-availability disk group based on a round-robin disk selection strategy. If ioutil is enabled, the disk with the smallest ioutil that meets the required amount of data to be stored is selected from the high-availability disk group as the target disk, completing this round of disk selection. If the available space of the disk with the largest remaining available space in the low-availability disk group meets the required amount of data to be stored, it is determined whether the random probability is less than the high-availability volume group selection probability. If so, it is determined whether to enable ioutil. If ioutil is not enabled, a disk is selected from the high-availability disk group based on a round-robin disk selection strategy. If ioutil is enabled, the disk with the smallest ioutil is selected from the high-availability disk group as the target disk, completing this round of disk selection. If the random probability is greater than or equal to the high-availability volume group selection probability, it is determined whether to enable ioutil. If ioutil is not enabled, disks are selected from the low-availability disk group based on a round-robin strategy. If ioutil is enabled, the disk with the lowest ioutil value that meets the amount of data to be stored is selected from the low-availability disk group as the target disk, completing this round of disk selection.

[0085] like Figure 3 As shown, Figure 3 A schematic diagram of a disk selection device based on HDFS provided in this application embodiment, the device comprising:

[0086] The acquisition module 301 is used to acquire the input and output operation time of the disk on the data node;

[0087] The calculation module 302 is used to calculate the disk performance index value based on the input and output operation time. The performance index is used to describe the disk's busy level.

[0088] Select module 303 is used to select the target disk based on performance metric values.

[0089] As an optional implementation of this application, before selecting the target disk based on the performance index value, the selection module 303 is further configured to: obtain the disk capacity and usage; calculate the remaining available space of the disk; and determine whether the difference in the remaining available space between any two disks is less than or equal to a preset threshold.

[0090] The selection module 303 is specifically used for: if the difference in remaining available space between any two disks is less than or equal to a preset threshold, then traverse all disks of the data node to query whether there is a first disk with remaining available space that meets the amount of data to be stored; if so, then select the disk with the smallest performance index value from the first disk as the target disk.

[0091] As an optional implementation of this application, the selection module 303 is specifically used to: if there are multiple disks with the smallest performance index values ​​in the first disk, then the disk with the smallest performance index value and the largest remaining available space is selected as the target disk.

[0092] As an optional implementation of this application, the selection module 303 is specifically used to: if the difference in remaining available space between the disk with the largest remaining available space and the disk with the smallest remaining available space on the data node is greater than a preset threshold, then determine whether the amount of data to be stored is greater than the available space of the disk with the largest remaining available space in the low available space disk group; if so, select the disk with the smallest performance index from other disks as the target disk, and the other disks are disks in the high available space disk group that meet the size of the amount of data to be stored; wherein, the division of the low available space disk group and the high available space disk group is based on the sum of the smallest remaining available space and the preset threshold.

[0093] As an optional implementation of this application, the selection module 303, after determining whether the amount of data to be stored is greater than the available space of the disk with the largest remaining available space in the low-availability disk group, is further configured to: if the amount of data to be stored is less than or equal to the available space of the disk with the largest remaining available space in the low-availability disk group, determine whether the random probability is less than the selection probability of the high-availability volume group; if so, select the disk with the smallest performance index from the high-availability disk group as the target disk; if not, select the disk with the smallest performance index that meets the requirements of the amount of data to be stored from the low-availability disk group as the target disk.

[0094] This application provides a disk selection device based on HDFS. The device first collects the input / output operation time of the disks on the data nodes, calculates the disk performance indicators based on the input / output operation times, and then selects the target disk based on the performance indicators. Thus, this application considers disk load when selecting a disk, calculates the disk performance indicator value, and guides the data nodes to select suitable disks based on the performance indicator value, achieving reasonable data allocation, balancing the busy levels of each disk, maintaining the data balance of the data nodes, and avoiding system bottlenecks caused by uneven data read / write operations.

[0095] In one embodiment, this application provides an electronic device, which may be a terminal, and its internal structure diagram may be as follows. Figure 4As shown, the electronic device includes a processor, memory, communication interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a stuttering detection method. The display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0096] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0097] In one embodiment, the HDFS-based disk selection device provided in this application can be implemented as a computer program, which can be implemented in the form of, for example... Figure 4 The device operates on the electronic device shown. The memory of the electronic device can store various program modules that constitute the HDFS-based disk selection device. The computer program comprised of the various program modules causes the processor to execute the steps of the HDFS-based disk selection methods described in the various embodiments of this application.

[0098] In one embodiment, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by the computer program, implements the steps of the HDFS-based disk selection method of the various embodiments of this application described in this specification.

[0099] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0100] It should be understood, in the several embodiments provided in this application, that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0101] In this application, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0102] In this application, memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0103] In this application, computer-readable media includes both permanent and non-permanent, removable and non-removable storage media. Storage media can implement information storage using any method or technology; the information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient media, such as modulated data signals and carrier waves.

[0104] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0105] The above are merely specific embodiments of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to these embodiments, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A disk selection method based on HDFS, characterized in that, include: The time taken for disk input / output operations on data nodes is collected using the / proc / diskstats file in the Linux system. Based on the input / output operation time, the disk performance index value is calculated by dividing the difference between the input / output operation time of two adjacent samples by the sampling time interval. The performance index is used to describe the disk's busyness. Select the target disk based on the performance metric values; Before selecting a target disk based on the performance index value, the method further includes: obtaining the storage space and used space of the disk; calculating the remaining available space of the disk; and determining whether the difference in remaining available space between any two disks is less than or equal to a preset threshold. The step of selecting a target disk based on the performance index value includes: if the difference in remaining available space between any two disks is less than or equal to the preset threshold, then traverse all disks of the data node to query whether there is a first disk with remaining available space that meets the amount of data to be stored; if so, then select the disk with the smallest performance index value from the first disks as the target disk. The step of selecting the target disk based on the performance metric value includes: If the difference in remaining available space between the disk with the largest remaining available space and the disk with the smallest remaining available space on the data node is greater than the preset threshold, then it is determined whether the amount of data to be stored is greater than the available space of the disk with the largest remaining available space in the low-availability disk group; if so, the disk with the smallest performance index is selected from other disks as the target disk, and the other disks are disks in the high-availability disk group that meet the amount of data to be stored; wherein, the division between the low-availability disk group and the high-availability disk group is based on the sum of the smallest remaining available space and the preset threshold; If the amount of data to be stored is less than or equal to the available space of the disk with the largest remaining available space in the low-availability disk group, then determine whether the random probability is less than the selection probability of the high-availability volume group; if so, then select the disk with the smallest performance index from the high-availability disk group as the target disk; if not, then select the disk with the smallest performance index that meets the requirements of the amount of data to be stored from the low-availability disk group as the target disk.

2. A disk selection device based on HDFS, characterized in that, include: The acquisition module is used to collect the time taken for disk input / output operations on data nodes through the / proc / diskstats file of the Linux system; The calculation module is used to calculate the disk's performance index value by dividing the difference between the input and output operation times of two adjacent acquisitions by the sampling time interval, based on the input and output operation time consumption. The performance index is used to describe the disk's busyness. The selection module is used to select the target disk based on the performance metric values; The selection module is also used to obtain the storage space and used space of the disk; calculate the remaining available space of the disk; and determine whether the difference in remaining available space between any two disks is less than or equal to a preset threshold. If the difference in remaining available space between any two disks is less than or equal to the preset threshold, then all disks of the data node are traversed to query whether there is a first disk with remaining available space that meets the amount of data to be stored; if so, then the disk with the smallest performance index value among the first disks is selected as the target disk. If the difference in remaining available space between the disk with the largest remaining available space and the disk with the smallest remaining available space on the data node is greater than the preset threshold, then it is determined whether the amount of data to be stored is greater than the available space of the disk with the largest remaining available space in the low-availability disk group; if so, the disk with the smallest performance index is selected from other disks as the target disk, and the other disks are disks in the high-availability disk group that meet the amount of data to be stored; wherein, the division between the low-availability disk group and the high-availability disk group is based on the sum of the smallest remaining available space and the preset threshold; If the amount of data to be stored is less than or equal to the available space of the disk with the largest remaining available space in the low-availability disk group, then determine whether the random probability is less than the selection probability of the high-availability volume group; if so, then select the disk with the smallest performance index from the high-availability disk group as the target disk; if not, then select the disk with the smallest performance index that meets the requirements of the amount of data to be stored from the low-availability disk group as the target disk.

3. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the HDFS-based disk selection method as described in claim 1.

4. A computer-readable storage medium, characterized in that, include: A computer program is stored on the computer-readable storage medium, and when executed by a processor, the computer program implements the HDFS-based disk selection method as described in claim 1.

5. A computer program product, characterized in that, include: The computer program product includes a computer program that, when run on a computer, causes the computer to implement the HDFS-based disk selection method as described in claim 1.

Images