HDFS (Hadoop Distributed File System)-based disk selection method and device, equipment, medium and product

By collecting disk input and output operations in HDFS, calculating performance indicator values, and selecting the appropriate disk, the problem of data read and write imbalance in HDFS is solved, and data balance of data nodes and system performance is improved.

CN119088313AActive Publication Date: 2024-12-06CHINA ELECTRONICS CLOUD DIGITAL INTELLIGENCE TECH CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411232814.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-04
Publication Date
2024-12-06
Estimated Expiration
2044-09-04

AI Technical Summary

Technical Problem

In HDFS, in a long-running cluster, due to data deletion and disk hot-swap removal, the disk space storage may be uneven, resulting in uneven data reading and writing, and system bottlenecks.

Method used

By collecting the input and output operations of disks on the data node, calculate the performance indicator value of the disk, select the appropriate disk based on the performance indicator value, consider the disk load, realize reasonable allocation and balance of data, and avoid the data reading and writing imbalance caused by unbalanced data System bottleneck.

Benefits of technology

By calculating the performance indicator values ​​of the disk, guide the data nodes to select the appropriate disk, realize the reasonable allocation of data, balance the busyness of each disk, maintain the data balance state of the data nodes, and avoid system bottlenecks caused by unbalanced data reading and writing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088313A_ABST
    Figure CN119088313A_ABST
Patent Text Reader

Abstract

The invention relates to an HDFS-based disk selection method and device, equipment, a medium and a product, and particularly relates to the technical field of data processing. Comprises: collecting input and output operation time consumption of a disk on a data node; according to the input and output operation time consumption, calculating a performance index value of the disk, the performance index being used for describing the busy degree of the disk; and selecting a target disk according to the performance index value. According to the embodiment of the invention, when the disk is selected, the disk load is considered, the performance index value of the disk is calculated, the data node is guided to select the proper disk based on the performance index value, the reasonable distribution of data is realized, the busy degree of each disk is balanced, the data balance state of the data node is maintained, and the system bottleneck caused by unbalanced data read-write is avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a disk selection method, device, equipment, medium and product based on HDFS. Background Art

[0002] HDFS is a distributed file system that usually stores data on multiple disks on multiple DataNode nodes. HDFS provides two optional disk selection policies: one is the round robin volume selection policy, and the other is the available space volume selection policy.

[0003] The disk selection strategy based on polling selects disks one by one. After traversing all disks, it starts selecting again from the beginning to ensure that each disk has a chance to be selected, thus avoiding the situation where some disks are overused while other disks are idle. The disk selection strategy based on available space gives priority to disks with more free space to store new data blocks, thus avoiding data storage imbalance to a certain extent.

[0004] Although the polling volume storage method can ensure that every disk can be used, it may cause uneven storage of disk space in a long-term cluster due to data deletion and disk hot swapping. Using the selection strategy based on available space, each newly written data will enter the newly added empty disk, while other disks are idle during this period, which will cause a read and write bottleneck on the new disk. Summary of the invention

[0005] In order to solve the above technical problems or at least partially solve the above technical problems, the present application provides a disk selection method, device, equipment, medium and product based on HDFS, which can select a suitable disk to avoid system bottlenecks caused by unbalanced data reading and writing.

[0006] In order to achieve the above purpose, the technical solutions provided by the embodiments of the present application are as follows:

[0007] In a first aspect, the present application provides a disk selection method based on HDFS, including: collecting the input and output operation time of the disk on the data node; calculating the performance index value of the disk based on the input and output operation time, the performance index is used to describe the busyness of the disk; and selecting the target disk based on the performance index value.

[0008] As an optional implementation of the present application, before selecting the target disk according to the performance indicator value, the method also includes: obtaining the capacity and usage of the disk; calculating the remaining available space of the disk; determining whether the difference in the remaining available space between any two disks is less than or equal to a preset threshold; selecting the target disk according to the performance indicator value, including: if the difference in the remaining available space between any two disks is less than or equal to the preset threshold, traversing all disks of the data node to query whether there is a first disk with remaining available space that meets the amount of data to be stored; if so, determining from the first disk to select the disk with the smallest performance indicator value as the target disk.

[0009] As an optional implementation of the present application, the target disk is selected according to the performance index value, including: if there are multiple disks with the smallest performance index values ​​in the first disk, the disk with the smallest performance index value and the largest remaining available space is selected as the target disk.

[0010] As an optional implementation of the present application, a target disk is selected based on a performance index value, including: if the difference in remaining available space between the disk with the largest remaining available space and the disk with the smallest remaining available space on the data node is greater than a preset threshold, then determining whether the amount of data to be stored is greater than the available space of the disk with the largest remaining available space in the low-available space disk group; if so, selecting the disk with the smallest performance index from other disks as the target disk, the other disks being disks in the high-available space disk group that meet the size of the data to be stored; wherein the basis for dividing the low-available space disk and the high-available space disk group is: the sum of the minimum remaining available space and the preset threshold.

[0011] As an optional implementation of the present application, after determining whether the amount of data to be stored is greater than the available space of the disk with the largest remaining available space in the low-available space disk group, the method also includes: if the amount of data to be stored is less than or equal to the available space of the disk with the largest remaining available space in the low-available space disk group, determining whether the random probability is less than the probability of selecting a high-available space volume group; if so, selecting the disk with the smallest performance index from the high-available space disk group as the target disk; if not, selecting a disk that meets the amount of data to be stored and has the smallest performance index from the low-available space disk group as the target disk.

[0012] In a second aspect, the present application provides a disk selection device based on HDFS, the device comprising:

[0013] The collection module is used to collect the input and output operation time of the disk on the data node;

[0014] A calculation module is used to calculate the performance index value of the disk according to the input and output operation time. The performance index is used to describe the busyness of the disk;

[0015] The selection module is used to select a target disk based on a performance indicator value.

[0016] As an optional implementation of the present application, the selection module, before selecting the target disk according to the performance indicator value, is further used to: obtain the capacity and usage of the disk; calculate the remaining available space of the disk; determine whether the difference in the remaining available space between any two disks is less than or equal to a preset threshold;

[0017] The selection module is specifically used to: if the difference in remaining available space between any two disks is less than or equal to a preset threshold, traverse all disks of the data node to query whether there is a first disk with remaining available space that meets the amount of data to be stored; if so, determine and select the disk with the smallest performance indicator value from the first disk as the target disk.

[0018] As an optional implementation of the present application, the selection module is specifically used to: if there are multiple disks with the smallest performance index values ​​in the first disk, then the disk with the smallest performance index value and the largest remaining available space is used as the target disk.

[0019] As an optional implementation of the present application, the selection module is also used to: if the difference in remaining available space between the disk with the largest remaining available space and the disk with the smallest remaining available space on the data node is greater than a preset threshold, then determine whether the amount of data to be stored is greater than the available space of the disk with the largest remaining available space in the low-available space disk group; if so, select the disk with the smallest performance indicator from other disks as the target disk, and the other disks are disks in the high-available space disk group that meet the amount of data to be stored.

[0020] As an optional implementation of the present application, the selection module is also used for: after determining whether the amount of data to be stored is greater than the available space of the disk with the largest remaining available space in the low-available space disk group, it is also used for: if the amount of data to be stored is less than or equal to the available space of the disk with the largest remaining available space in the low-available space disk group, then determining whether the random probability is less than the selection probability of the high-available space volume group; if so, selecting the disk with the smallest performance index from the high-available space disk group as the target disk; if not, selecting the disk that meets the amount of data to be stored and has the smallest performance index from the low-available space disk group as the target disk.

[0021] In a third aspect, the present application provides an electronic device comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the HDFS-based disk selection method is implemented as described in the first aspect or any one of its optional embodiments.

[0022] In a fourth aspect, the present application provides a computer-readable storage medium, comprising: a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the HDFS-based disk selection method as described in the first aspect or any one of its optional embodiments.

[0023] In a fifth aspect, the present application provides a computer program product, including: the computer program product includes a computer program, and when the computer program runs on a computer, the computer implements the HDFS-based disk selection method as described in the first aspect or any one of its optional embodiments.

[0024] Compared with the prior art, the technical solution provided by the embodiments of the present application has the following advantages:

[0025] The disclosed embodiment provides a disk selection method based on HDFS, which first collects the input and output operation time of the disk on the data node, calculates the performance index of the disk according to the input and output operation time, and then selects the target disk according to the performance index. In this way, the application considers the disk load when selecting the disk, calculates the performance index value of the disk, guides the data node to select a suitable disk based on the performance index value, realizes the reasonable distribution of data, balances the busyness of each disk, maintains the data balance state of the data node, and avoids the system bottleneck caused by the imbalance of data reading and writing. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0028] Figure 1 Schematic diagram for polling-based disk selection strategy;

[0029] Figure 2A A schematic diagram of a disk selection method based on HDFS in an embodiment of the present application Figure 1 ;

[0030] Figure 2B FIG2 is a flow chart of a disk selection method based on HDFS according to an embodiment of the present application;

[0031] Figure 3 A schematic diagram of the structure of a disk selection device based on HDFS provided in an embodiment of the present application;

[0032] Figure 4 A schematic diagram of the structure of an electronic device described in an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the technical terms required to be used in the embodiments or the prior art description are briefly introduced below:

[0034] HDFS adopts a master-slave architecture, consisting of NameNode (NN) and DataNode. NameNode is the master node, which is mainly responsible for managing the HDFS namespace, parsing client requests, and controlling client access to HDFS. DataNode is a slave node, mainly used for physical storage, and its data volume depends on the size of the cluster.

[0035] NameNode is the master node of HDFS. Its main functions are: responsible for managing the HDFS namespace, cluster information and data blocks; maintaining the entire HDFS file directory tree, the metadata of the file directory and the list of data blocks corresponding to each file; accepting client operation requests; managing the mapping relationship between files and data blocks, and between data blocks and DataNodes.

[0036] DataNode is responsible for processing read and write requests from file system clients. It creates, copies, and deletes data blocks under the unified scheduling of NameNode. DataNode encapsulates a structure in memory: the data storage component (FsDataset), which is responsible for dynamically maintaining all data block (Block) sets on this node. Because Block files exist in various storage disks and directories, it is also responsible for maintaining the storage directory (i.e., disk). In DataNode, each storage directory is encapsulated into a volume (FsVolume), simulating the "mount point" under the Linux system. Under normal circumstances, FsVolume can be considered as an abstraction of the disk. Sometimes it is necessary to access the real disk (such as retrieving disk available space), so it is necessary for the abstraction to interact directly with the real disk, which is mainly implemented by calling respective Linux commands in Java.

[0037] In HDFS, Block, i.e. its replica data, is the basic storage unit of a file. Files are connected in an orderly and regular manner through Block file blocks. Replica data can be understood as the specific storage embodiment of Block. HDFS configures the size of Block through ${dfs.Blocksize}, which is 128MB by default. When a new Block is generated, the replica location determines the storage location of the replica in the cluster, which is determined by the location selection strategy and storage strategy.

[0038] There are currently two disk selection policies for writing data to HDFS: round-robin disk selection policy and remaining available space disk selection policy. The disk selection policy used is specified by the configuration item dfs.datanode.fsdataset.volume.choosing.policy. The default is round-robin volume selection policy. The configuration items are described in Table 1 below.

[0039] Table 1

[0040]

[0041] The RoundRobinVolumeChoosingPolicy strategy is relatively simple, which is to select each volume in turn in a round-robin manner. Figure 1 As shown in the figure, there are two new replicas, Replica1 and Replica2, both of which need to be stored in the storage directory belonging to DISK. Using the polling disk selection strategy, Replica1 will be placed in the directory belonging to DISK 1, and Replica2 will be placed in the directory belonging to DISK 2. The processing of different types of replicas is similar.

[0042] This strategy is based on the concept of "round robin", that is, disks are selected in a certain order. When the last disk is reached, the selection starts from the beginning, and so on, to ensure that each disk has a chance to be selected, so as to achieve a balanced distribution of data on the disk. The design goal of this strategy is to solve the problem of uneven disk data space, especially when dealing with too many small files, the number of blocks of each disk of the same DataNode is the same but the occupied disk size is uneven. The implementation of the polling strategy is based on a simple principle: select disks in turn, and when all disks are traversed, start again from the beginning to ensure that each disk has a chance to be selected, so as to avoid the situation where some disks are overused and other disks are idle. This strategy helps to improve the overall efficiency and reliability of the storage system, especially when dealing with a large number of small files, it can better balance the load of different disks.

[0043] The Available Space Volume Choosing Policy takes into account the remaining available space of the disk. This policy aims to optimize data distribution by giving priority to disks with more remaining space to store new data blocks, thereby avoiding data storage imbalance to a certain extent. Specifically, when the remaining available space of all disks in the optional disk list differs within a certain configured threshold (10GB by default), it means that the data volume between the disks is relatively uniform at this time. At this time, the policy degenerates to RoundRobinVolumeChoosingPolicy, that is, each disk is selected in turn in order.

[0044] In the above two disk selection strategies, the polling volume storage method can ensure that every disk is used, but in a long-running cluster, due to data deletion and disk hot swapping, it may cause uneven disk space storage. When the polling strategy is used in a long-running cluster, DataNode sometimes fills its storage directory (disk / volume) unevenly, resulting in some disks being full while other disks are rarely used. This may happen due to a large number of write and delete operations, or due to the replacement of disks. In addition, if we use a selection strategy based on free space, each newly written data will go to the newly added empty disk, leaving other disks idle during this period, which will create a read and write bottleneck on the new disk.

[0045] In addition, DataNode can be configured with multiple storage directories to store data block files. These storage directories can be heterogeneous, aiming to improve the data block IO throughput. The data blocks under each storage directory are managed by the FsVolumelmpl object, and the FsVolumeList class saves all FsVolumelmpl objects on the DataNode and provides disk-like services for FsDatasetlmpl. This design improves the data block IO throughput and also provides greater flexibility for data storage and management.

[0046] When implementing AvailableSpaceVolumeChoosingPolicy, you can adjust the dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction configuration item to control the allocation ratio of new data blocks to ensure that data can be more evenly distributed on different disks. The default value of this configuration item is 0.75f, which is generally sufficient. However, according to actual conditions and needs, this value can be adjusted to optimize the storage distribution of data.

[0047] In order to solve some or all of the technical problems existing in the above-mentioned related technologies, the embodiments of the present application provide a disk selection method, device, electronic device, medium and product based on HDFS, wherein the method first collects the input and output operation time of the disk on the data node, calculates the performance index of each disk according to the input and output operation time, and then selects the target disk according to the performance index. In this way, the present application considers the disk load when selecting the disk, calculates the performance index value of the disk, guides the data node to select the appropriate disk based on the performance index value, realizes the reasonable distribution of data, balances the busyness of each disk, maintains the data balance state of the data node, and avoids the system bottleneck caused by the imbalance of data reading and writing.

[0048] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0049] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0050] A disk selection method based on HDFS provided in the embodiments of the present application can be implemented by a disk selection device based on HDFS or an electronic device, and the electronic device includes but is not limited to a vehicle terminal, a server, a personal computer, a laptop computer, a tablet computer, a smart phone, etc. The operating system of the electronic device may include Android, a mobile operating system (iOS) developed by Apple, an operating system (Windows) developed by Microsoft Corporation of the United States, a Linux (Unix) operating system, etc., which is not limited by the embodiments of the present application. The electronic device can be run alone to implement the present application, or it can be connected to the network and implement the present application through interactive operations with other computer devices in the network. Among them, the network where the electronic device is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN) network, etc.

[0051] It should be noted that the protection scope of the HDFS-based disk selection method described in the embodiment of the present application is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present application are included in the protection scope of the present application.

[0052] like Figure 2A As shown, Figure 2AA schematic diagram of a disk selection method based on HDFS in an embodiment of the present application Figure 1 The method can be performed by a disk selection device based on HDFS, wherein the device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 2A As shown, the method mainly includes the following steps S201 to S203:

[0053] S201, collecting the time consumption of disk input and output operations on the data node.

[0054] In some embodiments, disk information is read from the / proc / diskstats file to collect the time spent doing I / O operations on the disk on the data node, as shown in Table 2, the time spent doing I / Os (ms) field. In Linux, the I / O status of the disk can be viewed through the iostat command. The / proc / diskstats file displays the I / O statistics of the block device. Each line contains the following 14 fields.

[0055] Table 2

[0056] Number of columns meaning 1 majornumber Major device number 2 minormumber Minor device number 3 devicename Device Name 4 reads completed successfully The total number of successful read completions 5 readsmerged Merge read times 6 sectorsread Number of read sectors 7 timespentreading(ms) Reading time 8 writescompleted The total number of successful write completions 9 writesmerged Merge write times 10 sectorswritten Number of sectors written 11 timespentwriting(ms) Time to write 12 I / Os currently in progress The number of I / Os currently being processed 13 timespentdoingI / Os(ms) Input and output time (ms) 14 weightedtimespentdoingI / Os(ms) Weighted number of milliseconds spent on input / output operations

[0057] In some embodiments, a configuration item as shown in Table 3 is added to the configuration file hdfs-site.xml to enable the ioutil function. If the ioutil function is not enabled, the RoundRobinVolumeChoosingPolicy policy is used to select the target disk.

[0058] Table 3

[0059]

[0060] S202: Calculate the performance index value of each disk according to the input and output operation time.

[0061] In some embodiments, the time difference between adjacent input and output operations is calculated, and the performance index value ioutil of the disk is calculated in combination with the sampling time interval, as shown in the following formula (1). The performance index is used to describe the busyness of the disk. The larger the performance index value, the busier the disk.

[0062]

[0063] In formula (1), Δt represents the time difference between adjacent input and output operations, and T is the sampling time interval. An exemplary sampling time interval is 1 second, which is set by the configuration items shown in Table 4 below.

[0064] Table 4

[0065]

[0066] In some embodiments, when the data node is started, after the storageLocations of DiskIOUtilManager is set, the DiskIOUtilManager thread is started, in which the StorageLocation class saves the URI and Storage Type (SSD / DISK / RAM) of a storage directory. The device name corresponding to the directory (for example, / dev / sda, / dev / sdb) can be obtained through the path of the directory, and then a mapping between the device name and the IOStat object (a wrapper object that stores ioutil values) is established. After starting the DiskIOUtilManager thread, according to the default configuration IOUtilCollectInterval, collection will be performed every 1 second and the ioutil value will be calculated once. The calculated performance indicator value is stored corresponding to the device name and stored in the Map data structure.

[0067] S203: Select a target disk according to the performance indicator value.

[0068] The target disk has the smallest performance index value. In some embodiments, when executing step S203, the storage space and used space of the disk are obtained, and the remaining available space of the disk is calculated. The optional storage space minus the used space equals the remaining available space; it is determined whether the difference in the remaining available space between any two disks is less than a preset threshold. If so, it indicates that the data between the disks is relatively balanced. Then, when executing step S203, all disks of the data node are traversed to query whether there is a first disk with remaining available space that satisfies the amount of data to be stored; if so, the disk with the smallest performance index value is determined from the first disk as the target disk. Among them, the preset threshold is set to 10GB by default. If there are multiple disks with the smallest performance index values, the disk with the largest remaining available space is selected as the target disk.

[0069] Optionally, after calculating the remaining available space of the disks, determine the disk with the largest remaining available space and the disk with the smallest remaining available space; determine whether the difference in remaining available space between the disk with the largest remaining available space and the disk with the smallest remaining available space is less than or equal to a preset threshold; if so, it means that the difference in remaining available space between other disks must be less than the preset threshold, and then select the disk with the smallest performance index value and that meets the amount of data to be stored as the target disk.

[0070] In some embodiments, after determining whether the difference in remaining available space between the disk with the largest remaining available space and the disk with the smallest remaining available space is less than or equal to a preset threshold, if the difference in remaining available space is greater than the preset threshold, indicating that the data distribution between the disks is relatively dispersed, the amount of data to be stored is determined, and it is determined whether the amount of data to be stored is greater than the available space of the disk with the largest remaining available space in the low-available space disk group; if so, the disk with the smallest performance indicator is selected from other disks as the target disk, and the other disks are disks in the high-available space disk group that meet the amount of data to be stored.

[0071] The basis for dividing the low available space disk group and the high available space disk group is: the sum of the minimum remaining available space and the preset threshold. Optionally, after calculating the remaining available space of the disk, a minimum value (minimum remaining available space) is determined, and the sum of the minimum remaining available space and the preset threshold is calculated. Based on this, two disk groups are divided, and the one less than or equal to the sum is the low available space disk group, and the one greater than the sum is the high available space disk group. It should be noted that even if it is determined that there are multiple minimum remaining available spaces after calculating the remaining available space of the disk, only one remaining available space is selected and added to the preset threshold (10G), so as to divide the high available space disk group with relatively high remaining available space and the low available space disk group with relatively low remaining available space according to this sum.

[0072] Optionally, after determining whether the difference in remaining available space between the disk with the largest remaining available space and the disk with the smallest remaining available space is less than or equal to a preset threshold, first traverse all disks of the data node to query whether there is a first disk with remaining available space that satisfies the amount of data to be stored; if so, subsequent steps can be executed, after dividing the disk group with high available space and the disk group with low available space, determine whether the remaining available space of the disk with the largest remaining available space in the low available space disk group can satisfy the amount of data to be stored; if not, indicating that the disk cannot accommodate the amount of data to be stored, a disk that satisfies the amount of data to be stored and has the smallest performance indicator is selected from the high available disk group as the target disk.

[0073] It is understandable that if the difference between the maximum remaining available space and the minimum remaining available space is greater than the preset threshold, the data distribution of each disk of the data node is uneven, and it is judged that the disk with the largest remaining available space in the low available space disk group cannot store the current copy data, then consider the disks in other disk groups with slightly larger remaining available space (i.e., high available space disk group), preferably the disk with remaining available space that meets the amount of data to be stored, and then select the disk with the smallest performance index as the target disk, the target disk has a large remaining space and a low busyness, so as to store all data and disperse the I / O pressure. As shown in Table 5, the configuration items involved in the embodiment of the present application.

[0074] Table 5

[0075]

[0076] In some embodiments, after determining whether the amount of data to be stored is greater than the available space of the disk with the largest remaining available space in the low-available space disk group, if the amount of data to be stored is less than or equal to the available space of the disk with the largest remaining available space in the low-available space disk group, it means that the disk with the largest remaining available space in the low-available space disk group can accommodate the data to be stored, and then further determining whether the random probability is less than the probability of selecting a volume group with high available space, wherein the calculation formula for the probability of selecting a volume group with high available space is shown in the following formula (2):

[0077]

[0078] Wherein, p is the probability of selecting a high-availability space volume group, which indicates the probability of selecting a high-availability space disk group; m is the number of disks in the high-availability space disk group, p0 indicates the probability of selecting a high-availability space volume, which is the configuration value of the configuration item balanced-space-preference-fraction shown in Table 5; n is the number of disks in the low-availability space disk group, q is the probability of selecting a low-availability space volume, q=1-p0.

[0079] If the random probability is less than the probability of selecting a high-availability volume group, the disk with the smallest performance index is selected from other disks as the target disk, and the other disks are disks in the high-availability disk group that meet the amount of data to be stored. This means that even if the disk with the largest remaining available space in the low-availability disk group can accommodate the data to be stored, combined with the calculated probability of selecting a high-availability volume group, when the random probability is less than the probability of selecting a high-availability volume group, it is better to select the target disk from the high-availability disk group, because the disk with the largest remaining available space in the low-availability disk group can accommodate the data to be stored, which means that all disks in the high-availability disk group can accommodate the data to be stored, and then the disk with the smallest performance index is selected from the high-availability disk group as the target disk. If the random probability is greater than or equal to the probability of selecting a high-availability volume group, the disk with the smallest performance index that meets the amount of data to be stored and meets the minimum performance index is selected from the low-availability disk group as the target disk. Description: When the disk with the largest remaining available space in the low-available space disk group can accommodate the data to be stored, the calculated high-available space volume group selection probability is used to determine whether the random probability is less than the high-available space volume group selection probability. If not, first select some disks from the low-available space disk group that meet the amount of data to be stored, and then select the disk with the smallest performance index as the target disk.

[0080] In summary, the embodiment of the present application provides a disk selection method based on HDFS, which first collects the input and output operation time of the disk on the data node, calculates the performance index of the disk according to the input and output operation time, and then selects the target disk according to the performance index. In this way, the present application considers the disk load when selecting the disk, calculates the performance index value of the disk, guides the data node to select a suitable disk based on the performance index value, realizes the reasonable distribution of data, balances the busyness of each disk, maintains the data balance state of the data node, and avoids the system bottleneck caused by the imbalance of data reading and writing.

[0081] like Figure 2B As shown, Figure 2B A second flowchart of a disk selection method based on HDFS provided in an embodiment of the present application is provided. The method comprises the following steps:

[0082] Set the disk selection policy according to the data node parameter dfs.donatanode.fsdataset.volume.choosing.policy. If the disk selection policy is the round-robin disk selection policy RoundRobinVolumeChoosingPolicy, the next disk will be selected in rounds until the current round of disk selection ends.

[0083] If the disk selection policy is the disk selection policy based on available space AvailableSpaceVolumeChoosingPolicy, then determine whether the difference between the disk with the largest remaining space and the disk with the smallest remaining space is less than or equal to the preset threshold. Among them, whether the maximum remaining available space Max-minimum remaining available space Min is less than or equal to the preset threshold balanced-space-threshold. The default value of the preset threshold is 10G. If so, determine whether ioutil is enabled. If ioutil is not enabled, select disks based on the polling disk selection policy. If ioutil is enabled, determine whether there is a disk with a remaining available space greater than or equal to the amount of data to be stored. If not, this round of disk selection ends. If so, determine whether there is a disk with a remaining available space greater than or equal to the amount of data to be stored with a minimum ioutil. If there is a minimum ioutil, use the disk with a remaining available space greater than or equal to the amount of data to be stored and the smallest ioutil as the target disk to complete this round of disk selection. If there is no minimum ioutil, determine whether the ioutil of multiple disks is the same and all are minimum values. If so, select the disk with the largest remaining available space from these multiple disks as the target disk to complete this round of disk selection.

[0084] If the difference between the remaining available space of the disk with the largest remaining available space and the disk with the smallest remaining space is greater than the preset threshold, it is determined whether the available space of the disk with the largest remaining available space in the low-available space disk group meets the amount of data to be stored. If not, it is determined whether to enable ioutil. If ioutil is not enabled, a disk is selected from the high-available space disk group based on the polling disk selection strategy. If ioutil is enabled, a disk that meets the amount of data to be stored and has the smallest ioutil is selected from the high-available space disk group as the target disk to complete this round of disk selection. If the available space of the disk with the largest remaining available space in the low-available space disk group meets the amount of data to be stored, it is determined whether the random probability is less than the high-available space volume group selection probability. If so, it is determined whether to enable ioutil. If ioutil is not enabled, a disk is selected from the high-available space disk group based on the polling disk selection strategy. If ioutil is enabled, a disk with the smallest ioutil is selected from the high-available space disk group as the target disk to complete this round of disk selection. If the random probability is greater than or equal to the high-available space volume group selection probability, it is determined whether to enable ioutil. If ioutil is not enabled, disk selection is performed from the low available space disk group based on the round-robin disk selection strategy. If ioutil is enabled, a disk that meets the required data volume and has the smallest ioutil is selected as the target disk from the low available space disk group to complete this round of disk selection.

[0085] like Figure 3 As shown, Figure 3 A schematic diagram of the structure of a disk selection device based on HDFS provided in an embodiment of the present application, the device comprising:

[0086] The collection module 301 is used to collect the input and output operation time of the disk on the data node;

[0087] A calculation module 302 is used to calculate a performance index value of the disk according to the input and output operation time consumption, and the performance index is used to describe the busyness of the disk;

[0088] The selection module 303 is used to select a target disk according to the performance indicator value.

[0089] As an optional implementation of the present application, the selection module 303, before selecting the target disk according to the performance indicator value, is further used to: obtain the capacity and usage of the disk; calculate the remaining available space of the disk; determine whether the difference in the remaining available space between any two disks is less than or equal to a preset threshold;

[0090] The selection module 303 is specifically used to: if the difference in the remaining available space between any two disks is less than or equal to a preset threshold, traverse all the disks of the data node to query whether there is a first disk with remaining available space that meets the amount of data to be stored; if so, determine from the first disks to select the disk with the smallest performance indicator value as the target disk.

[0091] As an optional implementation of the present application, the selection module 303 is specifically used to: if there are multiple disks with the smallest performance index values ​​in the first disk, then use the disk with the smallest performance index value and the largest remaining available space as the target disk.

[0092] As an optional implementation of the present application, the selection module 303 is specifically used to: if the difference in remaining available space between the disk with the largest remaining available space and the disk with the smallest remaining available space on the data node is greater than a preset threshold, then determine whether the amount of data to be stored is greater than the available space of the disk with the largest remaining available space in the low-available space disk group; if so, select the disk with the smallest performance index from other disks as the target disk, and the other disks are disks in the high-available space disk group that meet the size of the data to be stored; wherein the basis for dividing the low-available space disk and the high-available space disk group is: the sum of the minimum remaining available space and the preset threshold.

[0093] As an optional implementation of the present application, the selection module 303, after determining whether the amount of data to be stored is greater than the available space of the disk with the largest remaining available space in the low-available space disk group, is also used to: if the amount of data to be stored is less than or equal to the available space of the disk with the largest remaining available space in the low-available space disk group, determine whether the random probability is less than the selection probability of the high-available space volume group; if so, select the disk with the smallest performance index from the high-available space disk group as the target disk; if not, select the disk that meets the amount of data to be stored and has the smallest performance index from the low-available space disk group as the target disk.

[0094] The embodiment of the present application provides a disk selection device based on HDFS, which first collects the input and output operation time of the disk on the data node, calculates the performance index of the disk according to the input and output operation time, and then selects the target disk according to the performance index. In this way, the present application considers the disk load when selecting the disk, calculates the performance index value of the disk, guides the data node to select a suitable disk based on the performance index value, realizes the reasonable distribution of data, balances the busyness of each disk, maintains the data balance state of the data node, and avoids the system bottleneck caused by the imbalance of data reading and writing.

[0095] In one embodiment, the present application provides an electronic device, which may be a terminal, and its internal structure diagram may be as follows: Figure 4As shown. The electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected by a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, an operator network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a method for detecting a jam is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a button, a trackball or a touchpad provided on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0096] Those skilled in the art will understand that Figure 4 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0097] In one embodiment, the HDFS-based disk selection device provided in the present application can be implemented in the form of a computer program. The computer program can be used in Figure 4 The computer program composed of each program module enables the processor to execute the steps of the disk selection method based on HDFS in each embodiment of the present application described in this specification.

[0098] In one embodiment, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by the computer program, implements the steps of the HDFS-based disk selection method of each embodiment of the present application described in this specification.

[0099] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code.

[0100] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0101] In the present application, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0102] In this application, memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0103] In this application, computer-readable media includes permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0104] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0105] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A disk selection method based on HDFS, It is characterized in that include: The disk input and output operations on the data collection node are time-consuming; Calculate the performance index value of the disk according to the time consumed by the input and output operations, where the performance index is used to describe the busyness of the disk; A target disk is selected according to the performance indicator value.

2. The method according to claim 1, It is characterized in that Before selecting the target disk according to the performance indicator value, the method further includes: Obtaining the storage space and usage space of the disk; Calculating the remaining available space of the disk; Determine whether the difference in remaining available space between any two disks is less than or equal to a preset threshold; The step of selecting a target disk according to the performance indicator value includes: If the difference between the remaining available space of any two disks is less than or equal to the preset threshold, all disks of the data node are traversed to check whether there is a first disk with remaining available space that satisfies the amount of data to be stored; If so, a disk with the smallest performance index value is determined and selected from the first disks as the target disk.

3. The method according to claim 2, It is characterized in that The step of selecting a target disk according to the performance indicator value includes: If there are multiple disks with the smallest performance index values ​​in the first disk, the disk with the smallest performance index value and the largest remaining available space is used as the target disk.

4. The method according to claim 2, It is characterized in that The step of selecting a target disk according to the performance indicator value includes: If the difference in remaining available space between the disk with the largest remaining available space and the disk with the smallest remaining available space on the data node is greater than the preset threshold, then determining whether the amount of data to be stored is greater than the available space of the disk with the largest remaining available space in the low-available-space disk group; If yes, select a disk with the smallest performance index from other disks as the target disk, and the other disks are disks in a high-available space disk group that meet the amount of data to be stored; The low available space disk group and the high available space disk group are divided based on the sum of the minimum remaining available space and the preset threshold.

5. The method according to claim 4, It is characterized in that After determining whether the amount of data to be stored is greater than the available space of the disk with the largest remaining available space in the low available space disk group, the method further includes: If the amount of data to be stored is less than or equal to the available space of the disk with the largest remaining available space in the low available space disk group, then determine whether the random probability is less than the probability of selecting a high available space volume group; If yes, select the disk with the smallest performance index from the high-available-space disk group as the target disk; If not, a disk that meets the amount of data to be stored and has the smallest performance index is selected from the low-free-space disk group as the target disk.

6. A disk selection device based on HDFS, It is characterized in that include: The collection module is used to collect the input and output operation time of the disk on the data node; A calculation module, used to calculate the performance index value of the disk according to the input and output operation time consumption, and the performance index is used to describe the busyness of the disk; The selection module is used to select a target disk according to the performance indicator value.

7. The device according to claim 6, It is characterized in that The selection module, before selecting the target disk according to the performance indicator value, is further used to: obtain the capacity and usage of the disk; calculate the remaining available space of the disk; and determine whether the difference in the remaining available space between any two disks is less than or equal to a preset threshold; The selection module is specifically used to: if the difference between the remaining available space of any two disks is less than or equal to the preset threshold, traverse all disks of the data node to query whether there is a first disk with remaining available space that satisfies the amount of data to be stored; If so, a disk with the smallest performance index value is determined and selected from the first disks as the target disk.

8. An electronic device, It is characterized in that include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the HDFS-based disk selection method as described in any one of claims 1 to 5 is implemented.

9. A computer-readable storage medium, It is characterized in that include: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the HDFS-based disk selection method according to any one of claims 1 to 5 is implemented.

10. A computer program product, It is characterized in that include: The computer program product comprises a computer program, and when the computer program is run on a computer, the computer is enabled to implement the disk selection method based on HDFS as claimed in any one of claims 1 to 5.

Citation Information

Patent Citations

  • Multiple disc management method and equipment

    CN102541460A

  • Data writing method and device of distributed file system

    CN106649401A

  • Kubernetes cluster container group scheduling method and device and medium

    CN116244085A

  • Reduction of maximum latency using dynamic self-tuning for redundant array of independent disks

    US20160334999A1

  • Smart Volume Manager for Storage Space Usage Optimization

    US20170090774A1