Distributed storage water level adjusting method and device, computer device and medium

Abstract

The application relates to the technical field of distributed storage, and discloses a distributed storage water level adjusting method and device, computer equipment and a medium, the adjusting method comprising the following steps: obtaining a preset cluster information and a preset cluster water level to obtain a preset data amount of a target cluster; performing data embedding on the target cluster according to the preset data amount; judging whether a water level outside a preset cluster water level deviation range exists in each disk of the target cluster; if at least one disk has a water level outside the deviation range, adjusting the preset data amount or adjusting the weight of a disk bearing the preset data corresponding to the water level outside the deviation range according to the deviation between the water level outside the deviation range and the preset cluster water level, building a new target cluster according to the preset cluster information, and performing data embedding on the new target cluster. Through the application, manual operation of an operator in a water level test process to make the water levels of all the disks in the cluster reach an ideal range is avoided.

Description

Technical Field

[0001] This invention relates to the field of distributed storage technology, and specifically to a distributed storage water level adjustment method, device, computer equipment, and medium. Background Technology

[0002] Distributed storage, as a data storage technology, disperses data across various device nodes in a cluster. Each node contains multiple disks, and the storage performance of each disk collectively determines the overall storage performance of the cluster. A water level test is typically performed to determine the storage performance of the cluster and its individual disks. The water level refers to the percentage of used storage space on each storage device relative to the total storage capacity. During a water level test, a certain amount of data is written to each disk while it is empty to adjust the water level. Once the water level of each disk reaches the ideal range, performance tests are then conducted on the cluster and its individual disks. This process of writing data to empty disks is called data pre-embedding. Typically, operators need to repeatedly adjust the amount of pre-embedded data and perform data pre-embedding to bring the water level of each disk to the ideal range, with each pre-embedding process taking tens of minutes or even several hours. Manually performing such operations is time-consuming and labor-intensive, representing a significant burden for operators. Summary of the Invention

[0003] In view of this, the present invention provides a distributed storage water level adjustment method, device, computer equipment and medium to solve the problem that operators need to manually adjust the amount of pre-embedded data and perform data pre-embedding operations, which are time-consuming and laborious, during the water level testing process.

[0004] In a first aspect, the present invention provides a distributed storage water level adjustment method, which is applied to nodes in a target cluster, and includes:

[0005] The amount of pre-embedded data for the target cluster is obtained based on the preset cluster information and preset cluster water level;

[0006] Data is pre-embedded in the target cluster according to the amount of pre-embedded data;

[0007] Obtain the water level of each disk in the target cluster and determine whether there is a water level outside the preset cluster water level deviation range for each disk.

[0008] If at least one disk's water level is outside the deviation range, adjust the amount of pre-embedded data or adjust the weight of the disk corresponding to the water level outside the deviation range that carries the pre-embedded data according to the deviation between the water level outside the deviation range and the preset cluster water level. Build a new target cluster according to the preset cluster information and pre-embed data for the new target cluster.

[0009] The distributed storage water level adjustment method provided by this invention avoids the time-consuming and laborious manual operation required by operators during water level testing to ensure that each disk in the cluster reaches the ideal range, thereby improving the overall efficiency of water level testing and reducing the burden on operators.

[0010] In one optional embodiment, the distributed storage water level adjustment method provided by the present invention further includes:

[0011] If the water level of each disk is within the deviation range, the distributed storage water level adjustment method ends.

[0012] This invention enables the distributed storage level adjustment method to terminate directly when the level of each disk is within the deviation range. This avoids the problem of only addressing situations where the level of at least one disk is outside the deviation range, while neglecting to address situations where the level of all disks is within the deviation range.

[0013] In one optional implementation, adjusting the amount of pre-embedded data or adjusting the weight of the disk carrying the pre-embedded data corresponding to water levels outside the deviation range includes:

[0014] If the water level of at least one disk in each disk is outside the deviation range and all water levels outside the deviation range are higher than the preset cluster water level, sort the water levels outside the deviation range in descending order to obtain the highest water level outside the deviation range, and reduce the amount of pre-buried data based on the difference between the highest water level and the deviation range.

[0015] If the water level of at least one disk in each disk is outside the deviation range and all water levels outside the deviation range are lower than the preset cluster water level, sort the water levels outside the deviation range in descending order to obtain the lowest water level outside the deviation range, and increase the amount of pre-buried data according to the difference between the lowest water level and the deviation range.

[0016] If at least two disks in each disk have water levels outside the deviation range, and the water levels outside the deviation range include both water levels higher than the preset cluster water level and water levels lower than the preset cluster water level, sort the water levels outside the deviation range in descending order to obtain the highest and lowest water levels outside the deviation range. Reduce the weight of the disk corresponding to the highest water level that carries the pre-embedded data, and increase the weight of the disk corresponding to the lowest water level that carries the pre-embedded data.

[0017] By determining whether there are any water levels outside the preset cluster water level range in each disk's water level, the system automatically adjusts the amount of pre-embedded data or the weight of the pre-embedded data on the disk corresponding to the water level outside the deviation range based on the possible judgment results. This avoids errors caused by manual operation and improves the overall efficiency of water level testing.

[0018] In one optional implementation, before obtaining the pre-embedded data volume of the target cluster based on preset cluster information and preset cluster water level, the following steps are included:

[0019] Build the target cluster based on the preset cluster information.

[0020] During the water level test, data needs to be pre-installed on all disks of the target cluster while they are empty. Some disks in an existing target cluster may not be empty; performing a water level test on such a cluster would reduce the overall accuracy of the test. By building a completely new target cluster based on pre-defined cluster information, where all disks are empty, the accuracy of the overall water level test is ensured.

[0021] In one optional implementation, the preset cluster information includes:

[0022] The number of disks in the target cluster, the storage capacity of each disk in the target cluster, and the percentage of the actual storage capacity of the target cluster.

[0023] The number of disks in the target cluster, the storage capacity of each disk in the target cluster, and the percentage of the actual storage capacity of the target cluster included in the preset cluster information are necessary information for calculating the amount of pre-embedded data in the target cluster. The existence of this information provides a basis for calculating the amount of pre-embedded data in the target cluster.

[0024] In one optional implementation, the amount of pre-embedded data for the target cluster is obtained based on preset cluster information and preset cluster water level, including:

[0025] The storage capacity of the target cluster is obtained based on the number of disks and the storage capacity of each disk.

[0026] The actual storage capacity of the target cluster is obtained by comparing the target cluster's storage capacity with the percentage of actual storage capacity.

[0027] The amount of data to be embedded is determined based on the actual storage capacity of the target cluster and the preset cluster water level.

[0028] By using the preset cluster water level and the preset cluster information including the number of disks in the target cluster, the storage space capacity of each disk in the target cluster, and the percentage of the actual storage space capacity of the target cluster, the amount of pre-embedded data in the target cluster can be calculated relatively accurately, ensuring the accuracy of the overall water level test.

[0029] In one optional implementation, the process further includes the following steps before building a new target cluster based on preset cluster information:

[0030] Clear the data in the target cluster;

[0031] Delete the target cluster.

[0032] The data pre-embedded in the target cluster will occupy a large amount of storage space in the target cluster. Before building a new target cluster based on the preset cluster information, clearing the data in the old target cluster and deleting the old target cluster can free up storage space for building the new target cluster and avoid affecting the subsequent data pre-embedding in the new target cluster.

[0033] In one optional embodiment, the distributed storage water level adjustment method provided by the present invention further includes:

[0034] The target cluster is subjected to performance testing according to the preset cluster water level, and the performance test results data are obtained.

[0035] Determine the performance of the target cluster based on the results data.

[0036] Since the performance of each disk collectively determines the performance of the entire cluster, when the level of each disk is adjusted to the ideal range, a performance test is performed on the target cluster. By comprehensively analyzing the various values ​​in the test results data, the performance of the target cluster can be evaluated relatively accurately.

[0037] In one alternative implementation, the performance test results data include:

[0038] The target cluster's bandwidth, target cluster's latency, and the bandwidth and latency of each disk in the target cluster.

[0039] The performance test results include the target cluster's bandwidth, target cluster latency, and the bandwidth and latency of each disk in the target cluster. These values ​​reflect the performance of the target cluster and its individual disks. By comprehensively analyzing these values, the performance of the target cluster can be evaluated relatively accurately.

[0040] In a second aspect, the present invention provides a distributed storage water level adjustment device, which is applied to nodes in a target cluster, and the device includes:

[0041] The cluster management module is used to build new target clusters based on preset cluster information;

[0042] The data pre-embedding module is used to obtain the amount of pre-embedding data for the target cluster based on the preset cluster information and preset cluster water level, and to pre-embedding data for the target cluster according to the amount of pre-embedding data.

[0043] The water level analysis module is used to obtain the water level of each disk in the target cluster and determine whether there is a water level outside the deviation range of the preset cluster water level. If the water level of at least one disk is outside the deviation range, the module adjusts the amount of pre-embedded data or the weight of the disk corresponding to the water level outside the deviation range carrying the pre-embedded data according to the deviation between the water level outside the deviation range and the preset cluster water level. The module also notifies the cluster management module to build a new target cluster according to the preset cluster information and notifies the data pre-embedding module to pre-embed data in the new target cluster according to the amount of pre-embedded data.

[0044] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the distributed storage water level adjustment method of the first aspect or any corresponding embodiment described above.

[0045] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the distributed storage level adjustment method of the first aspect or any corresponding embodiment described above. Attached Figure Description

[0046] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating a distributed storage water level adjustment method according to an embodiment of the present invention;

[0048] Figure 2 This is a flowchart illustrating another distributed storage water level adjustment method according to an embodiment of the present invention;

[0049] Figure 3 This is a flowchart illustrating another distributed storage water level adjustment method according to an embodiment of the present invention;

[0050] Figure 4 This is a flowchart illustrating another distributed storage water level adjustment method according to an embodiment of the present invention;

[0051] Figure 5 This is a schematic diagram of the working process of a distributed storage water level regulation system according to an embodiment of the present invention;

[0052] Figure 6This is a structural block diagram of a distributed storage water level regulating device according to an embodiment of the present invention;

[0053] Figure 7 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Distributed storage, as a data storage technology, disperses data across various device nodes in a cluster. Each node contains multiple disks, and the storage performance of each disk collectively determines the overall storage performance of the cluster. Water level testing is typically performed to determine the storage performance of the cluster and its individual disks. During water level testing, a certain amount of data is pre-loaded onto each disk with an empty disk to adjust the water level. Once the water level of each disk reaches the ideal range, performance testing is then conducted on the cluster and its individual disks. In general, it is necessary to repeatedly adjust the pre-loaded data and perform data pre-loading to adjust the water level of each disk to the ideal range, with each pre-loading taking tens of minutes or even several hours. Manually performing such operations is time-consuming and labor-intensive, placing a significant burden on operators.

[0056] This invention provides a distributed storage water level adjustment method. This method avoids the time-consuming and laborious manual operation required by operators during water level testing to ensure that each disk in the cluster reaches the ideal range, thereby improving the overall efficiency of water level testing and reducing the burden on operators.

[0057] According to an embodiment of the present invention, a distributed storage water level adjustment method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0058] This embodiment provides a distributed storage water level adjustment method, which can be used on nodes in a target cluster. Figure 1 This is a flowchart of a distributed storage water level adjustment method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0059] Step S101: Obtain the amount of pre-embedded data for the target cluster based on the preset cluster information and preset cluster water level.

[0060] The preset cluster information includes, but is not limited to, the number of disks in the target cluster, the storage capacity of each disk in the target cluster, and the percentage of the target cluster's actual storage capacity. The target cluster's storage capacity can be obtained from the number of disks and the storage capacity of each disk. The actual storage capacity of the target cluster can be obtained from the percentage of the target cluster's actual storage capacity. The amount of data to be embedded can be obtained from the actual storage capacity and the preset cluster water level.

[0061] Step S102: Pre-embed data for the target cluster according to the pre-embedded data volume.

[0062] Data is pre-embedded in the target cluster according to the pre-embedded data volume, and the pre-embedded data will be stored in a distributed manner on various disks of the target cluster.

[0063] Step S103: Obtain the water level of each disk in the target cluster and determine whether there is a water level outside the preset cluster water level deviation range in each disk.

[0064] Assuming the preset cluster water level is 80%, and the deviation range of the preset cluster water level is -5% to +5%, i.e., 75% to 85%. After pre-installing data, the water levels of the three disks in the target cluster are 74%, 80%, and 86%, respectively. 74% is less than 75%, and 86% is greater than 85%, meaning 74% and 86% are outside the deviation range of the preset cluster water level; 80% is between 75% and 85%, meaning 80% is within the deviation range of the preset cluster water level. For each disk in the target cluster, if the water level of each disk is within the deviation range, it indicates that the water level of each disk has been adjusted to the ideal range; otherwise, it indicates that the water level of the disks outside the deviation range still needs adjustment.

[0065] Step S104: If the water level of at least one disk is outside the deviation range, adjust the amount of pre-embedded data or adjust the weight of the disk carrying the pre-embedded data corresponding to the water level outside the deviation range according to the deviation between the water level outside the deviation range and the preset cluster water level. Build a new target cluster according to the preset cluster information and pre-embed data for the new target cluster.

[0066] Adjusting the amount of pre-embedded data will affect the water level of each disk during subsequent water level adjustment. Adjusting the weight of the disks carrying pre-embedded data corresponding to water levels outside the deviation range will affect the amount of pre-embedded data carried by these disks during subsequent water level adjustment. The weight of a disk carrying pre-embedded data is directly proportional to the amount of pre-embedded data the disk can carry; that is, the higher the weight, the larger the amount of pre-embedded data carried, and the lower the weight, the smaller the amount of pre-embedded data carried.

[0067] The amount of pre-embedded data cannot be adjusted by adding or deleting data after the data has been pre-embedded; the weight of the disks carrying the pre-embedded data corresponding to water levels outside the deviation range can only be adjusted when building the cluster. Therefore, it is necessary to build a new target cluster based on the preset cluster information and pre-embed data in the new target cluster so that the water levels of each disk in the target cluster can eventually be adjusted to within the ideal range.

[0068] If the water level of each disk is within the deviation range, it means that the water level of each disk has been adjusted to the ideal range, and the water level adjustment can be ended at this time.

[0069] The distributed storage water level adjustment method provided in this embodiment can automatically adjust the amount of pre-buried data or the weight of the disk carrying the pre-buried data corresponding to the water level outside the preset cluster water level deviation range, and automatically perform subsequent related operations to ensure that the water level of each disk in the target cluster is ultimately adjusted to the ideal range. This avoids the time-consuming and laborious manual operation by operators to ensure that each disk in the cluster reaches the ideal range, improves the overall efficiency of water level testing, and reduces the burden on operators.

[0070] In step S104 above, adjusting the amount of pre-embedded data or adjusting the weight of the disk carrying the pre-embedded data corresponding to the water level outside the deviation range includes:

[0071] If the water level of at least one disk in each disk is outside the deviation range and all water levels outside the deviation range are higher than the preset cluster water level, sort the water levels outside the deviation range in descending order to obtain the highest water level outside the deviation range, and reduce the amount of pre-buried data based on the difference between the highest water level and the deviation range.

[0072] For example, the preset cluster water level is 80%, and the deviation range of the preset cluster water level is -5% to +5%, i.e., 75% to 85%, with a pre-embedded data volume of 'a'. After pre-embedding the data, three disks in the target cluster have water levels outside the deviation range and higher than the preset cluster water level, namely 86%, 87%, and 88%, with 88% being the highest. The deviation between 88% and 85% is 3%. Therefore, the amount of pre-embedded data that needs to be reduced is (a ÷ 88% × 3%).

[0073] If the water level of at least one disk in each disk is outside the deviation range and all water levels outside the deviation range are lower than the preset cluster water level, sort the water levels outside the deviation range in descending order to obtain the lowest water level outside the deviation range, and increase the amount of pre-buried data according to the difference between the lowest water level and the deviation range.

[0074] For example, the preset cluster water level is 80%, and the deviation range of the preset cluster water level is -5% to +5%, i.e., 75% to 85%, with a pre-embedded data volume of 'a'. After pre-embedding the data, three disks in the target cluster have water levels outside the deviation range and lower than the preset cluster water level, namely 72%, 73%, and 74%, with 72% being the lowest. The deviation between 72% and 75% is 3%. Therefore, the amount of data that needs to be added to the pre-embedded data volume is (a ÷ 72% × 3%).

[0075] If at least two disks in each disk have water levels outside the deviation range, and the water levels outside the deviation range include both water levels higher than the preset cluster water level and water levels lower than the preset cluster water level, sort the water levels outside the deviation range in descending order to obtain the highest and lowest water levels outside the deviation range. Reduce the weight of the disk corresponding to the highest water level that carries the pre-embedded data, and increase the weight of the disk corresponding to the lowest water level that carries the pre-embedded data.

[0076] For example, the preset cluster water level is 80%, and the deviation range of the preset cluster water level is -5% to +5%, i.e., 75% to 85%. The weight of each disk in the target cluster carrying the pre-embedded data is usually 100% by default. After pre-embedding the data, one disk in the target cluster has a water level of 86%, which is outside the deviation range and higher than the preset cluster water level, while another disk has a water level of 74%, which is outside the deviation range and lower than the preset cluster water level. The weight of the disk corresponding to the 86% water level is reduced to (100% - (86% - 85%)), i.e., 99%, while the weight of the disk corresponding to the 74% water level is increased to (100% + 75% - 74%), i.e., 101%.

[0077] By determining whether there are any water levels outside the preset cluster water level range in each disk's water level, the amount of pre-embedded data or the weight of the pre-embedded data carried by the disks corresponding to water levels outside the deviation range in each disk is automatically adjusted based on the possible judgment results. This avoids errors caused by manual operation and improves the overall efficiency of water level testing.

[0078] In step S104 above, before building a new target cluster based on preset cluster information, the following steps are also included:

[0079] Clear the data in the target cluster and delete the target cluster.

[0080] The data pre-embedded in the target cluster will occupy a large amount of storage space in the target cluster. Before building a new target cluster based on the preset cluster information, clearing the data in the old target cluster and deleting the old target cluster can free up storage space for building the new target cluster and avoid affecting the subsequent data pre-embedding in the new target cluster.

[0081] This embodiment provides a distributed storage water level adjustment method, which can be used on nodes in the aforementioned target cluster. Figure 2 This is a flowchart of another distributed storage water level adjustment method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0082] Step S201: Build the target cluster according to the preset cluster information.

[0083] During the water level test, data needs to be pre-installed on all disks of the target cluster while they are empty. Some disks in an existing target cluster may not be empty; performing a water level test on such a cluster would reduce the overall accuracy of the test. By building a completely new target cluster based on pre-defined cluster information, where all disks are empty, the accuracy of the overall water level test is ensured.

[0084] Step S202: Obtain the pre-embedded data volume of the target cluster based on the preset cluster information and preset cluster water level. For details, please refer to [link to relevant documentation]. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0085] Step S203: Pre-embed data for the target cluster according to the pre-embedded data volume. For details, please refer to [link to relevant documentation]. Figure 1 Step S102 of the illustrated embodiment will not be described again here.

[0086] Step S204: Obtain the water level of each disk in the target cluster and determine whether any water level on any disk is outside the preset cluster water level deviation range. For details, please refer to [link to relevant documentation]. Figure 1Step S103 of the illustrated embodiment will not be described again here.

[0087] Step S205: If at least one disk's water level is outside the deviation range, adjust the amount of pre-buried data or adjust the weight of the disk corresponding to the water level outside the deviation range that carries the pre-buried data based on the deviation between the water level outside the deviation range and the preset cluster water level. Then, build a new target cluster based on the preset cluster information and pre-buy data for the new target cluster. For details, please refer to [link to relevant documentation]. Figure 1 Step S104 of the illustrated embodiment will not be described again here.

[0088] This embodiment provides a distributed storage water level adjustment method, which can be used on nodes in the aforementioned target cluster. Figure 3 This is a flowchart of another distributed storage water level adjustment method according to an embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps:

[0089] Step S301: Obtain the amount of pre-embedded data for the target cluster based on the preset cluster information and the preset cluster water level. The preset cluster information includes the number of disks in the target cluster, the storage capacity of each disk in the target cluster, and the percentage of the actual storage capacity of the target cluster.

[0090] Specifically, step S301 includes:

[0091] Step S3011: Obtain the storage space capacity of the target cluster based on the number of disks and the storage space capacity of each disk.

[0092] Step S3012: Obtain the actual storage capacity of the target cluster based on the storage capacity of the target cluster and the percentage of actual storage capacity.

[0093] Step S3013: Obtain the amount of pre-buried data based on the actual storage space capacity of the target cluster and the preset cluster water level.

[0094] The following examples will be used to explain steps S3011, S3012 and S3013.

[0095] Assume the target cluster contains 3 disks, each with a storage capacity of 1TB, the actual storage capacity of the target cluster accounts for 2 / 3, and the preset cluster water level is 80%.

[0096] Given that the target cluster contains 3 disks and each disk has a storage capacity of 1T, the storage capacity of the target cluster is (3×1T), which is 3T.

[0097] Based on the target cluster's storage capacity of 3T and the actual storage capacity ratio of 2 / 3, the actual storage capacity of the target cluster can be calculated as (2 / 3 × 3T), which is 2T.

[0098] Based on the actual storage capacity of the target cluster being 2T and the preset cluster water level being 80%, the amount of data to be embedded is (2T × 80%), which is 1.6T.

[0099] Step S302: Pre-embed data for the target cluster according to the pre-embedded data volume. For details, please refer to [link / reference]. Figure 1 Step S102 of the illustrated embodiment will not be described again here.

[0100] Step S303: Obtain the water level of each disk in the target cluster and determine whether any water level on any disk is outside the preset cluster water level deviation range. For details, please refer to [link to relevant documentation]. Figure 1 Step S103 of the illustrated embodiment will not be described again here.

[0101] Step S304: If at least one disk's water level is outside the deviation range, adjust the amount of pre-buried data or adjust the weight of the disk corresponding to the water level outside the deviation range that carries the pre-buried data based on the deviation between the water level outside the deviation range and the preset cluster water level. Then, build a new target cluster based on the preset cluster information and pre-buy data for the new target cluster. For details, please refer to [link to relevant documentation]. Figure 1 Step S104 of the illustrated embodiment will not be described again here.

[0102] This embodiment provides a distributed storage water level adjustment method, which can be used on nodes in the aforementioned target cluster. Figure 4 This is a flowchart of another distributed storage water level adjustment method according to an embodiment of the present invention, such as... Figure 4 As shown, the process includes the following steps:

[0103] Step S401: Obtain the pre-embedded data volume of the target cluster based on the preset cluster information and preset cluster water level. For details, please refer to [link to relevant documentation]. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0104] Step S402: Pre-embed data for the target cluster according to the pre-embedded data volume. For details, please refer to [link / reference]. Figure 1 Step S102 of the illustrated embodiment will not be described again here.

[0105] Step S403: Obtain the water level of each disk in the target cluster and determine whether any water level on any disk is outside the preset cluster water level deviation range. For details, please refer to [link to relevant documentation]. Figure 1 Step S103 of the illustrated embodiment will not be described again here.

[0106] Step S404: If at least one disk's water level is outside the deviation range, adjust the amount of pre-buried data or adjust the weight of the disks corresponding to the water levels outside the deviation range that carry pre-buried data based on the deviation between the water levels outside the deviation range and the preset cluster water levels. Then, build a new target cluster based on the preset cluster information and pre-buy data for the new target cluster. For details, please refer to [link to relevant documentation]. Figure 1 Step S104 of the illustrated embodiment will not be described again here.

[0107] Step S405: Perform a performance test on the target cluster according to the preset cluster water level, and obtain the performance test result data.

[0108] Step S406: Determine the performance of the target cluster based on the result data.

[0109] The performance test results include the target cluster's bandwidth, latency, and the bandwidth and latency of each disk in the target cluster. These data reflect the performance of the target cluster and its individual disks. By comprehensively analyzing these test results, the performance of the target cluster can be evaluated relatively accurately.

[0110] The embodiments of the present invention can be implemented in the following application scenarios in the following manner.

[0111] This embodiment provides a distributed storage water level regulation system, including a cluster management module, a data pre-embedding module, and a water level analysis module. The three modules of this system are configured according to... Figure 5 The flowchart shown illustrates distributed storage water level adjustment. Combined with... Figure 5 The above-described distributed storage water level adjustment process will be illustrated by example.

[0112] The target cluster can be built using the cluster management module based on the preset cluster information.

[0113] The data pre-embedding module determines the amount of pre-embedded data for the target cluster based on preset cluster information and a preset cluster water level. The preset cluster water level is 80%, and the preset cluster information includes: 3 disks in the target cluster, each disk has a storage capacity of 1TB, and the deployment strategy is a 2+1 erasure policy. Based on the 2+1 erasure policy, the actual storage space capacity percentage of the target cluster is determined to be 2 / 3. Therefore, the target cluster's storage space capacity is (3 × 1TB), which is 3TB. Consequently, the actual storage space capacity is (2 / 3 × 3TB), which is 2TB. Finally, the pre-embedded data amount is (2TB × 80%), which is 1.6TB.

[0114] The data pre-embedding module pre-embedding data into the target cluster at a volume of 1.6T. After the data pre-embedding is completed, the water level analysis module obtains the water level of each disk in the target cluster and determines whether there is a water level outside the preset cluster water level deviation range, which is 75% to 85%.

[0115] If at least one disk in the target cluster has a water level outside the deviation range, and all water levels outside the deviation range are higher than the preset cluster water level, sort the water levels outside the deviation range from highest to lowest, and obtain the highest water level outside the deviation range. Reduce the amount of pre-buried data based on the difference between the highest water level and the deviation range. For example, if the water levels of three disks in the target cluster are 86%, 87%, and 88%, respectively, they are all outside the deviation range of 75% to 85% and higher than the preset cluster water level of 80%. 88% is the highest water level among them, and the deviation between 88% and 85% is 3%, then the amount of pre-buried data that needs to be reduced is (3T ÷ 88% × 3%), which is 0.102T.

[0116] If at least one disk in the target cluster has a water level outside the deviation range, and all water levels outside the deviation range are lower than the preset cluster water level, sort the water levels outside the deviation range from highest to lowest, and obtain the lowest water level outside the deviation range. Increase the amount of pre-buried data by the difference between the lowest water level and the deviation range. For example, if the water levels of three disks in the target cluster are 72%, 73%, and 74%, respectively, they are all outside the deviation range of 75% to 85% and lower than the preset cluster water level of 80%. 72% is the lowest water level among them, and the deviation between 72% and 75% is 3%, then the amount of pre-buried data that needs to be increased is (3T ÷ 72% × 3%), which is 0.125T.

[0117] If at least two disks in each disk have water levels outside the deviation range, and the water levels outside the deviation range include both water levels higher than the preset cluster water level and water levels lower than the preset cluster water level, sort the water levels outside the deviation range in descending order to obtain the highest and lowest water levels outside the deviation range. Reduce the weight of the disk corresponding to the highest water level that carries the pre-embedded data, and increase the weight of the disk corresponding to the lowest water level that carries the pre-embedded data.

[0118] For example, the water levels of three disks in the target cluster are 74%, 80%, and 86%, respectively. Among them, 86% is outside the deviation range of 75% to 85% and higher than the preset cluster water level of 80%, and 74% is outside the deviation range of 75% to 85% and lower than the preset cluster water level of 80%. The weight of the disk corresponding to the 86% water level carrying the pre-embedded data is reduced to (100% - (86% - 85%)), which is 99%, and the weight of the disk corresponding to the 74% water level carrying the pre-embedded data is increased to (100% + 75% - 74%), which is 101%.

[0119] After adjusting the amount of pre-embedded data or adjusting the weight of the disk carrying the pre-embedded data corresponding to the water level outside the deviation range through the water level analysis module, the data in the target cluster is cleared and the target cluster is deleted. A new target cluster is built according to the preset cluster information and data is pre-embedded in the new target cluster.

[0120] If the water level of each disk is within the deviation range, it indicates that the water level of each disk has been adjusted to the ideal range. At this point, stop water level adjustment and use the cluster management module to perform performance tests on the target cluster according to the preset cluster water level. The performance test results include the bandwidth of the target cluster, the latency of the target cluster, the bandwidth of each disk in the target cluster, and the latency of each disk in the target cluster. These data reflect the performance of the target cluster and its individual disks. By comprehensively analyzing these test results, the performance of the target cluster can be evaluated relatively accurately.

[0121] This embodiment also provides a distributed storage water level regulating device, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0122] This embodiment provides a distributed storage water level regulation device, such as... Figure 6 As shown, it includes:

[0123] The cluster management module 601 is used to build a new target cluster based on preset cluster information.

[0124] The data pre-embedding module 602 is used to obtain the amount of pre-embedding data for the target cluster based on the preset cluster information and the preset cluster water level, and to pre-embedding data for the target cluster according to the amount of pre-embedding data.

[0125] The water level analysis module 603 is used to obtain the water level of each disk in the target cluster and determine whether there is a water level outside the deviation range of the preset cluster water level. If the water level of at least one disk is outside the deviation range, the module adjusts the amount of pre-embedded data or the weight of the disk corresponding to the water level outside the deviation range carrying the pre-embedded data according to the deviation between the water level outside the deviation range and the preset cluster water level. The module notifies the cluster management module 601 to build a new target cluster according to the preset cluster information and notifies the data pre-embedding module 602 to pre-embed data in the new target cluster according to the amount of pre-embedded data.

[0126] In some optional implementations, the cluster management module 601 is also used to build the target cluster according to the preset cluster information before the data pre-embedding module 602 obtains the amount of pre-embedding data of the target cluster according to the preset cluster information and the preset cluster water level.

[0127] In some optional implementations, the cluster management module 601 is also used to clear the data in the target cluster and delete the target cluster after receiving a notification from the water level analysis module 603 to build a new target cluster based on preset cluster information.

[0128] In some optional implementations, the cluster management module 601 further includes:

[0129] The performance testing unit is used to perform performance tests on the target cluster according to a preset cluster water level, obtain performance test result data, and determine the performance of the target cluster based on the result data. The performance test result data includes: the bandwidth of the target cluster, the latency of the target cluster, the bandwidth of each disk in the target cluster, and the latency of each disk in the target cluster.

[0130] In some optional implementations, the data pre-embedding module 602 is used to obtain the pre-embedding data volume of the target cluster based on preset cluster information and preset cluster water level, including:

[0131] The preset cluster information includes the number of disks in the target cluster, the storage capacity of each disk in the target cluster, and the percentage of the actual storage capacity of the target cluster.

[0132] The storage capacity of the target cluster is obtained based on the number of disks and the storage capacity of each disk.

[0133] The actual storage capacity of the target cluster is obtained by comparing the target cluster's storage capacity with the percentage of actual storage capacity.

[0134] The amount of data to be embedded is determined based on the actual storage capacity of the target cluster and the preset cluster water level.

[0135] In some optional implementations, the water level analysis module 603 is used to adjust the amount of pre-embedded data or the weight of the disk-borne pre-embedded data corresponding to water levels outside the deviation range, including:

[0136] If the water level of at least one disk in each disk is outside the deviation range and all water levels outside the deviation range are higher than the preset cluster water level, sort the water levels outside the deviation range in descending order to obtain the highest water level outside the deviation range, and reduce the amount of pre-buried data based on the difference between the highest water level and the deviation range.

[0137] If the water level of at least one disk in each disk is outside the deviation range and all water levels outside the deviation range are lower than the preset cluster water level, sort the water levels outside the deviation range in descending order to obtain the lowest water level outside the deviation range, and increase the amount of pre-buried data according to the difference between the lowest water level and the deviation range.

[0138] If at least two disks in each disk have water levels outside the deviation range, and the water levels outside the deviation range include both water levels higher than the preset cluster water level and water levels lower than the preset cluster water level, sort the water levels outside the deviation range in descending order to obtain the highest and lowest water levels outside the deviation range. Reduce the weight of the disk corresponding to the highest water level that carries the pre-embedded data, and increase the weight of the disk corresponding to the lowest water level that carries the pre-embedded data.

[0139] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0140] In this embodiment, the distributed storage water level adjustment device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0141] This invention also provides a computer device having the above-described features. Figure 6 The distributed storage water level regulation device shown.

[0142] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 7 As shown, the computer device includes one or more processors 701, memory 702, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 7 Take the 701 processor as an example.

[0143] Processor 701 may be a central processing unit, a network processor, or a combination thereof. Processor 701 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GPA), or any combination thereof.

[0144] The memory 702 stores instructions executable by at least one processor 701 to cause at least one processor 701 to perform the method shown in the above embodiments.

[0145] The memory 702 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 702 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 702 may optionally include memory remotely located relative to the processor 701, and these remote memories can be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0146] The memory 702 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state hard disk; the memory 702 may also include a combination of the above types of memory.

[0147] The computer device also includes a communication interface 703 for communicating with other devices or communication networks.

[0148] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state hard disk, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0149] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A distributed storage water level regulation method, characterized in that, The method, applied to nodes in the target cluster, includes: The amount of pre-embedded data for the target cluster is obtained based on the preset cluster information and the preset cluster water level; Data is pre-embedded in the target cluster according to the pre-embedded data volume; Obtain the water level of each disk in the target cluster, and determine whether there is a water level outside the deviation range of the preset cluster water level among the water levels of each disk; If the water level of at least one of the disks is outside the deviation range, the amount of pre-embedded data or the weight of the disk corresponding to the water level outside the deviation range carrying the pre-embedded data is adjusted according to the deviation between the water level outside the deviation range and the preset cluster water level. A new target cluster is built according to the preset cluster information, and data is pre-embedded in the new target cluster so that the water level of each disk in the target cluster can eventually be adjusted to within the deviation range of the preset cluster water level.

2. The distributed storage water level regulation method according to claim 1, characterized in that, The method further includes: If the water level of each disk is within the deviation range, the distributed storage water level adjustment method ends.

3. The distributed storage water level regulation method according to claim 1 or 2, characterized in that, The adjustment of the amount of pre-embedded data or the adjustment of the weight of the disk carrying the pre-embedded data corresponding to the water level outside the deviation range includes: If the water level of at least one of the disks is outside the deviation range and all water levels outside the deviation range are higher than the preset cluster water level, sort the water levels outside the deviation range in descending order to obtain the highest water level outside the deviation range, and reduce the amount of pre-embedded data based on the difference between the highest water level and the deviation range. If the water level of at least one of the disks is outside the deviation range and all water levels outside the deviation range are lower than the preset cluster water level, sort the water levels outside the deviation range in descending order to obtain the lowest water level outside the deviation range, and increase the amount of pre-embedded data according to the difference between the lowest water level and the deviation range. If the water level of at least two disks in the disks is outside the deviation range, and the water level outside the deviation range includes both water levels higher than the preset cluster water level and water levels lower than the preset cluster water level, the water levels outside the deviation range are sorted in descending order to obtain the highest and lowest water levels outside the deviation range. The weight of the disk corresponding to the highest water level in carrying the pre-embedded data is reduced, and the weight of the disk corresponding to the lowest water level in carrying the pre-embedded data is increased.

4. The distributed storage water level regulation method according to claim 1 or 2, characterized in that, Before obtaining the pre-embedded data volume of the target cluster based on preset cluster information and preset cluster water level, the process includes: The target cluster is built based on the preset cluster information.

5. The distributed storage water level regulation method according to claim 1 or 2, characterized in that, The preset cluster information includes: The number of disks in the target cluster, the storage capacity of each disk in the target cluster, and the percentage of the actual storage capacity of the target cluster.

6. The distributed storage water level adjustment method according to claim 5, characterized in that, The step of obtaining the pre-embedded data volume of the target cluster based on preset cluster information and preset cluster water level includes: The storage capacity of the target cluster is obtained based on the number of disks and the storage capacity of each disk; The actual storage capacity of the target cluster is obtained based on the storage capacity of the target cluster and the percentage of the actual storage capacity. The amount of pre-embedded data is obtained based on the actual storage space capacity of the target cluster and the preset cluster water level.

7. The distributed storage water level regulation method according to claim 1 or 2, characterized in that, Before building a new target cluster based on the preset cluster information, the following steps are also included: Clear the data in the target cluster; Delete the target cluster.

8. The distributed storage water level adjustment method according to claim 1 or 2, characterized in that, The method further includes: The target cluster is subjected to performance testing according to the preset cluster water level, and the performance test result data is obtained. The performance of the target cluster is determined based on the resulting data.

9. The distributed storage water level adjustment method according to claim 8, characterized in that, The performance test results include: The bandwidth of the target cluster, the latency of the target cluster, the bandwidth of each disk in the target cluster, and the latency of each disk in the target cluster.

10. A distributed storage water level regulating device, characterized in that, The device is applied to nodes in a target cluster and includes: The cluster management module is used to build new target clusters based on preset cluster information; The data pre-embedding module is used to obtain the amount of pre-embedded data for the target cluster based on the preset cluster information and the preset cluster water level, and to pre-embed the data for the target cluster according to the amount of pre-embedded data. The water level analysis module is used to obtain the water level of each disk in the target cluster, determine whether there is a water level outside the deviation range of the preset cluster water level; if the water level of at least one disk is outside the deviation range, adjust the amount of pre-embedded data or adjust the weight of the disk corresponding to the water level outside the deviation range carrying the pre-embedded data according to the deviation between the water level outside the deviation range and the preset cluster water level, notify the cluster management module to build a new target cluster according to the preset cluster information, and notify the data pre-embedding module to pre-embed data in the new target cluster according to the pre-embedded data amount, so that the water level of each disk in the target cluster can eventually be adjusted to within the deviation range of the preset cluster water level.

11. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the steps of the distributed storage water level adjustment method according to any one of claims 1 to 9 by executing the computer instructions.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the steps of the distributed storage water level adjustment method according to any one of claims 1 to 9.

Images