Node resource scheduling method and device of distributed storage cluster and medium
By obtaining load information in a distributed storage cluster and performing prediction model analysis, adjusting the number of nodes in the node pool, the problem of insufficient resource utilization is solved, and flexible and reliable resource scheduling and scientific resource allocation are achieved.
Patent Information
- Application Number
- CN202411251682.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-06
- Publication Date
- 2024-11-15
AI Technical Summary
In actual applications, distributed storage clusters cannot accurately predict the utilization of node resources, resulting in the inability to fully and effectively utilize resources, affecting cluster performance.
By obtaining the load information of each node in the distributed storage cluster and inputting it into a pre-established prediction model, the resource usage prediction for days, months, and years are carried out, and the number of nodes in the annual, monthly, and daytime backup node pool is adjusted. To achieve flexible and reliable scheduling of node resources.
Multi-level prediction and scheduling of node resources of distributed storage clusters is realized, ensuring sufficient resources are provided in the high-traffic stage, timely release resources in the low-traffic stage, avoid waste, and achieve scientific and reasonable resource allocation.
Smart Images

Figure CN118963969A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of distributed storage technology, and in particular to a node resource scheduling method, device and medium for a distributed storage cluster. Background Art
[0002] Distributed storage technology is widely used in various application scenarios due to its advantages such as improving the reliability, availability and access efficiency of storage systems and being easy to expand. It covers multiple fields such as big data storage, cloud storage services, data backup and recovery, large-scale file sharing, data analysis and mining.
[0003] However, in actual application scenarios of distributed storage clusters, insufficient utilization of node resources in the cluster or insufficient cluster node resources at special times may occur. If the resource utilization of nodes cannot be accurately predicted and resource scheduling cannot be carried out in a targeted manner, the resources of the distributed storage cluster cannot be fully and effectively utilized, affecting the performance of the cluster.
[0004] Therefore, technicians in this field are in urgent need of a node resource scheduling method for a distributed storage cluster to solve the problem that in the actual application of the distributed storage cluster, it is currently impossible to accurately predict the node resource utilization and schedule resources in advance. Summary of the invention
[0005] The purpose of the present invention is to provide a node resource scheduling method, device and medium for a distributed storage cluster to solve the problem that the node resource utilization cannot be accurately predicted and resource scheduling cannot be performed in advance in the actual application of the distributed storage cluster.
[0006] In order to solve the above technical problems, the present invention provides a node resource scheduling method for a distributed storage cluster, comprising:
[0007] Obtain the load information of each node in the distributed storage cluster;
[0008] The load information is input into a pre-established prediction model with a period of day, month and year respectively, so as to obtain the resource usage prediction results of the distributed storage cluster for the next day, next month and next year;
[0009] At the end of the current year, adjusting the number of nodes in the pre-established annual standby node pool based on the resource usage forecast results for the next year;
[0010] At the end of the month, the number of nodes in the pre-established monthly standby node pool is adjusted according to the resource usage forecast result of the next month; if the monthly standby node pool needs to add new nodes, they are retrieved from the annual standby node pool; if the monthly standby node pool needs to release nodes, the released nodes are added to the annual standby node pool;
[0011] At 24:00 on the same day, the number of nodes in the distributed storage cluster is adjusted according to the resource usage forecast results for the next day; if the distributed storage cluster needs to add new nodes, they are retrieved from the monthly standby node pool; if the distributed storage cluster needs to release nodes, the released nodes are added to the monthly standby node pool.
[0012] In a possible embodiment, the prediction model includes: a daily prediction model, a monthly prediction model and an annual prediction model;
[0013] The daily prediction model is obtained by training with historical daily data, the monthly prediction model is obtained by training with historical monthly data, and the annual prediction model is obtained by training with historical annual data.
[0014] The historical daily data is the load information corresponding to each node of the distributed storage cluster when the power consumption is the largest in a day, the historical monthly data is the load information corresponding to each node of the distributed storage cluster when the power consumption is the largest in a month, and the historical year data is the load information corresponding to each node of the distributed storage cluster when the power consumption is the largest in a year;
[0015] The load information is input into a pre-established prediction model with a period of day, month, and year, respectively, to obtain the resource usage prediction results of the distributed storage cluster for the next day, next month, and next year, including:
[0016] Inputting the load information into the daily prediction model on a daily basis to obtain the resource usage prediction result of the distributed storage cluster for the next day;
[0017] Inputting the load information into the monthly prediction model on a monthly basis to obtain the resource usage prediction result of the distributed storage cluster for the next month;
[0018] The load information is input into the annual prediction model on an annual basis to obtain the resource usage prediction result of the distributed storage cluster for the next year.
[0019] In a possible embodiment, the daily prediction model includes: a holiday prediction model and a non-holiday prediction model;
[0020] The holiday prediction model is obtained by training the historical day data corresponding to holidays, and the non-holiday prediction model is obtained by training the historical day data corresponding to non-holidays;
[0021] The load information is input into the daily prediction model on a daily basis to obtain the resource usage prediction result of the distributed storage cluster for the next day, including:
[0022] If the next day is a non-holiday, the load information is input into the non-holiday prediction model on a daily basis to obtain the resource usage prediction result of the distributed storage cluster for the next day;
[0023] If the next day is a holiday, the load information is input into the holiday prediction model on a daily basis to obtain the resource usage prediction result of the distributed storage cluster for the next day.
[0024] In a possible embodiment, inputting the load information into the daily prediction model on a daily basis to obtain the resource usage prediction result of the distributed storage cluster for the next day includes:
[0025] Inputting the load information into the daily prediction model on a daily basis, and using the output result of the daily prediction model as an intermediate prediction result;
[0026] If the next day is not a holiday, the intermediate prediction result is used as the resource usage prediction result of the distributed storage cluster for the next day;
[0027] If the next day is a holiday, the intermediate result is processed upward by a preset ratio to obtain the resource usage prediction result of the distributed storage cluster for the next day;
[0028] The preset ratio is obtained based on the historical day data corresponding to holidays and the historical day data corresponding to non-holidays.
[0029] In a possible embodiment, the prediction model is:
[0030] The forecast value of the next Nth period = the horizontal part of this period + N*the trend part of this period;
[0031] Where N is a positive integer and the period is days, months, or years;
[0032] The horizontal part of this cycle = the first smoothing coefficient * the actual value of demand in this cycle + (1-the first smoothing coefficient) * (the horizontal part of the previous cycle + the trend part of the previous cycle);
[0033] The trend part of this cycle = the second smoothing coefficient * (the horizontal part of this cycle - the horizontal part of the previous cycle) + (1-the second smoothing coefficient) * the trend part of the previous cycle;
[0034] The value ranges of the first smoothing coefficient and the second smoothing coefficient are both (0, 1); and the actual value of the demand in this period is a specific parameter value of the load information in this period.
[0035] In a possible embodiment, the initial value of the horizontal part in the prediction model is: the specific parameter value corresponding to the historical data of the first period in the time series in the historical data for training the prediction model;
[0036] The initial value of the trend part in the prediction model is: the specific parameter value corresponding to the historical data of the second period in time series in the historical data for training the prediction model.
[0037] In a possible embodiment, the initial value of the horizontal part in the prediction model is: the intercept of a pre-established target linear regression model;
[0038] The initial value of the trend part in the prediction model is: the slope of the target linear regression model;
[0039] The target linear regression model is a linear regression model established based on the historical data of the first M periods in time series in the historical data for training the prediction model; M is a positive integer greater than 1.
[0040] In order to solve the above technical problems, the present invention also provides a node resource scheduling device for a distributed storage cluster, comprising:
[0041] The load monitoring module is used to obtain the load information of each node in the distributed storage cluster;
[0042] A resource prediction module, used to input the load information into a pre-established prediction model with a period of day, month, and year, respectively, to obtain a resource usage prediction result of the distributed storage cluster for the next day, next month, and next year;
[0043] An annual adjustment module, used to adjust the number of nodes in the pre-established annual standby node pool at the end of the current year according to the resource usage forecast result for the next year;
[0044] A monthly adjustment module is used to adjust the number of nodes in the pre-established monthly standby node pool at the end of the month according to the resource usage forecast result of the next month; wherein, if the monthly standby node pool needs to add new nodes, they are retrieved from the annual standby node pool; if the standby node pool at the end of the month needs to release nodes, the released nodes are added to the annual standby node pool;
[0045] The daily adjustment module is used to adjust the number of nodes in the distributed storage cluster at 24:00 on the same day according to the resource usage prediction results of the next day; if the distributed storage cluster needs to add new nodes, they are retrieved from the monthly standby node pool; if the distributed storage cluster needs to release nodes, the released nodes are added to the monthly standby node pool.
[0046] In order to solve the above technical problems, the present invention also provides a node resource scheduling device for a distributed storage cluster, comprising:
[0047] Memory for storing computer programs;
[0048] The processor is used to implement the steps of the node resource scheduling method of the distributed storage cluster as described above when executing the computer program.
[0049] To solve the above technical problem, the present invention also provides a non-volatile storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the node resource scheduling method of the distributed storage cluster as described above are implemented.
[0050] The present invention provides a node resource scheduling method for a distributed storage cluster. By collecting the load information of each node in the distributed storage cluster and inputting it into a pre-established prediction model, the load situation of each node in the future period, that is, the resource usage situation, can be obtained, and then the resource adjustment can be carried out in a targeted manner to achieve full utilization of the node resources in the cluster within the prediction time period. Further, the method predicts and adjusts the node resources in three different time periods of day, month and year; the resource usage prediction results corresponding to the three different periods obtained by the prediction model are used to prepare the spare nodes that may be used in the next year in advance according to the prediction results of the next year, and form an annual spare node pool for calling at any time in the next year; the spare nodes that may be used in the next month are prepared in advance according to the prediction results of the next month, and the monthly spare node pool is formed, wherein the increase of nodes in the monthly spare node pool is obtained from the annual spare node pool, and the release of nodes is also based on the annual spare node pool; in addition, the number of nodes in the cluster is increased or released according to the prediction results of the next day, and the application of the next day has been prepared, and the increase and release of nodes in the cluster are based on the monthly spare node pool.
[0051] In summary, this method provides a multi-layer node resource prediction and scheduling mechanism with three time granularities, which progresses layer by layer from day to month to year. The node resource adjustment for the next day is based on the monthly standby node pool, and the node resource adjustment for the next month is based on the annual standby node pool, so as to achieve flexible and reliable node resource adjustment, which can cover the complete life cycle of the distributed storage cluster. Provide sufficient node resources for use during the high-traffic phase of the cluster, and release node resources in time during the low-traffic phase of the cluster to avoid resource waste and achieve scientific and reasonable node resource allocation.
[0052] The node resource scheduling device of the distributed storage cluster and the non-volatile storage medium provided by the present invention correspond to the above method and have the same effect as above. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0054] Figure 1 A flowchart of a node resource scheduling method for a distributed storage cluster provided by an embodiment of the present invention;
[0055] Figure 2 A schematic diagram of node resource scheduling provided by an embodiment of the present invention;
[0056] Figure 3 A schematic diagram of another node resource scheduling provided by an embodiment of the present invention;
[0057] Figure 4 A structural diagram of a node resource scheduling device for a distributed storage cluster provided by an embodiment of the present invention;
[0058] Figure 5 A structural diagram of another node resource scheduling device for a distributed storage cluster provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0060] The core of the present invention is to provide a node resource scheduling method, device and medium for a distributed storage cluster.
[0061] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0062] In the related technology, there are real-time monitoring methods for the usage rate of various components of the nodes in the distributed storage cluster, such as the central processing unit (CPU), memory, hard disk, network card and other components. When the usage rate of any component exceeds the preset threshold, an alarm can be issued to remind the operation and maintenance personnel to promptly solve the problem of insufficient resources in a certain node, so as to facilitate the operation and maintenance personnel to allocate node resources.
[0063] However, this solution is passive and lagging in the allocation of node resources, and is not predictable. Resource allocation can only be carried out after the node resources show insufficient characteristics. In addition, this allocation solution also relies on manual implementation by operation and maintenance personnel, which is not smart and convenient enough.
[0064] To solve the above problems, the present invention provides a node resource scheduling method for a distributed storage cluster, such as Figure 1 As shown, including:
[0065] S11: Obtain the load information of each node in the distributed storage cluster.
[0066] S12: Inputting the load information into a pre-established prediction model with a period of day, month, and year, respectively, to obtain a prediction result of resource usage of the distributed storage cluster for the next day, next month, and next year.
[0067] S13: At the end of the current year, the number of nodes in the pre-established annual standby node pool is adjusted based on the resource usage forecast results for the next year.
[0068] S14: At the end of the month, the number of nodes in the pre-established monthly standby node pool is adjusted according to the resource usage forecast result for the next month.
[0069] If the monthly standby node pool needs to add new nodes, they will be transferred from the annual standby node pool. If the monthly standby node pool needs to release nodes, the released nodes will be added to the annual standby node pool.
[0070] S15: At 24:00 on the same day, the number of nodes in the distributed storage cluster is adjusted according to the resource usage forecast result for the next day.
[0071] If the distributed storage cluster needs to add new nodes, they will be retrieved from the monthly standby node pool. If the distributed storage cluster needs to release nodes, the released nodes will be added to the monthly standby node pool.
[0072] The load information in step S11 is specifically information that can reflect the usage of node resources in the distributed storage cluster, that is, information that can reflect the node load. Specifically, as mentioned above, the resource usage of the node can be reflected by the usage rate of each component in the node. Exemplarily, the components and resources in the node include but are not limited to: CPU, memory, hard disk, network card, etc., then according to the usage rate of these components, the node's usage of these resources can be reflected. In addition, the bandwidth values of node components such as CPU, memory, hard disk, network card, etc. can also reflect the usage of these resources, which can also be used as a possible implementation plan for the above-mentioned load information.
[0073] It should be noted that although this embodiment does not limit the specific type of load information (such as the usage rate or bandwidth value of any component in the node), it also does not limit the number of values included in the load information (such as the load information may include the usage rate or bandwidth value of any component, or the usage rate or bandwidth value of multiple components, and may also include the usage rate and bandwidth value of multiple components).
[0074] In a possible implementation, the load information may include the above-mentioned information, such as the usage rate (and / or bandwidth value) of various components. This implementation will increase the computational difficulty in the subsequent prediction model training and prediction process, but can maximize the accuracy and resolution of the prediction of the node resource usage, resulting in better prediction and resource allocation effects.
[0075] However, considering that this method can generally achieve node resource allocation through the distributed storage cluster itself to avoid introducing additional processors, if this method requires too high a computational difficulty, it may have an adverse impact on the function of the distributed storage cluster itself. Therefore, from the perspective of simplifying the computational difficulty of subsequent prediction and resource allocation, a possible implementation scheme is to include only one parameter value in the load information.
[0076] Specifically, the CPU usage (or bandwidth value) can be selected as the parameter value of the load information. This is because in the actual application of the node, the CPU is generally the most important resource in the node, and the usage changes of other node components such as memory, hard disk, network card, etc. will also be reflected in the CPU usage. Therefore, the CPU usage can be used to represent the overall resource usage of the node.
[0077] In another possible embodiment, a parameter value may be finally obtained to characterize the resource usage of the entire node by performing data processing such as weighted calculation on the usage rate (or bandwidth value) of each component in the node.
[0078] For step S12, the resource usage of the distributed storage cluster for the next day, next month and next year is predicted based on the load information obtained in step S11 and the pre-established prediction model, and the resource usage prediction result is obtained. It should be noted that although this step describes the prediction process of three different time periods in one step. But in fact, it is not necessary to perform predictions of three different time periods every time a prediction is made. It can be seen from the contents of the subsequent steps S13~S15 that there are different triggering conditions for the allocation of node resources in different time periods. And it is not difficult to know that the prediction of resource usage in different time periods is to achieve node resource allocation in different time periods. Therefore, the prediction of different time periods in step S12 should be adapted to the trigger conditions of steps S13~S15. For example, the prediction of node resource usage for the next day can be performed only once a day, the prediction of the next month can be performed only once a month, and the prediction of the next year can be performed only once a year.
[0079] It should also be noted that this embodiment does not impose strict restrictions on the prediction model in step S12. In this embodiment, the prediction model is used to predict the amount of node resources required when the node resources are reasonably applied, or the number of standard nodes required, and other output results that can indirectly reflect the amount of node resources required based on load information reflecting the use of node resources as input. Among them, the standard node is a node whose node resource amount meets the preset standard. There are currently many solutions that can use historical samples of a certain data to perform model training to obtain corresponding outputs, such as various machine learning algorithms, artificial neural networks, models, etc., and this embodiment does not limit this.
[0080] As for steps S13~S15, as mentioned above, each has its own triggering condition, so there is no order requirement for the execution of steps S13~S15, and the execution is triggered when the corresponding triggering condition is met. It should be noted that the annual standby node pool mentioned in step S13 is a pre-created node pool, in which standby nodes are stored. That is, at the initial deployment of the distributed storage cluster, in addition to the nodes required for the deployment of the cluster, it is also necessary to reserve some standby nodes in advance. According to the different deployment cycles of these standby nodes, they belong to the annual standby node pool and the monthly standby node pool. Among them, since the node deployment for the next day occurs at 24:00 on the same day and needs to be put into use immediately, there is no need to set up an additional standby node pool for the "day" cycle, and the number of nodes in the cluster can be directly deployed according to the corresponding resource usage prediction results.
[0081] "Day" belongs to the time period of "month", so when the number of nodes in the cluster is allocated according to the resource usage forecast results of the next day, the monthly standby node pool can be used as the object. That is, when the cluster needs to increase the number of nodes, the corresponding number of standby nodes are called from the monthly standby node pool and added to the cluster. When the cluster needs to reduce the number of nodes, the corresponding number of nodes can be released from the monthly standby node pool to the monthly standby node pool as standby nodes for node resource scheduling on a later day of this month or for other clusters in this month. Similarly, for node resource allocation with a time period of "month", the corresponding number of standby nodes are called from the annual standby node pool to the monthly standby node pool, or the corresponding number of standby nodes are released to the annual standby node pool, which will not be repeated in this embodiment.
[0082] That is, the present embodiment does not limit the number of clusters to which the present method is applied, and the present method can be applied to one cluster or to multiple different clusters at the same time. At the same time, different clusters can set up monthly standby node pools and annual standby node pools respectively, or they can share the same monthly standby node pool and annual standby node pool.
[0083] It should also be noted that if Figure 2 and Figure 3 As shown, since the number of nodes in the annual standby node pool and the monthly standby node pool in this method are dynamically updated with the corresponding time period, a distributed storage cluster only needs to set up one annual standby node pool and one monthly standby node pool (multiple distributed storage clusters can also share one annual standby node pool and one monthly standby node pool). Considering that the maximum business volume that may occur in the same distributed storage cluster on a certain day of a year is always greater than or equal to the maximum business volume that may occur on a certain day of any month in the year; similarly, the maximum business volume that may occur in the same distributed storage cluster on a certain day of a month is always greater than or equal to the maximum business volume on any day of the month; therefore, the number of standby nodes in the annual standby node pool ≥ the number of standby nodes in the monthly standby node pool ≥ the number of nodes in the distributed storage cluster, that is, Figure 2 As shown in , the monthly standby node pool is a sub-pool of the annual standby node pool. Or in another possible implementation scenario, Figure 3 As shown in the figure, the monthly standby node pool and the annual standby node pool are two independent node pools and have no containment relationship with each other.
[0084] Furthermore, for the resource allocation of the node, this embodiment also provides another possible implementation scheme, and the method further includes:
[0085] S21: Dynamically adjust the traffic volume of each node according to the load information of each node.
[0086] It should be noted that the regulation of node traffic should meet a major premise: the greater the resource utilization rate reflected by the node load information, the more the traffic of the node needs to be reduced; the smaller the resource utilization rate reflected by the node load information, the more the traffic of the node needs to be increased.
[0087] However, this embodiment does not limit the specific dynamic adjustment strategy of node traffic volume, and the appropriate implementation scheme can be freely selected according to actual needs. This embodiment is a resource allocation scheme provided by the above method on the other hand of allocating the total amount of node resources, the purpose of which is to achieve "peak shaving and valley filling" of node resources, and to achieve full utilization of node resources under the premise of keeping the total amount of node resources unchanged. The combination of the two can bring better node resource allocation effect.
[0088] In addition, for the standby node mentioned above, this embodiment also provides a possible implementation scheme, and the method further includes:
[0089] S22: If the standby node is not added to the distributed storage cluster, power off the standby node.
[0090] From the above, it can be seen that this method realizes the allocation of cluster node resources through the spare node pool, but the spare node has no other use when it is not added to the cluster. Therefore, this embodiment powers off the spare node to avoid wasting electricity and other resources, thereby better meeting the energy-saving needs in actual application scenarios.
[0091] In summary, the present invention provides a node resource scheduling method for a distributed storage cluster, which uses the collected load information and the pre-established prediction model to predict the resource usage of the cluster for a period of time in the future. Specifically, in this method, the node resource usage is predicted and resource scheduling is performed in three different time periods: day, month, and year. For the year, which is the largest time span, the standby node is prepared in advance and stored in the annual standby node pool, so that the cluster can call the standby node or release its own node to the standby node pool as needed in the next year to realize the dynamic scheduling of node resources in the cluster. Further, for the month, which is smaller than the year and larger than the day, a corresponding standby node pool is also provided, that is, a monthly standby node pool, to provide node resource scheduling for the cluster in the next month. Among them, the month is the next level cycle unit of the year, and the scheduling of the standby nodes in the monthly standby node pool is based on the annual standby node pool, so as to realize the node resource scheduling based on the time level. Similarly, the day, as the minimum time period, is also a common operation cycle in the actual cluster scenario. According to the prediction results, the number of nodes in the cluster can be directly scheduled to meet the needs of the cluster operation work the next day. Among them, day is the next level period unit of month, and the scheduling of nodes required by the cluster on the next day is based on the monthly standby node pool, thereby realizing node resource scheduling based on time level.
[0092] In summary, this method provides a multi-layer node resource prediction and scheduling mechanism with three time granularities, which progresses layer by layer from day to month to year. The node resource adjustment for the next day is based on the monthly standby node pool, and the node resource adjustment for the next month is based on the annual standby node pool, so as to achieve flexible and reliable node resource adjustment, which can cover the complete life cycle of the distributed storage cluster. Provide sufficient node resources for use during the high-traffic phase of the cluster, and release node resources in time during the low-traffic phase of the cluster to avoid resource waste and achieve scientific and reasonable node resource allocation.
[0093] It should be noted that the present invention does not limit the specific implementation of the prediction model in the above embodiments, and the performance of the prediction model varies depending on the technology used in the prediction model. For example, through artificial neural networks, resource usage predictions for a variety of different time periods can be achieved. Exemplarily, when load information with a daily cycle is input to the prediction model, the prediction model outputs prediction results with a daily cycle; when load information with a monthly cycle is input to the prediction model, the prediction model outputs prediction results with a monthly cycle; when load information with a yearly cycle is input to the prediction model, the prediction model outputs prediction results with a yearly cycle. Thus, based on one prediction model, the prediction of resource usage in three different time periods can be achieved, but this solution has high requirements on the prediction model and is not easy to implement.
[0094] Based on this, this embodiment also provides a possible implementation scheme, and the above-mentioned prediction models include: a daily prediction model, a monthly prediction model and an annual prediction model.
[0095] Among them, the daily prediction model is trained through historical daily data, the monthly prediction model is trained through historical monthly data, and the annual prediction model is trained through historical annual data. The historical daily data is the load information corresponding to each node of the distributed storage cluster when the power consumption is the highest in one day, the historical monthly data is the load information corresponding to each node of the distributed storage cluster when the power consumption is the highest in one month, and the historical annual data is the load information corresponding to each node of the distributed storage cluster when the power consumption is the highest in one year.
[0096] Then the above step S12 specifically includes:
[0097] S121: Input the load information into the daily prediction model on a daily basis to obtain a resource usage prediction result of the distributed storage cluster for the next day.
[0098] S122: Input the load information into the monthly prediction model on a monthly basis to obtain a resource usage prediction result of the distributed storage cluster for the next month.
[0099] S123: Input the load information into the annual prediction model on an annual basis to obtain a resource usage prediction result of the distributed storage cluster for the next year.
[0100] It should be noted that there is no order restriction between steps S121~S123, and they can be executed in any order or in parallel, and this embodiment does not limit this. In addition, for the acquisition of the above-mentioned historical data corresponding to different time periods, in this embodiment, the load information when the node consumes the most power during the time period is used as the historical data of the node in a time period. Exemplarily, if the load information is specifically the CPU usage rate, then for the historical daily data of a certain node, the CPU usage rate of the node when the power consumption is the highest during the day is selected as the historical daily data of the node on that day, and the same applies to other historical data.
[0101] It is easy to understand that this embodiment selects the load information when the power consumption is maximum within the time period as historical data. Its purpose is to ensure that the prediction results output by the trained prediction model can meet the cluster's requirements for node resources under the maximum possible access volume, so as to ensure that the cluster has sufficient resources to use during the high business volume stage that may occur in the time period.
[0102] In addition, this embodiment trains corresponding prediction models for three different time periods: day, month, and year. On the one hand, it reduces the requirements for each prediction model and the difficulty of training. On the other hand, it is also beneficial to the prediction accuracy of the prediction model, thereby bringing better resource allocation effects.
[0103] Furthermore, based on the above embodiment, this embodiment also provides a possible implementation scheme, wherein the above daily prediction model includes: a holiday prediction model and a non-holiday prediction model. The holiday prediction model is obtained by training the historical day data corresponding to holidays, and the non-holiday prediction model is obtained by training the historical day data corresponding to non-holidays.
[0104] Then step S121 specifically includes:
[0105] S121-A1: If the next day is a non-holiday, the load information is input into the non-holiday prediction model on a daily basis to obtain a resource usage prediction result of the distributed storage cluster for the next day.
[0106] S121-A2: If the next day is a holiday, the load information is input into the holiday prediction model on a daily basis to obtain a resource usage prediction result of the distributed storage cluster for the next day.
[0107] That is, in this embodiment, in the prediction of node resource usage based on a daily time period, holidays and non-holidays are distinguished. This is because in actual application scenarios, the number of visits to the distributed storage cluster by customers often presents different characteristics based on holidays and non-holidays. For example, if the distributed storage cluster is applied to entertainment-related businesses or scenarios, the business volume on holidays is generally higher than that on non-holidays. Similarly, if the distributed storage cluster is applied to work-related businesses or scenarios, the business volume on non-holidays is generally higher than that on holidays. Based on this division, holidays and non-holidays can also be called working days and non-working days.
[0108] Based on the different access characteristics of holiday and non-holiday customers to the distributed storage cluster, this embodiment performs modeling and prediction for holidays and non-holidays respectively to obtain more accurate prediction results, thereby bringing about better node resource scheduling effects.
[0109] On the other hand, although the above-mentioned method of modeling and predicting separately for holidays and non-holidays can bring higher scheduling effects, it also requires more prediction models and higher implementation difficulty. Based on this, this embodiment also provides another separate prediction scheme for holidays and non-holidays, and the above-mentioned step S121 specifically includes:
[0110] S121-B1: Input the load information into the daily prediction model with a daily cycle, and use the output result of the daily prediction model as the intermediate prediction result.
[0111] S121-B2: If the next day is not a holiday, the intermediate prediction result is used as the resource usage prediction result of the distributed storage cluster for the next day.
[0112] S121-B3: If the next day is a holiday, the intermediate result is increased by a preset ratio to obtain a resource usage prediction result of the distributed storage cluster for the next day.
[0113] The preset ratio is obtained based on the historical data of days corresponding to holidays and the historical data of days corresponding to non-holidays. The preset ratio can be a positive value or a negative value. When the preset ratio is a negative value, the intermediate result is "floated up" by the preset ratio, and the final result is equivalent to the intermediate result being floated down, corresponding to the application scenario where the business volume on holidays is less than that on non-holidays.
[0114] In addition, in order to achieve higher prediction accuracy, the daily prediction model in this embodiment can be trained using non-holiday data.
[0115] As can be seen from the above, this embodiment can also distinguish between holidays and non-holidays and make separate predictions, bringing better prediction accuracy. And this method does not require two different prediction models, and only requires one daily prediction model to be implemented. In addition, the daily prediction model can further provide a non-holiday prediction model mentioned in the above embodiment, bringing higher prediction accuracy.
[0116] On the other hand, the above embodiments do not limit the specific method used to establish the prediction model in the present method. In some of the above embodiments, there may be application scenarios that require multiple different prediction models. In this case, if the prediction model is too complex, it may affect the normal operation of the distributed storage cluster, or introduce other processors with certain data processing capabilities, resulting in unnecessary costs.
[0117] In view of this, this embodiment provides an implementation scheme of a prediction model, and the above-mentioned prediction model is specifically: the prediction value of the next Nth period = the horizontal part of this period + N*the trend part of this period.
[0118] Where N is a positive integer and the period is day, month, or year.
[0119] The horizontal part of this cycle = the first smoothing coefficient * the actual value of demand in this cycle + (1-first smoothing coefficient) * (the horizontal part of the previous cycle + the trend part of the previous cycle).
[0120] The trend part of this cycle = the second smoothing coefficient * (the horizontal part of this cycle - the horizontal part of the previous cycle) + (1-the second smoothing coefficient) * the trend part of the previous cycle.
[0121] The value ranges of the first smoothing coefficient and the second smoothing coefficient are both (0, 1); the actual value of the demand in this period is the specific parameter value of the load information in this period.
[0122] Furthermore, the prediction model provided in this embodiment is expressed by mathematical formulas as follows:
[0123] (1);
[0124] (2);
[0125] (3);
[0126] in, is the predicted value for the next Nth period; is the initial forecast value, Corresponding to the horizontal part results of this period, The horizontal part results corresponding to the previous period; is the local growth value, Corresponding to the trend part results of this period, Corresponds to the trend part results of the previous period; is the first smoothing coefficient, is the second smoothing coefficient.
[0127] It should be noted that the prediction model provided in this embodiment is simple to implement and has low computational difficulty. Even if there are multiple prediction models at the same time, it will not bring too high a computational load, making this method easier to implement. Even if devices with data processing capabilities in a distributed storage cluster are reused to implement this method, it will not bring too high a computational pressure to the distributed storage cluster, and the adverse effects on the distributed storage cluster can be controlled within a controllable range.
[0128] In addition, the prediction model provided by this embodiment can also realize the prediction of multiple subsequent cycles, not limited to the prediction of the next cycle, bringing a wider range of application scenarios. However, in the actual application of this method, when the load information of the current time period is generally used for prediction, the prediction accuracy of the next cycle is the highest. Therefore, although the prediction model provided by this embodiment supports the prediction of the subsequent Nth cycle, N=1 is generally taken, that is, the prediction of the next cycle is performed.
[0129] Furthermore, for the prediction model provided in the above embodiment, the prediction of node resource usage depends on the continuous iteration of the horizontal part and the trend part. Therefore, this embodiment provides a possible implementation scheme for setting the initial values of the horizontal part and the trend part in the prediction model:
[0130] The initial value of the horizontal part in the prediction model is: the specific parameter value corresponding to the historical data of the first period in the time series in the historical data for training the prediction model.
[0131] The initial value of the trend part in the prediction model is: the specific parameter value corresponding to the historical data of the second period in the time series in the historical data for training the prediction model.
[0132] Taking days as a time period, all historical data is the collected load information of each node in the distributed storage cluster over a period of time. In order to ensure the training requirements of the prediction model, the sample data needs to be large enough, and a node needs at least multiple days of historical data. Correspondingly, for this node, among all the historical data corresponding to the node, the first day's historical data is used as the initial value of the horizontal part of the prediction model, that is, Similarly, the historical data of the second day is used as the initial value of the trend part of the prediction model, that is, Based on the setting of this initial value, subsequent iterations of the prediction model can be performed to achieve the prediction of node resource usage in any period.
[0133] In addition, this embodiment also provides another possible initial value setting scheme for the horizontal part and the trend part in the prediction model, which is:
[0134] The initial value of the horizontal part in the prediction model is: the intercept of the pre-established target linear regression model.
[0135] The initial value of the trend part in the forecast model is: the slope of the target linear regression model.
[0136] The target linear regression model is: a linear regression model established based on the historical data of the first M periods in time series in the historical data of the training prediction model; M is a positive integer greater than 2.
[0137] In this embodiment, days are also used as a time period for exemplary explanation. Then, a corresponding linear regression model can be established based on the first M historical days data of the training prediction model, and then the intercept and slope of the linear regression model are used as the initial values of the horizontal part and the trend part. As the principle of the linear regression model shows, at least two points are required to establish a linear regression model, so at least two historical days data are required to establish a target linear regression model, and the number of points must be a positive integer, that is, M is a positive integer greater than 1.
[0138] In the above embodiment, a node resource scheduling method of a distributed storage cluster is described in detail, and the present invention also provides a corresponding embodiment of a node resource scheduling device of a distributed storage cluster. It should be noted that the present invention describes the embodiment of the device part from two perspectives, one is based on the perspective of functional modules, and the other is based on the perspective of hardware.
[0139] Based on the perspective of functional modules, this embodiment provides a node resource scheduling device for a distributed storage cluster, such as Figure 4 As shown, including:
[0140] The load monitoring module 11 is used to obtain the load information of each node in the distributed storage cluster;
[0141] The resource prediction module 12 is used to input the load information into a pre-established prediction model with a period of day, month and year, so as to obtain the resource usage prediction results of the distributed storage cluster for the next day, month and year;
[0142] The annual adjustment module 13 is used to adjust the number of nodes in the pre-established annual standby node pool at the end of the current year according to the resource usage forecast result for the next year;
[0143] The monthly adjustment module 14 is used to adjust the number of nodes in the pre-established monthly standby node pool at the end of the month according to the resource usage forecast result of the next month; if the monthly standby node pool needs to add new nodes, they are retrieved from the annual standby node pool; if the standby node pool needs to release nodes at the end of the month, the released nodes are added to the annual standby node pool;
[0144] The daily adjustment module 15 is used to adjust the number of nodes in the distributed storage cluster at 24:00 on the same day according to the resource usage forecast results of the next day; if the distributed storage cluster needs to add new nodes, they are retrieved from the monthly standby node pool; if the distributed storage cluster needs to release nodes, the released nodes are added to the monthly standby node pool.
[0145] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.
[0146] Figure 5 A structural diagram of a node resource scheduling device for a distributed storage cluster provided by another embodiment of the present invention, such as Figure 5 As shown, a node resource scheduling device for a distributed storage cluster includes: a memory 20, for storing computer programs;
[0147] The processor 21 is configured to implement the steps of a node resource scheduling method for a distributed storage cluster as described in the above embodiment when executing a computer program.
[0148] The node resource scheduling device of a distributed storage cluster provided in this embodiment may include but is not limited to a mobile terminal, a personal computer, a workstation, etc.
[0149] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of a digital signal processor (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.
[0150] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 20 is at least used to store the following computer program 201, wherein, after the computer program is loaded and executed by the processor 21, it can implement the relevant steps of a node resource scheduling method for a distributed storage cluster disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 202 may include Windows, Unix, Linux, etc. Data 203 may include, but is not limited to, a node resource scheduling method for a distributed storage cluster, etc.
[0151] In some embodiments, a node resource scheduling device for a distributed storage cluster may also include a display screen 22 , an input / output interface 23 , a communication interface 24 , a power supply 25 , and a communication bus 26 .
[0152] Those skilled in the art will understand that Figure 5 The structure shown in does not constitute a limitation on a node resource scheduling device for a distributed storage cluster, and may include more or fewer components than those shown in the figure.
[0153] An embodiment of the present invention provides a node resource scheduling device for a distributed storage cluster, including a memory and a processor. When the processor executes a program stored in the memory, it can implement the following method: a node resource scheduling method for a distributed storage cluster.
[0154] In addition to the embodiments of the node resource scheduling method and device of a distributed storage cluster provided in the above embodiments, the present invention also provides an embodiment corresponding to a computer program product. A computer program product includes a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the node resource scheduling method of a distributed storage cluster described in any of the above embodiments can be implemented.
[0155] Since the embodiments of the computer program product part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the computer program product part, which will not be repeated here.
[0156] Finally, the present invention also provides an embodiment corresponding to a non-volatile storage medium. The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps recorded in the above method embodiment are implemented.
[0157] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc. Various media that can store program codes.
[0158] The above is a detailed introduction to a node resource scheduling method, device and medium for a distributed storage cluster provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referenced to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.
[0159] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.
Claims
1. A node resource scheduling method for a distributed storage cluster, characterized in that: include: Obtain the load information of each node in the distributed storage cluster; The load information is input into a pre-established prediction model with a period of day, month and year respectively, so as to obtain the resource usage prediction results of the distributed storage cluster for the next day, next month and next year; At the end of the current year, adjusting the number of nodes in the pre-established annual standby node pool based on the resource usage forecast results for the next year; At the end of the month, the number of nodes in the pre-established monthly standby node pool is adjusted according to the resource usage forecast result of the next month; if the monthly standby node pool needs to add new nodes, they are retrieved from the annual standby node pool; if the monthly standby node pool needs to release nodes, the released nodes are added to the annual standby node pool; At 24:00 on the same day, the number of nodes in the distributed storage cluster is adjusted according to the resource usage forecast results for the next day; if the distributed storage cluster needs to add new nodes, they are retrieved from the monthly standby node pool; if the distributed storage cluster needs to release nodes, the released nodes are added to the monthly standby node pool.
2. The node resource scheduling method of a distributed storage cluster according to claim 1, characterized in that: The prediction models include: daily prediction model, monthly prediction model and annual prediction model; The daily prediction model is obtained by training with historical daily data, the monthly prediction model is obtained by training with historical monthly data, and the annual prediction model is obtained by training with historical annual data. The historical daily data is the load information corresponding to each node of the distributed storage cluster when the power consumption is the largest in a day, the historical monthly data is the load information corresponding to each node of the distributed storage cluster when the power consumption is the largest in a month, and the historical year data is the load information corresponding to each node of the distributed storage cluster when the power consumption is the largest in a year; The load information is input into a pre-established prediction model with a period of day, month, and year, respectively, to obtain the resource usage prediction results of the distributed storage cluster for the next day, next month, and next year, including: Inputting the load information into the daily prediction model on a daily basis to obtain the resource usage prediction result of the distributed storage cluster for the next day; Inputting the load information into the monthly prediction model on a monthly basis to obtain the resource usage prediction result of the distributed storage cluster for the next month; The load information is input into the annual prediction model on an annual basis to obtain the resource usage prediction result of the distributed storage cluster for the next year.
3. The node resource scheduling method of a distributed storage cluster according to claim 2, characterized in that: The daily prediction model includes: a holiday prediction model and a non-holiday prediction model; The holiday prediction model is obtained by training the historical day data corresponding to holidays, and the non-holiday prediction model is obtained by training the historical day data corresponding to non-holidays; The load information is input into the daily prediction model on a daily basis to obtain the resource usage prediction result of the distributed storage cluster for the next day, including: If the next day is a non-holiday, the load information is input into the non-holiday prediction model on a daily basis to obtain the resource usage prediction result of the distributed storage cluster for the next day; If the next day is a holiday, the load information is input into the holiday prediction model on a daily basis to obtain the resource usage prediction result of the distributed storage cluster for the next day.
4. The node resource scheduling method of a distributed storage cluster according to claim 2, characterized in that: Inputting the load information into the daily prediction model on a daily basis to obtain the resource usage prediction result of the distributed storage cluster for the next day includes: Inputting the load information into the daily prediction model on a daily basis, and using the output result of the daily prediction model as an intermediate prediction result; If the next day is not a holiday, the intermediate prediction result is used as the resource usage prediction result of the distributed storage cluster for the next day; If the next day is a holiday, the intermediate result is processed upward by a preset ratio to obtain the resource usage prediction result of the distributed storage cluster for the next day; The preset ratio is obtained based on the historical day data corresponding to holidays and the historical day data corresponding to non-holidays.
5. The node resource scheduling method of a distributed storage cluster according to any one of claims 2 to 4, characterized in that: The prediction model is: The forecast value of the next Nth period = the horizontal part of this period + N*the trend part of this period; Where N is a positive integer and the period is days, months, or years; The horizontal part of this cycle = the first smoothing coefficient * the actual value of demand in this cycle + (1-the first smoothing coefficient) * (the horizontal part of the previous cycle + the trend part of the previous cycle); The trend part of this cycle = the second smoothing coefficient * (the horizontal part of this cycle - the horizontal part of the previous cycle) + (1-the second smoothing coefficient) * the trend part of the previous cycle; The value ranges of the first smoothing coefficient and the second smoothing coefficient are both (0, 1); and the actual value of the demand in this period is a specific parameter value of the load information in this period.
6. The node resource scheduling method of a distributed storage cluster according to claim 5, characterized in that: The initial value of the horizontal part in the prediction model is: the specific parameter value corresponding to the historical data of the first period in the time series in the historical data for training the prediction model; The initial value of the trend part in the prediction model is: the specific parameter value corresponding to the historical data of the second period in time series in the historical data for training the prediction model.
7. The node resource scheduling method of a distributed storage cluster according to claim 5, characterized in that: The initial value of the horizontal part in the prediction model is: the intercept of the pre-established target linear regression model; The initial value of the trend part in the prediction model is: the slope of the target linear regression model; The target linear regression model is a linear regression model established based on the historical data of the first M periods in time series in the historical data for training the prediction model; M is a positive integer greater than 1.
8. A node resource scheduling device for a distributed storage cluster, characterized in that: include: The load monitoring module is used to obtain the load information of each node in the distributed storage cluster; A resource prediction module, used to input the load information into a pre-established prediction model with a period of day, month, and year, respectively, to obtain a resource usage prediction result of the distributed storage cluster for the next day, next month, and next year; An annual adjustment module, used to adjust the number of nodes in the pre-established annual standby node pool at the end of the current year according to the resource usage forecast result for the next year; A monthly adjustment module is used to adjust the number of nodes in the pre-established monthly standby node pool at the end of the month according to the resource usage forecast result of the next month; wherein, if the monthly standby node pool needs to add new nodes, they are retrieved from the annual standby node pool; if the standby node pool at the end of the month needs to release nodes, the released nodes are added to the annual standby node pool; The daily adjustment module is used to adjust the number of nodes in the distributed storage cluster at 24:00 on the same day according to the resource usage prediction results of the next day; if the distributed storage cluster needs to add new nodes, they are retrieved from the monthly standby node pool; if the distributed storage cluster needs to release nodes, the released nodes are added to the monthly standby node pool.
9. A node resource scheduling device for a distributed storage cluster, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the node resource scheduling method for a distributed storage cluster as described in any one of claims 1 to 7 when executing the computer program.
10. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the node resource scheduling method of the distributed storage cluster as described in any one of claims 1 to 7 are implemented.
