Resource management method, device, electronic device and storage medium

Abstract

The present invention belongs to the technical field of resource management of distributed cluster systems, and specifically provides a resource management method, device, electronic device and storage medium, which aims to solve the problem that the existing distributed storage cluster cannot reasonably utilize resources when balancing data, resulting in low resource utilization and excessive load on physical storage nodes. To this end, the present invention predicts the predicted resource occupancy data of the server within a preset second time period from the current moment based on the historical resource occupancy data of the server obtained; obtains the current resource occupancy data of the server, and determines the maximum value between the current resource occupancy data and the predicted resource occupancy data as the maximum resource occupancy data of the server; calculates the available resource data of the server within the preset second time period based on the maximum resource occupancy data; and adjusts the resource occupancy of the server based on the preset second time period and the available resource data, so as to improve resource utilization and balance the load of each server.

Description

Technical Field

[0001] The present invention belongs to the technical field of resource management of distributed cluster systems, and specifically provides a resource management method, device, electronic equipment and computer-readable storage medium. Background Art

[0002] With the advent of the big data era, the Internet field generates a large amount of data every day, so distributed storage cluster technology is widely used. In a distributed storage cluster, data should be distributed as evenly as possible on the physical storage nodes in the distributed storage cluster to achieve better performance and resource utilization. Various load balancing algorithms are usually used to ensure that data is evenly written to each physical storage node.

[0003] However, in actual applications, various reasons such as cluster expansion, node downtime, network failure, disk failure, etc. can lead to uneven data distribution among physical storage nodes. Therefore, an automatic data balancing method is used to migrate data to achieve space balance between physical storage nodes.

[0004] When balancing data, manual triggering or timed triggering is usually used, and the maximum available resource data is limited to avoid affecting normal storage services. However, this method can only limit the maximum available resource data based on experience or current resource usage data. If too few resources are used for data balancing, the time required will be greatly extended, resulting in low resource utilization; if too many resources are used for data balancing, it may affect normal services, causing physical storage nodes to crash due to excessive load, and further leading to poor user experience.

[0005] Accordingly, the art needs a new resource management method to solve the above problems. Summary of the invention

[0006] The present invention aims to solve the above technical problems, that is, to solve the problem that the existing distributed storage cluster cannot reasonably utilize resources when balancing data, resulting in low resource utilization, and the load of physical storage nodes is unbalanced, resulting in downtime.

[0007] The present invention provides a resource management method, which is applied to each server in a distributed storage cluster. The resource management method includes: obtaining historical resource occupancy data of the server, wherein the historical resource occupancy data represents the resource occupancy of the server within a preset first time period from the current moment; based on the historical resource occupancy data, predicting the predicted resource occupancy data of the server within a preset second time period from the current moment; obtaining the current resource occupancy data of the server, and determining the maximum value between the current resource occupancy data and the predicted resource occupancy data as the maximum resource occupancy data of the server; based on the maximum resource occupancy data, calculating the available resource data of the server within a preset second time period; and adjusting the resource occupancy of the server based on the preset second time period and the available resource data.

[0008] In some embodiments, obtaining historical resource occupancy data of a server includes: obtaining original resource occupancy data of the server in each time period of each day within a preset first time period; and denoising the original resource occupancy data to obtain historical resource occupancy data.

[0009] In some embodiments, based on historical resource occupancy data, predicted resource occupancy data of a server within a preset second time period from the current moment is predicted, including: calculating the average value and standard deviation of the historical resource occupancy data of the server within a preset first time period; adding a preset multiple of the standard deviation to the average value as the predicted resource occupancy data of the server within the preset second time period.

[0010] In some embodiments, current resource occupancy data of a server is obtained, and the maximum value between the current resource occupancy data and the predicted resource occupancy data is determined as the maximum resource occupancy data of the server, including: obtaining the current resource occupancy data of the server, and comparing the current resource occupancy data with the predicted resource occupancy data; if the current resource occupancy data is greater than the predicted resource occupancy data, using the current resource occupancy data as the maximum resource occupancy data of the server; if the current resource occupancy data is less than or equal to the predicted resource occupancy data, using the predicted resource occupancy data as the maximum resource occupancy data of the server.

[0011] In some embodiments, based on the maximum resource occupancy data, the available resource data of the server within a preset second time period is calculated, including: obtaining the maximum resource available data of the server, multiplying the maximum resource available data by a preset percentage and then subtracting the maximum resource occupancy data as the available resource data of the server within the preset second time period.

[0012] In some embodiments, based on a preset second time duration and available resource data, the resource occupancy of the server is adjusted, including: when the available resource data is greater than or equal to a preset threshold, migrating the server data to other servers in the distributed storage cluster to ensure data balance among the servers; when the available resource data is less than the preset threshold, not adjusting the resource occupancy of the server.

[0013] In some embodiments, the resource usage data includes memory resource usage data, CPU resource usage data, disk usage data and / or bandwidth resource usage data.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any method described in the above technical solutions when executing the computer program.

[0015] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the method described in any one of the above technical solutions are implemented.

[0016] It can be understood by those skilled in the art that, when adopting the above-mentioned technical scheme, the present invention obtains historical resource occupancy data of the server, wherein the historical resource occupancy data represents the resource occupancy of the server within a preset first time period from the current moment; based on the historical resource occupancy data, predicts the predicted resource occupancy data of the server within a preset second time period from the current moment; obtains the current resource occupancy data of the server, and determines the maximum value between the current resource occupancy data and the predicted resource occupancy data as the maximum resource occupancy data of the server; based on the maximum resource occupancy data, calculates the available resource data of the server within a preset second time period; based on the preset second time period and the available resource data, adjusts the resource occupancy of the server, thereby realizing automatic configuration and adjustment of resources, avoiding manual intervention in resource configuration, improving resource utilization, reducing resource waste, balancing the load of each server, avoiding server downtime due to excessive load, and further improving user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a schematic flow chart of a resource management method according to the present invention;

[0018] Figure 2 is a flow chart of another resource management method according to the present invention;

[0019] Figure 3 is a schematic diagram of the structure of a resource management device according to the present invention;

[0020] Figure 4 is a schematic structural diagram of an electronic device according to the present invention. DETAILED DESCRIPTION

[0021] Some embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0022] In the description of the present application, ordinal numbers such as "first" and "second" are only used to describe different technical features of the same type, and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions or technical means between the various embodiments of the present application can be combined with each other as long as they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0023] In the description of the present invention, "module" and "processor" may include hardware, software or a combination of the two. A module may include hardware circuits, various suitable sensors, communication ports, and memory, and may also include software parts, such as program code, or a combination of software and hardware. The processor may be a central processing unit, a microprocessor, a digital signal processor, or any other suitable processor. The processor has data and / or signal processing functions. The processor may be implemented in software, hardware, or a combination of the two. Non-temporary computer-readable storage media include any suitable medium that can store program code, such as a disk, a hard disk, an optical disk, a flash memory, a read-only memory, a random access memory, and the like. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The term "at least one A or B" or "at least one of A and B" has a similar meaning to "A and / or B", and may include only A, only B, or A and B. The singular terms "one" and "the" may also include plural forms.

[0024] In recent years, with the continuous development of Internet technology, from daily social networks and e-commerce to smart cities, national security and other related fields, various data have exploded and the demand for data storage has increased. At the same time, the requirements for security, completeness and high availability in the field of data storage are also getting higher and higher.

[0025] Distributed storage is a widely used storage technology. Distributed storage is a data storage technology that uses the disk space on each machine in the enterprise through the network and forms a virtual storage device with these scattered storage resources. Data is stored in various corners of the enterprise. Distributed storage system is the mainstream technology for large-scale data storage. It can store data on different nodes to solve the problems of data backup, expansion, reduction and data migration of large-scale storage systems.

[0026] With the continuous development of big data technology, more and more computing resources are needed. In the field of big data technology, the amount of data accessed does not grow steadily, but peaks and valleys occur due to various factors. Distributed storage node technology can cope with the huge amount of data access to a certain extent, so that the data processing pressure is shared by each storage node. However, it is difficult to achieve balanced management of many storage nodes in distributed storage nodes. For example, some storage nodes bear too much data resource processing pressure, resulting in excessive responsibility; some storage nodes have a small load, resulting in resource waste.

[0027] It can be seen that in distributed storage node technology, storage node management is crucial. Although most existing distributed storage systems have data balancing functions, the time and resource occupancy of data balancing often need to be manually set. For example, Ceph can set the start time and resource occupancy of data balancing, but it cannot be adjusted automatically, and it is also impossible to predict the rationality of the set resource occupancy data; GlusterFS rebalance needs to be manually enabled, and resource occupancy can only be controlled by setting the migration level to control the files migrated each time, and its rationality is also impossible to predict. Therefore, it is necessary to provide a solution that can ensure the balanced distribution of resources of each storage node, thereby improving resource utilization.

[0028] A resource management method and device according to an embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

[0029] Figure 1 It is a flowchart of a resource management method according to the present invention. Figure 1 The resource management method can be performed by a resource management device, which can be implemented by software and / or hardware and can be integrated in an electronic device such as a server. Figure 1 As shown, the resource management method is applied to each server in a distributed storage cluster, including:

[0030] S101, obtaining historical resource occupancy data of a server, wherein the historical resource occupancy data represents resource occupancy of the server within a preset first time period from the current moment;

[0031] S102, predicting resource occupancy data of a server within a preset second time period from the current moment based on historical resource occupancy data;

[0032] S103, obtaining current resource occupancy data of the server, and determining the maximum value between the current resource occupancy data and the predicted resource occupancy data as the maximum resource occupancy data of the server;

[0033] S104, calculating available resource data of the server within a preset second time period based on the maximum resource occupancy data;

[0034] S105: Adjust resource occupancy of the server based on the preset second duration and available resource data.

[0035] Specifically, a distributed storage cluster builds a network topology of multiple storage servers to form a storage server cluster. In the distributed storage cluster, relevant algorithms are used to provide data reading, writing and disk storage functions. The distributed storage cluster uses multiple storage servers to share the storage load.

[0036] The server may be a server that provides various services, for example, a background server that receives a request sent by a terminal device that establishes a communication connection with the server, and the background server may receive and analyze the request sent by the terminal device, and generate a processing result. The server may be a single server, a server cluster consisting of several servers, or a cloud computing service center, which is not limited in the embodiments of the present invention.

[0037] It should be noted that the server can be hardware or software. When the server is hardware, it can be various electronic devices that provide various services for terminal devices. When the server is software, it can be multiple software or software modules that provide various services for terminal devices, or it can be a single software or software module that provides various services for terminal devices, which is not limited in the embodiments of the present invention.

[0038] The terminal device can be hardware or software. When the terminal device is hardware, it can be various electronic devices with a display screen and supporting communication with a server, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers; when the terminal device is software, it can be installed in the electronic devices described above. The terminal device can be implemented as multiple software or software modules, or as a single software or software module, and the embodiments of the present invention do not limit this.

[0039] The historical resource occupancy data is the resource usage of each server within a preset first time period from the current moment. The historical resource occupancy data may include but is not limited to CPU usage, memory usage, disk usage, etc. It should be noted that the historical resource occupancy data may be collected based on a preset time granularity, for example, resource occupancy data collected at a period of every minute, every hour, every day, every week, or every month.

[0040] The preset first duration refers to any period before the current moment, that is, a historical time period. The historical time period may be a period between the current moment and a certain moment in the past, for example, 10 minutes past the current moment, 1 hour past the current moment, 30 days past the current moment, etc. The historical time period may also be a period between a certain moment in the past and another moment, for example, a period between 3 hours and 30 minutes and 3 hours and 40 minutes in the past. It should be noted that the preset first duration may be a duration pre-set according to empirical data, or may be a duration obtained by adjusting the set duration according to actual needs, and the embodiment of the present invention does not limit this. For example, the preset first duration may be 15 minutes, 30 minutes, 1 hour, 2 hours, a day, a week, half a month, a month, three months, a year, etc. Preferably, in an embodiment of the present invention, the preset first duration is 30 days.

[0041] The preset second duration refers to any period after the current moment, that is, the next time period. The next time period can be a period between the current moment and a certain moment in the future, for example, 5 minutes in the future from the current moment, 3 hours in the future from the current moment, 180 days in the future from the current moment, etc. The next time period can also be a period between a certain moment in the future and another moment, for example, a period between 1 hour 20 minutes and 2 hours 50 minutes in the future. It should be noted that the preset second duration can be a duration pre-set according to empirical data, or a duration obtained by adjusting the set duration according to actual needs, and the embodiment of the present invention does not limit this. For example, the preset second duration can be 15 minutes, 30 minutes, 1 hour, 2 hours, a day, a week, half a month, a month, three months, a year, etc. Preferably, in an embodiment of the present invention, the preset first duration is 15 minutes.

[0042] The predicted resource occupancy data is the amount of resources that each server may occupy within a preset second time period (i.e., a period of time in the future, such as 1 hour, one day, etc.) from the current moment. The predicted resource occupancy data may include, but is not limited to, CPU usage, memory usage, disk usage, etc. It should be noted that the predicted resource occupancy data may be collected based on a preset time granularity, for example, resource occupancy data collected at a period of every minute, every hour, every day, every week, or every month.

[0043] The current resource occupancy data is the amount of resources occupied by each server at the current time (for example, 12:00, 14:00, etc.). The current resource occupancy data includes but is not limited to CPU usage, memory usage, disk usage, etc. It should be noted that the current resource occupancy data can be collected based on a preset time granularity, for example, resource occupancy data collected every minute, every hour, every day, every week, every month, etc. It should be noted that if there is no historical resource occupancy data, or if it is impossible to collect complete historical resource occupancy data, the current resource occupancy data can be used.

[0044] The maximum resource occupancy data is the maximum amount of resources occupied by each server. The maximum resource occupancy data may include but is not limited to CPU usage, memory usage, disk usage, etc. It should be noted that the maximum resource occupancy data may be obtained based on a comparison between the current resource occupancy data and the predicted resource occupancy data.

[0045] The available resource data is the amount of resources that each server can occupy. The available resource data may include but is not limited to CPU usage, memory usage, disk usage, etc. It should be noted that the available resource data may be calculated based on the maximum resource usage data.

[0046] It should be noted that the resource usage data of the present invention is not limited to the CPU usage, memory usage, and disk usage as described above, and may also include bandwidth usage, for example.

[0047] According to the technical solution provided by the embodiment of the present invention, by obtaining historical resource occupancy data of the server, wherein the historical resource occupancy data represents the resource occupancy of the server within a preset first time period from the current moment; based on the historical resource occupancy data, predicting the predicted resource occupancy data of the server within a preset second time period from the current moment; obtaining the current resource occupancy data of the server, and determining the maximum value between the current resource occupancy data and the predicted resource occupancy data as the maximum resource occupancy data of the server; based on the maximum resource occupancy data, calculating the available resource data of the server within a preset second time period; based on the preset second time period and the available resource data, adjusting the resource occupancy of the server, realizing automatic configuration and adjustment of resources, avoiding manual intervention in resource configuration, improving resource utilization, reducing resource waste, balancing the load of each server, avoiding the situation where the server crashes due to excessive load, and further improving the user experience.

[0048] In some embodiments, obtaining historical resource occupancy data of a server includes: obtaining original resource occupancy data of the server in each time period of each day within a preset first time period; and denoising the original resource occupancy data to obtain historical resource occupancy data.

[0049] Specifically, each time period may be, for example, 9:00 to 10:00, 10:00 to 11:00, 11:00 to 12:00, etc., or 10:15 to 10:30, 10:30 to 11:00, 11:00 to 11:15, etc., and this embodiment of the present invention is not limited to this.

[0050] The original resource occupancy data is the actual occupancy of various resources. Since there may be some uncontrollable factors in the data collection process, which may lead to abnormal data in a certain period or day, the original resource occupancy data needs to be denoised. Here, the denoising method may include but is not limited to removing the original resource occupancy data outside the preset range as noise data; or, calculating the average value μ and standard deviation σ of the original resource occupancy data in the most recent month, assuming that the interval of normal data is [μ-2σ, μ+2σ], then replace the data outside the interval with the average value μ.

[0051] According to the technical solution provided by the embodiment of the present invention, by preprocessing the original resource occupancy data, abnormal data can be removed, thereby making the obtained historical resource occupancy data more accurate.

[0052] In some embodiments, based on historical resource occupancy data, predicted resource occupancy data of a server within a preset second time period from the current moment is predicted, including: calculating the average value and standard deviation of the historical resource occupancy data of the server within a preset first time period; adding a preset multiple of the standard deviation to the average value as the predicted resource occupancy data of the server within the preset second time period.

[0053] Specifically, the preset multiple can be obtained based on the n-sigma algorithm, and the n-sigma algorithm has the advantages of simple calculation and high efficiency. Preferably, in an embodiment of the present invention, a 3-sigma algorithm is used to determine the fluctuation range of the predicted resource occupancy data, that is, [μ-3σ, μ+3σ] is taken as the fluctuation range. Here, the 3-sigma principle can be simply described as: statistically, the observed values ​​of random variables that conform to the standard normal distribution will fall in different intervals with a certain probability, and the probability of falling within the positive and negative three standard deviations reaches 99.73%. Therefore, it can be considered that the situation falling in other intervals is a low-probability event, that is, the data falling outside 3 standard deviations can be considered as abnormal data.

[0054] Optionally, in some embodiments, if the random variable does not conform to the standard normal distribution, then according to the Chebyshev inequality, the probability of falling within 3 standard deviations is also greater than 88.9%. Here, the Chebyshev principle can be simply described as follows: in any data set, the proportion (or portion) within m standard deviations of its mean is always at least 1-1 / m 2, where m is any positive number greater than 1. For m=2, m=3 and m=5, the following results are obtained: of all data, at least 3 / 4 (or 75%) of the data are within 2 standard deviations of the mean; of all data, at least 8 / 9 (or 88.9%) of the data are within 3 standard deviations of the mean; of all data, at least 24 / 25 (or 96%) of the data are within 5 standard deviations of the mean. Preferably, in an embodiment of the present invention, the preset multiple is 3, that is, to ensure that most resource occupancy data falls within the interval of plus or minus three standard deviations of the mean value calculated using historical resource occupancy data.

[0055] According to the technical solution provided by the embodiment of the present invention, by setting the allowed fluctuation range through any one of the above two methods, more accurate predicted resource occupancy data can be obtained, thereby improving the prediction accuracy.

[0056] In some embodiments, current resource occupancy data of a server is obtained, and the maximum value between the current resource occupancy data and the predicted resource occupancy data is determined as the maximum resource occupancy data of the server, including: obtaining the current resource occupancy data of the server, and comparing the current resource occupancy data with the predicted resource occupancy data; if the current resource occupancy data is greater than the predicted resource occupancy data, using the current resource occupancy data as the maximum resource occupancy data of the server; if the current resource occupancy data is less than or equal to the predicted resource occupancy data, using the predicted resource occupancy data as the maximum resource occupancy data of the server.

[0057] Specifically, after calculating the predicted resource occupancy data, the electronic device also obtains the current resource occupancy data of the server, and compares the current resource occupancy data with the predicted resource occupancy data to determine the maximum resource occupancy data of the server. Further, if the current resource occupancy data is greater than the predicted resource occupancy data, the current resource occupancy data is used as the maximum resource occupancy data of the server; if the current resource occupancy data is less than or equal to the predicted resource occupancy data, the predicted resource occupancy data is used as the maximum resource occupancy data of the server.

[0058] According to the technical solution provided by the embodiment of the present invention, by comparing the current resource occupancy data with the predicted resource occupancy data, the maximum resource occupancy data of the server can be accurately obtained, thereby improving the prediction accuracy.

[0059] In some embodiments, based on the maximum resource occupancy data, the available resource data of the server within a preset second time period is calculated, including: obtaining the maximum resource available data of the server, multiplying the maximum resource available data by a preset percentage and then subtracting the maximum resource occupancy data as the available resource data of the server within the preset second time period.

[0060] Specifically, the preset percentage may be the percentage of the reserved available resource data to the maximum resource available data. The preset percentage may be a percentage pre-set according to empirical data, or may be a percentage obtained by adjusting the set percentage according to actual needs, and the embodiment of the present invention does not limit this. For example, the preset percentage may be 5%, 10%, 15%, 20%, 30%, 50%, etc. Preferably, in the embodiment of the present invention, the preset percentage is 90%.

[0061] For example, taking 12:00 to 12:15 as an example, assuming that the historical resource occupancy data obtained for 30 days is 371, 364, 447, 378, 469, ..., 510, 433, 411, 372, then based on the historical resource occupancy data, the average value of the historical resource occupancy data calculated is 468.40, the standard deviation is 93.43, and the allowed fluctuation range is [188.1, 748.7]. Therefore, 748.7 is used as the maximum resource occupancy data occupied by normal business; further, assuming that the bandwidth upper limit of data migration (that is, the total bandwidth) is 1000M, the available bandwidth during data migration is 90% of the total bandwidth (10% is reserved), that is, 900M, and the calculated available bandwidth is subtracted from the maximum resource occupancy data to obtain the bandwidth available for data migration: 1000×90%-748.7=151.3, that is, the available resource data is 151.3M.

[0062] According to the technical solution provided by the embodiment of the present invention, by limiting the maximum resource occupancy data used, it is possible to avoid affecting normal storage services during data balancing, thereby ensuring the smooth progress of storage services.

[0063] In some embodiments, based on a preset second time duration and available resource data, the resource occupancy of the server is adjusted, including: when the available resource data is greater than or equal to a preset threshold, migrating the server data to other servers in the distributed storage cluster to ensure data balance among the servers; when the available resource data is less than the preset threshold, not adjusting the resource occupancy of the server.

[0064] Specifically, the preset threshold value may be a resource data available value pre-set according to empirical data, or may be a resource data available value obtained by adjusting the set resource data available value according to actual needs, and the embodiment of the present invention does not limit this. For example, the preset threshold value may be 0, 1, 10, 20, 50, 100, etc. Preferably, in the embodiment of the present invention, the preset threshold value is 0.

[0065] In an embodiment of the present invention, if the calculated available resource data is greater than or equal to 0, it means that the storage service is not yet saturated. At this time, the server data can be migrated to other servers in the distributed storage cluster to ensure data balance among the servers; if the calculated available resource data is less than 0, it means that the storage service is saturated. At this time, the server cannot perform data migration.

[0066] According to the technical solution provided in the embodiment of the present invention, by comparing the available resource data with the preset threshold, it is possible to quickly and accurately determine whether to perform data migration. Therefore, automatic configuration and adjustment of resources are achieved, manual intervention in resource configuration is avoided, resource utilization is improved, and user experience is further enhanced.

[0067] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0068] Figure 2 It is a flowchart of another resource management method according to the present invention. Figure 2 The resource management method can be executed by a resource management device, which can be implemented by software and / or hardware and can be integrated in an electronic device such as a server. Figure 2 As shown, the resource management method is applied to each server in a distributed storage cluster, including:

[0069] S201, obtaining original resource occupancy data of the server in each time period of each day within a preset first time period, and performing denoising processing on the original resource occupancy data to obtain historical resource occupancy data;

[0070] S202, calculating the average value and standard deviation of the historical resource occupancy data of the server within a preset first duration, and adding a preset multiple of the standard deviation to the average value as the predicted resource occupancy data of the server within a preset second duration;

[0071] S203, obtaining current resource occupancy data of the server, and comparing the current resource occupancy data with the predicted resource occupancy data to determine maximum resource occupancy data of the server;

[0072] S204, if the current resource occupancy data is greater than the predicted resource occupancy data, the current resource occupancy data is used as the maximum resource occupancy data of the server;

[0073] S205, if the current resource occupancy data is less than or equal to the predicted resource occupancy data, the predicted resource occupancy data is used as the maximum resource occupancy data of the server;

[0074] S206, obtaining maximum resource available data of the server, and multiplying the maximum resource available data by a preset percentage and then subtracting the maximum resource occupancy data as available resource data of the server within a preset second time period;

[0075] S207, when the available resource data is greater than or equal to a preset threshold, migrate the data of the server to other servers in the distributed storage cluster to ensure data balance among the servers;

[0076] S208: When the available resource data is less than a preset threshold, the resource occupancy of the server is not adjusted.

[0077] Specifically, after obtaining the original resource occupancy data of any server in the distributed storage cluster for each time period of each day within a preset first time period, the electronic device performs denoising processing on the original resource occupancy data to obtain historical resource occupancy data, calculates the average value and standard deviation of the historical resource occupancy data, and takes the average value plus a preset multiple of the standard deviation as the predicted resource occupancy data of the server within a preset second time period; then, the electronic device obtains the current resource occupancy data of the server, and compares the current resource occupancy data with the predicted resource occupancy data to determine the maximum resource occupancy data of the server; if the current resource occupancy data is greater than the predicted resource occupancy data, the current resource occupancy data is taken as the maximum resource occupancy data of the server; if the current resource occupancy data is less than or equal to the predicted resource occupancy data, the predicted resource occupancy data is taken as the maximum resource occupancy data of the server; further, the electronic device obtains the maximum resource available data of the server, and multiplies the maximum resource available data by a preset percentage and then subtracts the maximum resource occupancy data as the available resource data of the server within the preset second time period; when the available resource data is greater than or equal to a preset threshold, the electronic device migrates the data of the server to other servers in the distributed storage cluster to ensure data balance of each server; when the available resource data is less than the preset threshold, the electronic device does not adjust the resource occupancy of the server.

[0078] According to the technical solution provided in the embodiment of the present invention, the resource occupancy of the server can be adjusted based on the preset second time duration and available resource data, thereby realizing automatic configuration and adjustment of resources, avoiding manual intervention in resource configuration, and optimizing the utilization of resources required by each server, thereby reducing resource waste, reducing the operating costs of the server, and further improving the user experience.

[0079] The following are embodiments of the device disclosed herein, which can be used to execute the method embodiments disclosed herein. For details not disclosed in the device embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0080] Figure 3Schematic diagram of a resource management device according to the present invention. Figure 3 As shown, the resource management device is applied to each server in a distributed storage cluster, including:

[0081] The acquisition module 301 is configured to acquire historical resource occupancy data of the server, wherein the historical resource occupancy data represents the resource occupancy of the server within a preset first time period from the current moment;

[0082] The prediction module 302 is configured to predict the predicted resource occupancy data of the server within a preset second time period from the current moment based on the historical resource occupancy data;

[0083] The determination module 303 is configured to obtain current resource occupancy data of the server, and determine the maximum value between the current resource occupancy data and the predicted resource occupancy data as the maximum resource occupancy data of the server;

[0084] The calculation module 304 is configured to calculate the available resource data of the server within a preset second time period based on the maximum resource occupation data;

[0085] The adjustment module 305 is configured to adjust the resource occupancy of the server based on the preset second duration and the available resource data.

[0086] According to the technical solution provided by the embodiment of the present disclosure, by obtaining historical resource occupancy data of the server, wherein the historical resource occupancy data represents the resource occupancy of the server within a preset first time period from the current moment; based on the historical resource occupancy data, predicting the predicted resource occupancy data of the server within a preset second time period from the current moment; obtaining the current resource occupancy data of the server, and determining the maximum value between the current resource occupancy data and the predicted resource occupancy data as the maximum resource occupancy data of the server; based on the maximum resource occupancy data, calculating the available resource data of the server within a preset second time period; based on the preset second time period and the available resource data, adjusting the resource occupancy of the server, automatic configuration and adjustment of resources are realized, manual intervention in resource configuration is avoided, resource utilization is improved, resource waste is reduced, the load of each server is balanced, server downtime due to excessive load is avoided, and user experience is further improved.

[0087] In some embodiments, Figure 3 The acquisition module 301 acquires the original resource occupancy data of the server in each time period of each day within a preset first time period, and performs denoising processing on the original resource occupancy data to obtain historical resource occupancy data.

[0088] In some embodiments, Figure 3The prediction module 302 calculates the average value and standard deviation of the historical resource occupancy data of the server within a preset first duration, and adds the average value plus a preset multiple of the standard deviation as the predicted resource occupancy data of the server within a preset second duration.

[0089] In some embodiments, Figure 3 The determination module 303 obtains the current resource occupancy data of the server and compares the current resource occupancy data with the predicted resource occupancy data; if the current resource occupancy data is greater than the predicted resource occupancy data, the current resource occupancy data is used as the maximum resource occupancy data of the server; if the current resource occupancy data is less than or equal to the predicted resource occupancy data, the predicted resource occupancy data is used as the maximum resource occupancy data of the server.

[0090] In some embodiments, Figure 3 The calculation module 304 obtains the maximum resource available data of the server, and multiplies the maximum resource available data by a preset percentage and then subtracts the maximum resource occupancy data as the available resource data of the server within a preset second time period.

[0091] In some embodiments, when the available resource data is greater than or equal to a preset threshold, Figure 3 The adjustment module 305 migrates the data of the server to other servers in the distributed storage cluster to ensure data balance of each server; when the available resource data is less than the preset threshold, Figure 3 The adjustment module 305 does not adjust the resource occupancy of the server.

[0092] In some embodiments, the resource usage data includes memory resource usage data, CPU resource usage data, disk usage data and / or bandwidth resource usage data.

[0093] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0094] Figure 4 Schematic diagram of the structure of the electronic device 4 according to the present invention. Figure 4 As shown, the electronic device 4 of this embodiment includes: a processor 401, a memory 402, and a computer program 403 stored in the memory 402 and executable on the processor 401. When the processor 401 executes the computer program 403, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 401 executes the computer program 403, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0095] Exemplarily, the computer program 403 may be divided into one or more modules / units, which are stored in the memory 402 and executed by the processor 401 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program 403 in the electronic device 4.

[0096] The electronic device 4 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 4 may include, but is not limited to, a processor 401 and a memory 402. Those skilled in the art will appreciate that Figure 4 It is only an example of the electronic device 4 and does not constitute a limitation of the electronic device 4. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0097] The processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0098] The memory 402 may be an internal storage unit of the electronic device 4, for example, a hard disk or memory of the electronic device 4. The memory 402 may also be an external storage device of the electronic device 4, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 4. Further, the memory 402 may also include both an internal storage unit and an external storage device of the electronic device 4. The memory 402 is used to store computer programs and other programs and data required by the electronic device. The memory 402 may also be used to temporarily store data that has been output or is to be output.

[0099] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0100] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0101] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments invented herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0102] In the embodiments provided by the present invention, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. Multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0103] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0104] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0105] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. The computer program may include computer program code, and the computer program code may be in the form of source code, object code, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0106] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A resource management method, applied to each server in a distributed storage cluster, characterized in that: include: Obtaining historical resource occupancy data of the server, wherein the historical resource occupancy data represents resource occupancy of the server within a preset first time period from the current moment, and the preset first time period refers to any period before the current moment; Based on the historical resource occupancy data, predict the predicted resource occupancy data of the server within a preset second time period from the current moment, where the preset second time period refers to any time period after the current moment; Acquire current resource occupancy data of the server, and determine a maximum value between the current resource occupancy data and the predicted resource occupancy data as maximum resource occupancy data of the server; Calculating available resource data of the server within the preset second duration based on the maximum resource occupancy data includes: Acquire maximum resource available data of the server, and multiply the maximum resource available data by a preset percentage and then subtract the maximum resource occupancy data as available resource data of the server within the preset second time period; Adjusting the resource occupancy of the server based on the preset second duration and the available resource data includes: When the available resource data is greater than or equal to a preset threshold, the data of the server is migrated to other servers in the distributed storage cluster to ensure data balance among the servers; When the available resource data is less than the preset threshold, the resource occupancy of the server is not adjusted, and the resource occupancy data includes memory resource occupancy data, CPU resource occupancy data, disk occupancy data and / or bandwidth resource occupancy data.

2. The method according to claim 1, characterized in that The obtaining of historical resource occupancy data of the server includes: Obtaining original resource occupancy data of the server in each time period of each day within the preset first time period; The original resource occupancy data is subjected to denoising to obtain the historical resource occupancy data.

3. The method according to claim 1, characterized in that: The predicting, based on the historical resource occupancy data, the predicted resource occupancy data of the server within a preset second time period from the current moment, includes: Calculate the average value and standard deviation of the historical resource occupancy data of the server within the preset first time period; The average value plus the standard deviation of a preset multiple is used as the predicted resource occupancy data of the server within the preset second time period.

4. The method according to claim 1, characterized in that: The acquiring the current resource occupancy data of the server and determining the maximum value of the current resource occupancy data and the predicted resource occupancy data as the maximum resource occupancy data of the server includes: Acquire current resource occupancy data of the server, and compare the current resource occupancy data with the predicted resource occupancy data; If the current resource occupancy data is greater than the predicted resource occupancy data, taking the current resource occupancy data as the maximum resource occupancy data of the server; If the current resource occupancy data is less than or equal to the predicted resource occupancy data, the predicted resource occupancy data is used as the maximum resource occupancy data of the server.

5. A resource management device, applied to each server in a distributed storage cluster, characterized in that: include: an acquisition module configured to acquire historical resource occupancy data of the server, wherein the historical resource occupancy data represents resource occupancy of the server within a preset first time period from the current moment, and the preset first time period refers to any period before the current moment; A prediction module, configured to predict the predicted resource occupancy data of the server within a preset second time period from the current moment based on the historical resource occupancy data, where the preset second time period refers to any period after the current moment; a determination module configured to obtain current resource occupancy data of the server, and determine a maximum value between the current resource occupancy data and the predicted resource occupancy data as maximum resource occupancy data of the server; The calculation module is configured to calculate the available resource data of the server within the preset second time period based on the maximum resource occupation data, including: Acquire maximum resource available data of the server, and multiply the maximum resource available data by a preset percentage and then subtract the maximum resource occupancy data as available resource data of the server within the preset second time period; The adjustment module is configured to adjust the resource occupancy of the server based on the preset second duration and the available resource data, including: When the available resource data is greater than or equal to a preset threshold, the data of the server is migrated to other servers in the distributed storage cluster to ensure data balance among the servers; When the available resource data is less than the preset threshold, the resource occupancy of the server is not adjusted, and the resource occupancy data includes memory resource occupancy data, CPU resource occupancy data, disk occupancy data and / or bandwidth resource occupancy data.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images