Computer fixed resource division method based on big data

Abstract

The invention relates to the technical field of resource division, and particularly discloses a computer fixed resource division method based on big data, and the method comprises the steps: collecting the resource use condition in real time through a sensor and a monitoring tool deployed at each computing node; performing data cleaning and processing by using a big data platform based on the collected historical data, and training a resource demand prediction model by adopting machine learning and deep learning methods; on the basis of a prediction result, a distributed optimization technology is adopted, calculation tasks are distributed to a plurality of nodes, each node calculates and updates resource division according to own resource requirements and system loads, and dynamic adjustment of resource division is achieved through a scheduling algorithm; the energy consumption data of each computing node is acquired in real time through an energy efficiency monitoring system, and unnecessary energy consumption is reduced by adopting an energy consumption optimization algorithm on the premise of ensuring efficient allocation of resources. The resource use condition is collected in real time, resource division is dynamically adjusted based on the prediction result, and meanwhile energy consumption is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of resource partitioning, and more specifically, to a computer fixed resource partitioning method based on big data. Background Art

[0002] With the rapid development of cloud computing and big data technologies, efficient management and optimization of computer resources has become one of the core challenges of modern distributed systems. In traditional resource allocation methods, static configuration is often used to allocate resources. However, with the continuous changes in workloads and resource requirements in cloud computing environments, these methods are often unable to cope with the dynamic needs of large-scale systems. Cloud service providers usually need to dynamically allocate computing, storage, and network resources according to the real-time needs of users, while traditional centralized resource management methods are prone to become performance bottlenecks and lack flexibility. Distributed resource management effectively avoids the risk of single point failure by distributing resource allocation tasks to various computing nodes, and can use multiple computers in the network for parallel processing to improve system reliability and efficiency. However, the main problems faced by distributed resource allocation are how to accurately predict resource requirements, how to deal with resource imbalance in heterogeneous hardware environments, and how to reduce energy consumption and operating costs while ensuring efficient resource utilization.

[0003] In the big data and cloud computing environment, the continuous growth of data and the continuous expansion of system scale make traditional resource management and optimization methods seem incapable. Especially in terms of resource demand prediction, the existing static prediction methods cannot adapt to the environment with large load fluctuations. In order to improve the prediction accuracy, more and more research has begun to rely on big data analysis technologies such as machine learning and deep learning to optimize resource allocation by mining historical data and analyzing real-time data. In addition, energy consumption is also an issue that cannot be ignored in cloud computing. The energy consumption of data centers has accounted for a considerable part of the world's total energy consumption. How to reduce energy consumption through reasonable resource division and dynamic adjustment has become an important issue that cloud service providers need to solve while improving resource utilization. Summary of the invention

[0004] In order to overcome the above-mentioned defects of the prior art, the present invention provides a computer fixed resource partitioning method based on big data, which is to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A method for partitioning computer fixed resources based on big data comprises the following steps:

[0007] Step S1, collecting resource usage in real time through sensors and monitoring tools deployed on each computing node;

[0008] Step S2, based on the collected historical data, use the big data platform to clean and process the data, and use the deep learning method to train the resource demand prediction model;

[0009] Step S3, based on the prediction results, distributed optimization technology is used to assign computing tasks to nodes. Each node calculates and updates resource partitioning according to its own resource requirements and system load, and dynamically adjusts resource partitioning through a local scheduling algorithm.

[0010] Step S4, obtaining energy consumption data of each computing node in real time through the energy efficiency monitoring system, and using an energy consumption optimization algorithm to reduce energy consumption while ensuring efficient resource allocation;

[0011] Each node calculates and updates resource partitioning based on its own resource requirements and system load, and node i calculates and updates resource partitioning based on its predicted resource requirements. and the current load L i Calculate the amount of resources required The calculation formula is: Among them, α is the load factor, which is an adjustment factor used to control the load L i Impact on required resources; each node dynamically adjusts its resource allocation through a local scheduling algorithm: in, is the required amount of resources of node i at time step s; is the predicted resource demand of node i at time step s; L i (s′) is the load of node i at time step s; N is the size of the sliding window; α1 is the first weight coefficient, which controls the contribution of predicted demand to required resources; α2 is the second weight coefficient, which controls the contribution of historical load to required resources.

[0012] As a further solution of the present invention, in step S1, the resource usage is collected in real time by sensors and monitoring tools deployed on each computing node, including the following specific contents: in order to achieve efficient computer fixed resource division, detailed resource usage data is collected in real time from each computing node in the distributed computer system, and the resource usage data includes: CPU usage, memory usage, disk I / O status, network bandwidth, node load, task queue length, system response time and other aspects of resource consumption and workload data. First, real-time resource status data is obtained from the computing node, which is implemented by means of a distributed monitoring system and sensors, and an operating system-based monitoring agent is installed on each computing node, and node resource usage data is regularly collected through the monitoring agent. The monitoring agent will continue to run, regularly obtain information such as CPU usage, memory usage, disk I / O frequency, etc. of each node, and store these data in a local cache. For virtual machines and containers in a cloud environment, the monitoring agent will not only monitor the resource usage of the virtual machine itself, but also monitor the host resources on which they run, to ensure that the resource consumption of the entire computing environment can be accurately reflected.

[0013] Then, the collected data is transmitted. In order to improve the efficiency of data transmission, the HTTP protocol is used to realize the communication between the node and the centralized platform. The appropriate communication protocol is selected according to the actual scenario to ensure low latency and high bandwidth utilization. Data transmission in a distributed environment has certain security risks. The data may be tampered with or leaked during the transmission process. Therefore, TLS encrypted transmission is used to ensure the security of data during transmission.

[0014] As a further solution of the present invention, in step S2, based on the collected historical data, a big data platform is used to clean and process the data, and a deep learning method is used to train the resource demand prediction model, including the following specific contents: In a distributed computer system, data collection is usually achieved through a monitoring tool installed on each computing node. The monitoring tool regularly collects the operating status data of the node. Since the collected data may contain noise or missing values, the data must be preprocessed before data analysis. The preprocessing includes: missing value processing, outlier detection, data standardization and normalization. In the resource monitoring process, there are missing values ​​in the collected data, and the missing values ​​may be caused by various reasons such as network delays, node failures, sensor errors, etc.; the existence of the missing values ​​affects the accuracy of data analysis. For the missing values ​​of a certain feature, the mean is used to fill in, and the calculation formula is: Among them, x jis the value of the jth sampling point in the feature, and M is the number of samples. Outliers can affect data analysis. Outliers are detected by box plot method and deleted. Since the value ranges of different features may vary greatly, directly using these features to train the model may cause some features to have too much impact on model training. Z-score standardization and Min-Max normalization are used to eliminate the dimensional differences between different features.

[0015] The resource demand of each node is time-varying. In order to effectively predict the future resource demand of each node, a resource demand prediction model is established based on deep learning. By analyzing the historical resource usage data of the node, the resource demand of each node in the future is predicted. The resource demand prediction model includes the following specific steps:

[0016] Step Y1, the input layer is used to receive the historical resource usage data of the node. The data is a time series matrix containing resource usage of multiple time steps, such as CPU occupancy, memory usage, disk I / O, etc. Assuming that each time step has D features, the number of historical time steps is Q, and the shape of the input data is [M, Q, D], where M is the number of samples, then the tensor X is used as the input of the model and enters the convolution layer for feature extraction;

[0017] In step Y2, the convolution layer extracts local features from the input data by sliding the convolution kernel. The convolution kernel slides on the time step s and captures the short-term dependencies and patterns in the time series through the weighted summation operation. The convolution formula is: Among them, Y s is the output feature at time step s after the convolution operation, K j is the jth weight in the convolution kernel, k is the size of the convolution kernel, X s+j-1 is the local area of ​​the tensor X input at time step s, and b is the bias term in the convolution operation; the output of the convolution layer is a feature map with a shape of [M, Q-k+1, F], where F is the number of convolution kernels, indicating the number of features extracted;

[0018] Step Y3: The pooling layer downsamples the features extracted by the convolution layer to reduce the dimension of the data. The pooling formula is: P = max(X local ), where X local Represents the data of the local area in the feature map, P is the feature map output by the pooling layer, and the shape of the feature map is Where p is the size of the pooling window;

[0019] In step Y4, the fully connected layer is responsible for globally integrating the features output by the pooling layer and generating a numerical prediction result for the resource requirements of each node. Then the output layer uses the result of the fully connected layer as the final prediction value and uses the mean square error (MSE) as the loss function to measure the difference between the predicted value and the true value. The formula of the loss function is: Among them, O j is the prediction result of the jth sample, Y j is the true value of the jth sample, and L is the loss value.

[0020] As a further solution of the present invention, in step S3, based on the prediction results, distributed optimization technology is used to assign computing tasks to nodes. Each node calculates and updates resource division according to its own resource requirements and system load, and dynamic adjustment of resource division is achieved through a local scheduling algorithm, including the following specific contents: predicting the resource requirements of each node based on historical data, real-time monitoring and deep learning models, and predicting the resource requirements of each node in the future by modeling the historical usage of resources such as the node's CPU utilization, memory occupancy, disk I / O and network bandwidth.

[0021] As a further solution of the present invention, in step S4, the energy consumption data of each computing node is obtained in real time through the energy efficiency monitoring system, and the energy consumption optimization algorithm is adopted to reduce energy consumption under the premise of ensuring efficient allocation of resources, including the following specific contents: by deploying energy consumption sensors on each computing node, the power consumption of the node is monitored. The energy consumption sensor provides real-time energy consumption data of the node, including CPU power consumption, memory power consumption, storage device power consumption, and network communication power consumption. In addition, the energy consumption sensor can not only report energy consumption data in real time, but also collect the energy efficiency performance of the node under different load conditions to form the historical energy consumption data of the node. By establishing an energy consumption optimization algorithm model, the resource requirements of the computing tasks, the energy efficiency status of the nodes, and the load distribution of the system are comprehensively considered to complete all task allocations with the lowest energy consumption. The energy consumption optimization algorithm model specifically includes the following steps:

[0022] In step W1, the goal is to minimize the total energy consumption while ensuring that the task resource requirements are met and the load is balanced, that is: Among them, E total is the total energy consumption; x i,t is the task allocation amount. If task t is assigned to node i, then x i,t =1, otherwise, x i,t =0; R t is the resource requirement of task t, indicating the computing resources required by task t; L i is the current load of node i, indicating the amount of resources occupied; E i (R t,L i ) is the energy consumption of node i when processing task t, which depends on the resource requirement R of the task t and the node load L i ; N is the number of computing nodes; T is the number of tasks to be assigned;

[0023] Step W2, the energy consumption E of node i when processing task t i (R t ,L i ) is calculated by the following formula: E i (R t ,L i )=P i (L i +R t )·T t , where P i (L) is the power consumption of node i when the load is L, P i (L) Satisfaction in, is the idle power consumption of node i, α i is the first power consumption coefficient of node i, β i is the second power consumption coefficient of node i, and L is the load of the node;

[0024] Step W3, constraints: The resource requirements of each task t are met: in, is the maximum resource capacity of node i; each task is assigned to and only to one node: The total load of the node does not exceed its maximum resource capacity: Task allocation x i,t is a binary variable:

[0025] Step W4, combining the above objective function and constraints, we get the following optimization problem: Solved by integer linear programming.

[0026] The technical effects and advantages of a computer fixed resource partitioning method based on big data in the present invention are as follows: the present invention collects resource usage in real time by deploying sensors and monitoring tools at each computing node, uses deep learning to train prediction models based on historical data after cleaning and processing using a big data platform, allocates computing tasks to multiple nodes based on prediction results using distributed optimization technology and dynamically adjusts resource partitioning, and reduces energy consumption under the premise of ensuring efficient resource allocation through energy efficiency monitoring and energy consumption optimization algorithms. The method overcomes the defects of traditional resource allocation methods, can adapt to the dynamic changes of workloads and resource requirements in a cloud computing environment, and improves system reliability and efficiency; solves the problems of resource demand prediction and resource imbalance in heterogeneous hardware environments faced by distributed resource allocation; improves the accuracy of resource demand prediction, and helps cloud service providers reduce energy consumption while improving resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 The present invention is a flowchart of a method for partitioning computer fixed resources based on big data.

[0028] Figure 2 A schematic diagram of the structure of a distributed computer system is proposed for the present invention.

[0029] Figure 3 A flow chart of an energy consumption optimization algorithm model is proposed for the present invention.

[0030] Figure 4 This is a schematic diagram of the resource allocation process in cloud computing proposed by the present invention.

[0031] Figure 5 This is a schematic diagram of task scheduling and resource allocation in the cloud computing environment proposed by the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] Example 1

[0034] See also Figure 1 As shown in the flowchart, the embodiment of the present invention provides a method for partitioning computer fixed resources based on big data, which includes the following steps:

[0035] Step S1, collecting resource usage in real time through sensors and monitoring tools deployed on each computing node;

[0036] Step S2, based on the collected historical data, use the big data platform to clean and process the data, and use the deep learning method to train the resource demand prediction model;

[0037] Step S3, based on the prediction results, distributed optimization technology is used to assign computing tasks to nodes. Each node calculates and updates resource partitioning according to its own resource requirements and system load, and dynamically adjusts resource partitioning through a local scheduling algorithm.

[0038] Step S4, obtaining the energy consumption data of each computing node in real time through the energy efficiency monitoring system, and using the energy consumption optimization algorithm to reduce energy consumption while ensuring efficient allocation of resources.

[0039] Furthermore, in step S1, the resource usage is collected in real time by sensors and monitoring tools deployed on each computing node, including the following specific contents: in order to achieve efficient computer fixed resource division, detailed resource usage data is collected in real time from each computing node in the distributed computer system, and the resource usage data includes: CPU usage, memory usage, disk I / O status, network bandwidth, node load, task queue length, system response time and other resource consumption and workload data. First, real-time resource status data is obtained from the computing node, and with the help of a distributed monitoring system and sensors, a monitoring agent based on an operating system is installed on each computing node, and node resource usage data is regularly collected through the monitoring agent. The monitoring agent will continue to run, regularly obtain information such as CPU usage, memory usage, disk I / O frequency, etc. of each node, and store these data in a local cache. For virtual machines and containers in a cloud environment, the monitoring agent will not only monitor the resource usage of the virtual machine itself, but also monitor the host resources on which they run, to ensure that the resource consumption of the entire computing environment can be accurately reflected.

[0040] Then, the collected data is transmitted. In order to improve the efficiency of data transmission, the HTTP protocol is used to realize the communication between the node and the centralized platform. The appropriate communication protocol is selected according to the actual scenario to ensure low latency and high bandwidth utilization. Data transmission in a distributed environment has certain security risks. The data may be tampered with or leaked during the transmission process. Therefore, TLS encrypted transmission is used to ensure the security of data during transmission.

[0041] Furthermore, in step S2, based on the collected historical data, a big data platform is used to clean and process the data, and a deep learning method is used to train a resource demand prediction model, including the following specific contents: Figure 2As shown in the structural diagram, in a distributed computer system, data collection is usually achieved through a monitoring tool installed on each computing node. The monitoring tool regularly collects the node's operating status data. Since the collected data may contain noise or missing values, the data must be preprocessed before data analysis. The preprocessing includes: missing value processing, outlier detection, data standardization and normalization. In the resource monitoring process, the collected data has missing values, which may be caused by various reasons such as network delays, node failures, sensor errors, etc. The existence of the missing values ​​affects the accuracy of data analysis. For the missing values ​​of a certain feature, the mean is used to fill in the missing values. The calculation formula is: Among them, x j is the value of the jth sampling point in the feature, and M is the number of samples. Outliers can affect data analysis. Outliers are detected by box plot method and deleted. Since the value ranges of different features may vary greatly, directly using these features to train the model may cause some features to have too much impact on model training. Z-score standardization and Min-Max normalization are used to eliminate the dimensional differences between different features.

[0042] The resource demand of each node is time-varying. In order to effectively predict the future resource demand of each node, a resource demand prediction model is established based on deep learning. By analyzing the historical resource usage data of the node, the resource demand of each node in the future is predicted. The resource demand prediction model includes the following specific steps:

[0043] Step Y1, the input layer is used to receive the historical resource usage data of the node. The data is a time series matrix containing resource usage of multiple time steps, such as CPU occupancy, memory usage, disk I / O, etc. Assuming that each time step has D features, the number of historical time steps is Q, and the shape of the input data is [M, Q, D], where M is the number of samples, then the tensor X is used as the input of the model and enters the convolution layer for feature extraction;

[0044] In step Y2, the convolution layer extracts local features from the input data by sliding the convolution kernel. The convolution kernel slides on the time step s and captures the short-term dependencies and patterns in the time series through the weighted summation operation. The convolution formula is: Among them, Y s is the output feature at time step s after the convolution operation, K j is the jth weight in the convolution kernel, k is the size of the convolution kernel, X s+j-1 is the local area of ​​the tensor X input at time step s, and b is the bias term in the convolution operation; the output of the convolution layer is a feature map with a shape of [M, Q-k+1, F], where F is the number of convolution kernels, indicating the number of features extracted;

[0045] Step Y3: The pooling layer downsamples the features extracted by the convolution layer to reduce the dimension of the data. The pooling formula is: P = max(X local ), where X local Represents the data of the local area in the feature map, P is the feature map output by the pooling layer, and the shape of the feature map is Where p is the size of the pooling window;

[0046] In step Y4, the fully connected layer is responsible for globally integrating the features output by the pooling layer and generating a numerical prediction result for the resource requirements of each node. Then the output layer uses the result of the fully connected layer as the final prediction value and uses the mean square error (MSE) as the loss function to measure the difference between the predicted value and the true value. The formula of the loss function is: Among them, O j is the prediction result of the jth sample, Y j is the true value of the jth sample, and L is the loss value.

[0047] Furthermore, in step S3, based on the prediction results, distributed optimization technology is used to assign computing tasks to nodes. Each node calculates and updates resource allocation based on its own resource requirements and system load. Dynamic adjustment of resource allocation is achieved through a local scheduling algorithm, including the following specific contents: predicting the resource requirements of each node based on historical data, real-time monitoring, and deep learning models, and predicting the resource requirements of each node in the future by modeling the historical usage of resources such as the node's CPU usage, memory usage, disk I / O, and network bandwidth. In a distributed computer system, the resource requirements and task load of each node are dynamically changing, and each node has different resource constraints and performance requirements. Therefore, each node calculates resource allocation based on its own resource requirements and current load conditions. Node i calculates resource allocation based on its predicted resource requirements. and the current load L i Calculate the amount of resources required The calculation formula is: Among them, α is the load factor, which is an adjustment factor used to control the load L i Impact on required resources; Each node dynamically adjusts its resource division through a local scheduling algorithm to maintain optimal resource utilization. The calculation formula is:

[0048]

[0049] Where: is the required amount of resources of node i at time step s; is the predicted resource demand of node i at time step s; L i(s′) is the load of node i at time step s; N is the size of the sliding window; α1 is the first weight coefficient, which controls the contribution of predicted demand to required resources; α2 is the second weight coefficient, which controls the contribution of historical load to required resources.

[0050] Furthermore, in step S4, the energy consumption data of each computing node is obtained in real time through the energy efficiency monitoring system, and the energy consumption optimization algorithm is used to reduce energy consumption while ensuring efficient resource allocation, including the following specific contents: by deploying energy consumption sensors on each computing node, the power consumption of the node is monitored. The energy consumption sensor provides real-time energy consumption data of the node, including CPU power consumption, memory power consumption, storage device power consumption, and network communication power consumption. In addition, the energy consumption sensor can not only report energy consumption data in real time, but also collect the energy efficiency performance of the node under different load conditions to form the historical energy consumption data of the node. By establishing an energy consumption optimization algorithm model, the resource requirements of the computing tasks, the energy efficiency status of the nodes, and the load distribution of the system are comprehensively considered to complete all task allocations with the lowest energy consumption, see Figure 3 As shown in the flowchart, the energy consumption optimization algorithm model specifically includes the following steps:

[0051] In step W1, the goal is to minimize the total energy consumption while ensuring that the task resource requirements are met and the load is balanced, that is: Among them, E total is the total energy consumption; x i,t is the task allocation amount. If task t is assigned to node i, then x i,t =1, otherwise, x i,t =0; R t is the resource requirement of task t, indicating the computing resources required by task t; L i is the current load of node i, indicating the amount of resources occupied; E i (R t ,L i ) is the energy consumption of node i when processing task t, which depends on the resource requirement R of the task t and the node load L i ; N is the number of computing nodes; T is the number of tasks to be assigned;

[0052] Step W2, the energy consumption E of node i when processing task t i (R t ,L i ) is calculated by the following formula: E i (R t ,L i )=P i (L i +R t )·T t , where P i(L) is the power consumption of node i when the load is L, P i (L) Satisfaction in, is the idle power consumption of node i, α i is the first power consumption coefficient of node i, β i is the second power consumption coefficient of node i, and L is the load of the node;

[0053] Step W3, constraints: The resource requirements of each task t are met: in, is the maximum resource capacity of node i; each task is assigned to and only to one node: The total load of the node does not exceed its maximum resource capacity: Task allocation x i,t is a binary variable:

[0054] Step W4, combining the above objective function and constraints, we get the following optimization problem: Solved by integer linear programming.

[0055] See also Figure 4-Figure 5 ,The present invention overcomes the defects of the traditional resource ,allocation methods and can adapt to the dynamic changes of workload and ,resource requirements in the cloud computing environment.

[0056] The present invention collects resource usage in real time by deploying sensors and monitoring tools at each computing node, uses deep learning to train prediction models based on historical data after cleaning and processing using a big data platform, uses distributed optimization technology to allocate computing tasks to multiple nodes based on the prediction results and dynamically adjusts resource division, and also reduces energy consumption under the premise of ensuring efficient resource allocation through energy efficiency monitoring and energy consumption optimization algorithms. The method overcomes the defects of traditional resource allocation methods, can adapt to the dynamic changes of workloads and resource requirements in cloud computing environments, and improves system reliability and efficiency; solves the problems of resource demand prediction and resource imbalance in heterogeneous hardware environments faced by distributed resource allocation; improves the accuracy of resource demand prediction, and helps cloud service providers reduce energy consumption while improving resource utilization.

[0057] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0058] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for dividing computer fixed resources based on big data, characterized in that: The steps include: Step S1, collecting resource usage in real time through sensors and monitoring tools deployed on each computing node; Step S2, based on the collected historical data, use the big data platform to clean and process the data, and use the deep learning method to train the resource demand prediction model; Step S3, based on the prediction results, distributed optimization technology is used to assign computing tasks to nodes. Each node calculates and updates resource partitioning according to its own resource requirements and system load, and dynamically adjusts resource partitioning through a local scheduling algorithm. Step S4, obtaining energy consumption data of each computing node in real time through the energy efficiency monitoring system, and using an energy consumption optimization algorithm to reduce energy consumption while ensuring efficient resource allocation; Each node calculates and updates resource partitioning based on its own resource requirements and system load, and node i calculates and updates resource partitioning based on its predicted resource requirements. and the current load L i Calculate the amount of resources required The calculation formula is: Among them, α is the load factor, which is an adjustment factor used to control the load L i Impact on required resources; each node dynamically adjusts its resource allocation through a local scheduling algorithm: in, is the required resource amount of node i at time step s; is the predicted resource demand of node i at time step s; L i (s′) is the load of node i at time step s; N is the size of the sliding window; α1 is the first weight coefficient, which controls the contribution of predicted demand to required resources; α2 is the second weight coefficient, which controls the contribution of historical load to required resources.

2. A method for partitioning computer fixed resources based on big data according to claim 1, characterized in that In step S4, by establishing an energy consumption optimization algorithm model, the resource requirements of the computing tasks, the energy efficiency of the nodes and the load distribution of the system are comprehensively considered to complete all task allocations with the lowest energy consumption. The energy consumption optimization algorithm model specifically includes the following steps: In step W1, the goal is to minimize the total energy consumption while ensuring that the task resource requirements are met and the load is balanced, that is: Among them, E total is the total energy consumption; x i,t is the task allocation amount. If task t is assigned to node i, then x i,t =1, otherwise, x i,t =0; R t is the resource requirement of task t, indicating the computing resources required by task t; L i is the current load of node i, indicating the amount of resources occupied; E i (R t ,L i ) is the energy consumption of node i when processing task t, which depends on the resource requirement R of the task t and the node load L i ; N is the number of computing nodes; T is the number of tasks to be assigned; Step W2, the energy consumption E of node i when processing task t i (R t ,L i ) is calculated by the following formula: i (R t ,L i )=P i (L i +R t )·T t , where P i (L) is the power consumption of node i when the load is L, P i (L) Satisfaction in, is the idle power consumption of node i, α i is the first power consumption coefficient of node i, β i is the second power consumption coefficient of node i, and L is the load of the node; Step W3, constraints: The resource requirements of each task t are met: in, is the maximum resource capacity of node i; each task is assigned to and only to one node: The total load of the node does not exceed its maximum resource capacity: Task allocation x i,t is a binary variable: Step W4, combining the above objective function and constraints, we get the following optimization problem: Solved by integer linear programming.

3. A method for partitioning computer fixed resources based on big data according to claim 1, characterized in that In step S2, the resource demand of each node is time-varying. In order to effectively predict the future resource demand of each node, a resource demand prediction model is established based on deep learning. By analyzing the historical resource usage data of the node, the resource demand of each node in the future is predicted. The resource demand prediction model includes the following specific steps: Step Y1, the input layer is used to receive the historical resource usage data of the node. The data is a time series matrix containing resource usage of multiple time steps, such as CPU occupancy, memory usage, and disk I / O. Assuming that each time step has D features, the number of historical time steps is Q, and the shape of the input data is [M, Q, D], where M is the number of samples, the tensor X is used as the input of the model and enters the convolutional layer for feature extraction; In step Y2, the convolution layer extracts local features from the input data by sliding the convolution kernel. The convolution kernel slides on the time step s and captures the short-term dependencies and patterns in the time series through the weighted summation operation. The convolution formula is: Among them, Y s is the output feature at time step s after the convolution operation, K j is the jth weight in the convolution kernel, k is the size of the convolution kernel, X s+j-1 is the local area of ​​the tensor x input at time step s, and b is the bias term in the convolution operation; the output of the convolution layer is a feature map with a shape of [M, Q-k+1, F], where F is the number of convolution kernels, indicating the number of features extracted; Step Y3: The pooling layer downsamples the features extracted by the convolution layer to reduce the dimension of the data. The pooling formula is: P = max(X local ), where X local Represents the data of the local area in the feature map, P is the feature map output by the pooling layer, and the shape of the feature map is Among them, p is the size of the pooling window; In step Y4, the fully connected layer is responsible for globally integrating the features output by the pooling layer and generating a numerical prediction result for the resource requirements of each node. Then the output layer uses the result of the fully connected layer as the final prediction value and uses the mean square error (MSE) as the loss function to measure the difference between the predicted value and the true value. The formula of the loss function is: Among them, O j is the prediction result of the jth sample, Y j is the true value of the jth sample, and L is the loss value.

4. The method for partitioning computer fixed resources based on big data according to claim 1, characterized in that: In step S1, real-time collection of resource usage includes: real-time collection of resource consumption and workload data including CPU usage, memory usage, disk I / O status, network bandwidth, node load, task queue length and system response time from each computing node in the distributed computer system; installing an operating system-based monitoring agent on each computing node, and regularly collecting node resource usage data through the monitoring agent and storing it in a local cache; using the HTTP protocol to achieve communication between the node and the centralized platform, and using TLS encrypted transmission to ensure data security.

5. The method for partitioning computer fixed resources based on big data according to claim 1, characterized in that: In step S2, data cleaning and processing include: performing missing value processing, outlier detection, data standardization and normalization on the collected data; for missing values, using mean filling, the calculation formula is: Among them, x j is the value of the jth sampling point in the feature, and M is the number of samples; outliers are detected and deleted through the box plot method; Z-score standardization and Min-Max normalization are used to eliminate the dimensional differences between different features.

6. The method for partitioning computer fixed resources based on big data according to claim 1, characterized in that: In step S4, energy consumption sensors are deployed on each computing node to monitor the node power consumption, including real-time energy consumption data of CPU power consumption, memory power consumption, storage device power consumption and network communication power consumption, and the energy efficiency performance of the node under different load conditions is collected to form historical energy consumption data.

7. The method for partitioning computer fixed resources based on big data according to claim 1, characterized in that: In step S2, data collection is achieved through a monitoring tool installed on each computing node, and the monitoring tool regularly collects the operating status data of the node.

8. The method for partitioning computer fixed resources based on big data according to claim 1, characterized in that: In step S1, real-time resource status data is obtained from the computing nodes with the help of a distributed monitoring system and sensors. A monitoring agent based on an operating system is installed on each computing node. The monitoring agent regularly collects usage data of node resources. The monitoring agent runs continuously and regularly obtains information on the CPU usage, memory usage, and disk I / O frequency of each node.

Images