Computing power data management system and method based on distributed computing
Through distributed sensor networks, machine learning, reinforcement learning algorithms, and dynamically adjusting data center resources, the problem that traditional centralized computing is difficult to meet massive data processing is solved, and efficient computing power utilization and low-cost operation are achieved.
Patent Information
- Application Number
- CN202411293160.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-14
- Publication Date
- 2024-11-26
AI Technical Summary
Traditional centralized computing architectures are difficult to meet the needs of massive data processing and complex computing tasks, resulting in too long calculation time and a computing power data management solution based on distributed computing is needed.
Through a distributed sensor network, the data center environment data and computing node status data are collected in real time, and the temperature changes, load and computing power requirements are predicted using machine learning and reinforcement learning algorithms, dynamically adjust resource allocation strategies, optimize refrigeration systems and task scheduling, and realize intelligent Load balancing and resource utilization optimization.
It improves the overall computing power utilization rate, reduces power usage efficiency and operational costs, shortens task completion time, enhances system throughput and reliability, reduces user-perceived latency, and extends hardware life.
Smart Images

Figure CN119025283A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of distributed technology, and in particular to a computing power data management system and method based on distributed computing. Background Art
[0002] In recent years, with the continuous growth of computing needs, the traditional centralized computing architecture has been unable to meet the needs of massive data processing and complex computing tasks. Distributed computing is a computing method that is relative to centralized computing. With the development of computing technology, some applications require extremely huge computing power to complete. If centralized computing is used, it will take a considerable amount of time to complete. A solution for computing power data management based on distributed computing is urgently needed. Summary of the invention
[0003] Based on the above problems, the present invention proposes a computing power data management system and method based on distributed computing. Through the accurate computing power demand prediction and dynamic resource allocation of the solution of the present invention, the overall computing power utilization rate can be improved; combined with the coordinated optimization of environmental data and computing power allocation, the power supply efficiency can be reduced and energy consumption can be saved; through intelligent task scheduling and load balancing, the task completion time can be reduced.
[0004] In view of this, one aspect of the present invention proposes a computing power data management system based on distributed computing, comprising: a cloud server, a data center, and a plurality of computing nodes; The cloud server is configured as follows: Collect data center environmental data and node status data of each computing node in real time through a distributed sensor network; Collect and integrate computing power data of each computing node; The collected environmental data, node status data and computing power data are transmitted to the central control system for preprocessing and analysis to obtain first data; Based on the first data, a prediction model is constructed using a machine learning algorithm to predict future temperature changes, computing loads, and computing power requirements of the data center; Dynamically adjust the computing resource allocation strategy according to the computing load and the computing power demand; Optimizing refrigeration system parameters according to the temperature change; Continuously monitor the system operation status and computing power utilization efficiency, detect anomalies and trigger alarm mechanisms; Regularly evaluate the effectiveness of computing power management strategies and use reinforcement learning technology to optimize control strategies.
[0005] Another aspect of the present invention provides a computing power data management method based on distributed computing, comprising: Collect data center environmental data and node status data of each computing node in real time through a distributed sensor network; Collect and integrate computing power data of each computing node; The collected environmental data, node status data and computing power data are transmitted to the central control system for preprocessing and analysis to obtain first data; Based on the first data, a prediction model is constructed using a machine learning algorithm to predict future temperature changes, computing loads, and computing power requirements of the data center; Dynamically adjust the computing resource allocation strategy according to the computing load and the computing power demand; Optimizing refrigeration system parameters according to the temperature change; Continuously monitor the system operation status and computing power utilization efficiency, detect anomalies and trigger alarm mechanisms; Regularly evaluate the effectiveness of computing power management strategies and use reinforcement learning technology to optimize control strategies.
[0006] Optionally, the step of building a prediction model based on the first data using a machine learning algorithm to predict future temperature changes, computing load, and computing power requirements of the data center includes: Creating time-related features, generating lagged features, and constructing interactive features after performing time alignment, processing missing values, and feature standardization and normalization on the first data; Select multiple preset basic models and develop a model fusion solution based on the model properties of the multiple basic models; Input the time-related features, lag features, and interaction features into the corresponding basic models for training, and fuse the trained multiple models according to the model fusion scheme to obtain a prediction model; Based on the first data and the prediction model, future temperature changes, computing load and computing power requirements of the data center are predicted.
[0007] Optionally, the step of dynamically adjusting the computing power resource allocation strategy according to the computing load and the computing power requirement includes: Real-time monitoring of first real-time status data of each computing node; Collecting first real-time task queue information of each computing node, wherein the task queue information includes the type, quantity, priority, and estimated execution time of tasks to be executed; Combining historical computing load data and special event data of each computing node, using a time series model to predict first load change data of each computing node within a first preset period; Build a task resource consumption model to estimate the resource requirements of different types of tasks; Combining the first real-time task queue information and the task resource consumption model, obtaining first real-time task resource consumption demand data; Adjust the computing power resource allocation strategy within the first preset period according to the computing load, the computing power requirement, the first real-time status data, the first load change data and the first real-time task resource consumption requirement data.
[0008] Optionally, adjusting the computing power resource allocation strategy within the first preset period according to the computing load, the computing power requirement, the first real-time status data, the first load change data, and the first real-time task resource consumption requirement data includes: Analyze the change trend of the computing load and computing power demand of each computing node according to the computing load, the computing power demand, the first real-time status data, the first load change data, and the first real-time task resource consumption demand data to obtain an analysis result; Using a time series analysis method, predict the change in computing power demand of each computing node within the first preset period to obtain a prediction result; Perform the following operations based on the analysis and prediction results: Dynamically adjust the number and specifications of virtual machines or containers on each computing node; Migrate tasks across nodes to balance computing load and heat distribution; Select the corresponding computing nodes for different types of computing tasks; Assign high-load tasks to computing nodes with lower temperatures; Migrate tasks to computing nodes that are close to the temperature threshold; The power consumption of each computing node is controlled through dynamic voltage and frequency scaling.
[0009] Optionally, a computing resource utilization optimization step is also included, specifically: Analyze historical computing power usage patterns based on historical work data of each computing node, identify inefficient or wasteful resource allocation plans, and adjust them; Implement a computing resource oversale strategy to improve overall resource utilization; Implement energy-saving measures for idle or low-load computing nodes.
[0010] Optionally, computing power cost management steps are also included, specifically: Calculate the computing power cost of each computing task in real time; Dynamically adjust task priorities and resource allocation strategies based on cost-benefit analysis; Generate computing power usage reports to calculate cost allocation and billing.
[0011] Optionally, the step of continuously monitoring the system operation status and computing power utilization efficiency, detecting anomalies and triggering an alarm mechanism includes: Use unsupervised learning algorithms to detect abnormal patterns in environmental and performance data and identify performance bottlenecks and resource contention; When an anomaly is detected, the corresponding processing mechanism and alarm are triggered.
[0012] Optionally, the step of regularly evaluating the effect of the computing power management strategy and optimizing the control strategy using reinforcement learning technology includes: Define state space and action space; Define a multi-dimensional reward function that includes computing power utilization, energy efficiency, and task completion time; Use deep reinforcement learning algorithms to learn optimal computing power allocation and management strategies; Verify the effectiveness of the strategy in a simulated environment and gradually apply it to the actual system through A / B testing.
[0013] Optionally, a visualization step is also included, specifically: Generate data center heat maps in real time; Display historical trends and forecast results of key performance indicators; Provides an interactive interface for administrators to adjust policies and perform manual intervention.
[0014] By adopting the technical solution of the present invention, the computing power data management method based on distributed computing includes: collecting the environmental data of the data center and the node status data of each computing node in real time through a distributed sensor network; collecting and integrating the computing power data of each computing node; transmitting the collected environmental data, node status data and computing power data to the central control system for preprocessing and analysis to obtain first data; based on the first data, using a machine learning algorithm to build a prediction model to predict the future temperature changes, computing load and computing power requirements of the data center; dynamically adjusting the computing power resource allocation strategy according to the computing load and the computing power requirements; optimizing the refrigeration system parameters according to the temperature changes; continuously monitoring the system operation status and computing power utilization efficiency, detecting anomalies and triggering an alarm mechanism; regularly evaluating the effectiveness of the computing power management strategy, and optimizing the control strategy using reinforcement learning technology. The solution of the embodiment of the present invention can improve the overall computing power utilization rate through accurate computing power demand prediction and dynamic resource allocation; combine environmental data and collaborative optimization of computing power allocation to reduce power supply efficiency and save energy consumption; reduce task completion time through intelligent task scheduling and load balancing; quickly identify and handle performance bottlenecks to improve system throughput; dynamic resource expansion and contraction mechanism enables the system to better cope with load fluctuations, and the supported peak load can be increased; accurate cost analysis and resource allocation optimization can reduce overall operating costs; automated management reduces manual intervention and reduces labor costs; active anomaly detection and processing mechanism can reduce system failures caused by resource problems; intelligent heat pipe Management can extend the life of hardware; detailed computing power usage reports support more accurate cost allocation and improve billing accuracy; multi-dimensional performance indicator analysis can provide deeper system insights and support better decision-making; control strategies based on reinforcement learning can be continuously optimized, enabling the system to adapt to changing workloads and environmental conditions; using advanced machine learning algorithms, the accuracy of computing power demand prediction can be improved; more efficient resource allocation can reduce user-perceived delays, and more stable system performance can improve user satisfaction; by optimizing energy use and improving efficiency, the carbon emissions of data centers can be reduced; more efficient resource utilization and thermal management allow for higher density server deployment, which can increase computing density. The solution of this embodiment can not only significantly improve the efficiency and performance of distributed computing systems, but also reduce operating costs, improve reliability and sustainability; this comprehensive computing power data management method provides a powerful optimization tool for modern large-scale distributed computing environments, which can create significant technical advantages and economic value in a highly competitive market. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 is a schematic block diagram of a computing power data management system based on distributed computing provided by an embodiment of the present invention; Figure 2It is a flow chart of a computing power data management method based on distributed computing provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0016] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.
[0017] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.
[0018] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.
[0019] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0020] Refer to the following Figure 1 to Figure 2 To describe a computing power data management system and method based on distributed computing provided according to some embodiments of the present invention.
[0021] like Figure 1 As shown, an embodiment of the present invention provides a computing power data management system based on distributed computing, including: a cloud server, a data center, and multiple computing nodes; The cloud server is configured as follows: Collect data center environmental data and node status data of each computing node in real time through a distributed sensor network; Collect and integrate computing power data of each computing node; The collected environmental data, node status data and computing power data are transmitted to the central control system for preprocessing and analysis to obtain first data; Based on the first data, a prediction model is constructed using a machine learning algorithm to predict future temperature changes, computing loads, and computing power requirements of the data center; Dynamically adjust the computing resource allocation strategy according to the computing load and the computing power demand; Optimizing refrigeration system parameters according to the temperature change; Continuously monitor the system operation status and computing power utilization efficiency, detect anomalies and trigger alarm mechanisms; Regularly evaluate the effectiveness of computing power management strategies and use reinforcement learning technology to optimize control strategies.
[0022] It should be known that Figure 1 The block diagram of the computing power data management system based on distributed computing is for illustration only, and the number of modules shown does not limit the protection scope of the present invention. The computing power data management system based on distributed computing provided in this embodiment can be used to execute the corresponding embodiments of the computing power data management method based on distributed computing. For the specific implementation process, please refer to the description of the following method embodiments, which will not be repeated here.
[0023] See also Figure 2 Another embodiment of the present invention provides a computing power data management method based on distributed computing, comprising: Collect the environmental data of the data center (the environmental data includes temperature, humidity, air flow speed, etc.) and the node status data of each computing node (the node status data includes CPU temperature, power consumption, utilization, etc.) in real time through a distributed sensor network; In this step, temperature, humidity, and air flow velocity sensors are installed at key locations in the data center, and CPU temperature and power consumption monitoring modules are deployed on each computing node to ensure that sensors cover key areas such as hot and cold channels, cabinet rooms, and equipment air inlets and outlets. Low-power, high-reliability data acquisition terminals are developed to implement communication interfaces between sensors and acquisition terminals, such as I2C, SPI, UART, etc., and data caching mechanisms are designed to prevent data loss. Hierarchical network topologies are constructed, such as star or tree structures, to implement low-latency, high-bandwidth data transmission protocols, such as MQTT or CoAP, and network redundancy and failover mechanisms are designed. Preliminary data filtering and compression are performed on edge devices to implement data formatting and standardization, perform simple anomaly detection, and filter out obvious erroneous data. Secure data transmission channels are established, such as TLS encryption, to achieve Data block transmission and breakpoint continuation mechanism, design distributed data storage system, such as time series database; develop visual dashboard, display environmental data and node status in real time, realize multi-dimensional data correlation analysis, set configurable alarm threshold and notification mechanism; implement sensor calibration and regular maintenance plan, develop data consistency check algorithm, realize data compensation and interpolation processing, and handle missing or abnormal data; integrate with data center infrastructure management (DCIM) system, realize data interaction interface with computing power scheduling system, develop API, support third-party system integration; realize plug-in architecture, support rapid access of new types of sensors, design distributed data processing framework, support large-scale data analysis; realize dynamic load balancing, optimize data flow processing; implement equipment authentication and authorization mechanism, encrypt and store sensitive data, and establish a complete audit log system. The solution of this step can realize high-precision environmental monitoring, improve real-time performance and large-scale monitoring capabilities, have high reliability, improve energy efficiency, extend equipment life, predict faults, optimize resource utilization, and improve operation and maintenance efficiency. This high-precision, real-time environment and performance monitoring system provides a solid data foundation for the intelligent management of data centers, which not only significantly improves operational efficiency and reliability, but also provides possibilities for further intelligent decision-making and optimization.
[0024] Collect and integrate computing power data of each computing node (including CPU usage, memory usage, network bandwidth utilization, dedicated accelerator performance indicators, container resource usage, and storage I / O performance); In this step, specifically: develop a lightweight, cross-platform data collection agent that supports mainstream operating systems (Linux, Windows, macOS), implements containerized deployment, and supports Docker and Kubernetes environments; designs a plug-in architecture to facilitate the expansion of new data collection modules; uses operating system-specific APIs (such as Linux's / proc / stat and Windows's Performance Counters) to collect overall CPU usage and each core's usage, distinguish between user state, system state, and IO wait time, and implement CPU frequency and temperature monitoring; collects physical memory and virtual memory usage, monitors memory paging and swapping activities, and tracks memory leaks and large memory consuming processes; monitors inbound and outbound traffic on network interfaces, calculates real-time throughput and utilization, and analyzes network connection status and quality indicators (such as latency and packet loss rate); integrates GPU monitoring modules (such as NVML for NVIDIA GPUs and ROCm for AMD GPUs), and develops monitoring interfaces for FPGAs and other AI accelerators; collects computing utilization, memory usage, temperature, energy consumption, and other indicators; integrates container runtime APIs (Docker API, Kubernetes Metrics API), collect container-level CPU, memory, network, and storage usage data, implement container-to-host resource mapping, and support multi-tenant environments; use system tools (such as iostat) or custom modules to monitor disk I / O, measure key indicators such as IOPS, throughput, latency, distinguish between read and write operation performance, and monitor the health status of storage devices; perform data aggregation and preliminary analysis at the node level; implement data compression algorithms to reduce network transmission load; perform edge computing, identify and mark abnormal data; implement TLS encrypted transmission, design identity authentication and authorization mechanisms, and implement data integrity verification; develop high-concurrency data receiving services, implement load balancing and horizontal expansion mechanisms, and design data parsing and verification logic; select appropriate time series databases (such as InfluxDB, Prometheus); optimize data models and indexing strategies; implement data sharding and replication to ensure high availability; develop real-time analysis systems based on stream processing frameworks (such as Apache Flink, Spark Streaming), implement complex event processing (CEP) logic, design dynamic alarm thresholds and intelligent alarm mechanisms; develop interactive dashboards to support multi-dimensional data display; implement custom report generation functions; support data export and API interfaces to facilitate integration with other systems; develop anomaly detection models to identify performance anomalies; implement resource usage prediction models to support capacity planning; integrate recommendation systems to provide optimization suggestions; implement health monitoring of the collection system itself, develop automated problem diagnosis and repair mechanisms, and design disaster recovery and failover solutions.This comprehensive, high-precision computing data management system not only greatly improves resource utilization efficiency and system performance, but also provides a solid data foundation for intelligent operations and continuous optimization; it can also significantly reduce operating costs, improve service quality, and enhance decision-making capabilities, thereby creating significant technical advantages and economic value in the highly competitive cloud computing and big data markets.
[0025] The collected environmental data, node status data and computing power data are transmitted to the central control system for preprocessing and analysis to obtain first data; In this step, the following are specifically included: aggregating data within a short time window (such as 1 minute) on each node, encoding data using an efficient data serialization format (such as Protocol Buffers, Apache Avro); implementing incremental encoding to transmit only changed data; implementing an encrypted channel based on TLS 1.3, using two-way authentication to ensure the trustworthiness of the data source, and implementing data compression (such as GZIP, LZ4) to reduce the amount of transmission; using a UDP-based reliable transmission protocol (such as QUIC) to improve transmission efficiency, implementing adaptive flow control and congestion avoidance algorithms, and designing retransmission and error recovery mechanisms to ensure data integrity; designing a distributed receiving gateway to support horizontal expansion; implementing load balancing to ensure evenly distributed data flows; using a message queue (such as Apache Kafka) as a data buffer layer; implementing an efficient data parser to support parallel processing; performing data integrity and consistency checks; identifying and processing abnormal data formats; using a stream processing framework (such as Apache Flink) to build real-time processing pipelines; implement data cleaning, handle missing values and outliers; perform unit conversion and standardization; associate data from different sources (environment, state, computing power); implement time synchronization, handle timestamp differences between different data sources; generate comprehensive performance indicators (such as energy efficiency ratio PUE); calculate statistical features (mean, variance, trend, etc.); implement sliding window analysis to capture short-term fluctuations; generate composite features, such as resource utilization percentage; implement anomaly detection algorithms based on statistics and machine learning, use historical data to establish a normal behavior baseline, and mark and classify detected anomalies; use principal component analysis (PCA) or t-SNE for high-dimensional data visualization, implement online clustering algorithms, and identify similar performance patterns; generate low-dimensional representations of system states; deploy pre-trained machine learning models (such as time series prediction); implement online update mechanisms for models to adapt to changes in data distribution; generate short-term and long-term performance forecasts; calculate key performance indicators (KPIs) and service level objectives (SLOs); implement complex event processing (CEP) to detect performance event sequences; generate real-time dashboard data; implement multi-level storage strategies, hot data is stored in in-memory databases, and columnar storage (such as Apache Parquet) to optimize analytical queries and implement data aging and archiving mechanisms; design metadata storage to record data lineage; implement data version control to support rollback and auditing; manage data model evolution to ensure backward compatibility. This efficient and real-time data processing and analysis system can significantly improve the operational efficiency, reliability and decision-making capabilities of data centers; it not only provides a solid data foundation for intelligent management and optimization, but also helps organizations quickly extract valuable insights from massive data, thereby gaining significant advantages in the fiercely competitive digital economy.
[0026] Based on the first data, a prediction model is constructed using a machine learning algorithm (including but not limited to regression analysis, time series analysis, neural network, and ensemble learning method) to predict future temperature changes, computing load, and computing power requirements of the data center; Dynamically adjust computing resource allocation strategies (including task scheduling, load balancing, and resource expansion and contraction) according to the computing load and computing power requirements; Optimizing refrigeration system parameters according to the temperature change; In this step, the optimization of refrigeration system parameters includes: adjusting the cold air flow direction according to the thermal map analysis; dynamically adjusting the power output of the refrigeration equipment; and optimizing the cold and hot channel isolation strategy.
[0027] Continuously monitor the system operation status and computing power utilization efficiency, detect anomalies and trigger alarm mechanisms; Regularly evaluate the effectiveness of computing power management strategies and use reinforcement learning technology to optimize control strategies.
[0028] Through the technical solution of this embodiment, the overall computing power utilization rate can be improved through accurate computing power demand prediction and dynamic resource allocation; the coordinated optimization of environmental data and computing power allocation can reduce power supply efficiency and save energy consumption; through intelligent task scheduling and load balancing, the task completion time can be reduced; the rapid identification and processing of performance bottlenecks can improve system throughput; the dynamic resource expansion and contraction mechanism enables the system to better cope with load fluctuations, and the supported peak load can be increased; accurate cost analysis and resource allocation optimization can reduce overall operating costs; automated management reduces manual intervention and can reduce labor costs; active anomaly detection and processing mechanisms can reduce system failures caused by resource problems; intelligent thermal Management can extend the life of hardware; detailed computing power usage reports support more accurate cost allocation and improve billing accuracy; multi-dimensional performance indicator analysis can provide deeper system insights and support better decision-making; control strategies based on reinforcement learning can be continuously optimized, enabling the system to adapt to changing workloads and environmental conditions; using advanced machine learning algorithms, the accuracy of computing power demand prediction can be improved; more efficient resource allocation can reduce user-perceived delays, and more stable system performance can improve user satisfaction; by optimizing energy use and improving efficiency, the carbon emissions of data centers can be reduced; more efficient resource utilization and thermal management allow for higher density server deployment, which can increase computing density. The solution of this embodiment can not only significantly improve the efficiency and performance of distributed computing systems, but also reduce operating costs, improve reliability and sustainability; this comprehensive computing power data management method provides a powerful optimization tool for modern large-scale distributed computing environments, which can create significant technical advantages and economic value in a highly competitive market.
[0029] In some possible implementations of the present invention, the step of building a prediction model based on the first data using a machine learning algorithm to predict future temperature changes, computing load, and computing power requirements of the data center includes: The first data is time-aligned (to ensure consistent time intervals), missing values are handled (using methods such as interpolation or forward filling), feature standardization and normalization are performed to create time-related features (such as hours, days of the week, months, etc.), generate lagged features (to capture historical trends), and construct interactive features (such as the product of CPU usage and temperature); In this step, time alignment can ensure that all data indicators have consistent timestamps and sampling intervals, for example, data collected at intervals of 5 minutes. This is the basis for subsequent feature engineering and modeling; missing value processing refers to the use of interpolation or forward filling methods to fill missing values in the data, which can ensure the continuity and integrity of the data; feature standardization and normalization refers to standardizing (subtracting the mean, dividing by the standard deviation) and normalizing (scaling to the range of 0-1) the original features so that the features have the same dimension and numerical range, which helps to improve the convergence speed and generalization performance of the model; time-related feature creation refers to creating time-related features such as hours, days of the week, and months based on timestamp information. These features can help the model learn the periodic patterns in the data; lag feature generation refers to creating lag features within the historical time window, such as certain indicator values in the previous hour, day, and week, which can help the model learn the trends and dependencies in the data series; interactive feature construction refers to combining interactive features such as products and quotients between different features. For example, the product of CPU usage and temperature can reflect the impact of CPU load on temperature. This feature engineering helps capture the complex relationship between features. In general, these data preprocessing and feature engineering steps are designed to mine richer and more predictive features from the raw data, laying the foundation for subsequent model training and performance optimization.
[0030] Select multiple preset basic models (such as regression analysis model, time series analysis model, neural network model, ensemble learning model, etc.), and develop model fusion solutions based on the model attributes (such as functions, processing capabilities, etc.) of multiple basic models (including: model combination strategies, such as weighted average, voting, etc.; dynamic model selection mechanism, select the best model according to real-time performance; meta-learning system, automatically select the model suitable for a specific scenario; etc.); In this step, you can select a variety of different types of basic models according to business needs and data characteristics, such as linear regression, time series model, neural network, decision tree, etc.; train the selected multiple basic models separately, and evaluate the prediction performance of each model on the validation set; determine the specific strategy of model fusion according to the model attributes (such as function, processing power, etc.) of multiple basic models, such as weighted average, voting, stacking learning, etc., and the weight distribution can be determined according to the performance of each model on the validation set; if a dynamic model selection mechanism is adopted, it is necessary to design a method to monitor the performance of each model in real time and automatically select the best model according to performance indicators; for a meta-learning system, it is necessary to establish a meta-level model to automatically select the most appropriate combination of basic models according to data characteristics; use the selected fusion strategy to train the fusion model, and evaluate its prediction performance on the validation set; adjust the parameters of the fusion strategy, such as the weight coefficient, to further optimize the performance of the fusion model. Through such a model fusion method, we can improve prediction accuracy (different models have their own advantages and disadvantages, and through fusion, we can give full play to the advantages of each model and obtain more accurate prediction results), enhance model robustness (fusion of multiple basic models can improve the robustness of the overall model and reduce the limitations of a single model), enhance adaptability (dynamic model selection and meta-learning systems can automatically select the best model combination according to data characteristics to improve the adaptability of the model), and improve interpretability (adding the interpretability of the basic model to the fusion strategy can improve the interpretability of the overall model). In general, model fusion is an effective technical means to improve prediction performance. By making full use of the advantages of different models, more accurate and robust prediction results can be obtained.
[0031] Input the time-related features, lag features, and interaction features into the corresponding basic models for training, and fuse the trained multiple models according to the model fusion scheme to obtain the prediction model (including: rolling prediction mechanism, regularly updating the prediction results; multi-scale prediction mechanism, including short-term (hourly), medium-term (day-level) and long-term (weekly / monthly) predictions; build a probabilistic prediction framework to generate prediction intervals; etc.); In this step, different types of features are input into the corresponding basic models for training, such as: time-related features are input into the time series model, lagged features are input into the regression model, and interactive features are input into the neural network model; according to the previously designed model fusion scheme, the trained basic models are fused to obtain the final prediction model. The fusion strategy can include weighted average, voting mechanism, dynamic model selection, and meta-learning system; in practical applications, a rolling prediction method can be used to regularly update the prediction results, such as retraining and predicting the model with the latest data every certain period of time (such as 1 hour); short-term, medium-term and long-term multi-scale prediction models can be constructed for different needs; short-term predictions (hourly level) can be used for real-time monitoring and early warning, medium-term predictions (days) can be used for business planning, and long-term predictions (weeks / months) can be used for strategic decision-making; based on the prediction results, a probabilistic prediction framework can also be constructed to generate corresponding confidence intervals for the prediction results, which can better reflect the uncertainty of the prediction and provide richer information for decision-making. Through the above methods, making full use of various features and integrating multiple basic models can greatly improve the accuracy of predictions; rolling predictions and multi-scale predictions can enhance the model's adaptability to new data and improve overall robustness; the probabilistic prediction framework can provide richer information for decision-making, which helps to improve the scientificity and rationality of decision-making; incorporating the interpretability of the basic model into the fusion strategy can improve the interpretability of the overall model. In general, this method of comprehensive utilization of feature engineering and model fusion can effectively improve the overall performance of the prediction model and provide strong decision support for practical applications.
[0032] Based on the first data and the prediction model, future temperature changes, computing load and computing power requirements of the data center are predicted.
[0033] In this embodiment, it is possible to improve prediction accuracy, shorten prediction response time, enhance model adaptability and scalability, optimize resource utilization, improve abnormal warning accuracy, etc. This prediction system based on advanced machine learning not only greatly improves the operational efficiency and reliability of the data center, but also provides strong support for intelligent decision-making and continuous optimization; it enables the data center to shift from passive response to active management, significantly reducing costs and environmental impact while improving service quality, and provides key support for enterprises to maintain their competitive advantage in the digital economy era.
[0034] In some possible implementations of the present invention, the step of dynamically adjusting the computing power resource allocation strategy according to the computing load and the computing power requirement includes: Real-time monitoring of the first real-time status data of each computing node (including CPU, memory, storage, and network usage); In this step, the monitoring agent deployed on each computing node regularly collects key indicators such as CPU utilization, memory usage, disk I / O, network traffic, etc.; open source monitoring tools such as Prometheus and Telegraf can be used to quickly deploy and configure data collection rules; the collected real-time status data is transmitted to a centralized monitoring platform or time series database; data can be transmitted using message queue systems (such as Kafka), HTTP APIs, or directly stored in time series databases (such as InfluxDB); a dashboard is built on the monitoring platform to display the key indicators of each node in real time and visualize the running status of the node; Grafana can be used Open source tools such as RESTfull can be used to quickly build a visual dashboard; set threshold rules to monitor whether each indicator exceeds the normal range in real time and detect abnormal situations in time; combine machine learning models to intelligently detect abnormal patterns and improve the accuracy and sensitivity of monitoring; once an abnormal situation is found, immediately notify relevant personnel or systems through email, SMS or webhook; integrate alarm management system to achieve flexible notification strategies and multi-channel alarms; store the collected historical status data to support subsequent capacity planning, performance analysis and trend prediction; use the powerful query capabilities of time series database to quickly analyze and mine historical data. Through the above methods, key indicators of each node can be collected and displayed in real time, and abnormal situations can be quickly discovered and handled; abnormalities can be discovered and handled in time to avoid service interruptions caused by hardware or software problems; based on historical data analysis, system capacity can be better planned and system performance can be optimized; automated monitoring and alarm mechanisms can reduce the workload of operation and maintenance personnel; make full use of monitoring data to provide data support for system upgrades, resource allocation, etc. In general, this real-time monitoring solution can greatly improve the reliability and observability of the system and provide strong support for system operation and maintenance and optimization.
[0035] Collecting first real-time task queue information of each computing node, wherein the task queue information includes the type, quantity, priority, and estimated execution time of tasks to be executed; In this step, the monitoring agent deployed on each computing node regularly collects detailed information about the current task queue, including key indicators such as the type, quantity, priority, and estimated execution time of the tasks to be executed. The API interfaces provided by various computing frameworks or scheduling systems, such as Kubernetes' Pod status and Apache Spark's job queue, can be used to obtain the required task information. The collected task queue data is transmitted to a centralized monitoring platform or time series database. Message queue systems (such as Kafka), HTTP Data can be transmitted and stored through API or directly stored in a time series database (such as InfluxDB); a dashboard can be built on the monitoring platform to display the task queue overview of each node in real time, including task type, quantity, priority, and estimated execution time; open source tools such as Grafana can be used to quickly build a visual dashboard; threshold rules can be set to monitor in real time whether there are any abnormalities in the task queue, such as task backlogs, high-priority task accumulation, etc.; machine learning models can be combined to intelligently detect abnormal patterns in task queues to improve the accuracy and sensitivity of monitoring; once an abnormality in the task queue is found, the relevant personnel or system will be notified immediately by email, SMS, or webhook; an alarm management system can be integrated to implement flexible notification strategies and multi-channel alarms; the collected historical task queue data can be stored to support subsequent capacity planning, task scheduling optimization, and trend prediction; the powerful query capabilities of the time series database can be used to quickly analyze and mine historical data. Through the above method, the key indicators of the task queue of each node can be collected and displayed in real time, and abnormal situations can be quickly discovered and handled; based on the real-time data of the task queue, the scheduling strategy can be optimized and the task execution efficiency can be improved; using historical data analysis, the system capacity can be better planned and the task scheduling performance can be optimized; the automated monitoring and alarm mechanism reduces the workload of operation and maintenance personnel; and the task queue data can be fully utilized to provide data support for system upgrades, resource allocation, etc. In general, this task queue monitoring solution can help grasp the real-time operating status of the system and provide strong support for task scheduling optimization and system operation and maintenance.
[0036] Combine the historical computing load data of each computing node and special event data (such as promotional activities), and use a time series model (such as ARIMA, LSTM) to predict the first load change data of each computing node in the first preset period; In this step, the historical computing load data of each computing node is collected, including CPU usage, memory usage, disk I / O and other indicators; at the same time, relevant special event data, such as promotional activities, holidays, etc., are collected as external influencing factors; the collected data is preprocessed by cleaning, missing value processing, outlier processing and other preprocessing work to ensure data quality; according to business needs, relevant feature variables are selected, such as time series features (such as timestamps, cycles, etc.), event features (such as promotional activities, holidays, etc.); the features are encoded, normalized and other processing is performed to ensure that the data format is unified and can be used for modeling; according to the characteristics of the data, a suitable time series model is selected, such as ARIMA, LSTM, Prophet, etc.; the preprocessed data set is divided into a training set and a validation set, Use the training set to train the model and evaluate the model performance on the validation set; adjust the model parameters and continuously optimize the model performance until the prediction accuracy requirements are met; use the trained model to predict the load changes of each computing node in the future preset period based on the current time node and the feature values in the future (such as timestamps, event labels, etc.); predict the change trends of key indicators such as CPU usage, memory usage, and disk I / O; visualize the prediction results in the form of charts to facilitate intuitive analysis and understanding of future load changes; use Python visualization libraries such as Matplotlib and Plotly to generate corresponding charts; analyze the prediction results and combine them with actual business conditions to determine whether system resources or emergency plans need to be adjusted. Through the above method, the time series model combined with historical data and special event data can more accurately predict the changes in the load of computing nodes in the future; based on the prediction results, system resources can be planned and allocated in advance to avoid performance bottlenecks caused by load peaks; understanding the future load change trends in advance is conducive to formulating targeted emergency plans and strategies; make full use of data-driven prediction results to provide decision support for system upgrades, resource allocation, etc.; intuitive chart displays help managers quickly understand and control the load of the system. In general, this load forecasting solution based on time series models can help better plan and allocate system resources and improve overall operation and maintenance efficiency and service quality.
[0037] Build a task resource consumption model to estimate the resource requirements of different types of tasks; In this step, historical running data of various types of tasks are collected, including task type, input data size, execution time, CPU / memory / storage and other resource consumption; data is pre-processed such as cleaning, missing value processing, and outlier processing to ensure data quality; resource indicators that need to be modeled are determined according to business needs, such as CPU utilization, memory usage, disk I / O, etc.; relevant feature variables are selected according to task characteristics, such as task type, input data size, concurrency, etc.; features are encoded and scaled to ensure that the data format is unified and can be used for modeling; different feature combinations are tried to observe the impact on model performance and select the optimal feature set; according to the resource consumption characteristics of the task, select Appropriate machine learning models, such as linear regression, decision tree regression, neural network, etc.; divide the preprocessed data set into training set and validation set, use the training set to train the model, and evaluate the model performance on the validation set; adjust the model parameters and continuously optimize the model performance until the prediction accuracy requirements are met; deploy the trained model to the actual system and use new task data for verification; monitor the prediction performance of the model in the production environment and fine-tune the model parameters when necessary; use the trained model to predict the resource consumption required for the task based on the characteristic values of the new task (such as task type, input data scale, etc.); feed back the prediction results to the task scheduling system to support reasonable allocation of resources and improve resource utilization efficiency. Through the above method, the resource consumption of different types of tasks can be more accurately predicted by combining the machine learning model with historical task data; based on the prediction results, more accurate resource allocation can be carried out to avoid resource waste or shortage; through the reasonable use of resources, the hardware investment cost of servers, storage, etc. can be reduced; more accurate resource demand prediction is conducive to the dynamic expansion and contraction of the system on demand and improve scalability; make full use of data-driven prediction results to provide decision support for system upgrades, resource allocation, etc. In general, this machine learning-based task resource consumption model can help better plan and allocate system resources and improve overall operation and maintenance efficiency and service quality.
[0038] Combining the first real-time task queue information and the task resource consumption model, obtaining first real-time task resource consumption demand data; In this step, the first real-time task queue information includes task ID, task type, input data size, etc. For each task in the queue, the previously established task resource consumption model is used to input task-related features to predict the resource consumption of the task, such as CPU, memory, storage, etc. The task queue information and the task resource consumption prediction results are associated and integrated to form the resource consumption demand data of the first real-time task; the data is stored in a database or other storage medium for subsequent task scheduling and resource management. Through the above method, combined with the real-time information of the task queue and the prediction results of the resource consumption model, more accurate resource demand data can be obtained; mastering the resource consumption demand of the first real-time task can support more dynamic and accurate resource allocation and improve resource utilization efficiency; timely understanding the task resource demand can make resource reservation and scheduling in advance, shorten the response and processing time of the task; reasonably allocate resources to avoid performance degradation or task failure caused by resource shortage, and improve the reliability of the service; combined with historical data, the task scheduling strategy can be further optimized to achieve more intelligent resource management. In general, this method based on task queues and resource consumption models can help grasp the resource requirements of key tasks in real time, provide strong support for the system's dynamic resource management and scheduling, and improve overall operating efficiency and service quality.
[0039] Adjust the computing power resource allocation strategy within the first preset period according to the computing load, the computing power requirement, the first real-time status data, the first load change data and the first real-time task resource consumption requirement data.
[0040] Through the solution of this embodiment, it is possible to achieve resource utilization optimization, performance improvement, cost savings, scalability improvement, load balancing effect, fault recovery capability, resource prediction accuracy, automation level, energy efficiency, security improvement, user experience, operation and maintenance efficiency, business agility, and environmental impact; this dynamic and intelligent computing resource allocation strategy not only significantly improves the performance and efficiency of the system, but also greatly reduces operating costs and environmental impact; it enables data centers to respond to load changes more flexibly, provide more stable and high-quality services, and optimize resource use, creating significant competitive advantages for enterprises. Through continuous learning and optimization, this system can continuously adapt to new business needs and technological changes, providing strong support for the long-term development of enterprises.
[0041] In some possible implementations of the present invention, a dynamic task scheduling algorithm design step is also included, specifically: implementing a multi-objective scheduling algorithm to balance factors such as resource utilization, energy consumption, and task completion time; considering task affinity and anti-affinity to optimize data locality; implementing task priority and preemption mechanisms to ensure timely execution of critical tasks.
[0042] In some possible implementations of the present invention, the method for implementing the load balancing strategy includes: designing a dynamic load balancing algorithm, such as minimum number of connections, weighted polling, etc.; designing intelligent load prediction and distribution based on machine learning; considering network topology and latency, and optimizing cross-regional load balancing.
[0043] In some possible implementations of the present invention, the resource expansion and contraction mechanism includes: setting resource utilization thresholds to trigger automatic expansion and contraction; implementing predictive expansion to cope with load growth in advance; designing hierarchical expansion strategies, such as hot and cold node pools; In some possible implementations of the present invention, it also includes designing an intelligent resource allocation engine, specifically: developing a resource allocation strategy based on reinforcement learning to achieve multi-dimensional resource orchestration (considering factors such as CPU, memory, network, storage, etc.); designing an automatic evaluation and optimization cycle of the resource allocation strategy.
[0044] In some possible implementations of the present invention, adjusting the computing power resource allocation strategy within the first preset period according to the computing load, the computing power requirement, the first real-time status data, the first load change data, and the first real-time task resource consumption requirement data includes: Analyze the change trend of the computing load and computing power demand of each computing node according to the computing load, the computing power demand, the first real-time status data, the first load change data, and the first real-time task resource consumption demand data to obtain an analysis result; Using a time series analysis method, predict the change in computing power demand of each computing node within the first preset period to obtain a prediction result; Perform the following operations based on the analysis and prediction results: Dynamically adjust the number and specifications of virtual machines or containers on each computing node; Migrate tasks across nodes to balance computing load and heat distribution; Select the corresponding computing nodes for different types of computing tasks (such as CPU-intensive, memory-intensive, and I / O-intensive); Assign high-load tasks to computing nodes with lower temperatures; Migrate tasks to computing nodes that are close to the temperature threshold; The power consumption of each computing node is controlled through dynamic voltage and frequency scaling.
[0045] In this embodiment, the overall computing load information, computing power demand data, first real-time status data, first load change data and first real-time task resource consumption demand data of the system are collected and stored in real time; the collected data are analyzed to analyze the changing trend of the computing load and computing power demand of each computing node to obtain the analysis results; the computing power demand changes in a future period of time (i.e., within a first preset period) are predicted using methods such as time series analysis to obtain the prediction results; according to the analysis results and the prediction results, the computing power resource allocation strategy within the first preset period is dynamically adjusted, including: dynamically adjusting the number and specifications of virtual machines or containers on each computing node; performing cross-node task migration to balance Computing load and heat distribution; select corresponding computing nodes for different types of computing tasks (such as CPU-intensive, memory-intensive, and I / O-intensive); assign high-load tasks to computing nodes with lower temperatures; perform task migration on computing nodes close to the temperature threshold; control the power consumption of each computing node through dynamic voltage and frequency adjustment to increase or reduce the computing resources allocated to the first real-time task, reasonably allocate computing resources between multiple real-time tasks, and adjust the resource elastic scaling strategy to adapt to load changes; implement corresponding computing resource adjustment operations according to the adjusted resource allocation strategy; continuously monitor the system status and further optimize the resource allocation strategy according to the actual operation situation. Through the above methods, dynamic adjustment of resource allocation can better meet the computing power requirements of real-time tasks and reduce resource waste; reasonable allocation of computing resources can shorten the processing time of real-time tasks and improve system response speed; timely adjust resources according to load changes to avoid performance degradation or task failure caused by insufficient resources and improve service reliability; through dynamic elastic scaling, resources can be flexibly allocated according to demand changes to reduce overall operating costs; analyzing historical data trends can provide a basis for future resource planning and scheduling strategy optimization. In general, this dynamic resource allocation method based on comprehensive data analysis can help adjust computing resources in real time to better meet changing real-time task requirements and improve the overall performance and service quality of the system.
[0046] In some possible implementations of the present invention, a computing resource utilization optimization step is also included, specifically: Analyze historical computing power usage patterns based on historical work data of each computing node, identify inefficient or wasteful resource allocation plans, and adjust them; In this step, the historical work data of each computing node is collected and stored in real time, including indicators such as CPU utilization, memory usage, and task execution time; the collected historical work data is analyzed to identify the computing power usage pattern of each computing node, including: the fluctuation trend of resource utilization, resource consumption of different task types, and the correlation of resource usage between nodes; based on the analysis results, the computing nodes or task allocation schemes with low resource utilization efficiency are identified, such as: nodes that are in a low-load state for a long time, node allocation schemes with serious resource imbalance, and task allocations with serious resource waste; for the identified inefficient resource allocation, optimization strategies are formulated, such as: adjusting task allocation to allocate tasks with heavy loads to nodes with surplus resources; dynamically reducing the resource quota of low-load nodes and allocating excess resources to other high-load nodes; predicting future loads based on historical patterns and making resource elastic scaling in advance. According to the optimization strategy, the corresponding resource adjustment operations are implemented; the system status is continuously monitored, and the resource allocation strategy is further optimized based on the actual operation situation. The above method can identify and adjust inefficient resource allocation schemes, and make full use of the computing power resources of each computing node; dynamically adjust resource configuration according to load conditions to avoid resource waste and reduce overall operating costs; reasonably allocate resources to ensure that high-load tasks have sufficient computing power support and improve service reliability; based on the prediction of historical usage patterns, resource elasticity adjustments can be made in advance to quickly respond to load changes; through data analysis, historical rules can be mined to provide a basis for future resource management strategy optimization. In general, this dynamic resource optimization method based on historical work data analysis can help identify inefficient resource usage in the system and make timely adjustments, thereby improving overall resource utilization efficiency and service quality.
[0047] Implement a computing resource oversale strategy to improve overall resource utilization; In this step, the workload data of each computing node in the past period of time is collected, including indicators such as CPU utilization, memory usage, and task execution time; these historical data are analyzed to understand the resource usage characteristics of different types of tasks and the resource usage correlation between nodes; based on historical data, a workload prediction model is established using machine learning and other technologies to predict resource demand in the future; the prediction model should be able to capture the seasonality and trend characteristics of the workload to improve the prediction accuracy; based on the prediction model, the resource usage of each computing node in the future period of time is evaluated; a reasonable oversold coefficient is set, that is, a certain proportion of resources are appropriately over-allocated on the actual available resources of the node; the determination of the oversold coefficient requires balancing the risk of improving resource utilization and service quality; the actual workload situation is continuously monitored and compared with the prediction results; when the actual load exceeds the available resources of a node, the resource allocation is adjusted in time, for example: temporarily borrowing resources from other nodes, dynamically adjusting task allocation according to priority, notifying users to temporarily reduce resource usage, etc.; collect and analyze the implementation effect data of the resource oversold strategy, and continuously optimize the oversold coefficient and related parameters; according to the actual operation of the system, adjust the oversold strategy to improve resource utilization while ensuring service quality. By implementing this computing power resource overselling strategy, resources can be oversold reasonably, which can greatly improve the overall computing power resource utilization efficiency; improve resource utilization and reduce the actual cost of each unit of resources; dynamically adjust resource allocation to maximize resource utilization while ensuring service quality; overselling strategies based on load forecasting can better adapt to changes in workloads; continuously optimize overselling strategies to provide data support for future resource management decisions. In general, by implementing a computing power resource overselling strategy, overall resource utilization can be significantly improved, and operating costs can be reduced while ensuring service quality.
[0048] Implement energy-saving measures (such as frequency reduction or shutdown) for idle or low-load computing nodes.
[0049] In some possible implementations of the present invention, a computing power cost management step is also included, specifically: Calculate the computing power cost of each computing task in real time; Dynamically adjust task priorities and resource allocation strategies based on cost-benefit analysis; Generate computing power usage reports to calculate cost allocation and billing.
[0050] In this embodiment, real-time calculation of the computing power cost of each computing task includes: collecting resource usage of each computing task, including CPU time, memory usage, network I / O and other indicators; combining the actual cost of each resource (such as CPU cost, memory cost, etc.) to calculate the overall computing power cost of each task in real time; associating the computing power cost information of the task with the task metadata (such as task type, priority, user, etc.). Dynamically adjusting task priority and resource allocation strategy based on cost-effectiveness analysis includes: continuously monitoring the computing power cost of each task, and comparing and analyzing it with the business value or SLA requirements of the task; dynamically adjusting the priority of tasks to ensure that high-value or urgent tasks can get priority in resource allocation; adjusting resource allocation strategy in real time according to task priority and cost-effectiveness, such as: preemptive resource allocation (temporarily suspending low-priority tasks and allocating resources to high-priority tasks), elastic scaling (dynamically increasing or releasing resources according to task load). Generating computing power usage reports to count cost allocation and billing includes: collecting and summarizing computing power cost data for each computing task; counting cost allocation according to task attribution (user, department, project, etc.); generating regular computing power usage reports for cost accounting, billing, etc. By implementing this dynamic resource management strategy based on computing power cost, resource allocation can be dynamically adjusted according to task value and cost to give full play to the effectiveness of key tasks; accurate computing power cost accounting helps to reasonably allocate and control computing resource costs; quickly respond to changes in business needs and flexibly adjust resource allocation strategies; ensure that high-value or urgent tasks receive sufficient resource support and improve the service level of key businesses; detailed computing power usage reports provide a basis for future resource planning and investment decisions. In general, by calculating computing power costs in real time and dynamically adjusting resource allocation, the efficiency of computing resource utilization and business value can be greatly improved, while enhancing cost visibility and service quality assurance.
[0051] In some possible implementations of the present invention, the step of continuously monitoring the system operating status and computing power utilization efficiency, detecting anomalies and triggering an alarm mechanism includes: Use unsupervised learning algorithms to detect abnormal patterns in environmental and performance data and identify performance bottlenecks and resource contention; When an anomaly is detected, the corresponding processing mechanism and alarm are triggered.
[0052] In this embodiment, environmental indicators (such as CPU, memory, disk, network, etc.) and application performance indicators (such as response time, QPS, error rate, etc.) are collected; the collected raw data is cleaned, normalized and other pre-processing operations are performed to adapt to subsequent analysis algorithms; appropriate unsupervised learning algorithms are selected, such as anomaly detection, clustering, dimensionality reduction and other methods; the anomaly detection model is trained using historical data to learn normal data distribution patterns; environmental and performance data are continuously collected and input into the trained anomaly detection model in real time; the model will evaluate whether the current data is abnormal and identify possible performance bottlenecks and resource competition; once an abnormal situation is detected, the corresponding processing action is immediately triggered, such as: automatically expanding / reducing resources, adjusting application configuration parameters, and notifying relevant operation and maintenance personnel for further diagnosis and processing; at the same time, an alarm message is sent to promptly notify relevant personnel and systems. This method can automatically identify abnormal conditions in the environment and applications without relying on manual monitoring. After an abnormality is triggered, automatic remedial measures can be taken in a timely manner to shorten the failure time. The burden of manual inspections can be reduced, freeing operation and maintenance personnel from daily monitoring. It can quickly respond to sudden performance problems and improve the system's self-healing and adaptive capabilities. The abnormal detection results can provide a basis for subsequent performance analysis and bottleneck diagnosis. In general, the automated detection and processing of environmental and performance anomalies through unsupervised learning algorithms can greatly improve the reliability and availability of the system.
[0053] In some possible implementations of the present invention, the step of regularly evaluating the effect of the computing power management strategy and optimizing the control strategy using reinforcement learning technology includes: Define state space and action space; Define a multi-dimensional reward function that includes computing power utilization, energy efficiency, and task completion time; Use deep reinforcement learning algorithms to learn optimal computing power allocation and management strategies; Verify the effectiveness of the strategy in a simulated environment and gradually apply it to the actual system through A / B testing.
[0054] In this embodiment, the state space includes indicators such as the system's current computing power utilization, energy consumption, and task queues, and the action space includes optional computing power allocation strategies, such as increasing / decreasing CPU / GPU resource allocation; comprehensively considering indicators such as computing power utilization, energy efficiency, and task completion time, and designing appropriate weights to balance the importance of these indicators; selecting a suitable deep reinforcement learning model (such as DQN, DDPG, etc.); using historical data to train the model to learn the optimal computing power management strategy; building a simulated computing environment, including workloads, dynamic changes in resources, etc.; testing the optimized computing power management strategy in a simulated environment to evaluate its performance; gradually applying the optimization strategy to the actual system through A / B testing; and continuously collecting feedback data to further optimize and adjust the strategy. Through this method, computing resources can be reasonably allocated to maximize utilization and avoid resource waste; computing power can be dynamically adjusted according to workload to improve energy efficiency and reduce operating costs; computing resources can be intelligently allocated to ensure that key tasks can be completed in a timely manner; the system can autonomously adjust computing power management strategies according to environmental changes to improve stability; the strategy training process based on the reward function has a certain degree of interpretability, and the algorithm can be guided to learn the desired optimization goal by adjusting the reward function. In short, using reinforcement learning technology to optimize computing power management strategies can significantly improve the overall performance and energy efficiency of the system.
[0055] In some possible implementations of the present invention, a visualization step is also included, specifically: Generate data center heat maps in real time; Display historical trends and forecast results of key performance indicators; Provides an interactive interface for administrators to adjust policies and perform manual intervention.
[0056] In this embodiment, distributed temperature sensors are used to collect temperature data from various areas of the data center in real time, and the collected temperature data is input into the visualization system, and a heat map is generated on the map with the help of color coding; the heat map is updated in real time to intuitively display the current heat distribution of the data center; key performance indicators of the data center, such as CPU / memory utilization, network throughput, power consumption, etc. are collected; the historical data of these indicators are visualized and trend charts are generated; the time series prediction model is used to generate prediction results for these indicators for a period of time in the future; the prediction results are displayed on the dashboard together with the historical trends; a Web-based interactive management console is designed, and the aforementioned heat map and performance indicator dashboard are integrated on the console, allowing administrators to manually adjust the computing power allocation strategy and observe the effect in real time; alarm and anomaly detection functions are provided to facilitate administrators to discover and deal with problems in a timely manner. In the scheme of this embodiment, the heat map intuitively displays the heat distribution of the data center, and the trend chart clearly presents the historical changes of key indicators; the future trend prediction based on the time series prediction model can provide a basis for decision-making; administrators are allowed to manually intervene and adjust the computing power management strategy, improve management flexibility and response speed, and enhance system controllability; the system status is continuously monitored, abnormal situations are discovered and warned in a timely manner, and system reliability and stability are improved. In general, the solution of this embodiment can monitor the heat distribution and key performance indicators of the data center in real time, and provide administrators with a visual interactive control interface, thereby greatly improving the visibility, controllability and adaptability of data center operations.
[0057] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0058] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0059] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the above-mentioned units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.
[0060] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0061] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0062] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the above-mentioned methods of each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or CD-ROM and other media that can store program codes.
[0063] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (English: Read-Only Memory, abbreviated as: ROM), a random access memory (English: Random Access Memory, abbreviated as: RAM), a magnetic disk or an optical disk, etc.
[0064] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of the present application. At the same time, for general technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
[0065] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions without departing from the spirit and scope of the present invention, and can make various changes and modifications, including the combination of the above-mentioned different functions and implementation steps, including software and hardware implementation methods, all of which are within the scope of protection of the present invention.
Claims
1. A computing power data management system based on distributed computing, characterized in that: include: Cloud servers, data centers, and multiple computing nodes; The cloud server is configured as follows: Collect data center environmental data and node status data of each computing node in real time through a distributed sensor network; Collect and integrate computing power data of each computing node; The collected environmental data, node status data and computing power data are transmitted to the central control system for preprocessing and analysis to obtain first data; Based on the first data, a prediction model is constructed using a machine learning algorithm to predict future temperature changes, computing loads, and computing power requirements of the data center; Dynamically adjust the computing resource allocation strategy according to the computing load and the computing power demand; Optimizing refrigeration system parameters according to the temperature change; Continuously monitor the system operation status and computing power utilization efficiency, detect anomalies and trigger alarm mechanisms; Regularly evaluate the effectiveness of computing power management strategies and use reinforcement learning technology to optimize control strategies.
2. A computing power data management method based on distributed computing, characterized in that: include: Collect data center environmental data and node status data of each computing node in real time through a distributed sensor network; Collect and integrate computing power data of each computing node; The collected environmental data, node status data and computing power data are transmitted to the central control system for preprocessing and analysis to obtain first data; Based on the first data, a prediction model is constructed using a machine learning algorithm to predict future temperature changes, computing loads, and computing power requirements of the data center; Dynamically adjust the computing resource allocation strategy according to the computing load and the computing power demand; Optimizing refrigeration system parameters according to the temperature change; Continuously monitor the system operation status and computing power utilization efficiency, detect anomalies and trigger alarm mechanisms; Regularly evaluate the effectiveness of computing power management strategies and use reinforcement learning technology to optimize control strategies.
3. The computing power data management method based on distributed computing according to claim 2 is characterized in that: The step of using a machine learning algorithm to build a prediction model based on the first data to predict future temperature changes, computing load, and computing power requirements of the data center includes: Creating time-related features, generating lagged features, and constructing interactive features after performing time alignment, processing missing values, and feature standardization and normalization on the first data; Select multiple preset basic models and develop a model fusion solution based on the model properties of the multiple basic models; Input the time-related features, lag features, and interaction features into the corresponding basic models for training, and fuse the trained multiple models according to the model fusion scheme to obtain a prediction model; Based on the first data and the prediction model, future temperature changes, computing load and computing power requirements of the data center are predicted.
4. The computing power data management method based on distributed computing according to claim 3 is characterized in that: The step of dynamically adjusting the computing power resource allocation strategy according to the computing load and the computing power requirement includes: Real-time monitoring of first real-time status data of each computing node; Collecting first real-time task queue information of each computing node, wherein the task queue information includes the type, quantity, priority, and estimated execution time of tasks to be executed; Combining historical computing load data and special event data of each computing node, using a time series model to predict first load change data of each computing node within a first preset period; Build a task resource consumption model to estimate the resource requirements of different types of tasks; Combining the first real-time task queue information and the task resource consumption model, obtaining first real-time task resource consumption demand data; Adjust the computing power resource allocation strategy within the first preset period according to the computing load, the computing power requirement, the first real-time status data, the first load change data and the first real-time task resource consumption requirement data.
5. The computing power data management method based on distributed computing according to claim 4 is characterized in that: The adjusting the computing power resource allocation strategy within the first preset period according to the computing load, the computing power requirement, the first real-time status data, the first load change data, and the first real-time task resource consumption requirement data includes: Analyze the change trend of the computing load and computing power demand of each computing node according to the computing load, the computing power demand, the first real-time status data, the first load change data, and the first real-time task resource consumption demand data to obtain an analysis result; Using a time series analysis method, predict the change in computing power demand of each computing node within the first preset period to obtain a prediction result; Perform the following operations based on the analysis and prediction results: Dynamically adjust the number and specifications of virtual machines or containers on each computing node; Migrate tasks across nodes to balance computing load and heat distribution; Select the corresponding computing nodes for different types of computing tasks; Assign high-load tasks to computing nodes with lower temperatures; Migrate tasks to computing nodes that are close to the temperature threshold; The power consumption of each computing node is controlled through dynamic voltage and frequency scaling.
6. The computing power data management method based on distributed computing according to claim 5 is characterized in that: It also includes steps to optimize computing resource utilization, specifically: Analyze historical computing power usage patterns based on historical work data of each computing node, identify inefficient or wasteful resource allocation plans, and adjust them; Implement a computing resource oversale strategy to improve overall resource utilization; Implement energy-saving measures for idle or low-load computing nodes.
7. The computing power data management method based on distributed computing according to claim 6 is characterized in that: It also includes computing power cost management steps, specifically: Calculate the computing power cost of each computing task in real time; Dynamically adjust task priorities and resource allocation strategies based on cost-benefit analysis; Generate computing power usage reports to calculate cost allocation and billing.
8. The computing power data management method based on distributed computing according to claim 7 is characterized in that: The steps of continuously monitoring the system operation status and computing power utilization efficiency, detecting anomalies and triggering an alarm mechanism include: Use unsupervised learning algorithms to detect abnormal patterns in environmental and performance data and identify performance bottlenecks and resource contention; When an anomaly is detected, the corresponding processing mechanism and alarm are triggered.
9. The computing power data management method based on distributed computing according to claim 8 is characterized in that: The steps of regularly evaluating the effect of the computing power management strategy and optimizing the control strategy using reinforcement learning technology include: Define state space and action space; Define a multi-dimensional reward function that includes computing power utilization, energy efficiency, and task completion time; Use deep reinforcement learning algorithms to learn optimal computing power allocation and management strategies; Verify the effectiveness of the strategy in a simulated environment and gradually apply it to the actual system through A / B testing.
10. The computing power data management method based on distributed computing according to claim 9 is characterized in that: It also includes visualization steps, specifically: Generate data center heat maps in real time; Display historical trends and forecast results of key performance indicators; Provides an interactive interface for administrators to adjust policies and perform manual intervention.
Citation Information
Patent Citations
Method for intelligently predicting K8s resource scheduling
CN117891559A
Network resource scheduling optimization system of cloud environment
CN117931424A
Distributed computing power scheduling management system and method
CN118132228A
Thermal station load prediction and optimization control method and system based on distributed machine learning
CN118153779A
Method and system for improving computing power efficiency
CN118550711A
Cited By
Intelligent calculation center energy efficiency optimization system and method
CN119248621A
An intelligent center energy efficiency optimization system and method
CN119248621B
Marine environment multi-source data fusion processing method and system
CN119271519A
Workstation data monitoring method based on Windows system
CN119292871A
Distributed real-time data stream processing system and method
CN119356879A
