Intelligent operation and maintenance system and method for AI server

Abstract

The invention discloses an intelligent operation and maintenance system and method for an AI server, and relates to the technical field of intelligent operation and maintenance. During operation of the system, multi-dimensional state data generated in the operation process of the AI server is collected in real time and preprocessed in a unified mode, a consistency feature matrix is formed, and the consistency feature matrix is obtained based on real-time data and historical data; the method comprises the steps of calculating an intelligent operation and maintenance index SIOI of a server by combining a machine learning algorithm or a deep learning model, realizing early warning of potential faults of the server by constructing a fault risk assessment formula and setting a threshold value, and generating a resource optimization index by constructing a resource allocation optimization model and comprehensively considering task priority, hardware utilization rate and energy consumption factors. Based on historical data and operation feedback of server operation, a deep reinforcement learning algorithm or an online learning mechanism is adopted, parameters of a fault prediction model and a resource allocation model are automatically optimized, and the adaptation capability of the model is adjusted according to the dynamically changing operation environment and task characteristics.

Description

Technical Field

[0001] The present invention relates to the field of intelligent operation and maintenance technology, and specifically to an intelligent operation and maintenance system and method for an AI server. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, AI servers have become an important infrastructure to support model training and reasoning. AI servers bear the heavy responsibility of high-performance computing, and their operating status directly affects the efficiency and effectiveness of AI tasks. However, traditional server operation and maintenance methods rely on manual monitoring and experience judgment, which makes it difficult to respond to operating pressures and potential failures in complex environments in a timely manner, and cannot meet the AI ​​server's requirements for real-time, stability, and efficient resource utilization. Therefore, intelligent operation and maintenance that combines multi-dimensional data analysis and dynamic adjustment functions has become a key direction for the development of AI servers.

[0003] An intelligent operation and maintenance system and method for AI servers is designed to address the various shortcomings of traditional operation and maintenance methods. First, the existing system's monitoring of the server's operating status is limited to a single parameter and lacks multi-dimensional data fusion capabilities, resulting in inaccurate fault identification. Secondly, resource allocation lacks a dynamic adjustment mechanism and cannot be optimized in real time according to task priority and hardware status, resulting in low resource utilization efficiency. In addition, the lack of effective encryption and privacy protection mechanisms during data transmission and processing can easily lead to data security risks. These deficiencies make the existing operation and maintenance system unable to cope with the complex operating environment of AI servers and unable to achieve efficient and intelligent operation and maintenance goals.

[0004] The above status quo and shortcomings are mainly due to the gap between the high requirements of AI server operation and maintenance for multi-dimensional data analysis, dynamic resource scheduling and security protection and the existing technical means. Traditional operation and maintenance relies on fixed monitoring rules and static resource allocation methods, which are difficult to adapt to the real-time changes in AI tasks' requirements for high computing load and low latency. When these shortcomings occur, a series of abnormal effects may occur, including increased risk of server failure, increased waste of resources, delayed task execution, and data leakage. These problems not only reduce the operating efficiency of the server, but may also cause large-scale service interruptions and economic losses, posing a serious threat to the stability and availability of AI technology. Therefore, there is an urgent need for an intelligent operation and maintenance system to solve the above problems through multi-module collaboration and dynamic optimization strategies. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides an intelligent operation and maintenance system and method for an AI server, which solves the problems mentioned in the background technology.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: an intelligent operation and maintenance system for an AI server, comprising a data acquisition and preprocessing module, a fault prediction module, a resource dynamic optimization module, an intelligent control module, a data security protection module and a self-learning optimization module; The data collection and preprocessing module is used to collect and preprocess the multi-dimensional status data generated during the operation of the AI ​​server in real time to form a consistent feature matrix; The fault prediction module is used to calculate the server intelligent operation and maintenance index SIOI based on real-time data and historical data, combined with machine learning algorithms or deep learning models, and achieve early warning of potential server failures by constructing a fault risk assessment formula and threshold setting; The resource dynamic optimization module is used to dynamically adjust the server's resource allocation strategy according to the server's real-time operating status, task requirements and hardware load conditions, generate resource optimization indicators by building a resource allocation optimization model, and dynamically allocate GPU, memory and other computing resources according to the optimization results; The intelligent control module is used to receive the results output by the fault prediction module and the resource dynamic optimization module, and adjust task scheduling, hardware resource allocation and exception handling measures through instruction execution to ensure the stability and efficiency of system operation; The data security protection module is used to encrypt and protect the privacy of the collected, transmitted and processed data during the operation and maintenance of the AI ​​server, use a hash algorithm to verify the data integrity, achieve the security of data transmission through an encryption algorithm, and protect sensitive information through a data desensitization mechanism; The self-learning optimization module is used to automatically optimize the parameters of the fault prediction model and the resource allocation model based on the historical data and operation feedback of the server operation, using a deep reinforcement learning algorithm or an online learning mechanism, and adjust the adaptability of the model according to the dynamically changing operating environment and task characteristics, so as to continuously improve the prediction and optimization effects.

[0007] Preferably, the data acquisition and preprocessing module includes a multidimensional data acquisition unit, a data cleaning unit and a feature matrix construction unit; The multi-dimensional data acquisition unit is used to acquire a full range of multi-source data through the sensor interface, log system and API, and to collect the status data of the AI ​​server in real time during operation, including hardware performance parameters, energy consumption and temperature parameters and network load parameters, and to acquire hardware performance data groups, energy consumption and temperature data groups and network load data groups; The hardware data group includes: GPU utilization U GPU 、Memory usage M MEM , CPU utilization UCPU, disk read and write rate RDISK and IO operation rate IO RATE ; The energy consumption temperature data group includes the overall power consumption P SYS 、GPU core temperature T GPU 、CPU core temperature T CPU and fan speed F RPM ; The network load data set includes the data packet throughput T PNET 、Network delay L NET , transmission error rate E TRAN and bandwidth utilization B UTIL ; The data cleaning unit is used to detect and clean the collected hardware performance data group, energy consumption temperature data group and network load data group, remove obvious outliers through rule algorithms and statistical models, interpolate or complete missing data, normalize multidimensional data, and eliminate dimensional differences between different parameters; The feature matrix construction unit is used to construct a unified multi-dimensional feature matrix based on the cleaned and standardized data, align the data according to the time series, and form a feature input with a consistent timestamp and data format.

[0008] Preferably, the fault prediction module includes a fault risk assessment unit and an early warning output unit; The fault risk assessment unit is used to quantify the potential faults in the server operation state by constructing a fault risk assessment formula based on the feature matrix data collected in real time, and calculate and obtain: the operation health coefficient OHC, the energy consumption stability coefficient ESC and the network efficiency coefficient NEC. Based on the historical operation data, a machine learning algorithm or a deep learning model is used for training to construct an evaluation model suitable for AI server fault prediction and generate a server intelligent operation and maintenance index SIOI; The early warning output unit is used to automatically generate early warning information, obtain a level assessment plan, and output the fault type, impact range and occurrence time when the server intelligent operation and maintenance index SIOI exceeds a set threshold, providing a basis for early intervention in system operation.

[0009] Preferably, the operating health coefficient OHC is calculated by the following formula: ; Where U GPU 、M MEM , U CPU , R DISK and IO RATE They represent GPU utilization, memory usage, CPU utilization, disk read and write rates, and IO operation rates, respectively. is the maximum value of each parameter, α1, α2, α3, α4 and α5 are the weights of each parameter, reflecting the influence of different hardware parameters on the overall health status; The operating health coefficient OHC is calculated by the following formula: ; Where P SYS , T GPU , T CPU and F RPM Respectively represent the overall power consumption, GPU core temperature, CPU core temperature and fan speed, is the maximum value of each parameter, β1, β2, β3 and β4 are the weights of each parameter, indicating the influence of each energy consumption and temperature parameter on stability; The network efficiency coefficient NEC is calculated by the following formula: ; Where, L NET 、E TRAN , T PNET and B UTIL They represent network delay, transmission error rate, packet throughput and bandwidth utilization respectively. is the maximum value of each parameter, γ1, γ2, γ3 and γ4 are the weights of each parameter, indicating the influence of each network parameter on the network performance; The server intelligent operation index SIOI is calculated by the following formula: ; In the formula, OHC represents the operation health coefficient, ESC represents the energy consumption stability coefficient, NEC represents the network efficiency coefficient, and δ1, δ2, and δ3 represent comprehensive weights, which are set according to the requirements of actual application scenarios for hardware health, energy consumption stability, and network efficiency.

[0010] Preferably, the level evaluation scheme is used to compare the server intelligent operation and maintenance index SIOI with the set qualified threshold A and qualified threshold B from three aspects: resource optimization, hardware maintenance, and network adjustment, and take corresponding improvement measures according to different levels of the server intelligent operation and maintenance index SIOI; SIOI≥A, the first evaluation level is obtained. The overall operation status of the server is good, the hardware load is low, the energy consumption is stable, the network performance is high, and there is no obvious risk in the system. Adjust the task scheduling and postpone some non-high priority tasks to reduce resource usage. Check the performance of the cooling system, including fans and radiators, to prevent the temperature control ability from decreasing due to long-term operation. B ≤ SIOI < A, obtain the second evaluation level. There is pressure on the server running status. The hardware utilization rate is 20% - 50% higher than the qualified index. The power consumption and temperature are within 20% higher than the qualified index. There is a bottleneck in network performance. Move non-critical tasks and low-priority data analysis to off-peak periods to reduce the occupancy of CPU and IO resources by background tasks, and check the fan speed F RPMF , appropriately improve the cooling efficiency to reduce the core temperature T GPU , T CPU , reduce the disk read and write frequency; SIOI < B, obtain the third evaluation level. The server running status is in warning. The hardware load is close to or exceeds the safe operation threshold by more than 50%. The energy consumption and temperature are too high. The network performance has significantly declined. Immediately suspend non-critical tasks, including log processing and batch data backup, unload some low-priority tasks, give priority to ensuring resource allocation for high-priority tasks, and limit the usage rate of high-power devices within 50%.

[0011] Preferably, the resource dynamic optimization module includes an operation status analysis unit, a resource allocation optimization unit, and a dynamic adjustment output unit; The operation status analysis unit is used to analyze the task requirements, hardware load conditions, and resource utilization rates in real time according to the current running status of the AI server, obtain the dynamic running characteristics of the server, and provide a decision-making basis for resource optimization; The resource allocation optimization unit is used to build a resource allocation optimization model, comprehensively consider task priorities, hardware utilization rates, and energy consumption factors, calculate the optimal resource allocation plan through a multi-objective optimization algorithm, and generate resource optimization indicators to ensure that resource allocation meets task requirements and is efficient; The dynamic adjustment output unit is used to generate a dynamic allocation plan according to the resource optimization results, adjust the use and allocation of CPU, GPU, and memory computing resources, ensure the dynamic and real-time nature of server resource allocation, and improve the overall operation efficiency of the system.

[0012] Preferably, the intelligent control module includes a task scheduling control unit, a resource allocation control unit, and an exception handling control unit; The task scheduling control unit is used to generate a task scheduling strategy according to the task type and current running status of the system, adjust the execution order and allocation method of tasks, and ensure that the real-time nature and priority of task execution are met; The resource allocation control unit is used to receive the resource allocation indicators of the resource optimization module, perform real-time control over the allocation of server hardware resources according to the optimization results, and execute the allocation instructions to adjust the use status of computing resources; The exception handling control unit is used to execute corresponding exception handling measures according to the early warning information, including isolating faulty hardware, switching redundant resources and reducing system load, so as to ensure stable operation of the system under abnormal conditions.

[0013] Preferably, the data security protection module includes a data encryption unit and a privacy protection unit; The data encryption unit is used to encrypt the collected, transmitted and stored data, use symmetric encryption or asymmetric encryption algorithm to generate encrypted data streams to prevent external attacks or data leakage, verify the integrity of the collected and transmitted data, and use hash algorithm to generate verification values ​​to prevent data from being tampered with during transmission or storage; The privacy protection unit is used to desensitize data involving user privacy or sensitive information, hide sensitive content through field replacement and fuzzification methods, and ensure that the data meets privacy protection requirements during the operation and maintenance process.

[0014] Preferably, the self-learning optimization module includes a data feedback analysis unit and a model optimization unit; The data feedback analysis unit is used to analyze the operation rules of the system and the prediction effect of the model based on the real-time data and historical operation records of the server operation, generate model performance feedback data, and provide a basis for model optimization; The model optimization unit is used to automatically optimize the parameters of the fault prediction model and the resource allocation model based on feedback data, using a deep reinforcement learning algorithm or an online learning mechanism, improve the model's adaptability to dynamic environments, and generate new prediction and resource allocation strategies based on the optimized model.

[0015] An intelligent operation and maintenance method for an AI server includes the following steps: Step 1: Real-time collection and unified preprocessing of multi-dimensional status data generated during the operation of the AI ​​server to form a consistent feature matrix; Step 2: Based on real-time data and historical data, combined with machine learning algorithms or deep learning models, the server intelligent operation and maintenance index SIOI is calculated, and early warning of potential server failures is achieved by constructing a failure risk assessment formula and threshold setting; Step 3: Dynamically adjust the server's resource allocation strategy based on the server's real-time operating status, task requirements, and hardware load. By building a resource allocation optimization model, comprehensively consider task priority, hardware utilization, and energy consumption factors, generate resource optimization indicators, and dynamically allocate GPU, memory, and other computing resources based on the optimization results. Step 4: Receive the output results of the fault prediction module and the resource dynamic optimization module, and adjust task scheduling, hardware resource allocation and exception handling measures through instruction execution to ensure the stability and efficiency of system operation; Step 5: During the operation and maintenance of the AI ​​server, the collected, transmitted and processed data are encrypted and privacy protected. The data integrity is verified using a hash algorithm. The encryption algorithm is used to achieve data transmission security. At the same time, a data desensitization mechanism is used to protect sensitive information. Step 6: Based on the historical data and operation feedback of the server, use deep reinforcement learning algorithm or online learning mechanism to automatically optimize the parameters of the fault prediction model and resource allocation model, adjust the adaptability of the model according to the dynamically changing operating environment and task characteristics, and continuously improve the prediction and optimization effects.

[0016] The present invention provides an intelligent operation and maintenance system and method for an AI server, which has the following beneficial effects: (1) When the system is running, the multi-dimensional status data generated during the operation of the AI ​​server is collected and pre-processed in real time to form a consistent feature matrix. Based on real-time data and historical data, combined with machine learning algorithms or deep learning models, the server intelligent operation and maintenance index SIOI is calculated. By constructing a fault risk assessment formula and threshold setting, early warning of potential server failures is achieved. By constructing a resource allocation optimization model, task priority, hardware utilization and energy consumption factors are comprehensively considered to generate resource optimization indicators. Based on the historical data and operation feedback of the server operation, a deep reinforcement learning algorithm or online learning mechanism is used to automatically optimize the parameters of the fault prediction model and resource allocation model, and adjust the model's adaptability according to the dynamically changing operating environment and task characteristics.

[0017] (2) Based on the collaborative work of the data acquisition and preprocessing module, fault prediction module, resource dynamic optimization module, intelligent control module, data security protection module and self-learning optimization module, this system realizes key tasks such as real-time monitoring of multi-dimensional status data, fault prediction, resource optimization and security protection during the operation of the AI ​​server. Through the unified collection and preprocessing of multi-dimensional data, a consistent feature matrix is ​​formed, which provides high-quality input data for subsequent analysis and optimization; the fault prediction module realizes the early identification and risk management of potential server failures through the calculation and early warning function of the intelligent operation and maintenance index SIOI; the resource dynamic optimization module dynamically adjusts the server resource allocation strategy through a multi-objective optimization algorithm to ensure task execution efficiency and hardware operation stability. The completion of this series of tasks enables the system to fully perceive, predict and respond to the operating status of the server, thereby achieving the purpose of ensuring system stability and efficiency.

[0018] (3) Compared with existing operation and maintenance technical means, this system has achieved significant improvements in multi-dimensional data fusion, dynamic optimization capabilities and intelligence levels. First, traditional systems are usually limited to single-dimensional data monitoring and lack comprehensive analysis capabilities for hardware, energy consumption, network and other aspects. This system, by constructing a unified feature matrix, comprehensively covers key parameters such as GPU utilization, power consumption, and network latency, and realizes unified analysis of multi-dimensional data. Second, traditional resource scheduling methods are mostly based on fixed rules and lack dynamic adjustment capabilities, resulting in low resource utilization. This system realizes real-time optimization allocation of task priorities and hardware loads through the resource dynamic optimization module, effectively improving resource utilization efficiency. In addition, through the self-learning optimization module, this system can continuously optimize the prediction model and scheduling strategy based on operation feedback, solving the shortcomings of traditional technologies in adapting to dynamic operating environments. These improvements have significantly enhanced the system's adaptability to complex operating scenarios and operation and maintenance efficiency.

[0019] (4) Through the collaborative work and functional improvement of the above modules, this system has achieved significant improvement in many aspects. In terms of operating efficiency, the system has increased resource utilization by more than 20% through dynamic resource allocation and task optimization, while reducing the risk of failure caused by hardware overload. In terms of security, the system prevents data leakage and tampering through data encryption, privacy protection and integrity verification mechanisms, and enhances the security and reliability of the operation and maintenance process. In terms of intelligence, through the continuous model training and updating of the self-learning optimization module, the prediction accuracy and adaptability of the system have been greatly improved. In addition, the early warning capability of fault prediction significantly reduces the risk of server operation and avoids large-scale service interruptions caused by sudden failures. Overall, through the combination of modular design and intelligent functions, this system provides all-round support for the stable operation of AI servers and achieves the goal of efficient, reliable and safe intelligent operation and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a block diagram of an intelligent operation and maintenance system for an AI server of the present invention; Figure 2 The present invention is a schematic diagram of the steps of an intelligent operation and maintenance method for an AI server. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Example 1 The present invention provides an intelligent operation and maintenance system for AI servers, please refer to Figure 1 , including data acquisition and preprocessing module, fault prediction module, resource dynamic optimization module, intelligent control module, data security protection module and self-learning optimization module; The data collection and preprocessing module is used to collect and preprocess the multi-dimensional status data generated during the operation of the AI ​​server in real time to form a consistent feature matrix; The fault prediction module is used to calculate the server intelligent operation and maintenance index SIOI based on real-time data and historical data, combined with machine learning algorithms or deep learning models, and achieve early warning of potential server failures by constructing a fault risk assessment formula and threshold setting; The resource dynamic optimization module is used to dynamically adjust the server's resource allocation strategy according to the server's real-time operating status, task requirements and hardware load conditions, generate resource optimization indicators by building a resource allocation optimization model, and dynamically allocate GPU, memory and other computing resources according to the optimization results; The intelligent control module is used to receive the results output by the fault prediction module and the resource dynamic optimization module, and adjust task scheduling, hardware resource allocation and exception handling measures through instruction execution to ensure the stability and efficiency of system operation; The data security protection module is used to encrypt and protect the privacy of the collected, transmitted and processed data during the operation and maintenance of the AI ​​server, use a hash algorithm to verify the data integrity, achieve the security of data transmission through an encryption algorithm, and protect sensitive information through a data desensitization mechanism; The self-learning optimization module is used to automatically optimize the parameters of the fault prediction model and the resource allocation model based on the historical data and operation feedback of the server operation, using a deep reinforcement learning algorithm or an online learning mechanism, and adjust the adaptability of the model according to the dynamically changing operating environment and task characteristics, so as to continuously improve the prediction and optimization effects.

[0023] In this embodiment, the multi-dimensional status data generated during the operation of the AI ​​server is collected in real time and uniformly preprocessed to form a consistency feature matrix. Based on real-time data and historical data, combined with a machine learning algorithm or a deep learning model, the server intelligent operation and maintenance index SIOI is calculated. By constructing a fault risk assessment formula and threshold setting, early warning of potential server failures is achieved. By constructing a resource allocation optimization model, task priority, hardware utilization and energy consumption factors are comprehensively considered to generate resource optimization indicators. A hash algorithm is used to verify data integrity, and an encryption algorithm is used to achieve data transmission security. At the same time, a data desensitization mechanism is used to protect sensitive information. Based on the historical data and operation feedback of the server operation, a deep reinforcement learning algorithm or an online learning mechanism is used to automatically optimize the parameters of the fault prediction model and the resource allocation model. According to the dynamically changing operating environment and task characteristics, the adaptability of the model is adjusted to continuously improve the prediction and optimization effects.

[0024] Example 2 This embodiment is explained in Example 1, please refer to Figure 1 ,Specifically: the data acquisition and preprocessing module includes a multidimensional data acquisition unit, a data cleaning unit and a feature matrix construction unit; The multi-dimensional data acquisition unit is used to acquire a full range of multi-source data through the sensor interface, log system and API, and to collect the status data of the AI ​​server in real time during operation, including hardware performance parameters, energy consumption and temperature parameters and network load parameters, and to acquire hardware performance data groups, energy consumption and temperature data groups and network load data groups; The hardware data group includes: GPU utilization U GPU 、Memory usage M MEM , CPU utilization UCPU, disk read and write rate R DISK and IO operation rate IO RATE ; The energy consumption temperature data group includes the overall power consumption P SYS 、GPU core temperature T GPU 、CPU core temperature T CPU and fan speed F RPM ; The network load data set includes the data packet throughput T PNET 、Network delay L NET , transmission error rate E TRAN and bandwidth utilization B UTIL ; The data cleaning unit is used to detect and clean the collected hardware performance data group, energy consumption temperature data group and network load data group, remove obvious outliers through rule algorithms and statistical models, interpolate or complete missing data, normalize multidimensional data, and eliminate dimensional differences between different parameters; The feature matrix construction unit is used to construct a unified multi-dimensional feature matrix based on the cleaned and standardized data, align the data according to the time series, and form a feature input with a consistent timestamp and data format.

[0025] The fault prediction module includes a fault risk assessment unit and an early warning output unit; The fault risk assessment unit is used to quantify the potential faults in the server operation state by constructing a fault risk assessment formula based on the feature matrix data collected in real time, and calculate and obtain: the operation health coefficient OHC, the energy consumption stability coefficient ESC and the network efficiency coefficient NEC. Based on the historical operation data, a machine learning algorithm or a deep learning model is used for training to construct an evaluation model suitable for AI server fault prediction and generate a server intelligent operation and maintenance index SIOI; The early warning output unit is used to automatically generate early warning information, obtain a level assessment plan, and output the fault type, impact range and occurrence time when the server intelligent operation and maintenance index SIOI exceeds a set threshold, providing a basis for early intervention in system operation.

[0026] In this embodiment, the AI ​​server intelligent operation and maintenance system of the present invention realizes multi-dimensional real-time monitoring of the server's operating status, data cleaning and feature construction, accurate prediction of potential faults, and early warning functions through the organic combination of the data acquisition and preprocessing module, the fault prediction module and its various functional units. Through the real-time acquisition of the multi-dimensional data acquisition unit, covering hardware performance parameters, energy consumption and temperature parameters, and network load parameters, the system can fully perceive the operating status of the server; through the data cleaning unit, the elimination of abnormal values, the interpolation and completion of missing data, and the normalization processing, the accuracy and consistency of the data are guaranteed; the multi-dimensional feature matrix generated by the feature matrix construction unit provides high-quality input for subsequent analysis. Combined with the fault prediction module, the health status of the server is quantified using the operation health coefficient OHC, the energy consumption stability coefficient ESC, and the network efficiency coefficient NEC, and the quantitative analysis and prediction of potential faults are realized through the server intelligent operation and maintenance index SIOI. When SIOI exceeds the set threshold, the system can automatically generate early warning information and level assessment plans, providing operation and maintenance personnel with decision-making basis such as fault type, impact scope and occurrence time, significantly improving the efficiency and accuracy of fault handling, effectively avoiding system downtime or abnormal operation caused by sudden faults, and ensuring the stability and efficiency of the server.

[0027] Example 3 This embodiment is explained in Example 1, please refer to Figure 1 Specifically, the operating health coefficient OHC is calculated by the following formula: ; Where U GPU 、M MEM , U CPU , R DISK and IO RATE They represent GPU utilization, memory usage, CPU utilization, disk read and write rates, and IO operation rates, respectively. is the maximum value of each parameter, α1, α2, α3, α4 and α5 are the weights of each parameter, reflecting the influence of different hardware parameters on the overall health status; The operating health coefficient OHC is calculated by the following formula: ; Where P SYS , T GPU , T CPU and F RPM Respectively represent the overall power consumption, GPU core temperature, CPU core temperature and fan speed, is the maximum value of each parameter, β1, β2, β3 and β4 are the weights of each parameter, indicating the influence of each energy consumption and temperature parameter on stability; The network efficiency coefficient NEC is calculated by the following formula: ; Where, L NET 、E TRAN , T PNET and B UTIL They represent network delay, transmission error rate, packet throughput and bandwidth utilization respectively. is the maximum value of each parameter, γ1, γ2, γ3 and γ4 are the weights of each parameter, indicating the influence of each network parameter on the network performance; The server intelligent operation index SIOI is calculated by the following formula: ; In the formula, OHC represents the operation health coefficient, ESC represents the energy consumption stability coefficient, NEC represents the network efficiency coefficient, and δ1, δ2, and δ3 represent comprehensive weights, which are set according to the requirements of actual application scenarios for hardware health, energy consumption stability, and network efficiency.

[0028] The described level evaluation scheme is used to compare the Server Intelligent Operation and Maintenance Index (SIOI) with the set qualified thresholds A and B from three aspects: resource optimization, hardware maintenance, and network adjustment, and take corresponding improvement measures according to different levels of the SIOI; When SIOI≥A, obtain the first evaluation level. The overall operation status of the server is good, the hardware load is low, the energy consumption is stable, the network efficiency is high, and there are no obvious risks in the system. Adjust the task scheduling, delay some non-high-priority tasks to reduce resource occupancy, and check the performance of the cooling system, including fans and radiators, to prevent the temperature control ability from decreasing due to long-term operation; When B≤SIOI<A, obtain the second evaluation level. There is pressure on the server operation status. The hardware utilization rate is 20%-50% higher than the qualified index, the power consumption and temperature are within 20% higher than the qualified index, and there are bottlenecks in network performance. Move non-critical tasks and low-priority data analysis to the off-peak period to reduce the occupancy of CPU and IO resources by background tasks, and check the fan speed F RPMF , appropriately improve the cooling efficiency to reduce the core temperature T GPU , T CPU , and reduce the disk read and write frequency; When SIOI<B, obtain the third evaluation level. The server operation status is in warning. The hardware load is close to or exceeds the safe operation threshold by more than 50%, the energy consumption and temperature are too high, and the network performance has significantly declined. Immediately suspend non-critical tasks, including log processing and batch data backup, uninstall some low-priority tasks, give priority to ensuring resource allocation for high-priority tasks, and limit the usage rate of high-power-consuming devices within 50%.

[0029] In this embodiment, the present invention realizes the comprehensive quantification and level evaluation of the server operation status by constructing the quantification formulas of the Operation Health Coefficient (OHC), Energy Consumption Stability Coefficient (ESC), and Network Efficiency Coefficient (NEC), and combining the comprehensive evaluation of the Server Intelligent Operation and Maintenance Index (SIOI). According to different levels of SIOI, the system takes targeted improvement measures from three aspects: resource optimization, hardware maintenance, and network adjustment: when the operation status is good, optimize the task scheduling to reduce resource occupancy, and extend the hardware service life through the maintenance of the cooling system; when there is pressure on the operation status, relieve the system load by dynamically adjusting the task priority and optimizing the cooling efficiency; when the operation status is in warning, urgently suspend non-critical tasks and limit the usage rate of high-power-consuming devices, effectively reducing the failure risk. This solution significantly improves the system's perception ability and response efficiency to the operation status, can realize the full-link operation and maintenance optimization from slight pressure relief to severe risk prevention and control, ensure the efficient operation and stability of the server, and at the same time reduce the hardware loss and energy consumption cost.

[0030] Example 4 This embodiment is explained in Example 1, please refer to Figure 1 ,Specifically: the resource dynamic optimization module includes an operation status analysis unit, a resource allocation optimization unit and a dynamic adjustment output unit; The operation status analysis unit is used to analyze task requirements, hardware load conditions, and resource utilization in real time according to the current operation status of the AI ​​server, obtain the dynamic operation characteristics of the server, and provide a decision-making basis for resource optimization; The resource allocation optimization unit is used to construct a resource allocation optimization model, comprehensively consider task priority, hardware utilization and energy consumption factors, calculate the optimal resource allocation plan through a multi-objective optimization algorithm, and generate resource optimization indicators to ensure that resource allocation meets task requirements and is efficient; The dynamic adjustment output unit is used to generate a dynamic allocation plan according to the resource optimization results, adjust the usage allocation of CPU, GPU, and memory computing resources, ensure the dynamic and real-time nature of server resource allocation, and improve the overall operating efficiency of the system.

[0031] The intelligent control module includes a task scheduling control unit, a resource allocation control unit and an exception handling control unit; The task scheduling control unit is used to generate a task scheduling strategy according to the task type and current running status of the system, adjust the execution order and allocation method of the tasks, and ensure the real-time execution and priority of the tasks; The resource allocation control unit is used to receive the resource allocation index of the resource optimization module, perform real-time control on the allocation of server hardware resources according to the optimization result, and execute the allocation instruction to adjust the usage status of the computing resources; The exception handling control unit is used to execute corresponding exception handling measures according to the early warning information, including isolating faulty hardware, switching redundant resources and reducing system load, so as to ensure stable operation of the system under abnormal conditions.

[0032] The data security protection module includes a data encryption unit and a privacy protection unit; The data encryption unit is used to encrypt the collected, transmitted and stored data, use symmetric encryption or asymmetric encryption algorithm to generate encrypted data streams to prevent external attacks or data leakage, verify the integrity of the collected and transmitted data, and use hash algorithm to generate verification values ​​to prevent data from being tampered with during transmission or storage; The privacy protection unit is used to desensitize data involving user privacy or sensitive information, hide sensitive content through field replacement and fuzzification methods, and ensure that the data meets privacy protection requirements during the operation and maintenance process.

[0033] The self-learning optimization module includes a data feedback analysis unit and a model optimization unit; The data feedback analysis unit is used to analyze the operation rules of the system and the prediction effect of the model based on the real-time data and historical operation records of the server operation, generate model performance feedback data, and provide a basis for model optimization; The model optimization unit is used to automatically optimize the parameters of the fault prediction model and the resource allocation model based on feedback data, using a deep reinforcement learning algorithm or an online learning mechanism, improve the model's adaptability to dynamic environments, and generate new prediction and resource allocation strategies based on the optimized model.

[0034] In this embodiment, the present invention realizes the dynamic adjustment of AI server resource allocation, real-time optimization of operation control, comprehensive protection of data security, and continuous self-optimization of the model through the collaborative work of the resource dynamic optimization module, the intelligent control module, the data security protection module, and the self-learning optimization module. The resource dynamic optimization module perceives the task requirements and hardware load conditions in real time through the operation status analysis unit, and generates a dynamic allocation plan in combination with the multi-objective optimization model constructed by the resource allocation optimization unit, which significantly improves the efficiency and adaptability of resource allocation. The intelligent control module ensures the real-time and priority of task execution through task scheduling and exception handling functions, and responds quickly in abnormal situations, and ensures the stable operation of the system through hardware isolation and load adjustment. The data security protection module prevents data leakage and tampering through data encryption and privacy protection functions, and ensures the integrity and compliance of data transmission and storage. The self-learning optimization module optimizes the parameters of the prediction model and allocation strategy based on historical operation data and real-time feedback, so that the system has a high degree of adaptability to dynamic environments. The collaboration of these modules significantly improves the operating efficiency, security and intelligence level of the AI ​​server, and provides a strong technical guarantee for the stable operation and efficient operation and maintenance of the server in a complex environment.

[0035] Example 5 An intelligent operation and maintenance method for AI servers, please refer to Figure 2 , specifically: including the following steps: Step 1: Real-time collection and unified preprocessing of multi-dimensional status data generated during the operation of the AI ​​server to form a consistent feature matrix; Step 2: Based on real-time data and historical data, combined with machine learning algorithms or deep learning models, the server intelligent operation and maintenance index SIOI is calculated, and early warning of potential server failures is achieved by constructing a failure risk assessment formula and threshold setting; Step 3: Dynamically adjust the server's resource allocation strategy based on the server's real-time operating status, task requirements, and hardware load. By building a resource allocation optimization model, comprehensively consider task priority, hardware utilization, and energy consumption factors, generate resource optimization indicators, and dynamically allocate GPU, memory, and other computing resources based on the optimization results. Step 4: Receive the output results of the fault prediction module and the resource dynamic optimization module, and adjust task scheduling, hardware resource allocation and exception handling measures through instruction execution to ensure the stability and efficiency of system operation; Step 5: During the operation and maintenance of the AI ​​server, the collected, transmitted and processed data are encrypted and privacy protected. The data integrity is verified using a hash algorithm. The encryption algorithm is used to achieve data transmission security. At the same time, a data desensitization mechanism is used to protect sensitive information. Step 6: Based on the historical data and operation feedback of the server, use deep reinforcement learning algorithm or online learning mechanism to automatically optimize the parameters of the fault prediction model and resource allocation model, adjust the adaptability of the model according to the dynamically changing operating environment and task characteristics, and continuously improve the prediction and optimization effects.

[0036] In this embodiment, the present invention realizes real-time perception, dynamic adjustment and continuous optimization of the running status of the AI ​​server through a six-step intelligent operation and maintenance process. Step 1 generates a consistency feature matrix through real-time collection and preprocessing of multi-dimensional status data, providing a high-quality data basis for subsequent analysis; Step 2 accurately calculates the server intelligent operation and maintenance index SIOI by constructing a fault risk assessment formula, combining real-time data and historical data, and realizes early warning of potential faults; Step 3 dynamically adjusts the resource allocation strategy, balances task priority, hardware utilization and energy consumption factors through the construction of a resource optimization model, and improves the efficiency of server resource utilization; Step 4 ensures the accuracy and real-time performance of task scheduling and exception handling through intelligent control, and ensures the stable operation of the system; Step 5 prevents data leakage and tampering through data encryption and privacy protection mechanisms, and enhances the security of the operation and maintenance process; Step 6 realizes continuous optimization of the prediction model and allocation model through deep reinforcement learning or online learning mechanisms, and improves the system's adaptability to dynamic environments. The above-mentioned processes work together to significantly improve the operating efficiency, stability and security of the AI ​​server, and provide comprehensive support for efficient intelligent operation and maintenance in complex environments.

[0037] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent operation and maintenance system for an AI server, characterized in that: It includes data acquisition and preprocessing module, fault prediction module, resource dynamic optimization module, intelligent control module, data security protection module and self-learning optimization module; The data collection and preprocessing module is used to collect and preprocess the multi-dimensional status data generated during the operation of the AI ​​server in real time to form a consistent feature matrix; The fault prediction module is used to calculate the server intelligent operation and maintenance index SIOI based on real-time data and historical data, combined with machine learning algorithms or deep learning models, and achieve early warning of potential server failures by constructing a fault risk assessment formula and threshold setting; The resource dynamic optimization module is used to dynamically adjust the server's resource allocation strategy according to the server's real-time operating status, task requirements and hardware load conditions, generate resource optimization indicators by building a resource allocation optimization model, and dynamically allocate GPU, memory and other computing resources according to the optimization results; The intelligent control module is used to receive the results output by the fault prediction module and the resource dynamic optimization module, and adjust task scheduling, hardware resource allocation and exception handling measures through instruction execution to ensure the stability and efficiency of system operation; The data security protection module is used to encrypt and protect the privacy of the collected, transmitted and processed data during the operation and maintenance of the AI ​​server, use a hash algorithm to verify the data integrity, achieve the security of data transmission through an encryption algorithm, and protect sensitive information through a data desensitization mechanism; The self-learning optimization module is used to automatically optimize the parameters of the fault prediction model and the resource allocation model based on the historical data and operation feedback of the server operation, using a deep reinforcement learning algorithm or an online learning mechanism, and adjust the adaptability of the model according to the dynamically changing operating environment and task characteristics, so as to continuously improve the prediction and optimization effects.

2. According to claim 1, an intelligent operation and maintenance system for an AI server is characterized in that: The data acquisition and preprocessing module includes a multidimensional data acquisition unit, a data cleaning unit and a feature matrix construction unit; The multi-dimensional data acquisition unit is used to acquire a full range of multi-source data through the sensor interface, log system and API, and to collect the status data of the AI ​​server in real time during operation, including hardware performance parameters, energy consumption and temperature parameters and network load parameters, and to acquire hardware performance data groups, energy consumption and temperature data groups and network load data groups; The hardware data group includes: GPU utilization U GPU 、Memory usage M MEM , CPU utilization UCPU, disk read and write rate R DISK and IO operation rate IO RATE ; The energy consumption temperature data group includes the overall power consumption P SYS 、GPU core temperature T GPU 、CPU core temperature T CPU and fan speed F RPM ; The network load data set includes the data packet throughput T PNET 、Network delay L NET , transmission error rate E TRAN and bandwidth utilization B UTIL ; The data cleaning unit is used to detect and clean the collected hardware performance data group, energy consumption temperature data group and network load data group, remove obvious outliers through rule algorithms and statistical models, interpolate or complete missing data, normalize multidimensional data, and eliminate dimensional differences between different parameters; The feature matrix construction unit is used to construct a unified multi-dimensional feature matrix based on the cleaned and standardized data, align the data according to the time series, and form a feature input with a consistent timestamp and data format.

3. The intelligent operation and maintenance system for an AI server according to claim 2, characterized in that: The fault prediction module includes a fault risk assessment unit and an early warning output unit; The fault risk assessment unit is used to quantify potential faults in the server operating state by constructing a fault risk assessment formula based on the feature matrix data collected in real time, calculate and obtain: the operating health coefficient OHC, the energy consumption stability coefficient ESC, and the network efficiency coefficient NEC. Based on historical operating data, it is trained using machine learning algorithms or deep learning models to construct an evaluation model suitable for AI server fault prediction, and generate the server intelligent operation and maintenance index SIOI; The early warning output unit is used to automatically generate early warning information when the server intelligent operation and maintenance index SIOI exceeds the set threshold, obtain the level evaluation plan, and output the fault type, impact range, and occurrence time, providing a basis for early intervention in system operation.

4. The intelligent operation and maintenance system for an AI server according to claim 3, characterized in that: The operating health coefficient OHC is calculated and obtained through the following formula: ； Where U GPU 、M MEM , U CPU , R DISK and IO RATE They represent GPU utilization, memory usage, CPU utilization, disk read and write rates, and IO operation rates, respectively. is the maximum value of each parameter, α1, α2, α3, α4 and α5 are the weights of each parameter, reflecting the influence of different hardware parameters on the overall health status; The operating health coefficient OHC is calculated and obtained through the following formula: ； Where P SYS , T GPU , T CPU and F RPM Respectively represent the overall power consumption, GPU core temperature, CPU core temperature and fan speed, is the maximum value of each parameter, β1, β2, β3 and β4 are the weights of each parameter, indicating the influence of each energy consumption and temperature parameter on stability; The network efficiency coefficient NEC is calculated and obtained through the following formula: ； Where, L NET 、E TRAN , T PNET and B UTIL They represent network delay, transmission error rate, packet throughput and bandwidth utilization respectively. is the maximum value of each parameter, γ1, γ2, γ3 and γ4 are the weights of each parameter, indicating the influence of each network parameter on the network performance; The server intelligent operation and maintenance index SIOI is calculated and obtained through the following formula: ； In the formula, OHC represents the operating health coefficient, ESC represents the energy consumption stability coefficient, NEC represents the network efficiency coefficient, and δ1, δ2, and δ3 represent the comprehensive weights, which are set according to the requirements for hardware health, energy consumption stability, and network efficiency in the actual application scenario.

5. The intelligent operation and maintenance system for an AI server according to claim 3, characterized in that: The level evaluation plan is used to compare the server intelligent operation and maintenance index SIOI with the set qualified thresholds A and B from three aspects: resource optimization, hardware maintenance, and network adjustment, and take corresponding improvement measures according to different levels of the server intelligent operation and maintenance index SIOI; When SIOI≥A, the first evaluation level is obtained. The overall operating state of the server is good, the hardware load is low, the energy consumption is stable, the network efficiency is high, and there are no obvious risks in the system. Adjust the task scheduling, delay some non-high-priority tasks to reduce resource occupancy, and check the performance of the cooling system, including fans and radiators, to prevent the temperature control ability from decreasing due to long-term operation; B ≤ SIOI < A, obtain the second evaluation level. There is pressure on the server running status. The hardware utilization rate is 20% - 50% higher than the qualified index, the power consumption and temperature are within 20% higher than the qualified index, and there is a bottleneck in network performance. Move non-critical tasks and low-priority data analysis to off-peak periods to reduce the occupancy of CPU and IO resources by background tasks, and check the fan speed F RPMF , appropriately improve the cooling efficiency to reduce the core temperature T GPU , T CPU , reduce the disk read and write frequency; When SIOI<B, the third evaluation level is obtained. The server operating state is in a warning, the hardware load is close to or exceeds the safe operating threshold by more than 50%, the energy consumption and temperature are too high, and the network performance has significantly decreased. Immediately suspend non-critical tasks, including log processing and batch data backup, unload some low-priority tasks, give priority to ensuring resource allocation for high-priority tasks, and limit the usage rate of high-power devices within 50%.

6. The intelligent operation and maintenance system for an AI server according to claim 1, characterized in that: The resource dynamic optimization module includes an operating state analysis unit, a resource allocation optimization unit, and a dynamic adjustment output unit; The operating state analysis unit is used to analyze the task requirements, hardware load conditions, and resource usage rates in real time according to the current operating state of the AI server, obtain the dynamic operating characteristics of the server, and provide a decision-making basis for resource optimization; The resource allocation optimization unit is used to construct a resource allocation optimization model, comprehensively consider task priorities, hardware utilization, and energy consumption factors, calculate the optimal resource allocation plan through a multi-objective optimization algorithm, and generate resource optimization indicators to ensure that resource allocation meets task requirements and is efficient; The dynamic adjustment output unit is used to generate a dynamic allocation plan according to the resource optimization results, adjust the usage allocation of CPU, GPU, and memory computing resources, ensure the dynamic and real-time nature of server resource allocation, and improve the overall operating efficiency of the system.

7. The intelligent operation and maintenance system for an AI server according to claim 1, characterized in that: The intelligent control module includes a task scheduling control unit, a resource allocation control unit and an exception handling control unit; The task scheduling control unit is used to generate a task scheduling strategy according to the task type and current running status of the system, adjust the execution order and allocation method of the tasks, and ensure the real-time execution and priority of the tasks; The resource allocation control unit is used to receive the resource allocation index of the resource optimization module, perform real-time control on the allocation of server hardware resources according to the optimization result, and execute the allocation instruction to adjust the usage status of the computing resources; The exception handling control unit is used to execute corresponding exception handling measures according to the early warning information, including isolating faulty hardware, switching redundant resources and reducing system load, so as to ensure stable operation of the system under abnormal conditions.

8. The intelligent operation and maintenance system for an AI server according to claim 1, characterized in that: The data security protection module includes a data encryption unit and a privacy protection unit; The data encryption unit is used to encrypt the collected, transmitted and stored data, use symmetric encryption or asymmetric encryption algorithm to generate encrypted data streams to prevent external attacks or data leakage, verify the integrity of the collected and transmitted data, and use hash algorithm to generate verification values ​​to prevent data from being tampered with during transmission or storage; The privacy protection unit is used to desensitize data involving user privacy or sensitive information, hide sensitive content through field replacement and fuzzification methods, and ensure that the data meets privacy protection requirements during the operation and maintenance process.

9. The intelligent operation and maintenance system for an AI server according to claim 1, characterized in that: The self-learning optimization module includes a data feedback analysis unit and a model optimization unit; The data feedback analysis unit is used to analyze the operation rules of the system and the prediction effect of the model based on the real-time data and historical operation records of the server operation, generate model performance feedback data, and provide a basis for model optimization; The model optimization unit is used to automatically optimize the parameters of the fault prediction model and the resource allocation model based on feedback data, using a deep reinforcement learning algorithm or an online learning mechanism, improve the model's adaptability to dynamic environments, and generate new prediction and resource allocation strategies based on the optimized model.

10. An intelligent operation and maintenance method for an AI server, applied to an intelligent operation and maintenance system for an AI server according to any one of claims 1 to 9, characterized in that: The following steps are involved: Step 1: Real-time collection and unified preprocessing of multi-dimensional status data generated during the operation of the AI ​​server to form a consistent feature matrix; Step 2: Based on real-time data and historical data, combined with machine learning algorithms or deep learning models, the server intelligent operation and maintenance index SIOI is calculated, and early warning of potential server failures is achieved by constructing a failure risk assessment formula and setting thresholds; Step 3: Dynamically adjust the server's resource allocation strategy based on the server's real-time operating status, task requirements, and hardware load. By building a resource allocation optimization model, comprehensively consider task priority, hardware utilization, and energy consumption factors, generate resource optimization indicators, and dynamically allocate GPU, memory, and other computing resources based on the optimization results. Step 4: Receive the output results of the fault prediction module and the resource dynamic optimization module, and adjust task scheduling, hardware resource allocation and exception handling measures through instruction execution to ensure the stability and efficiency of system operation; Step 5: During the operation and maintenance of the AI ​​server, the collected, transmitted and processed data are encrypted and privacy protected. The data integrity is verified using a hash algorithm. The encryption algorithm is used to achieve data transmission security. At the same time, a data desensitization mechanism is used to protect sensitive information. Step 6: Based on the historical data and operation feedback of the server, use deep reinforcement learning algorithm or online learning mechanism to automatically optimize the parameters of the fault prediction model and resource allocation model, adjust the adaptability of the model according to the dynamically changing operating environment and task characteristics, and continuously improve the prediction and optimization effects.

Images