Big data calculation analysis mining and operation storage system based on intelligent machine room

Abstract

The invention discloses a big data calculation analysis mining and operation storage system based on a smart machine room, relates to the technical field of big data analysis, and solves the technical problems of equipment state evaluation simplification, equipment grading and monitoring strategy extensibility and insufficient anomaly analysis depth. According to the method, invalid data is removed, the data format is unified, the storage and transmission cost is reduced, a high-quality data basis is provided for subsequent analysis, a dynamic reference is constructed based on historical normal data of equipment, the influence of task types on performance is distinguished in combination with comparison of a real-time instruction period and a historical period, and the limitation of single threshold judgment in the prior art is avoided; the accuracy of equipment state evaluation is improved, importance classification is performed on the equipment based on a multi-dimensional quantitative scoring system of business influence degree, fault cost and operation role, and a differentiated monitoring period is adopted, so that the problem of resource waste or insufficient core equipment monitoring caused by extensive monitoring strategies in the prior art is solved, and the accuracy of equipment state evaluation is improved. And the monitoring precision and the system overhead are balanced.

Description

Technical Field

[0001] This invention relates to the field of big data analytics technology, specifically to a big data computing, analysis, mining, operation, and storage system based on a smart data center. Background Technology

[0002] With the rapid development of the digital economy, smart data centers, as core infrastructure for data storage, processing, and transmission, are experiencing continuous growth in scale and complexity. This involves a surge in the number of hardware facilities such as servers, network equipment, and cooling systems, and they must support the continuous operation of critical businesses such as financial transactions, medical services, and industrial control. To ensure the stable and efficient operation of data centers, traditional operation and maintenance models have gradually upgraded from manual inspections to intelligent management based on data collection and analysis.

[0003] In existing technologies, data center monitoring systems typically possess basic data acquisition capabilities, collecting equipment operating status, environmental parameters, and energy consumption data, and providing anomaly warnings through simple threshold comparisons. Some systems incorporate historical data comparisons, but often use a single indicator as a benchmark without considering differences in equipment type and business scenarios. Regarding anomaly analysis, they often focus on isolated anomalies of single devices, lacking in-depth analysis of data fluctuation amplitude, anomaly frequency, and device correlations, leading to high false alarm rates and difficulties in root cause identification. Furthermore, the classification of equipment importance is rather crude, employing a uniform monitoring cycle, resulting in insufficient monitoring of core equipment or waste of resources on non-core equipment. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a big data computing, analysis, mining, operation, and storage system based on a smart data center, which solves the problems of simplistic equipment status assessment, crude equipment classification and monitoring strategies, and insufficient depth of anomaly analysis.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a big data computing, analysis, mining, operation, and storage system based on a smart data center, comprising: The working status comparison and analysis module is used to receive preprocessed data transmitted by the data acquisition module in the computer room, acquire all processor devices in the computer room and the historical working data of each device in normal status, calculate the historical instruction cycle, compare the current real-time instruction cycle in the preprocessed data with the historical instruction cycle, if the current real-time instruction cycle increases, it is marked as a device to be analyzed and an abnormal status signal is generated, if it decreases, it is marked as a normal device and a normal status signal is generated, the normal status devices are classified by importance level to obtain core devices and non-core devices, core devices are monitored with a cycle t1 and non-core devices are monitored with a cycle t2, periodic monitoring information is generated and transmitted; The abnormal status analysis module is used to receive abnormal status signals, acquire the device to be analyzed and abnormal data, analyze from the perspectives of data fluctuation amplitude, frequency of abnormal occurrence, and correlation with related device data, calculate the absolute fluctuation value and relative fluctuation ratio of abnormal data and historical benchmark values, compare with the quantile benchmark interval to generate fluctuation analysis results, count the number and frequency of abnormal triggers within time t0, generate frequency analysis results by combining time series diagrams and business logs, compare the timestamps of abnormal devices and related devices, calculate the correlation to generate correlation analysis results; The comprehensive identification and analysis module is used to receive fluctuation analysis results, frequency analysis results, and correlation analysis results. Based on the fluctuation analysis results, it marks first-level anomaly candidates; based on the frequency analysis results, it marks second- or third-level anomaly candidates; and based on the correlation analysis results, it marks first-, second-, or third-level anomaly candidates. It then determines whether the anomaly is serious, general, or minor based on the combined markings of the candidates at each level, and transmits the anomaly identification results.

[0006] As a further embodiment of the present invention, it also includes a data acquisition module for a computer room, which is used to collect equipment operating status data, environmental parameter data, and power consumption data in the computer room, and to preprocess the collected data, including data cleaning, data standardization, and data compression, to obtain preprocessed data and transmit it. The analysis information output module is used to receive periodic monitoring information and anomaly identification results and display them to management personnel.

[0007] As a further aspect of the present invention, the importance level classification in the working status comparison and analysis module includes: The scores are based on three aspects: business impact, failure cost, and operational role. In terms of business impact, core business is assigned 5 points, supporting business is assigned 3 points, and auxiliary business is assigned 1 point. In the fault cost calculation, a recovery time of ≤10 minutes is assigned 5 points, 10-60 minutes is assigned 3 points, and >60 minutes is assigned 1 point; in the running role, a load percentage of ≥30% is assigned 5 points, 10-30% is assigned 3 points, and <10% is assigned 1 point. The overall score is calculated as follows: Business Impact × Weight 1 + Failure Cost × Weight 2 + Operating Role × Weight 3. Weights 1, 2, and 3 are set by the operator. Equipment with an overall score greater than 3 is considered core equipment; otherwise, it is considered non-core equipment.

[0008] As a further aspect of the present invention, the abnormal state analysis module includes data fluctuation amplitude analysis, which includes: Constructing quantile benchmark intervals for 95% confidence intervals based on historical data [CPI] avg -1.96σ, CPI avg +1.96σ], where CPI avgThe historical average instruction cycle is σ, and the historical data standard deviation is σ. If the relative fluctuation ratio exceeds this range, it is considered a significant fluctuation.

[0009] As a further aspect of the present invention, the anomaly occurrence frequency analysis in the anomaly state analysis module includes: By using time sequence diagrams to identify anomalies as periodic high-frequency anomalies, random occasional anomalies, or continuous irregular anomalies, and by comparing the anomaly timestamps with task scheduling records and user access peaks in the business logs, it can be determined whether the anomaly is business-driven.

[0010] As a further aspect of the present invention, the correlation analysis of relevant equipment data in the abnormal state analysis module includes: The associated devices include physically associated devices, logically associated devices, and environmental factors. The correlation between abnormal device indicators and associated device and environmental indicators is calculated using the Pearson correlation coefficient, and abnormal association rules are mined using the Apriori algorithm.

[0011] As a further aspect of the present invention, the anomaly identification processing in the comprehensive identification and analysis module includes: When the fluctuation analysis results show significant fluctuations, it is marked as a first-level anomaly candidate; when the frequency analysis results show periodic high-frequency anomalies, it is marked as a second-level anomaly candidate; when random anomalies occur during critical periods of core business, they are marked as second-level anomaly candidates; and when they occur during off-peak periods of non-core business, they are marked as third-level anomaly candidates. If a persistent, irregular anomaly is driven by business needs, it is marked as a Level 2 anomaly candidate; otherwise, it is marked as a Level 3 anomaly candidate. If the correlation analysis results show that the abnormal device is strongly correlated with environmental indicators and there are no significant changes in environmental parameters, it is marked as a Level 2 anomaly candidate; if there are significant changes in environmental parameters, it is marked as a Level 3 anomaly candidate. If the abnormal device is strongly correlated with upstream devices and the upstream devices are abnormal, it is marked as a Level 1 anomaly candidate; if the upstream devices are not abnormal, it is marked as a Level 2 anomaly candidate. If a single device is abnormal and there are no related impacts, it is marked as a Level 3 anomaly candidate.

[0012] As a further aspect of the present invention, the anomaly determination rule in the comprehensive identification and analysis module is as follows: The presence of a Level 1 anomaly candidate indicates a severe anomaly; the absence of a Level 1 anomaly candidate but the presence of a Level 2 anomaly candidate indicates a general anomaly; and the presence of only a Level 3 anomaly candidate indicates a minor anomaly.

[0013] As a further aspect of the present invention, in the working status comparison and analysis module, the value range of t1 is 5-15 minutes, and the value range of t2 is 30-60 minutes; if the fluctuation range of the indicators of normal equipment exceeds the historical average ±10% for three consecutive cycles, the monitoring cycle is temporarily shortened.

[0014] This invention provides a big data computing, analysis, mining, operation, and storage system based on a smart data center. Compared with existing technologies, it has the following advantages: This invention provides a high-quality data foundation for subsequent analysis by systematically preprocessing data to remove invalid data, unify data formats, and reduce storage and transmission costs. It constructs a dynamic benchmark based on historical normal equipment data and distinguishes the impact of task types on performance by comparing real-time command cycles with historical cycles. This avoids the limitations of single threshold judgment in existing technologies and improves the accuracy of equipment status assessment. Based on a multi-dimensional quantitative scoring system of business impact, failure cost, and operational role, it classifies equipment by importance and adopts differentiated monitoring cycles. This solves the problem of resource waste or insufficient monitoring of core equipment caused by extensive monitoring strategies in existing technologies and balances monitoring accuracy with system overhead.

[0015] This invention analyzes anomalies from three dimensions: data fluctuation amplitude, anomaly frequency, and device correlation. It combines Pearson correlation coefficient and time series analysis to uncover the essence of anomalies. Compared with isolated anomaly judgment in existing technologies, it significantly improves the efficiency of root cause location and reduces the false alarm rate. Through a comprehensive identification module, anomalies are classified into multiple levels and dynamically sorted according to their impact range and urgency. This helps maintenance personnel prioritize high-priority issues and solves the problem of disordered anomaly response in existing technologies, thereby improving maintenance efficiency. Attached Figure Description

[0016] Figure 1 This is a system block diagram of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Please see Figure 1 This application provides a big data computing, analysis, mining, operation, and storage system based on a smart data center, including: a data center data acquisition module, a working status comparison and analysis module, an abnormal status analysis module, a comprehensive identification and analysis module, and an analysis information output module, and combined with... Figure 1 It can be seen that the information between the above functional modules is transmitted in one direction only.

[0019] The data acquisition module for the computer room is primarily responsible for collecting various types of data within the computer room, including equipment operating status data, environmental parameter data, and power consumption data. It also performs preprocessing on the collected data, including data cleaning to remove invalid and erroneous data to ensure data quality; data standardization to convert data in different formats and units into a unified standard for easier subsequent analysis; and data compression to reduce data storage space and improve transmission efficiency, resulting in preprocessed data. This preprocessed data is then transmitted to the operational status comparison and analysis module.

[0020] The working status comparison and analysis module is used to judge the overall status of the computer room based on the acquired preprocessed data. It acquires all devices in the computer room and labels them as i, where i = 1, 2, ..., j, and j represents the type of computer room device, specifically the corresponding processor. Next, it acquires the historical working data corresponding to the computer room device, which represents all normal working data within time T. It calculates the instruction cycle based on the historical working data, where the instruction cycle represents the time required to execute an instruction, specifically the calculation time for different processing tasks. The obtained instruction cycle is compared with the preprocessed data, specifically comparing the current real-time instruction cycle in the preprocessed data with the historical instruction cycle. If the preprocessed data shows an increase in the instruction cycle value compared to the historical instruction cycle, it indicates an increase in task processing time. The corresponding device is then marked as a device to be analyzed, and an abnormal status signal is generated and transmitted to the abnormal status analysis module. Conversely, if the preprocessed data shows a decrease in the instruction cycle value compared to the historical instruction cycle, it indicates a decrease in task processing time. The corresponding device is then marked as a normal device, and a normal status signal is generated. Based on the generated normal status signals, the equipment in the computer room is classified according to importance level, resulting in core equipment and non-core equipment. The specific classification method is as follows: Scoring is conducted based on three aspects: business impact, failure cost, and operational role. For business impact, the business relationship level is analyzed: core businesses (such as bank transaction systems and hospital HIS systems) receive 5 points; supporting businesses (such as data backup and middleware) receive 5 points; and auxiliary businesses (such as office terminals) receive 5 points. For failure cost, the recovery time objective (RTO) requirement is analyzed: ≤10 minutes receives 5 points; 10-60 minutes receives 3 points; and >60 minutes receives 1 point. For operational role, the load share is analyzed: ≥30% of the total data center load receives 5 points; 10-30% receives 3 points; and <10% receives 1 point. Substitute the assigned business impact, failure cost, and operating role into the comprehensive score calculation formula. Comprehensive score = business impact × weight 1 + failure cost × weight 2 + operating role × weight 3. The specific values ​​of weight 1, 2, and 3 are set by the operator. If the comprehensive score is greater than 3 points, it is classified as core equipment; otherwise, it is classified as non-core equipment. For core equipment, monitoring is performed on a time period of t1, while for non-core equipment, monitoring is performed on a time period of t2. The values ​​of t1 and t2 are set by the operator, and t1 is less than t2. Periodic monitoring information is generated and transmitted to the analysis information output module.

[0021] The abnormal state analysis module analyzes the acquired abnormal state signals, identifies the corresponding devices to be analyzed, and obtains corresponding abnormal data through comparison with preprocessed data. It then conducts a multi-dimensional analysis of the abnormal data, delving into aspects such as data fluctuation amplitude, frequency of anomalies, and correlation with data from related devices. The specific analysis is as follows: To assess data fluctuation, historical baseline values ​​for the analyzed equipment are obtained, and the difference between outlier data and these historical baseline values ​​is calculated to obtain the absolute fluctuation value. The relative fluctuation ratio is then calculated using the formula (absolute fluctuation value / historical baseline value) × 100%. Based on historical data, a quantile baseline interval is constructed. [CPI] avg -1.96σ, CPI avg +1.96σ], where CPIavg is the historical average instruction cycle and σ is the historical data standard deviation. The two are compared. If the relative fluctuation ratio exceeds the quantile benchmark interval, it indicates that the data has significant fluctuations. Otherwise, it indicates that the data does not have significant fluctuations, and the fluctuation analysis results are generated. To determine the frequency of anomalies, the number of anomaly triggers within time t0 is obtained, with the specific value of time t0 set by the operator. The corresponding frequency of occurrence is then calculated. Simultaneously, a time series diagram is used to analyze whether the anomalies exhibit periodicity, time-periodity, or randomness, generating time series analysis information. This information specifically includes periodic high-frequency anomalies, random occasional anomalies, and continuous irregular anomalies. Next, the anomaly timestamps are compared with business logs, such as task scheduling records and user access peaks, to determine whether there are business-driven anomalies. Business correlation verification is performed, and frequency analysis results are generated. To assess the correlation of relevant equipment data, the abnormal timestamps of abnormal equipment and related equipment are compared to determine whether there is a temporal correlation. At the same time, the Pearson correlation coefficient is used to quantify the correlation between abnormal equipment indicators and related equipment and environmental indicators, and the correlation analysis results are generated, including strong correlation between abnormal equipment and environmental indicators, strong correlation between abnormal equipment and upstream equipment, and single equipment abnormality with no related impact. Simultaneously, the generated fluctuation analysis results, frequency analysis results, and correlation analysis results are transmitted to the comprehensive identification and analysis module.

[0022] The comprehensive identification and analysis module is used to identify anomalies by combining the obtained fluctuation analysis results, frequency analysis results, and correlation analysis results. First, the fluctuation analysis results are evaluated. If the fluctuation analysis results show that the data has significant fluctuations, the abnormal data is marked as a first-level anomaly candidate. If there are no significant fluctuations, the evaluation of the frequency analysis results is then carried out. For frequency analysis results, if the generated periodic high-frequency anomalies are identified, the corresponding abnormal data will be marked as a secondary anomaly candidate. If the anomalies are random and occasional, further judgment will be made based on the business scenario in which they occur. If they occur during critical periods of core business, they will be marked as a secondary anomaly candidate. If they occur during off-peak periods of non-core business, they will be temporarily marked as a tertiary anomaly candidate. If the anomalies are continuous and irregular, they will be verified through business correlation. If there are business-driven anomalies, they will be marked as a secondary anomaly candidate; otherwise, they will be marked as a tertiary anomaly candidate. For correlation analysis results, if the abnormal equipment shows a strong correlation with environmental indicators, then based on the historical changes in environmental parameters, if there are no significant recent changes in environmental parameters, the abnormal data is marked as a secondary anomaly candidate; if there are significant changes in environmental parameters, it is marked as a tertiary anomaly candidate. If the abnormal equipment is strongly correlated with upstream equipment, and the upstream equipment also shows anomalies, then the abnormal data is marked as a primary anomaly candidate; if the upstream equipment does not show anomalies, it is marked as a secondary anomaly candidate. If it is a single equipment anomaly with no associated impact, it is marked as a tertiary anomaly candidate.

[0023] Based on the above-mentioned anomaly candidate markers at all levels, when a Level 1 anomaly candidate exists, it is directly determined as a serious anomaly; when there is no Level 1 anomaly candidate but there is a Level 2 anomaly candidate, it is determined as a general anomaly; when there is only a Level 3 anomaly candidate, it is determined as a minor anomaly, and the final anomaly identification result is transmitted to the analysis information output module.

[0024] The analysis information output module is used to display the acquired anomaly identification results to the relevant management personnel.

[0025] The data in the above formulas are all calculated using numerical values, without substituting the units of the parameters. In addition, the contents not described in detail in this specification are all prior art known to those skilled in the art.

[0026] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A big data computing analysis mining and operation storage system based on a smart machine room, characterized in that, Comprise: The working state comparison analysis module is used for receiving the pretreatment data transmitted by the machine room data acquisition module, obtaining all processor devices in the machine room and historical working data of each device in normal state, and calculating historical instruction period. The current real-time instruction period in the pretreatment data is compared with the historical instruction period. If the current real-time instruction period increases, the device is marked as to be analyzed and an abnormal state signal is generated. If the current real-time instruction period decreases, the device is marked as normal and a normal state signal is generated. The normal state devices are classified by importance level to obtain core devices and non-core devices. The core devices are monitored at a period t1, and the non-core devices are monitored at a period t2. Periodic monitoring information is generated and transmitted. The abnormal state analysis module is used for receiving the abnormal state signal, obtaining the to-be-analyzed device and abnormal data, analyzing from the data fluctuation amplitude, abnormal occurrence frequency and correlation with related device data, calculating the absolute fluctuation value and the relative fluctuation proportion of the abnormal data and the historical reference value, comparing with the quantile reference interval to generate a fluctuation analysis result, counting the number of times and frequency of abnormal triggering within a time t0, combining a time sequence diagram and a business log to generate a frequency analysis result, and comparing the time stamps of the abnormal device and the related device to calculate the correlation to generate a correlation analysis result. The comprehensive identification analysis module is used for receiving the fluctuation analysis result, the frequency analysis result and the correlation analysis result, marking a first-level abnormal candidate according to the fluctuation analysis result, marking a second-level or third-level abnormal candidate according to the frequency analysis result, and marking a first-level, second-level or third-level abnormal candidate according to the correlation analysis result. The first-level, second-level and third-level candidates are determined as a serious abnormality, a general abnormality or a slight abnormality, and an abnormality identification result is transmitted. 2.The big data computing analysis mining and operation storage system based on the intelligent machine room of claim 1, wherein, The machine room data acquisition module is also included, which is used for collecting device running state data, environmental parameter data and power consumption data in the machine room, preprocessing the collected data, preprocessing including data cleaning, data standardization and data compression, obtaining pretreatment data and transmitting; The analysis information output module is used for receiving the periodic monitoring information and the abnormality identification result and displaying them to the management personnel. 3.The big data computing analysis mining and operation storage system based on intelligent machine room of claim 1, wherein, In the working state comparison analysis module, the importance level classification includes: Scoring from three aspects of business impact degree, fault cost and running role. In the business impact degree, core business is assigned 5 points, supporting business is assigned 3 points, and auxiliary business is assigned 1 point. In the fault cost, recovery time ≤10 minutes is assigned 5 points, 10-60 minutes is assigned 3 points, and >60 minutes is assigned 1 point. In the running role, load ratio ≥30% is assigned 5 points, 10-30% is assigned 3 points, and <10% is assigned 1 point. 4.The big data computing analysis mining and operation storage system based on intelligent machine room of claim 1, wherein, The comprehensive score = business impact degree × weight one + fault cost × weight two + running role × weight three, and the weight one, two and three are set by the operator. If the comprehensive score is >3 points, it is a core device, otherwise it is a non-core device. A quantile-based benchmark interval [CPI avg -1.96σ, CPI avg +1.96σ] is constructed based on historical data, where CPI avg is the historical average instruction cycle, and σ is the standard deviation of historical data. If the relative fluctuation ratio exceeds this interval, it is determined as a significant fluctuation. 5.The big data computing analysis mining and operation storage system based on intelligent machine room of claim 1, wherein, In the abnormal state analysis module, the data fluctuation amplitude analysis includes: In the abnormal state analysis module, the abnormal occurrence frequency analysis includes: Through the time sequence diagram, the abnormality is identified as a periodic high-frequency abnormality, a random occasional abnormality or a continuous irregular abnormality. The abnormal time stamp is compared with the task scheduling record and user access peak in the business log to determine whether it is a business-driven abnormality. 6.The big data computing analysis mining and operation storage system based on intelligent machine room of claim 1, wherein, In the abnormal state analysis module, the correlation analysis of the related device data includes: The related devices include physical related devices, logical related devices and environmental related factors. The correlation between the abnormal device indicators and the related devices and environmental indicators is calculated by using the Pearson correlation coefficient. The abnormal correlation rules are mined by using the Apriori algorithm. 7.The big data computing analysis mining and operation storage system based on intelligent machine room of claim 1, wherein, In the comprehensive identification analysis module, the abnormal identification process includes: When the fluctuation analysis result shows a significant fluctuation, it is marked as a first-level abnormal candidate. When the frequency analysis result is a periodic high-frequency abnormality, it is marked as a second-level abnormal candidate. When a random occasional abnormality occurs in a core business critical period, it is marked as a second-level abnormal candidate. When it occurs in a non-core business low peak period, it is marked as a third-level abnormal candidate. When a sustained irregular abnormality exists, it is marked as a second-level abnormal candidate if it is driven by business, otherwise it is marked as a third-level abnormal candidate. When the correlation analysis result shows that the abnormal device and the environmental indicators are strongly related and the environmental parameters have no significant changes, it is marked as a second-level abnormal candidate. When the environmental parameters have significant changes, it is marked as a third-level abnormal candidate. When the abnormal device is strongly related to the upstream device and the upstream device is abnormal, it is marked as a first-level abnormal candidate. When the upstream device is not abnormal, it is marked as a second-level abnormal candidate. When a single device is abnormal and has no related impact, it is marked as a third-level abnormal candidate. 8.The big data computing analysis mining and operation storage system based on the intelligent machine room of claim 1, wherein, In the comprehensive identification analysis module, the abnormal determination rules are: When there is a first-level abnormal candidate, it is determined as a serious abnormality. When there is no first-level abnormal candidate but there is a second-level abnormal candidate, it is determined as a general abnormality. When there is only a third-level abnormal candidate, it is determined as a slight abnormality. 9.The big data computing analysis mining and operation storage system based on the intelligent machine room of claim 1, wherein, In the working state comparison analysis module, the value range of t1 is 5-15 minutes and the value range of t2 is 30-60 minutes. If the fluctuation range of the normal device indicators exceeds the historical mean value ±10% for three consecutive periods, the monitoring period is temporarily shortened.

Images