Data center operation and maintenance management system based on big data

Abstract

The invention belongs to the technical field of data center management, and particularly relates to a data center operation and maintenance management system based on big data, which comprises a data acquisition module used for acquiring data from data sources of various devices of a data center in real time or at regular intervals; and the data processing module is used for receiving the data transmitted by the data acquisition module. The data center operation and maintenance management system based on big data can intelligently allocate the storage space according to the actual demand of the system and the resource use condition, and avoids resource waste and storage bottleneck. Through the technologies of data compression, duplicate removal and the like, the utilization rate of the storage space is further improved, the storage cost is reduced, meanwhile, a data storage strategy is optimized, a data backup and recovery mechanism is implemented, and it is ensured that system data can be recovered in time when faced with risks such as hardware faults, software errors or manual misoperation. And system instability and service interruption caused by data loss are avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of data center management, and in particular to a data center operation and maintenance management system based on big data. Background Art

[0002] Data centers are global collaborative networks of specific devices used to receive and store data information on the Internet infrastructure. Generally speaking, the most important part of data center operation and maintenance management is to ensure the speed at which the data center receives and reads data, because this speed directly affects the user experience.

[0003] With the development of business, the amount of data generated by data centers has increased dramatically, which has put higher requirements on the system's data processing and storage capabilities. If the system cannot process this data efficiently, it may lead to performance degradation and even affect the normal operation of the system. In addition, the storage of big data requires a lot of hardware resources and maintenance costs, resulting in low management efficiency. In view of this, we propose a data center operation and maintenance management system based on big data. Summary of the invention

[0004] The main purpose of the present invention is to provide a data center operation and maintenance management system based on big data, which can solve the problems raised in the above background technology.

[0005] To achieve the above objectives, the data center operation and maintenance management system based on big data proposed in the present invention includes: A data collection module, which is used to collect data in real time or periodically from data sources of various devices in the data center; A data processing module is used to receive the data transmitted by the data acquisition module, and perform cleaning, denoising, anomaly detection, data normalization and other processing on the collected raw data to improve data quality and reduce noise and errors; A monitoring and management module, which is used to monitor the hardware facilities of the data center, such as servers, network equipment, etc., and perform fault warning and maintenance management; A storage management module, which is used to centrally integrate the data processed by the data processing module and perform distributed storage according to the size and type of the data; A fault management module, which is used to handle faults and events in the data center, including problem location, troubleshooting, event recording and reporting, etc.; The analysis module is responsible for receiving data from the monitoring and management module in real time, including device status, network traffic, performance indicators, etc., and performing real-time analysis to assist operation and maintenance personnel in quickly discovering potential problems or anomalies and taking timely measures; Among them, the storage management module includes a storage optimization unit, a cache unit, a control unit and a data protection unit. The storage optimization unit is used for data compression, deduplication, hierarchical storage and other technical means to reduce storage space occupancy and improve the utilization rate of storage resources. The cache unit is used to store frequently accessed data in the cache, reduce the number of accesses to the main storage system, and improve the data access speed. The control unit is responsible for the allocation, scheduling and monitoring of storage resources to ensure the stable operation of the storage management module. The data protection unit is used to regularly back up important data to ensure that it can be quickly restored when the data is lost or damaged. At the same time, sensitive data is encrypted to prevent data leakage and illegal access.

[0006] Preferably, in the process of storing the data collected by the data collection module, the storage management module first performs data fusion on the collected data, then performs distributed storage of the data according to the size and importance of the data, then further performs deduplication and conversion processing on the distributed stored data, and then performs data storage optimization; In the process of data fusion, information from multiple data sources such as different sensors, different sources, and different processing methods is integrated to eliminate errors and deviations from a single data source. In the process of distributed storage of data, data is dispersed across multiple physical devices or servers, which can process data in parallel, improve reading and writing speeds and transmission speeds, and meet the needs of large-scale data processing.

[0007] Preferably, the storage optimization unit can also reasonably allocate storage resources according to the access frequency and value of the data, reduce storage costs, and optimize data storage layout and access paths.

[0008] Preferably, the cache unit can also dynamically adjust the cache strategy according to the data access pattern and the changes in hot data to ensure the efficient use of cache resources, while supporting multi-level cache architecture (such as CPU cache, memory cache, disk cache, etc.), providing different levels of cache services to meet the needs of different application scenarios.

[0009] Preferably, the control unit can also perform corresponding storage operations according to preset storage strategies, such as data backup, recovery, migration, etc., to facilitate the configuration, monitoring and management of storage resources.

[0010] Preferably, the data protection unit can also ensure the integrity and consistency of the data according to checksum, hash value and the like.

[0011] Preferably, the data acquisition module includes a data loading unit, a data receiving unit and a data transmission unit, the data loading unit is used to load the collected data, the data receiving unit is used to receive the loaded data, and the data transmission unit is used to transmit the data.

[0012] Preferably, the monitoring management module includes a real-time monitoring unit, an alarm strategy unit, an alarm notification unit and a privacy protection unit. The real-time monitoring unit is used to continuously and uninterruptedly monitor the operating status of the managed system, including but not limited to equipment performance, environmental parameters, security status, etc., and through various sensors and acquisition devices, the real-time monitoring unit can accurately collect various data of the system, such as CPU usage, memory occupancy, network traffic, temperature and humidity, etc., to provide a basis for subsequent analysis and processing. The real-time monitoring unit has a built-in intelligent algorithm to automatically analyze the collected data, detect potential anomalies or faults, and transmit the signal to the alarm strategy unit. The privacy protection unit has a strict access control mechanism to restrict illegal users from accessing monitoring data. Only authorized personnel can view and process relevant data. At the same time, for data involving personal privacy, the privacy protection unit supports anonymous processing technology, such as face blurring and data desensitization.

[0013] Preferably, the fault management module includes a fault detection unit, a fault location unit and a fault recovery unit. The fault detection unit is used to detect the operating status of the system, including hardware, software, network and other levels. The fault location unit accurately locates the detected fault based on the positioning technology and algorithm, and determines the specific location and cause of the fault. The fault recovery unit supports automated processing functions and can automatically execute a preset fault recovery process, thereby reducing manual intervention and reducing the risk of human error.

[0014] Preferably, the locating algorithm of the fault locating unit includes a rule-based fault locating algorithm, a statistics-based fault locating algorithm and an artificial intelligence-based fault locating algorithm.

[0015] The present invention provides a data center operation and maintenance management system based on big data. It has the following beneficial effects: (1) The data center operation and maintenance management system based on big data can intelligently allocate storage space according to the actual needs of the system and resource usage, avoid resource waste and storage bottlenecks, and further improve the utilization of storage space and reduce storage costs through data compression, deduplication and other technologies. At the same time, it optimizes data storage strategies and implements data backup and recovery mechanisms to ensure that system data can be restored in a timely manner when facing risks such as hardware failure, software errors or human errors, thereby avoiding system instability and service interruption caused by data loss.

[0016] (2) The data center operation and maintenance management system based on big data can integrate information from multiple data sources, eliminate errors and deviations from a single data source, and in the process of distributed storage of data, disperse the data across multiple physical devices or servers, so that data can be processed in parallel, improving the reading and writing speed and transmission speed, thus meeting the needs of large-scale data processing.

[0017] (3) The data center operation and maintenance management system based on big data can realize real-time monitoring and timely detect system anomalies. Combined with the alarm strategy unit and the alarm notification unit, it ensures that management personnel can quickly receive fault alarms and take measures, thereby effectively reducing the impact of system failures on the business and improving the stability and reliability of the system.

[0018] (4) The data center operation and maintenance management system based on big data can automatically detect and locate faults, reduce the time and cost of manual troubleshooting, and improve operation and maintenance efficiency. At the same time, the privacy protection unit ensures that sensitive information is not leaked during the operation and maintenance process, protects user privacy, and ensures the security of system data during transmission, storage, and processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0020] Figure 1 is a system block diagram of the present invention; Figure 2 This is a block diagram of the storage management module of the present invention; Figure 3 This is a block diagram of the monitoring management module of the present invention; Figure 4 This is a block diagram of the fault management module of the present invention; Figure 5 This is a data storage flow chart of the present invention; Figure 6 This is a data acquisition block diagram of the present invention; Figure 7 It is a schematic diagram of the data acquisition device of the present invention.

[0021] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] See also Figure 1-Figure 7 The present invention proposes a data center operation and maintenance management system based on big data, including a data acquisition module, a data processing module, a monitoring management module, a storage management module, a fault management module and an analysis module. The data acquisition module is used to collect data from the data sources of various devices in the data center in real time or regularly. The data processing module is used to receive the data transmitted by the data acquisition module, and perform cleaning, denoising, anomaly detection, data normalization and other processing on the collected raw data to improve data quality and reduce noise and errors. The monitoring management module is used to monitor the hardware facilities of the data center, such as servers, network equipment, etc., and perform fault warning and maintenance management. The storage management module is used to centrally integrate the data processed by the data processing module and perform distributed storage according to the size and type of the data. The fault management module is used to handle faults and events in the data center, including problem location, troubleshooting, event recording and reporting, etc. The analysis module is responsible for receiving data from the monitoring management module in real time, including device status, network traffic, performance indicators, etc., and performing real-time analysis to assist operation and maintenance personnel to quickly discover potential problems or anomalies and take timely measures.

[0024] In an embodiment of the present invention, in order to optimize the storage structure, specifically, the storage management module includes a storage optimization unit, a cache unit, a control unit and a data protection unit. The storage optimization unit is used for data compression, deduplication, hierarchical storage and other technical means to reduce storage space occupation and improve the utilization rate of storage resources. The cache unit is used to store frequently accessed data in the cache, reduce the number of accesses to the main storage system, and improve the data access speed. The control unit is responsible for the allocation, scheduling and monitoring of storage resources to ensure the stable operation of the storage management module. The data protection unit is used to regularly back up important data to ensure that it can be quickly restored when the data is lost or damaged. At the same time, sensitive data is encrypted to prevent data leakage. Leakage and illegal access, wherein, in the process of storing the data collected by the data acquisition module, the storage management module first performs data fusion on the collected data, and then performs distributed storage of the data according to the size of the data and the importance of the data, and then further performs deduplication and conversion processing on the distributed stored data, and then performs data storage optimization. In the process of data fusion, information from multiple data sources such as different sensors, different sources, and different processing methods is integrated to eliminate errors and deviations from a single data source. In the process of distributed storage of data, the data is dispersed on multiple physical devices or servers, and data can be processed in parallel, thereby improving the reading and writing speed and the transmission speed to meet the needs of large-scale data processing.

[0025] Furthermore, the storage optimization unit can reasonably allocate storage resources according to the access frequency and value of the data, reduce storage costs, and optimize data storage layout and access paths. The cache unit can also dynamically adjust the cache strategy according to the access pattern of the data and the changes in hot data to ensure efficient use of cache resources. At the same time, it supports multi-level cache architecture (such as CPU cache, memory cache, disk cache, etc.) and provides different levels of cache services to meet the needs of different application scenarios. The control unit can also perform corresponding storage operations according to preset storage strategies, such as data backup, recovery, migration, etc., to facilitate the configuration, monitoring and management of storage resources. The data protection unit can also ensure the integrity and consistency of the data based on checksums, hash values, etc.

[0026] Furthermore, the data acquisition module includes a data loading unit, a data receiving unit and a data transmission unit, wherein the data loading unit is used to load the collected data, the data receiving unit is used to receive the loaded data, and the data transmission unit is used to transmit the data.

[0027] In an embodiment of the present invention, the monitoring management module includes a real-time monitoring unit, an alarm strategy unit, an alarm notification unit and a privacy protection unit. The real-time monitoring unit is used to continuously and uninterruptedly monitor the operating status of the managed system, including but not limited to equipment performance, environmental parameters, security status, etc., and through various sensors and acquisition devices, the real-time monitoring unit can accurately collect various data of the system, such as CPU usage, memory occupancy, network traffic, temperature and humidity, etc., to provide a basis for subsequent analysis and processing. The real-time monitoring unit has a built-in intelligent algorithm to automatically analyze the collected data, detect potential anomalies or faults, and transmit the signal to the alarm strategy unit. The privacy protection unit has a strict access control mechanism to restrict illegal users from accessing monitoring data. Only authorized personnel can view and process relevant data. At the same time, for data involving personal privacy, it ensures that managers can quickly receive fault alarms and take measures, thereby effectively reducing the impact of system failures on business and improving the stability and reliability of the system. The privacy protection unit supports anonymous processing technologies, such as face blurring and data desensitization, to ensure the security of system data during transmission, storage and processing. Furthermore, the fault management module includes a fault detection unit, a fault location unit and a fault recovery unit. The fault detection unit is used to detect the operating status of the system, including hardware, software, network and other levels. The fault location unit accurately locates the detected fault based on the location technology and algorithm to determine the specific location and cause of the fault. The fault recovery unit supports the automatic processing function and can automatically execute the preset fault recovery process to reduce manual intervention and reduce the risk of human error. The location algorithm of the fault location unit includes a rule-based fault location algorithm, a statistics-based fault location algorithm and an artificial intelligence-based fault location algorithm. The rule-based fault location algorithm matches the actual fault phenomenon with the entries in the rule library through the preset fault rule library to locate the fault; the statistics-based fault location algorithm finds out the laws and characteristics of the fault through statistical analysis of the historical operation data of the system; the artificial intelligence-based fault location algorithm uses artificial intelligence technologies such as machine learning and deep learning to automatically analyze and learn system data to improve the accuracy and efficiency of fault location.

[0028] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the inventive concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A data center operation and maintenance management system based on big data, characterized in that: include: A data collection module, which is used to collect data in real time or periodically from data sources of various devices in the data center; A data processing module is used to receive the data transmitted by the data acquisition module, and perform cleaning, denoising, anomaly detection, data normalization and other processing on the collected raw data to improve data quality and reduce noise and errors; A monitoring and management module, which is used to monitor the hardware facilities of the data center, such as servers, network equipment, etc., and perform fault warning and maintenance management; A storage management module, which is used to centrally integrate the data processed by the data processing module and perform distributed storage according to the size and type of the data; A fault management module, which is used to handle faults and events in the data center, including problem location, troubleshooting, event recording and reporting, etc.; The analysis module is responsible for receiving data from the monitoring and management module in real time, including device status, network traffic, performance indicators, etc., and performing real-time analysis to assist operation and maintenance personnel in quickly discovering potential problems or anomalies and taking timely measures; Among them, the storage management module includes a storage optimization unit, a cache unit, a control unit and a data protection unit. The storage optimization unit is used for data compression, deduplication, hierarchical storage and other technical means to reduce the storage space occupancy. The cache unit is used to store frequently accessed data in the cache to reduce the number of accesses to the main storage system. The control unit is responsible for the allocation, scheduling and monitoring of storage resources. The data protection unit is used to regularly back up important data to ensure that it can be quickly restored when the data is lost or damaged. At the same time, sensitive data is encrypted to prevent data leakage and illegal access.

2. According to the big data-based data center operation and maintenance management system of claim 1, it is characterized by: In the process of storing the data collected by the data collection module, the storage management module first performs data fusion on the collected data, then performs distributed storage of the data according to the size and importance of the data, then further performs deduplication and conversion processing on the distributed stored data, and then performs data storage optimization; In the process of data fusion, information from multiple data sources such as different sensors, different sources, and different processing methods is integrated to eliminate errors and deviations from a single data source. In the process of distributed storage of data, data is dispersed across multiple physical devices or servers, and data can be processed in parallel.

3. The data center operation and maintenance management system based on big data according to claim 2 is characterized by: The storage optimization unit can also reasonably allocate storage resources and optimize data storage layout and access paths according to the access frequency and value of the data.

4. The data center operation and maintenance management system based on big data according to claim 2 is characterized by: The cache unit can also dynamically adjust the cache strategy according to the data access mode and the changes in hot data, and support a multi-level cache architecture to provide cache services at different levels.

5. The data center operation and maintenance management system based on big data according to claim 2, characterized in that: The control unit can also perform corresponding storage operations according to preset storage strategies, such as data backup, recovery, migration, etc.

6. The data center operation and maintenance management system based on big data according to claim 1, characterized in that: The data protection unit can also ensure the integrity and consistency of the data based on checksums, hash values, etc.

7. The data center operation and maintenance management system based on big data according to claim 1, characterized in that: The data acquisition module includes a data loading unit, a data receiving unit and a data transmission unit. The data loading unit is used to load the collected data, the data receiving unit is used to receive the loaded data, and the data transmission unit is used to transmit the data.

8. The data center operation and maintenance management system based on big data according to claim 1, characterized in that: The monitoring management module includes a real-time monitoring unit, an alarm strategy unit, an alarm notification unit and a privacy protection unit. The real-time monitoring unit is used to continuously and uninterruptedly monitor the operating status of the managed system, including but not limited to equipment performance, environmental parameters, security status, etc. The real-time monitoring unit has a built-in intelligent algorithm to automatically analyze the collected data, detect potential anomalies or faults, and transmit the signal to the alarm strategy unit. The privacy protection unit has a strict access control mechanism to restrict illegal users from accessing monitoring data. Only authorized personnel can view and process relevant data. At the same time, for data involving personal privacy, the privacy protection unit supports anonymous processing technology, such as face blurring and data desensitization.

9. The data center operation and maintenance management system based on big data according to claim 1, characterized in that: The fault management module includes a fault detection unit, a fault location unit and a fault recovery unit. The fault detection unit is used to detect the operating status of the system, including hardware, software, network and other levels. The fault location unit accurately locates the detected fault based on the positioning technology and algorithm, and determines the specific location and cause of the fault. The fault recovery unit supports automated processing functions and can automatically execute the preset fault recovery process, reducing manual intervention and reducing the risk of human error.

10. The data center operation and maintenance management system based on big data according to claim 1, characterized in that: The locating algorithm of the fault locating unit includes a rule-based fault locating algorithm, a statistics-based fault locating algorithm and an artificial intelligence-based fault locating algorithm.

Images