Inspection system for data center

Abstract

The invention provides an inspection system for a data center, and the system comprises an inspection robot which is used for carrying out the inspection of the equipment and environment of the data center, and finding a temperature abnormal region or an electric power energy consumption abnormal region; related data are sent to the robot management module; the robot agent module is used for acquiring temperature data of internal hardware of the server and a load of the server, and sending the temperature data and the load to the robot management module; according to an instruction issued by the robot management module, the load exceeding a first preset threshold value is adjusted, and the internal hardware exceeding a preset temperature is processed; and the robot management module is used for monitoring and managing the inspection robot, receiving the inspection data, analyzing the inspection data exceeding a second preset threshold value, sending an analysis result to operation and maintenance personnel, and adjusting the load of the server according to an instruction issued by the operation and maintenance personnel. Problems can be rapidly and accurately positioned and solved, and the operation and maintenance level of a data center is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of operation and maintenance management, and in particular to an inspection system for a data center. Background Art

[0002] Data Center (DC) is a set of complex facilities that store, manage, process and distribute data, including computer systems and supporting communication storage systems, redundant data communication connections, environmental control equipment, monitoring equipment and various safety devices. It is usually owned and operated by large enterprises, government agencies or cloud service providers. With the continuous maturity and advancement of cloud computing, big data and artificial intelligence technologies, data centers have also developed rapidly. Today's data centers are no longer just a simple place for unified hosting and maintenance of servers. It has evolved into a collection of high-performance computers that integrates big data computing and storage. Data centers involve hundreds of thousands or millions of servers or computing units such as GPUs and TPUs for AI processing. The number of resources is large, the heterogeneity is strong, and the management and maintenance are complex.

[0003] A survey shows that the annual power consumption of data centers exceeds 1.5% of the total electricity consumption of the whole society, and up to 10% of the cabinets in the data center operate at temperatures higher than the allowable range of equipment reliability. Abnormal cabinet temperature in the computer room will have a negative impact on the performance and reliability of the equipment. High temperature environment may cause equipment overheating, reduce performance, shorten life, and even damage the equipment. Therefore, timely monitoring and handling of abnormal cabinet temperature in the computer room is an important task to ensure the normal operation of equipment and maintain the computer room environment. Energy consumption analysis and temperature monitoring are essential to prevent server shutdown due to overheating and improve the energy efficiency of data centers.

[0004] Inspection robots are usually equipped with infrared thermal imagers. During inspection, they perform infrared scanning on the entire cabinet of the data center, count and display the temperature data in the cabinet. When abnormal temperature is found, the abnormal temperature event is notified to the operation and maintenance personnel as a work order. The operation and maintenance personnel find and locate the corresponding cabinet U unit and server based on the thermal imaging image in the work order, and check all related servers one by one. After determining the server, it is necessary to conduct further analysis in the server, check the CPU, hard disk and other hardware devices, and repair and replace them until they are normal.

[0005] The abnormal increase in temperature and power consumption in the data center are two interrelated issues. The power meter can count the energy consumption of the electrical equipment passing through it and display it on the attached screen. The inspection robot reads the power consumption value through its visible light camera, finds abnormal power consumption and notifies the operation and maintenance personnel. Similarly, the operation and maintenance personnel are also required to conduct subsequent analysis to find the software that uses a large amount of computing resources for intensive calculations for a long time to determine whether it is an abnormal situation.

[0006] It can be seen that this inspection system needs to be combined with manual analysis and judgment, which takes up a lot of energy and time of operation and maintenance personnel, and is highly dependent on the knowledge level and structure of operation and maintenance personnel, which restricts the automation level of data center operation and maintenance. Summary of the invention

[0007] In view of the shortcomings of the prior art, the present invention provides an inspection system for a data center, comprising:

[0008] The inspection robot is used to inspect the data center equipment and environment, find the abnormal temperature area or abnormal power energy consumption area of ​​the data center equipment; obtain the temperature and humidity data of the data center environment, and send the data of the abnormal temperature area or abnormal power energy consumption area and the temperature and humidity data to the robot management module;

[0009] The robot agent module is used to obtain the temperature data of the internal hardware of the server in the data center and the load of the server in real time, and send the temperature data and the load to the robot management module; the robot agent module adjusts the load exceeding the first preset threshold according to the instruction issued by the robot management module, processes the internal hardware of the server exceeding the preset temperature, and sends the temperature data and the load of the server to the robot management module;

[0010] The robot management module is used to monitor and manage the inspection robot, receive the data of the abnormal temperature area or the abnormal power consumption area, the temperature and humidity data, the internal hardware temperature data of the server, and the load of the server sent by the inspection robot and the robot agent module, analyze the inspection data that exceeds the second preset threshold according to the inspection data received from the inspection robot and the robot agent module, send the analysis results to the operation and maintenance personnel, and adjust the server load according to the instructions issued by the operation and maintenance personnel.

[0011] Furthermore, the inspection robot includes the following components: a vehicle chassis motion platform, a multi-stage lifting platform, an industrial computer, an on-board wireless communication device, a laser radar, an RFID reader, a positioning device, and is also equipped with a visible light camera, an infrared thermal imager, a temperature and humidity sensor, an environmental gas sensor, and a voice intercom device.

[0012] Furthermore, the inspection robot is used to inspect the data center equipment and environment, and find the abnormal temperature area or abnormal power consumption area of ​​the data center equipment, including:

[0013] The inspection robot uses an infrared thermal imager to perform infrared scanning on the surface of the data center cabinet to find areas with abnormal temperature, or uses a visible light camera to read the readings of the power meter installed on the data center cabinet to find areas with abnormal power consumption.

[0014] Furthermore, it also includes:

[0015] The inspection robot uses a visible light camera to scan the barcode or QR code preset on the cabinet to obtain the unique identification of the cabinet and determine the location of the cabinet; or

[0016] Use an RFID reader to read the RFID tag pre-installed on the cabinet, obtain the identification information of the cabinet, and determine the location of the cabinet;

[0017] The data of the abnormal temperature area or the abnormal power consumption area and the corresponding cabinet location are sent to the robot management module.

[0018] Furthermore, the temperature and humidity data of the data center environment are obtained, including:

[0019] The inspection robot obtains the temperature and humidity data of the environment at various locations in the data center based on temperature and humidity sensors and positioning devices.

[0020] Furthermore, the robot agent module is installed in the server of the data center to obtain the temperature data of the internal hardware of the server, the usage data of the computing and storage resources of the server, and the process and resource occupancy data of the server in real time.

[0021] Furthermore, the robot agent module sends the internal hardware data of the server exceeding the preset temperature to the robot management module.

[0022] Furthermore, the robot agent module adjusts the load exceeding the first preset threshold according to the instruction issued by the robot management module, including:

[0023] For server loads that exceed a first preset threshold, the server load is adjusted by calling an API interface provided by the underlying virtualization of the server to migrate virtual machines and containers, kill abnormal processes, and so on.

[0024] Furthermore, the robot management module is used to monitor and manage the inspection robot, including:

[0025] Monitor the running status of the inspection robot and set the inspection path of the inspection robot.

[0026] Further, according to the inspection data received from the inspection robot and the robot agent module, the inspection data exceeding the second preset threshold is sent to the operation and maintenance personnel, including:

[0027] The robot management module analyzes and judges the inspection data exceeding the second preset threshold, and obtains the current operation and maintenance suggestions based on the judgment rules;

[0028] The inspection data and operation and maintenance suggestions are sent to the operation and maintenance personnel.

[0029] The present invention provides an inspection system for a data center, which can quickly and accurately locate and solve problems, save operation and maintenance time, and automatically adjust server loads to improve the operation and maintenance level of the data center. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a structural schematic diagram of a patrol inspection system for a data center provided by an embodiment of the present invention;

[0031] Figure 2 The present invention is a schematic diagram of a workflow of an inspection system for a data center according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present invention, so the present invention is not limited to the specific implementation disclosed below.

[0033] like Figure 1 As shown, the present invention provides an inspection system for a data center, comprising: an inspection robot, a robot agent module, and a robot management module;

[0034] The inspection robot is used to inspect the data center equipment and environment, find the abnormal temperature area or abnormal power energy consumption area of ​​the data center equipment; obtain the temperature and humidity data of the data center environment, and send the data of the abnormal temperature area or abnormal power energy consumption area and the temperature and humidity data to the robot management module;

[0035] The robot agent module is used to obtain the temperature data of the internal hardware of the server in the data center and the load of the server in real time, and send the temperature data and the load to the robot management module; the robot agent module adjusts the load exceeding the first preset threshold according to the instruction issued by the robot management module, processes the internal hardware of the server exceeding the preset temperature, and sends the temperature data and the load of the server to the robot management module;

[0036] The robot management module is used to monitor and manage the inspection robot, receive the data of the abnormal temperature area or the abnormal power consumption area, the temperature and humidity data, the internal hardware temperature data of the server, and the load of the server sent by the inspection robot and the robot agent module, analyze the inspection data that exceeds the second preset threshold according to the inspection data received from the inspection robot and the robot agent module, send the analysis results to the operation and maintenance personnel, and adjust the server load according to the instructions issued by the operation and maintenance personnel.

[0037] Among them, the inspection robot, the main components include: chassis motion platform, multi-stage lifting platform, industrial computer, vehicle-mounted wireless communication device, laser radar, RFID reader, positioning device, and also equipped with visible light camera, infrared thermal imager, temperature and humidity sensor, environmental gas sensor, voice intercom device and other sensors to achieve comprehensive inspection, intelligent identification and perception of data center equipment and environment.

[0038] Furthermore, the inspection robot uses an infrared thermal imager to perform infrared scanning on the surface of the data center cabinet to find areas with abnormal temperature, or uses a visible light camera to read the readings of the power meter installed on the data center cabinet to find areas with abnormal power consumption.

[0039] The inspection robot uses a visible light camera to scan the barcode or QR code preset on the cabinet to obtain the unique identification of the cabinet and determine the location of the cabinet; or

[0040] Use an RFID reader to read the RFID tag pre-installed on the cabinet, obtain the identification information of the cabinet, and determine the location of the cabinet;

[0041] The data of the abnormal temperature area or the abnormal power consumption area and the corresponding cabinet location are sent to the robot management module.

[0042] Furthermore, the inspection robot obtains the temperature and humidity data of the environment at various locations in the data center based on the temperature and humidity sensors and the positioning device.

[0043] The robot agent module is a collection of software installed in the server of the data center, which obtains the temperature data of the internal hardware of the server, the usage data of the computing and storage resources of the server, and the process and resource usage data of the server in real time.

[0044] The robot agent module sends the internal hardware data of the server that exceeds the preset temperature to the robot management module. For the server load that exceeds the first preset threshold, the server load is adjusted by calling the API interface provided by the underlying virtualization of the server, migrating virtual machines and containers, killing abnormal processes. More specifically, the robot agent module runs in the server as a virtual machine, Docker image or process or a combination of these forms by using hardware virtualization and container virtualization technology. The robot agent module implements the function of obtaining the temperature of the internal hardware devices of the server in real time, including devices such as CPU, memory, graphics card, TPU, fan, etc. The robot agent module also implements the function of obtaining the computing and storage resource usage of the server, and obtains the occupancy of the server's CPU, memory, hard disk and other computing resources by the running entities such as processes and virtual machines in the server in real time. The robot agent module sends the above inspection results to the robot management module through the network. The robot agent module has the function of adjusting the server load, and migrates the specified virtual machines and containers by calling the API interface provided by the underlying virtualization of the server (such as open source KVM), thereby adjusting the server load.

[0045] Furthermore, the robot management module monitors the running status of the inspection robot and sets the inspection path of the inspection robot. The inspection data exceeding the second preset threshold is analyzed and judged, and the current operation and maintenance suggestions are obtained according to the analysis and judgment rules; the inspection data and operation and maintenance suggestions are sent to the operation and maintenance personnel. The operation and maintenance personnel are notified to adjust the server load and automatically execute.

[0046] Combination Figure 2 ,The workflow of this system is described as follows:

[0047] 1. The inspection robot uses an infrared thermal imager to perform an infrared scan on the surface of the data center cabinet to find areas with abnormal temperature. Alternatively, the inspection robot uses a visible light camera to read the readings of the power meter installed on the cabinet to find abnormal increases in power consumption.

[0048] 2. The inspection robot obtains identification information of the cabinet in the abnormal temperature area or the abnormally increased power consumption. This information uniquely identifies the cabinet. The identification information of the cabinet is obtained by using a combination of one or more of the following methods.

[0049] (1) The inspection robot uses a visible light camera to scan the barcode or QR code preset on the cabinet and uses the information contained in the barcode or QR code as the cabinet identification information.

[0050] (2) The inspection robot uses an RFID reader to read the RFID tag pre-installed on the cabinet and uses the read data as identification information.

[0051] (3) The inspection robot uses the current indoor geographic location information as the cabinet identification information.

[0052] 3. The inspection robot sends the identification information of the cabinet to the robot management platform.

[0053] 4. The robot management platform uses the identification information of the cabinet as the key value to find the corresponding server information in the cabinet in the built-in database. The server information contains relevant information of the pre-installed robot agent module in the server, such as the IP address, MAC address, and operating status of the robot agent module.

[0054] 5. The robot management platform sends control commands to the robot agent module, which executes the commands and checks the operating status of the server on which it is located.

[0055] (1) The robot agent module checks the current temperature of the main hardware components of the server, including the CPU, memory, graphics card, TPU, etc., and finds and locates components with over-high temperatures. Whether the temperature is too high is determined by comparing it with the set threshold. If it exceeds the threshold, it is determined to be too high. This system has been tested many times and found that when the CPU threshold is set to 65 degrees Celsius and the threshold of other modules is set to 50 degrees Celsius, the probability of misjudgment is low.

[0056] (2) The robot agent module checks the server resource usage of entities running in the server, and finds and locates processes, virtual machines, Dock images and other running entities that occupy too much computing resources or storage resources of the server. The judgment method is to compare the resource usage rate with the set threshold. If it exceeds the threshold, it is judged that the resource usage is too high. After running this method for a long time in the data center of the reporting unit, it was found that when the computing resource usage threshold is set to 90% and the storage resource usage threshold is set to 62%, the overall effect is good and the judgment error is small.

[0057] (3) The robot agent module sends the above-mentioned hardware device information with excessively high temperature and operating entity information with excessively high resource usage to the robot management platform through the network.

[0058] 6. The robot management platform continues to judge and handle according to the rules:

[0059] (1) If the number of hardware components with over-high temperatures inside the server is greater than three, the operation and maintenance personnel will be notified to shut down the server. After confirmation by the operation and maintenance personnel, the system will automatically power off the server and cool it down to avoid more serious consequences.

[0060] (2) If the number of running entities that occupy too much server computing resources exceeds 5, the operation and maintenance personnel will be notified for further processing, including but not limited to: 1: stop accepting new computing tasks; 2: migrate the virtual machines or containers that occupy too many resources to other servers; 3: kill the processes that occupy too many resources, or shut down the virtual machines and containers that occupy too many resources. After the operation and maintenance personnel confirm, the system completes these instructions to adjust the server load, reduce the server's workload, and thereby reduce the server's operating temperature and power consumption.

[0061] (3) After the above treatment measures are completed, the inspection robot re-inspects the areas with abnormally increased temperature and energy consumption found in step 1, and reports the re-inspection results to the operation and maintenance personnel.

[0062] (4) The robot agent module shuts down the server or migrates virtual machines and containers. Normally, the operation and maintenance personnel need to confirm on the human-machine interface provided by the robot management module. Here, this operation is further provided to the operation and maintenance personnel as a configuration option. When the above situation occurs, it can be directly processed without confirmation, thereby increasing the degree of automation of operation and maintenance.

[0063] The following is a comparative explanation of a typical embodiment. In the cabinet A of this embodiment, the fan outlet of a backup server B is blocked, and the server temperature is too high due to long-term operation. This server is for backup purposes and is configured to be automatically shut down by the robot agent module when an abnormal temperature is found.

[0064] 1. The inspection robot finds that the temperature of cabinet A is too high, obtains the identification information of cabinet A and sends it to the robot management module.

[0065] 2. The robot management module queries the robot agent module information pre-installed in all servers in cabinet A based on the identification information of cabinet A, and sends control commands to these robot agent modules.

[0066] 3. The robot agent module executes commands, obtains the temperature of the hardware devices of each server and reports it to the robot management module. The CPU temperature of server B is 70 degrees, the hard disk temperature is 51 degrees, the graphics card temperature is 53 degrees, and the fan temperature is 75 degrees.

[0067] 4. The robot management module comprehensively judges and finds that the temperature of server B is abnormal. It controls the robot agent module to power off server B and informs the operation and maintenance personnel of the temperature information of the fan, CPU, hard disk, graphics card and other components. All these operations are automatically controlled by the program and can be completed within 1 minute.

[0068] 5. Based on the device temperature information of server B provided by the system, the operation and maintenance personnel focused on checking the fan-related areas and quickly discovered that the air outlet was blocked.

[0069] In contrast, in traditional data center inspection robot systems, the inspection robot and the servers on the data center cabinets are strictly isolated, and no data and information interaction occurs. When the robot finds that the temperature on the surface of cabinet A is too high during inspection, it needs to check all the servers on cabinet A one by one. At least the servers adjacent to the over-temperature area need to be checked until server B with the over-temperature is found. The operation and maintenance personnel then go deep into the server and use special tool software or other methods to check the temperature of each key hardware module inside server B until the fan outlet is found to be blocked. The whole process takes hours or days.

[0070] It can be seen from the above embodiments that when the system locates the overtemperature situation in the data center, it shortens the positioning time from several hours to seconds, locates and solves the problem more quickly and accurately, saves operation and maintenance time, improves the operation and maintenance level of the data center, and has considerable economic value.

[0071] This system is designed to more quickly and accurately locate problems with excessive temperature and abnormal power consumption in the data center, and automatically adjust server loads.

[0072] 1. When the inspection robot finds a cabinet with excessively high temperature and abnormal power consumption, the corresponding robot agent module obtains the temperature of the hardware components of the server in the abnormal cabinet and the occupancy of computing resources by the running entities in the server. After system processing, the scope of the abnormality is further narrowed to the hardware components and running entities in the server.

[0073] 2. The system comprehensively analyzes abnormal situations and notifies operation and maintenance personnel to adjust server load. The robot agent module automatically shuts down servers, migrates virtual machines and containers, and kills abnormal processes.

[0074] 3. The robot agent module can be deployed to the server in the data center through virtual machines, containers, etc., which is convenient and fast. It can be selected on demand according to the needs of server management and has little interference with the original server system.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalents that do not depart from the spirit and scope of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A patrol inspection system for a data center, characterized in that: include: An inspection robot is used to inspect the data center equipment and environment to find abnormal temperature areas or abnormal power consumption areas of the data center equipment; Acquire the temperature and humidity data of the data center environment, and send the data of the abnormal temperature area or the abnormal power consumption area and the temperature and humidity data to the robot management module; The robot agent module is used to obtain the temperature data of the internal hardware of the server in the data center and the load of the server in real time, and send the temperature data and load to the robot management module; The robot agent module adjusts the load exceeding the first preset threshold value according to the instruction issued by the robot management module, processes the internal hardware of the server exceeding the preset temperature, and sends the temperature data and the load of the server to the robot management module; The robot management module is used to monitor and manage the inspection robot, receive the data of the abnormal temperature area or the abnormal power consumption area, the temperature and humidity data, the internal hardware temperature data of the server, and the load of the server sent by the inspection robot and the robot agent module, analyze the inspection data that exceeds the second preset threshold according to the inspection data received from the inspection robot and the robot agent module, send the analysis results to the operation and maintenance personnel, and adjust the server load according to the instructions issued by the operation and maintenance personnel.

2. The system according to claim 1, characterized in that The inspection robot includes the following components: a vehicle chassis motion platform, a multi-stage lifting platform, an industrial computer, an on-board wireless communication device, a laser radar, an RFID reader, a positioning device, and is also equipped with a visible light camera, an infrared thermal imager, a temperature and humidity sensor, an environmental gas sensor, and a voice intercom device.

3. The system according to claim 1, characterized in that The inspection robot is used to inspect the data center equipment and environment, and find the abnormal temperature area or power consumption area of ​​the data center equipment, including: The inspection robot uses an infrared thermal imager to perform infrared scanning on the surface of the data center cabinet to find areas with abnormal temperature, or uses a visible light camera to read the readings of the power meter installed on the data center cabinet to find areas with abnormal power consumption.

4. The system according to claim 3, characterized in that Also includes: The inspection robot uses a visible light camera to scan the barcode or QR code preset on the cabinet to obtain the unique identification of the cabinet and determine the location of the cabinet; or Use an RFID reader to read the RFID tag pre-installed on the cabinet, obtain the identification information of the cabinet, and determine the location of the cabinet; The data of the abnormal temperature area or the abnormal power consumption area and the corresponding cabinet location are sent to the robot management module.

5. The system according to claim 1, characterized in that Obtain temperature and humidity data of the data center environment, including: The inspection robot obtains the temperature and humidity data of the environment at various locations in the data center based on temperature and humidity sensors and positioning devices.

6. The system according to claim 1, characterized in that The robot agent module is installed in the server of the data center to obtain the temperature data of the internal hardware of the server, the usage data of the server's computing and storage resources, and the process and resource occupancy data in the server in real time.

7. The system according to claim 6, characterized in that The robot agent module sends the internal hardware data of the server that exceeds the preset temperature to the robot management module.

8. The system according to claim 1, characterized in that The robot agent module adjusts the load exceeding the first preset threshold according to the instruction issued by the robot management module, including: For server loads that exceed a first preset threshold, the server load is adjusted by calling an API interface provided by the underlying virtualization of the server, migrating virtual machines and containers, killing abnormal processes, and so on.

9. The system according to claim 1, characterized in that The robot management module is used to monitor and manage the inspection robot, including: Monitor the running status of the inspection robot and set the inspection path of the inspection robot.

10. The system according to claim 1, characterized in that According to the inspection data received from the inspection robot and the robot agent module, the inspection data exceeding the second preset threshold is sent to the operation and maintenance personnel, including: The robot management module analyzes and judges the inspection data exceeding the second preset threshold, and obtains the current operation and maintenance suggestions based on the judgment rules; The inspection data and operation and maintenance suggestions are sent to the operation and maintenance personnel.

Images