Automatic repair of computing equipment in data centers

Abstract

A management device for managing multiple computing devices in a data center may include: a network interface; a first module that periodically sends health status queries to the computing devices via the network interface; a second module configured to receive responses to the health status queries and collect and store health status data of the computing devices; a third module configured to create support tickets; and / or a fourth module configured to (i) create and periodically update a Cox proportional hazards (CPH) model based on the health status data; (ii) apply a deep neural network (DNN) to the input of the CPH model; (iii) determine the failure probability for each computing device; (iv) compare each failure probability to a threshold; and (v) cause the third module to generate a pre-failure support ticket for each computing device with a determined failure probability above the threshold.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Patent Application No. 16 / 879,157, filed May 20, 2020, which is a continuation to U.S. Patent Application No. 10,691,528, filed January 29, 2020, Serial No. 16 / 776,213, entitled “Computing System,” filed July 23, 2019, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] This disclosure relates generally to the field of computing, and more specifically to systems and methods for managing multiple computing devices in a data center. Background Technology

[0004] The background description is set forth below for the purpose of providing context only. Therefore, no aspect of this background description is intended or implied to be an admission of prior art to this disclosure, to the extent that it is not otherwise considered as such.

[0005] Many cryptocurrencies (such as Bitcoin and Litecoin) are based on a technology called blockchain, in which transactions are grouped into blocks. These blocks, along with previous blocks containing earlier transactions, are stored in a ledger (“blockchain”) and made immutable (i.e., practically unmodifiable) by including hash values. Hash values ​​are numbers calculated based on blocks and conform to the standards of a particular blockchain. Once a block and its hash value are confirmed by the cryptocurrency network, they are added to the blockchain. Hash values ​​can be used to verify whether any previous transactions or blocks on the blockchain have been altered or tampered with. This creates an immutable ledger of transactions and allows cryptocurrency networks to prevent someone from attempting to double-spend digital coins.

[0006] Many cryptocurrency networks consist of a large number of participants who repeatedly attempt to be the first to calculate a hash value that meets the requirements of the blockchain network. Depending on the blockchain, these participants may receive rewards (e.g., coin rewards or transaction fee rewards) for being the first to successfully calculate a hash value, and these rewards can incentivize them to continue participating.

[0007] Many blockchain networks need to solve computationally difficult problems as part of hash calculations. Difficult problems require a solution that is data that is difficult (expensive, time-consuming) to generate but easily verifiable by others and meets certain requirements. This is often referred to as "Proof-of-Work." A Proof-of-Work (PoW) system (or protocol or function) is a consensus mechanism. It prevents denial-of-service attacks and other service abuses, such as spam on the network, by requiring service requesters to perform some work (usually meaning computer processing time). For some blockchain networks attempting to compensate for increases in hash power occurring on the network, the difficulty level may be changed periodically.

[0008] Participants in the network operate standard PCs, servers, or dedicated computing devices. Due to the difficulty involved and the amount of computation required, dedicated computing devices are typically configured with specialized components to increase the speed of performing hashes (the device's hash rate) or other computations required by the blockchain network. Examples of specialized components include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), graphics processing units (GPUs), and accelerated processing units (APUs). Dedicated cryptocurrency mining software (e.g., cgminer) can also be used in conjunction with specialized components, such as software applications configured to compute the SHA-256 algorithm.

[0009] Specialized computing devices often operate for extended periods at high frequencies that generate significant amounts of heat. Even with cooling (e.g., high-speed fans), the heat and continuous operation can negatively impact the reliability and lifespan of components within these devices. For example, ASIC-based specialized computing devices contain a large number of hash chips (e.g., 100), which are more prone to failure as temperatures rise.

[0010] Many participants in blockchain networks simultaneously operate large numbers of (e.g., 1,000, 10,000, 50,000 or more) of different dedicated computing devices (e.g., dedicated computing devices from one manufacturer or different generations from different manufacturers) in large data centers. These data centers and large numbers of dedicated computing devices can be difficult to manage. Data centers accommodating large numbers of dedicated computing devices or other ASIC or GPU-based systems face different challenges than traditional data centers accommodating more general-purpose computers. This is due to significantly higher density, including higher power usage, higher heat generation, and almost continuous compute-intensive operations.

[0011] Continuous operation often leads to performance problems such as memory leaks. Memory leaks can degrade a computer's performance by reducing the amount of available memory. Memory leaks can be a problem when a program runs for a long time and consumes more and more memory over time. Eventually, too much available memory may be allocated, and all or part of the device or system may stop functioning properly. One or more applications running on the device may fail, and the system may slow down due to thrashing. Thrashing refers to the excessive use of a computer's virtual memory resources, leading to a persistent state of paging and page faults, which significantly slows down or inhibits application-level processing.

[0012] In large data centers, a large number of units may fail daily for both known and unknown reasons. A typical data center management solution is to determine when a computing device stops responding to requests (e.g., responding to network pings) and then restart the device (e.g., by going to the device and unplugging it). This is less than ideal because it may require data center technicians to spend a significant amount of time each day performing meticulous and manual restarts of all failed devices. Furthermore, significant losses can occur during the process when device performance deteriorates while the device is still able to respond to requests.

[0013] For at least these reasons, there is a need for systems and methods that allow for improved management of large numbers of computing devices. Summary of the Invention

[0014] Methods and systems for more easily managing data centers with multiple computing devices are envisioned. Example computing devices include, for example, ASIC computing devices, FPGA computing devices, and GPU computing devices, but other types of computing devices are also possible and anticipated.

[0015] In one embodiment, the method includes issuing automatic (e.g., without human intervention) status queries and repair instructions based on monitored attributes and predefined health attribute levels. A first health status query may be sent to a second computing device. The health status query may include, for example, hash rate or temperature. In response to no acceptable response to the first health status query being received within a first predetermined time, a first repair instruction is sent to the second computing device. After sufficient time has elapsed to complete the first repair instruction, a second health status query may be sent to the second computing device. In response to no acceptable response to the second health status query being received within a second predetermined time, a second repair instruction is sent to the second computing device. After sufficient time has elapsed to complete the second repair instruction, a third health status query may be distributed to the second computing device. In response to no acceptable response to the third health status query being received within an acceptable time period, a repair ticket may be generated.

[0016] In some embodiments, the first repair instruction may include resetting only the mining application running on the computing device, adjusting fan speed, voltage level, and operating frequency, and the second repair instruction may include resetting the entire computing device.

[0017] A system for managing computing devices operating in a data center is also envisioned. In one embodiment, the system may include a network interface for communicating with the managed computing devices and multiple modules configured together to automatically manage the computing devices. Modules may include, for example, a first module that sends status queries to the managed computing devices. An exemplary second module may be configured to receive and process responses to health status queries, and a third module may be configured to create support tickets in response to two or more failed repair attempts. A repair attempt can be determined to have failed when a predetermined amount of time has elapsed without receiving an acceptable response to the health status query.

[0018] The system can be implemented in software as instructions that can be executed by the processor of a computing device, and these instructions can be stored on a non-transitory computer-readable storage medium, such as a flash drive, CD-ROM, DVD-ROM, or hard disk.

[0019] In an embodiment, a management device for managing multiple computing devices in a data center may include: a network interface for communicating with the multiple computing devices; a first module that periodically sends health status queries to each computing device via the network interface; a second module configured to receive responses to the health status queries and collect and store health status data for each computing device; a third module configured to create support tickets; and / or a fourth module. The fourth module may be configured to: (i) create and periodically update a Cox proportional hazards (CPH) model based on the collected health status data; (ii) apply a deep neural network (DNN) to the input of the CPH model; (iii) determine a failure probability for each computing device; (iv) compare each determined failure probability to a predetermined threshold; and / or (v) cause the third module to generate a pre-failure support ticket for each computing device with a determined failure probability exceeding the predetermined threshold.

[0020] The foregoing and other aspects, features, details, utility and / or advantages of the embodiments of this disclosure will become apparent from reading the following description and from viewing the accompanying drawings. Attached Figure Description

[0021] Figure 1 This is a top view of an example of a data center used for computing devices.

[0022] Figure 2 This is a front view of an example pod in a data center used for computing devices.

[0023] Figure 3 This is an illustration of an example of a rack used for computing equipment in a data center.

[0024] Figure 4 This is an illustration of an example computing device.

[0025] Figure 5 This is a flowchart illustrating an example method for managing computing devices in a data center in accordance with the teachings of this disclosure.

[0026] Figure 6 This is an illustration of an example system for managing computing devices in a data center, based on the teachings of this disclosure.

[0027] Figure 7 This is a flowchart illustrating another example method for managing computing devices in a data center. Detailed Implementation

[0028] Embodiments of this disclosure will now be described in detail, with examples thereof described herein and illustrated in the accompanying drawings. While this disclosure will be described in conjunction with embodiments and / or examples, it is to be understood that they are not intended to limit this disclosure to those embodiments and / or examples. Rather, this disclosure covers alternatives, modifications, and equivalents.

[0029] Various embodiments for various apparatuses, systems, and / or methods are described herein. Numerous specific details are set forth to provide a thorough understanding of the overall structure, function, manufacture, and use of the embodiments described in the specification and illustrated in the accompanying drawings. However, those skilled in the art will understand that the embodiments can be practiced without such specific details. In other instances, well-known operations, components, and elements have not been described in detail so as not to obscure the embodiments described in the specification. Those skilled in the art will understand that the embodiments described and illustrated herein are non-limiting examples, and therefore it will be understood that the specific structural and functional details disclosed herein may be representative and not necessarily limit the scope of the embodiments.

[0030] Now for reference Figure 1The image shows a top view of an example data center 100 used for computing equipment. Data center 100 is configured with a large number of pods 110. Pods are standardized blocks of racks arranged in rows or (more typically) in pairs, sharing common infrastructure components such as power distribution units, network routers / switches, sealing systems, and air handling units. For example, a pod might have two parallel equipment racks spaced apart, each facing outwards. The equipment on the racks can be oriented to draw in cool air from outside the pod and exhaust hot air (heated by the computing equipment) into an empty space at the center of the pod, where the hot air then rises and is expelled from the data center. For example, a hot air duct might be present in the middle of each pod to collect the hot exhaust gases, and the pod then exhausts the hot exhaust gases from the data center via vents in the data center roof.

[0031] Turn now Figure 2 The diagram shows the front side of an example of compartment 110. Compartment 110 has multiple racks 210, each rack 210 having multiple shelves 230 for holding computing devices. For organizational and management purposes, the shelves can be grouped together in a switching section 220, each shelf in the switching section 220 being supported by the same network switch. Within each shelf of the shelves 230, there can be multiple bin locations 240, each bin location 240 holding a single computing device. Each computing device can be installed in a bin that has power and network connections.

[0032] Turn now Figure 3 The image shows a more detailed front view of a rack 230 within example rack 210. In this example, computing devices 310 are mounted in each case 240 within rack 230. In this example, computing device 310 is an ASIC computing device. ASIC computing devices typically include a controller board with a network port 320, one or more status indicator LEDs 330, and a pair of cooling fans (front fan 340 shown) that draw air through the center of the computing device. Multiple hash boards are present within the computing device that perform computations and generate heat.

[0033] Turn now Figure 4 The illustration shows an example embodiment of computing device 310. In this example, computing device 310 includes a controller 360 that oversees the operation of multiple hashboards 350 within the device. Controller 360 also includes a network port 320 for communication, a reset button 370 for resetting computing device 310, and one or more indicator LEDs 330. Two fans 340 (a front fan and a rear fan) draw cool air into computing device 310 and exhaust waste heat through the rear of computing device 310. Other types of computing devices are possible and anticipated.

[0034] As mentioned above, one problem faced by operators of large data centers is identifying and servicing computing devices that are not operating optimally. For a variety of reasons, including the time required for a device to degrade from full operational status to an unresponsive state, waiting for a device to fail and then manually restarting it or resetting it using a reset button is undesirable. For at least this reason, there is a need for improved systems and methods for managing large numbers of computing devices.

[0035] Turn now Figure 5 The diagram illustrates a flowchart of an example embodiment of an improved method for managing computing devices in a data center, in accordance with the teachings of this disclosure. A request for status information from one or more computing devices is made (step 500). This request may be sent from a management server to one or more computing devices 310 via a network connection (e.g., wired or wireless Ethernet). If the response is acceptable (step 510), another periodic request may be sent after a predetermined polling interval has elapsed (step 520). For example, the status request may be sent every 1, 5, 6, 10, or 30 minutes. The status request may request data about different operating parameters of the computing device, such as hash rate, temperature, fan speed, or the number of hardware errors. An example request may query the temperature of a particular computing device 310, and an acceptable response may be one indicating that the operating temperature is below the operating temperature specified by the manufacturer for that particular device type and model.

[0036] Some requests may be sent to the data provider rather than the monitored device. For example, in some embodiments, a hash rate request may be sent to the monitored device, but in other embodiments, the request may be sent to a database that stores information from the pool that the computing device is currently processing. For example, the database may be a local copy of the data provided by the pool. Hash rates are typically calculated in hashes per second, such as 3 PH / s (3 × 10⁻⁶ hashes per second). 15 (hashes), 13TH / s (13×10 per second) 12 (hash) or 90GH / s (90×10 per second) 9 The hash rate is reported by pools and may be provided periodically. Some pools track hash rates on a worker basis. This means that if more than one compute device is connected to the pool as a single worker, the reported hash rate information may be batch-overwritten across all of these compute devices. Some pools allow the use of a worker name specific to the compute device, which enables the pool to track the hash rate and report the hash rate separately for each compute device.

[0037] If the response does not indicate an acceptable state for the computing device (step 510), a first repair instruction is sent to the computing device (step 530). An example of a first repair instruction is to restart an application running on the computing device. For example, a mining application running on the computing device could be restarted. This is different from restarting the entire computing device. Another example of a repair instruction includes an instruction to increase fan speed or decrease operating voltage in response to a status response indicating excessively high temperature. Depending on the managed computing device, the repair instruction may also include running a computer program on the computing device.

[0038] After the first repair instruction has been sent, a waiting period occurs (step 540) to allow the target computing device to complete the execution of the first repair instruction. For example, a waiting period of five, ten, or fifteen minutes can be used to give the target computing device sufficient time to complete the execution (or repeat) of the first repair instruction. Then, another status request is sent to the computing device (step 550). If the response is acceptable (e.g., within the manufacturer's temperature or hash rate specifications) (step 560), the system waits until the next polling cycle (step 520) before making another status request (step 500). If the response is unacceptable (step 560), a second repair instruction is sent (step 570). An example of a second repair instruction is a full device reset instruction. Another example of a second repair instruction is an instruction to reduce the operating frequency in response to receiving a status response indicating that the temperature is too high.

[0039] After the second repair instruction has been sent, a waiting period occurs (step 580) to allow the target computing device to complete the execution of the second repair instruction. Then, another status request is sent to the computing device (step 590). If the response is acceptable (step 592) (e.g., within the manufacturer's temperature or hash rate specifications), the system waits until the next polling cycle (step 520) before making another status request (step 500). An acceptable response (steps 510, 560, 592) can also remove the computing device from the list of any existing faulty devices. If the response is unacceptable (step 592), a support ticket is generated (step 594). The support ticket can include not only the identifier of the computing device but also the history of the repair instructions executed and the resulting health status report. Advantageously, this can save support personnel time from manually executing repair instructions.

[0040] In some embodiments, a health status query may include running diagnostic instructions or a set of instructions. In other embodiments, a health status query may be a query on a database that stores status information collected periodically by the computing device (e.g., received from a pool that the computing device is processing).

[0041] Turn now Figure 6 This illustration shows an example embodiment of a system for managing computing devices in a data center. In this embodiment, the system includes a large number of computing devices 310. The computing devices 310 communicate with a management server 600 via a network interface 640 of a server. While wireless networks are possible, the current density of computing devices in a data center means that it is currently preferred to use a wired network, such as wired Ethernet, for communication between the management server 600 and the computing devices 310. In some embodiments, the computing devices 310 may include a controller 360 and a network interface for communicating with the management server 600 via the network 610. The controller 360 may be configured to send computing tasks to one or more computing or hash boards 350, each computing or hash board having multiple GPUs or ASIC chips 390 that can operate at frequencies specified by the controller. The computing devices 310 may further include multiple cooling fans 340 and a power supply 380. The voltage output to the ASIC chips 390 may vary based on settings configured by the controller 360. Higher voltage and frequency levels for the ASIC chips 390 will improve performance, but they may also increase heat and negatively impact lifespan.

[0042] Management server 600 can be a traditional PC or server, or a dedicated device. Management server 600 may be configured with one or more processors 620, volatile and non-volatile memory, such as flash memory or internal or external hard drives (e.g., network-attached storage accessible to server 600). Management server 600 is configured to execute management application 630 to assist users (e.g., data center technicians) in managing computing device 310. Management server 600 may be located within the same data center or facility as computing device 310, or it may be located remotely and connected to computing device 310 via the Internet.

[0043] The management application 630 is preferably implemented in software (e.g., instructions stored on a non-volatile storage medium such as a hard disk, flash drive, or DVD-ROM), but a hardware implementation is possible. The management application 630 may include multiple modules, such as: a user interface module 632 that displays data to and receives input from the user (e.g., via an external display or via the user's web browser); a query sending module 634 that sends status queries to obtain status data of the computing device 310 (e.g., periodically polling the health status of each device); a query response processing and repair module 638 that receives and processes status query responses and issues repair instructions when necessary; and a support ticket creation module 636 that creates a support ticket if the repair instructions fail to move the computing device to an acceptable state. If the problem is resolved, module 636 may generate a "resolved" ticket or update an existing ticket containing detailed information about the history of providing the computing device in the ticketing system. Modules 636 and 638 may also include intelligence (e.g., rules) to prevent the management application from getting stuck in a loop due to recurring problems with a device or a group of devices. For example, after a support ticket is sent, the ticket creation module 636 can be configured not to send subsequent repair instructions or create additional tickets until a technician indicates that the device has been repaired or the device correctly responds to a health status query (indicating that the device has been repaired).

[0044] In some embodiments, the user interface module 632 may provide a user with an interface to configure rules (or override predefined rules) for when and which repair instructions are sent. The management application 630 may be configured to automatically execute this step if conditions of the rules (e.g., symptoms or leading indicators of a problem) are determined to be met. In some embodiments, the computing system may be configured to learn from past data on the activity and / or profiles of a second computing device and take corrective / proactive actions, such as based on leading indicators, before a problem occurs. For example, in some learning-based embodiments, the management application 630 may be configured to initiate repairs based on leading indicators (e.g., detected hash rate decline) before a previously encountered problem (e.g., device hang) recurs.

[0045] In one embodiment, management application 630 may be configured to monitor for anomalies in key health metrics (e.g., hash rate, temperature) and, when a problem is identified, to identify the scale of the problem and escalate it appropriately, including notifying the appropriate individuals when an escalation beyond automatically issuing a repair instruction (self-healing) is required. For example, if a single device reports an elevated temperature, a first (and second, if necessary) repair instruction may be dispatched. However, if multiple devices in the same rack (e.g., more than 5%) begin to experience elevated temperatures, management application 630 may be configured to (1) turn on or increase the active cooling rate (e.g., evaporative coolers or air conditioners) for that area of ​​the data center, or (2) create a high-priority ticket for the rack, as multiple high-temperature health status reports may indicate a broader airflow problem in that part of the data center, rather than just a problem with a single computing device.

[0046] In another embodiment, management application 630 may be configured to apply artificial intelligence (AI) and machine learning (ML) to predict and create remedial mechanisms (e.g., repair instructions) before anomalies occur or reach a critical impact in computing devices. For example, in one embodiment, management application 630 may be configured to track the status report history of computing device 310 in data storage 650. When a problem pattern is detected (e.g., a series of multiple unacceptable health status responses within a predetermined time period), ticket creation module 636 may create a ticket even if the repair instruction is successful. These "repeated" devices may be on the verge of more catastrophic failure and may benefit from technicians inspecting and replacing worn components (such as fans or power supplies). Management application 630 may be further configured to provide a user with a list of repeated devices via user interface module 632.

[0047] In some embodiments, management application 630 can be configured to avoid generating multiple tickets when multiple machines are down in the same region due to the same problem. Regions can be defined by aggregation. Levels may increase the number of second compute devices affected in a defined region, from a single second compute device up to and including multiple second compute devices across multiple data centers. For example, defined levels may include: overall (all sites), site, cabin, rack, switch, and individual compute device. For example, if the "machine uptime" metric is healthy at the site, cabin, and rack levels but unhealthy at the switch level, a ticket may be generated at the switch level. If management application 630 detects health status numbers that are outside the acceptable range for a particular aggregation level, the application can generate a ticket along with remediation instructions to attempt to resolve the issue automatically. When multiple compute devices are down due to the same problem, such as in the case of a dead switch cutting off network communication with compute devices from multiple companies, a ticket may be generated for each customer.

[0048] In some embodiments, to help identify when computing device 310 may enter a fault state and what other indicators may be associated with the fault state, management application 630 may include artificial intelligence (AI) and machine learning (ML) module 654 to predict anomalies in computing device 310 and create remedial mechanisms (e.g., support tickets) before the anomaly occurs or reaches a critical impact. In some embodiments, AI / ML module 654 may be configured to use a Cox proportional hazards (CPH) model to predict the probability of a fault event in computing device 310 based on historical telemetry data (stored in data storage 650) and optional climate data (e.g., temperature and humidity readings in a data center).

[0049] CPH models are typically used in clinical settings to determine how multivariate factors can affect patient survival. The benefit of CPH models is their ability to simultaneously assess the impact of several factors on patient survival. Computational devices 310 that output a failure probability above a predetermined threshold (e.g., 80% probability) can generate support tickets, allowing data center personnel to inspect them and replace parts as needed. For AI and ML, models are built and trained using data from data storage 650.

[0050] In some embodiments, the initial model for CPH can be built using open-source packages such as Themeaux'sSurvival in R or Pilon's Lifelines in Python. The model's performance can then be tested to confirm its satisfaction based on a concordance index, a measure of the predictions made by the evaluation algorithm and used to score the survival model. The score is calculated as the proportion of concordant pairs divided by the total number of possible evaluation pairs.

[0051] The high-scoring model can then be enhanced by connecting a deep neural network (DNN) to the input of the CPH model. This method uses the input-output relationship associated with a simple feedforward neural network to model the examined survival data as the basis for a non-linear proportional hazards model. In one embodiment, this is the Faraggi and Simon approach using a neural network for regression. For example, the network can be configured with a single output node and a single-layer perceptron with inputs, hidden nodes, bias nodes, and outputs. A sigmoid non-linearity can be used between the inputs and hidden nodes. Advantageously, these DNN implementations of CPH may outperform traditional CPH and are more scalable due to the ability to adjust or arbitrarily choose the neural network architecture.

[0052] In some embodiments, AI module 654 may implement the Efron method for calculating probabilities, as this method has been shown to outperform other methods, being faster than exact calculation methods and tending to produce estimates that are closer than other methods. The implementation of the Efron method exists in TensorFlow 2.0, making it more reusable than previous CPH DNN work.

[0053] Turn now Figure 7 The diagram illustrates details of an embodiment of the operation of the AI / ML module 654. In this embodiment, data from multiple computing devices is collected and stored (step 700). A CPH model is created / updated based on the collected data (step 710). A DNN is applied to the input of the CPH module (step 720). The failure probability of each computing device is determined based on the model (step 730). If the failure probability is greater than a predetermined threshold (step 740) and no support ticket has been generated (step 760), a ticket is generated (step 770). If the failure probability is lower than the predetermined threshold (step 740), the process waits until the polling interval before repeating (step 750).

[0054] Throughout this specification, references to "various embodiments," "adopting embodiments," "in embodiments," or "embodiments," etc., indicate that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment. Therefore, phrases such as "in various embodiments," "adopting embodiments," "in embodiments," or "embodiments," appearing throughout this specification, do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Thus, a particular feature, structure, or characteristic illustrated or described in connection with one embodiment / example can be combined, in whole or in part, with features, structures, functions, and / or characteristics of one or more other embodiments / examples without limitation, as such combinations are not illogical or nonfunctional. Moreover, numerous modifications can be made to adapt particular situations or materials to the teachings of this disclosure without departing from the scope of this disclosure.

[0055] It should be understood that references to a single element are not necessarily so limited and may include one or more of such elements. Any directional references (e.g., addition, subtraction, up, down, upward, downward, left, right, left to right, top, bottom, above, below, vertical, horizontal, clockwise, and counterclockwise) are used only for identification purposes to aid the reader's understanding of this disclosure and do not impose limitations, particularly regarding the location, orientation, or use of embodiments.

[0056] Engagement references (e.g., attachment, coupling, connection, etc.) should be interpreted broadly and may include intermediate components between relative movements between elements and connections between elements. Therefore, an engagement reference does not necessarily mean that two elements are directly connected / coupled and in a fixed relationship with each other. The use of "for example" in the specification should be interpreted broadly and is used to provide non-limiting examples of embodiments of this disclosure, and this disclosure is not limited to such examples. The use of "and" and "or" should be interpreted broadly (e.g., considered as "and / or"). For example, but not limited to, the use of "and" does not necessarily require all listed elements or features, and the use of "or" is inclusive unless the construction is illogical.

[0057] While a process, system, and method may be described herein by combining one or more steps in a particular order, it should be understood that the method may also be practiced with steps in a different order, performing certain steps, performing additional steps, and / or omitting certain steps described.

[0058] All content contained in the above description or shown in the accompanying drawings should be interpreted as illustrative rather than restrictive. Changes in detail or structure may be made without departing from this disclosure.

[0059] It should be understood that the computer, system, and / or processor described herein may include conventional processing means known in the art capable of executing pre-programmed instructions stored in associated memory, all of which perform the functions described herein. To the extent that the methods described herein are embodied in software, the resulting software may be stored in associated memory and may also constitute components for performing the methods. The system or processor may further be of a type having ROM, RAM, RAM and ROM, and / or a combination of non-volatile and volatile memory, making it possible to store any software and also allowing the storage and processing of dynamically generated data and / or signals.

[0060] It should be further understood that the article of manufacture according to this disclosure may include a non-transitory computer-readable storage medium having a computer program encoded thereon for implementing the logic and other functions described herein. The computer program may include code for performing one or more methods disclosed herein. This embodiment may be configured to execute via one or more processors, such as multiple processors integrated into a single system or distributed across and connected together on a communication network, and the communication network may be wired and / or wireless. Code for implementing one or more features described in conjunction with one or more embodiments may, when executed by a processor, cause multiple transistors to change from a first state to a second state. Specific change patterns (e.g., which transistors change state while which do not) may be specified at least in part by logic and / or code.

Claims

1. A management device for managing multiple computing devices in a data center, wherein, The management device includes: A network interface for communicating with the plurality of computing devices. The first module sends a first health status query for one of the plurality of computing devices. The second module is configured to receive and process any response to the first health status query, and The third module is configured to create support tickets. The second module is configured to respond if no acceptable response to the first health status query is received within a first predetermined time: (a) Send a first repair command to the selected computing device. (b) Wait for at least sufficient time for the first repair instruction to complete. (c) Cause the first module to send a second health status query to the selected computing device; and (d) If no acceptable response to the query for the second health status is received within a second predetermined time: (i) Cause the first module to send a second repair command to the selected computing device. (ii) Wait for at least sufficient time for the second repair instruction to complete. (iii) Send a third health status query to the selected computing device; and (iv) In response to not receiving an acceptable response to the third health status query, the third module creates a support ticket that identifies the selected computing device and the health status of the selected computing device; The management device further includes a fourth module, which includes a deep neural network (DNN) implementation of the Cox proportional hazards (CPH) model. The DNN and CPH model are trained on historical state data from the plurality of computing devices. The fourth module is configured to output a predicted failure probability for the selected computing device. The third module is configured to create a prediction support ticket for the selected computing device if the predicted failure probability is greater than a predetermined threshold.

2. The management device according to claim 1, wherein, The first health status query, the second health status query, and the third health status query are queries of the hash rate of the selected computing device, wherein the first repair instruction is to restart the mining application operating on the selected computing device, and wherein the second repair instruction is to restart the selected computing device.

3. The management device according to claim 1, wherein, The first health status query, the second health status query, and the third health status query are queries of the temperature of the selected computing device, wherein the first repair instruction is to increase the fan speed of the selected computing device, and wherein the second repair instruction is to decrease the operating frequency of the selected computing device.

4. The management device according to claim 1, wherein, The plurality of computing devices are installed in a plurality of racks, wherein the plurality of racks are located in a plurality of compartments, wherein the second module is configured to detect: (i) Whether more than a first predetermined percentage of the computing devices within a specific rack have not provided an acceptable response to the health status query; or (ii) Whether more than a second predetermined percentage of the computing devices within a specific cabin have not provided an acceptable response to the health status query. In response, the third module creates a support ticket for the specific rack or compartment.

5. The management device according to claim 1, wherein, The plurality of computing devices are installed in a plurality of racks, wherein the health status query is temperature-dependent, wherein the second module is configured to detect whether computing devices in a particular rack of the plurality of racks that are above a predetermined threshold have not yet provided an acceptable response to the health status query, and in response to activate or increase the rate of active cooling of the area of ​​the data center including the particular rack.

6. The management device according to claim 1, wherein, The second module is configured to store health status query responses and detect repeat offenders among the plurality of computing devices, wherein the repeat offender has multiple unacceptable health status query responses within a predetermined time period, even if repair has been performed in accordance with the repair instructions.

7. A method for managing multiple computing devices in a data center, the method comprising: A first health status query is issued from the first computing device for the second computing device among the plurality of computing devices; If no acceptable response to the query for the first health status is received within a first predetermined time: (i) A first repair command is sent from the first computing device to the second computing device. (ii) Wait for at least enough time for the first repair instruction to complete. (iii) Issuing a second health status query for the second computing device from the first computing device, and (iv) In response to the failure to receive an acceptable response to the second health status query within a second predetermined time: (a) A second repair command is issued from the first computing device to the second computing device. (b) Wait for at least sufficient time for the second repair instruction to complete. (c) Issue a third health status query from the first computing device to the second computing device, and (d) In response to the absence of an acceptable response to the third health status query within a second predetermined time, a support ticket is created to identify the second computing device and the health status of the second computing device; Health status data is collected from each of the plurality of computing devices; A Cox proportional hazards CPH model is created based on health status data collected from each of the plurality of computing devices; A deep neural network (DNN) is applied as input to the CPH model; Determine the failure probability for each of the plurality of computing devices; as well as For each of the plurality of computing devices whose determined failure probability exceeds a predetermined threshold, a prediction support ticket is generated.

8. The method according to claim 7, wherein, The first repair instruction is to restart the mining application operating on the second computing device, and the second repair instruction is to restart the second computing device.

9. The method according to claim 7, wherein, The first health status query, the second health status query, and the third health status query are queries of the hash rate of the second computing device, wherein the first repair instruction is to restart the mining application operating on the second computing device, and wherein the second repair instruction is to restart the second computing device.

10. The method according to claim 7, wherein, The first health status query, the second health status query, and the third health status query are queries of the temperature of the second computing device, wherein the first repair instruction is to restart the mining application operating on the second computing device, and wherein the second repair instruction is to restart the second computing device.

11. The method according to claim 7, wherein, The first health status query, the second health status query, and the third health status query are queries of the temperature of the second computing device, wherein the first repair instruction is to increase the fan speed, and wherein the second repair instruction is to reduce the operating frequency of the second computing device.

12. The method according to claim 7, wherein, The first health status query, the second health status query, and the third health status query are for the fan speed of the second computing device, wherein the first repair instruction is to change the fan speed, and wherein the second repair instruction is to restart the second computing device.

13. The method of claim 7, further comprising: In response to the first repair instruction or the second repair instruction that generates an acceptable health status query response, a resolved ticket is generated.

14. The method of claim 7, further comprising: The system stores information about the number of repair instructions dispatched to the second computing device, and avoids submitting any more repair instructions to the second computing device if a predetermined threshold for repair attempts has been exceeded within the repair window.

15. A non-transitory computer-readable storage medium storing instructions executable by a processor of a computing device, the instructions causing the computing device, when executed, to: A first health status query is sent from the first computing device to the second computing device, and in response to no acceptable response to the first health status query being received within a first predetermined time: (i) Sending a first repair command from the first computing device to the second computing device. (ii) Wait for at least enough time for the first repair instruction to complete. (iii) Sending a second health status query from the first computing device to the second computing device; and (iv) In response to no acceptable response to the second health status query received within a second predetermined time: (a) Sending a second repair command from the first computing device to the second computing device. (b) Wait for at least sufficient time for the second repair instruction to complete. (c) Send a third health status query from the first computing device to the second computing device. (d) In response to the absence of an acceptable response to the query for the third health status from the second computing device within a second predetermined time, a support ticket identifying the second computing device and the health status of the second computing device is created; Health status data is collected from each of a plurality of computing devices in a data center, including the second computing device and the first computing device. A Cox proportional hazards CPH model is created based on health status data collected from each of the plurality of computing devices; A deep neural network (DNN) is applied as input to the CPH model; Determine the failure probability for each of the plurality of computing devices; as well as For each of the plurality of computing devices whose determined failure probability exceeds a predetermined threshold, a prediction support ticket is generated.

16. The storage medium according to claim 15, wherein, The first health status query, the second health status query, and the third health status query are queries of the hash rate of the second computing device, wherein the first repair instruction is to restart the mining application operating on the second computing device, and wherein the second repair instruction is to restart the second computing device.

17. The storage medium according to claim 15, wherein, The first health status query, the second health status query, and the third health status query are queries of the temperature of the second computing device, wherein the first repair instruction is to increase the fan speed of the second computing device, and wherein the second repair instruction is to reduce the operating frequency of the second computing device.

18. The storage medium according to claim 15, wherein, The instruction further causes the computing device to create a support ticket in response to detecting that the number of unhealthy computing devices in the rack exceeds a predetermined threshold.

19. The storage medium according to claim 15, wherein, The instruction further instructs the computing device to: store information about the number of repair instructions dispatched to the second computing device, and avoid submitting any further repair instructions to the second computing device if a predetermined threshold for repair attempts has been exceeded within the repair window.

Images