Automatic repair of computing devices in data center

Abstract

A management device for managing a plurality of computing devices in a data center may include a network interface; a first module that periodically sends a health status query to the computing device via the network interface; a second module configured to receive a response to the health status query and collect and store health status data for the computing device; a third module configured to create a support ticket, and / or a fourth module configured to (i) create and periodically update a Cox Proportional Risk (CPH) model based on health status data; (ii) applying a deep neural network (DNN) to the input of the CPH model; (iii) determining a probability of failure for each computing device; (iv) comparing each fault probability with a threshold value; and (v) causing the third module to generate a pre-failure support ticket for each computing device having the determined probability of failure above the threshold.

Description

[0001] This application is a divisional application of Chinese patent application No. 202080034743.4, entitled "Automatic Repair of Computing Equipment in Data Centers" (filed on July 22, 2020).

[0002] Cross-references to related applications

[0003] This application claims priority to U.S. Patent Application No. 16 / 879,157, filed May 20, 2020, which is a continuation to U.S. Patent Application No. 10,691,528, filed January 29, 2020, Serial No. 16 / 776,213, entitled “Computing System,” filed July 23, 2019, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0004] This disclosure relates generally to the field of computing, and more specifically to systems and methods for managing multiple computing devices in a data center. Background Technology

[0005] The background description is set forth below for the purpose of providing context only. Therefore, no aspect of this background description is intended or implied to be an admission of prior art to this disclosure, to the extent that it is not otherwise considered as such.

[0006] Many cryptocurrencies are based on a technology called blockchain, in which transactions are grouped into blocks. These blocks, along with previous blocks containing earlier transactions, are stored in a ledger (“blockchain”) and made immutable (i.e., practically unmodifiable) by including hash values. Hash values ​​are numbers calculated based on blocks and conform to the standards of a particular blockchain. Once a block and its hash value are confirmed by the cryptocurrency network, they are added to the blockchain. Hash values ​​can be used to verify whether any previous transactions or blocks on the blockchain have been altered or tampered with. This creates an immutable ledger of transactions and allows cryptocurrency networks to prevent someone from attempting to double-spend digital coins.

[0007] Many cryptocurrency networks consist of a large number of participants who repeatedly attempt to be the first to calculate a hash value that meets the requirements of the blockchain network. Depending on the blockchain, these participants may receive rewards (e.g., coin rewards or transaction fee rewards) for being the first to successfully calculate a hash value, and these rewards can incentivize them to continue participating.

[0008] Many blockchain networks need to solve computationally difficult problems as part of hash calculations. Difficult problems require a solution that is data that is difficult (expensive, time-consuming) to generate but easily verifiable by others and meets certain requirements. This is often referred to as "Proof-of-Work." A Proof-of-Work (PoW) system (or protocol or function) is a consensus mechanism. It prevents denial-of-service attacks and other service abuses, such as spam on the network, by requiring service requesters to perform some work (usually meaning computer processing time). For some blockchain networks attempting to compensate for increases in hashing power occurring on the network, the difficulty level may be changed periodically.

[0009] Participants in the network operate standard PCs, servers, or dedicated computing devices. Due to the difficulty involved and the amount of computation required, dedicated computing devices are typically configured with specialized components to increase the speed of performing hashes (the device's hash rate) or other computations required by the blockchain network. Examples of specialized components include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), graphics processing units (GPUs), and accelerated processing units (APUs). Dedicated cryptocurrency mining software (e.g., cgminer) can also be used in conjunction with specialized components, such as software applications configured to compute the SHA-256 algorithm.

[0010] Specialized computing devices often operate for extended periods at high frequencies that generate significant amounts of heat. Even with cooling (e.g., high-speed fans), the heat and continuous operation can negatively impact the reliability and lifespan of components within these devices. For example, ASIC-based specialized computing devices contain a large number of hash chips (e.g., 100), which are more prone to failure as temperatures rise.

[0011] Many participants in blockchain networks simultaneously operate large numbers of (e.g., 1,000, 10,000, 50,000 or more) of different dedicated computing devices (e.g., dedicated computing devices from one manufacturer or different generations from different manufacturers) in large data centers. These data centers and large numbers of dedicated computing devices can be difficult to manage. Data centers accommodating large numbers of dedicated computing devices or other ASIC or GPU-based systems face different challenges than traditional data centers accommodating more general-purpose computers. This is due to significantly higher density, including higher power usage, higher heat generation, and almost continuous compute-intensive operations.

[0012] Continuous operation often leads to performance issues such as memory leaks. Memory leaks can degrade a computer's performance by reducing the amount of available memory. Memory leaks can be a problem when a program runs for a long time and consumes more and more memory over time. Eventually, too much available memory may be allocated, and all or part of the device or system may stop functioning properly. One or more applications running on the device may fail, and the system may slow down due to thrashing. Thrashing refers to the excessive use of a computer's virtual memory resources, leading to a persistent state of paging and page faults, which significantly slows down or inhibits application-level processing.

[0013] In large data centers, a large number of units may fail daily for both known and unknown reasons. A typical data center management solution is to determine when a computing device stops responding to requests (e.g., responding to network pings) and then restart the device (e.g., by going to the device and unplugging it). This is less than ideal because it may require data center technicians to spend a significant amount of time each day performing meticulous and manual restarts of all failed devices. Furthermore, significant losses can occur during the process when a device's performance deteriorates while it is still able to respond to requests.

[0014] For at least these reasons, there is a need for systems and methods that allow for improved management of large numbers of computing devices. Summary of the Invention

[0015] Methods and systems for more easily managing data centers with multiple computing devices are envisioned. Example computing devices include, for example, ASIC computing devices, FPGA computing devices, and GPU computing devices, but other types of computing devices are also possible and anticipated.

[0016] In one embodiment, the method includes issuing automatic (e.g., without human intervention) status queries and repair instructions based on monitored attributes and predefined health attribute levels. A first health status query may be sent to a second computing device. The health status query may include, for example, hash rate or temperature. In response to no acceptable response to the first health status query being received within a first predetermined time, a first repair instruction is sent to the second computing device. After sufficient time has elapsed to complete the first repair instruction, a second health status query may be sent to the second computing device. In response to no acceptable response to the second health status query being received within a second predetermined time, a second repair instruction is sent to the second computing device. After sufficient time has elapsed to complete the second repair instruction, a third health status query may be distributed to the second computing device. In response to no acceptable response to the third health status query being received within an acceptable time period, a repair ticket may be generated.

[0017] In some embodiments, the first repair instruction may include resetting only the mining application running on the computing device, adjusting fan speed, voltage level, and operating frequency, and the second repair instruction may include resetting the entire computing device.

[0018] A system for managing computing devices operating in a data center is also envisioned. In one embodiment, the system may include a network interface for communicating with the managed computing devices and multiple modules configured together to automatically manage the computing devices. Modules may include, for example, a first module that sends status queries to the managed computing devices. An exemplary second module may be configured to receive and process responses to health status queries, and a third module may be configured to create support tickets in response to two or more failed repair attempts. A repair attempt can be determined to have failed when a predetermined amount of time has elapsed without receiving an acceptable response to the health status query.

[0019] The system can be implemented in software as instructions that can be executed by the processor of a computing device, and these instructions can be stored on a non-transitory computer-readable storage medium, such as a flash drive, CD-ROM, DVD-ROM, or hard disk.

[0020] In an embodiment, a management device for managing multiple computing devices in a data center may include: a network interface for communicating with the multiple computing devices; a first module that periodically sends health status queries to each computing device via the network interface; a second module configured to receive responses to the health status queries and collect and store health status data for each computing device; a third module configured to create support tickets; and / or a fourth module. The fourth module may be configured to: (i) create and periodically update a Cox proportional hazards (CPH) model based on the collected health status data; (ii) apply a deep neural network (DNN) to the input of the CPH model; (iii) determine a failure probability for each computing device; (iv) compare each determined failure probability to a predetermined threshold; and / or (v) cause the third module to generate a pre-failure support ticket for each computing device with a determined failure probability exceeding the predetermined threshold.

[0021] The foregoing and other aspects, features, details, utility and / or advantages of the embodiments of this disclosure will become apparent from reading the following description and from viewing the accompanying drawings. Attached Figure Description

[0022] Figure 1 This is a top view of an example of a data center used for computing devices.

[0023] Figure 2 This is a front view of an example pod in a data center used for computing devices.

[0024] Figure 3 This is an illustration of an example of a rack used for computing equipment in a data center.

[0025] Figure 4 This is an illustration of an example computing device.

[0026] Figure 5 This is a flowchart illustrating an example method for managing computing devices in a data center in accordance with the teachings of this disclosure.

[0027] Figure 6 This is an illustration of an example system for managing computing devices in a data center, based on the teachings of this disclosure.

[0028] Figure 7 This is a flowchart illustrating another example method for managing computing devices in a data center. Detailed Implementation

[0029] Embodiments of this disclosure will now be described in detail, with examples thereof described herein and illustrated in the accompanying drawings. While this disclosure will be described in conjunction with embodiments and / or examples, it is to be understood that they are not intended to limit this disclosure to those embodiments and / or examples. Rather, this disclosure covers alternatives, modifications, and equivalents.

[0030] Various embodiments for various apparatuses, systems, and / or methods are described herein. Numerous specific details are set forth to provide a thorough understanding of the overall structure, function, manufacture, and use of the embodiments described in the specification and illustrated in the accompanying drawings. However, those skilled in the art will understand that the embodiments can be practiced without such specific details. In other instances, well-known operations, components, and elements have not been described in detail so as not to obscure the embodiments described in the specification. Those skilled in the art will understand that the embodiments described and illustrated herein are non-limiting examples, and therefore it will be understood that the specific structural and functional details disclosed herein may be representative and not necessarily limit the scope of the embodiments.

[0031] Now for reference Figure 1The image shows a top view of an example data center 100 used for computing equipment. Data center 100 is configured with a large number of pods 110. Pods are standardized blocks of racks arranged in rows or (more typically) in pairs, sharing common infrastructure components such as power distribution units, network routers / switches, sealing systems, and air handling units. For example, a pod might have two parallel equipment racks spaced apart, each facing outwards. Equipment on the racks can be oriented to draw in cool air from outside the pod and exhaust hot air (heated by the computing equipment) into an empty space at the center of the pod, where the hot air then rises and is expelled from the data center. For example, a hot air duct might be present in the middle of each pod to collect the hot exhaust gases, and the pod then exhausts the hot exhaust gases from the data center via vents in the data center roof.

[0032] Turn now Figure 2 The front side of a sample compartment 110 is shown. Compartment 110 has multiple racks 210, each rack 210 having multiple shelves 230 for holding computing devices. For organizational and management purposes, the shelves can be grouped together in a switching section 220, each shelf in the switching section 220 being supported by the same network switch. Multiple bin locations 240 can exist within each rack in the shelf 230, each bin location 240 holding a single computing device. Each computing device can be installed in a bin that has power and network connections.

[0033] Turn now Figure 3 The image shows a more detailed front view of a rack 230 within example rack 210. In this example, computing devices 310 are mounted in each case 240 within rack 230. In this example, computing device 310 is an ASIC computing device. ASIC computing devices typically include a controller board with a network port 320, one or more status indicator LEDs 330, and a pair of cooling fans (front fan 340 shown) that draw air through the center of the computing device. Multiple hash boards are present within the computing device that perform computations and generate heat.

[0034] Turn now Figure 4 The illustration shows an example embodiment of computing device 310. In this example, computing device 310 includes a controller 360 that oversees the operation of multiple hashboards 350 within the device. Controller 360 also includes a network port 320 for communication, a reset button 370 for resetting computing device 310, and one or more indicator LEDs 330. Two fans 340 (a front fan and a rear fan) draw cool air into computing device 310 and exhaust waste heat through the rear of computing device 310. Other types of computing devices are possible and anticipated.

[0035] As mentioned above, one problem faced by operators of large data centers is identifying and servicing computing devices that are not operating optimally. For a variety of reasons, including the time required for a device to degrade from full operational status to an unresponsive state, waiting for a device to fail and then manually restarting it or resetting it using a reset button is undesirable. For at least this reason, there is a need for improved systems and methods for managing large numbers of computing devices.

[0036] Turn now Figure 5 The diagram illustrates a flowchart of an example embodiment of an improved method for managing computing devices in a data center, in accordance with the teachings of this disclosure. A request for status information from one or more computing devices is made (step 500). This request may be sent from a management server to one or more computing devices 310 via a network connection (e.g., wired or wireless Ethernet). If the response is acceptable (step 510), another periodic request may be sent after a predetermined polling interval has elapsed (step 520). For example, the status request may be sent every 1, 5, 6, 10, or 30 minutes. The status request may request data about different operating parameters of the computing device, such as hash rate, temperature, fan speed, or the number of hardware errors. An example request may query the temperature of a particular computing device 310, and an acceptable response may be one indicating that the operating temperature is below the operating temperature specified by the manufacturer for that particular device type and model.

[0037] Some requests may be sent to the data provider rather than the monitored device. For example, in some embodiments, a hash rate request may be sent to the monitored device, but in other embodiments, the request may be sent to a database that stores information from the pool that the computing device is currently processing. For example, the database may be a local copy of the data provided by the pool. Hash rates are typically calculated in hashes per second, such as 3 PH / s (3 × 10⁻⁶ PH / s). 15 13 TH / s (13 x 10^6 hashes) 12 (hashes) or 90 GH / s (90 x 10^6 GHz per second) 9 (Hash rate), and may be provided periodically by the pool. Some pools track hash rates on a worker basis. This means that if more than one compute device is connected to the pool as a single worker, the reported hash rate information may batch-cover all of these compute devices. Some pools allow the use of a worker name specific to the compute device, which allows the pool to track the hash rate and report the hash rate separately for each compute device.

[0038] If the response does not indicate an acceptable state for the computing device (step 510), a first repair instruction is sent to the computing device (step 530). An example of a first repair instruction is to restart an application running on the computing device. For example, a mining application running on the computing device could be restarted. This is different from restarting the entire computing device. Another example of a repair instruction includes an instruction to increase fan speed or decrease operating voltage in response to a status response indicating excessively high temperature. Depending on the managed computing device, the repair instruction may also include running a computer program on the computing device.

[0039] After the first repair instruction has been sent, a waiting period occurs (step 540) to allow the target computing device to complete the execution of the first repair instruction. For example, a waiting period of five, ten, or fifteen minutes can be used to give the target computing device sufficient time to complete the execution (or repeat) of the first repair instruction. Then, another status request is sent to the computing device (step 550). If the response is acceptable (e.g., within the manufacturer's temperature or hash rate specifications) (step 560), the system waits until the next polling cycle (step 520) before making another status request (step 500). If the response is unacceptable (step 560), a second repair instruction is sent (step 570). An example of a second repair instruction is a full device reset instruction. Another example of a second repair instruction is an instruction to reduce the operating frequency in response to receiving a status response indicating that the temperature is too high.

[0040] After the second repair instruction has been sent, a waiting period occurs (step 580) to allow the target computing device to complete the execution of the second repair instruction. Then, another status request is sent to the computing device (step 590). If the response is acceptable (step 592) (e.g., within the manufacturer's temperature or hash rate specifications), the system waits until the next polling cycle (step 520) before making another status request (step 500). An acceptable response (steps 510, 560, 592) can also remove the computing device from the list of any existing faulty devices. If the response is unacceptable (step 592), a support ticket is generated (step 594). The support ticket can include not only the identifier of the computing device but also the history of the repair instructions executed and the resulting health status report. Advantageously, this can save support personnel time from manually executing repair instructions.

[0041] In some embodiments, a health status query may include running diagnostic instructions or a set of instructions. In other embodiments, a health status query may be a query on a database that stores status information collected periodically by the computing device (e.g., received from a pool that the computing device is processing).

[0042] Turn now Figure 6 This illustration shows an example embodiment of a system for managing computing devices in a data center. In this embodiment, the system includes a large number of computing devices 310. The computing devices 310 communicate with a management server 600 via a network interface 640 of a server. While wireless networks are possible, the current density of computing devices in a data center means that it is currently preferred to use a wired network, such as wired Ethernet, for communication between the management server 600 and the computing devices 310. In some embodiments, the computing devices 310 may include a controller 360 and a network interface for communicating with the management server 600 via the network 610. The controller 360 may be configured to send computing tasks to one or more computing or hash boards 350, each computing or hash board having multiple GPUs or ASIC chips 390 that can operate at frequencies specified by the controller. The computing devices 310 may further include multiple cooling fans 340 and a power supply 380. The voltage output to the ASIC chips 390 may vary based on settings configured by the controller 360. Higher voltage and frequency levels for the ASIC chips 390 will improve performance, but they may also increase heat and negatively impact lifespan.

[0043] Management server 600 can be a traditional PC or server, or a dedicated device. Management server 600 may be configured with one or more processors 620, volatile and non-volatile memory, such as flash memory or internal or external hard drives (e.g., network-attached storage accessible to server 600). Management server 600 is configured to execute management application 630 to assist users (e.g., data center technicians) in managing computing device 310. Management server 600 may be located within the same data center or facility as computing device 310, or it may be located remotely and connected to computing device 310 via the Internet.

[0044] The management application 630 is preferably implemented in software (e.g., instructions stored on a non-volatile storage medium such as a hard disk, flash drive, or DVD-ROM), but a hardware implementation is possible. The management application 630 may include multiple modules, such as: a user interface module 632 that displays data to and receives input from the user (e.g., via an external display or via the user's web browser); a query sending module 634 that sends status queries to obtain status data of the computing device 310 (e.g., periodically polling the health status of each device); a query response processing and repair module 638 that receives and processes status query responses and issues repair instructions when necessary; and a support ticket creation module 636 that creates a support ticket if the repair instructions fail to move the computing device to an acceptable state. If resolved, module 636 may generate a "resolved" ticket or update an existing ticket with detailed records of the computing device's history in the ticketing system. Modules 636 and 638 may also include intelligence (e.g., rules) to prevent the management application from getting stuck in a loop due to recurring problems with a device or a group of devices. For example, after a support ticket is sent, the ticket creation module 636 can be configured not to send subsequent repair instructions or create additional tickets until a technician indicates that the device has been repaired or the device correctly responds to a health status query (indicating that the device has been repaired).

[0045] In some embodiments, the user interface module 632 may provide a user interface to configure rules (or override predefined rules) for when and which repair instructions are sent. The management application 630 may be configured to automatically execute this step if conditions of the rules (e.g., symptoms or leading indicators of a problem) are determined to be met. In some embodiments, the computing system may be configured to learn from past data on the activity and / or profiles of a second computing device and take corrective / proactive actions, such as based on leading indicators, before a problem occurs. For example, in some learning-based embodiments, the management application 630 may be configured to initiate repairs based on leading indicators (e.g., detected hash rate decline) before a previously encountered problem (e.g., device hang) recurs.

[0046] In one embodiment, management application 630 may be configured to monitor for anomalies in key health metrics (e.g., hash rate, temperature) and, when a problem is identified, to identify the scale of the problem and escalate it appropriately, including notifying the appropriate individuals when an escalation beyond automatically issuing a repair instruction (self-healing) is required. For example, if a single device reports an elevated temperature, a first (and second, if necessary) repair instruction may be dispatched. However, if multiple devices in the same rack (e.g., more than 5%) begin to experience elevated temperatures, management application 630 may be configured to (1) turn on or increase the active cooling rate (e.g., evaporative coolers or air conditioners) for that area of ​​the data center, or (2) create a high-priority ticket for the rack, as multiple high-temperature health status reports may indicate a broader airflow problem in that part of the data center, rather than just a problem with a single computing device.

[0047] In another embodiment, management application 630 may be configured to apply artificial intelligence (AI) and machine learning (ML) to predict and create remedial mechanisms (e.g., repair instructions) before anomalies occur or reach a critical impact in computing devices. For example, in one embodiment, management application 630 may be configured to track the status report history of computing device 310 in data storage 650. When a problem pattern is detected (e.g., a series of multiple unacceptable health status responses within a predetermined time period), ticket creation module 636 may create a ticket even if the repair instruction is successful. These "repeated" devices may be on the verge of more catastrophic failure and may benefit from technicians inspecting and replacing worn components (such as fans or power supplies). Management application 630 may be further configured to provide a user with a list of repeated devices via user interface module 632.

[0048] In some embodiments, management application 630 can be configured to avoid generating multiple tickets when multiple machines are down in the same region due to the same issue. Regions can be defined by aggregation. Levels may increase the number of second compute devices affected in a defined region, from a single second compute device up to and including multiple second compute devices across multiple data centers. For example, defined levels may include: overall (all sites), site, cabin, rack, switch, and individual compute device. For example, if the "machine uptime" metric is healthy at the site, cabin, and rack levels but unhealthy at the switch level, a ticket may be generated at the switch level. If management application 630 detects health status numbers that are outside the acceptable range for a particular aggregation level, the application can generate a ticket along with remediation instructions to attempt to resolve the issue automatically. When multiple compute devices are down due to the same issue, such as in the case of a dead switch cutting off network communication with compute devices from multiple companies, a ticket may be generated for each customer.

[0049] In some embodiments, to help identify when computing device 310 may enter a fault state and what other indicators may be associated with the fault state, management application 630 may include artificial intelligence (AI) and machine learning (ML) module 654 to predict anomalies in computing device 310 and create remedial mechanisms (e.g., support tickets) before the anomaly occurs or reaches a critical impact. In some embodiments, AI / ML module 654 may be configured to use a Cox proportional hazards (CPH) model to predict the probability of a fault event in computing device 310 based on historical telemetry data (stored in data storage 650) and optional climate data (e.g., temperature and humidity readings in a data center).

[0050] CPH models are typically used in clinical settings to determine how multivariate factors can affect patient survival. The benefit of CPH models is their ability to simultaneously assess the impact of several factors on patient survival. Computational devices 310 that output a failure probability above a predetermined threshold (e.g., 80% probability) can generate support tickets, allowing data center personnel to inspect them and replace parts as needed. For AI and ML, models are built and trained using data from data storage 650.

[0051] In some embodiments, the initial model for CPH can be built using open-source packages such as Themeaux's Survival in R or Pilon's Lifelines in Python. The model's performance can then be tested to confirm its satisfaction based on a concordance index, a measure of the predictions made by the evaluation algorithm and used to score the survival model. The score is calculated as the proportion of concordant pairs divided by the total number of possible evaluation pairs.

[0052] The high-scoring model can then be enhanced by connecting a deep neural network (DNN) to the input of the CPH model. This method uses the input-output relationship associated with a simple feedforward neural network to model the examined survival data as the basis for a non-linear proportional hazards model. In one embodiment, this is the Faraggi and Simon approach using a neural network for regression. For example, the network can be configured with a single output node and a single-layer perceptron with inputs, hidden nodes, bias nodes, and outputs. A sigmoid non-linearity can be used between the inputs and hidden nodes. Advantageously, these DNN implementations of CPH may outperform traditional CPH and are more scalable due to the ability to adjust or arbitrarily choose the neural network architecture.

[0053] In some embodiments, AI module 654 may implement the Efron method for calculating probabilities, as this method has been shown to outperform other methods, being faster than exact calculation methods and tending to produce estimates that are closer to the truth than other methods. The implementation of the Efron method exists in TensorFlow 2.0, making it more reusable than previous CPH DNN work.

[0054] Turn now Figure 7 The diagram illustrates details of an embodiment of the operation of the AI / ML module 654. In this embodiment, data from multiple computing devices is collected and stored (step 700). A CPH model is created / updated based on the collected data (step 710). A DNN is applied to the input of the CPH module (step 720). The failure probability of each computing device is determined based on the model (step 730). If the failure probability is greater than a predetermined threshold (step 740) and no support ticket has been generated (step 760), a ticket is generated (step 770). If the failure probability is lower than the predetermined threshold (step 740), the process waits until the polling interval before repeating (step 750).

[0055] Throughout this specification, references to "various embodiments," "adopting embodiments," "in embodiments," or "embodiments," etc., indicate that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment. Therefore, phrases such as "in various embodiments," "adopting embodiments," "in embodiments," or "embodiments," appearing throughout this specification, do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Thus, a particular feature, structure, or characteristic illustrated or described in connection with one embodiment / example can be combined, in whole or in part, with features, structures, functions, and / or characteristics of one or more other embodiments / examples without limitation, as such combinations are not illogical or nonfunctional. Moreover, numerous modifications can be made to adapt particular situations or materials to the teachings of this disclosure without departing from the scope of this disclosure.

[0056] It should be understood that references to a single element are not necessarily so limited and may include one or more of such elements. Any directional references (e.g., addition, subtraction, up, down, upward, downward, left, right, left to right, top, bottom, above, below, vertical, horizontal, clockwise, and counterclockwise) are used only for identification purposes to aid the reader's understanding of this disclosure and do not impose limitations, particularly regarding the location, orientation, or use of embodiments.

[0057] Engagement references (e.g., attachment, coupling, connection, etc.) should be interpreted broadly and may include intermediate components between relative movements between elements and connections between elements. Therefore, an engagement reference does not necessarily imply that two elements are directly connected / coupled and in a fixed relationship with each other. The use of "for example" in the specification should be interpreted broadly and is intended to provide non-limiting examples of embodiments of this disclosure, and this disclosure is not limited to such examples. The use of "and" and "or" should be interpreted broadly (e.g., considered as "and / or"). For example, but not limited to, the use of "and" does not necessarily require all listed elements or features, and the use of "or" is inclusive unless the construction is illogical.

[0058] While a process, system, and method may be described herein by combining one or more steps in a particular order, it should be understood that the method may also be practiced with steps in a different order, performing certain steps, performing additional steps, and / or omitting certain steps described.

[0059] All content contained in the above description or shown in the accompanying drawings should be interpreted as illustrative rather than restrictive. Changes in detail or structure may be made without departing from this disclosure.

[0060] It should be understood that the computer, system, and / or processor described herein may include conventional processing means known in the art capable of executing pre-programmed instructions stored in associated memory, all of which perform the functions described herein. To the extent that the methods described herein are embodied in software, the resulting software may be stored in associated memory and may also constitute components for performing the methods. The system or processor may further be of a type having ROM, RAM, RAM and ROM, and / or a combination of non-volatile and volatile memory, making it possible to store any software and also allowing the storage and processing of dynamically generated data and / or signals.

[0061] It should be further understood that the article of manufacture according to this disclosure may include a non-transitory computer-readable storage medium having a computer program encoded thereon for implementing the logic and other functions described herein. The computer program may include code for performing one or more methods disclosed herein. This embodiment may be configured to execute via one or more processors, such as multiple processors integrated into a single system or distributed across and connected together on a communication network, and the communication network may be wired and / or wireless. Code for implementing one or more features described in conjunction with one or more embodiments may, when executed by a processor, cause multiple transistors to change from a first state to a second state. Specific change patterns (e.g., which transistors change state while which do not) may be specified at least in part by logic and / or code.

Claims

1. A management device for managing multiple computing devices in a data center, wherein, The management device includes: A network interface for communicating with the plurality of computing devices. The first module periodically sends health status queries to each computing device in the computing device via the network interface. The second module is configured to receive responses to the health status query and to collect and store health status data for each computing device in the computing devices. The third module is configured to create support tickets, and The fourth module is configured as follows: (i) Based on the collected health status data, create and periodically update the Cox proportional hazards CPH model; (ii) Applying a deep neural network (DNN) to the input of the CPH model; (iii) Determine the failure probability for each of the plurality of computing devices; (iv) Compare each determined failure probability with a predetermined threshold; and (v) The third module generates a pre-failure support ticket for each of the plurality of computing devices having a determined failure probability exceeding the predetermined threshold.

2. The management device according to claim 1, wherein, If no acceptable response to a query for a specific health status is received within a first predetermined time, the third module is configured to: (i) Send the first repair command. (ii) Wait for at least sufficient time for the first repair instruction to complete. (iii) Send a second health status query, and (iv) In response to no acceptable response to the second health status query received within a second predetermined time: (a) Send a second repair command. (b) Wait for at least sufficient time for the second repair instruction to complete. (c) Send a third health status query, and (d) In response to the absence of an acceptable response to the third health status query within a third predetermined time, the third module creates a repair ticket.

3. The management device according to claim 1, wherein, The health status data includes the hash rate.

4. The management device according to claim 1, wherein, The health status data includes the temperature of the calculation device.

5. The management device according to claim 1, wherein, The second module is further configured to collect temperature and humidity data from the data center.

6. A method for managing multiple computing devices in a data center, the method comprising: Health status data is collected from each of the plurality of computing devices by periodically sending health status queries to each of the plurality of computing devices; Based on the collected health status data, a Cox proportional hazards CPH model is created and updated regularly. A deep neural network (DNN) is applied as input to the CPH model; Determine the failure probability for each of the plurality of computing devices; Each determined failure probability is compared with a predetermined threshold; as well as A support ticket is generated for each of the plurality of computing devices that has a determined failure probability exceeding the predetermined threshold.

7. The method according to claim 6, wherein, The collected health status data includes telemetry data.

8. The method according to claim 7, wherein, The collected health status data includes temperature and humidity readings from within the data center.

9. The method according to claim 6, wherein, The collected health status data includes the device fan speed.

10. The method according to claim 6, wherein, The collected health status data includes the device hash rate.

11. The method according to claim 6, wherein, The collected health status data includes device temperature.

12. The method according to claim 6, wherein, The predetermined threshold is 80%.

13. The method of claim 6, further comprising: If no acceptable response to the first health status query is received within a first predetermined time: (i) Issue the first repair command. (ii) Wait for at least sufficient time for the first repair instruction to complete. (iii) Issue a second health status query, and (iv) In response to no acceptable response to the second health status query received within a second predetermined time: (a) Issue a second repair instruction. (b) Wait for at least sufficient time for the second repair instruction to complete. (c) Issue a third health status query, and (d) If no acceptable response to the query for the third health status is received within a third predetermined time, a repair ticket is issued.

14. A non-transitory computer-readable storage medium storing instructions executable by a processor of a computing device, the instructions causing the computing device, when executed, to: Health status data is collected from each of the multiple computing devices by periodically sending health status queries to each of the multiple computing devices in the data center; Based on the collected health status data, a Cox proportional hazards CPH model is created and updated regularly. A deep neural network (DNN) is applied as input to the CPH model; Determine the failure probability of each of the plurality of computing devices; Each determined failure probability is compared with a predetermined threshold; as well as For each of the plurality of computing devices having a determined failure probability exceeding the predetermined threshold, a predictive support ticket is generated.

15. The non-transitory computer-readable storage medium according to claim 14, wherein, If no acceptable response to the first health status query is received within the first predetermined time: (i) Issue the first repair command. (ii) Wait for at least sufficient time for the first repair instruction to complete. (iii) Issue a second health status query, and (iv) In response to the failure to receive an acceptable response to the second health status query within a second predetermined time: (a) Issue a second repair instruction. (b) Wait for at least sufficient time for the second repair instruction to complete. (c) Issue a third health status query, and (d) If no acceptable response to the query for the third health status is received within a third predetermined time, a repair ticket is issued.

16. The non-transitory computer-readable storage medium according to claim 15, wherein, The first health status query, the second health status query, and the third health status query are queries of hash rate, wherein the first repair instruction is to restart the mining application, and wherein the second repair instruction is a device restart instruction.

17. The non-transitory computer-readable storage medium according to claim 14, wherein, The health status data includes the device fan speed.

18. The non-transitory computer-readable storage medium according to claim 14, wherein, The health status data includes the device hash rate.

19. The non-transitory computer-readable storage medium according to claim 14, wherein, The health status data includes the device temperature.

20. The non-transitory computer-readable storage medium according to claim 19, wherein, The health status data further includes temperature and humidity readings from within the data center.

Images