Fault processing method and device for heterogeneous accelerator card, electronic equipment and storage medium

Abstract

The invention discloses a fault processing method and device for a heterogeneous accelerator card, electronic equipment and a storage medium, and relates to the technical field of computers, and the method comprises the steps: if a fault heterogeneous accelerator card power-off instruction is received, the fault heterogeneous accelerator card power-off instruction comprises a to-be-removed heterogeneous accelerator card, when the to-be-removed heterogeneous accelerator card is in a distributed state, the to-be-removed heterogeneous accelerator card is in the distributed state; removing the to-be-removed heterogeneous acceleration card, and determining an undistributed target heterogeneous acceleration card from at least one heterogeneous acceleration card device of the fusion architecture whole cabinet; and replacing the to-be-removed heterogeneous accelerator card with the target heterogeneous accelerator card, and allocating the target heterogeneous accelerator card to the computing resource pool corresponding to the to-be-removed heterogeneous accelerator card. The technical problem that according to an existing heterogeneous accelerator card fault processing method, the whole heterogeneous accelerator card device needs to be powered off, and service operation of all computing resource pools in a whole machine cabinet is affected is solved, and the technical effect that the fault accelerator card can be replaced under the condition that the influence on the service function of the whole system can be reduced is achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a fault handling method, device, electronic device and storage medium for a heterogeneous accelerator card. Background Art

[0002] Traditional computing-centric architectures can no longer meet the requirements of high performance, high flexibility, and high scalability for emerging application scenarios such as artificial intelligence, machine learning, and intelligent computing. Therefore, data center architecture is undergoing a profound transformation, from computing-centric to data-centric converged architecture.

[0003] In the design of converged architecture, resource decoupling has become a key design principle. By decomposing the entire system into multiple independent pooled modules, such as computing resource pools, heterogeneous acceleration card resource pools, and IO resource pools, flexible configuration and efficient use of resources can be achieved. These pooled modules usually exist in the form of boxes, making system expansion and upgrades more convenient.

[0004] In a converged architecture cabinet design for IO expansion capabilities, the system is cleverly divided into an IOBox and multiple device boxes. As a centralized management point for IO resources, the IO Box is responsible for handling all IO-related operations, such as data transmission, device identification, and resource allocation. The device box includes multiple host boxes and heterogeneous acceleration card boxes, which are used to provide computing resources and heterogeneous acceleration capabilities respectively.

[0005] However, although this converged architecture performs well in resource allocation and scalability, it still faces some challenges in the maintenance and management of heterogeneous accelerator card resources. In particular, when a heterogeneous accelerator card fails and needs to be replaced, how to quickly repair and replace the faulty device without affecting the business functions of the entire system has become an urgent problem to be solved. Summary of the invention

[0006] The present invention provides a fault handling method, device, electronic device and storage medium for a heterogeneous acceleration card, so as to at least solve the technical problem that when a heterogeneous acceleration card device in a whole cabinet of a converged architecture fails, the handling method of the prior art needs to power off the entire heterogeneous acceleration card device, affecting the business operation of all computing resource pools in the whole cabinet, and bringing a large business impact to users.

[0007] The present invention provides a method for handling a fault of a heterogeneous acceleration card. The method comprises: judging whether a power-off instruction of a faulty heterogeneous acceleration card is received, wherein the power-off instruction of the faulty heterogeneous acceleration card includes a heterogeneous acceleration card to be removed; if the power-off instruction of the faulty heterogeneous acceleration card is received, then, if the heterogeneous acceleration card to be removed is in an allocated state, removing the heterogeneous acceleration card to be removed, and determining a target heterogeneous acceleration card in an unallocated state from at least one heterogeneous acceleration card device of a converged architecture whole cabinet; replacing the heterogeneous acceleration card to be removed with the target heterogeneous acceleration card, and allocating the target heterogeneous acceleration card to a computing resource pool corresponding to the heterogeneous acceleration card to be removed.

[0008] The present invention also provides a fault handling device for a heterogeneous acceleration card, comprising: a judgment module, used to judge whether a power-off instruction for a faulty heterogeneous acceleration card is received, wherein the power-off instruction for the faulty heterogeneous acceleration card includes a heterogeneous acceleration card to be removed; a fault handling module, used to remove the heterogeneous acceleration card to be removed if the power-off instruction for the faulty heterogeneous acceleration card is received, and determine a target heterogeneous acceleration card in an unallocated state from at least one heterogeneous acceleration card device of a converged architecture whole cabinet; a replacement module, used to replace the heterogeneous acceleration card to be removed with the target heterogeneous acceleration card, and allocate the target heterogeneous acceleration card to a computing resource pool corresponding to the heterogeneous acceleration card to be removed.

[0009] The present invention also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned methods for handling a fault of a heterogeneous acceleration card when executing the computer program.

[0010] The present invention also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned methods for handling faults of heterogeneous acceleration cards are implemented.

[0011] Through the present invention, it is determined whether a power-off instruction of a faulty heterogeneous acceleration card is received, wherein the power-off instruction of the faulty heterogeneous acceleration card includes the heterogeneous acceleration card to be removed. If a power-off instruction of a faulty heterogeneous acceleration card is received, the heterogeneous acceleration card to be removed is removed when the heterogeneous acceleration card to be removed is in an allocated state, and a target heterogeneous acceleration card in an unallocated state is determined from at least one heterogeneous acceleration card device of the whole cabinet of the converged architecture, and the heterogeneous acceleration card to be removed is replaced with the target heterogeneous acceleration card, and the target heterogeneous acceleration card is allocated to the computing resource pool corresponding to the heterogeneous acceleration card to be removed. Therefore, the technical problem that when a heterogeneous acceleration card device in the whole cabinet of the converged architecture fails, the processing method of the prior art needs to power off the entire heterogeneous acceleration card device, which affects the business operation of all computing resource pools in the whole cabinet and brings a greater business impact to users is solved, and the technical effect of completing the replacement operation of the faulty acceleration card while minimizing the impact on the business functions of the entire system, and promptly restoring the normal operation of the business, and reducing the maintenance pressure of the operation and maintenance personnel is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0013] Figure 1 A schematic diagram of a process flow of a method for troubleshooting a heterogeneous accelerator card provided by an embodiment of the present invention; Figure 2 A schematic diagram of the hardware architecture of a converged architecture system according to an embodiment of the present invention; Figure 3 A schematic diagram of a power supply structure of a heterogeneous acceleration card according to an embodiment of the present invention; Figure 4 A schematic diagram of a process flow of a method for handling a fault of a heterogeneous accelerator card according to an embodiment of the present invention; Figure 5 A schematic diagram of a fault handling device for a heterogeneous accelerator card provided by an embodiment of the present invention; Figure 6 FIG. 4 is a schematic diagram of a structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] It should be noted that, in the description of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0016] Before specifically introducing the fault handling method of the heterogeneous accelerator card of the present invention, the prior art handling method when the heterogeneous accelerator card fails and needs to be replaced is briefly introduced, which mainly manages the resource allocation and fault record of the heterogeneous accelerator card through the pooled system management controller (PSMC) in the IO Box. When a heterogeneous accelerator card fails, the PSMC can determine the location of the faulty device by querying the allocation relationship table according to the fault information reported by the Host, and record the fault log.

[0017] However, this solution has obvious limitations in the process of replacing faulty equipment. Since the entire heterogeneous acceleration card Box needs to be powered off before it can be replaced, this will cause the business operation of all Host resource pools in the entire cabinet to be interrupted, which will have a significant business impact on users.

[0018] Therefore, in order to solve this problem, it is necessary to develop a more efficient and reliable heterogeneous accelerator card resource maintenance and management solution to achieve rapid location, repair and replacement of faulty equipment while ensuring that the business continuity of the entire system is not affected.

[0019] Specifically, an embodiment of the present invention provides a method for handling a fault of a heterogeneous accelerator card, such as Figure 1 shown.

[0020] The embodiment of the present invention is a fusion architecture system for improving heterogeneous computing capabilities. The structure of the fusion architecture system is as follows: Figure 2As shown in the figure, the whole cabinet of the converged architecture has one IO resource pool, two heterogeneous acceleration card devices (i.e., heterogeneous acceleration card BOX0 and heterogeneous acceleration card BOX1), and a Host resource pool (including Host Box0, Host Box1, Host Box2, Host Box3, Host Box4, Host Box5, Host Box6, Host Box7, and Host Box8). Each Box is designed with a BMC (Baseboard Management Controller) module, which is responsible for the management of the corresponding Box. All BMCs are connected to the same network switch to form an internal LAN. The BMC in the IO Box serves as a pooled management controller PSMC, which can communicate with the BMC of each device Box through the network to realize the management and control functions of the entire system.

[0021] The IO resource pool contains 4 layers of PCIe Switch boards, each of which has two PCIe Switch chips, which expand, network, and allocate the PCIe resources of multiple CPUs in the HostBox. The PCIe Switch chip is an IO chip used to expand a set of PCIe signals into multiple sets of PCIe signals, and is used to expand CPU IO resources. Each PCIe Switch chip leads to 5 CDFP ports (high-speed signal ports), and the entire IO resource pool can eventually provide 40 (8*5) CDFP ports to the outside. Each device Box is also equipped with a corresponding CDFP port, and the device Box is connected to the IO resource pool through a cable.

[0022] The PSMC interacts with the BMC and PCIe Switch firmware of each device Box, summarizes each mapping relationship, and establishes an allocation relationship table between the heterogeneous accelerator card BOX and the Host Box. Through the allocation relationship table, it can be clearly displayed which heterogeneous accelerator cards in which heterogeneous accelerator card boxes are allocated under a Host resource pool. The allocation relationship of the embodiment of the present invention is as follows: { "HostBox_ID": "1", "Switch_CDFP_ID": "CDFP0-1", "Switch_Port": "0x000040", "Device": [ { "AcceleratorCardBox_ID": "0", "Slot": "AcceleratorCard0", "Switch_CDFP_ID": "CDFP1-1", "Switch_Port": "1P96", "Box_CDFP_ID": "CDFP0", "BusNumber": XX, "DeviceNumber": XX, "FunctionNumber": XX, "AssetInfo": { "BiosVersion": "XX", "BuildDate": "XX", "DeviceID": "XX", "FwVersion": "XX", "Manufacturer": "XX", "Model": "XX", "PartNumber": "XX", "SerialNumber": "XX", "SubSystemID": "XX", "UUID": "XX", "vendorID": "XX" } } ] } PSMC provides web pages and Redfish interfaces for allocating and releasing heterogeneous accelerator card resources. PSMC then updates the allocation relationship table between heterogeneous accelerator cards and Host BOX in the heterogeneous accelerator card device to ensure the accuracy of the allocation relationship.

[0023] In step S101, it is determined whether a power-off instruction of a faulty heterogeneous accelerator card is received, wherein the power-off instruction of the faulty heterogeneous accelerator card includes a heterogeneous accelerator card to be removed.

[0024] Optionally, in some embodiments, the faulty heterogeneous acceleration card power-off instruction further includes a first serial number of the heterogeneous acceleration card to be removed and a second serial number of a heterogeneous acceleration card device to which the heterogeneous acceleration card to be removed belongs.

[0025] The system first detects whether there is a power-off command input for the faulty heterogeneous accelerator card. The operation and maintenance personnel send the power-off command for the faulty heterogeneous accelerator card to the PSMC through the IPMI (Intelligent Platform Management Interface) command, Redfish interface or web page.

[0026] Specifically, when the system detects that a heterogeneous accelerator card has failed, the operation and maintenance personnel confirm that the heterogeneous accelerator card is to be removed and issue a power-off command for the failed heterogeneous accelerator card. After receiving the power-off command for the failed heterogeneous accelerator card, the converged architecture cabinet will parse it and extract key information, including the heterogeneous accelerator card to be removed, the first number of the heterogeneous accelerator card to be removed, and the second number of the heterogeneous accelerator card device to which it belongs.

[0027] Among them, the first number and the second number can be obtained by checking the fault log recorded by PSMC or other methods. The first number and the second number can be obtained by checking the fault log recorded by PSMC or other methods. The first number of the heterogeneous acceleration card to be removed and the second number of the heterogeneous acceleration card device to which it belongs can be obtained by: when the basic input and output system BIOS detects a PCIe fault in the heterogeneous acceleration card, it will report to PSMC to record the fault log. After the BMC of the heterogeneous acceleration card Box detects the fault of the heterogeneous acceleration card in an out-of-band manner, it will report to PSMC to record the fault log. Or the first number and the second number can be obtained by: when the host operating system Host OS detects the fault of the heterogeneous acceleration card in-band, etc., the first number of the heterogeneous acceleration card to be removed and the second number of the heterogeneous acceleration card device to which it belongs can be obtained.

[0028] Through the above technical solution, the first number of the heterogeneous accelerator card to be removed is determined, and the system can accurately identify and locate the heterogeneous accelerator card to be removed that needs to be powered off, avoiding the risk of other normally working accelerator cards being accidentally powered off due to misoperation. During the troubleshooting and recovery process, detailed power-off instruction information can help technicians locate the faulty heterogeneous accelerator card more quickly and take appropriate measures to repair it.

[0029] In step S102, if a power-off instruction of a faulty heterogeneous acceleration card is received, if the heterogeneous acceleration card to be removed is in an allocated state, the heterogeneous acceleration card to be removed is removed, and a target heterogeneous acceleration card in an unallocated state is determined from at least one heterogeneous acceleration card device in the entire cabinet of the converged architecture.

[0030] It should be understood that the converged architecture system extracts information about the heterogeneous acceleration card to be removed based on the received power-off instruction of the faulty heterogeneous acceleration card, and checks the status of the heterogeneous acceleration card to be removed.

[0031] If the heterogeneous accelerator card to be removed is in the allocated state, the converged architecture cabinet removes the heterogeneous accelerator card from its currently allocated computing resource pool (the computing resource pool is Figure 2 The host resource pool in the .

[0032] After the heterogeneous acceleration card to be removed is successfully powered off and removed, the converged architecture whole cabinet determines a target heterogeneous acceleration card in an unassigned state from at least one heterogeneous acceleration card device to replace the heterogeneous acceleration card to be removed.

[0033] Optionally, in some embodiments, after receiving the power-off instruction of the faulty heterogeneous acceleration card, it also includes: based on the first number and the second number, determining whether the heterogeneous acceleration card to be removed has a corresponding allocation relationship with at least one computing resource pool of the entire cabinet of the converged architecture; if the heterogeneous acceleration card to be removed has a corresponding allocation relationship with at least one computing resource pool of the entire cabinet of the converged architecture, determining that the heterogeneous acceleration card to be removed is in an allocated state; otherwise, determining that the heterogeneous acceleration card to be removed is in an unallocated state.

[0034] Specifically, the converged architecture system determines, based on the extracted first number and second number, whether the heterogeneous acceleration card to be removed has a corresponding allocation relationship with at least one computing resource pool of the converged architecture whole cabinet.

[0035] If the heterogeneous accelerator card to be removed has a corresponding allocation relationship with at least one computing resource pool, the accelerator card is determined to be in an allocated state. If the heterogeneous accelerator card to be removed does not have a corresponding allocation relationship with any computing resource pool, the accelerator card is determined to be in an unallocated state.

[0036] Through the above technical solution, according to the judgment results, the system can distinguish whether the heterogeneous acceleration card to be removed is in an allocated state or an unallocated state, which helps the system to manage heterogeneous acceleration card resources more accurately, avoid resource conflicts and waste, and automatically allocate unallocated heterogeneous acceleration cards to the computing resource pool corresponding to the original heterogeneous acceleration card to be removed during runtime to meet real-time changing performance requirements.

[0037] In step S103, the heterogeneous accelerator card to be removed is replaced with the target heterogeneous accelerator card, and the target heterogeneous accelerator card is allocated to the computing resource pool corresponding to the heterogeneous accelerator card to be removed.

[0038] Specifically, the converged architecture system replaces the position of the heterogeneous accelerator card to be removed with the target heterogeneous accelerator card, and allocates the target heterogeneous accelerator card to the computing resource pool originally corresponding to the heterogeneous accelerator card to be removed, so that the target accelerator card will assume the responsibilities originally assumed by the faulty accelerator card and provide acceleration capabilities for related computing instances or tasks.

[0039] Optionally, in some embodiments, after allocating the target heterogeneous acceleration card to the computing resource pool corresponding to the heterogeneous acceleration card to be removed, the method includes: updating an allocation relationship table of the target heterogeneous acceleration card in the entire cabinet of the converged architecture.

[0040] It is understandable that after allocating the target heterogeneous accelerator card to the computing resource pool corresponding to the heterogeneous accelerator card to be removed, the system obtains the allocation relationship table of the heterogeneous accelerator cards in the entire cabinet of the converged architecture, which is usually stored in the system's internal database or configuration file. Based on the previous replacement and allocation operations, the system finds the record of the heterogeneous accelerator card to be removed in the allocation relationship table and replaces it with the information of the target heterogeneous accelerator card, including the number of the target heterogeneous accelerator card, the number of the heterogeneous accelerator card device to which it belongs, etc.

[0041] The converged architecture system can save the updated allocation relationship table back to the internal database or configuration file to ensure that subsequent operations and scheduling of the system are based on the latest allocation information.

[0042] Through the above technical solution, updating the allocation relationship table ensures the consistency of the management and allocation of heterogeneous accelerator card resources within the system. It helps to avoid resource conflicts, incorrect allocation or omissions, thereby improving the stability and reliability of the system. By accurately recording the allocation of each heterogeneous accelerator card, the system can manage and utilize resources more effectively. When an accelerator card fails, the faulty heterogeneous accelerator card can be quickly located and replaced according to the updated allocation relationship table, reducing the fault recovery time and improving the system availability and business continuity.

[0043] Optionally, in some embodiments, after receiving the power-off instruction of the faulty heterogeneous acceleration card, the method further includes: when the heterogeneous acceleration card to be removed is in an unallocated state, controlling the heterogeneous acceleration card to be removed to power off.

[0044] It is understandable that if the heterogeneous acceleration card to be removed is in an unallocated state, the converged architecture system controls the power-off of the heterogeneous acceleration card to be removed according to the power-off instruction of the faulty heterogeneous acceleration card.

[0045] Through the above technical solution, it is ensured that the heterogeneous acceleration card to be removed is powered off when it is not used by any computing resource pool, thereby avoiding data loss or service interruption caused by sudden power failure.

[0046] Optionally, in some embodiments, when controlling the power-off of the heterogeneous acceleration card to be removed, it also includes: determining whether the heterogeneous acceleration card to be removed is powered off successfully; if the heterogeneous acceleration card to be removed is powered off successfully, generating a first power-off success signal to the pooled management controller of the entire cabinet of the converged architecture, and using the pooled management controller to send the power-off success information to a preset terminal to remind the operation and maintenance personnel to replace the new heterogeneous acceleration card.

[0047] Specifically, the converged architecture system monitors the power status of the heterogeneous acceleration card to be removed and detects whether it is successfully powered off. Once it is confirmed that the heterogeneous acceleration card to be removed is powered off successfully, the system generates a first power-off success signal to the pooled management controller of the entire converged architecture cabinet. After receiving the first power-off success signal, the pooled management controller generates corresponding power-off success information.

[0048] Then, the pooled management controller sends the power-off success information to the preset terminal (such as the mobile device and workstation of the operation and maintenance personnel) through the preset communication channel (such as email, SMS, operation and maintenance management system, etc.). The power-off success information includes the number of the heterogeneous accelerator card to be removed, the number of the heterogeneous accelerator card device where the heterogeneous accelerator card to be removed is located, the power-off time, and the reminder that the operation and maintenance personnel need to replace the new heterogeneous accelerator card.

[0049] After receiving the power-off success message, the O&M personnel know that the heterogeneous accelerator card to be removed has been safely powered off. They remove the heterogeneous accelerator card from the entire cabinet and install a new heterogeneous accelerator card.

[0050] If the power-off fails, you may need to retry the power-off command, log error information, notify the administrator, etc.

[0051] Through the above technical solution, it is possible to determine whether the heterogeneous acceleration card to be removed is powered off successfully to ensure that the acceleration card has indeed been safely powered off. Once it is confirmed that the heterogeneous acceleration card to be removed is powered off successfully, the power-off status can be quickly communicated to the operation and maintenance personnel without manual inspection or manual reporting. After receiving the power-off success information, the operation and maintenance personnel can immediately know when the faulty acceleration card can be safely removed and replaced with a new heterogeneous acceleration card, thereby reducing waiting time and improving the efficiency and response speed of operation and maintenance operations.

[0052] Optionally, in some embodiments, after using the pooled management controller to send a power-off success message to a preset terminal, it includes: determining whether the pooled management controller has received a power-on instruction; if the pooled management controller has received a power-on instruction, controlling the power-on of the new heterogeneous acceleration card based on the first number and the second number of the new heterogeneous acceleration card.

[0053] It can be understood that once the pooling management controller receives a power-on instruction, which includes the first number of the new heterogeneous accelerator card and the second number of the heterogeneous accelerator card device, the pooling management controller prepares to perform a power-on operation based on the first number of the new heterogeneous accelerator card and the second number of the heterogeneous accelerator card device. It should be noted that since the new heterogeneous accelerator card replaces the heterogeneous accelerator card to be removed, the first number and the second number of the heterogeneous accelerator card to be removed are also the first number and the second number of the new heterogeneous accelerator card.

[0054] The pooling management controller identifies the location of the new heterogeneous accelerator card that needs to be powered on according to the first number and the second number in the power-on instruction, sends the power-on instruction to the new heterogeneous accelerator card, and the system monitors the power status of the new heterogeneous accelerator card in real time to ensure that the power-on operation is successfully executed.

[0055] Through the above technical solution, the system can automatically trigger the power-on process of the new heterogeneous acceleration card by judging whether the pooled management controller has received the power-on instruction, thereby reducing manual intervention and improving the degree of automation. Once it is confirmed that the new heterogeneous acceleration card needs to be powered on, the system can immediately control its power-on based on the first number and the second number, ensuring that after the faulty acceleration card is removed, resources can be quickly restored and reallocated.

[0056] Optionally, in some embodiments, after controlling the power-on of the new heterogeneous acceleration card based on the first number and the second number, it includes: detecting whether the power-on of the new heterogeneous acceleration card is complete; if the power-on of the new heterogeneous acceleration card is complete, resetting the new heterogeneous acceleration card, and allocating the new heterogeneous acceleration card to the corresponding computing resource pool after the new heterogeneous acceleration card is successfully reset.

[0057] It is understandable that the converged architecture system continuously monitors the power status of the new heterogeneous acceleration card to determine whether its power-on is complete.

[0058] Once it is detected that the new heterogeneous accelerator card is powered on, the system prepares to perform the next reset operation, that is, the system sends a reset instruction to the new heterogeneous accelerator card, the system monitors the reset process of the new heterogeneous accelerator card, and detects whether the new heterogeneous accelerator card is reset successfully.

[0059] Once the new heterogeneous acceleration card is reset successfully, the system allocates the new heterogeneous acceleration card to the computing resource pool of the entire cabinet of the converged architecture.

[0060] Through the above technical solution, the need for manual intervention is reduced by automating the detection of power-on, reset and allocation processes. Operation and maintenance personnel can rely on the system's automated processing to ensure the correct deployment and configuration of new heterogeneous accelerator cards, thereby reducing operational complexity and error rates.

[0061] Optionally, in some embodiments, after the target heterogeneous acceleration card is allocated to the computing resource pool corresponding to the heterogeneous acceleration card to be removed, it also includes: controlling the power-off of the heterogeneous acceleration card to be removed, and determining whether the power-off of the heterogeneous acceleration card to be removed is successful; if the heterogeneous acceleration card to be removed is powered off successfully, generating a second power-off success signal to the pooled management controller of the entire cabinet of the converged architecture, and using the pooled management controller to send the second power-off success information to a preset terminal to remind the operation and maintenance personnel to replace the new heterogeneous acceleration card.

[0062] It should be understood that after the converged architecture system has completed the determination of the faulty heterogeneous acceleration card, the selection and replacement of the target heterogeneous acceleration card, and the allocation of the target heterogeneous acceleration card to the corresponding resource pool, it sends a power-off command to the heterogeneous acceleration card to be removed to control the power-off of the heterogeneous acceleration card to be removed. The converged architecture entire cabinet determines whether the heterogeneous acceleration card to be removed has been successfully powered off by monitoring the power status of the heterogeneous acceleration card to be removed.

[0063] Once it is confirmed that the heterogeneous acceleration card to be removed has been powered off successfully, the system generates a second power-off success signal to notify the pooled management controller of the entire cabinet of the converged architecture. After receiving the second power-off success signal, the pooled management controller generates corresponding second power-off success information and sends it to the preset terminal (such as the mobile device and workstation of the operation and maintenance personnel) through the preset communication channel (such as email, SMS, operation and maintenance management system, etc.).

[0064] The second power-off success information includes the number of the heterogeneous acceleration card to be removed, the power-off time, and a reminder of a new heterogeneous acceleration card that may need to be replaced by the operation and maintenance personnel.

[0065] After receiving the second power-off success message, the operation and maintenance personnel know that the heterogeneous accelerator card to be removed has been safely powered off and the system is ready for replacement. The operation and maintenance personnel perform a physical replacement operation, remove the faulty accelerator card from the entire cabinet, install a new heterogeneous accelerator card, and verify the new accelerator card to ensure that it works properly and meets system requirements, and may add it to the corresponding heterogeneous accelerator card device.

[0066] If the power-off fails, the system may need to execute error handling procedures, such as retrying the power-off command, recording error information, and notifying the administrator.

[0067] Through the above technical solution, after the new heterogeneous acceleration card is allocated to the computing resource pool, the system immediately controls the power-off of the acceleration card to be removed, ensuring that the computing resources will not be interrupted due to the replacement of the acceleration card, thereby improving the availability and continuity of the system. By judging whether the heterogeneous acceleration card to be removed is successfully powered off, the system can ensure that the acceleration card has indeed been safely powered off, preventing potential safety risks caused by incomplete power-off. Once it is confirmed that the acceleration card to be removed is successfully powered off, the system will automatically generate a second power-off success signal and send it to the preset terminal through the pooled management controller. This automated notification mechanism can quickly convey the power-off status to the operation and maintenance personnel, reminding them to replace the heterogeneous acceleration card to be removed. After receiving the power-off success information, the operation and maintenance personnel can immediately know when the heterogeneous acceleration card to be removed can be safely removed and replaced, reducing waiting time and improving the efficiency and response speed of operation and maintenance operations.

[0068] Optionally, in some embodiments, controlling the power-off of the heterogeneous acceleration card to be removed includes: generating a power-off instruction and determining a transmission mode of the power-off instruction; sending the power-off instruction to a controller corresponding to the heterogeneous acceleration card to be removed according to the transmission mode, so as to control the power-off of the heterogeneous acceleration card to be removed based on the power-off instruction through the controller corresponding to the heterogeneous acceleration card to be removed.

[0069] In some embodiments, the transmission mode includes cable transmission and / or network transmission.

[0070] Specifically, the power-off instruction of the heterogeneous accelerator card sent by the PSMC can be transmitted through the CDFP cable, or the power-off instruction can be directly sent to the BMC of the heterogeneous accelerator box through the network. The BMC of the heterogeneous accelerator box sends the power-off command to the CPLD of the heterogeneous accelerator box to control the power-off of the heterogeneous accelerator card. This does not rely on hardware design and is more flexible.

[0071] It can be understood that if the transmission mode is the cable transmission mode, the PSMC sends the power-off command to the BMC corresponding to the heterogeneous acceleration card to be removed through the CDFP cable. After receiving the power-off command, the BMC corresponding to the heterogeneous acceleration card to be removed sends the power-off command to the corresponding CPLD to control the power-off of the heterogeneous acceleration card.

[0072] If the transmission mode is the network transmission mode, the PSMC sends a power-off command to the BMC corresponding to the heterogeneous accelerator card to be removed through a network protocol (such as TCP / IP). The BMC corresponding to the heterogeneous accelerator card to be removed sends a power-off command to the corresponding CPLD, thereby controlling the power-off of the heterogeneous accelerator card to be removed.

[0073] It should be noted that not only the power-off instruction can be transmitted via a cable or a network, but the power-on instruction can also be transmitted via a cable or a network.

[0074] Through the above technical solution, by generating power-off instructions, determining the transmission mode, and sending power-off instructions to the BMC series operations of the heterogeneous acceleration card Box, the system can control the power-off of the acceleration card more flexibly and accurately, reduce the complexity of manual operations, improve the overall operation and maintenance efficiency, provide two or more optional transmission modes of cable transmission and network transmission, enhance the flexibility and adaptability of the system, help ensure that the heterogeneous acceleration card to be removed can be powered off safely and efficiently, and at the same time improve the efficiency and convenience of operation and maintenance operations.

[0075] Optionally, in some embodiments, the above-mentioned method for handling the failure of heterogeneous acceleration cards further includes: separately supplying power to all heterogeneous acceleration cards in the entire cabinet of the converged architecture.

[0076] It can be understood that the 16 heterogeneous acceleration cards in the multiple heterogeneous acceleration card devices in the embodiment of the present invention are powered separately, and the power on and off of different heterogeneous acceleration cards do not interfere with each other.

[0077] The power on and off of each heterogeneous accelerator card is controlled by the CPLD in the heterogeneous accelerator card box. Figure 3 As shown in the figure, CPLD receives three control signal inputs from IO Box CPLD (Host_CARDx_PWR_EN), heterogeneous accelerator card BoxBMC (BMC_CARDx_BTN_N), and heterogeneous accelerator card Box left ear powerbutton (FP_PWR_BTN_N), and outputs heterogeneous accelerator card power-on enable signals (CPLD_P12V_CARDx_EN, CPLD_P3V3_CARDx_EN) through internal logic processing. CPLD determines the power supply status of the heterogeneous accelerator card based on the powerenable (CPLD_P12V_CARDx_EN) and powergood (P3V3_CARDx_PWRGD) signals of the VR chip.

[0078] After the heterogeneous accelerator card Box CPLD receives the power-on command of the heterogeneous accelerator card, it enables the MP5991GLU to output P12V_CARDx, receives the P12V_CARDx_PWRGD signal, and enables the MPQ8626VR to output P3V3_CARDx. The above chips output the P12V and P3V3 required for the heterogeneous accelerator card to power the heterogeneous accelerator card.

[0079] When the enable signal (Host_CARDx_PWR_EN) on the IO Box CPLD is valid, the two signals of the IO Box CPLD (Host_CARDx_PWR_EN) and the heterogeneous accelerator card Box BMC (BMC_CARDx_BTN_N) will be ignored by the CPLD to prevent the heterogeneous accelerator card from being powered off abnormally.

[0080] The 40 CDFP ports of the IO Box and the CDFP ports of the device Box are connected through CDFP cables. The CDFP cables can transmit the power-on and power-off instructions (Host_CARDx_PWR_EN) of the CPLD to control the heterogeneous accelerator card, and transparently transmit the CPLD signals in the IO Box to the heterogeneous accelerator card Box CPLD.

[0081] Through the above technical solution, discrete power supply means that each heterogeneous accelerator card has an independent power supply, reducing the risk of failure of the entire system or multiple accelerator cards due to a single power failure. When a power failure occurs, only the affected accelerator card will be powered off, and other accelerator cards can continue to work, thereby improving the overall reliability and stability of the system. Discrete power supply enables the system to manage the power status of each accelerator card more flexibly. Discrete power supply allows the system to accurately control the power supply of each accelerator card according to actual needs.

[0082] In order to enable those skilled in the art to further understand the fault handling method of the heterogeneous accelerator card according to the embodiment of the present invention, the method is described in detail below in conjunction with specific embodiments. Figure 2 and 4 shown.

[0083] In step S401, when a heterogeneous accelerator card device fails or is in certain specific necessary scenarios and needs to be replaced, the operation and maintenance personnel determine the number of the heterogeneous accelerator card to be replaced and the number of the heterogeneous accelerator card Box where the heterogeneous accelerator card is located.

[0084] In step S402, the operation and maintenance personnel send a heterogeneous accelerator card power-off instruction to the PSMC through an IPMI command, a Redfish interface, or a web page. The instruction parameters include the heterogeneous accelerator card number and the heterogeneous accelerator card Box number where the heterogeneous accelerator card is located.

[0085] In step S403, the PSMC determines whether the heterogeneous accelerator card has been allocated to a Host BOX according to the allocation relationship table between the heterogeneous accelerator card that needs to be replaced and the computing resource pool.

[0086] In step S404, if it is in the allocation state, the PSMC removes the heterogeneous acceleration card that needs to be replaced from the Host BOX, and updates the allocation relationship table and the device asset information on the Host BMC.

[0087] In step S405, the PSMC selects an unallocated heterogeneous accelerator card from the computing resource pool to replace the heterogeneous accelerator card that needs to be replaced, and allocates an unallocated heterogeneous accelerator card to the HostBOX corresponding to the heterogeneous accelerator card that originally needs to be replaced to ensure its computing power requirements, and updates the allocation relationship table and the device asset information on the Host BMC side. If the heterogeneous accelerator card is in an unallocated state, steps S406 to S412 are directly executed. After allocating an unallocated heterogeneous accelerator card to the Host BOX corresponding to the heterogeneous accelerator card that originally needs to be replaced, the heterogeneous accelerator card that originally needs to be replaced is also controlled to be powered off, that is, steps S406 to S412 are executed.

[0088] In step S406, the PSMC sends a power-off instruction of the heterogeneous accelerator card to the IO Box CPLD through the I2C link according to the number of the heterogeneous accelerator card to be replaced and the number of the Box where the heterogeneous accelerator card is located.

[0089] In step S407, the IO Box CPLD sends a power-off instruction to the heterogeneous accelerator card Box CPLD through the CDFP cable to control the corresponding heterogeneous accelerator card to power off.

[0090] In step S408, after the heterogeneous acceleration card Box BMC detects through the CPLD that the heterogeneous acceleration card is powered off successfully, it reports a power-off success message to the PSMC, and the PSMC updates the power supply status of the heterogeneous acceleration card.

[0091] In step S409, the operation and maintenance personnel replace the heterogeneous accelerator card with a new one.

[0092] In step S410, after the operation and maintenance personnel completes replacing the new heterogeneous accelerator card, they send a power-on instruction for the heterogeneous accelerator card to the PSMC.

[0093] In step S411, after the PSMC detects that the new heterogeneous accelerator card is powered on, it sends a reset signal to the new heterogeneous accelerator card to reset the new heterogeneous accelerator card.

[0094] In step S412, after the new heterogeneous acceleration card is reset, it can continue to be allocated to the computing resource pool as a resource.

[0095] In summary, based on the analysis of the above specific embodiments, the present invention can achieve the following beneficial effects: (1) By receiving the power-off command of the faulty heterogeneous accelerator card and judging the status of the accelerator card to be removed, the faulty accelerator card can be quickly identified and processed. When it is confirmed that the accelerator card to be removed is in an allocated state, it can automatically select an unallocated target accelerator card from the resource pool for replacement and allocate it to the original computing resource pool, thereby ensuring the continuity and stability of computing resources. This not only improves the efficiency of fault handling, but also realizes the dynamic optimization allocation of resources, improving the flexibility and reliability of the overall system; (2) When controlling the power-off of the heterogeneous accelerator card to be removed, the power-off command is generated and the transmission mode (such as cable transmission or network transmission) is determined to ensure the accurate transmission and execution of the power-off command. At the same time, the method also includes the detection and confirmation of successful power-off, as well as a mechanism for using the pooled management controller to send power-off success information to the preset terminal, which helps the operation and maintenance personnel to understand the status of the accelerator card in a timely manner and perform safe physical replacement operations. In addition, the design of discrete power supply for all heterogeneous accelerator cards in the entire cabinet of the converged architecture further enhances the power supply stability and security of the system.

[0096] (3) After allocating the target heterogeneous accelerator card to the computing resource pool corresponding to the accelerator card to be removed, not only is the resource seamlessly replaced, but the accuracy and consistency of the internal resource information of the system is also ensured by updating the allocation relationship table, which helps the operation and maintenance personnel to better monitor and manage the accelerator card resources in the entire cabinet. At the same time, the fault handling process and information feedback mechanism provided by this method also greatly facilitate the daily maintenance and troubleshooting work of the operation and maintenance personnel, reducing the cost and complexity of operation and maintenance.

[0097] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0098] According to the fault handling method of the heterogeneous acceleration card proposed in the embodiment of the present invention, it is determined whether a power-off instruction of the faulty heterogeneous acceleration card is received, wherein the power-off instruction of the faulty heterogeneous acceleration card includes the heterogeneous acceleration card to be removed. If the power-off instruction of the faulty heterogeneous acceleration card is received, the heterogeneous acceleration card to be removed is removed when the heterogeneous acceleration card to be removed is in an allocated state, and the target heterogeneous acceleration card in an unallocated state is determined from at least one heterogeneous acceleration card device of the whole cabinet of the converged architecture, and the heterogeneous acceleration card to be removed is replaced by the target heterogeneous acceleration card, and the target heterogeneous acceleration card is allocated to the computing resource pool corresponding to the heterogeneous acceleration card to be removed. Thus, the technical problem that when a heterogeneous acceleration card device in the whole cabinet of the converged architecture fails, the processing method of the prior art needs to power off the entire heterogeneous acceleration card device, which affects the business operation of all computing resource pools in the whole cabinet and brings a greater business impact to the user can be solved, and the technical effect of completing the replacement operation of the faulty acceleration card and restoring the normal operation of the business in time while minimizing the impact on the business function of the whole system can be achieved, and reducing the maintenance pressure of the operation and maintenance personnel.

[0099] The embodiment of the present invention also provides a fault handling device for a heterogeneous accelerator card, such as Figure 5 As shown, the device includes: a judgment module 100, a fault processing module 200 and a replacement module 300.

[0100] Among them, the judgment module 100 is used to judge whether a power-off instruction of a faulty heterogeneous acceleration card is received, wherein the power-off instruction of the faulty heterogeneous acceleration card includes the heterogeneous acceleration card to be removed; the fault processing module 200 is used to remove the heterogeneous acceleration card to be removed if the power-off instruction of the faulty heterogeneous acceleration card is received, and determine the target heterogeneous acceleration card in the unallocated state from at least one heterogeneous acceleration card device of the whole cabinet of the converged architecture; the replacement module 300 is used to replace the heterogeneous acceleration card to be removed with the target heterogeneous acceleration card, and allocate the target heterogeneous acceleration card to the computing resource pool corresponding to the heterogeneous acceleration card to be removed.

[0101] Optionally, in some embodiments, the faulty heterogeneous acceleration card power-off instruction further includes a first serial number of the heterogeneous acceleration card to be removed and a second serial number of a heterogeneous acceleration card device to which the heterogeneous acceleration card to be removed belongs.

[0102] Optionally, in some embodiments, after receiving the power-off instruction of the faulty heterogeneous acceleration card, the fault handling module 200 is also used to: determine, based on the first number and the second number, whether the heterogeneous acceleration card to be removed has a corresponding allocation relationship with at least one computing resource pool of the entire cabinet of the converged architecture; if the heterogeneous acceleration card to be removed has a corresponding allocation relationship with at least one computing resource pool of the entire cabinet of the converged architecture, then determine that the heterogeneous acceleration card to be removed is in an allocated state; otherwise, determine that the heterogeneous acceleration card to be removed is in an unallocated state.

[0103] Optionally, in some embodiments, after receiving the power-off instruction of the faulty heterogeneous acceleration card, the fault processing module 200 is further used to: when the heterogeneous acceleration card to be removed is in an unallocated state, control the power-off of the heterogeneous acceleration card to be removed.

[0104] Optionally, in some embodiments, when controlling the power-off of the heterogeneous acceleration card to be removed, the fault handling module 200 is also used to: determine whether the heterogeneous acceleration card to be removed is powered off successfully; if the heterogeneous acceleration card to be removed is powered off successfully, generate a first power-off success signal to the pooled management controller of the entire cabinet of the converged architecture, and use the pooled management controller to send the power-off success information to a preset terminal to remind the operation and maintenance personnel to replace the new heterogeneous acceleration card.

[0105] Optionally, in some embodiments, after the pooled management controller sends a power-off success message to a preset terminal, the fault handling module 200 is further used to: determine whether the pooled management controller has received a power-on instruction; if the pooled management controller has received a power-on instruction, control the power-on of the new heterogeneous accelerator card based on the first number and the second number of the new heterogeneous accelerator card.

[0106] Optionally, in some embodiments, after controlling the power-on of the new heterogeneous acceleration card based on the first number and the second number, the fault handling module 200 is further used to: detect whether the power-on of the new heterogeneous acceleration card is complete; if the power-on of the new heterogeneous acceleration card is complete, reset the new heterogeneous acceleration card, and allocate it to the corresponding computing resource pool after the new heterogeneous acceleration card is successfully reset.

[0107] Optionally, in some embodiments, after the target heterogeneous acceleration card is allocated to the computing resource pool corresponding to the heterogeneous acceleration card to be removed, the allocation module 300 is also used to: control the power-off of the heterogeneous acceleration card to be removed, and determine whether the heterogeneous acceleration card to be removed is powered off successfully; if the heterogeneous acceleration card to be removed is powered off successfully, a second power-off success signal is generated to the pooled management controller of the entire cabinet of the converged architecture, and the second power-off success information is sent to the preset terminal using the pooled management controller to remind the operation and maintenance personnel to replace the new heterogeneous acceleration card.

[0108] Optionally, in some embodiments, the fault handling module 200 is further used to: generate a power-off instruction and determine a transmission mode of the power-off instruction; and send the power-off instruction to a controller corresponding to the heterogeneous acceleration card to be removed according to the transmission mode, so as to control the power-off of the heterogeneous acceleration card to be removed based on the power-off instruction through the controller corresponding to the heterogeneous acceleration card to be removed.

[0109] Optionally, in some embodiments, the transmission mode includes cable transmission and / or network transmission.

[0110] Optionally, in some embodiments, the above-mentioned heterogeneous acceleration card fault handling device 10 further includes: a power supply module, which is used to separately supply power to all heterogeneous acceleration cards in the entire cabinet of the converged architecture.

[0111] Optionally, in some embodiments, after allocating the target heterogeneous acceleration card to the computing resource pool corresponding to the heterogeneous acceleration card to be removed, the allocation module 300 is further used to: update the allocation relationship table of the target heterogeneous acceleration card in the entire cabinet of the converged architecture.

[0112] It should be noted that, for the description of the features in the embodiment corresponding to the fault handling device of the heterogeneous acceleration card, reference may be made to the relevant description of the embodiment corresponding to the fault handling method of the heterogeneous acceleration card, which will not be described in detail here.

[0113] An embodiment of the present invention further provides an electronic device, Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device may include: A memory 601 , a processor 602 , and a computer program stored in the memory 601 and executable on the processor 602 .

[0114] When the processor 602 executes the program, the fault handling method for the heterogeneous acceleration card provided in the above embodiment is implemented.

[0115] Furthermore, the electronic device further comprises: The communication interface 603 is used for communication between the memory 601 and the processor 602 .

[0116] The memory 601 is used to store computer programs that can be executed on the processor 602 .

[0117] The memory 601 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0118] If the memory 601, the processor 602 and the communication interface 603 are implemented independently, the communication interface 603, the memory 601 and the processor 602 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0119] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.

[0120] The processor 602 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0121] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned heterogeneous acceleration card fault handling method embodiments when running.

[0122] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0123] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0124] The above is a detailed introduction to the fault handling method of a heterogeneous acceleration card provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A method for handling a fault of a heterogeneous accelerator card, characterized in that: The following steps are involved: Determining whether a power-off instruction of a faulty heterogeneous accelerator card is received, wherein the power-off instruction of the faulty heterogeneous accelerator card includes the heterogeneous accelerator card to be removed; If a power-off instruction of the faulty heterogeneous accelerator card is received, if the heterogeneous accelerator card to be removed is in an allocated state, the heterogeneous accelerator card to be removed is removed, and a target heterogeneous accelerator card in an unallocated state is determined from at least one heterogeneous accelerator card device of the converged architecture whole cabinet; The target heterogeneous accelerator card is used to replace the heterogeneous accelerator card to be removed, and the target heterogeneous accelerator card is allocated to the computing resource pool corresponding to the heterogeneous accelerator card to be removed.

2. The method for handling a fault of a heterogeneous accelerator card according to claim 1, characterized in that: The failed heterogeneous acceleration card power-off instruction further includes a first number of the to-be-removed heterogeneous acceleration card and a second number of a heterogeneous acceleration card device to which the to-be-removed heterogeneous acceleration card belongs.

3. The method for handling a fault of a heterogeneous accelerator card according to claim 2, characterized in that: After receiving the power-off instruction of the faulty heterogeneous acceleration card, the method further includes: Based on the first number and the second number, determining whether the heterogeneous acceleration card to be removed has a corresponding allocation relationship with at least one computing resource pool of the converged architecture whole cabinet; If there is a corresponding allocation relationship between the heterogeneous acceleration card to be removed and at least one computing resource pool of the converged architecture whole cabinet, it is determined that the heterogeneous acceleration card to be removed is in the allocated state; otherwise, it is determined that the heterogeneous acceleration card to be removed is in the unallocated state.

4. The method for handling a fault of a heterogeneous accelerator card according to claim 1, characterized in that: After receiving the power-off instruction of the faulty heterogeneous acceleration card, the method further includes: When the heterogeneous acceleration card to be removed is in an unallocated state, the heterogeneous acceleration card to be removed is controlled to be powered off.

5. The method for handling a fault of a heterogeneous accelerator card according to claim 4, characterized in that: When controlling the power-off of the heterogeneous acceleration card to be removed, the method further includes: Determine whether the heterogeneous acceleration card to be removed is powered off successfully; If the heterogeneous acceleration card to be removed is powered off successfully, a first power-off success signal is generated to the pooled management controller of the converged architecture whole cabinet, and the pooled management controller is used to send power-off success information to a preset terminal to remind the operation and maintenance personnel to replace the new heterogeneous acceleration card.

6. The method for handling a fault of a heterogeneous accelerator card according to claim 5, characterized in that: After the pooling management controller is used to send the power-off success information to the preset terminal, the method includes: Determining whether the pooling management controller has received a power-on instruction; If the pooling management controller receives the power-on instruction, it controls the new heterogeneous accelerator card to power on based on the first number and the second number of the new heterogeneous accelerator card.

7. The method for handling a fault of a heterogeneous accelerator card according to claim 6, characterized in that: After controlling the new heterogeneous accelerator card to be powered on based on the first number and the second number, the method includes: Detect whether the new heterogeneous acceleration card is powered on; If the new heterogeneous acceleration card is powered on, the new heterogeneous acceleration card is reset, and is allocated to the corresponding computing resource pool after the new heterogeneous acceleration card is successfully reset.

8. The method for handling a fault of a heterogeneous accelerator card according to claim 1, characterized in that: After allocating the target heterogeneous accelerator card to the computing resource pool corresponding to the heterogeneous accelerator card to be removed, the method further includes: Controlling the to-be-removed heterogeneous accelerator card to be powered off, and determining whether the to-be-removed heterogeneous accelerator card is powered off successfully; If the heterogeneous acceleration card to be removed is powered off successfully, a second power-off success signal is generated to the pooled management controller of the converged architecture whole cabinet, and the pooled management controller is used to send the second power-off success information to a preset terminal to remind the operation and maintenance personnel to replace the new heterogeneous acceleration card.

9. The method for handling a fault of a heterogeneous accelerator card according to claim 4 or 8, characterized in that: The controlling the heterogeneous accelerator card to be removed to power off includes: Generate a power-off instruction and determine a transmission mode of the power-off instruction; According to the transmission mode, the power-off instruction is sent to a controller corresponding to the heterogeneous acceleration card to be removed, so that the controller corresponding to the heterogeneous acceleration card to be removed controls the power-off of the heterogeneous acceleration card to be removed based on the power-off instruction.

10. The method for handling a fault of a heterogeneous accelerator card according to claim 9, characterized in that: The transmission mode includes cable transmission and / or network transmission.

11. The method for handling a fault of a heterogeneous accelerator card according to claim 1, characterized in that: Also includes: Provide discrete power supply for all heterogeneous acceleration cards in the entire cabinet of the converged architecture.

12. The method for handling a fault of a heterogeneous accelerator card according to claim 1, characterized in that: After allocating the target heterogeneous accelerator card to the computing resource pool corresponding to the heterogeneous accelerator card to be removed, the method includes: Update the allocation relationship table of the target heterogeneous acceleration card in the converged architecture cabinet.

13. A fault handling device for a heterogeneous accelerator card, characterized in that: include: A judgment module, used to judge whether a power-off instruction of a faulty heterogeneous accelerator card is received, wherein the power-off instruction of the faulty heterogeneous accelerator card includes a heterogeneous accelerator card to be removed; a fault processing module, configured to, if receiving a power-off instruction for the faulty heterogeneous acceleration card, remove the heterogeneous acceleration card to be removed if the heterogeneous acceleration card to be removed is in an allocated state, and determine a target heterogeneous acceleration card in an unallocated state from at least one heterogeneous acceleration card device of the converged architecture whole cabinet; The replacement module is used to replace the to-be-removed heterogeneous acceleration card with the target heterogeneous acceleration card, and allocate the target heterogeneous acceleration card to the computing resource pool corresponding to the to-be-removed heterogeneous acceleration card.

14. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the fault handling method for a heterogeneous acceleration card as claimed in any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the fault handling method for a heterogeneous acceleration card as described in any one of claims 1 to 12.

Images