Server system, faulty equipment locating method, computer system, program product and storage medium

Abstract

The present application discloses a server system and a method for locating a faulty device, a computer system, a program product, and a storage medium. The method includes: in response to detecting a link failure, triggering a system management interrupt; parsing the advanced error reporting register of the faulty device to determine the fault information of the faulty device; determining the device asset information of the faulty device based on the fault information of the faulty device; in response to the device asset information of the faulty device not containing the silkscreen information of the faulty device, determining the physical location of the faulty device based on the fault information of the faulty device and the allocation relationship table. By implementing the server system disclosed in the embodiments of the present application and its method, device, system, computer program product, and readable storage medium for locating a faulty device, the utilization rate of server hardware resources can be greatly improved; the device with PCIe failure in the entire cabinet can be quickly located, the normal operation of the business can be restored in time, and the maintenance pressure of operation and maintenance personnel can be reduced.

Description

Technical Field

[0001] The present application relates to the field of computing technology, and in particular to a server system and a method, system, computer program product and readable storage medium for locating a faulty device thereof. Background Art

[0002] In order to meet the resource allocation requirements of scenarios such as artificial intelligence, machine learning, and intelligent computing, data centers are increasingly configuring pooled heterogeneous resources to improve the utilization of server hardware resources and meet heterogeneous computing requirements. Due to the pooling of heterogeneous resources, the allocation relationship between heterogeneous resources such as graphics processing units (GPUs) and computing resources is uncertain. For example: at time A, computing resource pool No. 0 is allocated with graphics processor No. 0 and graphics processor No. 1 in graphics processor resource pool No. 0. After a period of time, graphics processor No. 0 and graphics processor No. 1 are removed from computing resource No. 0. At the subsequent time B, computing resource pool No. 0 is allocated with graphics processor No. 2 and graphics processor No. 3 in graphics processor resource pool No. 1. For computing resource pool No. 0, at times A and B when there is a demand for graphics processor resources, two graphics processor resources are obtained. However, there is no fixed corresponding allocation relationship between computing resource No. 0 and graphics processor. Therefore, the basic input and output system firmware in computing resource No. 0 only saves the link identification information of the allocated graphics processor, but cannot save the silkscreen information of the allocated graphics processor. This will result in the inability to locate the specific faulty GPU when the assigned GPU fails. If the faulty GPU needs to be accurately located, maintenance time and costs will increase. Summary of the invention

[0003] In order to solve the problem of physical location of faulty equipment when pooled heterogeneous resources fail, this application provides the following technical solutions:

[0004] In a first aspect, a server system is provided, comprising:

[0005] At least one central processing unit, at least one switching device, at least one graphics processing unit and a network switch, wherein the at least one central processing unit is arranged in a first chassis, the at least one switching device is arranged in a third chassis, and the at least one graphics processing unit is arranged in a second chassis;

[0006] at least one central processing unit, at least one graphics processing unit, at least one switching device, and a network switch;

[0007] The at least one central processing unit is arranged in at least one first chassis, the at least one graphics processing unit is arranged in at least one second chassis, the at least one switching device is arranged in a third chassis, and the at least one first chassis, the at least one second chassis and the third chassis are all arranged in the same server cabinet;

[0008] Any first chassis of the at least one first chassis, any second chassis of the at least one second chassis, and the third chassis are respectively connected to the network switch through a first cable;

[0009] Any first chassis among the at least one first chassis and any second chassis among the at least one second chassis are respectively connected to the third chassis through a second cable.

[0010] Further, any first chassis of the at least one first chassis has: a first network port and a first communication port;

[0011] Any second chassis of the at least one second chassis has: a second network port, at least one second communication port;

[0012] The third chassis has: a third network port, at least one third communication port and at least one fourth communication port;

[0013] Any third communication port among the at least one third communication port is connected to a first communication port via a first cable, any fourth communication port among the at least one fourth communication port is connected to a second communication port among the at least one second communication port via a first cable, and the first network port, the second network port and the third network port are respectively connected to the network switch via a second cable.

[0014] Further, any first chassis of the at least one first chassis includes: a first baseboard management controller, a central processing unit and a first connector;

[0015] The first baseboard management controller is connected to the central processing unit, and the central processing unit is connected to the first connector;

[0016] The first connector serves as the first communication port;

[0017] The first baseboard management controller has a first baseboard management controller network port, and the first baseboard management controller network port serves as the first network port.

[0018] Furthermore, the first connector is a high-speed pluggable input / output connector.

[0019] Further, any second chassis of the at least one second chassis includes: a second baseboard management controller, at least one graphics processor and at least one second connector;

[0020] The second baseboard management controller is connected to the at least one graphics processor, and any graphics processor of the at least one graphics processor is connected to a second connector of the at least one second connector in a one-to-one correspondence;

[0021] The at least one second connector serves as the at least one second communication port;

[0022] The second baseboard management controller has a second baseboard management controller network port, and the second baseboard management controller network port serves as the second network port.

[0023] Furthermore, the second connector is a high-speed pluggable input / output connector.

[0024] Further, the third chassis includes: a third baseboard management controller, a transmission expansion device, a switching unit, at least one third connector and at least one fourth connector;

[0025] The switching unit is connected to the third baseboard management controller via the transmission expansion device;

[0026] The third baseboard management controller has a third baseboard management controller network port, and the third baseboard management controller network port serves as the third network port;

[0027] The switching unit has at least one computing communication port and at least one graphics card communication port;

[0028] Any one of the at least one computing communication port is connected to one of the at least one third connectors, and the at least one third connector serves as the at least one third communication port;

[0029] Any one of the at least one graphics card communication port is connected to one of the at least one fourth connectors, and the at least one fourth connector serves as the at least one fourth communication port.

[0030] Further, the switching unit includes at least one switching device;

[0031] The at least one switching device is fully connected.

[0032] Further, any of the at least one switching device has:

[0033] A host port, serving as the computing communication port;

[0034] A device port, serving as a communication port for the graphics card;

[0035] The internal interconnection port is used to connect the at least one switching device to a full connection.

[0036] Furthermore, the third connector and the fourth connector are high-speed pluggable input / output connectors.

[0037] Furthermore, the second cable is a CDFP high-speed cable.

[0038] In a second aspect, a method for locating a faulty device in a server system is provided, wherein the server system includes at least one central processing unit, at least one graphics processing unit, at least one switching device, and a network switch; the at least one central processing unit is arranged in at least one first chassis, the at least one graphics processing unit is arranged in at least one second chassis, and the at least one switching device is arranged in a third chassis; the at least one first chassis, the at least one second chassis, and the third chassis are all arranged in the same server cabinet; any first chassis in the at least one first chassis, any second chassis in the at least one second chassis, and the third chassis are respectively connected to the network switch through cables; any first chassis in the at least one first chassis and any second chassis in the at least one second chassis are respectively connected to the third chassis through cables, and the third chassis includes a third baseboard management controller, and the method includes:

[0039] triggering a system management interrupt in response to the at least one central processor detecting a server link failure;

[0040] Parsing the advanced error reporting register of the faulty device to determine the fault information of the faulty device;

[0041] Determine the equipment asset information of the faulty equipment according to the fault information of the faulty equipment;

[0042] In response to the fact that the device asset information of the faulty device does not include the slot silk-screen information of the faulty device, the physical location of the faulty device is determined by the third baseboard management controller according to the fault information of the faulty device and an allocation relationship table, wherein the allocation relationship table at least includes: a dynamic mapping relationship and a fixed interconnection relationship, the dynamic mapping relationship includes the connection relationship between the at least one switching device, and the fixed interconnection relationship includes the correspondence between the slot silk-screen information and the at least one second communication port.

[0043] Further, any first chassis among the at least one first chassis has: a first network port and a first communication port; any second chassis among the at least one second chassis has: a second network port and at least one second communication port; the third chassis has: a third network port, at least one third communication port and at least one fourth communication port; any third communication port among the at least one third communication port is connected to one of the first communication ports via a cable, any fourth communication port among the at least one fourth communication port is connected to one of the at least one second communication ports via a cable, and the first network port, the second network port and the third network port are respectively connected to the network switch via cables; the fault information of the faulty device includes link identification information of the faulty device;

[0044] Wherein, determining the physical location of the faulty device according to the fault information of the faulty device and the allocation relationship table includes:

[0045] Determine the port number of the switching device according to the link identification information of the faulty device;

[0046] Determine a fourth communication port based on the port number and the dynamic mapping relationship of the switching device;

[0047] Based on the fourth communication port and the fixed interconnection relationship, the second communication port and the faulty device slot silkscreen information connected to the corresponding second communication port are determined, and then the faulty device slot is determined.

[0048] Further, any first chassis in the at least one first chassis includes: a first baseboard management controller, a central processing unit and a first connector; the first baseboard management controller is connected to the central processing unit, and the central processing unit is connected to the first connector; the first connector serves as the first communication port; the first baseboard management controller has a first baseboard management controller network port, and the first baseboard management controller network port serves as the first network port; any second chassis in the at least one second chassis includes: a second baseboard management controller, at least one graphics processor and at least one second connector; the second baseboard management controller is connected to the at least one graphics processor, and any graphics processor in the at least one graphics processor is connected to a second connector in the at least one second connector; the at least one second connector serves as the at least one second communication port; the second baseboard management controller has a second baseboard management controller network port port, the second baseboard management controller network port serves as the second network port; the third chassis further comprises: a transmission extension device, a switching unit, at least one third connector and at least one fourth connector; the switching unit is connected to the third baseboard management controller through the transmission extension device; the third baseboard management controller has a third baseboard management controller network port, and the third baseboard management controller network port serves as the third network port; the switching unit has at least one computing communication port and at least one graphics card communication port; any computing communication port of the at least one computing communication port is connected to a third connector of the at least one third connector, and the at least one third connector serves as the at least one third communication port; any graphics card communication port of the at least one graphics card communication port is connected to a fourth connector of the at least one fourth connector, and the at least one fourth connector serves as the at least one fourth communication port;

[0049] Before the system management interrupt is triggered in response to detecting a link failure, the method further includes:

[0050] Powering on the server system, and establishing a network connection between the first baseboard management controller, the second baseboard management controller, and the third baseboard management controller;

[0051] Acquire the fixed interconnection relationship, and configure the fixed interconnection relationship in a configuration file of the third baseboard management controller;

[0052] Acquire a connection relationship of at least one switching device in the third chassis as a dynamic configuration relationship;

[0053] An allocation relationship table is determined according to the fixed interconnection relationship and the dynamic configuration relationship.

[0054] Further, configuring the fixed interconnection relationship in a configuration file of the third baseboard management controller includes:

[0055] Numbering hardware devices in at least one first chassis, at least one second chassis, and a third chassis; wherein the hardware devices include: a central processing unit, a graphics processing unit, a switching device, a first connector, a second connector, a third connector, and a fourth connector;

[0056] Numbering the ports of the switching device;

[0057] Associating the port of the switching device with the numbers of the third connector and the fourth connector;

[0058] Associating the number of the third connector with the number of the first connector, and associating the number of the fourth connector with the number of the second connector;

[0059] Associating the serial number of the second connector with the slot silkscreen information;

[0060] The above-mentioned relationship is regarded as a fixed interconnection relationship.

[0061] Further, obtaining the connection relationship of at least one switching device in the third chassis as a dynamic configuration relationship includes:

[0062] Obtain the internal interconnection relationship between switching devices;

[0063] Obtaining a mapping relationship between a port number of a switching device and a device link identifier;

[0064] Acquire a mapping relationship between asset information of any graphics processor of at least one graphics processor and at least one second connector;

[0065] The internal interconnection relationship, the mapping relationship between the port number of the switching device and the device link identifier, and the mapping relationship between the asset information of any graphics processor of the at least one graphics processor and at least one third connector are used as dynamic mapping relationships.

[0066] Furthermore, the method further comprises:

[0067] Parsing the fault information of the faulty device to obtain the fault type and fault level of the faulty device, wherein the fault type at least includes: an unrecoverable fault, and the fault level at least includes: a non-fatal fault and a fatal fault;

[0068] In response to the fault type of the faulty device being an unrecoverable fault and the fault level of the faulty device being a non-fatal fault, the faulty device is reset.

[0069] Further, in response to the fault type of the faulty device being an unrecoverable fault and the fault level of the faulty device being a fatal fault, a device fault alarm is issued.

[0070] In a third aspect, a computer system is provided, comprising a memory, a processor, and a server system fault device locating program stored in the memory and executable on the processor. When the processor executes the server system fault device locating program, the server system fault device locating method described in the second aspect is implemented.

[0071] In a fourth aspect, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method for locating a faulty device in a server system as described in the second aspect is implemented.

[0072] In a fifth aspect, a computer-readable storage medium is provided, on which a server system faulty device locating program is stored. When the server system faulty device locating program is executed by a processor, the server system faulty device locating method recorded in the second aspect is implemented.

[0073] The technical solution provided by the embodiments of the present application has the following beneficial effects: by implementing the server system and its faulty equipment locating method, apparatus, system, computer program product and readable storage medium disclosed in the embodiments of the present application, it is conducive to the flexible reconfiguration and rapid upgrading of key equipment such as processors and graphics processors, which can greatly improve the utilization rate of server hardware resources; the equipment with PCIe failure in the entire cabinet can be quickly located, and for the graphics processor equipment, if it is an unrecoverable and non-fatal failure, the graphics processor equipment can be hot-removed and hot-added, the equipment can be reset, and the normal operation of the business can be restored in time, reducing the maintenance pressure of operation and maintenance personnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0075] Figure 1 is a schematic diagram of a server system provided in an embodiment of the present application;

[0076] Figure 2 is a schematic diagram of full connection between switching devices provided in an embodiment of the present application;

[0077] Figure 3 It is a schematic diagram of a method for locating a faulty device in a server system provided in an embodiment of the present application;

[0078] Figure 4It is a schematic diagram of a device for locating a faulty device in a server system provided in an embodiment of the present application;

[0079] Figure 5 is a schematic diagram of a computer system provided in an embodiment of the present application;

[0080] Figure 6 It is a flow chart of a method for locating a faulty device in a server system provided in an embodiment of the present application;

[0081] Figure 7 It is a flow chart of another method for locating faulty equipment in a server system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0082] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in the implementation mode of this application will be clearly and completely described below in conjunction with the drawings in the implementation mode of this application. Obviously, the described implementation mode is only a part of the embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0083] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure shall have the usual meanings understood by persons with ordinary skills in the field to which the present disclosure belongs. The words "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one", "one" or "the" do not indicate a quantity limitation, but indicate that there is at least one. The numbers in the drawings of the specification only indicate the distinction between various functional components or modules, and do not indicate the logical relationship between components or modules. Words such as "include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0084] Hereinafter, various embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. It should be noted that in the accompanying drawings, the same reference numerals are given to components having substantially the same or similar structures and functions, and repeated descriptions thereof will be omitted.

[0085] In order to solve the problem that it is difficult to physically locate the faulty device when a failure occurs in pooled heterogeneous resources, this application provides the following technical solutions.

[0086] In some embodiments, Figure 1 As shown, the server system includes:

[0087] At least one central processor 120, at least one graphics processor 320, at least one switching device 230, and a network switch 4;

[0088] The at least one central processor 120 is disposed in at least one first chassis 10, the at least one graphics processor 320 is disposed in at least one second chassis 30, the at least one switching device 230 is disposed in a third chassis 2, and the at least one first chassis 10, the at least one second chassis 30 and the third chassis 2 are all disposed in the same server cabinet (not shown);

[0089] Any first chassis 10 of the at least one first chassis 10, any second chassis 30 of the at least one second chassis 30, and the third chassis 2 are respectively connected to the network switch 4 through a first cable;

[0090] Any first chassis 10 of the at least one first chassis 10 and any second chassis 30 of the at least one second chassis 30 are respectively connected to the third chassis 2 through a second cable.

[0091] The at least one first chassis 10 constructs a first chassis 1, and the first chassis 1 is a computing resource pool;

[0092] The third chassis 2 constructs a third chassis 2, and the third chassis 2 is an input and output resource pool;

[0093] The at least one second chassis 30 constructs a second chassis 3, and the second chassis 3 is a graphics processor resource pool.

[0094] The first chassis 1 has: a first chassis network port 1 L and at least one first input / output port 1 IO . First chassis network port 1 L Communicates with the network switch 4.

[0095] The third chassis 2 has: a third chassis network port 2 L , at least one second input / output port 2 IO1 and at least one third input / output port 2 IO2 . Third chassis network port 2 L Communicates with the network switch 4. At least one second input and output port 2 IO1The number of first input and output ports with at least one 1 IO Any second input / output port 2 of at least one second input / output port IO1 Correspondingly connected to a first input / output port 1 of at least one first input / output port IO , forming a communication connection between the first chassis 1 and the third chassis 2.

[0096] The second chassis 3 has: a second chassis network port 3 L and at least one fourth input / output port 3 IO . Second chassis network port 3 L Communicates with the network switch 4. At least one fourth input and output port 3 IO The number of input and output ports with at least one third 2 IO2 Any fourth input / output port 3 of at least one fourth input / output port IO Correspondingly connected to a third input / output port 2 of at least one third input / output port IO2 , forming a communication connection between the second chassis 3 and the third chassis 2.

[0097] Usually, the first chassis, the third chassis, and the second chassis are arranged in different chassis.

[0098] The first chassis 1 , the third chassis 2 and the second chassis 3 , which are respectively connected to the network switch 4 , form a local area network.

[0099] Preferably, at least one first input / output port 1 IO , at least one second input / output port 2 IO1 , at least one third input / output port 2 IO2 , and at least one fourth input-output port 3 IO It is a high-speed connection port, which is used to achieve extremely high data transmission speed, and can reach a data rate of 25 Gbps per channel on 16 channels, thereby achieving a total data transmission speed of 400 Gbps. Preferably, the high-speed connection port is a CDFP port. The above-mentioned ports connected to each other are connected through CDFP cables.

[0100] The first chassis, the third chassis, and the second chassis are arranged in different chassis, and each chassis can be arranged in the same cabinet. A baseboard management controller is arranged in each resource pool, which is responsible for managing the corresponding resource pool. The firmware of the baseboard management controller adopts the OpenBMC architecture. The baseboard management controllers in the first chassis, the third chassis, and the second chassis are all connected to the network switch to form a local area network. The pooled management controller can communicate with the baseboard management controllers in each resource pool to realize the management and control of the entire system.

[0101] Specifically, the first chassis 1 includes at least one first chassis 10;

[0102] Wherein, any first chassis 10 in at least one first chassis has: a first network port 10 L and the first communication port 10 IO ;

[0103] The first network port 10 of any first chassis 10 L After interconnection, it acts as the first chassis network port 1 L ;

[0104] The first communication port 10 of any first chassis 10 IO , as at least one first input / output port 1 IO One of the first input and output ports 1 IO .

[0105] The first chassis 10 includes: a first baseboard management controller 110, a central processing unit 120 and a first connector 130;

[0106] The first baseboard management controller 110 is connected to the central processor 120, and the central processor 120 is connected to the first connector 130. The first connector 130 serves as the first chassis communication port 10 IO ;

[0107] The first baseboard management controller 110 has a first baseboard management controller network port 110 L , as the first chassis network port 10 L .

[0108] Specifically, the third chassis 2 includes: a third baseboard management controller 21, a transmission expansion device 22, and a switching unit 23;

[0109] The third baseboard management controller 21 has: a third baseboard management controller network port 21 L ;

[0110] The switching unit 23 is connected to the third baseboard management controller 21 through the transmission expansion device 22; the third baseboard management controller network port 21 L As the third chassis network port 2 L ;

[0111] The switching unit 23 also has: at least one computing communication port 23 IO1 , and at least one heterogeneous communication port 23 IO2 ;

[0112] At least one computing communication port 23 IO1The number of second input and output ports with at least one 2 IO1 The number corresponds to;

[0113] At least one heterogeneous communication port 23 IO2 The number of input and output ports with at least one third 2 IO2 The number corresponds to .

[0114] The switching unit 23 includes at least one switching device 230;

[0115] At least one switching device 230 is fully connected, such as Figure 2 shown. Figure 2 The figure shows the fully connected form between 8 switching devices.

[0116] Any of the at least one switching device 230 has:

[0117] Host port H, used as computing communication port 23 IO1 ;

[0118] Device port D, used as heterogeneous communication port 23 IO2 ;

[0119] The internal interconnection port F is used to connect at least one switching device to a full connection.

[0120] The switching unit includes several layers of PCIe switching boards, each layer of PCIe switching boards has two switching devices, which are input and output devices for expanding a group of PCIe signals into multiple groups of PCIe signals, and are used to expand, network, and allocate PCIe resources in the first chassis. Each switching device can lead to 5 high-speed signal ports (preferably CDFP ports) for connecting to the first chassis and the second chassis to form a fixed topology. Preferably, a CDFP cable is used to connect the high-speed signal port to the first and second chassis.

[0121] The pooling management controller uses I 2 The C link is connected to the switch device. The firmware of the switch device provides basic read and write register functions, pool management controller, and 2 The C-link sends commands to the switching device to read and write the internal registers of the switching device. This enables unified management and configuration of the switching device. This enables monitoring of device temperature, port traffic information, port type, and device routing information.

[0122] Specifically, the second chassis 3 includes at least one second chassis 30;

[0123] Among them, any second chassis 30 in at least one second chassis 30 has: a second network port 30 L and at least one second communication port 30IO ;

[0124] The second network port 30 of any second chassis 30 L After interconnection, it can be used as the second chassis network port 3 L ;

[0125] Any communication port 30 of the second chassis 30 IO , as a fourth input / output port 3 of at least one fourth input / output port IO .

[0126] Further, the second chassis 30 includes: a second baseboard management controller 310, at least one graphics processor 320 and at least one third connector 330;

[0127] The second baseboard management controller 310 is connected to any one of the at least one graphics processor 320, and any one of the at least one graphics processor 320 is connected to a third connector 330 of the at least one third connector;

[0128] Any third connector 330 of the at least one third connector serves as a fourth input / output port 330 of the at least one fourth input / output port. IO .

[0129] The first baseboard management controller 110 , the third baseboard management controller 21 and the second baseboard management controller 310 are all connected to the network switch 4 to form a local area network. Data can be transmitted between the first baseboard management controller 110 , the third baseboard management controller 21 and the second baseboard management controller 310 through the network switch 4 .

[0130] In other embodiments, Figure 3 As shown, the method for locating a faulty device in a server system is applied to the server system described in the first aspect, and includes:

[0131] S100: In response to detecting a link failure, triggering a system management interruption;

[0132] S200: parsing the advanced error reporting register of the faulty device to determine the fault information of the faulty device;

[0133] S300: Determine device asset information of the faulty device according to the fault information of the faulty device;

[0134] S400: In response to the device asset information of the faulty device not including the silk screen information of the faulty device, determining the physical location of the faulty device according to the fault information of the faulty device and an allocation relationship table, wherein the allocation relationship table includes at least: a dynamic mapping relationship and a fixed interconnection relationship.

[0135] The fault information of the faulty device includes at least: link identification information of the faulty device;

[0136] Link identification information includes the bus, device, and function numbers of the device in the link. In a PCIe link, link identification information refers to the PCIe link identification information of the device. Usually, a set of BDF values ​​is used to represent the link identification information of the device in the PCIe link. Schematically, the bus number of the graphics processor device is: 0000, the device number is 02, and the function number is 00.0. The link identification information of the graphics processor device in the PCIe link, that is, BDF, can be expressed as: 0000:02:00.0. Usually, the link identification information of the device can be obtained through the lspci command.

[0137] However, since the GPU resources and the CPU resources are respectively set in different chassis, the BIOS cannot record the GPU slots that are not in the same chassis in the device asset information. Therefore, it is necessary to use the link identification information and the allocation relationship table to determine the physical location of the faulty GPU.

[0138] Usually, when the entire system is powered on, the basic input and output system will enumerate and process PCIe devices, and assign a set of link identification information (BDF) to each PCIe device. After that, the basic input and output system will assemble the identified asset information into a Json data format and write it to the shared memory, including the link identification information of the PCIe device in the first chassis and the slot silkscreen information corresponding to the device, and the link identification information of the allocated graphics processor device. The first baseboard management controller obtains the asset information reported by the basic input and output system from the shared memory and stores it in the file system of the baseboard management controller, schematically, saved as the assetInfo.json file.

[0139] Determine the location of the faulty device based on the fault information of the faulty device, including:

[0140] S410: Determine the port number of the switching device according to the link identification information of the faulty device;

[0141] S420: Determine a third communication port based on the port number of the switching device and the dynamic mapping relationship;

[0142] S430: Based on the third communication port and the fixed interconnection relationship, determine the fourth communication port and the faulty device slot connected to the corresponding fourth communication port, and then determine the physical location of the faulty device.

[0143] In response to detecting a link failure, before triggering a system management interrupt, the method further includes:

[0144] S010: Power on the server system, and establish a network connection between the first baseboard management controller, the second baseboard management controller, and the third baseboard management controller;

[0145] S020: Obtain a fixed interconnection relationship, and configure the fixed interconnection relationship as a configuration file in a third baseboard management controller;

[0146] S030: Obtain a dynamic configuration relationship between the third chassis and the second chassis;

[0147] S040: Determine an allocation relationship table according to the fixed interconnection relationship and the dynamic configuration relationship.

[0148] The allocation relationship table can be pre-configured and saved in the configuration file of the third baseboard management controller and the firmware of the switching device. It can also be configured when the system is powered on and saved in the configuration file of the third baseboard management controller and the firmware of the switching device.

[0149] The fixed interconnection relationship is configured in a configuration file of the third baseboard management controller, including:

[0150] S021: numbering the devices in the first chassis, the third chassis, and the second chassis;

[0151] S022: Associating the switch device port number in the third chassis with the switch device number;

[0152] S023: Associating the port number of the switch device in the third chassis with the first chassis number in the first chassis, the second subsystem number in the second chassis, and the communication port number of any second subsystem;

[0153] S024: Associating the communication port number of any second subsystem with the device slot;

[0154] S025: The above association relationship is regarded as a fixed interconnection relationship.

[0155] The dynamic configuration relationship between the third chassis and the second chassis is obtained, including:

[0156] S031: Acquire the internal interconnection relationship between the port numbers of the switching device maintained by the firmware of the switching device;

[0157] S032: Acquire a mapping relationship between a switch device port number and a device link identifier maintained by the firmware of the switch device;

[0158] S033: Acquire a mapping relationship between asset information of any graphics processor of at least one graphics processor maintained by a third baseboard management controller and at least one third connector;

[0159] S034: taking the internal interconnection relationship, the mapping relationship between the switch device port number and the device link identifier, and the mapping relationship between the asset information of any graphics processor of the at least one graphics processor and the at least one third connector as a dynamic mapping relationship.

[0160] The method for locating a faulty device in a server system further includes:

[0161] S500: Analyze the fault information of the faulty device to obtain the fault type and fault level of the faulty device, wherein the fault type at least includes: unrecoverable fault, and the fault level at least includes: non-fatal fault and fatal fault;

[0162] S600: In response to the fault type of the faulty device being an unrecoverable fault and the fault level of the faulty device being a non-fatal fault, replacing the faulty device with an idle device of the same type in the resource pool where the faulty device is located;

[0163] S700: Update the allocation relationship table.

[0164] Further, S700ʹ: in response to the fault type of the faulty device being an unrecoverable fault and the fault level of the faulty device being a fatal fault, a device fault alarm is issued.

[0165] The system management is based on the server system structure. The first baseboard management controller, the second baseboard management controller and the third baseboard management controller have LLDP functions. The pooled management controller can find the IP information of the corresponding baseboard management controller according to the broadcast of each baseboard management controller.

[0166] In order to maintain the allocation relationship between the system graphics processor and the computing resource pool, the hardware devices need to be numbered.

[0167] Schematically, HostBox_ID numbers are set for the first chassis, and are numbered from 0 to 7 respectively.

[0168] The graphics card resource pool sets the GPUBox_ID number to 0 and 1 respectively.

[0169] Set the Switch_CDFP_ID number for the CDFP port of the switch device as follows:

[0170] First layer: CDFP0-0~CDFP0-4; CDFP1-0~CDFP1-4;

[0171] The second layer: CDFP2-0~CDFP2-4; CDFP3-0~CDFP3-4;

[0172] The third layer: CDFP4-0~CDFP4-4; CDFP5-0~CDFP5-4;

[0173] Fourth layer: CDFP6-0~CDFP6-4; CDFP7-0~CDFP7-4.

[0174] Set the Switch_Port number for the switch device port defined by the firmware in the switch device, such as 0x200060 corresponding to a group of registers inside the switch device.

[0175] Set the Box_CDFP_ID number for the third connector, such as CDFP0-CDFP15.

[0176] Through the above numbering settings, the mapping relationship between Switch_CDFP_ID and Switch_Port, the mapping relationship between Switch_CDFP_ID and HostBox_ID, GPUBox_ID, Box_CDFP_ID, and the mapping relationship between Box_CDFP_ID and the silk screen of the graphics processor slot are configured as fixed interconnection relationships in the configuration file of the third baseboard management controller.

[0177] The internal interconnection relationship between the switching device port numbers maintained by the switching device firmware, the mapping relationship between the switching device port numbers and the device link identifiers maintained by the switching device firmware, and the mapping relationship between the asset information of any graphics processor in at least one graphics processor maintained by the third baseboard management controller and at least one third connector are dynamically maintained by the firmware of the switching chip as dynamic configuration relationships.

[0178] On the basis of having the latest numbering information mentioned above, when a graphics processor fails, the switch device port (Switch_Port) number is first determined according to the link identification information, the internal interconnection relationship of the switch device is determined according to the switch device port number, the CDFP port number of the switch device is determined according to the internal interconnection relationship of the switch device, and then the third network port connected thereto is determined, thereby determining the slot of the graphics processor.

[0179] The pooled management controller, i.e., the third baseboard management controller, aggregates the fixed interconnection relationship and the dynamic configuration relationship to form an allocation relationship table. Schematically, taking the Json format data structure as an example, the allocation relationship between the graphics processor device and the computing resources is shown as follows:

[0180] {

[0181] "HostBox_ID": "0",

[0182] "Switch_CDFP_ID": "CDFP0-0",

[0183] "Switch_Port": "0x000040",

[0184] "Device": [

[0185] {

[0186] "GPUBox_ID": "0",

[0187] "Slot": "GPU3",

[0188] "Switch_CDFP_ID": "CDFP0-1",

[0189] "Switch_Port": "0x000050",

[0190] "Box_CDFP_ID": "CDFP0",

[0191] "BusNumber": XX,

[0192] "DeviceNumber": XX,

[0193] "FunctionNumber": XX,

[0194] "AssetInfo": {

[0195] "BiosVersion": "XX",

[0196] "BuildDate": "XX",

[0197] "DeviceID": "XX",

[0198] "FwVersion": "XX",

[0199] "Manufacturer": "XX",

[0200] "Model": "XX",

[0201] "PartNumber": "XX",

[0202] "SerialNumber": "XX",

[0203] "SubSystemID": "XX",

[0204] "UUID": "XX",

[0205] "vendorID": "XX"

[0206] }

[0207] },

[0208] {

[0209] "GPUBox_ID": "1",

[0210] "Slot": "GPU5",

[0211] "Switch_CDFP_ID": "CDFP3-2",

[0212] "Switch_Port": "0x400060",

[0213] "Box_CDFP_ID": "CDFP1",

[0214] "BusNumber": YY,

[0215] "DeviceNumber": YY,

[0216] "FunctionNumber": YY,

[0217] "AssetInfo": {

[0218] "BiosVersion": "XX",

[0219] "BuildDate": "XX",

[0220] "DeviceID": "XX",

[0221] "FwVersion": "XX",

[0222] "Manufacturer": "XX",

[0223] "Model": "XX",

[0224] "PartNumber": "XX",

[0225] "SerialNumber": "XX",

[0226] "SubSystemID": "XX",

[0227] "UUID": "XX",

[0228] "vendorID": "XX"

[0229] }

[0230] } ]

[0232] }

[0233] The pooling management controller provides a display interface and Redfish interface for adjusting GPU resource allocation and adjustment. For example, to remove GPU 0 from the computing resource with HostBox_ID 0 and allocate it to the computing resource with HostBox_ID 1, you need to call I through the Redfish interface. 2 C command interacts with the firmware of the switch device to set the corresponding switch device register. After the setting is successful, the pooling management controller updates the allocation relationship table to ensure the correct allocation relationship.

[0234] It should be understood that although Figure 3 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 3 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0235] In some other embodiments, a method for locating a faulty device in a server system is applied to a server system, wherein the server system includes: at least one central processing unit 120, at least one graphics processing unit 320, at least one switching device 230, and a network switch 4; the at least one central processing unit 120 is arranged in at least one first chassis 10, the at least one graphics processing unit 320 is arranged in at least one second chassis 30, and the at least one switching device 230 is arranged in a third chassis 2; the at least one first chassis 10, the at least one second chassis 30 and the third chassis 2 are all arranged in the same server cabinet; any first chassis 10 of the at least one first chassis 10, any second chassis 30 of the at least one second chassis 30 and the third chassis 2 are respectively connected to the network switch 4 through a first cable; any first chassis 10 of the at least one first chassis 10 and any second chassis 30 of the at least one second chassis 30 are respectively connected to the third chassis 2 through a second cable, and the third chassis 2 includes a third baseboard management controller 21.

[0236] Methods for locating faulty equipment in the server system include:

[0237] triggering a system management interrupt in response to the at least one central processor detecting a server link failure;

[0238] Parsing the advanced error reporting register of the faulty device to determine the fault information of the faulty device;

[0239] Determine the equipment asset information of the faulty equipment according to the fault information of the faulty equipment;

[0240] In response to the fact that the device asset information of the faulty device does not include the slot silk-screen information of the faulty device, the physical location of the faulty device is determined by the third baseboard management controller according to the fault information of the faulty device and an allocation relationship table, wherein the allocation relationship table at least includes: a dynamic mapping relationship and a fixed interconnection relationship, the dynamic mapping relationship includes the connection relationship between the at least one switching device, and the fixed interconnection relationship includes the correspondence between the slot silk-screen information and the at least one second communication port.

[0241] In addition, any of the at least one first chassis 10 has: a first network port 10 L and the first communication port 10 IO Any of the at least one second chassis 30 has: a second network port 30 L , at least one second communication port 30 IO The third chassis 2 has: a third network port (third baseboard management controller network port 21 L ), at least one third communication port (computing communication port 23 IO1 ) and at least one fourth communication port (heterogeneous communication port 23 IO2 ); any third communication port of the at least one third communication port and one of the first communication ports 10 IO Any of the at least one fourth communication port and the at least one second communication port 30 are connected to each other through a corresponding connection of the second cable. IO A second communication port 30 IO The first network port 10 is connected via a corresponding cable. L , the second network port 30 L and the third network ports are respectively connected to the network switch 4 through the first cable; the fault information of the faulty device includes the link identification information of the faulty device;

[0242] At this time, determining the physical location of the faulty device according to the fault information of the faulty device and the allocation relationship table includes:

[0243] Determine the port number of the switching device according to the link identification information of the faulty device;

[0244] Determine a fourth communication port based on the port number and the dynamic mapping relationship of the switching device;

[0245] Based on the fourth communication port and the fixed interconnection relationship, the second communication port and the faulty device slot silkscreen information connected to the corresponding second communication port are determined, and then the faulty device slot is determined.

[0246] Further, any first chassis 10 in at least one first chassis 10 includes: a first baseboard management controller 110, a central processing unit 120 and a first connector 130; the first baseboard management controller 110 is connected to the central processing unit 120, and the central processing unit 120 is connected to the first connector 130; the first connector 130 serves as the first communication port; the first baseboard management controller 110 has a first baseboard management controller network port, and the first baseboard management controller network port serves as the first network port; any second chassis 30 in at least one second chassis 30 includes: a second baseboard management controller 310, at least one graphics processor 320 and at least one second connector 330; the second baseboard management controller 310 is connected to the at least one graphics processor 320, and any graphics processor 320 in the at least one graphics processor 320 is connected to one of the at least one second connectors 330; the at least one second connector 330 serves as the at least one second communication port; The second baseboard management controller 310 has a second baseboard management controller network port, and the second baseboard management controller network port serves as the second network port; the third chassis 2 also includes: a transmission extension device 22, a switching unit 23, at least one third connector and at least one fourth connector; the switching unit 23 is connected to the third baseboard management controller 21 through the transmission extension device; the third baseboard management controller 21 has a third baseboard management controller network port, and the third baseboard management controller network port serves as the third network port; the switching unit 23 has at least one computing communication port and at least one graphics card communication port; any computing communication port of the at least one computing communication port is connected to a third connector of the at least one third connector, and the at least one third connector serves as the at least one third communication port; any graphics card communication port of the at least one graphics card communication port is connected to a fourth connector of the at least one fourth connector, and the at least one fourth connector serves as the at least one fourth communication port;

[0247] At this time, before the system management interrupt is triggered in response to detecting a link failure, the method further includes:

[0248] Powering on the server system, and establishing a network connection between the first baseboard management controller, the second baseboard management controller, and the third baseboard management controller;

[0249] Acquire the fixed interconnection relationship, and configure the fixed interconnection relationship in a configuration file of the third baseboard management controller;

[0250] Acquire a connection relationship of at least one switching device in the third chassis as a dynamic configuration relationship;

[0251] An allocation relationship table is determined according to the fixed interconnection relationship and the dynamic configuration relationship.

[0252] Further, configuring the fixed interconnection relationship in a configuration file of the third baseboard management controller includes:

[0253] Numbering hardware devices in at least one first chassis, at least one second chassis, and a third chassis; wherein the hardware devices include: a central processing unit, a graphics processing unit, a switching device, a first connector, a second connector, a third connector, and a fourth connector;

[0254] Numbering the ports of the switching device;

[0255] Associating the port of the switching device with the numbers of the third connector and the fourth connector;

[0256] Associating the number of the third connector with the number of the first connector, and associating the number of the fourth connector with the number of the second connector;

[0257] Associating the serial number of the second connector with the slot silkscreen information;

[0258] The above-mentioned relationship is regarded as a fixed interconnection relationship.

[0259] Further, obtaining the connection relationship of at least one switching device in the third chassis as a dynamic configuration relationship includes:

[0260] Obtain the internal interconnection relationship between switching devices;

[0261] Obtaining a mapping relationship between a port number of a switching device and a device link identifier;

[0262] Acquire a mapping relationship between asset information of any graphics processor of at least one graphics processor and at least one second connector;

[0263] The internal interconnection relationship, the mapping relationship between the port number of the switching device and the device link identifier, and the mapping relationship between the asset information of any graphics processor of the at least one graphics processor and at least one third connector are used as dynamic mapping relationships.

[0264] Furthermore, the fault location method further includes: parsing the fault information of the faulty device, obtaining the fault type and fault level of the faulty device, wherein the fault type at least includes: an unrecoverable fault, and the fault level at least includes: a non-fatal fault and a fatal fault;

[0265] In response to the fault type of the faulty device being an unrecoverable fault and the fault level of the faulty device being a non-fatal fault, the faulty device is reset.

[0266] Further, in response to the fault type of the faulty device being an unrecoverable fault and the fault level of the faulty device being a fatal fault, a device fault alarm is issued.

[0267] Device fault alarm, including device fault log.

[0268] The fault log contains the chassis where the failed PCIe device is located and the slot position in the chassis.

[0269] Based on the equipment failure alarm, operation and maintenance personnel can promptly determine the physical location of the faulty equipment and replace it.

[0270] In other embodiments, Figure 4 As shown, the server system fault equipment positioning device includes:

[0271] an interrupt triggering module, configured to trigger a system management interrupt in response to at least one central processor detecting a server link failure;

[0272] A fault determination module, used for parsing the advanced error reporting register of the faulty device and determining the fault information of the faulty device;

[0273] An asset identification module, used to determine the equipment asset information of the faulty equipment according to the fault information of the faulty equipment;

[0274] A physical positioning module is used to determine the physical location of the faulty device through a third baseboard management controller in response to the fact that the device asset information of the faulty device does not contain the slot silk-screen information of the faulty device, based on the fault information of the faulty device and an allocation relationship table, wherein the allocation relationship table at least includes: a dynamic mapping relationship and a fixed interconnection relationship, the dynamic mapping relationship includes a connection relationship between at least one switching device, and the fixed interconnection relationship includes a corresponding relationship between the slot silk-screen information and at least one second communication port.

[0275] For the specific limitations of the server system fault equipment locating device described above, please refer to the limitations of the server system fault equipment locating method described above, which will not be repeated here. Each module in the above-mentioned server system fault equipment locating device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0276] In other embodiments, Figure 5 As shown, the computer system includes a memory, a processor, and a server system fault device location program stored in the memory and executable on the processor. When the processor executes the server system fault device location program, the server system fault device location method described above is implemented. The server system fault device location method described above is not repeated here.

[0277] In some other embodiments, a computer program product includes a computer program, and when the computer program is executed by a processor, the method for locating a faulty device in a server system described above is implemented. The method for locating a faulty device in a server system described above will not be described in detail here.

[0278] In other embodiments, a computer-readable storage medium stores a server system fault device location program, and when the server system fault device location program is executed by a processor, the server system fault device location method described above is implemented. The server system fault device location method described above is not repeated here.

[0279] The technical solution provided by the embodiments of the present application has the following beneficial effects: by implementing the server system and its faulty equipment locating method, apparatus, system, computer program product and readable storage medium disclosed in the embodiments of the present application, it is conducive to the flexible reconfiguration and rapid upgrading of key equipment such as CPU and GPU, which can greatly improve the utilization rate of server hardware resources; the equipment with PCIe failure in the whole cabinet can be quickly located, and for GPU equipment, if it is an unrecoverable and non-fatal failure, the GPU equipment can be hot-removed and hot-added, the equipment can be reset, and the normal operation of the business can be restored in time, reducing the maintenance pressure of operation and maintenance personnel.

[0280] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here. Example

[0281] Figure 6 A flow chart showing a method for locating a faulty device in a server system is shown.

[0282] When the entire system is powered on and the host is turned on, the basic input and output system of the first chassis will enumerate and process the PCIe devices, and assign a set of link identification information to each PCIe device. After that, the basic input and output system will assemble the identified asset information into a Json data format and write it to the shared memory, including the link identification information of the PCIe device in the first chassis, the slot silkscreen information corresponding to the device, and the link identification information of the allocated graphics processor device. The first baseboard management controller obtains the asset information reported by the basic input and output system from the shared memory and stores it in the file system of the baseboard management controller, assuming it is saved as the assetInfo.json file.

[0283] When the central processing unit in the first chassis detects a high-level fault report of a certain graphics processing unit device, a system management interrupt is triggered and the interrupt is handed over to the basic input and output system of the first chassis for processing.

[0284] The basic input and output system of the first chassis parses the advanced error reporting register of the faulty graphics processor, determines the error information of the faulty device, and records the link identification information of the faulty device.

[0285] The basic input and output system of the first chassis reports the link identification information and error information of the faulty graphics processor device to the first baseboard management controller through the IPMI interface.

[0286] The first baseboard management controller queries the device asset information sent by the basic input and output system when the computer is turned on according to the link identification information. If the corresponding silk screen information can be found, the device is a PCIe device in the first chassis. Then, the relevant alarm log information is recorded and reported to the pooled resource manager.

[0287] If the corresponding silk-screen information cannot be found, the device is an assigned graphics processor device, and the fault information is reported to the pooled resource manager through the IPMI or Redfish interface. The pooled resource manager queries the allocation relationship table between the entire system graphics processor device and the first chassis according to the HostBox_ID number and link identification information, determines the Box number, silk-screen information, Box CDFP number and graphics processor asset information, and the pooled management controller records the alarm log and synchronizes the log information to the first baseboard management controller.

[0288] The pooling management controller analyzes the error information of the faulty GPU device. If it is an unrecoverable and non-fatal fault, the pooling management controller will 2 C command to interact with the firmware of the switching device to perform hot removal and hot addition operations on the faulty graphics processor device and reset the device.

[0289] If it is a fatal failure, the operation and maintenance personnel can determine the Box number and corresponding slot of the failed GPU device based on the log and repair it. The operation and maintenance personnel can query the log information through the first baseboard management controller and the pooled management controller.

[0290] When allocating GPU resources to the entire system, the allocation relationship table between the GPU device and the first chassis needs to be updated at the same time. For example, when the GPU numbered 0 is removed from the first chassis numbered 0 and allocated to the first chassis numbered 1, the I 2 C command interacts with the firmware of the switching device to set the relevant registers of the corresponding switching firmware. After the setting is successful, the pooling management controller updates the allocation relationship table between the graphics processor device and the first chassis to ensure the accuracy of the allocation relationship and to accurately locate the GPU when a failure occurs. Example

[0291] Figure 7 A flow chart showing another method for locating a faulty device in a server system is shown.

[0292] When the entire system is powered on and the host is turned on, the basic input and output system of the first chassis will enumerate and process the PCIe devices, and assign a set of link identification information to each PCIe device. After that, the basic input and output system will assemble the identified asset information into a Json data format and write it to the shared memory, including the link identification information of the PCIe device in the first chassis, the slot silkscreen information corresponding to the device, and the link identification information of the allocated graphics processor device. The first baseboard management controller obtains the asset information reported by the basic input and output system from the shared memory and stores it in the file system of the baseboard management controller, assuming it is saved as the assetInfo.json file.

[0293] The pooling management controller establishes an allocation relationship table between the graphics processor device and the first chassis, takes the first chassis as a dimension, and sends the allocation relationship to the first baseboard management controller through a redfish interface.

[0294] The first baseboard management controller queries the device asset information assetInfo.json file according to the obtained allocation relationship (including the link identification information) and the link identification information, and sets the silk screen information in the allocation relationship to the corresponding device asset information.

[0295] When the central processing unit in the first chassis detects a high-level fault report of a certain graphics processing unit device, a system management interrupt is triggered and the interrupt is handed over to the basic input and output system of the first chassis for processing.

[0296] The basic input and output system of the first chassis parses the advanced error reporting register of the faulty graphics processor, determines the error information of the faulty device, and records the link identification information of the faulty device.

[0297] The basic input and output system of the first chassis reports the link identification information and error information of the faulty graphics processor device to the first baseboard management controller through the IPMI interface.

[0298] The first baseboard management controller queries the device asset information sent by the basic input and output system when the computer is turned on according to the link identification information. If the corresponding silk screen information can be found, the device is a PCIe device in the first chassis. Then, the relevant alarm log information is recorded and reported to the pooled resource manager.

[0299] If the corresponding silk-screen information cannot be found, the device is an assigned graphics processor device, and the fault information is reported to the pooled resource manager through the IPMI or Redfish interface. The pooled resource manager queries the allocation relationship table between the entire system graphics processor device and the first chassis according to the HostBox_ID number and link identification information, determines the Box number, silk-screen information, Box CDFP number and graphics processor asset information, and the pooled management controller records the alarm log and synchronizes the log information to the first baseboard management controller.

[0300] The pooling management controller analyzes the error information of the faulty GPU device. If it is an unrecoverable and non-fatal fault, the pooling management controller will use I 2 C command interacts with the firmware of the switching device to perform hot removal and hot addition operations on the faulty graphics processor device and reset the device.

[0301] If it is a fatal failure, the operation and maintenance personnel can determine the Box number and corresponding slot of the failed GPU device based on the log and repair it. The operation and maintenance personnel can query the log information through the first baseboard management controller and the pooled management controller.

[0302] When allocating GPU resources to the entire system, the allocation relationship table between the GPU device and the first chassis needs to be updated at the same time. For example, when the connection between the GPU numbered 0 and the first chassis numbered 0 is removed and allocated to the first chassis numbered 1, the I 2 C command interacts with the firmware of the switching device to set the relevant registers of the corresponding switching firmware. After the setting is successful, the pooling management controller updates the allocation relationship table between the graphics processor device and the first chassis to ensure the accuracy of the allocation relationship and to accurately locate the GPU when a failure occurs.

[0303] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program loaded on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a memory, or installed from a ROM. When the computer program is executed by an external processor, the above-mentioned functions defined in the method of the embodiment of the present application are executed.

[0304] It should be noted that the computer-readable medium of the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In an embodiment of the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In an embodiment of the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0305] The computer-readable medium may be included in the server; or it may exist independently without being installed in the server. The computer-readable medium carries one or more programs. When the one or more programs are executed by the server, the server: in response to detecting that the peripheral mode of the terminal is not activated, obtains the frame rate of the application on the terminal; when the frame rate meets the screen-off condition, determines whether the user is obtaining the screen information of the terminal; in response to the judgment result that the user is not obtaining the screen information of the terminal, controls the screen to enter the immediate dimming mode.

[0306] Computer program code for performing the operation of the embodiments of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0307] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0308] The technical solution provided by the present application is described in detail above. The principle and implementation method of the present application are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present application.

[0309] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A server system, characterized in that: include: at least one central processing unit, at least one graphics processing unit, at least one switching device, and a network switch; The at least one central processing unit is arranged in at least one first chassis, the at least one graphics processing unit is arranged in at least one second chassis, the at least one switching device is arranged in a third chassis, and the at least one first chassis, the at least one second chassis and the third chassis are all arranged in the same server cabinet; Any first chassis of the at least one first chassis, any second chassis of the at least one second chassis, and the third chassis are respectively connected to the network switch through a first cable; Any first chassis of the at least one first chassis and any second chassis of the at least one second chassis are respectively connected to the third chassis through a second cable, and the third chassis includes a third baseboard management controller; Any first chassis of the at least one first chassis has: a first network port and a first communication port; Any second chassis of the at least one second chassis has: a second network port, at least one second communication port; The third chassis has: a third network port, at least one third communication port and at least one fourth communication port; Any third communication port of the at least one third communication port is connected to one of the first communication ports via a first cable, any fourth communication port of the at least one fourth communication port is connected to one of the at least one second communication ports via a first cable, and the first network port, the second network port and the third network port are respectively connected to the network switch via a second cable; the fault information of the faulty device includes link identification information of the faulty device; The server system is configured as follows: triggering a system management interrupt in response to the at least one central processor detecting a server link failure; Parsing the advanced error reporting register of the faulty device to determine the fault information of the faulty device; Determine the equipment asset information of the faulty equipment according to the fault information of the faulty equipment; In response to the device asset information of the faulty device not including the slot silkscreen information of the faulty device, determining the physical location of the faulty device through the third baseboard management controller according to the fault information of the faulty device and the allocation relationship table, wherein the allocation relationship table at least includes: a dynamic mapping relationship and a fixed interconnection relationship, the dynamic mapping relationship includes the connection relationship between the at least one switching device, and the fixed interconnection relationship includes the corresponding relationship between the slot silkscreen information and the at least one second communication port; Wherein, determining the physical location of the faulty device according to the fault information of the faulty device and the allocation relationship table includes: Determine the port number of the switching device according to the link identification information of the faulty device; Determine a fourth communication port based on the port number and the dynamic mapping relationship of the switching device; Based on the fourth communication port and the fixed interconnection relationship, the second communication port and the faulty device slot silkscreen information connected to the corresponding second communication port are determined, and then the faulty device slot is determined.

2. The server system according to claim 1, characterized in that: Any first chassis of the at least one first chassis comprises: a first baseboard management controller, a central processing unit and a first connector; The first baseboard management controller is connected to the central processing unit, and the central processing unit is connected to the first connector; The first connector serves as the first communication port; The first baseboard management controller has a first baseboard management controller network port, and the first baseboard management controller network port serves as the first network port.

3. The server system according to claim 2, characterized in that: The first connector is a high-speed pluggable input / output connector.

4. The server system according to claim 1, characterized in that: Any second chassis of the at least one second chassis comprises: a second baseboard management controller, at least one graphics processor and at least one second connector; The second baseboard management controller is connected to the at least one graphics processor, and any graphics processor of the at least one graphics processor is connected to a second connector of the at least one second connector in a one-to-one correspondence; The at least one second connector serves as the at least one second communication port; The second baseboard management controller has a second baseboard management controller network port, and the second baseboard management controller network port serves as the second network port.

5. The server system according to claim 4, characterized in that: The second connector is a high-speed pluggable input / output connector.

6. The server system according to claim 1, characterized in that: The third chassis further comprises: a transmission expansion device, a switching unit, at least one third connector and at least one fourth connector; The switching unit is connected to the third baseboard management controller via the transmission expansion device; The third baseboard management controller has a third baseboard management controller network port, and the third baseboard management controller network port serves as the third network port; The switching unit has at least one computing communication port and at least one graphics card communication port; Any one of the at least one computing communication port is connected to one of the at least one third connectors, and the at least one third connector serves as the at least one third communication port; Any one of the at least one graphics card communication port is connected to one of the at least one fourth connectors, and the at least one fourth connector serves as the at least one fourth communication port.

7. The server system according to claim 6, characterized in that: The switching unit includes at least one switching device; The at least one switching device is fully connected.

8. The server system according to claim 7, characterized in that: Any of the at least one switching device has: A host port, serving as the computing communication port; A device port, serving as a communication port for the graphics card; The internal interconnection port is used to connect the at least one switching device to a full connection.

9. The server system according to any one of claims 6 to 8, characterized in that: The third connector and the fourth connector are high-speed pluggable input / output connectors.

10. The server system according to claim 1, characterized in that The second cable is a CDFP high-speed cable.

11. A method for locating a faulty device in a server system, characterized in that: The server system comprises at least one central processing unit, at least one graphics processing unit, at least one switching device, and a network switch; The at least one central processing unit is arranged in at least one first chassis, the at least one graphics processing unit is arranged in at least one second chassis, and the at least one switching device is arranged in a third chassis; the at least one first chassis, the at least one second chassis and the third chassis are all arranged in the same server cabinet; Any first chassis of the at least one first chassis, any second chassis of the at least one second chassis, and the third chassis are respectively connected to the network switch through a first cable; Any first chassis of the at least one first chassis and any second chassis of the at least one second chassis are respectively connected to the third chassis through a second cable, and the third chassis includes a third baseboard management controller; Any first chassis of the at least one first chassis has: a first network port and a first communication port; Any second chassis of the at least one second chassis has: a second network port, at least one second communication port; The third chassis has: a third network port, at least one third communication port and at least one fourth communication port; Any third communication port of the at least one third communication port is connected to one of the first communication ports via a first cable, any fourth communication port of the at least one fourth communication port is connected to one of the at least one second communication ports via a first cable, and the first network port, the second network port and the third network port are respectively connected to the network switch via a second cable; The fault information of the faulty device includes link identification information of the faulty device; The method comprises: triggering a system management interrupt in response to the at least one central processor detecting a server link failure; Parsing the advanced error reporting register of the faulty device to determine the fault information of the faulty device; Determine the equipment asset information of the faulty equipment according to the fault information of the faulty equipment; In response to the device asset information of the faulty device not including the slot silkscreen information of the faulty device, determining the physical location of the faulty device through the third baseboard management controller according to the fault information of the faulty device and the allocation relationship table, wherein the allocation relationship table at least includes: a dynamic mapping relationship and a fixed interconnection relationship, the dynamic mapping relationship includes the connection relationship between the at least one switching device, and the fixed interconnection relationship includes the corresponding relationship between the slot silkscreen information and the at least one second communication port; Wherein, determining the physical location of the faulty device according to the fault information of the faulty device and the allocation relationship table includes: Determine the port number of the switching device according to the link identification information of the faulty device; Determine a fourth communication port based on the port number and the dynamic mapping relationship of the switching device; Based on the fourth communication port and the fixed interconnection relationship, the second communication port and the faulty device slot silkscreen information connected to the corresponding second communication port are determined, and then the faulty device slot is determined.

12. The method for locating a faulty device in a server system according to claim 11, characterized in that: Any first chassis in the at least one first chassis includes: a first baseboard management controller, a central processing unit and a first connector; the first baseboard management controller is connected to the central processing unit, and the central processing unit is connected to the first connector; the first connector serves as the first communication port; the first baseboard management controller has a first baseboard management controller network port, and the first baseboard management controller network port serves as the first network port; any second chassis in the at least one second chassis includes: a second baseboard management controller, at least one graphics processor and at least one second connector; the second baseboard management controller is connected to the at least one graphics processor, and any graphics processor in the at least one graphics processor is connected to a second connector in the at least one second connector; the at least one second connector serves as the at least one second communication port; the second baseboard management controller has a second baseboard management controller network port , the second baseboard management controller network port serves as the second network port; the third chassis further comprises: a transmission extension device, a switching unit, at least one third connector and at least one fourth connector; the switching unit is connected to the third baseboard management controller through the transmission extension device; the third baseboard management controller has a third baseboard management controller network port, and the third baseboard management controller network port serves as the third network port; the switching unit has at least one computing communication port and at least one graphics card communication port; any computing communication port of the at least one computing communication port is connected to a third connector of the at least one third connector, and the at least one third connector serves as the at least one third communication port; any graphics card communication port of the at least one graphics card communication port is connected to a fourth connector of the at least one fourth connector, and the at least one fourth connector serves as the at least one fourth communication port; Before the system management interrupt is triggered in response to detecting a link failure, the method further includes: Powering on the server system, and establishing a network connection between the first baseboard management controller, the second baseboard management controller, and the third baseboard management controller; Acquire the fixed interconnection relationship, and configure the fixed interconnection relationship in a configuration file of the third baseboard management controller; Acquire a connection relationship of at least one switching device in the third chassis as a dynamic configuration relationship; An allocation relationship table is determined according to the fixed interconnection relationship and the dynamic configuration relationship.

13. The method for locating a faulty device in a server system according to claim 12, characterized in that: The configuration file for configuring the fixed interconnection relationship in the third baseboard management controller includes: Numbering hardware devices in at least one first chassis, at least one second chassis, and a third chassis; wherein the hardware devices include: a central processing unit, a graphics processing unit, a switching device, a first connector, a second connector, a third connector, and a fourth connector; Numbering the ports of the switching device; Associating the port of the switching device with the numbers of the third connector and the fourth connector; Associating the number of the third connector with the number of the first connector, and associating the number of the fourth connector with the number of the second connector; Associating the serial number of the second connector with the slot silkscreen information; The above-mentioned relationship is regarded as a fixed interconnection relationship.

14. The method for locating a faulty device in a server system according to claim 12, characterized in that: The obtaining of a connection relationship of at least one switching device in the third chassis as a dynamic configuration relationship includes: Obtain the internal interconnection relationship between switching devices; Obtaining a mapping relationship between a port number of a switching device and a device link identifier; Acquire a mapping relationship between asset information of any graphics processor of at least one graphics processor and at least one second connector; The internal interconnection relationship, the mapping relationship between the port number of the switching device and the device link identifier, and the mapping relationship between the asset information of any graphics processor of the at least one graphics processor and at least one third connector are used as dynamic mapping relationships.

15. The method for locating a faulty device in a server system according to claim 11, characterized in that: The method further comprises: Parsing the fault information of the faulty device to obtain the fault type and fault level of the faulty device, wherein the fault type at least includes: an unrecoverable fault, and the fault level at least includes: a non-fatal fault and a fatal fault; In response to the fault type of the faulty device being an unrecoverable fault and the fault level of the faulty device being a non-fatal fault, the faulty device is reset.

16. The method for locating a faulty device in a server system according to claim 15, characterized in that: In response to the fault type of the faulty device being an unrecoverable fault and the fault level of the faulty device being a fatal fault, a device fault alarm is issued.

17. A computer system, characterized in that: The method comprises a memory, a processor and a server system fault device locating program stored in the memory and executable on the processor. When the processor executes the server system fault device locating program, the server system fault device locating method described in any one of claims 11 to 16 is implemented.

18. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for locating a faulty device in a server system as described in any one of claims 11 to 16 is implemented.

19. A computer-readable storage medium, characterized in that: A server system faulty device locating program is stored thereon, and when the server system faulty device locating program is executed by a processor, the server system faulty device locating method according to any one of claims 11 to 16 is implemented.

Images