A method and a server system for implementing live operation of a processor
By working in concert with the baseboard management controller and FPGA, the system identifies and takes offline processor faults, offloading computing tasks to the DPU. This solves the problem of uninterrupted service when replacing a processor in a multi-processor shared DPU server, enabling seamless processor replacement and continuous service operation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INSPUR SUZHOU INTELLIGENT TECH CO LTD
- Filing Date
- 2023-06-08
- Publication Date
- 2026-07-31
AI Technical Summary
In servers with multiple processors sharing a DPU, how can we perform live replacement of a faulty processor without interrupting business operations, thus ensuring continuous service operation?
Through the collaborative work of the baseboard management controller and FPGA, processor faults are identified and offline requests are triggered. The computing power tasks are offloaded to the DPU, the power supply signal is pulled low simultaneously, the hardware link of the processor is interrupted, and with the assistance of the business layer software, the network and storage computing power are migrated to the DPU, realizing seamless replacement of the processor.
It enables seamless processor replacement, ensuring continuous business operation. The design is reliable and the structure is simple, making it suitable for live maintenance of multi-channel or multi-node servers.
Smart Images
Figure CN116610430B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of server technology, and more specifically to a method and server system for implementing live processor maintenance. Background Technology
[0002] The customer is considering a design that can house multiple compute nodes within a single chassis, primarily to allow all nodes to share a single DPU. Currently, DPUs are very expensive, and because the computing power of smart network interface cards (NICs) increases much faster than that of processors, it is not necessary for each compute node to be equipped with its own DPU. Instead, the design allows multiple compute nodes to share a single DPU, which can significantly reduce the hardware cost of the DPU allocated to each node or processor.
[0003] The DPU itself requires power from the server when the server is powered off, even when the server is plugged in with the AC power cord. This ensures the DPU doesn't lose power when the entire server is shut down, allowing customer business and management functions to run on the DPU. The DPU handles network offloading for cloud users and allocates and manages cloud disks and computing resources. Multi-processor or multi-node servers need to be able to perform individual hot maintenance. This requires live maintenance of critical components on individual nodes, especially processors, to ensure continuous business operation and enable seamless replacement of critical components, particularly processors.
[0004] After multiple processors share a DPU, there is a need to migrate computing power between the DPU and the processor. Under the premise of ensuring smooth migration, how to design the server to operate under power and meet the customer's need to replace the faulty processor without interrupting business is the technical problem to be solved by this application. Summary of the Invention
[0005] After multiple processors share a DPU, there is a need to migrate computing power between the DPU and the processor. Under the premise of ensuring smooth migration, how to design the server to operate under power and meet the customer's need to replace the faulty processor without interrupting business? This invention provides a highly reliable and flexible operation and maintenance server based on domestic processors and its implementation method.
[0006] In a first aspect, the present invention provides a method for implementing live maintenance of a processor, comprising the following steps:
[0007] After the processor experiences an unrecoverable error and offloads the computing task from the processor to the DPU, the processor generates a system management interrupt.
[0008] After the baseboard management controller identifies the processor's fault information and obtains the system management interrupt, it triggers the FPGA to send a shutdown request to the processor that generated the fault information.
[0009] After receiving the processor's response based on the offline request, the FPGA triggers the processor to reset and simultaneously pulls the processor power supply signal low.
[0010] After the processor power supply signals are all pulled low, the BIOS sends a system control interrupt to the operating system and stops the processor from operating its external I / O ports;
[0011] After the operating system recognizes the system control interrupt sent by the BIOS through its own driver, it stops any information interaction and business communication between the operating system and the processor, and completes the interrupt operation of the processor hardware link.
[0012] When an unrecoverable error occurs in the processor, the BIOS collects information from the processor's internal registers. The processor's internal registers have different record formats for different faults. After the BIOS collects this fault information and reads the system management interrupt issued by the processor, it sends it to the baseboard management controller via IPMI command. The baseboard management controller stores the fault in the log and at the same time triggers the FPGA to enable the processor to go offline.
[0013] As a further limitation of the technical solution of the present invention, after the processor experiences an unrecoverable error and offloads the computing task from the processor to the DPU, the step of the processor generating a system management interrupt includes:
[0014] After the processor experiences an unrecoverable error and offloads the computing power task from the processor to the DPU, the business layer software informs the processor through the operating system that the computing power migration is complete, and the processor generates a system management interrupt.
[0015] When a server is running services and its processor has an unrecoverable error requiring replacement, to avoid service interruption, the network computing, I / O computing, and storage computing tasks currently performed on the processor are offloaded to applications on the Data Processing Unit (DPU). In conjunction with these applications, the DPU can efficiently complete these tasks. This means that tasks that would normally require establishing virtual machines and integrating software on the server host with the processor for data transmission, analysis, and processing are moved to the DPU. The network and storage computing power consumed by the processor on the server host is taken over by the DPU. In other words, when the processor encounters an unrecoverable or fatal error requiring shutdown, the system first works with the business layer software running on the server operating system to offload the currently running services, especially network and storage computing power, to the DPU.
[0016] As a further limitation of the technical solution of the present invention, after the baseboard management controller identifies the processor's fault information and obtains the system management interrupt, the step of triggering the FPGA to issue a shutdown request to the processor that generated the fault information includes:
[0017] BIOS collects information from the processor's internal registers;
[0018] When the BIOS collects the processor's fault information and reads the system management interrupt issued by the processor, it sends it to the baseboard management controller via IPMI command;
[0019] After the baseboard management controller obtains the processor's fault information and receives the system management interrupt from the BIOS, it records the obtained fault information to the log and triggers the FPGA to send a shutdown request to the processor that generated the fault information.
[0020] As a further limitation of the technical solution of the present invention, after the baseboard management controller obtains the processor's fault information and receives the system management interrupt from the BIOS, it records the obtained fault information to the log and simultaneously triggers the FPGA to issue a shutdown request to the processor that generated the fault information. The steps include:
[0021] The baseboard management controller actively and periodically polls the interrupt register information inside the FPGA;
[0022] The baseboard management controller determines whether a fatal fault has occurred in the processor based on the acquired interrupt register information;
[0023] If so, after the baseboard management controller receives the system management interrupt issued by the processor from the BIOS, it obtains information from the processor's internal registers through PECI; among them, the processor's internal registers have different recording formats for different faults;
[0024] The baseboard management controller records the acquired fault information to the log and simultaneously triggers the FPGA to send a shutdown request to the processor that generated the fault information.
[0025] If not, after the baseboard management controller receives a system management interrupt from the BIOS, it will record the fault information obtained from the BIOS into the log and simultaneously trigger the FPGA to send a shutdown request to the processor that generated the fault information.
[0026] The baseboard management controller actively and periodically polls the interrupt register information inside the FPGA. When the baseboard management controller reads that the processor has a fatal fault and obtains the system management interrupt issued by the processor from the BIOS, it will obtain the information of the processor's internal registers through the PECI bus and record the fault information in the log. At the same time, it will trigger the FPGA to enable the processor to go offline.
[0027] As a further limitation of the technical solution of the present invention, after the processor generates a system management interrupt following the step of the processor generating an unrecoverable error and completing the migration of computing power from the processor to the DPU, the method further includes:
[0028] When a fatal fault occurs in the processor, the fatal fault enable pin connected to the FPGA is pulled low; after the FPGA recognizes that the fatal fault enable pin has been pulled low, it modifies the information in the interrupt register inside the FPGA.
[0029] When a fatal fault occurs in the processor, the fatal fault enable pin is pulled low. This pin is connected to the FPGA. After the FPGA recognizes that the fatal fault enable pin is pulled low, it will modify the information in the interrupt register inside the FPGA.
[0030] As a further limitation of the technical solution of the present invention, the method also includes:
[0031] The BIOS sends a system control interrupt to the operating system and simultaneously sends the system control interrupt to the baseboard management controller.
[0032] After receiving a system control interrupt, the baseboard management controller sends an instruction to the FPGA to characterize the interrupt operation that completes the processor hardware link.
[0033] The FPGA illuminates the offline status indicator of the server node board. When the offline status indicator is lit, it means that the processor on the node can be removed or replaced while powered on.
[0034] The maintenance personnel successfully removed the faulty processor from the powered node. At this point, other nodes were functioning normally, and the critical services running on the faulty processor were offloaded to the DPU, ensuring no service interruption due to the processor failure. Simultaneously, the operator replaced the processor while the node was powered on. As a further limitation of the technical solution of this invention, the method also includes:
[0035] After the processor that was removed or replaced while powered on is replaced and installed, the BIOS code is executed to boot the processor into the operating system, causing the processor to generate a system management interrupt.
[0036] After receiving a system management interrupt, the BIOS determines and identifies the processor that needs to be brought online and informs the board management controller of the identification result through IPMI command;
[0037] The baseboard management controller triggers the FPGA to send an online request to the replaced processor;
[0038] After receiving the processor's response based on the online request, the FPGA enables the power supply signal of the replaced processor to take effect and ensures that the power supply of all modules of the replaced processor is in the power supply complete state.
[0039] After the FPGA reads the information that the processor power supply is complete, it sends a reset command to the replaced processor to perform a hardware-level processor soft reset, enabling the processor to reset.
[0040] After the processor completes a soft reset, it re-detects and identifies the memory and initiates communication with the memory.
[0041] Once boot is complete, the BIOS sends a system control interrupt to the operating system to complete the processor's online process.
[0042] As a further limitation of the technical solution of the present invention, after the BIOS sends a system control interrupt to the operating system to complete the processor online action, the method further includes:
[0043] After receiving a system control interrupt, the business layer software on the operating system will reload the computing tasks that were offloaded to the DPU onto the newly replaced processor.
[0044] Secondly, the technical solution of the present invention also provides a server system for realizing live operation and maintenance of processors, including an expansion board, a management board and multiple nodes;
[0045] The expansion board is equipped with a DPU, and each node has a processor and an operating system installed; each processor is connected to a power supply.
[0046] Each node's processor is connected to the DPU, which is used by the operating system's business layer to offload computing tasks from the faulty processor to the DPU when an unrecoverable error occurs in the processor.
[0047] The management board is equipped with a baseboard management controller; each processor has an internal register for storing processor fault information; the baseboard management controller on the management board is connected to the internal registers of the processor;
[0048] The FPGA has an internal interrupt register. The baseboard management controller on the management board is connected to the interrupt register inside the FPGA. Each node's processor has a fatal fault enable pin connected to the FPGA for determining fatal faults in the processor. When a fatal fault occurs in the processor, the fatal fault enable pin is pulled low and connected to the FPGA. After the FPGA recognizes the fatal fault and the enable pin is pulled low, it modifies the information in the interrupt register.
[0049] Based on the server system, perform the method described in the first aspect to achieve live processor maintenance.
[0050] As a further limitation of the technical solution of the present invention, the management board is also provided with an IO port expansion selector, which is connected to the baseboard management controller and the FPGA on the management board respectively;
[0051] The baseboard management controller and FPGA are connected to the processor via I / O port expansion selectors.
[0052] Because of the limitation on the number of external interfaces of the FPGA, and the fact that the entire system can only operate on one processor at a time when performing processor online and offline operations, if another processor also fails and needs to be replaced while one processor is being replaced, the replacement of the first faulty processor (i.e., the processor is offline and then online) can only be completed before the other processor is replaced. Therefore, the processor signals will first be connected to the IO port expansion selector, and then connected to the FPGA or BMC.
[0053] As a further limitation of the technical solution of the present invention, the processor is provided with PECI bus pin, processor in-place pin, processor power supply completion pin, processor reset pin, and processor online / offline requirement pin;
[0054] The processor's PECI bus pin is connected to the baseboard management controller via an I / O port expansion selector. When the FPGA detects a fatal fault in the processor, the baseboard management controller obtains the fatal fault information from the FPGA's interrupt register and then retrieves the fault information from the processor's internal register via the PECI bus pin.
[0055] The processor in-place pin is connected to the FPGA via an I / O port expansion selector for identifying and determining the processor's presence.
[0056] The processor power supply completion pin is connected to the FPGA through the I / O port expansion selector. The FPGA enables the power supply of different modules of the processor. When the FPGA recognizes that all power supplies are effective and stable, it informs the processor that the power supply is complete through the processor power supply completion pin.
[0057] The processor reset pin is connected to the FPGA via an I / O port expansion selector for FPGA processor reset.
[0058] The processor online / offline request pin is used to send online and offline requests to the IO port expansion selector when the processor encounters a fault and needs to be offline, or when the processor is replaced and needs to be online. It is then connected to the FPGA.
[0059] When the processor experiences a fatal failure, the power pin is connected to the FPGA via an isolation control circuit and an I / O port expansion selector in sequence.
[0060] The isolation control circuit includes a MOSFET, the drain of which is connected to the processor's critical fault enable pin, which is also connected to the power supply via a pull-up resistor; the source of the MOSFET is connected to the I / O port expansion selector, and the gate of the MOSFET is connected to the FPGA.
[0061] The isolation logic design for external fatal faults primarily involves pulling up the processor's fatal fault enable pin to a 1.0V supply voltage via a resistor. Since this fatal fault enable pin is active low, it is connected to an I / O port expansion selector via a selector switch, which in turn connects to the FPGA. The drain of the selector switch is connected to the fatal fault enable pin, and the source of the selector switch is connected to the FPGA via the I / O port expansion selector. The control terminal (gate) of the selector switch is also connected to the FPGA. Depending on whether the processor is replaced, the FPGA controls this pin and the external isolation circuitry to switch it on and off. The FPGA's operating logic is as follows:
[0062] When the processor is working normally and there is no fatal fault, the pin states during a fatal fault are as follows:
[0063] When a fatal fault occurs, the critical fault enable pin is at a high level, and the isolation device (in this case, a selector switch) is closed. When a fatal fault occurs, the critical fault enable pin is at a low level, which means that the processor needs to perform an offline operation. This requires disconnecting the external isolation device of the critical fault enable pin.
[0064] When a processor encounters an unrecoverable error or a fatal failure, the processor needs to be replaced. The pin states during a fatal failure are as follows:
[0065] Before replacing the processor, if a fatal fault occurs that necessitates processor replacement, the fatal fault enable pin will be at a low level. In this case, it is necessary to perform a processor offline operation and disconnect the isolation device.
[0066] If the processor needs to be replaced due to other unrecoverable errors, the fatal fault enable pin will be at a high level. In this case, the processor needs to be taken offline and the isolator will be disconnected.
[0067] After the processor is replaced, the processor replacement is complete, the isolation device is closed, the processor is online, and the fatal fault enable pin is at a high level. Before the processor completes the online process, the system will not pay attention to the status of the fatal fault enable pin of the processor. It should be noted that before replacing the faulty processor, the software application layer business needs to be migrated first, and then the hardware level offline operation needs to be performed.
[0068] Before the processor is fully online, the entire system will not shut down due to any abnormal errors or faults triggered by that processor. Similarly, fatal fault signals and their corresponding logic circuits need to be isolated. Fatal faults caused by the replacement processor will not cause a system shutdown until the replacement processor is fully online. After the replacement processor completes its reset, the isolated circuit corresponding to the fatal fault enable pin needs to be enabled. Specifically, the FPGA sends a signal to the control terminal (the gate of the selector switch) of the fatal fault isolation circuit to close the isolation circuit. The fatal fault enable pin of the replaced processor is then connected to the isolation circuit. After this process is completed, the firmware sends a system control interrupt to the operating system, thus completing the processor online process.
[0069] As can be seen from the above technical solutions, the present invention has the following advantages: a multi-channel server or multi-node server with shared DPU, combined with processor, BIOS and operating system, and FPGA control, can realize live maintenance of a single node, especially seamless replacement of the processor, ensuring the continuous operation of services.
[0070] Furthermore, the design principle of this invention is reliable, the structure is simple, and it has a very wide range of application prospects.
[0071] Therefore, it is evident that the present invention has outstanding substantive features and significant progress compared with the prior art, and the beneficial effects of its implementation are also obvious. Attached Figure Description
[0072] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0073] Figure 1 This is a schematic flowchart of a method according to an embodiment of the present invention.
[0074] Figure 2 This is a schematic flowchart illustrating the fault diagnosis method of one embodiment of the present invention.
[0075] Figure 3 This is a schematic flowchart illustrating the fatal fault determination method of one embodiment of the present invention.
[0076] Figure 4 This is a schematic flowchart illustrating the powered installation of a processor in a method according to an embodiment of the present invention.
[0077] Figure 5 This is a control block diagram of the isolation line for fatal faults in an embodiment of the present invention.
[0078] Figure 6 This is a rear view diagram of the server system provided by the present invention.
[0079] Figure 7 This is a front view schematic diagram of the server system provided by the present invention.
[0080] Figure 8 This is a top view schematic diagram of the server system provided by the present invention.
[0081] Figure 9 This is a schematic diagram of the system connection for heat dissipation design in an embodiment of the present invention.
[0082] Figure 10 This is a schematic diagram of the specific signals connecting the nodes and the management board in the system. Detailed Implementation
[0083] To achieve a multi-node shared DPU design, the current DPU's uplink X16 PCIe lanes come from four processors on four nodes, with each processor providing one X4 PCIe lane. Combined with the FPGA, firmware, and operating system on the management board, this allows multiple nodes to share a single DPU. Furthermore, it enables timely migration of the processor's computing power to the DPU in case of a processor failure or anomaly on a node, allowing for rapid processor replacement without powering down or interrupting services. To facilitate a better understanding of the technical solutions in this invention, the following description, in conjunction with the accompanying drawings of the embodiments, provides a clear and complete overview of the technical solutions. Obviously, the described embodiments are merely some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort should fall within the scope of protection of this invention.
[0084] like Figure 1 As shown, this embodiment of the invention provides a method for implementing live processor maintenance, comprising the following steps:
[0085] Step 1: After the processor experiences an unrecoverable error and offloads the computing task from the processor to the DPU, the processor generates a system management interrupt;
[0086] Step 2: After the baseboard management controller identifies the processor's fault information and obtains the system management interrupt, it triggers the FPGA to send a shutdown request to the processor that generated the fault information.
[0087] Step 3: After receiving the processor's response based on the offline request, the FPGA triggers the processor to reset and simultaneously pulls the processor power supply signal low;
[0088] Step 4: After all the processor power supply signals are pulled low, the BIOS sends a system control interrupt to the operating system and stops the processor from operating its external I / O ports;
[0089] Step 5: After the operating system recognizes the system control interrupt sent by the BIOS through its own driver, it stops any information interaction and business communication between the operating system and the processor, and completes the interrupt operation of the processor hardware link.
[0090] When an unrecoverable error occurs in the processor, the BIOS collects information from the processor's internal registers. The processor's internal registers have different record formats for different faults. After the BIOS collects this fault information and reads the system management interrupt issued by the processor, it sends it to the baseboard management controller via IPMI command. The baseboard management controller stores the fault in the log and at the same time triggers the FPGA to enable the processor to go offline.
[0091] It should be noted that after the processor encounters an unrecoverable error and offloads computing tasks from the processor to the DPU, the business layer software notifies the processor of the computing power migration completion through the operating system, and the processor generates a system management interrupt. When the server is running business operations and the processor has an unrecoverable error requiring replacement, to avoid business interruption, the network computing power, I / O computing power, and storage computing power tasks running on the processor are released to the applications on the DPU. Combined with the applications, the DPU can also efficiently complete these tasks. That is, the work that originally needed to be done on the server host with the processor, involving the establishment of various virtual machines and software for data transmission, analysis, and processing, is transferred to the DPU. The network and storage computing power consumed on the server host and running on the processor is taken over by the DPU. In other words, when the processor encounters an unrecoverable or fatal error and needs to be taken offline, it first works with the business layer software running on the server operating system to handle the situation, offloading the running business operations, especially network and storage computing power, to the DPU. The specific process of unloading computing tasks is not an innovation of this application; existing computing power unloading steps can be used, and will not be elaborated here.
[0092] In some embodiments, such as Figure 2 As shown, after the baseboard management controller identifies the processor's fault information and obtains a system management interrupt, the steps that trigger the FPGA to send a shutdown request to the processor that generated the fault information include:
[0093] Step 21: The BIOS collects information from the processor's internal registers;
[0094] Step 22: After the BIOS collects the processor's fault information and reads the system management interrupt issued by the processor, it sends it to the baseboard management controller via IPMI command;
[0095] Step 23: After the baseboard management controller obtains the processor's fault information and the system management interrupt from the BIOS, it records the obtained fault information to the log and triggers the FPGA to send a shutdown request to the processor that generated the fault information.
[0096] It should be further explained that when a fatal fault occurs in the processor, the fatal fault enable pin connected to the FPGA is pulled low; after the FPGA detects the fatal fault and the enable pin is pulled low, it modifies the information in the interrupt register internally. For example... Figure 3 As shown, the steps of the baseboard management controller obtaining processor fault information and receiving system management interrupt from the BIOS, recording the obtained fault information in the log, and simultaneously triggering the FPGA to send a shutdown request to the processor that generated the fault information include:
[0097] Step 231: The baseboard management controller actively and periodically polls the interrupt register information inside the FPGA;
[0098] Step 232: The baseboard management controller determines whether a fatal fault has occurred in the processor based on the acquired interrupt register information;
[0099] If yes, proceed to step 233; if no, proceed to step 235.
[0100] Step 233: After the baseboard management controller receives the system management interrupt issued by the processor from the BIOS, it obtains the information of the processor's internal registers through PECI; among them, the processor's internal registers have different recording formats for different faults;
[0101] Step 234: The baseboard management controller records the acquired fault information to the log and simultaneously triggers the FPGA to send a shutdown request to the processor that generated the fault information;
[0102] Step 235: After the baseboard management controller obtains the system management interrupt from the BIOS, it records the fault information obtained from the BIOS into the log and triggers the FPGA to send a shutdown request to the processor that generated the fault information.
[0103] The baseboard management controller actively and periodically polls the interrupt register information inside the FPGA. When the baseboard management controller reads that the processor has a fatal fault and obtains the system management interrupt issued by the processor from the BIOS, it will obtain the information of the processor's internal registers through the PECI bus and record the fault information in the log. At the same time, it will trigger the FPGA to enable the processor to go offline.
[0104] In some embodiments, the method further includes:
[0105] Step 6: The BIOS sends a system control interrupt to the operating system and simultaneously sends the system control interrupt to the baseboard management controller; after receiving the system control interrupt, the baseboard management controller sends an interrupt operation instruction to the FPGA to indicate the completion of the processor hardware link; the FPGA performs offline status indicator lighting on the server node board. When the offline status indicator lights up, it means that the processor on the node can be removed or replaced while powered on.
[0106] The maintenance personnel successfully removed the faulty processor from the powered node. At this point, other nodes were functioning normally, and the critical services running on the faulty processor were offloaded to the DPU, ensuring no service interruption due to the processor failure. Simultaneously, the operator replaced the processor while the node was powered on. Figure 4 As shown, in some embodiments, the method further includes:
[0107] Step 7: After the processor that was removed or replaced while powered on is replaced and installed, the BIOS code is executed to boot the processor into the operating system, causing the processor to generate a system management interrupt;
[0108] Step 8: After receiving the system management interrupt, the BIOS determines and identifies the processor that needs to be brought online and informs the board management controller of the identification result through the IPMI command;
[0109] In this step, after receiving a system management interrupt, the BIOS processes and analyzes it to determine which processor needs to be brought online. The BIOS performs some checks to identify which processor is not online. Then, the BIOS notifies the baseboard management controller via IPMI commands. The baseboard management controller triggers the FPGA to send a online request to the replaced processor. It should be noted that the replaced processor has just been installed in the processor slot. The FPGA identifies the processor's presence through the processor's "in-place" pin. The BIOS can identify which processors in the current system are online and which are offline. The newly installed processor is still offline because it has not completed a series of online action instructions.
[0110] Step 9: The baseboard management controller triggers the FPGA to send an online request to the replaced processor;
[0111] Step 10: After receiving the processor's response based on the online request, the FPGA enables the power supply signal of the replaced processor to take effect, and ensures that the power supply of all modules of the replaced processor is in the power supply complete state.
[0112] The processor power supply signals described here refer to all the power supply signals of the processor. Different modules of the processor require different voltages, which are all output by externally designed voltage regulators (VR). Before the processor is taken offline, the power supply input to the processor is pulled low. This can be done by pulling the enable pin of the processor power supply low through the FPGA. The corresponding power supply voltage regulator (VR) is still working. After the processor is replaced, the FPGA will pull the enable pin of the processor power supply high to complete the processor power supply, and then cooperate with the subsequent online actions to complete the processor online.
[0113] Step 11: After the FPGA reads the information that the processor power supply is complete, it sends a reset command to the replaced processor to perform a hardware-level processor soft reset and enable processor reset.
[0114] It should be noted that, due to the limited number of FPGA pins, the multiple processors in this application are actually connected to the FPGA via an I / O port expansion selector. When the power supply to each module of the processor is ready, the FPGA reads the information that the processor power supply is complete, sends a reset command to the I / O port expansion selector, and then sends it to the replaced processor to perform a hardware-level processor soft reset, enabling the processor reset. After a 30ms delay, the reset operation is completed.
[0115] Step 12: After the processor completes a soft reset, it re-detects and identifies the memory and initiates communication with the memory;
[0116] The processor's internal registers only become effective after the processor has completed a soft reset.
[0117] Step 13: Booting complete. The BIOS sends a system control interrupt to the operating system to complete the processor online process.
[0118] Step 14: After receiving a system control interrupt, the business layer software on the operating system will reload the computing tasks that were unloaded into the DPU onto the newly replaced processor.
[0119] This invention also provides a server system for implementing live processor maintenance, including an expansion board, a management board, and multiple nodes;
[0120] The expansion board is equipped with a DPU, and each node has a processor and an operating system installed; each processor is connected to a power supply.
[0121] Each node's processor is connected to the DPU, which is used by the operating system's business layer to offload computing tasks from the faulty processor to the DPU when an unrecoverable error occurs in the processor.
[0122] The management board is equipped with a baseboard management controller; each processor has an internal register for storing processor fault information; the baseboard management controller on the management board is connected to the internal registers of the processor;
[0123] The FPGA has an internal interrupt register. The baseboard management controller on the management board is connected to the interrupt register inside the FPGA. Each node's processor has a fatal fault enable pin connected to the FPGA for determining fatal faults in the processor. When a fatal fault occurs in the processor, the fatal fault enable pin is pulled low and connected to the FPGA. After the FPGA recognizes the fatal fault and the enable pin is pulled low, it modifies the information in the interrupt register.
[0124] The server system performs the method described in the above embodiments to achieve live processor maintenance.
[0125] In some embodiments, the management board is also provided with an IO port expansion selector, which is connected to the baseboard management controller and the FPGA on the management board respectively;
[0126] The baseboard management controller and FPGA are connected to the processor via I / O port expansion selectors.
[0127] The server system provided in this embodiment has a total of 4 nodes, each with 1 processor, for a total of 4 processors. The signals from the processors are shown below, and each interconnect signal has 4 sets. Due to the limitation on the number of external interfaces of the FPGA, and the fact that the entire system can only operate on one processor at a time when performing processor online and offline operations, if another processor also fails and needs to be replaced while one processor is malfunctioning, the replacement of the first malfunctioning processor (i.e., the processor is offline and then online) can only be completed before the other processor can be replaced. Therefore, the signals of the 4 processors will first be connected to the IO port expansion selector, and then connected to the FPGA or BMC.
[0128] It should be noted that the processor is equipped with PECI bus pins, processor in-place pins, processor power-on completion pins, processor reset pins, and processor online / offline requirement pins.
[0129] The processor's PECI bus pin is connected to the baseboard management controller via an I / O port expansion selector. When the FPGA detects a fatal fault in the processor, the baseboard management controller obtains the fatal fault information from the FPGA's interrupt register and then retrieves the fault information from the processor's internal register via the PECI bus pin.
[0130] The processor in-place pin is connected to the FPGA via an I / O port expansion selector for identifying and determining the processor's presence.
[0131] The processor power supply completion pin is connected to the FPGA through the I / O port expansion selector. The FPGA enables the power supply of different modules of the processor. When the FPGA recognizes that all power supplies are effective and stable, it informs the processor that the power supply is complete through the processor power supply completion pin.
[0132] The processor reset pin is connected to the FPGA via an I / O port expansion selector for FPGA processor reset.
[0133] The processor online / offline request pin is used to send online and offline requests to the IO port expansion selector when the processor encounters a fault and needs to be offline, or when the processor is replaced and needs to be online. It is then connected to the FPGA.
[0134] The isolation circuit for the fatal fault enable pin is as follows: Figure 5 As shown, the fatal fault logic design mainly involves pulling the processor's fatal fault enable pin up to the 1.0V supply voltage via a resistor. Since this fatal fault enable pin is active low, it is connected to an I / O port expansion selector (the selector is also an isolation device, here a MOSFET) via a selector switch, and then connected to the FPGA. The drain of the MOSFET is connected to the fatal fault enable pin, and the source of the MOSFET is connected to the FPGA via the I / O port expansion selector. The gate of the MOSFET is connected to the FPGA. Depending on whether the processor is replaced, the FPGA controls this pin and the external isolation circuitry to switch it on and off. The FPGA's operating logic is as follows:
[0135] When the processor is working normally and there is no fatal fault, the pin states during a fatal fault are as follows:
[0136] When a fatal fault occurs, the critical fault enable pin is at a high level, and the isolation device (in this case, a selector switch) is closed. When a fatal fault occurs, the critical fault enable pin is at a low level, which means that the processor needs to perform an offline operation. This requires disconnecting the external isolation device of the critical fault enable pin.
[0137] When a processor encounters an unrecoverable error or a fatal failure, the processor needs to be replaced. The pin states during a fatal failure are as follows:
[0138] Before replacing the processor, if a fatal fault occurs that necessitates processor replacement, the fatal fault enable pin will be at a low level. In this case, it is necessary to perform a processor offline operation and disconnect the isolation device.
[0139] If the processor needs to be replaced due to other unrecoverable errors, the fatal fault enable pin will be at a high level. In this case, the processor needs to be taken offline and the isolator will be disconnected.
[0140] After the processor is replaced, the processor replacement is complete, the isolation device is closed, the processor is online, and the fatal fault enable pin is at a high level. Before the processor completes the online process, the system will not pay attention to the status of the fatal fault enable pin of the processor. It should be noted that before replacing the faulty processor, the software application layer business needs to be migrated first, and then the hardware level offline operation needs to be performed.
[0141] Before the processor is fully online, the entire system will not shut down due to any abnormal errors or faults triggered by that processor. Similarly, fatal fault signals and their corresponding logic circuits need to be isolated. Fatal faults caused by the replacement processor will not cause a system shutdown until the replacement processor is fully online. After the replacement processor completes its reset, the isolation circuit corresponding to the fatal fault enable pin needs to be enabled. Specifically, the FPGA sends a signal to the control terminal (gate) of the MOSFET to close the isolation circuit. The fatal fault enable pin of the replaced processor is then connected to the isolation circuit. After this process is completed, the firmware sends a system control interrupt to the operating system, thus completing the processor online process.
[0142] The following is a design for the management, interconnection, and upgrade of different compute nodes in high-end servers and multi-node servers. The following example uses four compute nodes for illustration.
[0143] The following multi-node server design based on domestically produced processors can meet the design requirements of multiple nodes sharing a DPU. A total of four nodes are located in the server's backplane: node 0, node 1, node 2, and node 3. Figure 6 As shown. The server front panel is used for expanding storage and PCIe devices, as well as DPU expansion, such as... Figure 7 As shown.
[0144] The server chassis contains a hard drive backplane for expanding hard drives, a PCIe expansion board for expanding PCIe devices, and a PCIe expansion board for expanding the DPU. In this embodiment of the invention, only one DPU, one fan wall, one management board, and four nodes are provided. The fan wall is used for cooling the entire system. The management board is used for monitoring and managing each node within the system, controlling fan speed, node power, and live maintenance control of the nodes. Each node has a single processor, a baseboard controller, and a power supply. The baseboard controller on each node manages the key components on that node and collects sensor information, etc. Figure 8 As shown.
[0145] The power management section primarily manages the baseboard control card on the management board, which connects to the power supply of each node via PMBUS. It acquires information such as temperature, input / output voltage, input / output current, and input / output power consumption, storing the information from the four PSUs. These values are then used by the baseboard controllers of the four nodes for reading via PMBUS, enabling effective and backup power control. Figure 9 As shown, the fan management in the thermal design involves the baseboard controller on each node acquiring sensor information for heat dissipation from its respective node. The baseboard controller on the management board connects to the baseboard controller on each node via I2C, collecting the BMC status of each node to perform thermal design. After acquiring sensor information, it integrates and evaluates the data and performs linear intelligent zone control. The baseboard controller on the management board automatically controls the fans via PWM according to the fan control strategy. When the baseboard controller on the management board experiences a wiring malfunction or hangs, the FPGA takes over the fan speed control. However, at this time, it will be controlled according to a fixed speed. To ensure normal system operation, the speed is generally set relatively high, such as full speed or designed at 85% speed. The process of removing the processor with a charge and installing the processor with a charge is as described in the above embodiment.
[0146] like Figure 10 This is a schematic diagram of the specific signals connecting the nodes and the management board in the system. 1. PECI_0~3: Connected to the BMC through the IO port selector. When the FPGA detects a fatal fault in the processor, the BMC can obtain the fatal fault information from the FPGA's interrupt register, and then the BMC obtains the fault information from the processor's internal register through PECI;
[0147] 2. Processor in place_0~3: Connected to the FPGA via an I / O port selector, used for identifying and determining the presence of the processor;
[0148] 3. Processor power supply complete_0~3: After connecting to the FPGA through the IO port selector, the FPGA enables the power supply of different modules of the processor to be effective. When the FPGA recognizes that all power supplies are effective and stable, it will notify the processor that the power supply is complete through this pin.
[0149] 4. Processor Reset_0~3: Connected to the FPGA via the IO port selector, used to reset the processor in the FPGA.
[0150] 5. Processor online / offline requirements_0~3: When a processor encounters a fault and needs to be taken offline, or when a processor needs to be replaced and taken online, the processor will send online and offline requirements to the IO port selector, which will then connect to the FPGA;
[0151] 6. Fatal Fault Enable_0~3: When a fatal fault occurs in the processor, this signal will be pulled low. This pin is connected to the I / O port selector, and then to the FPGA, for the determination of fatal faults in the processor. In addition to the processor encountering general unrecoverable errors, there is also a fatal fault, which is included in the category of unrecoverable errors, but it has a separate pin. With the help of external logic circuit design, this pin is a general input / output I / O pin, which can be enabled externally. Then the processor will trigger the corresponding check and protection strategy.
[0152] In addition, high-end multi-socket servers, such as 4-way, 8-way, 16-way, and 32-way servers, and multi-node servers, such as 2-node, 4-node, and 8-node servers, combined with the above design, can seamlessly replace critical components, such as processors and PCIe devices, while the system is powered on, ensuring the continuous operation of business and improving the customer's user experience.
[0153] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the invention should also be covered within the protection scope of the invention. Therefore, the protection scope of the invention should be determined by the scope of the claims.
Claims
1. A method for implementing live maintenance of a processor, the method comprising: Includes the following steps: After the processor experiences an unrecoverable error and offloads the computing task from the processor to the DPU, the processor generates a system management interrupt; when the processor experiences a fatal failure, the fatal failure enable pin connected to the FPGA is pulled low; after the FPGA recognizes that the fatal failure enable pin has been pulled low, it modifies the information in the interrupt register inside the FPGA. After the baseboard management controller identifies the processor's fault information and obtains the system management interrupt, it triggers the FPGA to send a shutdown request to the processor that generated the fault information. Specifically, this includes: the BIOS collecting information from the processor's internal registers; after the BIOS collects the processor's fault information and reads the system management interrupt issued by the processor, it sends it to the baseboard management controller via an IPMI command; after the baseboard management controller obtains the processor's fault information and obtains the system management interrupt from the BIOS, it records the obtained fault information in the log and simultaneously triggers the FPGA to send a shutdown request to the processor that generated the fault information. After receiving the processor's response based on the offline request, the FPGA triggers the processor to reset and simultaneously pulls the processor power supply signal low. After the processor power supply signals are all pulled low, the BIOS sends a system control interrupt to the operating system and stops the processor from operating its external I / O ports; After the operating system recognizes the system control interrupt sent by the BIOS through its own driver, it stops any information interaction and business communication between the operating system and the processor, and completes the interrupt operation of the processor hardware link.
2. The method for implementing live processor maintenance according to claim 1, characterized in that, After the baseboard management controller obtains the processor's fault information and receives the system management interrupt from the BIOS, it records the obtained fault information to the log and triggers the FPGA to send a shutdown request to the processor that generated the fault information. The steps include: The baseboard management controller actively and periodically polls the interrupt register information inside the FPGA; The baseboard management controller determines whether a fatal fault has occurred in the processor based on the acquired interrupt register information; If so, after the baseboard management controller receives the system management interrupt issued by the processor from the BIOS, it obtains information from the processor's internal registers through PECI; among them, the processor's internal registers have different recording formats for different faults; The baseboard management controller records the acquired fault information to the log and simultaneously triggers the FPGA to send a shutdown request to the processor that generated the fault information. If not, after the baseboard management controller receives a system management interrupt from the BIOS, it will record the fault information obtained from the BIOS into the log and simultaneously trigger the FPGA to send a shutdown request to the processor that generated the fault information.
3. The method for implementing live-line maintenance of a processor according to claim 1, characterized in that, The method also includes: The BIOS sends a system control interrupt to the operating system and simultaneously sends the system control interrupt to the baseboard management controller. After receiving a system control interrupt, the baseboard management controller sends an instruction to the FPGA to characterize the interrupt operation that completes the processor hardware link. The FPGA illuminates the offline status indicator of the server node board. When the offline status indicator is lit, it means that the processor on the node can be removed or replaced while powered on.
4. The method for implementing live-line maintenance of a processor according to claim 3, characterized in that, The method also includes: After the processor that was removed or replaced while powered on is replaced and installed, the BIOS code is executed to boot the processor into the operating system, causing the processor to generate a system management interrupt. After receiving a system management interrupt, the BIOS determines and identifies the processor that needs to be brought online and informs the board management controller of the identification result through IPMI command; The baseboard management controller triggers the FPGA to send an online request to the replaced processor; After receiving the processor's response based on the online request, the FPGA enables the power supply signal of the replaced processor to take effect and ensures that the power supply of all modules of the replaced processor is in the power supply complete state. After the FPGA reads the information that the processor power supply is complete, it sends a reset command to the replaced processor to perform a hardware-level processor soft reset, enabling the processor to reset. After the processor completes a soft reset, it re-detects and identifies the memory and initiates communication with the memory. Once boot is complete, the BIOS sends a system control interrupt to the operating system to complete the processor's online process.
5. The method for implementing live processor maintenance according to claim 4, characterized in that, After the BIOS sends a system control interrupt to the operating system to complete the processor online process, the following steps are also included: After receiving a system control interrupt, the business layer software on the operating system will reload the computing tasks that were offloaded to the DPU onto the newly replaced processor.
6. A server system for implementing live processor maintenance, characterized in that, Includes expansion board, management board and multiple nodes; The expansion board is equipped with a DPU, and each node has a processor and an operating system installed; each processor is connected to a power supply. Each node's processor is connected to the DPU, which is used by the operating system's business layer to offload computing tasks from the faulty processor to the DPU when an unrecoverable error occurs in the processor. The management board is equipped with a baseboard management controller; each processor has an internal register for storing processor fault information; the baseboard management controller on the management board is connected to the internal registers of the processor; The FPGA has an internal interrupt register. The baseboard management controller on the management board is connected to the interrupt register inside the FPGA. Each node's processor has a fatal fault enable pin connected to the FPGA for determining fatal processor faults. When a fatal fault occurs in the processor, the fatal fault enable pin is pulled low and connected to the FPGA. After the FPGA recognizes that the fatal fault enable pin has been pulled low, it modifies the information in the interrupt register. The server system performs the method described in any one of claims 1-5 to achieve live processor maintenance.
7. The server system for implementing live processor maintenance according to claim 6, characterized in that, The management board is also equipped with an IO port expansion selector, which is connected to the baseboard management controller and the FPGA on the management board respectively; The baseboard management controller and FPGA are connected to the processor via I / O port expansion selectors.
8. The server system for implementing live processor maintenance according to claim 7, characterized in that, The processor's fatal fault enable pin is connected to the FPGA via an isolation control circuit and an I / O port expansion selector in sequence. The isolation control circuit includes a MOSFET, the drain of which is connected to the processor's fatal fault enable pin, which is also connected to the power supply via a pull-up resistor; the source of the MOSFET is connected to the I / O port expansion selector, and the gate of the MOSFET is connected to the FPGA.