Memory fault processing method and system

Abstract

The invention discloses a memory fault processing method and system, and relates to the technical field of servers, and the method comprises the steps: obtaining first serial bus communication information between a processor and a plurality of memory banks; determining a memory overheating fault type according to first serial bus communication information between the processor and the plurality of memory banks, and generating a log file of the controller according to memory overheating fault type data; obtaining error data of the first serial bus communication information in response to the situation that the log file of the controller does not record data of a memory bank heat dissipation problem or a processor fault problem; and determining the fault type of the error of the first serial bus communication information according to the error data of the first serial bus communication information. According to the method and the device, memory faults caused by over-temperature of the memory and the processor can be checked without starting the box for reproduction, so that the detection time cost and the research and development manpower are reduced.

Description

Technical Field

[0001] This application relates to the field of server technology, and in particular to a memory fault handling method and system. Background Technology

[0002] In the relevant solution design, after the memory overheating problem occurs, apart from memory temperature being too high and can be read by command, memory overheating caused by the processor itself and memory overheating caused by I3C link failure requires opening the case for measurement. Not all measurement points are on the front, and it is even necessary to disconnect the power and resolder the wires for reproduction and verification, which increases the difficulty of detection. In some cases, it may take several months to reproduce and analyze the situation.

[0003] Related design schemes such as Figure 1 As shown, after a memory overheating event occurs, the controller (BMC, Baseboard Management Controller) can record the event. However, further verification and analysis are needed to determine the cause of the overheating. Typically, the temperature of all memory modules is first checked to see if overheating is present. If overheating is indeed present, the cooling strategy is analyzed. If no overheating is found, it's necessary to determine whether the overheating is caused by the processor itself or by an unresponsive I3C link. This requires opening the chassis for measurement. The first step is to measure the connection from the CPU to the FPGA (Field-Programmable Gate Array). If the memory overheating signal of the array is pulled low, further measurement is needed to determine how many I3C links are abnormal. Here, two difficulties arise: First, if there are many cables and structural heat dissipation layouts inside the chassis, it may be difficult to locate the measurement point; second, even if the measurement point can be located, it is possible to measure which I3C link is abnormal, but because the level is low after an I3C link is abnormal, and because three memory modules are connected on one I3C bus, it is usually necessary to solder wires to determine which one is faulty. This method increases the time cost and R&D manpower cost for production line shipment. Summary of the Invention

[0004] This application provides a memory fault handling method and system. The method includes: acquiring first serial bus communication information between a processor and several memory modules; determining the memory overheating fault type based on the first serial bus communication information; generating a controller log file based on the memory overheating fault type data; and storing error data of the first serial bus communication information, wherein the memory overheating fault type includes memory module overheating problems and processor failure problems; in response to the controller log file not recording data on memory module overheating problems or processor failure problems, acquiring error data of the first serial bus communication information; and determining the fault type of the first serial bus communication information error based on the error data of the first serial bus communication information. This application can troubleshoot memory faults caused by memory overheating and processor issues without opening the chassis for reproduction, reducing the time cost and R&D manpower required for detection.

[0005] This application provides a memory fault handling system, which includes a controller, a processor, a switching integrated circuit, a logic unit, a plurality of switches, and a plurality of memory modules. The first output terminal of the controller is connected to the switching integrated circuit, the switching integrated circuit is connected to a plurality of switches respectively, each switch is connected to a plurality of memory modules respectively, the second output terminal of the controller is connected to the logic unit, the logic unit is connected to a plurality of switches respectively, the logic unit is connected to the processor, and the processor is connected to a plurality of switches respectively. The system also includes a memory card, which is connected to the logic unit. The processor is connected to several switches via a first serial bus link; Each switch is connected to several memory modules via a first serial bus link; The controller and logic unit are connected via a second serial bus link.

[0006] This application also provides a memory fault handling method, which is applied to the controller of a memory fault handling system, and the method includes: Obtain the first serial bus communication information between the processor and several memory modules; The memory overheating fault type is determined based on the first serial bus communication information between the processor and several memory modules. A log file of the controller is generated based on the memory overheating fault type data, and the error data of the first serial bus communication information is stored. The memory overheating fault types include memory module heat dissipation problems and processor failure problems. If the controller's log file does not record data related to memory module overheating or processor malfunction, then error data is obtained from the first serial bus communication information. The fault type of the first serial bus communication information error is determined based on the error data of the first serial bus communication information.

[0007] The application further provides a memory fault processing system, which comprises a controller, a processor, a switching integrated circuit, a logic device, a plurality of switches and a plurality of memory banks, the first output end of the controller is connected with the switching integrated circuit, the switching integrated circuit is connected with the plurality of switches correspondingly, each switch is connected with the plurality of memory banks correspondingly, the second output end of the controller is connected with the logic device, the logic device is connected with the plurality of switches correspondingly, the logic device is connected with the processor, the processor is connected with the plurality of switches correspondingly, and the controller is used for: acquiring first serial bus communication information between the processor and the plurality of memory banks; determining a memory overheating fault type according to the first serial bus communication information between the processor and the plurality of memory banks, generating a log file of the controller according to memory overheating fault type data, and storing error data of the first serial bus communication information, wherein the memory overheating fault type comprises a memory bank heat dissipation problem and a processor fault problem; in response to the fact that the memory bank heat dissipation problem or the processor fault problem data is not recorded in the log file of the controller, error data of the first serial bus communication information is acquired; the fault type of the error of the first serial bus communication information is determined according to the error data of the first serial bus communication information.

[0008] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps of the memory fault processing method, and the method comprises: acquiring first serial bus communication information between the processor and the plurality of memory banks; determining a memory overheating fault type according to the first serial bus communication information between the processor and the plurality of memory banks, generating a log file of the controller according to memory overheating fault type data, and storing error data of the first serial bus communication information, wherein the memory overheating fault type comprises a memory bank heat dissipation problem and a processor fault problem; in response to the fact that the memory bank heat dissipation problem or the processor fault problem data is not recorded in the log file of the controller, error data of the first serial bus communication information is acquired; the fault type of the error of the first serial bus communication information is determined according to the error data of the first serial bus communication information.

[0009] By the present application, since the method comprises acquiring first serial bus communication information between a processor and a plurality of memory banks; determining a memory overheating fault type according to the first serial bus communication information between the processor and the plurality of memory banks, generating a log file of a controller according to the memory overheating fault type data, and storing error data of the first serial bus communication information, wherein the memory overheating fault type comprises a memory bank heat dissipation problem and a processor fault problem; in response to no memory bank heat dissipation problem or processor fault problem data being recorded in the log file of the controller, acquiring error data of the first serial bus communication information; and determining a fault type of the error of the first serial bus communication information according to the error data of the first serial bus communication information. The present application can troubleshoot memory fault problems caused by memory overheating and processor reasons without the need of opening a machine box, thereby reducing the time cost and research and development manpower of detection.

[0010] The technical scheme of the present application can troubleshoot problems caused by memory overheating and processor reasons without the need of opening a machine box. If the problem is caused by the processor, the processor can be directly replaced without the need of opening a machine box for measurement, cross verification and reproduction. If the problem is caused by an I3C link, the problem data can be retained, and which memory channel has the problem can be recorded, and whether the problem is caused by the memory or the system can be roughly analyzed, thereby providing data support for further reproduction and cross verification analysis, and avoiding problems in cross verification of which memory bank has the problem on the same I3C path.

[0011] The technical scheme of the present application can record problem data, and cross verification and reproduction problems in the early stage can be reproduced without the need of a logic analyzer and an oscilloscope, thereby facilitating reproduction operation of production line personnel without the need of too much intervention of research and development personnel. For problems of the processor, production line personnel can determine replacement of the processor by reading the log of the controller. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0013] Figure 1 The structural diagram of the related memory fault processing system provided for the embodiments of the present application is shown in the figure; Figure 2 The structural diagram of the memory fault processing system provided for the embodiments of the present application is shown in the figure; Figure 3 The first flowchart of the memory fault processing method provided for the embodiments of the present application is shown in the figure; Figure 4A second flowchart of the memory fault processing method provided by the embodiment of the present application is shown in FIG. 2. Figure 5 A specific flowchart of the memory fault processing method provided by the embodiment of the present application is shown in FIG. 3. Figure 6 An exemplary system that can be used to implement various embodiments of the present application is provided by the embodiment of the present application. DETAILED DESCRIPTION

[0014] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0015] It should be noted that, in the description of the present application, the terms “comprise”, “contain” or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms “first”, “second” and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0016] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0017] Modern society has entered an era of rapid development of informationization and intelligentization. As the core equipment of informationization system, servers play a crucial role in modern information technology architecture.

[0018] Servers are an indispensable part of modern computing and communication architecture. The efficient operation of servers supports various online services and enterprise operations, enabling information and resources to be quickly and securely shared and processed globally. With the development of technology, the functionality and efficiency of servers are constantly improving to meet the growing demand for data processing.

[0019] In servers, memory is a key component for temporary data storage. Memory plays an important role in communication between CPU and other components, especially when processing large amounts of data or running large programs. The capacity and speed of memory are particularly important. Memory also plays an indispensable role in the operation of servers.

[0020] The memory problem of the server is the most important problem in the server. Once the memory problem occurs, the operation of the whole system will be affected. Each problem of the memory needs attention. In the operation process of the server, the processor CPU alarm memhot problem will be encountered. However, there are many reasons for triggering the memory overheating memhot problem: the first reason is that the processor CPU itself has a problem, which needs to be replaced; the second reason is that the memory temperature is too high, which needs to check whether the heat dissipation strategy is correct and whether there is a blocked air duct in the server case; the third reason is that the I3C bus of the processor CPU accessing the memory bank cannot respond, which needs to check whether there is a hidden danger in the current design. If the operation of the production line personnel is improper, the phenomenon will disappear, and the research and development personnel need to invest a lot of manpower and material resources to reproduce and analyze.

[0021] In the related scheme design, the following disadvantages exist: 1. In addition to the memhot problem caused by memory overheating, the memory overheating problem caused by the processor CPU and the I3C link exception needs to be measured in the open case to achieve it; 2. The measurement in the open case needs to find the measurement point. If it is on the back of the board card, it needs to be powered off and reproduced to capture, so as to make a rough judgment; 3. After the problem occurs, a soldering line and an oscilloscope or a logic analyzer are needed to reproduce and capture the problem waveform to judge the cause of the problem and replace the problem component.

[0022] Embodiments of the present application provide a memory fault processing method, as shown in Figure 2 、 Figure 3 The method is applied to a controller in a memory fault processing system. The system also includes a processor, a switching integrated circuit, a logic device, a plurality of switches, and a plurality of memory banks. The first output end of the controller is connected with the switching integrated circuit. The switching integrated circuit is connected with the plurality of switches correspondingly. Each switch is connected with the plurality of memory banks correspondingly. The second output end of the controller is connected with the logic device. The logic device is connected with the plurality of switches correspondingly. The logic device is connected with the processor. The processor is connected with the plurality of switches correspondingly. The method includes the following steps: Obtaining first serial bus communication information between the processor and the plurality of memory banks; According to the first serial bus communication information between the processor and the plurality of memory banks, the memory overheating fault type is determined. According to the memory overheating fault type data, the log file of the controller is generated, and the error data of the first serial bus communication information is stored. The memory overheating fault type includes the memory bank heat dissipation problem and the processor fault problem; If the controller's log file does not record data related to memory module overheating or processor malfunction, then error data is obtained from the first serial bus communication information. The fault type of the first serial bus communication information error is determined based on the error data of the first serial bus communication information.

[0023] Understandably, this application designs each group of I3C links to the memory module to connect to the logic FPGA, and uses the logic FPGA to monitor and capture data. At the same time, the logic FPGA adds a memory card (SD card) to store problem data, which facilitates the reading of data due to problems caused by the I3C link not responding, thereby analyzing the reasons for the I3C link not responding.

[0024] The first serial bus is an improved I2C serial bus, also known as the I3C serial bus.

[0025] Embodiments of this application provide a memory fault handling method, such as... Figure 4 As shown, the method is applied to the controller of a memory fault handling system. The system also includes a processor, a switching integrated circuit, a logic unit, several switches, and several memory modules. The first output terminal of the controller is connected to the switching integrated circuit, the switching integrated circuit is connected to several switches respectively, and each switch is connected to several memory modules respectively. The second output terminal of the controller is connected to the logic unit, the logic unit is connected to several switches respectively, the logic unit is connected to the processor, and the processor is connected to several switches respectively. The method includes: Step S01: Obtain the first serial bus communication information between the processor and several memory modules.

[0026] Step S02: Determine the memory overheating fault type based on the first serial bus communication information between the processor and several memory modules, generate a controller log file based on the memory overheating fault type data, and store the error data of the first serial bus communication information. The memory overheating fault type includes memory module heat dissipation problems and processor failure problems.

[0027] In step S021, the system also includes a memory card, which is connected to the logic unit; Memory overheating events are monitored using logic devices; Determine if the logic unit has received a memory overheat interrupt event; If yes, then determine whether the first serial bus communication information between the processor and several memory modules includes a start bit and an end bit; if no, then stop storing the first serial bus communication information data. Determine whether the first serial bus communication information between the processor and several memory modules includes start and end bits, including: In response to the first serial bus communication information between the processor and the plurality of memory banks including a start bit and an end bit, it is determined whether the first serial bus communication information between the processor and the plurality of memory banks includes a new start bit and an end bit after the previous end bit ends transmission; In response to the first serial bus communication information between the processor and the plurality of memory banks including a start bit or an end bit, it is determined whether the clock line and the data line of the first serial bus communication link between the switch and the plurality of memory banks are both at a high level; The determination of whether the first serial bus communication information between the processor and the plurality of memory banks includes a new start bit and an end bit after the previous end bit ends transmission includes: In response to the first serial bus communication information between the processor and the plurality of memory banks including a new start bit and an end bit after the previous end bit ends transmission, the storage of the data of the first serial bus communication information is stopped; In response to the first serial bus communication information between the processor and the plurality of memory banks including a start bit or an end bit after the previous end bit ends transmission, the error data of the first serial bus communication information is stored in the storage card, wherein the error data of the first serial bus communication information includes the waveform of the clock line or the data line, the transmission timestamp, and the physical identifier of the memory bank; The determination of whether the clock line and the data line of the first serial bus communication between the switch and the plurality of memory banks are both at a high level includes: In response to the clock line and the data line of the first serial bus communication between the switch and the plurality of memory banks being at a high level, the error data of the first serial bus communication information is stored in the storage card; In response to the clock line or the data line of the first serial bus communication between the switch and the plurality of memory banks being at a low level, it is determined whether the low level duration of the clock line or the data line of the first serial bus communication between the switch and the plurality of memory banks is greater than a first threshold (50 ms).

[0028] Specifically, the logic device FPGA simultaneously performs data capture and monitoring on the memory overheating event and the I3C data; After detecting the memory overheating event, it is determined whether to store the I3C data; If the memory overheating signal is triggered, there are two complete start bits and stop bits in the I3C link, the first data is not saved, and only the second data is saved in the I3C error register in the storage SD card for temporary storage; if there is no complete start bit and stop bit from the first I3C link data, it proves that the I3C link has a problem, and both data are stored in the I3C error register in the SD card; After the memory overheating event occurs, it is determined whether the I3C bus (a first serial bus between the switch and the plurality of memory banks) is idle (bus idle: the clock line SCL is high and the data line SDA is high), if the I3C bus is idle, the current I3C data is retained in the I3C error register of the SD card. If the I3C bus is not idle, it is determined whether the SCL (clock line) and SDA (data line) are always low.

[0029] In step S022, in response to the low level duration of the clock line or the data line of the first serial bus communication between the switch and the plurality of memory banks being greater than a first threshold, error data of the first serial bus communication information is stored in the memory card. In response to the low level duration of the clock line or the data line of the first serial bus communication between the switch and the plurality of memory banks being less than or equal to the first threshold and the clock line and the data line of the first serial bus communication between the plurality of memory banks having high-low frequency change data, the temperature of all memory banks is obtained, and it is determined whether the temperature of each memory bank is greater than a second threshold (80℃). The determination of whether the temperature of each memory bank is greater than the second threshold includes: In response to the temperature of any one memory bank being greater than the second threshold, it is determined that the memory overheating fault type is a memory bank heat dissipation problem. In response to the temperature of all memory banks being less than or equal to the second threshold, it is determined that the memory overheating fault type is a processor fault problem.

[0030] Specifically, if the data level of one clock line or data line is always low (the signal line is continuously low: the duration exceeds the normal communication clock period or the Clock Stretching time allowed by the protocol (for example, >1ms)), the FPGA records a piece of data on the I3C link into the error register. If the I3C bus is not idle, and there is high-low frequency change data on the clock line SCL and the data line SDA, the controller BMC needs to read the temperature of each memory bank, and if there is an overheated memory bank, the controller BMC records the overheated memory bank in the log log of the controller.

[0031] If the memory temperature is normal, it proves that the I3C link communication is normal, and the cause of the memory overheating event is the processor CPU itself, and the controller BMC records the processor CPU problem in the log log of the controller.

[0032] In step S03, in response to the controller's log file not recording memory bank heat dissipation problem or processor fault problem data, error data of the first serial bus communication information is obtained.

[0033] Step S04, determining the fault type of the first serial bus communication information error according to the error data of the first serial bus communication information.

[0034] Step S041, in response to the clock line of the first serial bus communication link being interrupted, determining the fault type of the first serial bus communication information error as a system problem, wherein the system problem includes a processor problem and a first serial bus communication abnormal interference problem; In response to the memory bank of the first serial bus (I3C) communication link not responding to transmission, the fault type of the first serial bus communication information error is determined as a memory bank fault problem; In response to the switch reading data from any one memory bank abnormally, the fault type of the first serial bus communication information error is determined as a system problem.

[0035] Specifically, the researchers can first exclude whether it is a processor CPU problem or a memory bank over-temperature problem by collecting the BMC one-key log; If the controller BMC one-key log does not record the processor CPU problem and the memory bank over-temperature problem, the content in the SD card of the logic device FPGA needs to be exported, and by reading the error I3C record data in the SD card, it can be recorded in detail when accessing which memory bank error occurs, and it can be judged whether it is a memory bank problem or a system problem.

[0036] According to the error I3C record data, the error can be classified, if the clock line of the first serial bus is not sent, it can be judged as a system problem; if the memory bank response is not normal, it can be judged as a memory problem; if the memory bank data is read abnormally, it can be judged as a system problem, and further investigation is needed to find out the reason for the data communication error.

[0037] Step S05, when the memory over-temperature fault type is a processor fault problem, the processor fault problem is processed; The processor fault problem is processed, including: When the processor appears over-temperature error, the fan speed corresponding to the processor is increased and / or the working frequency of the processor is reduced through the controller; When the processor sends an unmaskable interrupt or an alarm signal to the controller, the error code of the processor is obtained, and the processor is restarted according to the error code of the processor; When the processor sends a machine check exception signal to the controller, the working mode of the processor is set to a safe mode through the controller, and the error log of the processor is recorded.

[0038] Step S06, when the memory overheating fault type is a memory bank heat dissipation problem, then the memory bank heat dissipation problem is processed; The memory bank heat dissipation problem is processed, including: The temperature of each memory bank is obtained by the controller; In response to the temperature of any one memory bank being greater than a third threshold value (70°C ~ 75°C), it is determined that the memory bank has a heat dissipation problem, the memory bank is set as a target memory bank, and the target memory bank is subjected to heat dissipation processing; The target memory bank is subjected to heat dissipation processing, including: The memory bandwidth of the target memory bank is reduced by the controller and / or the access load of the target memory bank is reduced by reducing the working frequency of the processor; In response to the target memory bank being a non-critical memory bank, the target memory bank is set to a disabled mode; The temperature of the target memory bank is re-acquired at a first interval threshold value (2-5 seconds), and in response to the temperature of the target memory bank being less than or equal to the third threshold value, the memory bandwidth and access load of the target memory bank are restored.

[0039] Step S07, when the memory overheating fault type is a first serial bus link communication problem, then the first serial bus link communication problem is processed; The first serial bus link communication problem is processed, including: In response to the low level duration of the clock line and the data line of the first serial bus link being greater than a first threshold value (1ms), the universal input and output interface of the first serial bus link and the reset pin of the first serial bus buffer integrated circuit are controlled by the controller to reset the first serial bus link communication; In response to the dynamic address allocation of the first serial bus link failing, the dynamic address allocation process is sent to the first serial bus link by the controller to re-perform dynamic address allocation; In response to the first serial bus master controller of the processor being abnormal, the system is reset and / or the first serial bus master controller and the first serial bus link of the processor are re-initialized by the controller.

[0040] Step S08, in response to the input and output interface voltage of the power supply being abnormal or the voltage regulator efficiency being less than a fourth threshold value (85%), it is determined that the fault type of the memory overheating is a power supply problem, and the input and output interface voltage of the power supply is dynamically adjusted to a fifth threshold value (set according to the memory type, for example 1.144V); In response to the controller incorrectly identifying the first serial bus link communication data or incorrectly identifying the memory overheating event, it is determined that the fault type of the memory overheating is a controller firmware fault problem, the firmware of the controller is upgraded and / or the firmware of the controller is reset, and an alarm prompt is given. In response to the system error rate being greater than a sixth threshold value (10 times / hour / memory channel), a fault type of memory overheating is determined to be a system behavior problem, and the system memory frequency is dynamically adjusted and / or the load scheduling of each memory bank is optimized by the controller; In response to the thermal response characteristics of each memory bank being inconsistent, a fault type of memory overheating is determined to be a memory bank compatibility problem, and incompatible memory banks are isolated by the controller, and an alarm is prompted.

[0041] Here, as shown in Figure 5 The application can locate and record the root cause of memory overheating memhot, and the problem root cause can be classified and recorded by the controller BMC and the logic device FPGA in cooperation with reading data; For processor CPU problems and high memory temperature problems, the BMC can record in detail; For abnormal problems caused by I3C links, the logic device FPGA can record problem data and perform simple analysis.

[0042] In addition, the memory fault processing method further comprises: In response to the temperature reported by the digital temperature sensor on any one memory bank being greater than or equal to a seventh threshold value (85°C), the controller receives real-time temperature information and related states, adjusts the duty cycle PWM curve of the memory bank fan, and sends a network shared channel protocol (NCSI) or modern platform management protocol (PLDM) command to notify the server; In response to the sensor of the memory bank being damaged or having a calibration error, reporting a false high temperature, the controller determines whether the memory bank is a real overheating event through multi-source data cross-validation, and reports a "sensor failure" event; In response to the absence of a heat-conducting pad, air duct blockage, or insufficient fan speed of the memory bank, the controller increases the duty cycle PWM of all fans in the server to 100%, calculates the percentage of the decrease in heat dissipation efficiency, and reports a "high temperature exhaust / air outlet temperature" alarm; In response to the server room air conditioner being inoperable, causing temperature rise of each device in the server, the controller starts a hierarchical response according to the temperature preset threshold value; The hierarchical response according to the temperature preset threshold value comprises: When the air temperature at the server inlet is greater than or equal to a first preset value (35°C), the controller performs a pre-warning; When the air temperature at the server inlet is greater than or equal to a second preset value (40°C), the controller actively enhances heat dissipation; When the air temperature at the server inlet is greater than or equal to a third preset value (45°C), the controller performs frequency reduction processing on the memory, dynamically reduces the frequency of the processor, and actively reduces or disables the dynamic overclocking function of the processor; When the processor temperature ≥ the fourth preset value (90°C), the controller triggers to stop assigning new Pod services to the node and stop the non-critical services of the server; When the air temperature at the server inlet ≥ the fifth preset value (50°C), the controller reports a "soon to be overheated" prompt, and the data center management system initiates memory load migration; When the air temperature at the server inlet ≥ the sixth preset value (55°C), the controller executes the chassis power supply shutdown instruction to forcibly shut down the server; In response to the heat generated by the processor or voltage regulation module being conducted to the memory area through the circuit board, the fan speed is increased to prevent heat accumulation, and the high-bandwidth memory stick is preferentially scheduled to a channel away from the heat source.

[0043] Here, the complete server intelligent thermal management and memory failure response strategy is included, covering a closed-loop system from memory temperature monitoring, sensor reliability verification, heat dissipation control, multi-level overheating response to resource scheduling linkage, suitable for modern data centers, AI high-density servers, or intelligent server platforms supporting Redfish + OpenBMC architecture.

[0044] The memory failure processing method provided by the embodiments of the present application can also be improved and optimized without departing from the technical solutions of the present application, and these improvements and optimizations should also be considered within the scope of protection of the present application.

[0045] The present application can also use the logic device FPGA to monitor and analyze the waveforms of other signal buses, such as the universal asynchronous receiver-transmitter UART and the second serial bus I2C and serial peripheral interface SPI transmitted data.

[0046] The technical solutions provided by the embodiments of the present application have the following beneficial effects: The present application can troubleshoot memory overheating and memory failure problems caused by processor reasons without opening the chassis, reducing the time cost and research and development manpower of detection.

[0047] The technical solutions of the present application can troubleshoot problems caused by memory overheating and processor itself without opening the chassis. If it is a processor problem, the processor can be replaced directly without the need to measure, cross-verify and reproduce by opening the chassis; if it is an I3C link problem, the problem data can be preserved, and the problem occurring in which memory channel can be recorded, and whether it is a memory problem or a system problem can be roughly analyzed. For further reproduction and cross-verification analysis, there is data support, and the problem of cross-verification of the same I3C path when reproducing is avoided.

[0048] The technical scheme of the application can record problem data, cross-verify and reproduce problems in the early stage, can not pass through a logic analyzer and an oscilloscope, facilitates production line personnel to reproduce operation, and does not need R&D personnel to intervene too much, and for problems of a processor, only production line personnel need to read a controller log to determine to replace the processor.

[0049] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and a necessary general hardware platform, and of course, it can also be realized by hardware, but in many cases, the former is a better embodiment.

[0050] The embodiment of the application further provides a memory fault processing system, the system comprising a controller, a processor, a switching integrated circuit, a logic device, a plurality of switches and a plurality of memory banks, the first output end of the controller being connected with the switching integrated circuit, the switching integrated circuit being connected with the plurality of switches in correspondence respectively, each switch being connected with the plurality of memory banks in correspondence, the second output end of the controller being connected with the logic device, the logic device being connected with the plurality of switches in correspondence respectively, the logic device being connected with the processor, the processor being connected with the plurality of switches in correspondence respectively, the controller being used for: acquiring first serial bus communication information between the processor and the plurality of memory banks; determining a memory overheating fault type according to the first serial bus communication information between the processor and the plurality of memory banks, generating a log file of the controller according to the memory overheating fault type data, and storing error data of the first serial bus communication information, wherein the memory overheating fault type comprises a memory bank heat dissipation problem and a processor fault problem; in response to the log file of the controller not recording memory bank heat dissipation problem or processor fault problem data, acquiring error data of the first serial bus communication information; determining a fault type of the first serial bus communication information error according to the error data of the first serial bus communication information.

[0051] the controller is used for monitoring a memory overheating event through the logic device; judging whether the logic device receives a memory overheating interrupt event; if yes, judging whether the first serial bus communication information between the processor and the plurality of memory banks comprises a start bit and an end bit; if no, stopping storing the first serial bus communication information data; the judging whether the first serial bus communication information between the processor and the plurality of memory banks comprises a start bit and an end bit, comprising: in response to the first serial bus communication information between the processor and the plurality of memory banks including a start bit and an end bit, determining whether the first serial bus communication information between the processor and the plurality of memory banks includes a new start bit and an end bit after the previous end bit ends transmission; in response to the first serial bus communication information between the processor and the plurality of memory banks including a start bit or an end bit, determining whether the clock line and the data line of the first serial bus communication link between the switch and the plurality of memory banks are both at a high level; the determining whether the first serial bus communication information between the processor and the plurality of memory banks includes a new start bit and an end bit after the previous end bit ends transmission includes: in response to the first serial bus communication information between the processor and the plurality of memory banks including a new start bit and an end bit after the previous end bit ends transmission, stopping storing data of the first serial bus communication information; in response to the first serial bus communication information between the processor and the plurality of memory banks including a start bit or an end bit after the previous end bit ends transmission, storing error data of the first serial bus communication information in the storage card, wherein the error data of the first serial bus communication information includes a waveform of the clock line or the data line, a transmission time stamp, and a physical identifier of the memory bank; the determining whether the clock line and the data line of the first serial bus communication between the switch and the plurality of memory banks are both at a high level includes: in response to the clock line and the data line of the first serial bus communication between the switch and the plurality of memory banks being both at a high level, storing error data of the first serial bus communication information in the storage card; in response to the clock line or the data line of the first serial bus communication between the switch and the plurality of memory banks being at a low level, determining whether a low level duration of the clock line or the data line of the first serial bus communication between the switch and the plurality of memory banks is greater than a first threshold.

[0052] the controller is configured to: in response to the low level duration of the clock line or the data line of the first serial bus communication between the switch and the plurality of memory banks being greater than the first threshold, store error data of the first serial bus communication information in the storage card; In response to the low-level duration of the clock line or data line of the first serial bus communication between the switch and the plurality of memory modules being less than or equal to a first threshold and the clock line and data line of the first serial bus communication between the plurality of memory modules continuously having high and low frequency change data, the temperature of all memory modules is acquired, and it is determined whether the temperature of each memory module is greater than a second threshold. The determination of whether the temperature of each memory module is greater than the second threshold includes: If the temperature of any memory module exceeds the second threshold, the memory overheating fault type is determined to be a memory module heat dissipation problem. If the temperature of all memory modules is less than or equal to the second threshold, then the memory overheating fault type is determined to be a processor fault problem.

[0053] The controller is configured to: in response to an interruption in the clock line of the first serial bus communication link, determine that the fault type of the first serial bus communication information error is a system problem, wherein the system problem includes processor problems and abnormal interference problems in the first serial bus communication; If the memory module in the first serial bus communication link does not respond with a transmission response, then the fault type of the first serial bus communication information error is determined to be a memory module failure problem. In response to an abnormal data read from any memory module by the switch, the fault type of the first serial bus communication information error is determined to be a system problem.

[0054] The beneficial effects of the technical solutions provided in this application are: This application can troubleshoot memory failures caused by overheating or processor issues without opening the computer case for reproduction, reducing testing time and R&D manpower.

[0055] This application's technical solution allows for troubleshooting of memory overheating and processor-related issues without requiring the computer to be opened for reproduction. If the problem is with the processor, it can be directly replaced without opening the computer for measurement, cross-validation, or reproduction. If the problem is with the I3C link, it can retain the problem data and record which memory channel the problem occurred in. It also provides a general direction for analysis, indicating whether the problem is with the memory or the system. This data supports further reproduction and cross-validation analysis, preventing issues when cross-validating which memory module on the same I3C path is faulty during reproduction.

[0056] The technical solution of this application can record problem data. Early cross-validation and problem reproduction can be performed without the need for logic analyzers and oscilloscopes, making it easier for production line personnel to reproduce the problem without much intervention from R&D personnel. For problems with the processor, production line personnel only need to read the controller log to determine whether to replace the processor.

[0057] Embodiments of this application also provide a memory fault handling system, such as... Figure 2 As shown, the memory fault handling system includes a controller, a processor, a switching integrated circuit, a logic unit, several switches, and several memory modules. The first output terminal of the controller is connected to the switching integrated circuit, the switching integrated circuit is connected to each of the several switches, each switch is connected to each of the several memory modules, the second output terminal of the controller is connected to the logic unit, the logic unit is connected to the several switches, the logic unit is connected to the processor, and the processor is connected to each of the several switches. The system also includes a memory card, which is connected to the logic unit. The processor is connected to the plurality of switches via a first serial bus link; Each switch is connected to the plurality of memory modules via a first serial bus link; The controller and the logic unit are connected via a second serial bus link.

[0058] The second serial bus is an I2C serial bus.

[0059] The beneficial effects of the technical solutions provided in this application are: This application can troubleshoot memory failures caused by overheating or processor issues without opening the computer case for reproduction, reducing testing time and R&D manpower.

[0060] This application's technical solution allows for troubleshooting of memory overheating and processor-related issues without requiring the computer to be opened for reproduction. If the problem is with the processor, it can be directly replaced without opening the computer for measurement, cross-validation, or reproduction. If the problem is with the I3C link, it can retain the problem data and record which memory channel the problem occurred in. It also provides a general direction for analysis, indicating whether the problem is with the memory or the system. This data supports further reproduction and cross-validation analysis, preventing issues when cross-validating which memory module on the same I3C path is faulty during reproduction.

[0061] The technical solution of this application can record problem data. Early cross-validation and problem reproduction can be performed without the need for logic analyzers and oscilloscopes, making it easier for production line personnel to reproduce the problem without much intervention from R&D personnel. For problems with the processor, production line personnel only need to read the controller log to determine whether to replace the processor.

[0062] For a description of the features in the embodiment corresponding to the memory fault handling system, please refer to the relevant description of the embodiment corresponding to the memory fault handling method, which will not be repeated here.

[0063] Embodiments of this application also provide an electronic device, which includes a memory fault handling system. The memory fault handling system includes a controller, a processor, a switching integrated circuit, a logic unit, a plurality of switches, and a plurality of memory modules. A first output terminal of the controller is connected to the switching integrated circuit, the switching integrated circuit is connected to each of the plurality of switches, each switch is connected to each of the plurality of memory modules, a second output terminal of the controller is connected to the logic unit, the logic unit is connected to the plurality of switches, the logic unit is connected to the processor, and the processor is connected to each of the plurality of switches. The system also includes a memory card, which is connected to the logic unit. The processor and the plurality of switches are connected via a first serial bus link I3C (Improved Inter-Integrated Circuit); Each switch is connected to the plurality of memory modules via a first serial bus link; The controller and the logic unit are connected via a second serial bus link, I2C.

[0064] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in the memory fault handling method embodiment, the method including: Obtain the first serial bus communication information between the processor and the plurality of memory modules; The memory overheating fault type is determined based on the first serial bus communication information between the processor and the plurality of memory modules. The controller's log file is generated based on the memory overheating fault type data, and the error data of the first serial bus communication information is stored. The memory overheating fault type includes memory module heat dissipation problems and processor failure problems. If no data on memory module overheating or processor malfunction is recorded in the controller's log file, then error data of the first serial bus communication information is obtained. The fault type of the first serial bus communication information error is determined based on the error data of the first serial bus communication information.

[0065] like Figure 6 As shown, embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in the memory fault handling method embodiments at runtime, the method including: Obtain the first serial bus communication information between the processor and the plurality of memory modules; The memory overheating fault type is determined based on the first serial bus communication information between the processor and the plurality of memory modules. The controller's log file is generated based on the memory overheating fault type data, and the error data of the first serial bus communication information is stored. The memory overheating fault type includes memory module heat dissipation problems and processor failure problems. If no data on memory module overheating or processor malfunction is recorded in the controller's log file, then error data of the first serial bus communication information is obtained. The fault type of the first serial bus communication information error is determined based on the error data of the first serial bus communication information.

[0066] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0067] Embodiments of this application also provide a computer program product, which includes a computer program. When executed by a processor, the computer program implements the steps in the memory fault handling method embodiment, the method including: Obtain the first serial bus communication information between the processor and the plurality of memory modules; The memory overheating fault type is determined based on the first serial bus communication information between the processor and the plurality of memory modules. The controller's log file is generated based on the memory overheating fault type data, and the error data of the first serial bus communication information is stored. The memory overheating fault type includes memory module heat dissipation problems and processor failure problems. If no data on memory module overheating or processor malfunction is recorded in the controller's log file, then error data of the first serial bus communication information is obtained. The fault type of the first serial bus communication information error is determined based on the error data of the first serial bus communication information.

[0068] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the steps in the memory fault handling method embodiments, the method including: Obtain the first serial bus communication information between the processor and the plurality of memory modules; The memory overheating fault type is determined based on the first serial bus communication information between the processor and the plurality of memory modules. The controller's log file is generated based on the memory overheating fault type data, and the error data of the first serial bus communication information is stored. The memory overheating fault type includes memory module heat dissipation problems and processor failure problems. If no data on memory module overheating or processor malfunction is recorded in the controller's log file, then error data of the first serial bus communication information is obtained. The fault type of the first serial bus communication information error is determined based on the error data of the first serial bus communication information.

[0069] This application can troubleshoot memory failures caused by overheating or processor issues without opening the computer case for reproduction, reducing testing time and R&D manpower.

[0070] This application's technical solution allows for troubleshooting of memory overheating and processor-related issues without requiring the computer to be opened for reproduction. If the problem is with the processor, it can be directly replaced without opening the computer for measurement, cross-validation, or reproduction. If the problem is with the I3C link, it can retain the problem data and record which memory channel the problem occurred in. It also provides a general direction for analysis, indicating whether the problem is with the memory or the system. This data supports further reproduction and cross-validation analysis, preventing issues when cross-validating which memory module on the same I3C path is faulty during reproduction.

[0071] The technical solution of this application can record problem data. Early cross-validation and problem reproduction can be performed without the need for logic analyzers and oscilloscopes, making it easier for production line personnel to reproduce the problem without much intervention from R&D personnel. For problems with the processor, production line personnel only need to read the controller log to determine whether to replace the processor.

[0072] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0073] The present application provides a detailed description of a memory fault handling method and system. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A memory fault handling method, characterized in that, The method is applied to a controller in a memory fault handling system. The system further includes a processor, a switching integrated circuit, a logic unit, several switches, and several memory modules. A first output terminal of the controller is connected to the switching integrated circuit. The switching integrated circuit is connected to each of the several switches, and each switch is connected to one of the several memory modules. A second output terminal of the controller is connected to the logic unit, which is connected to each of the several switches. The logic unit is connected to the processor, and the processor is connected to each of the several switches. The method includes: Obtain the first serial bus communication information between the processor and the plurality of memory modules; The memory overheating fault type is determined based on the first serial bus communication information between the processor and the plurality of memory modules. The controller's log file is generated based on the memory overheating fault type data, and the error data of the first serial bus communication information is stored. The memory overheating fault type includes memory module heat dissipation problems and processor failure problems. If no data on memory module overheating or processor malfunction is recorded in the controller's log file, then error data of the first serial bus communication information is obtained. The fault type of the first serial bus communication information error is determined based on the error data of the first serial bus communication information.

2. The memory fault handling method according to claim 1, characterized in that, The system also includes a memory card connected to the logic unit. The step of determining the memory overheating fault type based on the first serial bus communication information between the processor and the plurality of memory modules includes: The logic unit monitors memory overheating events. Determine whether the logic unit has received a memory overheat interrupt event; If yes, then determine whether the first serial bus communication information between the processor and the plurality of memory modules includes a start bit and an end bit; if no, then stop storing the first serial bus communication information data. The determination of whether the first serial bus communication information between the processor and the plurality of memory modules includes a start bit and an end bit includes: In response to the fact that the first serial bus communication information between the processor and the plurality of memory modules includes a start bit and an end bit, a determination is made as to whether the first serial bus communication information between the processor and the plurality of memory modules includes a new start bit and an end bit after the transmission ends at the previous end bit. In response to the fact that the first serial bus communication information between the processor and the plurality of memory modules includes a start bit or an end bit, it is determined whether the clock line and data line levels of the first serial bus communication link between the switch and the plurality of memory modules are both high. The determination of whether the first serial bus communication information between the processor and the plurality of memory modules includes new start and end bits after the previous end bit ends the transmission includes: In response to the fact that the first serial bus communication information between the processor and the plurality of memory modules includes a new start bit and an end bit after the previous end bit ends the transmission, the storage of the first serial bus communication information data is stopped. In response to the first serial bus communication information between the processor and the plurality of memory modules including a start bit or an end bit after the previous end bit ends the transmission, the error data of the first serial bus communication information is stored in the memory card, wherein the error data of the first serial bus communication information includes the waveform of the clock line or data line, the transmission timestamp, and the physical identifier of the memory module.

3. The memory fault handling method according to claim 2, characterized in that, The determination of whether the clock line and data line levels of the first serial bus communication between the switch and the plurality of memory modules are both high includes: In response to the clock line and data line levels of the first serial bus communication between the switch and the plurality of memory modules being both high, the erroneous data of the first serial bus communication information is stored in the memory card. In response to a low level on the clock line or data line of the first serial bus communication between the switch and the plurality of memory modules, a determination is made as to whether the duration of the low level on the clock line or data line of the first serial bus communication between the switch and the plurality of memory modules is greater than a first threshold. The determination of whether the low-level duration of the clock line or data line for communication between the switch and the plurality of memory modules is greater than a first threshold includes: If the low-level duration of the clock line or data line of the first serial bus communication between the switch and the plurality of memory modules is greater than a first threshold, then the erroneous data of the first serial bus communication information is stored in the memory card. In response to the low-level duration of the clock line or data line of the first serial bus communication between the switch and the plurality of memory modules being less than or equal to a first threshold and the clock line and data line of the first serial bus communication between the plurality of memory modules continuously having high and low frequency change data, the temperature of all memory modules is acquired, and it is determined whether the temperature of each memory module is greater than a second threshold. The determination of whether the temperature of each memory module is greater than the second threshold includes: If the temperature of any memory module exceeds the second threshold, the memory overheating fault type is determined to be a memory module heat dissipation problem. If the temperature of all memory modules is less than or equal to the second threshold, then the memory overheating fault type is determined to be a processor fault problem.

4. The memory fault handling method according to claim 1, characterized in that, The step of determining the fault type of the first serial bus communication information error based on the error data of the first serial bus communication information includes: In response to an interruption in the clock line of the first serial bus communication link, the fault type of the first serial bus communication information error is determined to be a system problem, wherein the system problem includes processor problems and abnormal interference problems in the first serial bus communication; If the memory module in the first serial bus communication link does not respond with a transmission response, then the fault type of the first serial bus communication information error is determined to be a memory module failure problem. In response to an abnormal data read from any memory module by the switch, the fault type of the first serial bus communication information error is determined to be a system problem.

5. The memory fault handling method according to claim 1, characterized in that, After determining the memory overheating fault type based on the first serial bus communication information between the processor and the plurality of memory modules, the process includes: When the memory overheating fault type is a processor fault, then the processor fault is handled. The handling of processor malfunctions includes: When the processor experiences an over-temperature error, the controller increases the fan speed corresponding to the processor and / or decreases the processor's operating frequency. When the processor sends a non-maskable interrupt or alarm signal to the controller, the error code of the processor is obtained, and the processor is restarted according to the error code. When the processor sends a machine check anomaly signal to the controller, the controller sets the processor's operating mode to safe mode and records the processor's error log.

6. The memory fault handling method according to claim 1, characterized in that, After determining the memory overheating fault type based on the first serial bus communication information between the processor and the plurality of memory modules, the method further includes: When the memory overheating fault type is a memory module heat dissipation problem, then the memory module heat dissipation problem should be addressed. The measures taken to address the memory module overheating issue include: The temperature of each memory module is obtained through the controller; If the temperature of any memory module exceeds the third threshold, it is determined that the memory module has a heat dissipation problem, the memory module is set as the target memory module, and heat dissipation treatment is performed on the target memory module; The heat dissipation treatment of the target memory module includes: The memory bandwidth of the target memory module is reduced by the controller and / or the access load of the target memory module is reduced by reducing the operating frequency of the processor; If the target memory module is a non-critical memory module, then the target memory module is set to a disabled mode; The temperature of the target memory module is reacquired according to the first interval threshold. If the temperature of the target memory module is less than or equal to the third threshold, the memory bandwidth and access load of the target memory module are restored.

7. The memory fault handling method according to claim 1, characterized in that, After determining the fault type of the first serial bus communication information error based on the error data of the first serial bus communication information, the process includes: When the memory overheating fault type is a first serial bus link communication problem, then the first serial bus link communication problem is handled. The handling of the first serial bus link communication problem includes: If the low-level duration of both the clock line and data line of the first serial bus link is greater than the first threshold, the controller controls the general-purpose input / output interface of the first serial bus link and the reset pin of the first serial bus buffer integrated circuit to reset the communication of the first serial bus link. If the dynamic address allocation of the first serial bus link fails, the controller sends a dynamic address allocation process to the first serial bus link to re-allocate the dynamic address. In response to an anomaly in the processor's first serial bus master controller, the system is reset and / or the processor's first serial bus master controller and first serial bus link are reinitialized via the controller.

8. The memory fault handling method according to claim 1, characterized in that, The method further includes: If the input / output interface voltage of the power supply is abnormal or the efficiency of the voltage regulator is less than the fourth threshold, the fault type of the memory overheating is determined to be a power supply problem, and the input / output interface voltage of the power supply is dynamically adjusted to the fifth threshold. In response to the controller incorrectly identifying the first serial bus link communication data or incorrectly identifying the memory overheating event, the fault type of the memory overheating is determined to be a firmware fault problem of the controller. The firmware of the controller is upgraded and / or the firmware of the controller is reset, and an alarm is issued. If the system error rate exceeds the sixth threshold, the fault type of the memory overheating is determined to be a system behavior problem, and the system memory frequency is dynamically adjusted and / or the load scheduling of each memory module is optimized by the controller. If the thermal response characteristics of the various memory modules are inconsistent, the fault type of the memory overheating is determined to be a memory module compatibility issue. The controller then isolates the incompatible memory modules and issues an alarm.

9. The memory fault handling method according to claim 1, characterized in that, The processor is connected to the plurality of switches via a first serial bus link; Each switch is connected to the plurality of memory modules via a first serial bus link; The controller and the logic unit are connected via a second serial bus link.

10. A memory fault handling system, characterized in that, The system includes a controller, a processor, a switching integrated circuit, a logic unit, several switches, and several memory modules. The first output of the controller is connected to the switching integrated circuit. Each switching integrated circuit is connected to one of the several switches, and each switch is connected to one of the several memory modules. The second output of the controller is connected to the logic unit. The logic unit is connected to the processor, and the processor is connected to one of the several switches. The controller is used for: Obtain the first serial bus communication information between the processor and the plurality of memory modules; The memory overheating fault type is determined based on the first serial bus communication information between the processor and the plurality of memory modules. The controller's log file is generated based on the memory overheating fault type data, and the error data of the first serial bus communication information is stored. The memory overheating fault type includes memory module heat dissipation problems and processor failure problems. If no data on memory module overheating or processor malfunction is recorded in the controller's log file, then error data of the first serial bus communication information is obtained. The fault type of the first serial bus communication information error is determined based on the error data of the first serial bus communication information.

Images