Fault rapid positioning method and device in MGX system

Abstract

The invention provides a fault rapid positioning method and device in an MGX system, and belongs to the field of server system management and fault diagnosis. The method comprises the following steps: determining an idle universal serial bus interface on a mainboard, and selecting a target USB interface and a fixed IP network segment configuration corresponding to the target USB interface from the idle USB interface; the BMC establishes a physical connection with a device where the HMC is located through the target USB interface, and constructs a virtual local area network channel; in the virtual local area network channel, performing out-of-band communication between the BMC and the HMC; the BMC obtains list information of each component in the MGX system from the HMC, and classifies the list information according to component types; the HMC pre-collects and stores the state information of each category according to the classification result; and the BMC periodically reads the state information based on the Redfish protocol, and identifies the fault in the MGX system according to the state information. According to the method and the device provided by the invention, the fault can be quickly and accurately positioned.

Description

Technical Field

[0001] The present application relates to the technical field of server system management and fault diagnosis, and in particular to a method and device for quickly locating faults in an MGX system. Background Art

[0002] As a core platform for high-performance computing and AI training, the MGX (Modular GPU eXtension) system uses NVLink Switch to achieve high-speed interconnection of multiple GPUs. With bandwidth and latency advantages far exceeding PCIe, it supports large-scale parallel computing tasks (such as deep learning model training and scientific computing simulations). With the surge in demand for AI computing power, the number of GPUs and interconnection complexity in the MGX system continue to increase. The impact of hardware failures (such as GPU data transmission errors, NVLink link interruptions, and GPU crashes) on business continuity is becoming increasingly significant, necessitating efficient fault location methods to ensure stable system operation.

[0003] Currently, fault location in the MGX system relies primarily on two technologies. The first is an OS-based in-band diagnostic method, which uses tools (such as nvidia-smi) under the system operating system (such as Linux) to query GPU status, NVLink link topology, and error counts, and analyzes the cause of the fault in combination with GPU driver logs (such as / var / log / nvidia / ). The second is the traditional out-of-band management method, which uses the IPMI protocol in some scenarios to collect basic hardware status (such as temperature and power supply) through the BMC (baseboard management controller).

[0004] Existing technologies have significant limitations in locating MGX system faults. For OS-based in-band diagnostic methods, core fault information (such as NVLink error logs) must be obtained through the OS and driver. If a GPU crash causes the OS to crash or the driver to fail, critical data will be lost, making it impossible to trace the root cause of the fault. Logging into the OS to query status requires interrupting or affecting ongoing business tasks (such as AI training), posing a risk of business interruption and failing to meet high-availability scenario requirements. Fault information is scattered across multiple data sources, including tool output, driver logs, and hardware registers, requiring manual integration and analysis. The location process is time-consuming and difficult to meet real-time requirements. Furthermore, different manufacturers have different ways of querying the status of GPUs and NVLink switches (such as command parameters and log formats), lacking standardized interfaces, increasing the complexity of cross-platform fault location. Traditional out-of-band management methods can only obtain surface status. Key information such as the core NVLink link status and GPU internal errors still needs to rely on the OS and driver to obtain, resulting in insufficient fault diagnosis depth. In addition, the IPMI protocol uses a binary data format, which is complex to parse and has poor compatibility with heterogeneous hardware from multiple vendors. It is difficult to support the refined fault location requirements of multiple GPUs and high-speed interconnect links in the MGX system. Summary of the Invention

[0005] In view of this, the present application provides a method and apparatus for rapid fault location in an MGX system, to solve the problems of fuzzy positioning, low diagnostic efficiency, unclear maintenance guidance, etc. caused by reliance on a single data source in traditional fault diagnosis.

[0006] Specifically, this application is implemented through the following technical solutions:

[0007] In a first aspect, the present application provides a method for quickly locating a fault in an MGX system, wherein the MGX system includes a management controller (BMC) and a hardware management controller (HMC) provided on a mainboard. The method includes:

[0008] The management controller BMC determines an idle universal serial bus (USB) interface on the mainboard, and selects a target USB interface and a fixed IP network segment configuration corresponding to the target USB interface.

[0009] The BMC establishes a physical connection with the device where the HMC is located through the target USB interface, and constructs a USB virtual local area network channel based on a fixed IP segment configuration. In the USB virtual local area network channel, the BMC and the HMC use the Redfish protocol for out-of-band communication;

[0010] The BMC obtains the inventory information of each component in the MGX system from the HMC based on the Redfish protocol and classifies the components according to the component type;

[0011] The HMC pre-collects status information of each category according to the classification result and stores it in the MGX system;

[0012] The BMC periodically reads the status information based on the Redfish protocol and identifies faults in the MGX system according to the status information.

[0013] A second aspect of the present application provides a device for quickly locating a fault in an MGX system, the device comprising:

[0014] A mainboard, on which a management controller BMC is provided;

[0015] A hardware management controller HMC, communicating with the BMC;

[0016] A universal serial bus (USB) interface, used to establish a USB virtual local area network channel based on a fixed IP segment configuration between the BMC and the device where the HMC is located;

[0017] a storage module, configured to store component status information and fault diagnosis information collected by the HMC;

[0018] The processor module is configured to execute any one of the methods for quickly locating faults in the MGX system.

[0019] The method and device for rapid fault location in the MGX system provided in this application realizes stable out-of-band communication between BMC and HMC that is independent of the business network by constructing a USB virtual LAN channel based on an idle USB interface and adopting a fixed IP segment configuration. This communication method does not rely on the host operating system and the business network. Even if the system crashes or the network is interrupted, it can still ensure the transmission of management data, solving the problem of data loss during failure caused by traditional reliance on the OS or business network. On this basis, standardized out-of-band communication is carried out with the help of the Redfish protocol, unifying the interactive interface between BMC and HMC. In the USB virtual LAN out-of-band channel of this solution, the standardized HTTP / HTTPS request-response mechanism of the Redfish protocol can adapt to the hardware differences between HMCs and BMCs of multiple manufacturers, ensuring that the status data of core components such as GPUs and NVSwitch are reliably transmitted and parsed in independent channels, avoiding the compatibility barriers caused by private communication protocols of devices from different manufacturers, and providing a unified interface support for the integration and analysis of fault data of cross-platform hardware. BMC can obtain status data and make abnormal judgments in a targeted manner according to component type, replacing the traditional manual query and analysis model, improving the timeliness and accuracy of fault discovery, and laying the foundation for rapid fault location. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Flowchart of Example 1 of the method for rapid fault location in the MGX system provided by this application;

[0021] Figure 2 This is a structural diagram of Embodiment 2 of the apparatus for rapid fault location in the MGX system provided in this application. DETAILED DESCRIPTION

[0022] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different drawings represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with this application.

[0023] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms "a," "the," and "the" used in this application are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0024] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0025] Specific embodiments are given below to introduce the technical solutions of the present application in detail.

[0026] Example 1

[0027] It should be noted that the MGX system provided in this application includes a management controller BMC and a hardware management controller HMC arranged on a mainboard. The MGX system is a hardware system (similar to a server / computing node cluster) designed for scenarios such as high-performance computing and deep learning. It integrates components such as multiple GPUs and high-speed interconnects, providing hardware support for large-scale parallel computing and AI training and inference. It can efficiently handle complex computing tasks such as training large language models and performing scientific simulations. The motherboard is the hardware foundation of the MGX system, hosting components such as the CPU, memory, various interfaces (such as USB and PCIe), and a management controller. It is responsible for the physical connection and signal transmission between hardware components. The management controller (BMC) is a dedicated management chip / module on the motherboard that can run independently of the host operating system. It monitors and manages the motherboard and associated hardware (such as temperature, voltage, and device status), and can also perform out-of-band communication (communication methods that do not rely on the host service network). The hardware management controller (HMC) focuses on system hardware-level management and can deeply collect and store status data of multiple components (such as GPUs, PCIe links, NVSwitches, etc.). It cooperates with the BMC to complete hardware monitoring and fault diagnosis, providing underlying data support for functions such as fault location.

[0028] It's important to note that the BMC and HMC work together in a collaborative and specialized manner. The BMC focuses on basic motherboard and system-level management and communication scheduling (for example, building virtual networks via USB and communicating using the Redfish protocol), while the HMC specializes in in-depth hardware status collection and storage (such as fine-grained status data for GPUs and PCIe links). Working together, the BMC can initiate data queries, log collection, and other commands to the HMC, which responds with hardware status feedback, providing a closed-loop data loop for fault location and system health management.

[0029] Based on the configuration of this application, from system power-on initialization to full-load operation, the BMC and HMC work together to monitor hardware temperature, voltage, link status, and more, providing early warning of potential failures (such as overtemperature triggering frequency reduction or link error count exceeding the limit alarm). It should also be noted that the HMC pre-stores the status of multiple components, and the BMC can quickly pull data and, combined with protocol parsing and log analysis, accurately locate faults such as GPUs and PCIe links, shortening troubleshooting time (compared to systems without independent management controllers, this reduces the time cost of manually troubleshooting hardware one by one). In addition, a management channel independent of the host OS (such as the out-of-band network established by the BMC via USB) allows remote access to the BMC and HMC for hardware status query and fault diagnosis even if the host OS crashes, ensuring controllable system operations and reducing the risk of business interruption.

[0030] Figure 1 This is a flowchart of the first embodiment of the method for rapid fault location in the MGX system provided by this application. Figure 1 The method provided in this embodiment may include:

[0031] S101: Determine an idle Universal Serial Bus (USB) interface on a mainboard, and select a target USB interface and a fixed IP network segment configuration corresponding to the target USB interface.

[0032] It should be noted that the Universal Serial Bus (USB) is a standardized hardware interface used for data transmission and power supply between devices. In the MGX system, the USB interface can be used as a management channel to enable the BMC to communicate with other hardware (such as the HMC), and it has the characteristics of plug-and-play and strong compatibility. Fixed IP segment configuration refers to the allocation of fixed IP addresses and subnet masks to network devices (such as BMC and HMC connected via USB), so that the devices are in the same logical network, ensuring stable and predictable communication (different from the connection fluctuations that may be caused by dynamic IP).

[0033] Specifically, the BMC calls the underlying hardware driver to traverse all USB physical interfaces on the motherboard (such as USB 3.0 and USB-C interfaces). By querying the interface status register, it selects interfaces that are idle (no device in use, no faults or errors), creating a list of available USB interfaces. Then, based on preconfigured rules (such as interface bandwidth priority and physical location priority), it selects the optimal interface from this list. For example, it prioritizes USB 3.0 interfaces (which offer higher bandwidth and meet the needs of large log transfers). If multiple interfaces of the same type are available, the physical interface closer to the motherboard management area (which offers shorter cabling and less signal interference) is selected and marked as the target USB interface. Finally, the BMC reads the system's pre-stored IP segment mapping table (for example, USB interface A corresponds to IP segment 192.168.10.0 / 24, and USB interface B corresponds to IP segment 192.168.20.0 / 24). Based on the selected target USB interface, the BMC automatically associates and loads the corresponding fixed IP segment configuration (including parameters such as IP address, subnet mask, and gateway). In the IP segment mapping table, each USB interface corresponds to a unique fixed IP segment, thereby avoiding the risk of IP conflicts when multiple USB interfaces are enabled at the same time and ensuring the stability of communication between the BMC and HMC.

[0034] Based on the above description, in the MGX system, the BMC and HMC require a stable out-of-band communication channel. However, the motherboard USB interface may be occupied by other devices (such as external storage and debugging tools), and dynamic IP configuration can easily lead to communication interruptions (such as IP conflicts and address changes after restart), affecting fault diagnosis data transmission. In this step, the BMC actively scans and filters idle USB interfaces, selects the target interface based on preconfigured rules, and binds it to a fixed IP network segment to build a reliable communication foundation. This avoids dynamic IP conflicts and address drift, ensuring long-term and stable communication between the BMC and HMC. By prioritizing high-bandwidth, low-interference USB interfaces, data transmission delays are reduced, accelerating the fault diagnosis process. In addition, standardized USB interfaces are compatible with various hardware scenarios (new and old motherboards, different HMC devices). When adding new devices later, this logic can be reused to quickly establish communication, reducing O&M complexity.

[0035] S102: The BMC establishes a physical connection with the device where the HMC is located through the target USB interface, and constructs a USB virtual local area network channel based on a fixed IP network segment configuration. In the USB virtual local area network channel, the BMC and the HMC use the Redfish protocol for out-of-band communication.

[0036] It should be noted that a physical connection refers to the connection between the BMC and the device where the HMC is located, connecting the physical pins of the target USB interface via a USB cable to establish a data transmission path at the hardware level. This is the basis for the virtual network channel. The USB virtual LAN channel (LAN over USB) is a technology that virtualizes the USB interface into an Ethernet network interface, allowing the USB interface originally used for peripheral connection to transmit network data like a network cable. Simply put, it uses USB as a network cable to build a virtual LAN between devices, allowing the BMC and HMC to achieve network communication through the USB interface without the need for additional physical network cables. During this process, the BMC needs to configure the network segment of its own virtual network interface based on the fixed IP network segment configuration corresponding to the target USB interface. At the same time, the USB virtual network interface of the device where the HMC is located must also be configured with an IP address in the same network segment. The two achieve stable network layer communication through the static IP network segment matching mechanism.

[0037] Specifically, the operator plugs one end of a USB cable into the target USB port on the motherboard housing the BMC and the other end into the USB port on the device housing the HMC (it's important to confirm in advance that the HMC supports LAN over USB and that the port is available). This completes the physical connection at the hardware level, enabling electrical signal transmission between the two ends. The BMC firmware then invokes the USB controller driver to switch the target USB port's operating mode to USB Ethernet mode (this means using the USB network driver to virtualize the USB port into an Ethernet network port, giving it the data link layer communication capabilities of a network card). The corresponding function must also be enabled on the USB port on the HMC (either automatically identified by the HMC or enabled through underlying configuration) so that both ends use the USB channel as a network channel. Furthermore, the BMC sets an IP address (e.g., 192.168.10.2) and subnet mask (e.g., 255.255.255.0) for the virtualized USB network interface based on the fixed IP segment configuration selected by S101. At the same time, the corresponding virtual interface on the device where the HMC resides must be configured with an IP address in the same network segment (e.g., 192.168.10.3) to ensure communication between both ends within the same virtual local area network (VLAN). Finally, the BMC sends a simple network request (e.g., a ping command) to the device where the HMC resides through the virtual network interface. If a response packet is received from the HMC, the USB VLAN channel is successfully established. If it fails, a retry attempt or an error is triggered (e.g., to check the IP configuration or whether the USB cable is loose). This establishes a dual physical and virtual network channel between the BMC and HMC, forming a stable out-of-band communication link that can be used to transmit Redfish protocol data, fault diagnosis logs, and more.

[0038] In specific implementation, a USB virtual LAN channel is constructed based on a fixed IP segment configuration, including:

[0039] (1) In the MGX system, the management controller BMC is configured to have a USB Ethernet function module and is set to the master device mode.

[0040] It should be noted that the USB Ethernet function module refers to the feature enabled in the BMC firmware / driver to emulate a USB interface as an Ethernet network card. Simply put, this allows a USB interface, previously limited to connecting USB flash drives and mice, to have network communication capabilities similar to a network cable interface, capable of transmitting IP packets. Master mode is a USB interface operating mode in which the BMC acts as the host, actively identifying and managing connected devices (such as the device housing the HMC). This is similar to connecting a mobile phone to a computer via USB. The computer is the master device, capable of charging and transferring data. In slave mode, the BMC actively manages the USB connection to the HMC, so it is set as the master device.

[0041] Specifically, enable the USB Ethernet function in the BMC system settings (or underlying driver configuration) (the path varies for different BMC firmware, such as loading the driver through the iplink command or selecting the function in the web management interface). Furthermore, set the BMC's target USB interface mode to the master device so that the BMC can actively control the connected HMC device instead of passively waiting for a connection.

[0042] (2) Connecting the BMC to the device where the hardware management controller HMC is located through the target USB interface, and configuring a static IP address for the USB virtual local area network channel; wherein, after the configuration is completed, the BMC and the HMC are in the same fixed network segment.

[0043] As described above, the operation and maintenance personnel use a USB cable, plugging one end of the target USB port on the BMC (configured in master mode) and the other end into the USB port on the device where the HMC is located (which must support USB slave mode). On the BMC side, configure a static IP address and subnet mask for the virtual Ethernet adapter. On the HMC side, configure a static IP address in the same network segment for the corresponding USB network interface.

[0044] (3) Starting a virtual Ethernet adapter on the BMC, identifying and initializing the USB virtual LAN channel.

[0045] It should be noted that the virtual Ethernet adapter is a software-emulated network card within the BMC system. It encapsulates the USB interface data stream into Ethernet frames through the driver layer, enabling the BMC to leverage the USB Ethernet function and implement network data transmission and reception operations through the USB interface, consistent with those of a real network cable network card. Specifically, the BMC system invokes the virtual Ethernet adapter startup command, and after the adapter starts, it automatically identifies the hardware parameters of the USB virtual LAN channel and completes initialization, allowing the BMC to transmit and receive network data through the adapter.

[0046] (4) After the initialization is completed, a network communication link is established between the BMC and the HMC based on the static IP address.

[0047] Specifically, the BMC sends a TCP SYN packet (a request to establish a connection) to the HMC's IP address through the virtual adapter. After receiving the packet, the HMC replies with a SYN+ACK packet, and the BMC replies with an ACK packet. This completes the TCP three-way handshake and establishes a stable network connection. Redfish protocol data can then be transmitted over this link.

[0048] It should be noted that the TCP / IP protocol stack automatically completes the process of addressing → connection → data transmission. Because step (2) has ensured the same network segment, the data packet sent by the BMC will be directly delivered to the corresponding network interface of the HMC through the virtual adapter + USB channel, without the need for complex routing, thus achieving the purpose of efficient transmission.

[0049] In this step, a USB virtual network is physically isolated from the service network. Even if the host OS crashes or the service network is disconnected, the BMC can still access the HMC via USB and retrieve fault logs (such as EROT information), ensuring uninterrupted troubleshooting. Static IP configuration combined with master mode avoids dynamic IP conflicts. Master mode allows the BMC to proactively manage the connection, reducing the risk of connection interruptions. It's also important to note that the USB virtual LAN established in this step is an out-of-band communication channel, completely decoupled from the host OS and service network. Even if the MGX system crashes due to GPU crashes, NVLink link interruptions, or the service network is disconnected due to overload or configuration errors, the BMC and HMC can still maintain communication via the physical USB connection and the virtual LAN. This feature completely eliminates the single point of failure associated with traditional in-band diagnostics (which rely on the OS) and some out-of-band solutions (which rely on the service network). Even if a core system component fails, the BMC can still retrieve the hardware status data required for troubleshooting (such as EROT logs and FPGA register snapshots) through the out-of-band channel. In addition, the motherboard's idle USB interface is utilized without additional wiring, making it suitable for the high-density hardware environment of the MGX system (multi-GPU, multi-module stacking). When adding new nodes, the management channel can be expanded by reusing the USB connection logic, making operation and maintenance simpler.

[0050] It's important to note that the Redfish protocol is an open standard network management protocol based on HTTP / HTTPS and JSON. Developed by the DMTF (Distributed Management Task Force), it's specifically designed for managing hardware such as servers and storage devices. Its core value lies in enabling unified management of heterogeneous hardware from multiple vendors (such as BMCs and HMCs) through standardized resource models and communication interfaces. For example, a BMC can initiate standardized hardware status query requests to an HMC using the Redfish protocol. The HMC then returns operational data on components like the GPU and NVSwitch in JSON format. This eliminates the need to develop customized drivers for different vendors' devices, significantly reducing the compatibility costs of cross-platform management.

[0051] Out-of-band communication refers to an independent communication method that does not rely on the host operating system (OS) and the business network. In this application, the BMC and the HMC implement out-of-band communication through the USB virtual LAN (the channel constructed by S102): Even if the "business network" that the GPU and CPU of the MGX system rely on when running the business is interrupted, or the host OS crashes due to a fault, the BMC can still maintain communication with the HMC through the USB physical connection to ensure that key operations such as hardware status collection and fault log pulling are not interrupted. The application of the Redfish protocol in this out-of-band channel further ensures the reliable transmission and parsing of fault data (such as EROT logs, FPGA register information) in an independent channel through a standardized HTTP / HTTPS request-response mechanism, providing a unified data interaction interface for multi-dimensional fault analysis. Specifically, in the virtual LAN channel, the BMC and the HMC use the Redfish protocol for out-of-band communication, including:

[0052] (1) The BMC sends a Redfish request based on the HTTPS protocol to the HMC based on the USB virtual LAN channel, and calls the corresponding resource path to access the operating status information of the GPU unit, PCIe interconnect link and NVSwitch module.

[0053] It should be noted that the resource path is the unique address for accessing hardware resources in the Redfish protocol, similar to a web URL. Each path corresponds to a specific hardware component, and the BMC accurately obtains data through the path.

[0054] Specifically, the BMC sends HTTPS-encrypted Redfish requests to the HMC over an established USB virtual LAN channel. These requests precisely target the GPU unit, PCIe interconnect, or NVSwitch module via standardized resource paths to obtain the target hardware's operating status information. HTTPS encryption ensures secure communication, while resource paths enable the BMC to directly locate and access the required data, avoiding redundant queries.

[0055] (2) After receiving the Redfish request, the HMC returns a JSON format response containing the device operation status, which includes the device unique identifier, operation status code, link connection topology and error count information.

[0056] After receiving the request, the HMC first verifies its legitimacy (e.g., identity authentication and permission checks), extracts the corresponding information from the locally cached hardware status data, and returns it to the BMC in a JSON format according to the Redfish protocol. The response includes key information such as the device's unique identifier, operational status code, link connection topology, and error counts (e.g., the number of correctable errors). A single response provides multi-dimensional status data, significantly improving information acquisition efficiency.

[0057] (3) The BMC dynamically adjusts the polling frequency of each resource path based on the response content according to the preset collection scheduling strategy, and automatically triggers a retry mechanism when a communication anomaly is detected.

[0058] Specifically, the BMC dynamically adjusts the polling frequency of each resource path based on the preset collection scheduling strategy and the response content returned by the HMC: when the hardware status is normal, a low frequency (such as once every 30 seconds) is used to reduce resource consumption; when an abnormality occurs (such as high GPU temperature), it switches to a high frequency (such as once every 1 second) to track changes in real time; at the same time, if a communication timeout or error is detected, the BMC automatically triggers a retry mechanism (such as a maximum of 3 retries, each with an exponentially increasing interval), ensuring that data can still be stably obtained during occasional network fluctuations, providing reliable information support for subsequent fault location.

[0059] In this step, the BMC can directly access the target hardware's status through the resource path. This enables more accurate fault monitoring and rapid fault location compared to traditional methods. Regardless of the HMC vendor, as long as it supports Redfish, the BMC uses the same resource path for access, eliminating the need to develop dedicated drivers for each vendor and reducing integration costs. Furthermore, dynamic polling frequency allows the BMC to focus more on problematic components during failures and relax monitoring during normal operation. This adaptive monitoring approach improves efficiency, and out-of-band channels and retry mechanisms ensure data is not lost during failures.

[0060] S103: The BMC obtains inventory information of each component in the MGX system from the HMC based on the Redfish protocol, and classifies the components according to their types.

[0061] It should be noted that parts list information refers to the structured data list of MGX system hardware components obtained by the BMC from the HMC via the Redfish protocol. It contains basic information such as each component's unique identifier (such as UUID), hardware model, physical slot, function type, health status, and firmware version. Classification by component type means grouping the list information by hardware functional attributes, for example, grouping GPU units responsible for graphics computing into one category, PCIe interconnect links responsible for high-speed data transmission into another category, and NVSwitch modules responsible for multi-GPU collaborative scheduling into another category. This classification method provides the data foundation for the subsequent HMC to implement differentiated status pre-collection strategies for different types of components (such as GPUs focusing on temperature and error counts, PCIe links focusing on speed and topology). It also allows the BMC to quickly locate the hardware category to which the fault belongs based on the component type. Specifically, the BMC sends a request to the HMC via the Redfish protocol, accessing a dedicated resource path corresponding to the HMC's pre-stored system hardware inventory. Upon receiving the request, the HMC returns structured data in JSON format that complies with the Redfish specification. Each hardware component contains standard Redfish attribute fields (such as Type, Id, and Name). The BMC then parses the returned inventory information, extracts the Type field for each component, and groups components of the same type based on the field value, generating a classification result. For example, the GPU class includes all GPU units (GPU-0, GPU-1, etc.); the PCIe Link class includes all PCIe interconnect links (PCIeLink-1, PCIeLink-2, etc.); and the NVSwitch class includes all NVSwitch modules (NVSwitch-0, NVSwitch-1, etc.). This classification approach enables the BMC to quickly locate the hardware category to which a fault belongs based on component type, avoiding the efficiency loss caused by indiscriminate system-wide scanning.

[0062] It's important to note that after classification, the BMC can develop differentiated strategies for different component types, ensuring more accurate and efficient status collection. For example, for GPUs, the focus is on monitoring temperature and memory error counts; for PCIe links, the focus is on monitoring speed and link width. Different MGX systems may have different hardware configurations (e.g., some may have eight GPUs, while others may have four). By automatically acquiring and classifying the inventory, the BMC eliminates the need for hard-coded hardware information and can adapt to different system configurations. For example, when a new NVSwitch module is added, the BMC automatically classifies it as an NVSwitch and monitors it according to that class's strategy, eliminating the need for manual program modification. Furthermore, when a fault occurs, the BMC can quickly identify the fault type based on the classification. For example, when an abnormal Health attribute value is detected, the BMC first determines the hardware category of the fault, determining whether it is a GPU or PCIe link, using the Type field. It then invokes the corresponding diagnostic logic (such as checking the GPU's EROT log or PCIe register information), avoiding a full system scan and saving time.

[0063] S104 : The HMC pre-collects status information of each category according to the classification result and stores the information in the MGX system.

[0064] It should be noted that the HMC is classified according to GPU units, PCIe interconnect links, NVSwitch modules, etc., and actively collects the real-time status of each type of component (such as temperature, error count, link rate, etc.), and stores it in a unified format to facilitate quick retrieval when the BMC needs it later.

[0065] Specifically, the HMC pre-collects status information of each category according to the classification result and stores it in the MGX system, including:

[0066] (1) The HMC collects the operating status data of various components through the underlying driver interface between the GPU unit, PCIe interconnect chip and NVSwitch module.

[0067] It should be noted that the underlying driver interface is a dedicated communication interface between the HMC and hardware components (such as GPUs and PCIe chips). It is supported by drivers provided by hardware manufacturers and can directly read the underlying hardware status (such as GPU register values ​​and PCIe link signal strength). This is more real-time and low-level than data obtained through the operating system.

[0068] Specifically, the HMC proactively reads the real-time hardware status of each component categorized by the BMC through the corresponding underlying driver interface. For GPU units, the NVIDIA-provided GPU driver interface is used to collect information such as temperature, video memory usage, and ECC error counts. For PCIe interconnects, the PCIe driver interface is used to collect information such as link rate, link width, and error packet counts. For NVSwitch modules, the NVSwitch driver interface is used to collect information such as port connection status, bandwidth utilization, and topology changes.

[0069] By communicating directly with the hardware registers or control chips through the underlying driver interface and bypassing the operating system, the most original hardware status can be obtained. This way, the data is highly real-time and accurate, and can capture instantaneous hardware status changes.

[0070] (2) The HMC locally caches the acquired state data and organizes and stores the state data in a structured manner based on the resource path format defined in the Redfish protocol.

[0071] It should be noted that the local cache here refers to the HMC's built-in storage area (such as memory or local hard disk), which is used to temporarily store collected status data and has much faster access speed than remote storage. The resource path format defined in the Redfish protocol is a standardized data storage path. Each path corresponds to a specific status data (such as the health status of GPU-0), which facilitates direct query by the BMC through the path.

[0072] Specifically, a key-value pair (Key-Value) storage method can be used. The Key is the Redfish resource path, and the Value is the corresponding status data. This structure allows the BMC to directly obtain the Value (data) by simply providing the Key (path) when making subsequent queries, which is very efficient.

[0073] When a fault occurs, the BMC needs to obtain the hardware status immediately. The HMC collects and stores data in advance, which can quickly locate the cause of the fault when it occurs. The HMC collects and stores data centrally, so the BMC does not need to communicate with each hardware separately, reducing the waste of resources caused by repeated collection.

[0074] S105: The BMC periodically reads the status information based on the Redfish protocol, and identifies faults in the MGX system according to the status information.

[0075] It should be noted that periodic reading allows the BMC to detect hardware anomalies in a timely manner and issue early warnings before failures occur.

[0076] Specifically, the BMC periodically reads the status information based on the Redfish protocol and identifies a fault in the MGX system according to the status information, including:

[0077] (1) The BMC periodically initiates a Redfish status read request to the HMC according to a preset polling cycle and resource path table to obtain the latest status data of each GPU unit, PCIe interconnect link and NVSwitch module in the MGX system.

[0078] It's important to note that the polling cycle is set by the system based on the importance of the hardware (for example, for a core GPU, the polling cycle is set to 5 seconds; for an NVSwitch, it's set to 30 seconds), balancing real-time performance with resource consumption. The resource path table is simply a standardized list of addresses. The BMC doesn't need to remember the physical location of the hardware; it can directly query by path. This ensures that the BMC continuously and stably obtains the latest status data, avoiding delayed fault detection due to missed detections.

[0079] (2) BMC performs structured analysis on the acquired status data and extracts the status data fields.

[0080] It should be noted that the status data returned by the HMC is in JSON format. The BMC uses parsing tools (such as a JSON parsing library) to extract key fields. For example, it can extract temperature from GPU data, rate from PCIe link data, and bandwidth utilization from NVSwitch data. This allows for rapid screening of the core data required for fault diagnosis while eliminating redundant information (such as hardware model and manufacturer information).

[0081] (3) Compare the extracted status data field with the locally preset operating threshold range to determine whether there is an abnormal state.

[0082] It should be noted that the operating threshold range is the normal operating parameter range of the hardware preset locally by the BMC (preset by the hardware manufacturer or operation and maintenance personnel based on device characteristics, such as the normal GPU temperature range of 0-85°C and the normal PCIe link rate of 16GT / s). Exceeding the range is considered an abnormality.

[0083] This step, from periodic data reading to extracting key information and then comparing thresholds to identify anomalies, requires no human intervention. This shifts fault location from passively waiting for alerts to proactive real-time monitoring, improving fault identification efficiency. Furthermore, standardized resource paths ensure a consistent data source, structured parsing ensures accurate extracted fields, and threshold comparison ensures objective judgment logic, avoiding misjudgments caused by differing analysis standards among different individuals and improving the accuracy and consistency of fault diagnosis.

[0084] Furthermore, identifying faults in the MGX system based on the status information includes: the BMC parsing the status information returned by the HMC and determining whether there is an abnormal state; when an abnormal state is detected, the BMC automatically triggers a fault log download operation to obtain fault diagnosis data including EROT fault information, FPGA register information, and EROT log information from the HMC. That is, when the BMC determines that an abnormal state exists based on the comparison results, it automatically triggers a log collection process, which specifically includes:

[0085] When the BMC determines that any component is in an abnormal state, based on the device identification information corresponding to the abnormal state, the BMC calls the log collection interface through the Redfish protocol and initiates a log download request to the HMC. According to the request, the HMC extracts the EROT fault information, FPGA register information, and EROT log information corresponding to the target component from the local cache. The extracted fault diagnosis data is compressed and packaged, and returned to the BMC through the USB virtual LAN channel. The BMC classifies and stores the received fault diagnosis data according to the device number and timestamp.

[0086] It should be noted that the log collection interface is a Redfish protocol standard interface provided by the HMC, which is used to trigger log collection and download; EROT fault information is a record of hardware emergency runtime, containing key parameters at the moment of failure (such as temperature and voltage when the GPU crashed); FPGA register information is the internal register status of the field programmable gate array, reflecting the hardware's underlying configuration and operating status (such as the signal strength and clock frequency of the PCIe link), and is used to locate hardware-level faults; EROT log information is a sequence of events recorded during hardware operation (such as the GPU temperature change curve and error count growth trend), which is used to analyze the fault development process.

[0087] Specifically, the BMC sends a log download request to the HMC via the Redfish protocol's standard log collection interface based on the device identification information of the abnormal component (such as GPU-0). This request explicitly specifies the required log type (such as EROT fault information or FPGA register values) and the time range (usually a few minutes before and after the abnormality), ensuring that the HMC can accurately extract relevant data. For example, if the BMC detects that the temperature of GPU-0 exceeds 85°C, it immediately requests all operating logs for that GPU within the past five minutes.

[0088] After receiving the request, the HMC quickly extracts three types of critical information from its local cache: EROT fault information, which records critical hardware parameters during an emergency (such as temperature and voltage at the moment of failure); FPGA register information, which represents underlying configuration data reflecting the hardware's real-time operating status (such as PCIe link signal strength and training status); and EROT log information, which records hardware events in a time series (such as temperature trends and error count growth). This data has been pre-collected and cached by the HMC in step S105, enabling a rapid response to the BMC's request and preventing data loss due to further hardware damage.

[0089] Furthermore, the HMC compresses and packages the extracted data (e.g., in ZIP or TGZ format) and returns it to the BMC via a USB virtual LAN channel. This compression process significantly reduces the data size (e.g., compressing 100MB of raw data to 20MB) and shortens transmission time. The out-of-band channel ensures reliable access to fault logs even in the event of a business network outage.

[0090] Finally, after receiving the compressed package, the BMC categorizes and stores the data by device number and timestamp. For example, a directory structure like / logs / GPU-0 / 20250708T103000Z / is created to store decompressed register dump files, temperature curves, and other data by type. This structured storage method not only facilitates operations and maintenance personnel to quickly retrieve fault data for a specific device at a specific time, but also lays the foundation for subsequent automated fault analysis.

[0091] In addition, it should be noted that when an abnormal state is detected, the BMC automatically triggers the fault log download operation. After obtaining fault diagnosis data including EROT fault information, FPGA register information, and EROT log information from the HMC, the following are included:

[0092] The BMC simultaneously parses the EROT fault information and the EROT log information; wherein, the BMC extracts the Message attribute and the Resolution attribute from the EROT fault information, determines the faulty GPU unit and the fault category based on the GPU number, fault location, and fault type contained in the Message attribute, and generates maintenance guidance information in combination with the operation suggestions provided by the Resolution attribute; the BMC extracts the contextual events and time series information of the fault occurrence time point from the EROT log information, identifies the propagation process of the abnormal state and the associated link components; the BMC parses the FPGA register information, obtains the underlying hardware status parameters associated with the faulty GPU, and compares them with the abnormal events in the EROT log to determine whether there are register abnormality characteristics; the BMC integrates the fault diagnosis data based on the acquired GPU fault location information, the abnormal event sequence extracted from the log, the FPGA register status characteristics, and the generated maintenance guidance information to generate a structured standard fault log file.

[0093] It should be noted that the Message attribute is the core field in the EROT fault information, containing a concise description of the fault (such as GPU-0 temperature is too high). The Resolution attribute is the repair suggestion provided in the EROT fault information (such as checking the GPU fan). Context events are related events before and after the fault in the EROT log (such as the temperature rising from 70°C to 90°C within 5 minutes), which are used to restore the fault development process. Time series information is a sequence of events recorded in chronological order, reflecting the evolution path of the fault (such as event A → event B → fault occurrence). Register anomaly characteristics are parameters in the FPGA register that deviate from the normal range (such as abnormal PCIe link training register values), which are direct evidence of hardware-level faults.

[0094] It should be noted that, first, the BMC will parse the EROT fault information and EROT log information simultaneously. In the EROT fault information, the BMC focuses on extracting the Message attribute and the Resolution attribute. The Message attribute directly contains key information such as the GPU number, fault location, and fault type. For example, if the temperature of GPU-0 is too high, the BMC can quickly identify the specific GPU unit and fault category that has failed by parsing this attribute. The Resolution attribute is a repair suggestion field predefined by the hardware manufacturer. By parsing this attribute, the BMC can obtain structured maintenance guidance information (such as checking the GPU fan or cleaning the radiator) and directly output it to the operation and maintenance personnel.

[0095] Next, the BMC extracts contextual events and time series information from the EROT log information at the time of the failure. This information records the evolution from normal to abnormal. For example, the log may show that before the GPU temperature is too high, the bandwidth utilization of the NVSwitch port is continuously 100%, and then the GPU temperature begins to rise sharply. By analyzing the sequence and correlation of these events, the BMC can identify the propagation path of the abnormal state, such as NVSwitch congestion that blocks GPU data transmission, which in turn causes excessive temperature. This step helps locate the source of the fault.

[0096] The BMC then parses the FPGA register information to obtain the underlying hardware status parameters associated with the faulty GPU. FPGA registers record the real-time operating status of the hardware. The BMC compares these register parameters with abnormal events in the EROT log to identify any register anomalies. For example, if the log indicates that the GPU temperature is too high, but the register indicates that the GPU's power management module has triggered frequency reduction protection, this completes the chain from overtemperature to protection triggering. This cross-validation of underlying parameters and higher-level events allows for more accurate fault location and avoids misdiagnosis.

[0097] Finally, the BMC integrates all the information obtained in the previous three steps to generate a structured, standard fault log file. This file contains fault location information (such as which GPU and type of fault), the abnormal event sequence (how the fault developed), FPGA register state characteristics (underlying hardware evidence), and generated repair instructions. This log uses a standardized JSON format, making it easy for operations personnel to quickly read and understand it, and also facilitates subsequent integration into AI systems for automated analysis.

[0098] Furthermore, the BMC parses the FPGA register information, obtains the underlying hardware status parameters associated with the faulty GPU, and compares them with the abnormal events in the EROT log to determine whether there are register abnormal characteristics, including: the BMC extracts hardware status data from the FPGA register information; based on the hardware status data, the BMC combines the abnormal event timestamp and event type recorded in the EROT log to perform a time and event sequence comparison; the BMC analyzes the abnormal threshold triggering situation of the register parameters to determine whether there are register status abnormal characteristics; the BMC determines the severity and impact range of the fault based on the correlation between the register abnormal characteristics and the abnormal events.

[0099] Specifically, the BMC extracts hardware status data from FPGA registers. FPGA registers are low-level hardware monitors that record key GPU-related parameters, such as PCIe link signal strength, the operating status of the GPU power management module, and memory controller operating parameters. This data is stored in binary format, and the BMC converts it into readable information using a pre-set mapping table. The BMC then compares this hardware status data with the abnormal events in the EROT log by time and event sequence. The EROT log records key events before and after the failure (such as NVSwitch congestion at 10:30 and GPU overtemperature at 10:32), each with a precise timestamp. Through time alignment, the BMC can clearly see the correlation between register parameter changes and logged events. For example, one minute after the NVSwitch congestion occurs, the PCIe link signal strength register value drops from -6dBm to -9dBm, and the logged PCIe error count begins to increase.

[0100] The BMC then analyzes register parameter abnormality thresholds. Each register parameter has a preset normal range (for example, the normal PCIe link rate should be 16GT / s, and the GPU temperature safety threshold is 90°C). The BMC checks whether the parameter exceeds the threshold. For example, if the PCIe link rate register value drops to 8GT / s (below the standard value) or the GPU temperature register shows 92°C (above the safety threshold), it will be marked as a register abnormality.

[0101] Finally, the BMC determines the severity and impact of the fault based on the correlation between the register anomaly characteristics and the abnormal event. If the register anomaly is completely synchronized with a critical fault in the log (such as a GPU computing interruption) and the parameter deviation is large (such as a temperature exceeding a threshold by 10°C), it is considered a critical fault, potentially affecting the entire GPU unit. If the register parameter fluctuation is only minor (such as a link rate that briefly drops to 14GT / s and then recovers) and the correlation with the warning event in the log is weak, it is considered a minor anomaly and requires only continued monitoring. This classification helps operations personnel prioritize urgent issues and optimize resource allocation.

[0102] The method provided in this embodiment achieves multi-dimensional technological breakthroughs by building an independent and reliable out-of-band communication and automated diagnostic process. First, the out-of-band communication mechanism, based on a USB virtual LAN channel and the Redfish protocol, completely eliminates reliance on the host operating system and service network. Even in the event of a system crash or driver failure, the BMC can still reliably access hardware status data pre-stored by the HMC, ensuring that fault information is not lost and resolving the data fragmentation issue inherent in traditional in-band diagnostics. Second, by pre-collecting and structured storage of status data (such as GPU temperature and PCIe link speed) by component type on the HMC, combined with the BMC's dynamically adjusted polling strategy, real-time monitoring and precise screening of hardware status are achieved, eliminating the blindness of manual troubleshooting and reducing fault detection time from hours to seconds. Furthermore, a multi-source data cross-validation mechanism (EROT fault information extraction to provide repair guidance, EROT log restoration of the fault propagation path, and FPGA registers providing underlying hardware evidence) significantly improves fault location accuracy. For example, by correlating register anomalies with log events, it is possible to quickly distinguish between heat dissipation issues and power module failures, reducing misdiagnosis. Finally, standardized fault log integration and classified storage not only provides clear maintenance guidance for operation and maintenance personnel, but also lays a data foundation for subsequent AI diagnostic model training, significantly reducing the complexity of cross-platform fault location and improving the overall reliability and operation and maintenance efficiency of the MGX system.

[0103] Example 2

[0104] Corresponding to the aforementioned embodiment of a method for quickly locating a fault in an MGX system, the present application also provides an embodiment of an apparatus for quickly locating a fault in an MGX system.

[0105] Figure 2 This is a structural diagram of the second embodiment of the fault rapid location device in the MGX system provided by this application. Figure 2 , the device provided in this embodiment includes:

[0106] A mainboard, on which a management controller BMC is provided;

[0107] A hardware management controller HMC, communicating with the BMC;

[0108] A universal serial bus (USB) interface, used to establish a USB virtual local area network channel based on a fixed IP segment configuration between the BMC and the device where the HMC is located;

[0109] a storage module, configured to store component status information and fault diagnosis information collected by the HMC;

[0110] The processor module is configured to execute any one of the methods for quickly locating faults in the MGX system.

[0111] The device of this embodiment can be used to perform Figure 1 The steps, specific implementation principles and implementation processes of the method embodiment shown are similar and will not be repeated here.

[0112] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for quickly locating a fault in an MGX system, characterized in that: The MGX system includes a management controller BMC and a hardware management controller HMC provided on a mainboard, and the method includes: Determine an idle Universal Serial Bus (USB) interface on the motherboard, and select a target USB interface and a fixed IP network segment configuration corresponding to the target USB interface; The BMC establishes a physical connection with the device where the HMC is located through the target USB interface, and constructs a USB virtual local area network channel based on a fixed IP segment configuration. In the USB virtual local area network channel, the BMC and the HMC use the Redfish protocol for out-of-band communication; The BMC obtains the inventory information of each component in the MGX system from the HMC based on the Redfish protocol and classifies the components according to the component type; The HMC pre-collects status information of each category according to the classification result and stores it in the MGX system; The BMC periodically reads the status information based on the Redfish protocol and identifies faults in the MGX system according to the status information.

2. The method according to claim 1, characterized in that The method of constructing a USB virtual LAN channel based on a fixed IP segment configuration includes: In the MGX system, the management controller BMC is configured to have a USB Ethernet function module and is set to master device mode; Connecting the BMC to the device where the hardware management controller HMC is located through the target USB interface, and configuring a static IP address for the USB virtual local area network channel; wherein, after the configuration is completed, the BMC and the HMC are in the same fixed network segment; Starting a virtual Ethernet adapter on the BMC to identify and initialize the USB virtual LAN channel; After the initialization is completed, a network communication link is established between the BMC and the HMC based on the static IP address.

3. The method according to claim 1, characterized in that In the USB virtual local area network channel, the BMC and the HMC use the Redfish protocol to perform out-of-band communication, including: The BMC sends a Redfish request based on the HTTPS protocol to the HMC based on the USB virtual LAN channel, calling the corresponding resource path to access the operating status information of the GPU unit, PCIe interconnect link and NVSwitch module; After receiving the Redfish request, the HMC returns a JSON format response containing the device operation status, including the device unique identifier, operation status code, link connection topology and error count information; The BMC dynamically adjusts the polling frequency of each resource path based on the response content according to a preset collection scheduling strategy, and automatically triggers a retry mechanism when a communication anomaly is detected.

4. The method according to claim 1, wherein The HMC pre-collects status information of each category according to the classification result and stores it in the MGX system, including: The HMC collects the operating status data of various components through the underlying driver interface between the GPU unit, PCIe interconnect chip and NVSwitch module; The HMC locally caches the acquired state data and, based on the resource path format defined in the Redfish protocol, structuredly organizes and stores the state data.

5. The method according to claim 1, wherein The BMC periodically reads the status information based on the Redfish protocol and identifies a fault in the MGX system according to the status information, including: The BMC periodically initiates a Redfish status read request to the HMC based on a preset polling cycle and resource path table to obtain the latest status data of each GPU unit, PCIe interconnect link, and NVSwitch module in the MGX system; The BMC performs structured parsing on the acquired status data and extracts the status data fields; The extracted status data fields are compared with the locally preset operating threshold range to determine whether there is an abnormal state.

6. The method according to claim 1, characterized in that The identifying a fault in the MGX system according to the status information includes: The BMC analyzes the status information returned by the HMC and determines whether there is an abnormal state; When an abnormal state is detected, the BMC automatically triggers the fault log download operation and obtains fault diagnosis data including EROT fault information, FPGA register information, and EROT log information from the HMC.

7. The method according to claim 6, characterized in that When an abnormal state is detected, the BMC automatically triggers a fault log download operation. After obtaining fault diagnosis data including EROT fault information, FPGA register information, and EROT log information from the HMC, the following steps are performed: The BMC simultaneously parses the EROT fault information and the EROT log information. The BMC extracts the Message attribute and the Resolution attribute from the EROT fault information. Based on the GPU number, fault location, and fault type contained in the Message attribute, the BMC determines the faulty GPU unit and fault type. Furthermore, the BMC generates maintenance guidance information based on the action suggestions provided by the Resolution attribute. The BMC extracts contextual events and time series information at the time of the fault occurrence from the EROT log information, and identifies the propagation process of the abnormal state and the associated link components; The BMC parses the FPGA register information, obtains underlying hardware status parameters associated with the faulty GPU, and compares the information with abnormal events in the EROT log to determine whether there are register abnormal characteristics. The BMC integrates the fault diagnosis data based on the acquired GPU fault location information, the abnormal event sequence extracted from the log, the FPGA register state characteristics, and the generated maintenance guidance information to generate a structured standard fault log file.

8. The method according to claim 7, characterized in that The BMC parses the FPGA register information, obtains the underlying hardware status parameters associated with the faulty GPU, and compares them with the abnormal events in the EROT log to determine whether there are register abnormal characteristics, including: The BMC extracts hardware status data from the FPGA register information; Based on the hardware status data, the BMC performs a time and event sequence comparison in combination with the abnormal event timestamp and event type recorded in the EROT log; The BMC analyzes the abnormal threshold triggering of the register parameter to determine whether there is an abnormal register status feature; The BMC determines the severity and impact range of the fault based on the correlation between the abnormal characteristics of the register and the abnormal event.

9. The method according to claim 6, characterized in that When an abnormal state is detected, the BMC automatically triggers a fault log download operation to obtain fault diagnosis data including EROT fault information, FPGA register information, and EROT log information from the HMC, including: When the BMC determines that any component is in an abnormal state, based on the device identification information corresponding to the abnormal state, the BMC calls the log collection interface through the Redfish protocol and initiates a log download request to the HMC; The HMC extracts EROT fault information, FPGA register information, and EROT log information corresponding to the target component from a local cache according to the request; Compressing and packaging the extracted fault diagnosis data, and returning it to the BMC through the USB virtual local area network channel; The BMC classifies and stores the received fault diagnosis data according to the device number and time stamp.

10. A device for quickly locating faults in an MGX system, characterized in that: The device is used to perform the method according to any one of claims 1 to 9, and the device includes: A mainboard, on which a management controller BMC is provided; A hardware management controller HMC, communicating with the BMC; A universal serial bus (USB) interface, used to establish a USB virtual local area network channel based on a fixed IP segment configuration between the BMC and the device where the HMC is located; a storage module, configured to store component status information and fault diagnosis information collected by the HMC; A processor module configured to execute the method according to any one of claims 1 to 9.

Images