A method and device for detecting and recovering faults in an AI server operating system

Abstract

The present invention relates to the field of operating systems, and in particular to a method and device for detecting and recovering faults in an AI server operating system. The method mainly includes: using a hard disk file system as a boot method when the operating system is started, exchanging a first heartbeat packet between a monitoring management module and a hard disk monitoring module in the hard disk file system, and obtaining the startup status of the operating system based on the first heartbeat packet; establishing a second heartbeat packet interaction between the monitoring management module and the BIOS, and the monitoring management module feeding back the startup status to the BIOS through the second heartbeat packet. When the startup fails, the BIOS switches the boot method to the memory file system, and starts the operating system through the system root file pre-stored in the memory file system. The present invention can effectively reduce the impact of two fatal server operating system failures: the failure of the operating system kernel to start normally and the failure of the root file system and the failure of loading normally, thereby avoiding losses caused by server startup failures.

Description

Technical Field

[0001] The present invention relates to the field of operating systems, and in particular to a method and device for detecting and recovering faults in an AI server operating system. Background Art

[0002] Currently, artificial intelligence (AI) servers primarily run on Linux. During the boot process, Linux can experience startup failures such as crashes or abnormal restarts. This can occur when server hardware fails, the operating system's hardware device driver is incompatible with the hardware, or core Linux components are damaged.

[0003] The two core components of the Linux operating system are the Linux kernel and the root file system. The root file system is typically stored on the hard drive. This can be damaged by sudden power outages or unplugging the hard drive. Furthermore, due to the nature of the hard drive storage medium, root file system corruption can also occur due to factors such as bad blocks. If the root file system is damaged, the operating system may experience startup failures.

[0004] In current server use, after the operating system boots normally, the server's Base-Board Management Controller (BMC) can be used to monitor the operating status of the server and operating system. However, the BMC cannot detect abnormalities in the operating system's kernel startup and root file system that cause the server to fail to boot normally. The BMC's event logs are also unable to record any abnormalities. This means that critical abnormalities are not detected, and the fault cannot be automatically recovered, significantly impacting the actual use of the server.

[0005] In view of this, how to overcome the defects of the existing technology and solve the problem that the existing technology cannot detect and recover the abnormal kernel startup and root file system that cannot be started normally is a problem to be solved in this technical field. Summary of the Invention

[0006] In view of the above defects or improvement needs of the prior art, the present invention solves the problem that the prior art cannot detect and recover the abnormal kernel startup and the root file system abnormality that cause the abnormal startup.

[0007] The embodiment of the present invention adopts the following technical solutions:

[0008] In the first aspect, the present invention provides a method for fault detection and recovery of an AI server operating system, specifically: using the hard disk file system as the boot mode when the operating system is started, exchanging a first heartbeat packet between the monitoring management module and the hard disk monitoring module in the hard disk file system, and obtaining the startup status of the operating system based on the first heartbeat packet; establishing a second heartbeat packet interaction between the monitoring management module and the BIOS, and the monitoring management module feeds back the startup status to the BIOS through the second heartbeat packet. When the startup fails, the BIOS switches the boot mode to the memory file system, and starts the operating system through the system root file pre-stored in the memory file system.

[0009] Preferably, the monitoring management module and the hard disk monitoring module in the hard disk file system exchange a first heartbeat packet, and the startup status of the operating system is obtained according to the first heartbeat packet, specifically including: the monitoring management module and the hard disk monitoring module periodically exchange the first heartbeat packet, and when the first heartbeat packet interaction is normal, the hard disk monitoring module starts normally and then feeds back a first success identification packet to the monitoring management module; when the monitoring management module receives a specified number of first success identification packets within a specified waiting time, it indicates that the startup status of the operating system is successful startup; when the monitoring management module does not receive the first success identification packet within the specified waiting time, it indicates that the startup status of the operating system is failed startup.

[0010] Preferably, the monitoring and management module feeds back the startup status to the BIOS through the second heartbeat packet, specifically including: periodic interaction of the second heartbeat packet between the monitoring and management module and the BIOS; when the interaction of the second heartbeat packet is normal, the BIOS sends a second request packet to the monitoring and management module after normal startup, and requests the startup status of the operating system from the monitoring and management module through the second request packet; when the startup status is a successful startup, the monitoring and management module returns a second success identification packet to the BIOS; when the startup status is a failed startup, the monitoring and management module returns a second failure identification packet to the BIOS.

[0011] Preferably, the monitoring management module feeds back the startup status to the BIOS via the second heartbeat packet, and also includes: when the operating system fails to start, the monitoring management module controls the server to repower on, and triggers the BIOS to resend the second request packet to the monitoring management module by repowering on.

[0012] Preferably, the BIOS switches the boot mode to the memory file system, specifically including: the BIOS modifies the startup boot item to the memory file system, and boots the operating system through the system root file pre-stored in the memory file system.

[0013] Preferably, after the operating system is started through the system root file pre-stored in the memory file system, it also includes: the monitoring management module and the memory monitoring module periodically interact with the third heartbeat packet, and when the third heartbeat packet interaction is normal, the memory monitoring module starts normally and then feeds back the third success identification packet to the monitoring management module; when the monitoring management module receives the third success identification packet sent by the memory monitoring module, it indicates that the startup is successful; when the monitoring management module does not receive the third success identification packet sent by the memory monitoring module within the specified time period, it indicates that the startup failed.

[0014] Preferably, when the startup status is successful startup, the operating system is started through the system root file pre-stored in the memory file system, and the BIOS also includes: backing up the startup file and the system root file in the hard disk file system, and copying the system root file pre-stored in the memory file system to the hard disk file system. At the next startup, the system root file that was successfully started last time is used to start the system.

[0015] Preferably, when the startup status is startup failure, the operating system is started through the root file pre-stored in the memory file system, and also includes: each time the monitoring management module receives the third failure identification packet, it records the number of startup failures; the memory monitoring module sends a fourth request packet to the monitoring management module, and the monitoring management module feeds back the number of startup failures to the memory monitoring module through the fourth reply packet; when the number of startup failures is greater than the specified number, no restart is attempted, and the IP configuration of the server is restored for remote fault handling.

[0016] In a second aspect, the present invention provides a device for AI server operating system fault detection and recovery, specifically comprising: at least one processor and a memory, the at least one processor and the memory being connected via a data bus, the memory storing instructions that can be executed by the at least one processor, and after being executed by the processor, the instructions are used to complete the AI ​​server operating system fault detection and recovery method in the first aspect.

[0017] In a third aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, which are executed by one or more processors to complete the method provided by the method described in the first aspect.

[0018] In a fourth aspect, a chip is provided, comprising: a processor and an interface, for calling and running a computer program stored in a memory, and executing the method provided in the first aspect.

[0019] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer or a processor, causes the computer or the processor to execute the method provided in the first aspect.

[0020] Compared with the existing technology, the present invention has the following advantages: heartbeat packet interaction between the monitoring and management module and the hard disk monitoring module implements heartbeat and watchdog functions during server startup, and restores the operating system through the system root file pre-stored in the memory file system. Furthermore, in a preferred embodiment of the present invention, the monitoring and management module and the memory monitoring module further monitor the startup process of the memory file system. This approach effectively reduces the impact of two critical server operating system failures: the failure of the operating system kernel to boot properly and the failure of the root file system to load properly, thereby avoiding losses caused by server startup failures. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0022] Figure 1 A flowchart of a method for detecting and recovering faults in an AI server operating system provided by an embodiment of the present invention;

[0023] Figure 2 A flowchart of another method for detecting and recovering faults in an AI server operating system provided by an embodiment of the present invention;

[0024] Figure 3 A flowchart of another method for detecting and recovering faults in an AI server operating system provided by an embodiment of the present invention;

[0025] Figure 4 A flowchart of another method for detecting and recovering faults in an AI server operating system provided by an embodiment of the present invention;

[0026] Figure 5 A schematic diagram of the message interaction process between the functional modules in the method provided in an embodiment of the present invention;

[0027] Figure 6 A schematic diagram of a specific implementation process of a BIOS module in a specific scenario of the method provided by an embodiment of the present invention;

[0028] Figure 7 A schematic diagram of the data processing process of the BIOS module in a specific scenario of the method provided by an embodiment of the present invention;

[0029] Figure 8 A schematic diagram of the data processing process of the hard disk monitoring module in a specific scenario of the method provided by an embodiment of the present invention;

[0030] Figure 9 A schematic diagram of the data processing process of the memory monitoring module in a specific scenario of the method provided by an embodiment of the present invention;

[0031] Figure 10 A schematic diagram of the data processing process of the system repair module in a specific scenario of the method provided by an embodiment of the present invention;

[0032] Figure 11 A schematic diagram of the structure of an AI server operating system fault detection and recovery device provided by an embodiment of the present invention;

[0033] The accompanying drawings are numerals as follows:

[0034] 11: Processor; 12: Memory. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0036] The present invention is an architecture of a specific functional system. Therefore, the specific embodiments mainly illustrate the functional logical relationship between the various structural modules, and do not limit the specific software and hardware implementation methods.

[0037] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0038] The operating system (OS) currently used in server applications is primarily the Linux operating system. The two core components of the Linux operating system are the Linux kernel and the root file system. Hardware incompatibility or hardware driver anomalies can cause the Linux kernel to fail to boot. For example, the OS kernel may freeze at startup or restart abnormally, resulting in fatal server downtime or repeated restarts.

[0039] Currently, the root files of the Linux operating system are stored on the hard disk. Factors such as the hard disk media lifespan, damage to the hard disk's Logical Volume Manager (LVM) partition data, abnormal server restarts, and sudden power cycles can cause root file system corruption. As a result, the root file system cannot be mounted properly during the Linux kernel startup process, causing the operating system to fail to boot. The server will freeze or enter emergency mode. At this time, the network card will not have an IP address configured, and services cannot be executed, affecting the normal use of the server. This is a fatal server problem.

[0040] The monitoring and management module in the current server cannot monitor and record the situation where the above two operating systems cannot start normally. In addition, due to the lack of IP configuration, the fault cannot be handled remotely. Users can only manually analyze the fault and attempt recovery on site, which seriously affects the operation and maintenance efficiency of the server.

[0041] Example 1:

[0042] To solve the above problems, this embodiment provides an AI server operating system fault detection and recovery method to perceive and automatically recover such server anomalies, thereby improving server reliability and operation and maintenance efficiency.

[0043] like Figure 1 As shown, the specific steps of the method provided by the embodiment of the present invention are as follows:

[0044] Step 101: Using the hard disk file system as the boot mode when the operating system is started, the monitoring management module exchanges a first heartbeat packet with the hard disk monitoring module in the hard disk file system, and the startup status of the operating system is obtained according to the first heartbeat packet.

[0045] After the server is powered on, the Basic Input Output System (BIOS) first powers on and tests each hardware component. After all hardware tests pass, it enters the Boot Device Selection (BDS) stage. In the Boot Device Selection stage, the BIOS obtains the root file and kernel of the operating system from the boot device according to the pre-specified boot method, and uses the root file to start the operating system. In actual implementation, the boot device used as a boot method can be a hard disk, optical disk, USB device, memory, etc.; in normal use, the hard disk file system is usually used as the boot method at startup. In the existing technology, the monitoring and management module cannot monitor the above process, and therefore cannot perform recovery and fault handling in the event of a startup failure.

[0046] In the method provided in this embodiment, a hard disk monitoring module (hard disk OS agent) is added to the hard disk file system. If the hard disk file system is used as the boot method when the operating system is started (hereinafter referred to as the hard disk boot method), after the BIOS is started, the root file in the specified folder of the hard disk file system is used to start the operating system. After the hard disk monitoring module is normally started, it interacts with the monitoring and management module through the first heartbeat packet, and sends the first success identification packet to the monitoring and management module, indicating that the OS is successfully started from the hard disk. If the hard disk monitoring module is not normally started, the monitoring and management module cannot receive the first heartbeat packet sent by the hard disk monitoring module in the hard disk file system, indicating that the hard disk file system failed to load or the kernel startup failed. The monitoring and management module needs to perform system restart, fault processing, damaged file recovery and fault log recording as needed.

[0047] Step 102: A second heartbeat packet interaction is established between the monitoring management module and the BIOS. The monitoring management module feeds back the startup status to the BIOS through the second heartbeat packet. When the startup fails, the BIOS switches the boot mode to the memory file system and starts the operating system through the system root file pre-stored in the memory file system.

[0048] After the BIOS is started, a second heartbeat packet is established between the BIOS and the monitoring and management module to ensure normal communication, and then a second request packet is sent to the monitoring and management module to query the startup status.

[0049] If the operating system is successfully started, the monitoring management module replies to the BIOS with a second success identification packet. After the BIOS receives the second success identification packet, it knows that the operating system has been successfully started and can execute subsequent operations normally.

[0050] If the operating system fails to boot, the monitoring and management module sends a second failure identification packet to the BIOS. Upon receiving the second failure identification packet, the BIOS detects the operating system failed to boot and initiates a reboot or recovery. In this embodiment, when the operating system is rebooted, the possibly damaged root file in the hard disk file system is not used. Instead, the correct root file is pre-stored in the memory file system. The BIOS modifies the boot entry to the memory file system and boots the operating system using the system root file pre-stored in the memory file system.

[0051] If the system can be started normally using the root file in the memory file system, the BIOS will back up the startup file and system root file in the hard disk file system, and copy the system root file stored in the memory file system to the hard disk file system. At the next startup, the system root file that was successfully started last time will be used to start the system.

[0052] Furthermore, in order to prevent root files in the hard disk file system and the memory file system from being damaged simultaneously, the hard disk file system and the memory file system are not stored on the same hard disk, or are not stored on the same physical or logical sector of the same hard disk.

[0053] After steps 101 and 102 provided in this embodiment, the AI ​​server operating system fault detection and recovery can be completed.

[0054] In this embodiment, the hard disk monitoring module in the hard disk file system is used to monitor the operating system startup process in the hard disk boot mode, and is responsible for communicating with the monitoring management module and feeding back the startup result of the hard disk boot mode to the monitoring management module. Figure 2 As shown, the following method can be used to complete the interaction between the monitoring management module and the hard disk monitoring module in the hard disk file system.

[0055] Step 201: The monitoring management module and the hard disk monitoring module periodically exchange first heartbeat packets. When the first heartbeat packet interaction is normal, the hard disk monitoring module starts normally and then feeds back a first success identification packet to the monitoring management module.

[0056] The monitoring and management module establishes a heartbeat relationship by exchanging first heartbeat packets with the hard disk monitoring module within the hard disk operating system. In practice, the monitoring and management module receives first heartbeat packets from the hard disk monitoring module every one minute. Furthermore, after the operating system boots successfully via the hard disk boot method, the hard disk monitoring module within the hard disk file system also sends back a first success identification packet to the monitoring and management module, indicating that it is operating normally.

[0057] Step 202: When the monitoring management module receives a specified number of first success identification packets within a specified waiting time, it indicates that the startup status of the operating system is successful.

[0058] If the monitoring and management module can receive the first heartbeat packet on time, it indicates that the communication between the monitoring and management module and the hard disk monitoring module is normal; if the monitoring and management module also receives the first success identification packet on time, it indicates that the operating system started by the hard disk boot method runs normally during the time period and the operating system is started successfully.

[0059] Step 203: When the monitoring management module does not receive the first success identification packet within the specified waiting time, it indicates that the startup status of the operating system is startup failure.

[0060] If the monitoring and management module cannot receive the first heartbeat packet on time, it indicates that the operating system may not start normally under the hard disk boot mode, or there is a communication failure between the monitoring and management module and the hard disk monitoring module, the operating system kernel cannot start, or the root file in the hard disk file system cannot be loaded; if the monitoring and management module can receive the first heartbeat packet on time, but cannot receive the first success identification packet on time, it also indicates that the operating system has not started normally under the hard disk boot mode.

[0061] After steps 201 to 203 provided in this embodiment, the monitoring management module can obtain the startup status of the operating system in the hard disk booting mode, so as to realize the monitoring of the startup status of the operating system in the hard disk booting mode.

[0062] In actual implementation, after the monitoring and management module obtains the startup status of the hard disk boot mode, it can use the startup status flag to record the status of the hard disk boot mode. When the BIOS queries the monitoring and management module for the startup status, the monitoring and management module can feedback the corresponding startup status to the BIOS based on the value of the startup status flag. Figure 3 As shown, the interaction between the monitoring management module and the BIOS can be completed in the following ways.

[0063] Step 301: The monitoring management module and the BIOS periodically exchange second heartbeat packets. When the second heartbeat packet interaction is normal, the BIOS starts normally and sends a second request packet to the monitoring management module to request the startup status of the operating system.

[0064] After the BIOS is started, it sends a second heartbeat packet to the monitoring and management module to establish a heartbeat. If the heartbeat is established normally, it means that the server startup abnormality is not caused by hardware failure.

[0065] At the same time, the BIOS sends a second request packet to the monitoring and management module through the Intelligent Platform Management Interface (IPMI) to query whether the operating system is started normally in the hard disk boot mode. The monitoring and management module listens to the second request packet sent by the BIOS and returns an identification packet corresponding to the current boot status.

[0066] Step 302: When the startup status is successful, the monitoring management module returns a second success identification packet to the BIOS.

[0067] After receiving the second success identification packet fed back by the monitoring management module, the BIOS learns that the operating system is started normally in the hard disk booting mode.

[0068] Step 303: When the startup status is startup failure, the monitoring management module returns a second failure identification packet to the BIOS.

[0069] After receiving the second failure identification packet fed back by the monitoring management module, the BIOS learns that the operating system fails to start in the hard disk booting mode.

[0070] On the other hand, in actual implementation, to avoid repeated requests, the BIOS only sends a second request packet to the monitoring and management module once after the operating system boots to obtain the operating system's boot status. Therefore, if the operating system fails to boot, the BIOS will not send a second request packet to the monitoring and management module again, and will only exchange a second heartbeat packet to confirm that communication between the two is normal. At this time, to query the boot status after the next operating system boot, the monitoring and management module controls the server to power on again. This power-on triggers the BIOS to resend the second request packet to the monitoring and management module.

[0071] After steps 301 to 303 provided in this embodiment, the interaction between the monitoring management module and the BIOS is completed, so that the BIOS can learn the startup status of the operating system so that the BIOS can perform corresponding subsequent operations according to the startup status.

[0072] Furthermore, to prevent the operating system from automatically shutting down or repeatedly restarting within a short period of time after startup, the monitoring and management module can repeatedly send back identification packets corresponding to the startup status to the BIOS over a period of time. The BIOS then determines the startup status based on the identification packets received over a period of time. In a specific scenario, if a specified number of second success identification packets are received within a specified time period after the BIOS begins booting the operating system, it indicates that the operating system has successfully started. If the specified number of second success identification packets are not received within the specified time period, or if a second failure identification packet is received within the specified time period, it indicates that the operating system has failed to start.

[0073] If the operating system boots successfully in hard disk boot mode, the BIOS does not need to perform any additional processing. If the operating system boots unsuccessfully in hard disk boot mode, the BIOS switches the boot mode to the memory file system and attempts to restart or recover the operating system.

[0074] If the memory file system is used as the boot method when the operating system is started (hereinafter referred to as the memory boot method) to start the operating system, the startup failure may occur again. Therefore, in the method provided in this embodiment, a memory monitoring module (memory OS agent) is also set in the memory file system, and the monitoring management module obtains the startup status of the operating system under the memory boot method through interaction with the memory monitoring module. Figure 4 As shown, the following method can be used to complete the interaction between the monitoring management module and the memory monitoring module.

[0075] Step 401: The monitoring management module and the memory monitoring module periodically exchange a third heartbeat packet. When the third heartbeat packet interaction is normal, the memory monitoring module starts normally and then feeds back a third success identification packet to the monitoring management module.

[0076] Similar to the hard drive monitoring module, after the memory monitoring module starts up, it exchanges a third heartbeat packet with the monitoring and management module to confirm that the memory monitoring module is operating normally and that communication between them is working properly. Simultaneously, the memory monitoring module sends a third success packet to the monitoring and management module, indicating that the operating system has successfully started in memory boot mode.

[0077] Step 402: When the monitoring management module receives the third success identification packet sent by the memory monitoring module, it indicates that the startup is successful.

[0078] If the monitoring and management module can receive the third heartbeat packet on time, it indicates that the communication between the monitoring and management module and the memory monitoring module is normal; if the monitoring and management module also receives the third success identification packet on time, it indicates that the operating system started by memory booting is running normally within this time period and the operating system is started successfully.

[0079] Step 403: When the monitoring management module does not receive the third success identification packet sent by the memory monitoring module within the specified time period, it indicates that the startup fails.

[0080] If the monitoring and management module cannot receive the third heartbeat packet on time, it indicates that the operating system may not start normally under the memory boot mode, or there is a communication failure between the monitoring and management module and the memory monitoring module, the operating system kernel cannot start, or the root file in the memory file system cannot be loaded; if the monitoring and management module can receive the third heartbeat packet on time, but cannot receive the third success identification packet on time, it also indicates that the operating system does not start normally under the memory boot mode. At this time, the monitoring and management module can also log the fault situation for subsequent fault handling.

[0081] After steps 401 to 403 provided in this embodiment, the startup status of the operating system in the memory booting mode can be obtained, thereby realizing monitoring of the startup status of the operating system in the memory booting mode.

[0082] Furthermore, after booting the operating system using the memory boot method, if a boot failure occurs again, it is necessary to continue using the memory boot method to restart the operation. To avoid repeated restarts caused by multiple failures, in this embodiment, each time the monitoring and management module receives the third failure identification packet, it records the number of boot failures; the memory monitoring module sends a fourth request packet to the monitoring and management module, and the monitoring and management module feedbacks the number of boot failures to the memory monitoring module via a fourth reply packet; when the number of boot failures exceeds the specified number, no reboot attempt is made, and the server's IP configuration is restored to facilitate remote fault handling.

[0083] In actual implementation, after booting in memory, an error check and repair are performed, and the number of failed boots in memory is checked. If the number is greater than or equal to the specified number, the IP address is configured and the system remains in the memory file system, awaiting further troubleshooting. If the number is less than the specified number, the root file in the memory file system is used to recover any damaged files in the hard disk file system. The system then changes the boot method back to the hard disk file system for the next boot, allowing the system to boot from the hard disk file system.

[0084] The method for AI server operating system fault detection and recovery provided in this embodiment has the following beneficial effects:

[0085] 1. The method provided in this embodiment can realize automatic perception and repair of the unbootable problem caused by server kernel and root file system failure.

[0086] 2. The method provided in this embodiment can enable the monitoring and management module to monitor and log kernel anomalies and root file system anomalies.

[0087] 3. The method provided in this embodiment does not require modification of the OS kernel and does not modify any content of the original root file system, and can achieve fault detection and recovery. It is extremely versatile and highly secure.

[0088] 4. The method provided in this embodiment does not require the support of additional hardware storage media and is implemented using pure software, thereby reducing the cost of fault detection and recovery.

[0089] 5. The method provided in this embodiment implements the server's heartbeat and watchdog functions through software, which can effectively reduce the impact of two fatal server operating system failures: the inability of the in-band OS kernel to start normally and the inability of the root file system to load normally. It is extremely versatile.

[0090] 6. The kernel and driver error log check and root file repair script content in the root file system can be modified and customized in real time, and the system's functional scalability is strong.

[0091] 7. The self-made memory file system adopts a read-only file system, which improves the reliability of the software system.

[0092] Example 2:

[0093] The AI ​​server operating system fault detection and recovery method provided in Example 1 can, in certain specific implementations, be implemented using the specific implementations of this embodiment. It should be understood that the specific implementations provided in this embodiment are merely intended to illustrate the specific implementation of the method in Example 1 in certain specific scenarios and are not intended to limit the scope of protection.

[0094] During a normal boot, the hard disk monitoring module within the hard disk file system boots normally and sends the first heartbeat packet 0x55 to the monitoring and management module. Simultaneously, it sends the first success identification packet 0x88 to the monitoring and management module, indicating that the operating system booted successfully using the hard disk boot method. The BIOS sends the second request packet 0xAA to the monitoring and management module, requesting the operating system boot status. The monitoring and management module then returns the second success identification packet 0xAA to the BIOS, indicating that the operating system booted successfully using the hard disk boot method according to the normal process.

[0095] If the kernel fails to boot or the disk root file system fails to load, the monitoring and management module cannot receive the 0x55 heartbeat packet sent by the hard disk monitoring module. In this case, the monitoring and management module sets the boot status flag osbootfailflag to 1, restarts the server system, and sends a second failure identification packet 0xBB to the BIOS to trigger the BIOS to switch the boot mode to memory boot mode.

[0096] The BIOS sends a second request packet (0xAA) to the monitoring and management module. If the monitoring and management module returns a second failure flag packet (0xBB), indicating that the operating system failed to boot in the current hard disk boot mode, the BIOS modifies the kernel startup file (grub.cfg) to change the boot mode from hard disk boot mode to memory boot mode.

[0097] After the system boots in ram mode, the BIOS adds the ram file system initramfs.tar.gz.xz and the ram file system startup configuration file grub-initramfs.cfg to the / boot directory of the OS. The BIOS also adds the file system check and recovery script fsck.sh to the ram file system initramfs.tar.gz.xz to recover the kernel module and the hard disk root file system.

[0098] The BIOS backs up the original kernel startup file grub.cfg in the operating system boot directory / boot on the hard disk as grub.cfg.bak, and copies the kernel startup configuration file grub-initramfs.cfg pre-existing in / boot as grub.cfg. The BIOS then starts the operating system using the initramfs.tar.gz.xz file pre-existing in the memory file system / boot.

[0099] After booting the operating system using memory boot, the memory monitoring module in the memory file system starts up and sends the third heartbeat packet 0x55 to the monitoring and management module. Simultaneously, it sends the third success flag 0x99 to the monitoring and management module, indicating a successful boot from the memory file. If the monitoring and management module detects the 0x99 packet sent by the memory monitoring module, it sets osbootfailflag to 0.

[0100] The kernel and disk root file repair scripts within the in-memory file system begin executing, performing error checking and repair. Simultaneously, the memory monitoring module sends a fourth request packet (0xCC) to the monitoring and management module, requesting the number of in-memory file system boot failures. If the number is five or more, the server configures the IP address and remains in the in-memory file system. If the number is less than five, the grub.cfg file is restored, the OS is restarted, and the server boots using the in-memory boot method the next time it boots.

[0101] Example 3:

[0102] In the AI ​​server operating system fault detection and recovery method provided in Example 1, each step can be organized into multiple functional modules provided in this embodiment for implementation. It should be understood that the specific implementation methods provided in this embodiment are only used to illustrate the specific implementation process of the method in Example 1 in certain specific scenarios and are not intended to limit the scope of protection.

[0103] The functional modules used in this embodiment are: BIOS module, hard disk monitoring module, memory monitoring module, system repair module, monitoring management module and repair script. Figure 5 1, which shows the functional modules used in this embodiment and the message interactions between the functional modules. For ease of explanation, the messages exchanged between different modules in this embodiment are the same as those in Example 2.

[0104] (1)BIOS module.

[0105] like Figure 6 As shown in the figure, after the BIOS completes hardware initialization in the server, before loading the OS, it sends a 0xAA message to the monitoring and management module, requesting whether the OS has booted successfully. If the BIOS receives a 0xAA message, indicating successful OS boot, it continues the previous process and boots the OS using the hard disk. If the BIOS receives a 0xBB message, indicating OS boot failure, the BIOS backs up the current OS boot configuration file, grub.cfg, and selects the boot configuration file pre-stored in the in-memory file system, initramfs.tar.gz.xz, to boot the OS from the in-memory file system.

[0106] (2) Hard disk monitoring module.

[0107] like Figure 7As shown in the figure, the root file in the hard disk file system is loaded normally. The hard disk monitoring module sends 0x55 to the monitoring management module every 1 minute and receives 0x55 from the monitoring management module at the same time. If the hard disk monitoring module receives 0x55 from the monitoring management module, it sends a heartbeat confirmation packet 0x88 to the monitoring management module, indicating that the hard disk file system has been successfully accessed.

[0108] (3)Memory monitoring module.

[0109] like Figure 8 As shown, the root file in the memory file system is loaded normally. The memory monitoring module sends 0x55 to the monitoring and management module every minute and receives 0x55 from the monitoring and management module. If the memory monitoring module receives 0x55 from the monitoring and management module, it sends 0x99 to the monitoring and management module, indicating that the memory file system has been successfully entered. At the same time, the memory monitoring module sends the monitoring and management module 0xCC command to obtain the number of failed in-memory boot mode startups. This is used for memory file system error checking and for the repair script to determine whether to remain in the memory file system.

[0110] (4) System repair module.

[0111] like Figure 9 As shown, the root file system in the memory file system is loaded normally, and the repair script is executed. If the repair is successful, the memory monitoring module obtains the number of failed boot attempts using the memory boot method. If this number is greater than or equal to five, the network port IP address is configured and the system remains in the memory file system. Otherwise, the grub.cfg file in the hard disk file system is restored, and the system is restarted. The system will then boot normally from the hard disk root file system. If the repair fails, the OS is directly restarted.

[0112] (5) Monitoring and management module.

[0113] like Figure 10As shown, the monitoring and management module boots normally, executing interactive processes with the BIOS, hard disk monitoring module, and memory monitoring module, monitoring messages sent by the BIOS, hard disk monitoring module, and memory monitoring module. If it receives a 0xAA message from the BIOS, it determines the value of osbootflag based on whether it receives a heartbeat packet from the hard disk monitoring module. If no heartbeat packet is received, osbootflag is set to 1. If osbootflag is 1, the monitoring and management module sends 0xBB to the hard disk monitoring module and restarts the OS, allowing it to boot from the in-memory file system the next time and perform kernel and root file system error checking and repair. If osbootflag is 0, and if the monitoring and management module receives a 0x88 message from the hard disk monitoring module, it sets osbootdiskflag to 1; if it receives a 0x99 message from the memory monitoring module, it sets osbootramflag to 1. The monitoring and management module also logs events related to the OS entering the in-memory file system for user access.

[0114] (6) Repair the script.

[0115] In a specific implementation, the system recovery process can be pre-written into a script file, and the error checking and recovery process of the kernel or hard disk file system can be controlled by the script.

[0116] The error checking and recovery logic of the kernel driver module is as follows: The current kernel log is collected through the Linux dmesg command, and the error information in the kernel log is checked. If some drivers such as xx1, xx2, xxN report errors, it indicates that there is a problem with the current kernel driver. In this case, the / boot / grub.cfg.bak file is modified and the parameter configuration blacklist = xx1, xx2, xxN is added to the kernel startup parameters to disable the problematic driver module, thereby fixing the kernel startup abnormality caused by the driver problem during the kernel startup process.

[0117] The logic for checking and repairing hard disk file system errors is as follows: First, the root file system type corresponding to the OS in the hard disk file system is obtained. Based on the root file system type, the corresponding fsck tool is called to check and repair the disk's root file system, repairing any root file system errors. If fsck is successful, it returns 0, indicating success; otherwise, it returns 1, indicating failure.

[0118] Example 4:

[0119] Based on the AI ​​server operating system fault detection and recovery methods provided in Examples 1 to 3 above, the present invention also provides an AI server operating system fault detection and recovery device that can be used to implement the above methods, such as Figure 11 , which is a schematic diagram of the device architecture of an embodiment of the present invention. The device for detecting and recovering an AI server operating system fault in this embodiment includes one or more processors 11 and a memory 12. Figure 11 A processor 11 is taken as an example.

[0120] The processor 11 and the memory 12 may be connected via a bus or other means. Figure 11 The bus connection is taken as an example.

[0121] Memory 12, as a nonvolatile computer-readable storage medium for the AI ​​server operating system fault detection and recovery method, can be used to store nonvolatile software programs, nonvolatile computer executable programs, and modules, such as the AI ​​server operating system fault detection and recovery methods described in Examples 1 through 3. Processor 11 executes the nonvolatile software programs, instructions, and modules stored in memory 12 to execute the various functional applications and data processing of the AI ​​server operating system fault detection and recovery device, thereby implementing the AI ​​server operating system fault detection and recovery methods described in Examples 1 through 3.

[0122] The memory 12 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory 12 may optionally include a memory remotely located relative to the processor 11, and such remote memory may be connected to the processor 11 via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0123] The program instructions / modules are stored in the memory 12, and when executed by one or more processors 11, the method for detecting and recovering the AI ​​server operating system fault in the above-mentioned embodiments 1 to 3 is executed, for example, the above-described Figures 1-4 The steps shown.

[0124] An embodiment of the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors, for example Figure 11 A processor 11 in the embodiment 1 can enable the one or more processors to execute the AI ​​server operating system fault detection and recovery method in embodiments 1 to 3, for example, executing the above-described Figures 1-4 The steps shown can also be implemented Figure 5-10 The various modules and units described.

[0125] It is worth noting that the information interaction, execution process, etc. between the modules and units within the above-mentioned devices and systems are based on the same concept as the processing method embodiment of the present invention. The specific content can be found in the description of the method embodiment of the present invention and will not be repeated here.

[0126] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), a disk or an optical disk, etc.

[0127] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for detecting and recovering faults in an AI server operating system, characterized in that: include: The hard disk file system is used as the boot mode when the operating system is started, the monitoring management module exchanges a first heartbeat packet with the hard disk monitoring module in the hard disk file system, and the startup status of the operating system is obtained according to the first heartbeat packet; A second heartbeat packet interaction is established between the monitoring and management module and the BIOS. The monitoring and management module feeds back the startup status to the BIOS through the second heartbeat packet. When the startup fails, the BIOS switches the boot mode to the memory file system and starts the operating system through the system root file pre-stored in the memory file system; wherein, the monitoring and management module is the BMC.

2. The method for detecting and recovering an AI server operating system fault according to claim 1, wherein: The monitoring management module exchanges a first heartbeat packet with the hard disk monitoring module in the hard disk file system, and obtains the startup status of the operating system according to the first heartbeat packet, specifically including: The monitoring management module and the hard disk monitoring module periodically exchange the first heartbeat packet. When the first heartbeat packet interaction is normal, the hard disk monitoring module starts normally and then feeds back the first success identification packet to the monitoring management module. When the monitoring management module receives a specified number of first success identification packets within a specified waiting time, it indicates that the startup status of the operating system is startup success; When the monitoring management module does not receive the first success identification packet within the specified waiting time, it indicates that the startup status of the operating system is startup failure.

3. The method for detecting and recovering an AI server operating system fault according to claim 1, wherein: The monitoring and management module feeds back the startup status to the BIOS via the second heartbeat packet, specifically including: The monitoring management module and the BIOS periodically exchange a second heartbeat packet. When the second heartbeat packet interaction is normal, the BIOS starts normally and sends a second request packet to the monitoring management module to request the startup status of the operating system from the monitoring management module through the second request packet. When the startup status is successful, the monitoring management module returns a second success identification packet to the BIOS; When the startup status is startup failure, the monitoring management module returns a second failure identification packet to the BIOS.

4. The method for detecting and recovering an AI server operating system fault according to claim 3, wherein: The monitoring management module feeds back the startup status to the BIOS via the second heartbeat packet, and further includes: When the operating system fails to start, the monitoring and management module controls the server to re-power on, and the re-power on triggers the BIOS to re-send the second request packet to the monitoring and management module.

5. The method for detecting and recovering an AI server operating system fault according to claim 1, wherein: The BIOS switches the boot mode to the memory file system, specifically including: The BIOS changes the boot item to the memory file system and boots the operating system using the system root file pre-stored in the memory file system.

6. The method for detecting and recovering an AI server operating system fault according to claim 1, wherein: After the operating system is started through the system root file pre-stored in the memory file system, it also includes: The monitoring management module and the memory monitoring module periodically exchange a third heartbeat packet. When the third heartbeat packet interaction is normal, the memory monitoring module starts normally and then feeds back a third success identification packet to the monitoring management module. When the monitoring management module receives the third success identification packet sent by the memory monitoring module, it indicates that the startup is successful; When the monitoring management module does not receive the third success identification packet sent by the memory monitoring module within the specified time period, it indicates that the startup has failed.

7. The method for detecting and recovering an AI server operating system fault according to claim 6, wherein: When the startup status is successful, the operating system is started by using the system root file pre-stored in the memory file system, further comprising: The BIOS backs up the startup files and system root files in the hard disk file system, and copies the system root files pre-stored in the memory file system to the hard disk file system. During the next startup, the system root files that were successfully started the previous time are used to start the system.

8. The method for detecting and recovering an AI server operating system fault according to claim 6, wherein: When the startup status is startup failure, the method of starting the operating system by using the system root file pre-stored in the memory file system further includes: Each time the monitoring management module receives the third failure identification packet, it records the number of startup failures; The memory monitoring module sends a fourth request packet to the monitoring management module, and the monitoring management module feeds back the number of startup failures to the memory monitoring module via a fourth reply packet; If the number of startup failures exceeds the specified number, no restart attempts are made and the server's IP configuration is restored to facilitate remote troubleshooting.

9. A device for detecting and recovering AI server operating system faults, characterized by: The method comprises at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to complete the method for AI server operating system fault detection and recovery according to any one of claims 1 to 8.

10. A non-volatile computer storage medium, characterized in that The computer storage medium stores computer-executable instructions, which are executed by one or more processors to complete the method for AI server operating system fault detection and recovery as described in any one of claims 1 to 8.

Images