Fault detection and recovery method and device for AI server operating system

Abstract

The invention relates to the field of operating systems, in particular to an AI server operating system fault detection and recovery method and device. The method mainly comprises the steps that a hard disk file system serves as a guiding mode when the operating system is started, a first heartbeat packet is interacted between a monitoring management module and a hard disk monitoring module in the hard disk file system, and the starting state of the operating system is obtained according to the first heartbeat packet; a second heartbeat packet interaction is established between the monitoring management module and the BIOS, the monitoring management module feeds back the starting state to the BIOS through the second heartbeat packet, when starting fails, the BIOS switches the guiding mode to the memory file system, and the operating system is started through a system root file pre-stored in the memory file system. According to the method, the influence of two fatal server operating system faults, namely incapability of normally starting an operating system kernel and incapability of normally loading a root file system, can be effectively reduced, and the loss caused by the starting fault of the server is avoided.

Description

Technical Field

[0001] The present invention relates to the field of operating systems, and in particular to a method and device for detecting and recovering faults of an AI server operating system. Background Art

[0002] In the current artificial intelligence (AI) server usage scenarios, the main operating system is the Linux operating system. During the startup process of the Linux operating system, startup failures such as downtime or abnormal restart may occur. When the server hardware fails, the operating system's hardware device driver is incompatible with the hardware, or the Linux operating system's core components are damaged, startup failures may occur during the server operating system's kernel startup process.

[0003] The two core components of the Linux operating system are the Linux kernel and the root file system. The root file system is generally stored in the hard disk. The root file system may be damaged due to sudden power failure of the server or plugging and unplugging of the hard disk. At the same time, due to the hard disk storage medium, the root file system may also be damaged due to factors such as bad blocks. When the root file system is damaged, the operating system will also fail to start.

[0004] In the current use of servers, after the operating system is started normally, the server's monitoring and management module (Base-board Management Controller, abbreviated as BMC) can be used to detect the running status of the server and the operating system. However, the monitoring and management module cannot detect the problem of abnormal startup caused by the abnormal kernel startup of the operating system itself and the abnormal root file system, and the event log of the monitoring and management module cannot record any abnormality. On the one hand, this fatal abnormality cannot be perceived, and on the other hand, the fault cannot be automatically recovered, which has a great impact on the actual use of the server.

[0005] In view of this, how to overcome the defects of the prior art and solve the problem that the prior art cannot detect and recover the abnormal kernel startup and the root file system abnormality that cause the abnormal startup is a problem to be solved in this technical field. Summary of the invention

[0006] In view of the above defects or improvement needs of the prior art, the present invention solves the problem that the prior art cannot detect and recover the abnormal kernel startup and the root file system abnormality that cause the abnormal startup.

[0007] The embodiment of the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a method for fault detection and recovery of an AI server operating system, specifically: using a hard disk file system as a boot mode when the operating system is started, a first heartbeat packet is exchanged between a monitoring management module and a hard disk monitoring module in the hard disk file system, and the startup status of the operating system is obtained according to the first heartbeat packet; a second heartbeat packet interaction is established between the monitoring management module and the BIOS, and the monitoring management module feeds back the startup status to the BIOS through the second heartbeat packet. When the startup fails, the BIOS switches the boot mode to the memory file system, and starts the operating system through the system root file pre-stored in the memory file system.

[0009] Preferably, the monitoring management module exchanges a first heartbeat packet with the hard disk monitoring module in the hard disk file system, and the startup status of the operating system is obtained according to the first heartbeat packet, which specifically includes: the monitoring management module and the hard disk monitoring module periodically exchange the first heartbeat packet, and when the first heartbeat packet interaction is normal, the hard disk monitoring module feeds back a first success identification packet to the monitoring management module after normal startup; when the monitoring management module receives a specified number of first success identification packets within a specified waiting time, it indicates that the startup status of the operating system is a successful startup; when the monitoring management module does not receive the first success identification packet within the specified waiting time, it indicates that the startup status of the operating system is a failed startup.

[0010] Preferably, the monitoring and management module feeds back the startup status to the BIOS via a second heartbeat packet, specifically including: periodic interaction of the second heartbeat packet between the monitoring and management module and the BIOS; when the second heartbeat packet interaction is normal, the BIOS sends a second request packet to the monitoring and management module after normal startup, and requests the startup status of the operating system from the monitoring and management module via the second request packet; when the startup status is a successful startup, the monitoring and management module returns a second success identification packet to the BIOS; when the startup status is a failed startup, the monitoring and management module returns a second failure identification packet to the BIOS.

[0011] Preferably, the monitoring management module feeds back the startup status to the BIOS via a second heartbeat packet, and also includes: when the operating system fails to start, the monitoring management module controls the server to repower on, and triggers the BIOS to resend the second request packet to the monitoring management module by repowering on.

[0012] Preferably, the BIOS switches the boot mode to the memory file system, specifically including: the BIOS modifies the startup boot item to the memory file system, and boots the operating system through the system root file pre-stored in the memory file system.

[0013] Preferably, after the operating system is started through the system root file pre-stored in the memory file system, it also includes: the monitoring management module and the memory monitoring module periodically interact with each other through a third heartbeat packet; when the third heartbeat packet interaction is normal, the memory monitoring module feeds back a third success identification packet to the monitoring management module after normal startup; when the monitoring management module receives the third success identification packet sent by the memory monitoring module, it indicates that the startup is successful; when the monitoring management module does not receive the third success identification packet sent by the memory monitoring module within a specified time period, it indicates that the startup has failed.

[0014] Preferably, when the startup status is successful startup, the operating system is started through the system root file pre-stored in the memory file system, and also includes: the BIOS backs up the startup file and the system root file in the hard disk file system, and copies the system root file pre-stored in the memory file system to the hard disk file system, and at the next startup, the system root file that was successfully started last time is used to start the system.

[0015] Preferably, when the startup status is a startup failure, the operating system is started through the root file pre-stored in the memory file system, and also includes: each time the monitoring management module receives the third failure identification packet, the memory monitoring module records the number of startup failures; the memory monitoring module sends a fourth request packet to the monitoring management module, and the monitoring management module feeds back the number of startup failures to the memory monitoring module through a fourth reply packet; when the number of startup failures is greater than the specified number, no restart is attempted, and the IP configuration of the server is restored to facilitate remote fault handling.

[0016] In a second aspect, the present invention provides an AI server operating system fault detection and recovery device, specifically: including at least one processor and a memory, the at least one processor and the memory are connected via a data bus, the memory stores instructions that can be executed by at least one processor, and after the instructions are executed by the processor, they are used to complete the AI ​​server operating system fault detection and recovery method in the first aspect.

[0017] In a third aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the method provided by the method described in the first aspect.

[0018] In a fourth aspect, a chip is provided, comprising: a processor and an interface, for calling and running a computer program stored in the memory from a memory, and executing the method provided in the first aspect.

[0019] According to a fifth aspect, a computer program product comprising instructions is provided. When the instructions are executed on a computer or a processor, the computer or the processor executes the method provided in the first aspect.

[0020] Compared with the prior art, the beneficial effects of the present invention are: heartbeat packet interaction is performed between the monitoring management module and the hard disk monitoring module, the heartbeat and watchdog functions are realized when the server is started, and the operating system is restored through the system root file pre-stored in the memory file system; at the same time, in the preferred embodiment of the present invention, the startup process of the memory file system is further monitored by the monitoring management module and the memory monitoring module. Using this method, the impact of the two fatal server operating system failures, namely, the failure of the operating system kernel to start normally and the failure of the root file system and the failure of loading normally, can be effectively reduced, and the loss caused by the server startup failure can be avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A flow chart of a method for fault detection and recovery of an AI server operating system provided in an embodiment of the present invention;

[0023] Figure 2 A flowchart of another method for AI server operating system fault detection and recovery provided by an embodiment of the present invention;

[0024] Figure 3 A flowchart of another method for AI server operating system fault detection and recovery provided by an embodiment of the present invention;

[0025] Figure 4 A flowchart of another method for AI server operating system fault detection and recovery provided by an embodiment of the present invention;

[0026] Figure 5 A schematic diagram of the message interaction process between the functional modules in the method provided in the embodiment of the present invention;

[0027] Figure 6 A schematic diagram of a specific implementation process of a BIOS module in a specific scenario of the method provided by an embodiment of the present invention;

[0028] Figure 7 A schematic diagram of the data processing process of the BIOS module in a specific scenario of the method provided by an embodiment of the present invention;

[0029] Figure 8 A schematic diagram of the data processing process of a hard disk monitoring module in a specific scenario of the method provided by an embodiment of the present invention;

[0030] Fig. 9 A schematic diagram of the data processing process of the memory monitoring module in a specific scenario of the method provided by an embodiment of the present invention;

[0031] Fig.10 A schematic diagram of the data processing process of the system repair module in a specific scenario of the method provided by an embodiment of the present invention;

[0032] Fig.11 A schematic diagram of the structure of an AI server operating system fault detection and recovery device provided by an embodiment of the present invention;

[0033] The reference numerals are as follows:

[0034] 11: processor; 12: memory. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0036] The present invention is an architecture of a specific functional system, so the specific embodiments mainly illustrate the functional logical relationship between the various structural modules, and do not limit the specific software and hardware implementation methods.

[0037] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0038] The operating system (OS) currently used in server applications is mainly the Linux operating system. The two core components of the Linux operating system are the Linux kernel and the root file system. Due to hardware compatibility or hardware driver anomalies, the Linux kernel may fail to start. For example, the OS kernel startup may get stuck or restart abnormally, causing fatal problems such as server downtime or repeated restarts.

[0039] At present, the root files of the Linux operating system are stored on the hard disk. The root file system may be damaged by factors such as the life of the hard disk medium, the damage of the hard disk logical volume manager (LVM) partition data, abnormal server restart, and sudden power on and off. As a result, the root file system cannot be mounted normally during the Linux kernel startup process, resulting in the inability to start the operating system. The server will be stuck or enter emergency mode. At this time, the network card has no IP configured, and the business cannot be executed, affecting the normal use of the server. It is a fatal problem for the server.

[0040] The monitoring and management module in the current server cannot monitor and record the situation where the above two operating systems cannot start normally, and because there is no IP configuration, the fault cannot be handled remotely. Users can only manually analyze the fault and try to recover on site, which seriously affects the operation and maintenance efficiency of the server.

[0041] Embodiment 1:

[0042] In order to solve the above problems, this embodiment provides an AI server operating system fault detection and recovery method to perceive and automatically recover such server anomalies, thereby improving the reliability and operation and maintenance efficiency of the server.

[0043] like Figure 1 As shown, the specific steps of the method provided by the embodiment of the present invention are as follows:

[0044] Step 101: Using the hard disk file system as the boot mode when the operating system is started, the monitoring management module exchanges a first heartbeat packet with the hard disk monitoring module in the hard disk file system, and the startup state of the operating system is obtained according to the first heartbeat packet.

[0045] After the server is powered on, the Basic Input Output System (BIOS) first powers on and detects each hardware component. After all hardware tests pass, it enters the Boot Device Selection (BDS) stage. In the boot device selection stage, the BIOS obtains the root file and kernel of the operating system from the boot device according to the pre-specified boot method, and uses the root file to start the operating system. In actual implementation, the boot device used as a boot method can be a hard disk, a CD, a USB device, a memory, etc.; in normal use, the hard disk file system is usually used as the boot method at startup. In the prior art, the monitoring and management module cannot monitor the above process, and therefore cannot perform recovery and fault handling in the event of a startup failure.

[0046] In the method provided in this embodiment, a hard disk monitoring module (hard disk OS agent) is added to the hard disk file system. If the hard disk file system is used as the boot mode when the operating system is started (hereinafter referred to as the hard disk boot mode), after the BIOS is started, the root file in the folder specified by the hard disk file system is used to start the operating system. After the hard disk monitoring module is normally started, it interacts with the monitoring and management module through the first heartbeat packet, and sends the first success identification packet to the monitoring and management module, indicating that the OS is successfully started from the hard disk. If the hard disk monitoring module is not started normally, the monitoring and management module cannot receive the first heartbeat packet sent by the hard disk monitoring module in the hard disk file system, indicating that the hard disk file system fails to load or the kernel fails to start, and the monitoring and management module needs to restart the system, handle faults, recover damaged files, and record fault logs as needed.

[0047] Step 102: A second heartbeat packet interaction is established between the monitoring management module and the BIOS. The monitoring management module feeds back the startup status to the BIOS through the second heartbeat packet. When the startup fails, the BIOS switches the boot mode to the memory file system and starts the operating system through the system root file pre-stored in the memory file system.

[0048] After the BIOS is started, a second heartbeat packet is established between the BIOS and the monitoring and management module to ensure normal communication, and then a second request packet is sent to the monitoring and management module to query the startup status.

[0049] If the operating system is successfully started, the monitoring management module replies to the BIOS with a second success identification packet. After the BIOS receives the second success identification packet, it is informed that the operating system has been successfully started, and subsequent operations can be performed normally.

[0050] If the operating system fails to start, the monitoring management module replies to the BIOS with a second failure identification packet. After receiving the second failure identification packet, the BIOS learns that the operating system fails to start, and restarts or recovers. In this embodiment, when the operating system is restarted, the root file that may have been damaged in the hard disk file system is not used, but the correct root file is pre-stored in the memory file system. The BIOS modifies the boot item to the memory file system, and starts the operating system through the system root file pre-stored in the memory file system.

[0051] If the system can be started normally using the root file in the memory file system, the BIOS will back up the startup file and system root file in the hard disk file system, and copy the system root file pre-stored in the memory file system to the hard disk file system. At the next startup, the system root file that was successfully started the last time will be used to start the system.

[0052] Furthermore, in order to prevent the root files in the hard disk file system and the memory file system from being damaged simultaneously, the hard disk file system and the memory file system are not stored on the same hard disk, or are not stored on the same physical or logical sector of the same hard disk.

[0053] After steps 101 and 102 provided in this embodiment, the AI ​​server operating system fault detection and recovery can be completed.

[0054] In this embodiment, the hard disk monitoring module in the hard disk file system is used to monitor the operating system startup process in the hard disk boot mode, and is responsible for communicating with the monitoring management module and feeding back the startup result of the hard disk boot mode to the monitoring management module. Figure 2 As shown, the following method can be used to complete the interaction between the monitoring management module and the hard disk monitoring module in the hard disk file system.

[0055] Step 201: The monitoring management module and the hard disk monitoring module periodically exchange the first heartbeat packet. When the first heartbeat packet interaction is normal, the hard disk monitoring module starts normally and then feeds back the first success identification packet to the monitoring management module.

[0056] The monitoring management module establishes a heartbeat by sending the first heartbeat packet to the hard disk monitoring module in the hard disk operating system. In actual implementation, the monitoring management module receives the first heartbeat packet sent by the hard disk monitoring module at intervals of 1 minute; at the same time, after the operating system is normally started through the hard disk boot method, the hard disk monitoring module in the hard disk file system will also feedback the first success identification packet to the monitoring management module, indicating that it is operating normally.

[0057] Step 202: When the monitoring management module receives a specified number of first success identification packets within a specified waiting time, it indicates that the startup status of the operating system is successful.

[0058] If the monitoring and management module can receive the first heartbeat packet on time, it indicates that the communication between the monitoring and management module and the hard disk monitoring module is normal; if the monitoring and management module also receives the first success identification packet on time, it indicates that the operating system started by the hard disk boot method runs normally during this time period and the operating system is started successfully.

[0059] Step 203: When the monitoring management module does not receive the first success identification packet within the specified waiting time, it indicates that the startup status of the operating system is startup failure.

[0060] If the monitoring and management module cannot receive the first heartbeat packet on time, it indicates that the operating system may not start normally under the hard disk boot mode, or there is a communication failure between the monitoring and management module and the hard disk monitoring module, the operating system kernel cannot start, or the root file in the hard disk file system cannot be loaded; if the monitoring and management module can receive the first heartbeat packet on time, but cannot receive the first success identification packet on time, it also indicates that the operating system does not start normally under the hard disk boot mode.

[0061] After steps 201 to 203 provided in this embodiment, the monitoring management module can obtain the startup status of the operating system in the hard disk boot mode, so as to realize the monitoring of the startup status of the operating system in the hard disk boot mode.

[0062] In actual implementation, after the monitoring management module obtains the startup status of the hard disk boot mode, it can use the startup status flag to record the status of the hard disk boot mode. When the BIOS queries the monitoring management module for the startup status, the monitoring management module can feedback the corresponding startup status to the BIOS based on the value of the startup status flag. Figure 3 As shown, the interaction between the monitoring management module and the BIOS can be completed in the following manner.

[0063] Step 301: The monitoring management module and the BIOS periodically exchange second heartbeat packets. When the second heartbeat packet interaction is normal, the BIOS starts normally and sends a second request packet to the monitoring management module to request the startup status of the operating system from the monitoring management module through the second request packet.

[0064] When the BIOS is started, the second heartbeat packet is sent to the monitoring management module to establish a heartbeat. If the heartbeat is established normally, it means that the abnormal server startup is not caused by hardware failure.

[0065] At the same time, the BIOS sends a second request packet to the monitoring management module through the Intelligent Platform Management Interface (IPMI) to query whether the operating system is started normally in the hard disk boot mode. The monitoring management module monitors the second request packet sent by the BIOS and returns an identification packet corresponding to the current startup status.

[0066] Step 302: When the startup status is successful, the monitoring management module returns a second success identification packet to the BIOS.

[0067] After the BIOS receives the second success identification packet fed back by the monitoring management module, it is learned that the operating system starts normally in the hard disk booting mode.

[0068] Step 303: When the startup status is startup failure, the monitoring management module returns a second failure identification packet to the BIOS.

[0069] When the BIOS receives the second failure identification packet fed back by the monitoring management module, it is learned that the operating system fails to start in the hard disk booting mode.

[0070] On the other hand, in actual implementation, in order to avoid repeated requests, the BIOS only sends the second request packet to the monitoring management module once after completing the booting of the operating system to obtain the startup status of the operating system. Therefore, when the operating system fails to boot, the BIOS will no longer send the second request packet to the monitoring management module, and only perform the second heartbeat packet interaction to confirm that the communication status between the two is normal. At this time, in order to query the startup status after the next boot of the operating system, the monitoring management module controls the server to power on again, and the power on again triggers the BIOS to resend the second request packet to the monitoring management module.

[0071] After steps 301 to 303 provided in this embodiment, the interaction between the monitoring management module and the BIOS is completed, so that the BIOS can learn the startup status of the operating system so that the BIOS can perform corresponding subsequent operations according to the startup status.

[0072] Furthermore, in order to avoid automatic shutdown or repeated restart within a short period of time after the operating system is started, the monitoring management module can feed back the identification packets corresponding to the startup status to the BIOS multiple times within a period of time, and the BIOS determines the startup status based on the identification packets received within a period of time. In a specific scenario, if the BIOS receives a specified number of second success identification packets within a specified time period after the BIOS starts to guide the operating system to start, it indicates that the operating system has started successfully; if the specified number of second success identification packets are not received within the specified time period, or the second failure identification packet is received within the specified time, it indicates that the operating system has failed to start.

[0073] When the operating system boots successfully in hard disk boot mode, the BIOS does not need to perform any additional processing. When the operating system boots unsuccessfully in hard disk boot mode, the BIOS switches the boot mode to the memory file system and attempts to restart or recover the operating system.

[0074] If the memory file system is used as the boot mode when the operating system is started (hereinafter referred to as the memory boot mode) to start the operating system, the startup failure may occur again. Therefore, in the method provided in this embodiment, a memory monitoring module (memory OS agent) is also set in the memory file system, and the monitoring management module obtains the startup status of the operating system in the memory boot mode through interaction with the memory monitoring module. Figure 4 As shown, the following method can be used to complete the interaction between the monitoring management module and the memory monitoring module.

[0075] Step 401: the monitoring management module and the memory monitoring module periodically exchange the third heartbeat packet. When the third heartbeat packet interaction is normal, the memory monitoring module starts normally and then feeds back the third success identification packet to the monitoring management module.

[0076] Similar to the hard disk monitoring module, after the memory monitoring module is started, it sends a third heartbeat packet to the monitoring management module to confirm that the memory monitoring module is running normally and the communication between the two is normal. At the same time, the memory monitoring module sends a third success identification packet to the monitoring management module, indicating that the operating system is successfully started in the memory boot mode.

[0077] Step 402: When the monitoring management module receives the third success identification packet sent by the memory monitoring module, it indicates that the startup is successful.

[0078] If the monitoring and management module can receive the third heartbeat packet on time, it indicates that the communication between the monitoring and management module and the memory monitoring module is normal; if the monitoring and management module also receives the third success identification packet on time, it indicates that the operating system started by the memory boot method runs normally during this time period and the operating system is started successfully.

[0079] Step 403: When the monitoring management module does not receive the third success identification packet sent by the memory monitoring module within the specified time period, it indicates that the startup fails.

[0080] If the monitoring and management module cannot receive the third heartbeat packet on time, it indicates that the operating system may not start normally in the memory boot mode, or there is a communication failure between the monitoring and management module and the memory monitoring module, the operating system kernel cannot start, or the root file in the memory file system cannot be loaded; if the monitoring and management module can receive the third heartbeat packet on time, but cannot receive the third success identification packet on time, it also indicates that the operating system does not start normally in the memory boot mode. At this time, the monitoring and management module can also log the fault situation for subsequent fault handling.

[0081] After steps 401-403 provided in this embodiment, the startup state of the operating system in the memory booting mode can be obtained, so as to realize the monitoring of the startup state of the operating system in the memory booting mode.

[0082] Furthermore, after the operating system is started using the memory boot method, if a startup failure occurs again, it is necessary to continue to use the memory boot method to restart the operation. In order to avoid repeated restarts caused by multiple failures, in this embodiment, the monitoring and management module records the number of startup failures each time it receives the third failure identification packet; the memory monitoring module sends a fourth request packet to the monitoring and management module, and the monitoring and management module feeds back the number of startup failures to the memory monitoring module through the fourth reply packet; when the number of startup failures is greater than the specified number, no restart is attempted, and the server's IP configuration is restored for remote fault handling.

[0083] In actual implementation, after booting in memory boot mode, error checking and repair are performed, and the number of failed boots in memory boot mode is queried. If it is greater than or equal to the specified number, the IP is configured and the system stays in the memory file system, waiting for subsequent fault processing; if it is less than the specified number, the root file in the memory file system is used to restore the files that may be damaged in the hard disk file system, and then the boot mode at the next startup is changed back to the hard disk file system so that the system can be started from the hard disk file system next time.

[0084] The method for AI server operating system fault detection and recovery provided in this embodiment has the following beneficial effects:

[0085] 1. The method provided in this embodiment can realize automatic perception and repair of the unbootable problem caused by server kernel and root file system failure.

[0086] 2. The method provided in this embodiment can enable the monitoring management module to monitor and log kernel anomalies and root file system anomalies.

[0087] 3. The method provided in this embodiment does not require modification of the OS kernel, and does not modify any content of the original root file system, and can achieve fault detection and recovery, and is extremely versatile and highly secure.

[0088] 4. The method provided in this embodiment does not require the support of additional hardware storage media and is implemented using pure software, thereby reducing fault detection and recovery costs.

[0089] 5. The method provided in this embodiment implements the heartbeat and watchdog functions of the server through software, which can effectively reduce the impact of two fatal server operating system failures: the in-band OS kernel cannot be started normally and the root file system cannot be loaded normally. It is extremely versatile.

[0090] 6. The kernel and driver error log check and root file repair script content in the root file system can be modified and customized in real time, and the system's functional scalability is strong.

[0091] 7. The self-made memory file system adopts a read-only file system, which improves the reliability of the software system.

[0092] Embodiment 2:

[0093] Based on the AI ​​server operating system fault detection and recovery method provided in Example 1, in some specific implementations, it can be implemented through the specific implementation of this embodiment. It can be understood that the specific implementation provided in this embodiment is only used to illustrate the specific implementation process of the method in Example 1 in certain specific scenarios, and is not intended to limit the scope of protection.

[0094] In the case of normal startup, the hard disk monitoring module in the hard disk file system starts normally, sends the first heartbeat packet 0x55 to the monitoring and management module, and sends the first success identification packet 0x88 to the monitoring and management module, indicating that the operating system is successfully started under the hard disk boot system. BIOS sends the second request packet 0xAA to the monitoring and management module to request the operating system startup status. At this time, the monitoring and management module returns the second success identification packet 0xAA to the BIOS, indicating that the operating system has been started using the hard disk boot method according to the normal process.

[0095] When the kernel cannot be started or the disk root file system cannot be loaded, the monitoring and management module cannot receive the 0x55 heartbeat packet sent by the hard disk monitoring module. At this time, the monitoring and management module sets the boot status flag osbootfailflag = 1, restarts the server system, and sends the second failure identification packet 0xBB to the BIOS to trigger the BIOS to switch the boot mode to the memory boot mode.

[0096] BIOS sends a second request packet 0xAA to the monitoring management module. If the monitoring management module returns a second failure identification packet 0xBB, it indicates that the operating system fails to start in the current hard disk boot mode. At this time, BIOS modifies the kernel boot file grub.cfg and changes the boot mode from hard disk boot mode to memory boot mode.

[0097] After changing to the memory boot mode, BIOS adds a new memory file system initramfs.tar.gz.xz and a memory file system startup configuration file grub-initramfs.cfg in the / boot directory of the OS, and adds a file system check and recovery script fsck.sh in the memory file system initramfs.tar.gz.xz to recover the kernel module and the hard disk root file system.

[0098] BIOS backs up the original kernel startup file grub.cfg in the hard disk operating system boot directory / boot as grub.cfg.bak, and copies the kernel startup configuration file grub-initramfs.cfg pre-existing in / boot as grub.cfg, and starts the operating system through the memory file system initramfs.tar.gz.xz file pre-existing in / boot.

[0099] After the operating system is started using the memory boot method, the memory monitoring module in the memory file system starts and sends the third heartbeat packet 0x55 to the monitoring management module, and at the same time sends the third success flag 0x99 to the monitoring management module, indicating that the startup from the memory file is successful. If the monitoring management module monitors the 0x99 message sent by the memory monitoring module, it will set osbootfailflag = 0.

[0100] The kernel and disk root file repair scripts in the memory file system start to execute, and error checking and repair are performed. At the same time, the memory monitoring module sends the fourth request packet 0xCC to the monitoring management module to request the number of failed startups of the memory file system. If the number is greater than or equal to 5 times, the IP is configured and stays in the memory file system; if the number is less than 5 times, grub.cfg is restored, the OS is restarted, and the memory boot method is used the next time the server starts.

[0101] Embodiment 3:

[0102] In the method for AI server operating system fault detection and recovery provided in Example 1, each step can be organized into multiple functional modules provided in this embodiment for implementation. It can be understood that the specific implementation methods provided in this embodiment are only used to illustrate the specific implementation process of the method in Example 1 in certain specific scenarios, and are not intended to limit the scope of protection.

[0103] The functional modules used in this embodiment are: BIOS module, hard disk monitoring module, memory monitoring module, system repair module, monitoring management module and repair script. Figure 5 , the functional modules used in this embodiment, and the message interactions between the functional modules. For ease of explanation, the messages interacted between different modules in this embodiment are the same as those in Example 2.

[0104] (1)BIOS module.

[0105] like Figure 6 As shown, the BIOS completes the hardware device initialization in the server, and before loading the OS, it sends a 0xAA message to the monitoring and management module, and then requests the monitoring and management module whether the OS has been successfully started. If 0xAA is received, it indicates that the OS has been successfully started, and the BIOS continues to execute according to the original process and uses the hard disk boot method to start the operating system. If 0xBB is received, it indicates that the OS has failed to start. At this time, the BIOS backs up the current operating system boot configuration file grub.cfg, and selects the boot configuration file pre-stored in the memory file system initramfs.tar.gz.xz to start, and guides the operating system to start from the memory file system.

[0106] (2) Hard disk monitoring module.

[0107] like Figure 7As shown, the root file in the hard disk file system is loaded normally, and the hard disk monitoring module sends 0x55 to the monitoring management module in a period of 1 minute, and receives 0x55 sent by the monitoring management module at the same time. If the hard disk monitoring module receives 0x55 sent by the monitoring management module, it sends a heartbeat confirmation packet of 0x88 to the monitoring management module, indicating that the hard disk file system has been entered normally.

[0108] (3)Memory monitoring module.

[0109] like Figure 8 As shown in the figure, the root file in the memory file system is loaded normally. The memory monitoring module sends 0x55 to the monitoring management module in a one-minute cycle, and receives 0x55 sent by the monitoring management module at the same time. If the memory monitoring module receives 0x55 sent by the monitoring management module, it sends 0x99 to the monitoring management module, indicating that the memory file system has been entered normally. At the same time, the memory monitoring module sends a 0xCC command to the monitoring management module to obtain the number of failed startups in the memory boot mode, for error checking of the memory file system, and for the repair script to determine whether to stay in the memory file system.

[0110] (4) System repair module.

[0111] like Fig. 9 As shown in the figure, the root file system in the memory file system is loaded normally, and the repair script is executed. If the repair is successful, the memory monitoring module obtains the number of failed startups in the memory boot mode. If it is greater than or equal to 5 times, configure the network port IP address and stay in the memory file system; otherwise, restore the grub.cfg in the hard disk file system, and then restart the system. The system will start normally from the hard disk root file system next time. If the repair fails, restart the OS directly.

[0112] (5) Monitoring and management module.

[0113] like Fig.10As shown, the monitoring and management module starts normally, executes the interactive process with the BIOS, hard disk monitoring module and memory monitoring module, and monitors the messages sent to the monitoring and management module by the BIOS, hard disk monitoring module and memory monitoring module. If 0xAA sent by the BIOS is monitored, the value of osbootflag is determined according to whether the heartbeat packet message sent by the hard disk monitoring module is received. If no heartbeat is received, osbootflag is set to 1. If the variable osbootflag is 1, the monitoring and management module sends 0xBB to the hard disk monitoring module and restarts the OS, so that it will start from the memory file system next time and perform kernel and root file system error checking and repair. If osbootflag is 0, if the monitoring and management module monitors the message sent by the hard disk monitoring module as 0x88, the monitoring and management module sets osbootdiskflag to 1; if the monitoring and management module monitors the message sent by the memory monitoring module as 0x99, the monitoring and management module sets osbootramflag to 1. At the same time, the event log of the OS entering the memory file system recorded by the monitoring and management module is available for user query.

[0114] (6) Repair the script.

[0115] In a specific implementation, the system recovery process can be pre-written into a script file, and the error checking and recovery process of the kernel or hard disk file system can be controlled by the script.

[0116] The error checking and recovery logic of the kernel driver module is as follows: collect the current kernel log through the Linux dmesg command, check the error information of the kernel log, if some drivers xx1, xx2, xxN, etc. have errors, it indicates that there is a problem with the driver of the current kernel, then modify the / boot / grub.cfg.bak file, add the parameter configuration blacklist=xx1, xx2, xxN in the kernel startup parameters to disable the problematic driver module, thereby repairing the kernel startup abnormality caused by the driver problem during the kernel startup process.

[0117] The logic of hard disk file system error checking and repairing is as follows: first, obtain the root file system type corresponding to the OS in the hard disk file system, call the corresponding fsck tool according to the root file system type to check and repair the disk root file system, and repair the root file system error. If fsck repairs successfully, it returns 0, indicating success, otherwise it returns 1, indicating that the repair failed.

[0118] Embodiment 4:

[0119] Based on the AI ​​server operating system fault detection and recovery methods provided in the above embodiments 1 to 3, the present invention also provides an AI server operating system fault detection and recovery device that can be used to implement the above method, such as Fig.11 , which is a schematic diagram of the device architecture of an embodiment of the present invention. The device for detecting and recovering an AI server operating system fault in this embodiment includes one or more processors 11 and a memory 12. Fig.11 A processor 11 is taken as an example.

[0120] The processor 11 and the memory 12 may be connected via a bus or other means. Fig.11 The example of connecting through bus is taken in the following.

[0121] The memory 12 is a non-volatile computer-readable storage medium as a method for detecting and recovering an AI server operating system fault, and can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the method for detecting and recovering an AI server operating system fault in Embodiments 1 to 3. The processor 11 executes various functional applications and data processing of the device for detecting and recovering an AI server operating system fault by running the non-volatile software programs, instructions and modules stored in the memory 12, that is, implementing the method for detecting and recovering an AI server operating system fault in Embodiments 1 to 3.

[0122] The memory 12 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 12 may optionally include a memory remotely arranged relative to the processor 11, and these remote memories may be connected to the processor 11 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0123] The program instructions / modules are stored in the memory 12, and when executed by one or more processors 11, the method for detecting and recovering the AI ​​server operating system fault in the above-mentioned embodiments 1 to 3 is executed, for example, the above-described Figure 1-Figure 4 The steps shown.

[0124] An embodiment of the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors, such as Fig.11 A processor 11 in the embodiment 1 can enable the above one or more processors to execute the AI ​​server operating system fault detection and recovery method in embodiments 1 to 3, for example, executing the above described Figure 1-Figure 4 The steps shown can also be implemented Figure 5-Figure 10 The various modules and units described.

[0125] It is worth noting that the information interaction, execution process, etc. between the modules and units within the above-mentioned devices and systems are based on the same concept as the processing method embodiment of the present invention. The specific contents can be found in the description of the method embodiment of the present invention and will not be repeated here.

[0126] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0127] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for fault detection and recovery of an AI server operating system, characterized in that: include: The hard disk file system is used as the boot mode when the operating system is started, the monitoring management module exchanges a first heartbeat packet with the hard disk monitoring module in the hard disk file system, and the startup state of the operating system is obtained according to the first heartbeat packet; A second heartbeat packet interaction is established between the monitoring management module and the BIOS. The monitoring management module feeds back the startup status to the BIOS through the second heartbeat packet. When the startup fails, the BIOS switches the boot mode to the memory file system and starts the operating system through the system root file pre-stored in the memory file system.

2. The method for detecting and recovering an AI server operating system fault according to claim 1, characterized in that: The monitoring management module exchanges a first heartbeat packet with the hard disk monitoring module in the hard disk file system, and obtains the startup state of the operating system according to the first heartbeat packet, specifically including: The monitoring management module and the hard disk monitoring module periodically exchange the first heartbeat packet. When the first heartbeat packet interaction is normal, the hard disk monitoring module starts normally and then feeds back the first success identification packet to the monitoring management module. When the monitoring management module receives a specified number of first success identification packets within a specified waiting time, it indicates that the startup state of the operating system is successful; When the monitoring management module does not receive the first success identification packet within the specified waiting time, it indicates that the startup status of the operating system is startup failure.

3. The method for AI server operating system fault detection and recovery according to claim 1, characterized in that: The monitoring management module feeds back the startup status to the BIOS through the second heartbeat packet, specifically including: The monitoring management module and the BIOS periodically exchange a second heartbeat packet. When the second heartbeat packet interaction is normal, the BIOS starts normally and sends a second request packet to the monitoring management module, and requests the monitoring management module for the startup status of the operating system through the second request packet. When the startup status is successful, the monitoring management module returns a second success identification packet to the BIOS; When the startup status is startup failure, the monitoring management module returns a second failure identification packet to the BIOS.

4. The method for AI server operating system fault detection and recovery according to claim 3, characterized in that: The monitoring management module feeds back the startup status to the BIOS via the second heartbeat packet, and further includes: When the operating system fails to start, the monitoring management module controls the server to re-power on, and the re-power on triggers the BIOS to re-send the second request packet to the monitoring management module.

5. The method for detecting and recovering an AI server operating system fault according to claim 1, characterized in that: The BIOS switches the boot mode to the memory file system, specifically including: The BIOS changes the boot item to the memory file system and starts the operating system through the system root file pre-stored in the memory file system.

6. The method for AI server operating system fault detection and recovery according to claim 1, characterized in that: After the operating system is started by using the system root file pre-stored in the memory file system, the method further includes: The monitoring management module and the memory monitoring module periodically exchange a third heartbeat packet. When the third heartbeat packet interaction is normal, the memory monitoring module starts normally and then feeds back a third success identification packet to the monitoring management module. When the monitoring management module receives the third success identification packet sent by the memory monitoring module, it indicates that the startup is successful; When the monitoring management module does not receive the third success identification packet sent by the memory monitoring module within the specified time period, it indicates that the startup fails.

7. The method for AI server operating system fault detection and recovery according to claim 6, characterized in that: When the startup status is successful, the operating system is started by using the system root file pre-stored in the memory file system, and further includes: The BIOS backs up the boot files and system root files in the hard disk file system, and copies the system root files pre-stored in the memory file system to the hard disk file system. At the next startup, the system root files that were successfully started the previous time are used to start the system.

8. The method for AI server operating system fault detection and recovery according to claim 6, characterized in that: When the startup state is startup failure, the operating system is started by using the root file pre-stored in the memory file system, and further includes: Each time the monitoring management module receives the third failure identification packet, it records the number of startup failures; The memory monitoring module sends a fourth request packet to the monitoring management module, and the monitoring management module feeds back the number of startup failures to the memory monitoring module through a fourth reply packet; When the number of startup failures exceeds the specified number, no restart attempt is made and the server's IP configuration is restored for remote troubleshooting.

9. A device for detecting and recovering faults in an AI server operating system, characterized in that: The method comprises at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor, and after the instructions are executed by the processor, they are used to complete the method for AI server operating system fault detection and recovery according to any one of claims 1 to 8.

10. A non-volatile computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, which are executed by one or more processors to complete the method for AI server operating system fault detection and recovery as described in any one of claims 1-8.

Images