Fault processing method and device, electronic device, and computer-readable storage medium

Abstract

The disclosure provides a fault processing method and device, electronic equipment and computer readable storage medium, relates to the technical field of data processing, and particularly relates to the technical field of cloud computing, virtual machines, computer hardware faults and the like. The specific implementation scheme is as follows: memory fault information reported by a memory hardware that has occurred a fault is acquired, the memory fault information comprises a fault host computer physical address of the memory hardware that has occurred a fault in a host computer; a user state process corresponding to the fault host computer physical address is determined through reverse mapping according to the fault host computer physical address; in the case that the user state process is a virtual machine simulator process, a fault host computer virtual address corresponding to the fault host computer physical address is acquired according to address information of a virtual memory region; and the fault host computer virtual address is sent to the user state process, so that the user state process instructs a fault virtual machine to run in the host computer according to the fault host computer virtual address.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and more particularly to the fields of cloud computing, virtual machines, and computer hardware failure. Specifically, embodiments of this disclosure relate to a fault handling method and apparatus, an electronic device, and a computer-readable storage medium. Background Technology

[0002] In the field of cloud computing, virtualization technology is used to divide large servers into multiple small virtual machines for use by multiple customers.

[0003] These small virtual machines may contain VFIO (a device passthrough technology for Linux operating systems). In order to support DMA (Direct Memory Access) of VFIO devices within the virtual machine, it is necessary to pin (lock) the HPA (Host Physical Address) memory corresponding to the GPA (Guest Physical Address) participating in DMA transfer within the virtual machine to prevent this memory from being swapped to the swap partition by the host machine or moved due to memory consolidation. Summary of the Invention

[0004] This disclosure provides a fault handling method and apparatus, an electronic device, and a computer-readable storage medium.

[0005] According to a first aspect of this disclosure, a fault handling method is provided, the method comprising:

[0006] Obtain memory fault information reported by the faulty memory hardware, the memory fault information including the faulty host physical address of the faulty memory hardware on the host machine;

[0007] Based on the physical address of the faulty host machine, the user-mode process corresponding to the physical address of the faulty host machine is determined through reverse mapping;

[0008] If the user-mode process is a virtual machine emulator process, the virtual address of the faulty host corresponding to the physical address of the faulty host is obtained based on the address information of the virtual memory region.

[0009] The faulty host virtual address is sent to the user-mode process so that the user-mode process can use the faulty host virtual address to instruct the faulty virtual machine running on the host to isolate the memory hardware.

[0010] According to a second aspect of this disclosure, a fault handling apparatus is provided, the apparatus comprising:

[0011] The information collection module is used to obtain memory fault information reported by the faulty memory hardware, including the faulty host physical address of the faulty memory hardware on the host machine.

[0012] The reverse mapping module is used to determine the user-mode process corresponding to the physical address of the faulty host through reverse mapping; and when the user-mode process is a virtual machine emulator process, it obtains the virtual address of the faulty host corresponding to the physical address of the faulty host based on the address information of the virtual memory region.

[0013] The fault injection module is used to send the faulty host virtual address to the user-mode process, so that the user-mode process can instruct the faulty virtual machine running on the host to isolate the memory hardware according to the faulty host virtual address.

[0014] According to a third aspect of this disclosure, an electronic device is provided, the electronic device comprising:

[0015] At least one processor; and

[0016] A memory communicatively connected to at least one of the aforementioned processors; wherein,

[0017] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable at least one processor to perform the fault handling method.

[0018] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the above-described fault handling method.

[0019] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described fault handling method.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0021] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0022] Figure 1 This is a schematic flowchart of a fault handling method provided in an embodiment of this disclosure;

[0023] Figure 2This invention discloses an overall architecture diagram of the host machine and virtual machine in an embodiment of the present disclosure;

[0024] Figure 3 This is a flowchart illustrating some steps of another fault handling method provided in this embodiment of the present disclosure;

[0025] Figure 4 This is a flowchart illustrating some steps of another fault handling method provided in this embodiment of the present disclosure;

[0026] Figure 5 This is a flowchart illustrating some steps of another fault handling method provided in this embodiment of the present disclosure;

[0027] Figure 6 This is a flowchart illustrating some steps of another fault handling method provided in this embodiment of the present disclosure;

[0028] Figure 7 This is a flowchart illustrating some steps of another fault handling method provided in this embodiment of the present disclosure;

[0029] Figure 8 This is a schematic diagram of the structure of a fault handling device provided in an embodiment of this disclosure;

[0030] Figure 9 This is a block diagram of an electronic device used to implement the fault handling method of the embodiments of this disclosure. Detailed Implementation

[0031] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0032] In some related technologies, QEMU (an emulator whose source code is distributed under the GPL license) is used as a virtual machine emulator.

[0033] When QEMU starts a virtual machine, it does not know which GPAs the virtual machine will use to participate in the virtual machine's DMA transfer. Therefore, QEMU will pin all the memory corresponding to the virtual machine. The pinned memory cannot be moved. This means that if these memory fail, the host machine cannot isolate the memory that failed by copying the contents of the memory that failed to the new memory page.

[0034] Therefore, once a memory failure occurs, the only option is to wait for the host machine to crash or for the application within the virtual machine to malfunction.

[0035] The fault handling methods and apparatus, electronic devices, and computer-readable storage media provided in the embodiments of this disclosure are intended to solve at least one of the above-mentioned technical problems of the prior art.

[0036] Figure 1 A flowchart illustrating a fault handling method provided in an embodiment of this disclosure is shown. Figure 1 As shown in the figure, the fault handling method provided in the embodiments of this disclosure may include steps S110, S120, S130 and S140.

[0037] In step S110, memory fault information reported by the faulty memory hardware is obtained. The memory fault information includes the faulty host physical address of the faulty memory hardware on the host machine.

[0038] In step S120, the user-mode process corresponding to the physical address of the faulty host is determined by reverse mapping based on the physical address of the faulty host.

[0039] In step S130, if the user-mode process is a virtual machine emulator process, the virtual address of the faulty host corresponding to the physical address of the faulty host is obtained based on the address information of the virtual memory region.

[0040] In step S140, the virtual address of the faulty host machine is sent to the user-mode process so that the user-mode process can instruct the faulty virtual machine isolation memory hardware running on the host machine according to the virtual address of the faulty host machine.

[0041] For example, a host machine refers to a physical computer or server that runs a hypervisor. It can manage and allocate physical resources, such as CPU (Central Processing Unit), memory, storage, and network connections, for virtual machines to use. The host machine creates and runs virtual machines on its own, providing an isolated execution environment for each virtual machine.

[0042] Figure 2 The overall architecture diagram of the host machine and virtual machine in this embodiment of the present disclosure is shown, such as... Figure 2 As shown, when the host machine's memory hardware experiences a failure, the memory hardware will report the failure event to the host's MCE (Machine Check Exception) driver by sending memory failure information. The memory failure information will carry the HPA (Host Physical Address) of the faulty memory hardware on the host machine, i.e., the faulty HPA.

[0043] Other modules on the host machine can register callback functions with the MCE driver, and the MCE driver will call each callback function in turn to handle the fault event.

[0044] The fault handling method provided in this disclosure can be executed by a module on the host machine, or by an electronic device such as a terminal device or server controlling the module on the host machine to execute it. Specifically, a callback function implementing the fault handling method provided in this disclosure can be registered with the MCE driver on the host machine. This allows the MCE driver to call the registered callback function after receiving memory fault information reported by the memory hardware, thereby running the fault handling method provided in this disclosure on the host machine.

[0045] In some possible implementations, in step S110, such as Figure 2 As shown, after receiving memory fault information reported by the faulty memory hardware, the MCE driver will call the callback function and send the memory fault information to the callback function.

[0046] In some possible implementations, this can also be achieved by registering a callback function with the MCE.

[0047] In some possible implementations, in step S120, such as Figure 2 As shown, the callback function can call the rmap (reverse mapping) function of the host operating system kernel to find the user-space process corresponding to the faulty HPA.

[0048] User mode is a run level in an operating system, primarily used for running user programs. The user-mode process corresponding to a faulty HPA is the process on the host machine that uses the faulty HPA.

[0049] In some operating system kernels (such as Linux), both processes and threads perform tasks, which are managed by a unified data structure called task_struct. task_struct contains various attributes of a process, such as process PID (Process ID, process controller), state, priority, scheduling information, etc., and is used to manage the process's lifecycle, scheduling, memory management, file system operations, etc.

[0050] The `comm` field of `task_struct` stores the name of the process. The name of the user-space process can be obtained through the `comm` field of the `task_struct` of the user-space process corresponding to the faulty HPA.

[0051] The user program corresponding to a user-mode process can be identified by its name.

[0052] If the user-mode process corresponding to the faulty HPA is not the virtual machine emulator process, it means that the user program using the faulty HPA is a user program other than the virtual machine emulator.

[0053] If the user-mode process corresponding to the faulty HPA is a virtual machine emulator process, it means that the user program currently using the faulty HPA is a virtual machine emulator. In other words, a virtual machine running on the host machine is using the faulty HPA. If the faulty memory hardware is not isolated, the application in the virtual machine may fail, or even cause the host machine to crash.

[0054] Therefore, when the user-mode process corresponding to the faulty HPA is a virtual machine emulator process, the HVA (Host Virtual Address) corresponding to the faulty HPA is calculated through the address information in the VMA (virtual memory area) area in rmap, which is the faulty HVA.

[0055] The host machine uses VMAs to manage the virtual memory space. Each process's virtual memory is composed of multiple VMAs. Therefore, the faulty HVA corresponding to a faulty HPA can be obtained through the VMAs.

[0056] In some possible implementations, QEMU (an emulated processor whose source code is distributed under the GPL license) is used as a virtual machine emulator; therefore, if a user-mode process is a virtual machine emulator process, it is also a QEMU process.

[0057] In some possible implementations, in step S130, the faulty HVA is sent to the user-space process (which is also the virtual machine emulator process) obtained through rmap via the send_sig_mceerr method.

[0058] like Figure 2 As shown, after the virtual machine emulator process obtains the faulty HVA, it looks up the GPA (Guest Physical Address) mapped to the faulty HVA, i.e., the faulty GPA. It then fills the virtual MCE register of the faulty virtual machine corresponding to the virtual machine emulator process with the faulty GPA, and then injects an MCE interrupt or CMCI (Corrected Machine Check Interrupt) interrupt into the faulty virtual machine to pass the faulty GPA to the faulty virtual machine.

[0059] After receiving an MCE or CMC interrupt, the faulty virtual machine uses the Soft Offline or Hard Offline function of memory to isolate the memory corresponding to the faulty GPA.

[0060] In the fault handling method provided in this embodiment, the fault HPA of the faulty memory hardware is used to obtain the virtual machine emulator process and the corresponding fault HVA corresponding to the faulty HPA. The memory fault is injected into the virtual machine by sending the faulty HVA to the virtual machine emulator process. The virtual machine then isolates the faulty memory hardware, preventing all memory corresponding to the virtual machine from being pinned, thus preventing the virtual machine from accessing the faulty memory hardware, which could lead to errors in the application within the virtual machine or even cause the host machine to crash, thereby reducing the downtime rate of the host machine.

[0061] The fault handling method provided in the embodiments of this disclosure will be described in detail below.

[0062] In some possible implementations, the memory hardware will report the fault event to the MCE driver in the host by sending memory fault information. In addition to carrying the HPA of the faulty memory hardware in the host machine, i.e. the faulty HPA, the memory fault level will also be carried in the memory fault information.

[0063] In some possible implementations, the level of memory fault can include CE, AO, and AR.

[0064] Here, CE stands for Correctable Error, AO stands for Action Optional, and AR stands for Action Required, indicating that action must be taken to resolve the fault.

[0065] In some possible implementations, a signal indicating the level of the memory fault is sent along with the fault HVA to the virtual machine emulator process. Specifically, it could be the BUS_MCEERR_AO signal, indicating that the memory fault is one that can be handled; it could be the BUS_MCEERR_AR signal, indicating that the memory fault is one that must be handled; or it could be the BUS_MCEERR_CE signal, indicating that the memory fault is a correctable error.

[0066] After acquiring the fault HVA and the signal indicating the level of memory fault, the virtual machine emulator process looks up the fault GPA mapped by the fault HVA, fills the virtual MCE register of the faulty virtual machine with the fault GPA and the memory fault level, and then injects an MCE interrupt or CMCI interrupt into the faulty virtual machine to pass the fault GPA and the memory fault level to the faulty virtual machine.

[0067] When a faulty virtual machine receives an MCE or CMC interrupt, and the operating system is a CentOS image, the mcelog user-space daemon within the virtual machine is used to determine the level of memory fault.

[0068] In the event of a memory fault level of AO or AR, immediately invoke the virtual machine's Hard Offline function to isolate the memory that has failed and kill the virtual machine process that is using the faulty memory hardware.

[0069] When the memory fault level is CE, the number of received CE memory faults is counted. If the number of CE memory faults reaches a preset threshold, the virtual machine's Soft Offline function is invoked to isolate the memory that has experienced a memory fault.

[0070] If the host machine has a memory UE (Uncorrectable Error) fault prediction function, it can predict the risk of CE becoming UE within the host machine, and then send the fault GPA corresponding to the CE memory fault with the risk of becoming UE to the virtual machine. After receiving it, the virtual machine can set the preset threshold to 1, that is, when it receives 1 CE memory fault with the risk of becoming UE, it will isolate the memory that has the memory fault.

[0071] After receiving an MCE or CMC interrupt, the faulty virtual machine uses the CEC kernel-mode driver within the virtual machine to determine the level of memory fault, in the case of the virtual machine's operating system (Ubuntu series images).

[0072] In the event of a memory fault level of AO or AR, immediately invoke the virtual machine's Hard Offline function to isolate the memory that has failed and kill the virtual machine process that is using the faulty memory hardware.

[0073] When the memory fault level is CE, the number of received CE memory faults is counted. If the number of CE memory faults reaches a preset threshold, the virtual machine's Soft Offline function is invoked to isolate the memory that has experienced a memory fault.

[0074] If the host machine has a memory UE failure prediction function, it can predict the risk of CE becoming UE within the host machine, and then send the failure GPA corresponding to the CE memory failure with the risk of becoming UE failure to the virtual machine. After receiving it, the virtual machine can set the preset threshold to 1, that is, when it receives 1 CE memory failure with the risk of becoming UE failure, it will isolate the memory that has failed.

[0075] By sending the memory fault level to the virtual machine emulator process, the virtual machine can adopt different handling methods for memory faults of different fault levels, thereby minimizing the use of memory isolation and reducing its impact on virtual machine operation.

[0076] As mentioned above, in some possible implementations, virtual machines running on the host machine may be subjected to various operations such as cold reboot, hot reboot, deletion, creation, and hot migration.

[0077] After a virtual machine undergoes cold restart, hot restart, deletion, creation, or hot migration, the isolation it provides for the memory hardware will disappear. To prevent the virtual machine from still accessing the memory hardware that has experienced memory failure after these operations, which could lead to errors in the application within the virtual machine or even crashes of the host machine, it is necessary to save the acquired memory failure information so that the virtual machine can still isolate the memory hardware that has experienced memory failure after these operations.

[0078] The acquired memory fault information can be saved as historical fault information. This historical fault information will not be deleted due to cold restart, hot restart, deletion, creation, or hot migration of the virtual machine. After cold restart, hot restart, deletion, creation, or hot migration, the virtual machine can isolate the memory hardware that has experienced a memory fault based on this historical fault information.

[0079] In some possible implementations, the physical address of the faulty host, the identifier of the faulty process, and the correspondence between the physical address of the faulty host and the identifier of the faulty process are saved as historical fault information.

[0080] Among them, the fault process identifier is a user-mode process, that is, the process control character of the virtual machine emulator process corresponding to the fault HPA obtained through reverse mapping.

[0081] Specifically, while sending the fault HVA to the virtual machine emulator process, the fault HPA and the corresponding virtual machine emulator process are written into the fault information file as historical fault information.

[0082] Each line of the fault information file contains historical fault information, including two fields: HPA and PID. The faulty HPA is written to the HPA field, and the faulty process identifier is written to the PID field.

[0083] Each time the fault handling method corresponding to the embodiment of this disclosure is executed, the acquired fault HPA and the corresponding virtual machine simulator process need to be written into the fault information file as historical fault information.

[0084] In some possible implementations, by being aware of operations performed on the virtual machine, historical fault information can be modified or the faulty memory hardware can be re-isolated based on these operations.

[0085] Figure 3The diagram illustrates a flowchart of an implementation that modifies historical fault information or re-isolates faulty memory hardware based on operations performed on the virtual machine. Figure 2 As shown, modifying historical fault information or re-isolating faulty memory hardware based on operations performed on the virtual machine may include steps S310, S320, and S330.

[0086] In step S310, virtual machine operations for one or more virtual machines running on the host machine and the operation identifiers corresponding to the virtual machine operations are obtained.

[0087] In step S320, a search is performed in the historical fault information based on the virtual machine operation and operation identifier;

[0088] In step S330, based on the search results, the historical fault information is modified and / or the virtual machine isolation memory hardware that indicates the virtual machine operation action is indicated by the operation identifier;

[0089] Among them, the operation identifier is the process control character of the virtual machine emulator process corresponding to the virtual machine whose operation is performed.

[0090] In some possible implementations, in step S310, a character device can be created, and the created character device can be opened in Libvirt to realize the operation on the virtual machine and obtain the operation identifier corresponding to the virtual machine operation.

[0091] Libvirt is an open-source toolset for managing virtualization platforms. It provides a unified interface for managing virtual machines, so Libvirt can be used to be aware of operations performed on virtual machines.

[0092] The operation identifier is the PID of the virtual machine emulator process corresponding to the virtual machine whose operation is performed.

[0093] For example, in the case of a virtual machine hot restart, since the virtual machine emulator process corresponding to the virtual machine does not change before and after the hot restart, the operation identifier is the PID of the virtual machine emulator process corresponding to the hot-restarted virtual machine.

[0094] When a virtual machine is deleted, the operation identifier is the PID of the virtual machine emulator process corresponding to the deleted virtual machine.

[0095] The operation identifier for creating a virtual machine is the PID of the virtual machine emulator process corresponding to the created virtual machine.

[0096] Virtual machine migration is equivalent to deleting the virtual machine from the host machine, while virtual machine cold restart is equivalent to first deleting the virtual machine and then recreating it.

[0097] In some possible implementations, in step S320, a retrieval is performed based on the PID field of the operation identifier in the fault information file.

[0098] In some possible implementations, in step S330, based on the search results in the PID field, if the virtual machine emulator process of the virtual machine corresponding to the faulty HPA changes, the historical fault information is modified. If the new virtual machine uses the faulty memory hardware, the memory hardware needs to be re-isolated.

[0099] The following section provides a detailed introduction to the processing of historical fault information for different virtual machine operations.

[0100] Figure 4 This diagram illustrates a flowchart of one implementation method for processing historical fault information or isolating memory hardware when the virtual machine operation is a hot restart. Figure 4 As shown, it may include steps S410 and S420.

[0101] In step S410, if the virtual machine operation is a hot restart, a search is performed in the historical fault information based on the operation identifier;

[0102] In step S420, if a fault process identifier that matches the operation identifier is retrieved from the historical fault information, the fault host virtual address corresponding to the fault host physical address is obtained, and the fault host virtual address is sent to the virtual machine emulator process corresponding to the operation identifier.

[0103] In some possible implementations, in step S410, when the virtual machine operation is a hot restart, since the virtual machine emulator process corresponding to the virtual machine will not change before and after the hot restart, and the memory corresponding to the virtual machine will not change, the historical fault information in the fault information file does not need to be changed.

[0104] However, after a virtual machine is hot-rebooted, the memory isolation performed in the virtual machine will also disappear, so it is necessary to re-isolate the faulty memory hardware.

[0105] The system searches historical fault information in the operation identification fault information file to determine whether the virtual machine isolated the faulty memory hardware before the hot restart. In other words, the memory corresponding to the virtual machine before the hot restart included the faulty memory hardware.

[0106] In some possible implementations, in step S420, if a fault process identifier that matches the operation identifier is retrieved from the historical fault information, it indicates whether the virtual machine isolated the faulty memory hardware before the hot restart. In other words, the memory corresponding to the virtual machine before the hot restart included the faulty memory hardware. Therefore, it is necessary to re-isolate the faulty memory hardware.

[0107] Since the fault HPA is stored in the historical fault information, it is only necessary to obtain the fault HVA corresponding to the fault HPA using the method described above, send the fault HVA to the virtual machine emulator process corresponding to the operation identifier, and perform memory isolation using the method described above. That is, the virtual machine emulator process corresponding to the operation identifier looks up the fault GPA mapped by the fault HVA through the fault HVA, fills the virtual MCE register of the fault virtual machine corresponding to the virtual machine emulator process with the fault GPA, and then injects an MCE interrupt or CMCI interrupt into the fault virtual machine to pass the fault GPA to the fault virtual machine. After receiving the MCE interrupt or CMC interrupt, the fault virtual machine uses the soft offline function of memory to isolate the memory corresponding to the fault GPA.

[0108] Figure 5 The diagram illustrates a flowchart of one implementation method for processing historical fault information or isolating memory hardware when the virtual machine operation is to delete the virtual machine. Figure 5 As shown, it may include steps S510 and S520.

[0109] In step S510, if the virtual machine operation is to delete the virtual machine, a search is performed in the historical fault information based on the operation identifier;

[0110] In step S520, if a fault process identifier that matches the operation identifier is found in the historical fault information, the fault process identifier that matches the operation identifier in the historical fault information is modified to an invalid value.

[0111] In some possible implementations, in steps S510 and S520, when the virtual machine operation is to delete the virtual machine, since the virtual machine is deleted, the virtual machine emulator process corresponding to the virtual machine will naturally disappear as well. Therefore, it is necessary to delete the fault process identifier that is consistent with the operation identifier in the fault information file, so as to prevent other processes from being assigned the operation identifier and causing the process to be incorrectly recorded as historical fault information.

[0112] However, since the memory hardware failure still exists, the faulty HPA in this historical fault information cannot be deleted to prevent other processes from using the faulty HPA. Therefore, it is only necessary to modify the faulty process identifier in the historical fault information that matches the operation identifier to an invalid value, such as -1, so as to retain the faulty HPA while breaking the correspondence between the PID of the virtual machine emulator process of the virtual machine to be deleted and the faulty HPA.

[0113] Figure 5 The diagram illustrates a flowchart of one implementation method for processing historical fault information or isolating memory hardware when the virtual machine operation is to create a virtual machine. Figure 6 As shown, it may include steps S610, S620, and S630.

[0114] In step S610, when the virtual machine operation is to create a virtual machine, the physical address of the faulty host machine corresponding to the faulty process identifier with invalid value in the historical fault information is obtained;

[0115] In step S620, the fault process control character of the user-mode process corresponding to the retrieved faulty host physical address is determined by reverse mapping.

[0116] In step S630, if the fault process control character and the operation identifier are consistent, the fault process identifier corresponding to the fault host physical address retrieved from the historical fault information is modified to the operation identifier.

[0117] In some possible implementations, in step S610, if the virtual machine operation is to create a virtual machine, all rows in the fault information file with invalid PID fields are searched for to obtain the fault HPA of the HPA field of these rows, thereby obtaining all unused faulty memory hardware.

[0118] In some possible implementations, in step S620, the acquired faulty HPA is reverse-mapped in the manner described above to obtain the user-mode process corresponding to the faulty HPA, that is, the process of the application that uses the faulty memory hardware corresponding to the faulty HPA.

[0119] In some possible implementations, in step S630, if the user-mode process matches the operation identifier, it indicates that the memory corresponding to the created virtual machine includes the faulty memory hardware. It is necessary to obtain the faulty HVA corresponding to the faulty HPA using the method described above, send the faulty HVA to the virtual machine emulator process corresponding to the operation identifier, and perform memory isolation using the method described above. That is, the virtual machine emulator process corresponding to the operation identifier finds the faulty GPA mapped by the faulty HVA through the faulty HVA, fills the virtual MCE register of the faulty virtual machine corresponding to the virtual machine emulator process with the faulty GPA, and then injects an MCE interrupt or CMCI interrupt into the faulty virtual machine to pass the faulty GPA to the faulty virtual machine. After receiving the MCE interrupt or CMC interrupt, the faulty virtual machine uses the soft offline function of memory to perform memory isolation on the memory corresponding to the faulty GPA.

[0120] At the same time, since a virtual machine has re-pinned the faulty memory hardware, it is necessary to update the historical fault information, that is, to write the PID of the virtual machine emulator process corresponding to the created virtual machine into the PID of the line where the faulty HPA is located.

[0121] In some possible implementations, virtual machine migration is equivalent to deleting the virtual machine from the host machine. Therefore, virtual machine migration only needs to follow... Figure 5 The method described above can be used to process it.

[0122] A virtual machine cold restart is equivalent to first deleting the virtual machine and then recreating it. Therefore, a virtual machine cold restart only requires following these steps: Figure 6 and Figure 5 The methods described above can be used for processing. In particular, after a cold restart, the PID of the virtual machine emulator process corresponding to the virtual machine is the same as the PID of the virtual machine emulator process that created the virtual machine.

[0123] In some possible implementations, if the host machine restarts, the PIDs of all virtual machine emulator processes will change. Therefore, it is necessary to modify all historical faults in the fault information file and re-implement memory isolation.

[0124] Figure 7 The diagram illustrates a process for modifying historical fault information and re-establishing memory isolation after a host machine reboot. Figure 7 As shown, it may include steps S710, S720, and S730.

[0125] In step S710, a host machine restart is detected. Before the virtual machine running on the host machine restarts, the fault process identifier in the historical fault information is modified to an invalid value.

[0126] In step S720, after the virtual machine running on the host machine restarts, the user-mode process corresponding to the physical address of the faulty host machine is determined by reverse mapping based on the physical address of the faulty host machine in the historical fault information.

[0127] In step S730, if the user-mode process is a virtual machine emulator process, the fault process identifier corresponding to the physical address of the faulty host in the historical fault information is modified to the process control character of the user-mode process.

[0128] In some possible implementations, in step S710, when the host machine restarts but the virtual machine running on the host machine does not restart, the fault process identifier in all historical fault information is modified to an invalid value, such as -1, so that the virtual machine emulator process of the virtual machine can be rewritten into the historical fault information after the virtual machine restarts.

[0129] Specifically, you can modify / etc / rc.local to change the fault process identifier in all historical fault information to an invalid value.

[0130] In some possible implementations, in steps S720 and S730, after the virtual machine restarts, since restarting the virtual machine is equivalent to recreating the virtual machine on the host machine, the restarted virtual machine needs to be treated as a newly created virtual machine and processed using the above method.

[0131] Therefore, steps S720 and S730 are essentially repetitions of steps S620 and S630, and will not be elaborated further here.

[0132] If the memory hardware fault is cleared, such as when the host machine replaces the faulty memory module, all historical fault information will become invalid. Therefore, it is necessary to delete the fault information corresponding to the host machine's physical address for the memory hardware fault in order to avoid affecting the operation of the host machine.

[0133] Based on and Figure 1 The method shown follows the same principle. Figure 8 A schematic diagram of the structure of a fault handling device provided in an embodiment of this disclosure is shown, such as... Figure 8 As shown, the fault handling device 80 may include:

[0134] The information collection module 810 is used to obtain memory fault information reported by the faulty memory hardware. The memory fault information includes the physical address of the faulty memory hardware on the host machine.

[0135] The reverse mapping module 820 is used to determine the user-mode process corresponding to the physical address of the faulty host through reverse mapping; when the user-mode process is a virtual machine emulator process, it obtains the virtual address of the faulty host corresponding to the physical address of the faulty host based on the address information of the virtual memory region.

[0136] The fault injection module 830 is used to send the virtual address of the faulty host machine to the user-mode process, so that the user-mode process can indicate the faulty virtual machine isolation memory hardware running on the host machine according to the virtual address of the faulty host machine.

[0137] In the fault handling device provided in this embodiment, the virtual machine emulator process corresponding to the faulty HPA and the corresponding faulty HVA are obtained through the faulty HPA of the faulty memory hardware. The memory fault is injected into the virtual machine by sending the faulty HVA to the virtual machine emulator process. The virtual machine isolates the faulty memory hardware, avoids all memory corresponding to the virtual machine being pinned, prevents the virtual machine from accessing the faulty memory hardware, causing the application in the virtual machine to malfunction, or even causing the host machine to crash, thereby reducing the downtime rate of the host machine.

[0138] It is understood that the above-described modules of the fault handling device in the embodiments of this disclosure have the ability to implement... Figure 1 The embodiments shown illustrate the functions of corresponding steps in the fault handling method. These functions can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned functions. These modules can be software and / or hardware, and each module can be implemented individually or integrated from multiple modules. For a detailed description of the functions of each module in the aforementioned fault handling device, please refer to [link to relevant documentation]. Figure 1 The corresponding descriptions of the fault handling methods in the embodiments shown are not repeated here.

[0139] In some possible implementations, the fault handling device also includes a fault information storage module, which is used to: store the faulty host physical address, the faulty process identifier, and the correspondence between the faulty host physical address and the faulty process identifier as historical fault information; wherein, the faulty process identifier is the process control character of the user-mode process.

[0140] In some possible implementations, the fault handling device further includes a fault information processing module, which includes: an information acquisition unit for acquiring virtual machine operations targeting one or more virtual machines running on the host machine and the corresponding operation identifiers of the virtual machine operations; a retrieval unit for retrieving information from historical fault information based on the virtual machine operations and the operation identifiers; and an information modification unit for modifying historical fault information based on the retrieval results and / or indicating the virtual machine isolation memory hardware acting on the virtual machine operation through the operation identifier; wherein, the operation identifier is the process control character of the virtual machine emulator process corresponding to the virtual machine acting on the virtual machine.

[0141] In some possible implementations, the retrieval unit is used to: in the case of a hot restart of the virtual machine, retrieve information based on the operation identifier in the historical fault information; the information modification unit is used to: in the case of retrieving a fault process identifier that matches the operation identifier from the historical fault information, obtain the fault host virtual address corresponding to the physical address of the fault host, and send the fault host virtual address to the virtual machine emulator process corresponding to the operation identifier.

[0142] In some possible implementations, the retrieval unit is used to: when the virtual machine operation is to delete the virtual machine, retrieve information based on the operation identifier in the historical fault information; the information modification unit is used to: when a fault process identifier that matches the operation identifier is retrieved from the historical fault information, modify the fault process identifier that matches the operation identifier in the historical fault information to an invalid value.

[0143] In some possible implementations, the retrieval unit is used to: obtain the physical address of the faulty host machine corresponding to the faulty process identifier with an invalid value in the historical fault information when the virtual machine operation is to create a virtual machine; the information modification unit is used to: determine the faulty process control character of the user-mode process corresponding to the retrieved faulty host machine physical address through reverse mapping based on the retrieved faulty host machine physical address; if the faulty process control character is consistent with the operation identifier, modify the faulty process identifier corresponding to the retrieved faulty host machine physical address in the historical fault information to the operation identifier.

[0144] In some possible implementations, the fault handling device also includes a host restart module, used to: detect a host restart, modify the fault process identifier in the historical fault information to an invalid value; determine the user-mode process corresponding to the fault host physical address through reverse mapping based on the fault host physical address in the historical fault information; and, if the user-mode process is a virtual machine emulator process, modify the fault process identifier corresponding to the fault host physical address in the historical fault information to the process control character of the user-mode process.

[0145] In some possible implementations, the fault handling device also includes a fault clearing module, which is used to: delete the historical fault information of the memory hardware corresponding to the faulty host physical address on the host machine when the memory hardware fault is cleared.

[0146] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0147] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0148] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0149] The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a fault handling method as provided in the embodiments of this disclosure.

[0150] Compared with existing technologies, this electronic device obtains the virtual machine emulator process and the corresponding faulty HVA corresponding to the faulty memory hardware through the faulty HPA of the faulty memory hardware. By sending the faulty HVA to the virtual machine emulator process, the memory fault is injected into the virtual machine, which then isolates the faulty memory hardware. This prevents all memory corresponding to the virtual machine from being pinned, thus preventing the virtual machine from accessing the faulty memory hardware, which could lead to errors in the application within the virtual machine or even cause the host machine to crash, thereby reducing the downtime rate of the host machine.

[0151] The readable storage medium is a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform the fault handling method provided in the embodiments of this disclosure.

[0152] Compared with existing technologies, this readable storage medium obtains the virtual machine emulator process and the corresponding faulty HVA corresponding to the faulty memory hardware through the faulty HPA of the faulty memory hardware. By sending the faulty HVA to the virtual machine emulator process, the memory fault is injected into the virtual machine, which then isolates the faulty memory hardware. This prevents all memory corresponding to the virtual machine from being pinned, thus preventing the virtual machine from accessing the faulty memory hardware, which could lead to errors in the application within the virtual machine or even cause the host machine to crash, thereby reducing the downtime rate of the host machine.

[0153] The computer program product includes a computer program that, when executed by a processor, implements the fault handling method provided in the embodiments of this disclosure.

[0154] Compared with existing technologies, this computer program product obtains the virtual machine emulator process and the corresponding faulty HVA corresponding to the faulty memory hardware through the faulty HPA of the faulty memory hardware. By sending the faulty HVA to the virtual machine emulator process, the memory fault is injected into the virtual machine, which then isolates the faulty memory hardware. This prevents all memory corresponding to the virtual machine from being pinned, thus preventing the virtual machine from accessing the faulty memory hardware, which could lead to errors in the application within the virtual machine or even cause the host machine to crash, thereby reducing the downtime rate of the host machine.

[0155] Figure 9 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0156] like Figure 9 As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0157] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0158] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as fault handling methods. For example, in some embodiments, the fault handling method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the fault handling method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform fault handling methods by any other suitable means (e.g., by means of firmware).

[0159] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0160] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0161] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0162] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0163] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0164] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0165] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0166] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A fault handling method, comprising: The system obtains memory fault information reported by the faulty memory hardware to the host machine for checking abnormal drivers. The memory fault information includes the physical address of the faulty memory hardware on the host machine and the level of memory fault. The memory fault levels include correctable errors, faults that can be addressed, and faults that require intervention. The reverse mapping function of the host operating system kernel is invoked to determine the user-mode process corresponding to the physical address of the faulty host based on the physical address of the faulty host. If the user-mode process is a virtual machine emulator process, the virtual address of the faulty host corresponding to the physical address of the faulty host is obtained based on the address information of the virtual memory region. The faulty host virtual address is sent to the user-mode process so that the user-mode process can run the faulty virtual machine on the host according to the faulty host virtual address and the memory fault level indication; The faulty virtual machine is configured to: determine the level of the received memory fault using a corresponding fault judgment component or daemon process based on the type of the operating system of the faulty virtual machine, and isolate the memory hardware in a differentiated manner based on the judgment result; The differentiated isolation of the memory hardware includes: when the memory fault is classified as a fault that can be handled or a fault that must be handled, the hardware is isolated by calling the hard offline function of the faulty virtual machine, and the virtual machine process using the faulty memory hardware is killed; when the memory fault is classified as a correctable error, the number of memory faults is accumulated, and when the number of memory faults reaches a preset threshold, the soft offline function of the faulty virtual machine is used to isolate the memory hardware.

2. The method according to claim 1, further comprising: The physical address of the faulty host, the identifier of the faulty process, and the correspondence between the physical address of the faulty host and the identifier of the faulty process are saved as historical fault information. The fault process identifier is the process control character of the user-mode process.

3. The method according to claim 2, further comprising: Obtain virtual machine operations for one or more virtual machines running on the host machine, and the operation identifiers corresponding to the virtual machine operations; Based on the virtual machine operation and the operation identifier, a search is performed in the historical fault information; Based on the search results, the historical fault information is modified and / or the virtual machine is isolated from the memory hardware by indicating the virtual machine operation action through the operation identifier; The operation identifier is the process control character of the virtual machine emulator process corresponding to the virtual machine whose operation is performed by the virtual machine.

4. The method according to claim 3, wherein, The step of retrieving information from historical fault information based on the virtual machine operation and the operation identifier includes: In the case where the virtual machine operation is a hot restart, the historical fault information is retrieved based on the operation identifier; The step of modifying the historical fault information based on the search results and / or isolating the memory hardware of the virtual machine by indicating the virtual machine operation through the operation identifier includes: If a fault process identifier that matches the operation identifier is found in the historical fault information, the fault host virtual address corresponding to the fault host physical address is obtained, and the fault host virtual address is sent to the virtual machine emulator process corresponding to the operation identifier.

5. The method according to claim 3, wherein, The step of retrieving information from historical fault information based on the virtual machine operation and the operation identifier includes: When the virtual machine operation is to delete the virtual machine, the historical fault information is retrieved based on the operation identifier. The step of modifying the historical fault information based on the search results and / or isolating the memory hardware of the virtual machine by indicating the virtual machine operation through the operation identifier includes: If a fault process identifier that matches the operation identifier is found in the historical fault information, the fault process identifier that matches the operation identifier in the historical fault information is modified to an invalid value.

6. The method according to claim 5, wherein, The step of retrieving information from historical fault information based on the virtual machine operation and the operation identifier includes: When the virtual machine operation is to create a virtual machine, obtain the physical address of the faulty host machine corresponding to the faulty process identifier with an invalid value in the historical fault information; The step of modifying the historical fault information based on the search results and / or isolating the memory hardware of the virtual machine by indicating the virtual machine operation through the operation identifier includes: Based on the retrieved physical address of the faulty host machine, the faulty process control character of the user-mode process corresponding to the retrieved physical address of the faulty host machine is determined through reverse mapping. If the fault process control character is consistent with the operation identifier, the fault process identifier corresponding to the fault host physical address retrieved from the historical fault information is modified to the operation identifier.

7. The method according to claim 2, further comprising: If the host machine restarts, the fault process identifier in the historical fault information will be modified to an invalid value before the virtual machine running on the host machine restarts. After the virtual machine running on the host machine restarts, the user-mode process corresponding to the physical address of the faulty host machine is determined by reverse mapping based on the physical address of the faulty host machine in the historical fault information. If the user-mode process is a virtual machine emulator process, the fault process identifier corresponding to the faulty host physical address in the historical fault information is modified to the process control character of the user-mode process.

8. The method according to claim 2, further comprising: If the memory hardware fault is cleared, the fault information corresponding to the faulty physical address of the memory hardware on the host machine will be deleted.

9. A fault handling device, comprising: The information collection module is used to obtain memory fault information reported by the faulty memory hardware to the host machine's machine check abnormal driver. The memory fault information includes the faulty host machine physical address of the faulty memory hardware and the level of memory fault. The memory fault levels include correctable errors, faults that can be addressed, and faults that require intervention. The reverse mapping module is used to call the reverse mapping function of the host operating system kernel to determine the user-mode process corresponding to the physical address of the faulty host based on the physical address of the faulty host; if the user-mode process is a virtual machine emulator process, it obtains the virtual address of the faulty host corresponding to the physical address of the faulty host based on the address information of the virtual memory region. The fault injection module is used to send the faulty host virtual address to the user-mode process, so that the user-mode process can run the faulty virtual machine on the host according to the faulty host virtual address and the level indication of the memory fault. The faulty virtual machine is configured to: determine the level of the received memory fault using a corresponding fault judgment component or daemon process based on the type of the operating system of the faulty virtual machine, and isolate the memory hardware differentially based on the judgment result; specifically, it is configured to: when the level of the memory fault is a fault that can be handled or a fault that must be handled, call the hard offline function of the faulty virtual machine to isolate the memory hardware and kill the virtual machine process that is using the faulty memory hardware; If the memory fault is classified as a correctable error, the number of memory faults is accumulated, and when the number of memory faults reaches a preset threshold, the soft offline function of the faulty virtual machine is invoked to isolate the memory hardware.

10. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

11. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

12. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.

Images