Hardware memory error tolerant software system

Abstract

Systems and methods to implement a hardware memory error-tolerant software system. For example, the system can include a host device that, in response to one or more requests to access a hardware memory, instantiates a kernel agent, determines, by the kernel agent based on received information, whether the request to access the memory will result in accessing a corrupted memory location, and in response to determining that the request will access the corrupted memory location, skips an operation related to the corrupted memory location. The system can also include a system to detect software vulnerabilities to hardware memory errors.

Description

[0001] Cross Reference to Related Applications

[0002] This application is a continuation of U.S. Application No. 17 / 551,767, filed December 15, 2021, the disclosure of which is incorporated herein by reference. BACKGROUND

[0003] Cloud computing has impacted the way enterprise manages computing needs. Cloud computing provides reliability, flexibility, scalability, and redundancy in an economically efficient manner, enabling enterprises to manage their information technology needs without the traditional capital investment and maintenance considerations of hosting hardware themselves. As cloud computing infrastructure grows to meet increasing demand, the impact of this shift is that memory errors occurring in the cloud, if not controlled and / or repaired, can negatively impact customer and user experience and decrease their trust in the infrastructure. For example, uncorrectable memory errors on a host machine can cause the host to abruptly shut down or crash, also causing all hosted virtual machines to abruptly terminate. As memory allocations to different virtual machines rise to the order of terabytes, uncorrectable memory errors can potentially impact thousands of virtual machines or applications, requiring extremely long periods of time to reestablish service. Hardware error recovery capabilities are typically provided via CPU (e.g., machine check architecture). Software error recovery capabilities require different approaches and implementations than those used for hardware systems. SUMMARY

[0004] Aspects of the disclosed technology can include a hardware memory error-tolerant software system, a system for detecting vulnerabilities in a software system, and / or processes related to such systems.

[0005] For example, one aspect of the disclosed technology can include a hardware memory error-tolerant software system, comprising: one or more processing devices; an input to receive information indicative of a detected hardware memory error storage location; and a memory storing instructions that can control operations related to the one or more processing devices. For example, the instructions can cause the one or more processing devices to: in response to one or more requests to access the memory, instantiate a kernel agent that determines, based on the received information, whether the requests to access the memory will cause access to a corrupted memory location; and in response to a determination that the requests will access the corrupted memory location, skip operations related to the corrupted memory location.

[0006] According to this aspect of the disclosed technology, the received information can include information identifying a kernel control path of an operating system of the computing device. Further, the identified kernel control path can include one of a kernel control path that performs housekeeping operations (e.g., periodic or event-driven operations performed by the kernel to maintain the internal state and services of the kernel, such as those related to a Linux kernel, for example) or optimization operations (e.g., CPU, memory, or disk usage).

[0007] According to this aspect of the technology, the request to access the memory can include a request to access a page in the memory. Further, the instructions can cause the one or more processing devices to skip an operation related to the page in response to a determination that the request will access a corrupted memory location. Where the skipped operation can include not running a given operation or referencing a fix-up handler that causes a given instruction to be redirected to a different memory location.

[0008] According to this aspect of the disclosed technology, the information from the hardware memory error detection system can include causing a processor to: instantiate a plurality of virtual machines on a host machine, the plurality of virtual machines being associated with a memory address space for running one or more processes; instantiate a memory injection utility, the memory injection utility injecting one or more hardware memory errors into the memory address space; adjust one of a frequency of injection of the one or more hardware memory errors or an error density level of the one or more hardware memory errors injected into the memory address space; detect one or more memory access errors associated with the plurality of virtual machines based on the one or more hardware memory errors injected into the memory address space, the one or more memory access errors providing an indication of possible malfunction in executing the one or more processes; and associate the detected one or more memory access errors with one or more memory locations associated with the memory address space. Further, according to this aspect of the disclosed technology, the error detection system can include: an analysis utility that aggregates the associated memory access errors to identify memory locations at which the one or more hardware memory errors were detected; and an output that provides the identified memory locations as information for input to a hardware memory error tolerance software system.

[0009] According to this aspect of the disclosed technology, the instructions can cause the one or more devices to randomly adjust at least one memory address. In another example, the instructions can cause the one or more devices to change a number of machines instantiated while changing an error density level. Further, the instructions can cause the one or more devices to change a workload of the instantiated plurality of machines.

[0010] Another aspect of the disclosed technology can include a system for detecting software system vulnerabilities resulting from hardware memory errors, comprising: a host machine having one or more processing devices and a memory storing instructions that cause the one or more processing devices to: instantiate a plurality of virtual machines on the host machine, the plurality of virtual machines being associated with a memory address space for running one or more processes; instantiate a memory injection utility that injects one or more hardware memory errors into the memory address space; adjust one of an injection frequency of the one or more hardware memory errors or an error density level of the one or more hardware memory errors injected into the memory address space; detect one or more memory access errors associated with the plurality of virtual machines based on the one or more hardware memory errors injected into the memory address space, the one or more memory access errors providing an indication of possible failure in executing the one or more processes; and associate the detected one or more memory access errors with one or more memory locations associated with the memory address space; and an analysis utility that aggregates the associated memory access errors to identify memory locations at which the one or more hardware memory errors were detected.

[0011] According to this aspect of the disclosed technology, the instructions can cause the one or more devices to randomly adjust the at least one memory address. Further, the error density level includes a number of errors randomly injected into memory locations associated with the memory address space. Moreover, the instructions can cause the one or more devices to vary a number of the plurality of virtual machines instantiated while varying the error density level. Additionally, the instructions can cause the one or more processing devices to vary workloads of the plurality of virtual machines instantiated.

[0012] Further, according to this aspect of the disclosed technology, the memory address space can include an entirety of an address space used by the plurality of virtual machines instantiated or a host machine address space. Additionally, the instantiated memory injection utility can filter the injected hardware memory errors to be one of a single rank in a dual in-line memory module (DIMM), a column in the DIMM, or a data line in the DIMM.

[0013] Another aspect of the disclosed technology can include a method of operating a hardware memory error-tolerant software system. The method can include instantiating, on a host computer, a kernel agent in response to one or more requests to access hardware memory; determining, by the kernel agent based on received information, whether the request to access memory will cause access to a corrupted memory location; and in response to determining that the request will access the corrupted memory location, skipping, by the host computer, operations related to the corrupted memory location. According to this aspect, the received information can include information identifying a kernel control path of an operating system of the computing device, the identified kernel control path including one of a kernel control path that performs housekeeping operations or optimization operations. According to such a method, the request to access memory can include a request to access a page in memory, and the instructions cause the one or more processing devices to skip, in response to the determination that the request will access the corrupted memory location, operations related to the page. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 The functionality of a host computer according to one aspect of the disclosed technology is illustratively depicted.

[0015] Figure 2 A system according to one aspect of the disclosed technology is illustratively depicted.

[0016] Figure 3 A method according to one aspect of the disclosed technology is illustratively depicted.

[0017] Figure 4 A method according to one aspect of the disclosed technology is illustratively depicted.

[0018] Figure 5 A method according to one aspect of the disclosed technology is illustratively depicted.

[0019] Figure 6 A system environment according to one or more aspects of the disclosed technology is illustratively depicted.

[0020] Figure 7 A system environment according to one or more aspects of the disclosed technology is illustratively depicted. DETAILED DESCRIPTION

[0021] SUMMARY

[0022] This technology relates to identifying memory errors and mitigating their impact on host or software systems, including allowing such systems to recover from or avoid the effects of such errors. For example, the disclosed technology advantageously allows software systems to recover from memory errors rather than causing the host and all managed virtual machines (VMs) to crash due to memory errors. The disclosed technology includes a mechanism that reacts to memory errors (e.g., memory check anomalies detected and reported by hardware) by causing applications that might access memory locations associated with the memory error to avoid accessing those memory locations. For example, a memory injection utility can inject memory errors at one or more memory locations corresponding to guest memory associated with one or more VMs. If a given VM accesses a corrupted memory location, applications associated with those memory locations can skip those memory locations during application execution. For example, applications can be modified to access different regions of memory during application execution. In this way, aspects of the disclosed technology can prevent potential system crashes.

[0023] A host machine is a device with memory and a processor configured to host one or more virtual machines. The host machine may implement a host operating system running a host kernel. A virtual machine simulates a real computer system and operates based on the computer architecture and functionality of a real or hypothetical computer system; this may include simulating specialized hardware and / or software. The operating system of a virtual machine is its guest operating system (“guest OS”), which may include a guest kernel. The memory allocated to a virtual machine is called its guest memory. Guest memory may correspond to a portion of the underlying physical memory of the host machine running the virtual machine.

[0024] During its operational lifetime, some or all of the memory devices on a host machine can fail for a variety of reasons, e.g., as a result of hardware defects or degradation over time or repeated use. Correctable errors typically do not affect the normal operation of the host machine. Correctable memory errors typically involve a single bit error (e.g., a bit flips from a one to a zero, or vice versa) and are corrected by supplementing the erroneous bit. Uncorrectable memory errors can be fatal to the host machine. For example, an uncorrectable memory error can occur in a memory device when a stored data bit inadvertently flips from one binary state to another. This can occur, for example, due to manufacturing defects of the memory device and / or due to magnetic or electrical interference (e.g., cosmic radiation) and temperature effects, which can cause bits to randomly flip. In contrast to uncorrectable memory errors that cause a system to crash or shut down, correctable memory errors typically do not affect the normal operation of the system. Memory errors can occur substantially at random locations in the DRAM chip and at random locations in the software system. While the host machine can implement error monitoring and handling techniques to recover from relatively small errors, recovery is not always feasible.

[0025] An uncorrectable memory error can occur when a processor of the host machine is accessing memory as part of the execution of one or more instructions. By way of example, the instructions can be part of a software routine that the host machine is configured to execute by its host kernel or an application running on the host machine. As part of executing the instructions, the processor accesses memory coupled to the host machine. The accessed memory can be memory reserved for the host machine or memory allocated to a virtual machine running on the host machine. Memory allocated to a virtual machine hosted by the host machine is referred to as its guest memory. The host maps the guest memory to the host's physical memory. When the processor accesses memory on the host machine, it can do so while operating in a kernel scenario or a user scenario. In the kernel scenario, the processor executes instructions that are part of a routine or software component for the host kernel of the host machine. The kernel can access the guest memory for a variety of different reasons. Generally, the kernel can access the guest memory as part of executing a routine to copy data from the guest memory.

[0026] If unchecked, uncorrectable memory errors can cause the host machine to crash or shut down with little or no warning or clue as to the source of the device failure. The impact of these uncorrectable memory errors is particularly severe for a host machine that hosts virtual machines and especially when each virtual machine can be allocated with gigabytes or terabytes of guest memory.

[0027] Some processors of a host machine can implement a machine check architecture that provides a mechanism for detecting and reporting errors caused by the processor or hardware connected to the processor. The machine check architecture generally refers to the portion of the processor that is configured to recognize and raise a machine check exception (MCE) that can be received and interpreted by a host kernel. While a host kernel, such as a Linux kernel-based kernel, can be configured to receive and handle MCEs that correspond to some uncorrectable memory errors without defaulting to a panic behavior, many uncorrectable memory errors cause the kernel to default to a panic behavior. When the kernel defaults to a panic behavior, the kernel can freeze or become unresponsive. The kernel can also cause the host machine executing the kernel to restart or shut down abruptly. If the host machine shuts down, all virtual machines (VMs) and software applications supported by the VMs also shut down.

[0028] Memory poisoning recovery aims to enable a software system to recover from memory errors, rather than having the hosted virtual machines crash along with the host. Aspects of the disclosed technology can include a memory poisoning recovery system and techniques. For example, the disclosed technology can include techniques or methods that enable determining which portions or portion of a software application, code, or instructions are susceptible to hardware memory errors. Hardware memory errors can be injected into the memory of a host device or host machine running one or more VMs. A VM on the host machine running the application can request access to a memory location that has been injected with an error due to the error injection process. Accessing the memory location with the error can cause the VM to crash and thus the software application to crash. Such crashes can be captured via a crash dump. Such crash dumps can be analyzed to detect a pattern that indicates a given portion of the software or code that crashed due to accessing the hardware memory location that was injected with an error. In turn, the software application or code can be designed such that when it is deployed in a production environment and encounters such memory errors, the VM, application, or host crashes are avoided.

[0029] Aspects of the disclosed technology include methods and systems for detecting and exposing software system vulnerabilities caused by memory errors. For example, the disclosed technology can include a memory error injection utility that injects hardware memory errors at various levels of randomness and density. It can also include a simulation and stress qualification system that can simulate various VM packaging and / or workload patterns under different injected memory error conditions. It can also include a system that can simulate a crashing process and related OS kernel and detect and report a vulnerable software stack.

[0030] A memory injection utility can reside, for example, on a host machine. The host machine can run one or more virtual machines as guests. These guests can be configured to run various software applications that utilize guest memory allocated to the guests by the host machine. The memory injection utility can inject hardware memory errors into locations of physical memory to corrupt the memory locations. The corrupted memory locations can thus correspond to a portion of guest memory (e.g., a corrupted virtual memory page). When a given guest accesses the corrupted virtual memory page, it should report an error or crash. The reported errors or crashes can be stored and subsequently reported. By randomly injecting hardware memory errors while the virtual machines are running applications that utilize the host's memory, a relationship between the software applications and the types of hardware memory errors that affect operation (e.g., cause errors or crashes) can be determined. By adjusting the density and address randomness of the injected memory errors, a simulation of random memory errors that can occur in a production environment can be achieved.

[0031] The simulation and stress qualification system can then inject various types of random memory errors on a set of machines over a given time period and, for example, repeatedly to reach a run time (e.g., 1 month of 120 servers is equivalent to about 10 machine years), allowing potentially vulnerable points in the software system (e.g., guests running software applications) to be reliably exposed that correspond to the probability that such vulnerabilities would occur in a production workload or production environment. Hardware errors related to the exposed vulnerabilities can be instrumented, aggregated, and analyzed via a crash measurement and analysis system. Instrumented or instrumentation refers to configuring kernel crash dumps to expose the actual stack traces of the crash instructions. Aggregated or aggregation refers to ordering and counting the call stack traces of the crash instructions from one or more rounds or cycles of the process that injected the error, or vice versa. Additionally, the stack traces can also be deduplicated or filtered to indicate frequency, such that the more or most frequent stack traces are given priority in terms of mitigating their impact. For example, the analysis can include extracting patterns from the aggregated data that indicate that a certain software application is more prone to a certain memory error. Common patterns can generally include call stacks that are frequently or most susceptible to memory errors. The vulnerabilities can then be prioritized for repair efforts based on the results of the analysis. For example, if a particular application error causes a large number of errors or crashes in the software system, such errors can be given a higher priority than other errors in terms of error repair.

[0032] Another aspect of the disclosed technology includes systems and techniques for recovery from software vulnerabilities related to memory accesses to user space memory from an OS kernel. Based on the analysis results of the simulation and stress qualification system, a memory poisoning checker assembly approach can be envisioned such that the OS kernel avoids accessing memory related to vulnerabilities in the software system. For example, one or more memory poisoning checker assembly routines can be envisioned and applied to kernel control paths related to the corrupted memory locations. These memory poisoning checkers can be viewed as kernel agents. For example, prior to accessing a memory location related to a vulnerability, the kernel agent first attempts to check whether a memory error exists in the memory page to be accessed and, if a memory error is detected, skips the operation on the memory page or operations related to the memory page.

[0033] In fact, the memory poisoning checker assembly approach or routines can be configured to detect memory errors without crashing the host. For example, the instruction pointer within the machine check exception (MCE) handler can be configured to fix the instruction pointer to avoid potentially corrupted memory locations. A fix-up table can be used, for example, to redirect one or more instructions to access different memory locations than the potentially corrupted memory locations.

[0034] Example process or method

[0035] Figure 1 A functional diagram of a host machine in accordance with at least one aspect of the disclosed technology is illustratively depicted. As shown, the host machine 100 includes an error injection utility 110 and a guest machine 120. For simplicity, only one guest machine is shown. However, in a typical case, the host machine 100 can host or run several VMs 120. The error injection utility 110 provides a hardware injection mechanism that allows for injection of hardware memory errors that cannot be corrected. The error injection utility 110 can include, for example, the EINJ functionality available in Linux. The utility 110 can be configured to inject hardware memory errors 130 into the guest workload memory or memory address space corresponding to a VM or guest OS memory page. Alternatively, the injection utility 110 can be configured to inject hardware memory errors 136 into the entire host memory address space.

[0036] The host machine 100 provides a memory poisoning recovery (MPR) framework for injecting / clearing hardware memory errors in the memory with hardware support by means of processor memory controller (e.g., registers in each memory controller). The MPR framework encompasses various memory accesses by varying parameters to simulate various memory hot / cold and to exert stress on various memory hot / cold and to exercise various kernel memory bug paths. The parameters can include one or more of the following: error density, access frequency, allocation type, address range, row / column, guest image, and guest memory backup. Error density refers to the number of errors randomly injected into the memory. Access frequency refers to the time gap between memory access cycles of the guest memory workload. Allocation type refers to workload separation of TMPFS (virtual memory file system), TMPFS with transparent hugepages, and anonymous memory, respectively. Address range refers to the injection of errors into random memory addresses, including the following selectable ranges: entire host physical address space or address space of the guest workload. Filtering parameters or functions allow filtering of addresses to be injected as a single row, single column, or data line (DQ) in the DIMM, or a collection of rows / columns / DQs to allow simulation of memory poisoning as close as possible to actual memory poisoning in production environments. Guest image refers to centos, ubuntu, redhat, sles, and specific versions. Guest memory backup refers to how the guest memory is backed up on the host, e.g., shmem, tmpfs, or hugetlb.

[0037] Figure 2A system 200 according to one aspect of the disclosed technology is depicted. As shown, the system 200 includes a regulator 210 communicatively coupled to an error injection module 220. The error injection module 220 is coupled to a memory 230 to enable injection of memory errors based on adjustments to the regulation parameters of the regulator 210. The error injection module 220 may, for example, inject errors into a machine check exception (MCE) register of the memory 230 indicating that a particular bit location in the memory (e.g., DRAM) is experiencing an uncorrectable memory error. In some examples, the error can be injected into a register of a memory controller of a CPU core. The memory 230 is followed by a data collector 240 that collects data related to the operation of the memory 230. The data collector 240 may, for example, collect data related to a crash and output the data as a crash dump 250. The data collector 240 can receive information indicating a memory location at which a hardware memory error was detected. The information can include information identifying a kernel control path of an operating system of the computing device, the identified kernel control path including one of a kernel control path performing a housekeeping operation or an optimization operation. Thus, the information can identify one or memory locations that, if accessed, can or will cause an application, VM, or host to crash or shut down.

[0038] The system 200 also includes a workload generator 260. The workload generator 260 provides functionality for adjusting VM packing, work modes, and other metrics that affect the workload experienced by the memory 230. For example, the workload generator 260 can cause instantiation of multiple VMs (as depicted at block 270) to put stress on the system, while the regulator 210 can set different parameters for different time periods to vary the conditions of the injected memory. The VMs can be configured to run real-world applications, such as a banking application, under real-world conditions. By adjusting the errors injected into the memory under different workload conditions, real conditions can be created to expose potentially vulnerable areas in the software system (e.g., software application).

[0039] The system 200 also includes an analyzer 280 that is provided with the collected data, including the crash dumps 250. The analyzer 280 aggregates the data it receives and extracts patterns from the aggregated data. For example, the analyzer can determine that the software system is designed such that certain memory accesses occur more frequently than others, e.g., hot pages. Conversely, the analyzer can determine that certain pages are accessed much less frequently, e.g., cold pages. The analyzer 280 can also determine the type of memory support required by the software system, e.g., an application running on a VM. More generally, the analyzer 280 collects and aggregates stack traces from crash dumps resulting from crashes caused by memory errors. From the aggregated stack traces, the analyzer identifies the most frequent code paths, instruction locations that will be affected by the memory error. These code paths / instruction locations are then cross-referenced with the source code to help understand the actual software virtual address space behind the source of the memory error (why the error is occurring frequently there) and how to make the memory accesses tolerant to the memory error. The analyzer 280 also collects information such as thread / process name, execution context (kernel / user space), backing memory type of the page with the error (e.g., HugeTLB or TMPFS or anonymous memory), and the time of the crash (thus the time from when the error was injected into the system until it caused a crash).

[0040] Figure 3 A process or method 300 according to one aspect of the disclosed technology is illustratively depicted. The method begins with creating a set of VMs (block 305). As described above, the VMs are created on a host machine. At block 310, errors are injected into the memory of the host machine running the VMs. The errors are injected at random densities and / or other parameter settings. At block 315, workloads are created to run on the VMs at different frequencies. At block 320, the host machine runs the VMs for N hours under the workload and error injection conditions and collects data such as crash dumps of crashes in the host and guests or VMs. The run time N and the number of VMs and host machines can be selected so as to simulate the long term behavior of a software system. For example, 120 servers running for about 1 month under certain conditions can achieve 10 machine years. Blocks 305, 310, 315, and 320 are repeated and, as shown at block 325, the crash dumps are aggregated. At block 330, the crashes are analyzed and grouped by the memory access that triggered the crash.

[0041] At block 340, the output of the analysis can be used to design a kernel agent code fix, such as, for example, redesigning the code to address the highest crash frequency memory kernel memory accesses. By way of example, a memory poisoning checker assembly routine can be developed based on the analysis to check for hardware memory errors prior to accessing memory related to portions of code or instructions that exhibited vulnerabilities during the analysis. These memory poisoning checker routines are in fact invoked from these vulnerable kernel code paths, which are kernel agents. These kernel agents are kernel threads responsible for specific tasks such as page table scans for memory access cold / hot detection; sparse page compression; and so forth.

[0042] Figure 4 is a diagram illustratively depicting how such a method can operate. Based on the analysis output from the analyzer, as depicted at block 410, vulnerable portions of the software application, code or instructions can be flagged. When a memory access request is received from the flagged code / instructions, etc. at block 416, the kernel agent checks for a memory error on the page that the flagged code / instructions wants to access at block 422 - for example, the OS kernel can scan the memory array for the memory location that the request is accessing. If there is an error on the page, then at block 428, the operation related to the corrupted memory page is skipped and a potential uncorrectable memory error is avoided. This thus avoids a potential crash of the host machine. If there is no error on the page, then the memory access request can be allowed, as indicated at block 434.

[0043] In another example, the output of the analyzer can be used to flag instructions or code such that when an error is detected, a fix-up handler can be invoked. For example, for instructions flagged as vulnerable requests for access to a memory location with an error, the fix-up handler checks a fix-up table that redirects the instructions to a different memory location than the one included in the access request. This thus avoids a potentially corrupted memory page or location and a potential crash of the host machine. This aspect of the technology can be implemented in kernel agents in pseudo-code as follows, for example:

[0044]

[0045] Figure 5 is a diagram showing an example of a method 500 that can be implemented in accordance with this aspect of the disclosed technology. As shown, in step 510, the error injection process described above is run on the memory address space. At block 520, memory access pages that are detected as vulnerable to memory errors and that crash frequently are collected or identified. Such vulnerable memory access pages can then be used by a fix-up handler to skip those pages (e.g., “poisoned memory pages”) in a production environment at block 530.

[0046] Figure 6 This is a block diagram of an example system 600 according to aspects of this disclosure. System 600 includes one or more computing devices 610A-K, including devices 610A, 610K, and optionally one or more other devices (not shown). In some implementations, system 600 includes a single computing device 610A that operates as a host 100, for example, as described above, as a simulation and stress assessment system. System 600 also includes a network 640 and one or more cloud computing systems 650A-M, which may include cloud computing systems 650A and 650M. In some implementations, system 600 includes a single cloud computing system 650A. Computing devices 610A-K may include computing devices located at customer locations that utilize cloud computing services. For example, if computing devices 610A-K are located at a business enterprise, computing devices 610A-K may use cloud systems 650A-M as part of providing one or more services to computing devices 610A-K that provide software applications.

[0047] like Figure 6 As shown, computing devices 610A-K may each include one or more processors 612A-K; memory 616A-K for storing data (D) 634A-K and instructions (I) 632A-K; a display 620A-K; a communication interface 624A-K; and an input system 628A-K, which are shown interconnected via a network 640. Each computing device 610A-K may be coupled to or connected to a corresponding storage device 636A-K, which may include local or remote storage for storing data, such as on a storage area network (SAN).

[0048] Each computing device 610A-K may include a standalone computer (e.g., a desktop or laptop) or server. Network 640 may include a data bus or similar component within the computing device, and / or may include one or more of a local area network (LAN), a virtual private network (VPN), a wide area network (WAN), or other types of networks described below with respect to network 640. Memory 616A-K stores information accessible by one or more processors 612A-K, including instructions 632A-K and data 634A-K that can be executed or otherwise used by the processors 612A-K. Memory 616A-K may be any type capable of storing information accessible by a corresponding processor, including computing device-readable media, or other media that store data readable by means of an electronic device, such as hard disk drives, memory cards, ROM, RAM, DVDs or other optical discs, and other write-only and read-only memories. Systems and methods may include different combinations of the foregoing, whereby different portions of instructions and data are stored on different types of media.

[0049] Instructions 632A-K can be any set of instructions executed directly (such as machine code) or indirectly (such as scripts, bytecode, interpretation threads, etc.) by the processor. One or more instructions executed by the processor can represent

[0050] Processors 612A-K can retrieve, store, or modify data 634A-K in accordance with instructions 632A-K. For example, data 634A-K associated with memory 616A-K can include data used to support services for one or more client devices, applications, etc. Such data can include data used to support hosting web-based applications, file sharing services, communication services, games, shared video or audio files, or any other web-based services.

[0051] Each processor 612A-K can be any one or any combination of a general-purpose processor and / or a special-purpose processor. Processors 612A-K are configured to implement a machine check architecture or other mechanism for identifying memory errors and reporting memory errors to a host kernel. Examples of general-purpose processors include CPUs. Alternatively, one or more processors can be special-purpose devices, such as FPGAs or ASICs, including tensor processing units (TPUs). Although Figure 6 Although the processor, memory, and other elements of each computing device 610A-K are shown functionally within single blocks, it will be understood by those skilled in the art that the processor, computing device, or memory can actually include multiple processors, computing devices, or memories, which can or can not be located within the same physical housing. In one example, one or more of computing devices 610A-K can include one or more server computing devices having multiple computing devices, such as a load-balanced server farm, that exchange information as clients to the network to receive, process, and transmit data to and from other computing devices as part of the operation of a business.

[0052] Computing devices 610A-K can include displays 620A-K, e.g., monitors with screens, touchscreens, projectors, televisions, or other devices operable to display information. Displays 620A-K can provide user interfaces that allow control of computing devices 610A-K and access to user-space applications and / or data-related VMs supported in one or more cloud systems 650A-M, e.g., on hosts in the cloud systems. Such control can include, for example, using the computing devices to cause data to be uploaded to cloud systems 650A-M through input systems 628A-K for processing, to be accumulated on storage 636A-K, or more generally, to manage different aspects of a customer's computing system. In some examples, computing devices 610A-K can also access APIs that allow them to specify workloads or jobs to run on VMs in the cloud as part of IaaS (Infrastructure as a Service) or SaaS (Software as a Service). While input systems 628A-K can be used to upload data, e.g., USB ports, computing devices 610A-K can also include mice, keyboards, touchscreens, or microphones that can be used to receive commands and / or data.

[0053] Network 640 can include various configurations and protocols, including short-range communication protocols such as Bluetooth™, Bluetooth™ LE, the Internet, the World Wide Web, intranets, virtual private networks, wide area networks, local networks, private networks using communication protocols proprietary to one or more companies, Ethernet, WiFi, HTTP, and / or the like, and various combinations thereof. Such communication can be facilitated by any communication infrastructure, such as modems and wireless interfaces, that are capable of transmitting and receiving data, among other things. Computing devices 610A-K can interface to network 640 through communications interface 624A-K, which can include the hardware, drivers, and software necessary to support the given communication protocol.

[0054] Cloud computing systems 650A-M can include one or more data centers, which can be connected via high-speed communication or computing networks. A data center can include a dedicated space within a building that houses computing systems and their related components, such as storage systems and communication systems. Typically, a data center will include racks of communication equipment, servers / hosts, and disks. The servers / hosts and disks include physical computing resources used to provide virtual computing resources, such as VMs. If a given cloud computing system includes more than one data center, these data centers can be located in different geographic locations that are relatively close to one another, chosen to provide services in a timely and cost-efficient manner as well as to provide redundancy and maintain high availability. Similarly, different cloud computing systems are typically provided at different geographic locations.

[0055] AsFigure 6 As shown, computing system 650A-M can include host machines 652A-M, storage 654A-M, and infrastructure 660A-M. Infrastructure 660A-M can include data centers within cloud computing system 650A-M. Infrastructure 660A-M can include one or more host machines, as well as switches, physical links (e.g., fiber optic), and other equipment used to interconnect host machines within a data center with storage 654A-M. Storage 654A-M can include disks or other storage devices that are partitioned to provide physical or virtual storage for virtual machines running on processing devices within a data center. Storage 654A-M can be provided as a SAN within a data center hosting virtual machines supported by storage 654A-M or in a different data center that does not share a physical location with the virtual machines it supports. One or more hosts or other computer systems within a given data center can be configured to act as a supervisory agent or hypervisor in creating and managing virtual machines related to one or more host machines in the given data center. Generally, a host or computer system configured to act as a hypervisor will contain instructions necessary to manage operations, for example, necessary to provide IaaS, PaaS (Platform as a Service), or SaaS to customers or users in response to service requests from, for example, computing devices 610A-K.

[0056] Figure 7 is a block diagram of a distributed system 700 according to aspects of the present disclosure, showing an example virtual machine monitor or hypervisor 820 interacting with a host operating system 620 and a virtual machine layer 840. Distributed system 700, such as that shown in FIG. 6, can implement various aspects of the functionality described herein. For example, distributed system 700 can facilitate the creation and management of virtual machines and the provisioning of virtual machines to customers or users. Distributed system 700 can also facilitate the creation and management of virtual machines and the provisioning of virtual machines to customers or users in response to service requests from, for example, computing devices 610A-K. Figure 6The distributed system shown by cloud system 650A-M, including a collection 704 of host machines 610A-M (e.g., hardware resources 610A-M) that support or execute virtual computing environment 800. Virtual computing environment 800 includes a virtual machine monitor (VMM) 820 and a virtual machine (VM) layer 840 that runs one or more virtual machines (VMs) 850A-N configured to execute instances 862A-N of one or more software applications 860. Each host machine 610A-M can include one or more physical central processing units (pCPUs) 712 and associated memory devices 716. While each hardware resource or host 610A-M is shown as having a single physical processor 712, any hardware resource 610A-M can include multiple physical processors 712 and / or multiple memory devices 716. Host machines 610A-M also include physical memory devices 716 that can be partitioned into virtual memory by a host operating system (OS) 618A-N and assigned for use by VMs 850A-N in VM layer 840, or even VMM 820 or host OS 620. Physical memory devices 716 can include random access memory (RAM) and / or disk memory (including, e.g., as shown by memory 654A-M accessible by infrastructure 660A-M). Figure 6

[0057] A host operating system (OS) 620 can execute on a given one of host machines 610A-M, or can be configured to operate across a collection of host machines 610A-M, including multiple host machines. For convenience, Figure 7 Host OS 620 is shown as operating across the collection of machines 610A-M. Further, while host OS 620 is shown as part of virtual computing environment 800, each host machine 610A-M is equipped with its own OS 718A-N. However, from the perspective of the virtual environment, the OS on each machine appears to VMM 820 and VM layer 840 as collective OS 620 and is managed as collective OS 620.

[0058] ​In some examples, the VMM 820 corresponds to a hypervisor 820 (e.g., a compute engine) that includes at least one of software, firmware, or hardware configured to create, instantiate / deploy, and execute VMs 850A-N. A computer associated with the VMM 820 that executes one or more VMs 850A-N is often referred to as a host machine (as used above), while each VM 850A-N can be referred to as a guest. Here, the VMM 820 or hypervisor is configured to provide each VM 850A-N with a corresponding guest operating system (OS) 854 (e.g., 854A-N) having a virtual operating platform, and manage execution of the corresponding guest OS 854A-N on the VM 850A-N. In some examples, multiple instances of various operating systems can share virtualized resources. For example, Linux ® a first VM 850A of a Windows ® a second VM 850B of a Windows ® a third VM 850C of a Windows

[0059] The distributed system 700 enables users to launch VMs 850A-N (through one or more computing devices 610A-M) on-demand, i.e., by sending a command or request 670 Figure 6 ) to the distributed system 700 (including the cloud system 650) via the network 640. For example, the command / request 670 can include an image or snapshot associated with the host operating system 620, and the distributed system 700 can use the image or snapshot to create a root resource for a corresponding VM. Here, the image or snapshot within the command / request 670 can include a boot loader, the host operating system 620, and a root file system. In response to receiving the command / request 670, the distributed system 700 can instantiate the corresponding VM and automatically start the VM upon instantiation.

[0060] A VM emulates a real computer system (e.g., from a host machine in the host machines 610A-M) and operates based on the computer architecture and functionality of the real computer system or an imaginary computer system, which can involve specialized hardware, software, or a combination thereof. In some examples, the distributed system 700 authorizes and authenticates a user device prior to launching one or more VMs 850A-N. An instance 862A-N of a software application 860 (or simply an instance) refers to a VM 850A-N hosted on the distributed system 700.

[0061] The host OS 620 virtualizes underlying host machine hardware and manages concurrent execution of one or more VM instances 862A-N. For example, the host OS 620 can manage the VM instances 862A-N, and each VM instance 862A-N can include an emulated version of the underlying host machine hardware or a different computer architecture. The emulated version of the hardware related to each VM instance is referred to as virtual hardware 852A-N. The virtual hardware 852A-N can include one or more virtual central processing units (vCPUs) (“virtual processors”) that emulate one or more physical processors 712 of the host machine 610A-M. The virtual processors can be interchangeably referred to as “computing resources” related to the VM instances 862A-N. The computing resources can include target computing resource levels required to execute the corresponding individual service instances 862A-N.

[0062] The virtual hardware 852A-N can also include virtual memory that communicates with the virtual processors and stores guest instructions (e.g., guest software) that are executable by the virtual processors to perform operations. For example, the virtual processors can execute instructions from the virtual memory that cause the virtual processors to execute the corresponding individual service instances 862A-N of the software applications 860. Here, the individual service instances 862A-N can be referred to as guest instances that are unable to determine whether it is being executed by the virtual hardware 852A-N or the physical data processing hardware 712. The host machine’s processors can include processor-level mechanisms to enable the virtual hardware 852A-N to efficiently execute the software instances 862A-N of the applications 860 by allowing guest software instructions to execute directly on the host machine’s processors without code rewriting, recompilation, or instruction emulation. The virtual memory can be interchangeably referred to as “memory resources” related to the VM instances 862A-N. The memory resources can include target memory resource levels required to execute the corresponding individual service instances 862A-N.

[0063] The virtual hardware 852A-N can also include at least one virtual storage device that provides runtime capacity for services on the physical storage hardware 716. The at least one virtual storage device can be referred to as a storage resource related to the VM instances 850. The storage resource can include target storage resource levels required to execute the corresponding individual service instances 862A-N. The guest software executing on each VM instance 862A-N can also be assigned a network boundary (e.g., assigned a network address) through which the respective guest software can communicate with other processes that can be accessible through an internal network or infrastructure 660 Figure 6 ), an external network 640 Figure 6 ), or both. The network boundary can be referred to as a network resource related to the VM instances 850A-N.

[0064] The guest OS 854A-N executing on each VM 850A-N includes software that controls execution of the corresponding individual service instance 862 (e.g., one or more of 862A-N of the application 860) by the VM 850A-N. The guest OS 854A-N executing on a VM instance can be the same as or different from other guest OS 854A-N executing on other VM instances 862A-N. In some implementations, a VM instance does not require a guest OS to execute an individual service instance 862. The host OS 620 can also include virtual memory reserved for the kernel 626 of the host OS 620. The kernel 626 can include kernel extensions and device drivers, and can perform certain privileged operations that are prohibited to processes running in the user process space of the host OS 620. Examples of privileged operations include accessing different address spaces, accessing special function processor units in the host machine (such as memory management units), and the like. The communication process 624 running on the host OS 620 can provide a portion of the VM network communication functionality and can execute in either the user process space or the kernel process space in relation to the kernel 626.

[0065] The kernel 626 can implement an MCE handler for processing MCEs raised by the processors of the host machine 610A-M. Similarly, the guest OS of each VM 850A-N can implement a guest MCE handler for receiving and processing emulated MCEs.

[0066] As shown in Figure 7 The host machine can include instructions 740A-N that can function according to one or more aspects of the disclosed technology. The instructions 740A through 740N can operate as a kernel agent or fix-up handler that is aware of potentially vulnerable instructions or code running at the VM layer 840 and takes action to skip over potentially corrupt memory locations on the memory device 716. The kernel agent or fix-up handler can be used to cause applications or instructions running at the VM layer to skip over other operations related to potentially corrupt memory locations or be directed to uncorrupted memory locations. Either operation will avoid crashing the VM or the host.

[0067] Aspects of the disclosed technology can include one or more of the following combinations of features:

[0068] F1. A hardware memory error tolerance software system, comprising:

[0069] one or more processing devices;

[0070] an input to receive information indicative of a memory location at which a hardware memory error was detected; and

[0071] a memory storing instructions that cause the one or more processing devices to:

[0072] in response to one or more requests to access the memory, instantiate a kernel agent that determines, based on the received information, whether the requests to access the memory will cause access to a corrupt memory location, and

[0073] in response to a determination that the requests will access a corrupt memory location, skip operations related to the corrupt memory location.

[0074] F2. The system of F1, wherein the received information includes information identifying a kernel control path of an operating system of the computing device, and the identified kernel control path includes a kernel control path that performs one of a housekeeping operation or an optimization operation.

[0075] F3. The system of any one of F1 or F2, wherein the requests to access the memory include requests to access a page in the memory.

[0076] F4. The system of any one of F1 to F3, wherein the instructions cause the one or more processing devices to, in response to the determination that the requests will access a corrupt memory location, skip operations related to the page.

[0077] F5. The system of F1, wherein the input receives the information from a hardware memory error detection system, including:

[0078] instantiating a plurality of virtual machines on a host machine, the plurality of virtual machines related to a memory address space for running one or more processes,

[0079] instantiating a memory injection utility that injects one or more hardware memory errors into the memory address space,

[0080] adjusting one of a frequency of injection of the one or more hardware memory errors or an error density level of the one or more hardware memory errors injected into the memory address space,

[0081] detecting one or more memory access errors related to the plurality of virtual machines based on the one or more hardware memory errors injected into the memory address space, the one or more memory access errors providing an indication of possible malfunction in executing the one or more processes, and

[0082] associating the detected one or more memory access errors with one or more memory locations related to the memory address space; and

[0083] an analysis utility that aggregates associated memory access errors to identify memory locations at which the one or more hardware memory errors are detected; and

[0084] an output that provides the identified memory locations as the information for input to the hardware memory error-tolerant software system.

[0085] F6. The system of F5, wherein the instructions cause the one or more devices to adjust the at least one memory address randomly.

[0086] F7. The system of any one of F5 or F6, wherein the error density level comprises a number of errors randomly injected into memory related to the memory address space.

[0087] F8. The system of any one of F5 to F7, wherein the instructions cause the one or more devices to vary a number of the plurality of machines instantiated while varying the error density level.

[0088] F9. The system of any one of F5 to F8, wherein the instructions cause the one or more devices to vary a workload of the plurality of machines instantiated.

[0089] F10. A system for detecting software system vulnerabilities resulting from hardware memory errors, comprising:

[0090] a host machine having one or more processing devices and a memory storing instructions that cause the one or more processing devices to:

[0091] instantiate a plurality of virtual machines related to a memory address space for running one or more processes,

[0092] instantiate a memory injection utility that injects one or more hardware memory errors into the memory address space,

[0093] adjust one of an injection frequency of the one or more hardware memory errors or an error density level of the one or more hardware memory errors injected into the memory address space,

[0094] detect one or more memory access errors related to the plurality of virtual machines based on the one or more hardware memory errors injected into the memory address space, the one or more memory access errors providing an indication of possible failure in executing the one or more processes, and

[0095] associating the detected one or more memory access errors with one or more memory locations related to the memory address space; and

[0096] an analysis utility that aggregates associated memory access errors to identify memory locations at which the one or more hardware memory errors were detected.

[0097] F11. The system of F10, wherein the instructions cause the one or more devices to randomly adjust the at least one memory address.

[0098] F12. The system of F10 or F11, wherein the error density level comprises a number of errors randomly injected into memory locations related to the memory address space.

[0099] F13. The system of any of F10 to F12, wherein the instructions cause the one or more devices to change a number of the plurality of virtual machines instantiated while changing the error density level.

[0100] F14. The system of any of F10 to F13, wherein the instructions cause the one or more processing devices to change a workload of the plurality of virtual machines instantiated.

[0101] F15. The system of any of F10 to F14, wherein the memory address space comprises an address space used by the plurality of virtual machines instantiated.

[0102] F16. The system of any of F10 to F15, wherein the memory address space comprises all of the host machine address space.

[0103] F17. The system of any of F10 to F16, wherein an instantiated memory injection utility filters the hardware memory errors injected into one of a single rank in a dual inline memory module (DIMM), a column in a DIMM, or a data line in a DIMM.

[0104] F18. A method of operating a hardware memory error tolerance software system, comprising:

[0105] in response to one or more requests to access hardware memory, instantiating a kernel agent on a host computer,

[0106] determining, by the kernel agent based on received information, whether the requests to access memory will cause access to a corrupt memory location, and

[0107] in response to determining that the request will access a corrupted memory location, skipping, by the host computer, operations related to the corrupted memory location.

[0108] F19. The method of F18, wherein the received information comprises information identifying a kernel control path of an operating system of the computing device, the identified kernel control path comprising one of a kernel control path that performs housekeeping operations or optimization operations.

[0109] F20. The method of F18 or F19, wherein the request to access memory comprises a request to access a page in memory, and the instructions cause the one or more processing devices to skip, in response to the determination that the request will access a corrupted memory location, operations related to the page.

[0110] Aspects of the disclosure can be implemented in digital electronic circuitry, in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of the foregoing. Software implementations of aspects of the disclosure could be implemented for example, as one or more computer programs running on a computer system.

[0111] In this specification, the phrase "configured to" is used in different contexts of different cases related to a part of a computer system, hardware, or computer program. When it is said that a system is configured to perform one or more operations, it means that the system has appropriate software, firmware, and / or hardware installed on the system, which, when operating, cause the system to perform the one or more operations. When it is said that certain hardware is configured to perform one or more operations, it means that the hardware includes one or more circuits that, when operating, receive an input and generate an output according to the input and corresponding to the one or more operations. When it is said that a computer program is configured to perform one or more operations, it means that the computer program includes one or more program instructions that, when executed by one or more computers, cause the one or more computers to perform the one or more operations.

[0112] The foregoing alternative examples are not mutually exclusive of one another unless otherwise specified, but can be implemented in various combinations to achieve unique advantages. As such, the foregoing description of the embodiments should be considered as merely illustrative of the subject matter defined by the claims that follow this detailed description, rather than as placing limitations upon the subject matter defined by the claims. Additionally, the examples described herein and the provision of clauses expressed in terms such as "containing", "comprising", and the like are to be construed in an inclusive sense, meaning that the stated integer elements are essential to the structure of the combinations that the claims recite. Additionally, the use of "or" means "and / or" unless otherwise stated. Furthermore, the use of "a" or "an" are employed to describe integers. Both the singular and the plural of such integers are consequently possible unless otherwise indicated. Also, the use of "about" means that variations of less than or more than and including the number specified are also possible, unless otherwise indicated. Also, the use of "comprising" means "including, but not limited to" and "consisting of means "including and limited to." Moreover, different reference numerals can designate the same or similar elements.

Claims

1. A hardware memory error tolerance software system, comprising: one or more processing devices; an input to receive information indicative of a memory location at which a hardware memory error was detected; and a memory storing instructions that cause the one or more processing devices to: in response to one or more requests to access memory, instantiate a kernel agent that determines, based on the received information, whether the request to access memory will cause access to a corrupted memory location, in response to a determination that the request will access the corrupted memory location, skip operations related to the corrupted memory location, wherein the request to access memory comprises a request to access a page in memory, and wherein the instructions cause the one or more processing devices to skip operations related to the page in response to the determination that the request will access the corrupted memory location.

2. The system of claim 1, wherein the received information comprises information that identifies a kernel control path of an operating system of a computing device, the identified kernel control path comprising a kernel control path that performs one of a housekeeping operation or an optimization operation.

3. The system of claim 1, wherein the input receives the information from a hardware memory error detection system, comprising: instantiating a plurality of virtual machines on a host machine, the plurality of virtual machines related to a memory address space for running one or more processes, instantiating a memory injection utility that injects one or more hardware memory errors into the memory address space, adjusting one of a frequency of injection of the one or more hardware memory errors or an error density level of the one or more hardware memory errors injected into the memory address space, detecting one or more memory access errors related to the plurality of virtual machines based on the one or more hardware memory errors injected into the memory address space, the one or more memory access errors providing an indication of possible malfunction in executing the one or more processes, and associating the detected one or more memory access errors with one or more memory locations related to the memory address space; and an analysis utility that aggregates the associated memory access errors to identify a memory location at which the one or more hardware memory errors were detected; and an output that provides the identified memory location as the information for input to the hardware memory error tolerance software system.

4. The system of claim 3, wherein the instructions cause the one or more devices to adjust the at least one memory address randomly.

5. The system of claim 3, wherein the error density level comprises a number of errors injected randomly into memory related to the memory address space.

6. The system of claim 3, wherein the instructions cause the one or more devices to vary a number of the plurality of virtual machines instantiated while varying the error density level.

7. The system of claim 3, wherein the instructions cause the one or more devices to vary a workload of the plurality of virtual machines instantiated.

8. A system for detecting software system vulnerabilities caused by hardware memory errors, comprising: a host machine having one or more processing devices and a memory storing instructions that cause the one or more processing devices to: instantiate a plurality of virtual machines on the host machine, the plurality of virtual machines associated with a memory address space for running one or more processes, instantiate a memory injection utility that injects one or more hardware memory errors into the memory address space, adjust one of an injection frequency of the one or more hardware memory errors or an error density level of the one or more hardware memory errors injected into the memory address space, detect one or more memory access errors associated with the plurality of virtual machines based on the one or more hardware memory errors injected into the memory address space, the one or more memory access errors providing an indication of possible malfunction in executing the one or more processes, and associate the detected one or more memory access errors with one or more memory locations associated with the memory address space; and an analysis utility that aggregates the associated memory access errors to identify memory locations at which the one or more hardware memory errors were detected.

9. The system of claim 8, wherein the instructions cause the one or more devices to randomly adjust the at least one memory address.

10. The system of claim 8, wherein the error density level comprises a number of errors randomly injected into memory locations associated with the memory address space.

11. The system of claim 8, wherein the instructions cause the one or more devices to vary a number of the plurality of virtual machines instantiated while varying the error density level.

12. The system of claim 8, wherein the instructions cause the one or more processing devices to vary a workload of the plurality of virtual machines instantiated.

13. The system of claim 8, wherein the memory address space comprises an address space used by the plurality of virtual machines instantiated.

14. The system of claim 8, wherein the memory address space comprises all of the host machine address space.

15. The system of claim 8, wherein the memory injection utility instantiated filters the hardware memory errors injected to be one of a single rank in a dual in-line memory module (DIMM), a column in a DIMM, or a data line in a DIMM.

16. A method of operating a hardware memory error tolerant software system, comprising: in response to one or more requests to access hardware memory, instantiating a kernel agent on a host computer, determining, by the kernel agent based on received information, whether the requests to access memory will cause access to a corrupt memory location, and in response to a determination that the request will access a corrupted memory location, skipping, by the host computer, operations related to the corrupted memory location, wherein the request to access memory comprises a request to access a page in memory, and in response to the determination that the request will access the corrupted memory location, operations related to the page are skipped.

17. The method of claim 16, wherein the received information comprises information identifying a kernel control path of an operating system of the computing device, the identified kernel control path comprising a kernel control path that performs one of a housekeeping operation or an optimization operation.

Images