Data storage using memory aperture flash order

A battery-backed memory aperture with a flash order prioritizes critical data storage in the event of power outages, addressing data integrity challenges in networked systems by ensuring critical data is preserved and reducing redundancy needs.

JP7586895B2Active Publication Date: 2024-11-19MICROSOFT TECHNOLOGY LICENSING LLC
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
JP2022511208
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-08-21
Filing Date
2020-06-15
Publication Date
2024-11-19
Estimated Expiration
2040-06-15

AI Technical Summary

Technical Problem

System architects face a multitude of trade-offs when choosing data storage devices, including volatile and non-volatile options, which can lead to complexity and unpredictability in ensuring data integrity, especially in networked environments like cloud-based virtual machines, where power outages can compromise critical data.

Method used

Implementing a data storage subsystem with a battery-backed memory aperture that follows a flash order for copying data to non-volatile storage, prioritizing critical data storage before non-critical data to ensure integrity, and using battery characteristics to size and resize the aperture for optimal data preservation.

Benefits of technology

This approach reduces the risk of data loss by ensuring critical data is flushed to non-volatile storage before power failures, thereby maintaining data integrity and reducing the need for redundant copies, while balancing cost and performance considerations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007586895000001
    Figure 0007586895000001
  • Figure 0007586895000002
    Figure 0007586895000002
  • Figure 0007586895000003
    Figure 0007586895000003
Patent Text Reader

Abstract

The combination of operational steps and device characteristics helps preserve data against integrity threats. Data is divided into critical and non-critical data based on criteria such as customer requirements, workload criticality, or virtual machine criticality. Data may be generated on compute nodes for storage on storage nodes, for example. Critical data is stored at physical addresses in battery-backed memory apertures and is flushed before non-critical data according to a flash order imposed by or on the battery-backed memory, e.g., bottom-up NVDIMM flash order. Redundant copies of data (especially non-critical data) can also be maintained in case the data is not flushed in time. Battery-backed memory apertures are sized and located according to their battery characteristics and can be relocated or resized as conditions change. Flash defragmentation is performed to optimize aperture usage, especially within the portion holding critical data.
Need to check novelty before this filing date? Find Prior Art

Description

[Background technology]

[0001]

[0001] System architects and others who design, implement, modify, or optimize computing systems recognize that data storage choices often involve tradeoffs. For example, some devices capable of storing digital data, such as some dynamic random access memory devices, support relatively fast data storage operations but cannot reliably retain data in the event of a power outage. These devices are generally referred to as "volatile" storage devices. There are many volatile storage devices with different technical characteristics, such as capacity, cost, expected working life, operating speed, difficulty of replacement, electrical requirements, and compatibility with hardware or software standards or best practices. Other data storage devices, such as electromechanical hard disk drives and solid state drives, reliably retain data values ​​even after power is removed. These devices are generally referred to as "non-volatile" storage devices. Again, there are many non-volatile storage devices with different technical characteristics.

[0002]

[0002] Thus, system architects and others who must choose among available storage mechanisms and procedures are faced with an inordinate number of storage device choices, with many interdependent tradeoffs between technology choices and system performance, cost, usability, reliability, and other characteristics. Thus, to reduce complexity and increase predictability, it may be useful to narrow and prioritize storage architecture choices by targeting specific performance assumptions, goals, or findings, but serious consideration is required. Summary of the Invention

[0003]

[0003] Some embodiments described herein provide improved data preservation tools and techniques, particularly in networked, multi-mode environments such as clouds running virtual machines. In some embodiments, memory allocation is based on the flash order of a battery-backed memory aperture, and based on associated information about battery characteristics, reducing the risk of losing critical data. Such storage allocation can also reduce data redundancy requirements without compromising data integrity.

[0004]

[0004] In one embodiment, a computing system has a data storage subsystem including a battery-backed memory having an aperture. The aperture has a flush order. The flush order is an order for copying data from the aperture to aperture-associated non-volatile storage in response to a data integrity threat, such as a power outage or reboot. The flush order defines a first-to-flush end of the aperture and a last-to-flush end of the aperture.

[0005] In this example, a data storage circuit is in operative communication with the battery backed-up memory. The data storage circuit may include a processor and firmware and may be configured to perform data storage steps. In this example, these data storage steps may include: (a) receiving a request to store a data set A-data in an aperture, where the A-data includes data designated as critical data; (b) identifying a portion A-memory of the aperture's unallocated memory, where the A-memory is large enough to hold the A-data, and where the A-memory has an address closer to the flash end of the aperture than any other address of any other unallocated memory of the aperture; and (c) marking the A-memory as allocated and inputting a copy of the A-data into the A-memory. These steps (a)-(c) may be repeated multiple times, one for each different data set that contains critical data.

[0006]

[0006] In this example, the data storage step may also include: (d) receiving a request to store a data set Z-Data in the aperture, the Z-Data not including any data designated as critical data; (e) identifying a portion of the unallocated memory of the aperture, Z-Memory, the Z-Memory being large enough to hold the Z-Data, the Z-Memory having an address closer to the flash end of the aperture than any other address of any other unallocated memory of the aperture; and (f) marking the Z-Memory as allocated and inputting a copy of the Z-Data into the Z-Memory. These steps (d)-(f) may be repeated for each different data set that does not contain critical data, or may be omitted when only critical data is being stored.

[0007]

[0007] The labels (a)-(f) serve here merely as identifiers and do not necessarily specify an order of operations. To give an example, Z-data may be stored in an aperture before any A-data is stored.

[0008]

[0008] In this context, one example of a data preservation subsystem is one that, rather than storing data in a battery-backed memory without consideration of the data's criticality, enters the critical data into the battery-backed memory prior to the non-critical data in a flush-order, thereby increasing the likelihood of successfully flushing the critical data and thereby preserving it. Alternative forms of data preservation, such as maintaining replicas or other redundant copies of the critical data external to the threatened computing system, may be reduced or eliminated given the capabilities of the data preservation subsystems described herein.

[0009] In operation, an embodiment of the data storage described herein receives a plurality of requests, each request seeking to store a respective set of data within an aperture of a battery-backed memory. The aperture has a flush order, which is an order in which data is copied from the aperture to an aperture-associated non-volatile storage in response to a data integrity threat. The flush order defines a flush leading edge of the aperture and a flush trailing edge of the aperture. In this example, each respective set of data includes data designated as critical data. For at least two of the requests, the embodiment identifies a respective portion of unallocated memory of the aperture, each portion of the unallocated memory being large enough to hold a respective set of data. Each identified portion of unallocated memory has an address that is closer to the flush leading edge of the aperture than any other address of any other unallocated memory of the aperture. For at least one of the requests, this embodiment marks each identified portion of unallocated memory as allocated and places a copy of the respective data set therein.

[0010] Continuing with this example, at some point after storing at least one critical data, the embodiment detects a data integrity threat, such as an impending reboot or an impending external power outage that would result in a switch to battery power. In response to detecting the threat, the embodiment copies the critical data from the aperture to non-volatile storage by flushing all of the critical data copied to the aperture. This flush preserves all of the critical data copied to the aperture, thus preserving the critical data (and possibly some or all of the non-critical data) without any reliance on having redundant copies of the data outside of the aperture or on the non-volatile backup memory.

[0011]

[0011] Other technical activities and characteristics related to the teachings herein will be apparent to those skilled in the art. The examples provided are merely illustrative. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Instead, this Summary is provided to introduce in a simplified form some technical concepts that are further described below in the Detailed Description. The innovation is defined by the claims as fairly understood, and if this Summary conflicts with the claims, the claims will control. [Brief description of the drawings]

[0012]

[0012] A more particular description will be provided with reference to the accompanying drawings, which merely illustrate selected aspects and are therefore not intended to be exhaustive in determining the scope of the claims. [Figure 1] A block diagram showing an overall computer system and also showing the storage media configured therein. [Diagram 2]FIG. 1 is a block diagram illustrating an environment including a compute node and at least one storage node. [Diagram 3] FIG. 1 is a block diagram illustrating certain aspects of a computing environment. [Figure 4] FIG. 1 is a block diagram showing an embodiment of a system in which a data storage function is configured. [Diagram 5] FIG. 2 is a block diagram showing some examples of battery characteristics. [Figure 6] FIG. 1 is a block diagram showing some examples of criteria for determining data criticality. [Figure 7] FIG. 1 is a block diagram illustrating an example of a computing system. [Figure 8] FIG. 1 is an architecture diagram showing compute nodes networked together with various storage nodes bearing various virtual machines and having battery-backed memory apertures. [Figure 9] Architectural diagram showing an operating system running various virtual machines and using a storage-backed memory aperture. [Figure 10] 1 is a flow chart illustrating steps in a data storage method. [Figure 11] 4 is a flow chart further illustrating steps in a data preservation method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013]

[0024] big picture

[0025] Innovations can expand beyond their origins, but understanding the origins of innovations can help appreciate their true meaning more deeply. In the present case, the teachings described herein were motivated by the technical challenge of storing big data for machine learning. Internet of Things devices pump out data in large volumes and at rapid rates, and machine learning tools and techniques can process this deluge of data in useful ways. Some machine learning databases contain millions of records within the actual database, so high read speeds are important. Despite the large amounts of data involved, data integrity is also important, both in the sense that unauthorized intentional changes to individual data are undesirable, and in the sense that data changes caused by reboots or unexpected power outages are undesirable.

[0014]

[0026] Other factors also play a role in this motivational situation. Specifically, cost concerns discourage or discourage simply putting all data into NVDIMMs or similar battery-backed high-speed random access memories. Even if it were possible to put all data into NVDIMMs, it would not address the risk to data integrity that would result from a battery failure. Battery backups are not as reliable as, for example, storage on a hard disk or DVD or tape or some similar non-volatile medium. Battery life varies over time. Data could be replicated across multiple NVDIMM devices for added safety, but again, cost is a key factor. Furthermore, the level of acceptable risk to data integrity is not always constant. For example, different customers may well have different service level agreements (SLAs) or other service targets or guarantees.

[0015]

[0027] In short, some of the data preservation tools and techniques described herein were motivated in part by technical challenges presented in efforts to preserve large sets of machine learning data, although those skilled in the art will recognize that the teachings provided herein have beneficial applicability in many other technical scenarios as well.

[0016]

[0028] Certain embodiments described herein implement a combination of operational steps and device characteristics to preserve data against integrity threats. Data is divided into critical and non-critical data based on criteria such as customer requirements, workload criticality, or virtual machine criticality. Critical data is stored at physical addresses in the battery-backed memory and is flushed prior to non-critical data. Certain embodiments recognize the risk that the battery power may not be sufficient to safely flush both the critical and non-critical data, and address this risk by, for example, keeping a redundant copy of the non-critical data in case the non-critical data is not flushed in time. The aperture of the battery-backed memory is sized according to the battery characteristics and may be resized as conditions change. Defragging is performed to optimize aperture usage, particularly within the portion holding the critical data. Other aspects of the innovative data preservation are also discussed herein.

[0017]

[0029] The embodiments described herein may be viewed in a broader context by some. To give an example, concepts such as allocation, data, order, power, storage, and request may be considered relevant to a particular embodiment. However, the availability of a broad context does not claim, and does not claim, exclusive rights to abstract ideas herein. Rather, this disclosure focuses on providing embodiments that are appropriately embodied to have a technical effect of fully or partially solving a particular technical problem, such as how to balance the cost of NVDIMMs against service level guarantee data availability requirements. Other configured storage media, systems, and processes involving allocation, data, order, power, storage, or request are not within the scope of this application. Thus, by properly understanding this disclosure, ambiguity, mere abstraction, lack of technical character, and attendant proof problems are also avoided.

[0018]

[0030] More generally, those skilled in the art will recognize that not every part of the disclosure, or every particular detail herein, necessarily must meet legal criteria such as enablement, written description, or best mode. Also, the embodiments are not limited to the particular motivating examples, requests, responses, operating systems or other kernels, software development environments, interface standards, software processes, development tools, identifiers, files, data structures, notations, control flows, pseudocode, naming conventions, node architectures, or other implementation options described herein. Any apparent discrepancy with any other patent disclosure, even from the owner of the innovation, is of no use in interpreting the scope of the claims presented in this disclosure.

[0019]

[0031] Technical features

[0032] The technical advantages of the embodiments described herein will be apparent to those skilled in the art and will be apparent to the broader attentive reader in a number of ways. Certain embodiments address technical activities such as memory allocation, flushing data from volatile storage to non-volatile storage, sizing or resizing memory apertures based on battery characteristics, and memory defragmentation, each of which are deeply rooted in computing technology. Some of the technical mechanisms discussed include, for example, NVDIMM or other battery-backed memory, Unified Extensible Firmware Interface, compute nodes, storage nodes, memory maps, flash orders, and virtual machines. Some of the technical effects discussed include, for example, increasing the likelihood that critical data will be preserved against integrity threats, reducing reliance on data redundancy to perform data preservation, and efficiently sizing the aperture of a battery-backed memory taking into account criteria such as battery characteristics and the amount of memory to reserve for non-critical data. In other words, purely psychological processes are expressly excluded. Certain embodiments improve the functioning of computing systems and services by preserving data against integrity threats while balancing criteria such as NVDIMM cost, customer service level assurance requirements, and node storage availability for data redundancy. Other advantages based on the technical properties of the present teachings will be apparent to those skilled in the art from the description provided.

[0020]

[0033] Acronyms, Abbreviations, Names, and Symbols

[0034] Some acronyms, abbreviations, names, and symbols are defined below. Others are defined elsewhere in this specification or do not require definition here in order to be understood by those of ordinary skill in the art.

[0021]

[0035] ACPI: Advanced Configuration and Power Interface

[0036] ALU: Arithmetic and Logic Unit

[0037] API: Application Program Interface

[0038] BIOS: Basic Input / Output System

[0039] BMC: Baseboard Management Controller

[0040] CD:Compact Disc

[0041] CPU: Central Processing Unit

[0042] EFI: Extensible Firmware Interface

[0043] DRAM: Dynamic Random Access Memory

[0044] DVD: Digital Versatile Disc or Digital Video Disc

[0045] FPGA: Field Programmable Gate Array

[0046] FPU: Floating Point Processing Unit

[0047] GPU: Graphical Processing Unit

[0048] GUI: Graphical User Interface

[0049] HDD: Hard Disk Drive (e.g. solid state, electromechanical, optical)

[0050] IaaS or IAAS: Infrastructure as a Service

[0051] ID: Identification or identity information

[0052] IoT: Internet of Things

[0053] LAN: Local Area Network

[0054] NVDIMM: Non-Volatile Dual In-line Memory Module

[0055] NVM e: Non-Volatile Memory Express

[0056] OS: Operating System

[0057] P aa S or PAAS: Platform as a Service

[0058] RAM: Random Access Memory

[0059] ROM: Read Only Memory

[0060] SATA: Serial ATA (Computer Bus Interface)

[0061] SLA: Service Level Agreement

[0062] SMM: System Management Mode

[0063] TCP / IP: Transmission Control Protocol / Internet Protocol

[0064] UEFI: Unified Extensible Firmware Interface

[0065] VM: Virtual Machine

[0066] WAN: Wide Area Network

[0067] Some additional terminology

[0022]

[0068] Reference is made herein to exemplary embodiments as illustrated in the drawings, and specific language is used herein to describe the same. However, alterations of the features illustrated herein and further modifications, as well as other technical applications of the abstract principles illustrated by the specific embodiments herein, will occur to those skilled in the relevant art(s) and having access to the present disclosure, and are deemed to fall within the scope of the appended claims.

[0023]

[0069] In this disclosure, the meanings of terms are clarified, and the claims should be read with careful attention to these clarifications. Specific examples are given, but those skilled in the relevant art(s) will understand that other examples may also fit the meaning of the terms used and fall within the scope of one or more claims. Terms do not necessarily have the same meanings in this specification as they have in general usage (especially outside of technology), in a particular industry usage, or in a particular dictionary or set of dictionaries. Reference numbers may be used with various wordings to help indicate the depth of a term. The absence of a reference number in a given text does not necessarily mean that the contents of a figure are not discussed by that text. The inventor asserts and exercises the right to a specific and selected lexicography. Terms in quotation marks are explicitly defined, but terms may also be implicitly defined without the use of quotation marks. Terms may also be defined, either explicitly or implicitly, in the detailed description herein and / or elsewhere in the application.

[0024]

[0070] As used herein, a "computer system" (also known as a "computing system") may include, for example, one or more servers, motherboards, processing nodes, laptops, tablets, personal computers (handheld or otherwise), personal digital assistants, smartphones, smart watches, smart bands, cell phones or other mobile devices having at least a processor and memory, video game systems, augmented reality systems, holographic projection systems, televisions, wearable computing systems, and / or other device(s) that provide one or more processors that are controlled at least in part by instructions. The instructions may be in the form of firmware or other software in memory and / or specialized circuitry.

[0025]

[0071] A "multithreaded" computer system is one that supports multiple threads of execution. The term "thread" should be understood to include code that can be scheduled or that is subject to scheduling and possibly synchronization. Threads may be known by other names outside of this disclosure, such as, for example, a "task," a "process," or a "coroutine." However, a distinction is made herein between threads and processes, with threads defining paths of execution within a process. Also, the threads of a process share a given address space, while different processes have different respective address spaces. The threads of a process may execute in parallel, sequentially, or a combination of parallel and sequential execution (e.g., time-sliced).

[0026]

[0072] A "processor" is a thread processing unit, such as a core in a simultaneous multithreading implementation. A processor includes hardware. A given chip can hold one or more processors. Processors can be general purpose or can be individually tailored for a specific use, such as vector processing, graphics processing, signal processing, floating point arithmetic processing, cryptography, I / O processing, machine learning, etc.

[0027]

[0073] "Kernel" includes operating systems, hypervisors, virtual machines, BIOS or UI code, and similar hardware interface software.

[0028]

[0074] "Code" means processor instructions, data (including constants, variables, and data structures), or both instructions and data. "Code" and "software" are used interchangeably herein. Executable code, interpreted code, and firmware are some examples of code.

[0029]

[0075] "Program" is used broadly herein to include applications, kernels, drivers, interrupt handlers, firmware, state machines, libraries, and other code written by a programmer (also called a developer) and / or automatically generated.

[0030]

[0076] "Service" means the provision of consumable programs in a cloud computing environment or other network or computing system environment that provides resources or access to resources to multiple programs.

[0031]

[0077] "Cloud" means pooled computing, storage, and networking resources that are elastically available for pay-per-use, on-demand services. Clouds may be private, public, community, or hybrid, and cloud services may be provided as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), Software as a Service (SaaS), or other services. Unless otherwise specified, any discussion of reading from or writing to a file may include reading / writing a local file, or reading / writing over a network, which may be a cloud network or other network, or both (reading / writing locally and over a network).

[0032]

[0078] "IoT" or "Internet of Things" means any networked collection of addressable embedded computing nodes. Such nodes are examples of computer systems as defined herein that also have at least two of the following characteristics: (a) a non-local human readable display; (b) a non-local keyboard; (c) the primary source of input is a sensor that tracks the source of non-verbal data; (d) a non-local rotating disk storage - RAM or ROM chips provide local-only memory; (e) a drive other than a CD or DVD; (f) embedded in a household appliance or fixture; (g) embedded in an implantable or wearable medical device; (h) embedded in a vehicle; (i) embedded in a process automation control system; or (j) designed for one of the following: environmental monitoring, urban infrastructure monitoring, industrial equipment monitoring, energy usage monitoring, human or animal health monitoring, physical security protection, or physical transportation system monitoring. IoT storage may be subject to unauthorized access through the cloud, through other networks, or through direct local access attempts.

[0033]

[0079] In some situations, for example in discussions of cybersecurity consisting of the confidentiality-integrity-availability triad, it is reasonable to distinguish between the meaning of "data availability" and the meaning of "data integrity." However, as used herein, "data integrity" is meant to encompass both the ability to access data and the ability to access data with the correct value. That is, power loss to a storage device, overwriting of values, irreversible encryption or encoding or compression, tampering, misplacement, scrambling, inaccessibility, and data value uncertainty are all examples of threats to data integrity.

[0034]

[0080] Unless otherwise expressly stated, as used herein, "defrag" refers to flash defragmentation ("defrag" is an abbreviation of "defragment"). Flash defragmentation is the process of reducing the amount of unallocated memory near the flash end of the aperture, or increasing the distance of unallocated memory from the flash end of the aperture, or both. Flash defragmentation may involve, for example, copying data, or moving data, or inserting data into memory that has not yet been allocated.

[0035]

[0081] "Flash defrag" differs from "consolidation defrag" in that they have different goals. Flash defrag has the goal of prioritizing the flushing of critical data over non-critical data, or over the flushing of garbage or irrelevant data in unallocated memory. Consolidation defrag, in contrast, has the goal of reducing the time it takes to pull data from storage, specifically from spinning disk platters or from tape, for use. Consolidation defrag attempts to consolidate all data belonging to a particular owner, for example all data in a particular file. These are not equivalent processes.

[0036]

[0082] For example, let A or B represent critical data owned by A or B, respectively, underlines represent unallocated space in memory, and memory aperture bounds are represented by curly braces. Then a change from {ABA_} to {AAB_} is an integrated defrag, not a flash defrag. Also, a change from {A_B_} to {AB__} is a flash defrag (if the flash tip is at the left edge), not an integrated defrag.

[0037]

[0083] As used herein, "include" allows for additional elements (i.e., include means comprise) unless otherwise specified.

[0038]

[0084] "Optimize" means to improve, but not necessarily to perfection. For example, further improvements may be possible in a program or algorithm that has been optimized.

[0039]

[0085] "Process" is sometimes used herein as a term in the computing science arts, and in its technical sense encompasses a computational resource user. A computational resource user may also include or be referred to as, for example, a coroutine, a thread, a task, an interrupt handler, an application process, a kernel process, a procedure, or an object method. In practice, a "process" is a computational entity identified by a system utility such as Windows® Task Manager, Linux® ps, or similar utilities in other operating system environments (trademarks of Microsoft Corporation, Linus, and Torvalds, respectively). "Process" is also used herein as a term of art in patent law, for example, when writing a process claim as opposed to a system claim or a product (configured storage medium) claim. Similarly, "method" is sometimes used herein as a term of art in the computing science arts (a type of "routine") and as a term of art in patent law ("process"). "Process" and "method" are used interchangeably herein in the patent law sense. However, one of ordinary skill in the art will understand which meaning is intended in a particular case, and will further understand that a given claimed process or method (in the patent law sense) may be implemented using one or more processes or methods (in the computing science sense).

[0040]

[0086] "Automatically" means through the use of automation (e.g., general-purpose computing hardware configured by software to perform certain operations and technical effects as discussed herein) as opposed to without automation. Specifically, steps performed "automatically" are not performed manually on paper or through human thought, but may be initiated by a human or guided interactively by a human. Automatic steps are performed by a machine to achieve one or more technical effects that would not be achieved without the technical interaction thus provided.

[0041]

[0087] Those skilled in the art will understand that the technical effect is a presumptive objective of a technical embodiment. For example, while an embodiment involves computation, the mere fact that some computation can be performed without a technical component (e.g., with paper and pencil, or even as a mental process) does not remove the presence of the technical effect and does not change the specific and technical nature of the embodiment. Data storage operations, such as identifying unallocated memory, copying data to memory, marking memory as allocated, flushing data from volatile storage to non-volatile storage, and many other operations discussed herein, are considered to be digital in nature. The human mind cannot directly interface with a CPU or other processor, or with RAM or other digital storage, to read and write the necessary data and perform the data storage steps taught herein. This will all be well understood by those skilled in the art in light of the present disclosure, but others may need to be made aware of it or made aware of it.

[0042]

[0088] "Computationally" similarly means that a computing device (at least a processor + memory) is used, and excludes obtaining a result solely by human thought or merely by human action. For example, performing arithmetic with paper and pencil is not performing arithmetic computationally as understood herein. The computational result may be faster, broader, deeper, more accurate, more consistent, more comprehensive, and / or otherwise provide a technical effect beyond the scope of human performance alone. A "computational step" is a step that is performed computationally. Neither "automatically" nor "computationally" necessarily means "immediately." "Computationally" and "automatically" are used interchangeably herein.

[0043]

[0089] "Proactively" means without a direct request from a user. In fact, a user may not even be aware that a proactive step according to an embodiment is possible until the results of that step are presented to the user. Unless otherwise specified, any computational and / or automated steps described herein may be performed proactively.

[0044]

[0090] Throughout this document, the use of the optional plural "(s), (es), or (ies)" means that there is one or more of the indicated feature. For example, "processor(s)" means "one or more processors" or, equivalently, "at least one processor."

[0045]

[0091] Solely under U.S. law and practice, the use of the word "step" in this specification, claims, or elsewhere is not intended to invoke means-plus-function, step-plus-function, or 35 U.S.C. § 112(6) / 112(f) claim construction, and any assumption to that effect is hereby expressly rebutted.

[0046]

[0092] As far as U.S. law and practice is concerned, claims are not intended to invoke a means-plus-function interpretation unless they use the phrase "means for." Any claim language that is intended to be interpreted as a means-plus-function language will expressly state that intent by using the phrase "means for." When a means-plus-function interpretation is applied, whether by the use of "means for" and / or by a court's legal interpretation of the claim language, the means described in the specification for a given noun or a given verb should be understood to be linked to that claim language and further linked together herein by appearing within the same block of a block diagram in the drawings, by designation by the same or similar name, by denotation by the same reference number, by functional relationships depicted in any of the figures, or by functional relationships pointed out in the body of this disclosure. For example, if a claim limitation recites a "zac widget" and that claim limitation is subject to means-plus-function interpretation, all structure identified anywhere in this specification in any figure block, paragraph, or example that recites "Zac widget" or that is linked together by any reference number assigned to Zac widget or that is disclosed as having a functional relationship to the structure or operation of Zac widget shall be considered to be, at a minimum, part of the structure identified in the application to Zac widget and help define a set of equivalents for the Zac widget structure.

[0047]

[0093] Those skilled in the art will appreciate that the disclosure of the innovation discusses various data values ​​and data structures, and that such items constitute memory by virtue of being resident in memory (RAM, disk, etc.), and that the disclosure of the innovation also discusses various algorithmic steps, and that these algorithmic steps may, in a given embodiment, be embodied in executable code, and that such code may also reside in memory, and that this code effectively constitutes any general-purpose processor that executes it, thereby functionally transforming the general-purpose processor into a special-purpose processor that is special-purpose hardware.

[0048]

[0094] Thus, one skilled in the art would not make the mistake of treating (a) a memory recited in a claim and (b) a data structure or data value or code recited in that claim as non-overlapping items. Even if a claim does not explicitly recite the residency of any data structure or data value or piece of code, it will be implicitly understood that the recited data structure and data value and code reside in the memory. Thus, an explicit recitation of such residency is not necessary. However, they are not prohibited, and one or two choices may be recited for emphasis, and not thereby excluding all other data values ​​and data structures and code from residency. Similarly, it will be implicitly understood that a code function recited in a claim constitutes a processor, regardless of whether that constituent quality is explicitly recited in the claim.

[0049]

[0095] Throughout this document, unless otherwise stated, whenever a step in a process is mentioned, it is assumed that the step may be performed directly by the party of interest and / or indirectly by the party of interest through an intermediary mechanism and / or entity and still fall within the scope of the step, i.e., direct performance of a step by the party of interest is not required unless direct performance is an explicitly stated requirement. For example, steps involving actions by a subject element upon a destination or other subject, such as allocating, copying, defragmenting, designating, detecting, determining, destroying, executing, failing, flushing, identifying, retaining, marking, moving, placing, saving, providing, receiving, reducing, reserving, residing, resizing, restoring, saving, sending, designating, storing (and allocates, allocated, copies, copied, etc.), may involve intermediary actions by some other party, such as transferring, copying, uploading, downloading, encoding, decoding, compressing, decompressing, encrypting, decrypting, authenticating, invoking, and others, including any actions described in this document, but may still be understood to be performed directly by the subject element.

[0050]

[0096] Whenever data or instructions are mentioned, it is implicitly understood that these items constitute computer-readable memory and / or computer-readable storage media, thereby converting them into specific articles, as opposed to, for example, a simple presence on paper, a presence in the human mind, or merely a signal propagating on a wire. For patent protection purposes only in the United States, memory or other computer-readable storage media are not propagating signals, carrier waves, or mere energy, which fall outside the scope of patentable subject matter under the United States Patent and Trademark Office (USPTO) interpretation of In re Nuijten. There are no claims in the United States that cover signals per se or mere energy, and in light of this disclosure, any claim construction that asserts otherwise is face unreasonable. Unless specifically stated otherwise in claims granted outside the United States, no claim covers signals per se or mere energy.

[0051]

[0097] Moreover, notwithstanding anything expressly stated to the contrary in this specification, it will be understood that a clear distinction is made between (a) computer-readable storage media and computer-readable memory, on the one hand, and (b) transmission media, also called signal media, on the other hand. Transmission media are propagating signal or carrier wave computer-readable media. In contrast, computer-readable storage media and computer-readable memory are neither propagating signal nor carrier wave computer-readable media. Unless otherwise stated in the claims, "computer-readable medium" means computer-readable storage media, and not the propagating signal itself or just energy.

[0052]

[0098] An "embodiment" is an example herein. The term "embodiment" is not interchangeable with "the invention." An embodiment may freely share or borrow features to create other embodiments (provided the result is operable), even if the resulting combination of features is not itself explicitly described herein. The requirement to explicitly and individually describe every permissible combination is unnecessary for a person of ordinary skill in the art and runs counter to the policy of acknowledging that patent specifications are written for a reader who is skilled in the art. Formal combinatorial calculations and informal common intuition regarding the number of possible combinations that can be obtained from even a small number of combinatorial features indicate that a large number of combinations of features exist for the embodiments described herein. Thus, the requirement to explicitly describe every combination runs counter to the policy of requiring patent specifications to be concise and for the reader to be knowledgeable in the relevant technical field.

[0053]

[0099] List of Reference Numbers

[0100] The following lists are provided for convenience and for the aid of the drawings, and are presented as part of the text of the specification describing the innovation with reference to multiple items. Items not listed in this list may, however, be part of a given embodiment. To facilitate readability of the text, a given reference number is listed near some, but not all, of the items referenced in the text. The same reference number may be used to refer to different examples or instances of a given item. The list of reference numbers is as follows:

[0054]

[0101] 100 Operating Environment, also known as Computing Environment

[0102] 102 Computer systems, also called calculation systems or computing systems

[0103] 104 users

[0104] 106 Peripheral Equipment

[0105] 108 For example, entire networks, including LANs, WANs, software-defined networks, clouds, and other wired or wireless networks

[0106] 110 Processor

[0107] 112 Computer Read Possible storage media, e.g. RAM, hard disk.

[0055]

[0108] 114 Removable Computer-Readable Storage Medium

[0109] 116 Processor executable instructions that may reside on a removable storage medium or in other memory (volatile or non-volatile or both).

[0056]

[0110] 118 Data

[0111] 120 Kernel(s), e.g., operating system(s), BIOS, UEFI, device drivers

[0112] 122 Tools, such as virus protection software, firewalls, packet sniffer software, intrusion detection systems, intrusion prevention systems, debuggers, profilers, compilers, interpreters, decompilers, assemblers, disassemblers, source code editors, auto-completion software, simulators, fuzzers, repository access tools, version control tools, optimizers, collaboration tools, software development tools and tool suites, hardware development tools and tool suites, diagnostics, etc.

[0057]

[0113] 124 applications, such as word processors, web browsers, spreadsheets, games, email tools.

[0114] 126 Display screen. Also called "display".

[0058]

[0115] 128 computing hardware not otherwise associated with the other reference numerals 106, 108, 110, 112, 114. In FIG. 8, numeral 128 also denotes processor 110 and memory 112 hardware.

[0059]

[0116] 200 For example, the computational function in a computation node.

[0117] 202 compute nodes.

[0118] 204 A data source, e.g., a program that outputs or otherwise generates data.

[0060]

[0119] 206 Virtual Machine: A computing construct that performs hardware virtualization and includes an operating system, and may include, for example, working memory resources, CPU resources, IO resources, and non-volatile memory.

[0061]

[0120] 208 System code, e.g., operating system, hypervisor, BIOS, UEFI, firmware.

[0121] 210 For example, a storage function in a storage node.

[0062]

[0122] 212 Storage Nodes

[0123] 214 Non-volatile storage, e.g., NVDIMM, disk, flash memory.

[0063]

[0124] 216 Battery

[0125] 218 Battery Backup Memory

[0126] 220 Memory aperture, which may refer to a portion of a physical address space associated with a digital data storage location in a physical device, or may refer to the digital data storage location itself.

[0064]

[0127] 222 Data storage request

[0128] 224 Response to data storage request

[0129] 226 Data duplication

[0130] 300 Aspects of the Computing Environment

[0131] 302 Cloud, which may also be called a "cloud computing environment."

[0065]

[0132] 304 Data storage circuitry. Includes electronic circuitry and any firmware that controls these electronic circuits or uses the data storage circuitry, but does not necessarily include non-volatile storage to which data is flashed.

[0066]

[0133] 306 Data storage subsystem, which includes non-volatile storage to which data is flushed.

[0134] 308 Hypervisor.

[0067]

[0135] 310 UEFI. May refer to Unified Extensible Firmware Interface, or firmware having such an interface, and may include or provide functionality for, for example, SEC (Security), PEI (Pre-EFI Initialization), DXE (Driver Execution Environment), BDS (Boot Device Selection), and SMM (System Management Mode).

[0068]

[0136] 312 Flash Order

[0137] 314 Data Integrity Threats

[0138] 316 Storage relationships between two or more nodes

[0139] 318 Data Criticality. Can refer to whether data is designated as critical or whether data meets the criteria to be designated as critical even if it has not yet been so designated. This can be a Boolean or other indication, e.g., a value within a range of two or more values ​​that indicates how to prioritize the flushing of the data.

[0069]

[0140] 402 Volatile memory, e.g., the DRAM itself.

[0141] 404 NVDIMM

[0142] 406 Battery Characteristics

[0143] 408 defragger (memory defragmentation) code.

[0070]

[0144] 410 The defragmented portion of memory, i.e. the portion that is subject to defragmentation.

[0145] 412 A data set that contains critical data (and may also have some non-critical data).

[0071]

[0146] 414 Memory allocated (or allocated to hold) critical data 412.

[0147] 416 A data set that does not contain any critical data.

[0072]

[0148] 418 Memory allocated (or allocated to) hold non-critical data 416.

[0149] 420 firmware.

[0073]

[0150] 422 allocator software, i.e., software that identifies unallocated space in an aperture, places critical data therein toward the flush leading edge of the aperture, and marks this space as allocated. The allocator can also defragment the aperture. The allocator can also identify space for non-critical data and place non-critical data toward the flush trailing edge of the aperture.

[0074]

[0151] 424 A physical address in memory 112. It can refer to the address itself (e.g., 0x0000) or a storage location (e.g., a memory cell) at the address.

[0075]

[0152] 502 Battery capacity, e.g., milliamp-hours.

[0153] 504 A reliability value, e.g., mean time between failures, or remaining predicted life, or an enumerated value (e.g., 2 on a scale where 10 is best and 0 is worst), or a reliability grade (e.g., medium reliability).

[0076]

[0154] 506 The age of the battery, e.g., time since installation, number of charging cycles it has undergone, number of times it has been used between flashes.

[0155] 600 Examples of data criticality criteria.

[0077]

[0156] 602 Data Criticality Criteria.

[0157] 604 The priority assigned to the virtual machine that generated the data.

[0158] 606 A customer, e.g., a cloud tenant.

[0078]

[0159] 608 A status that can be assigned to a customer, for example, to be made visible in an SLA.

[0160] 610 Workload, e.g., processing tasks that are assigned to a particular compute node.

[0079]

[0161] 612 The priority assigned to the workload that contains the data, or from the run in which the data was generated.

[0162] 702 Physical machines, e.g., processors, circuits, chips, batteries, displays, power supplies, housings, etc., including any embedded firmware or software stored in the non-volatile memory of the physical machines. Computing system 102 includes one or more physical machines 702.

[0080]

[0163] 704 Nodes in a cloud or other network. Compute nodes 202 and storage nodes 212 are examples of nodes 704.

[0164] 706 Containers, i.e. computing constructs that provide user space virtualization and do not themselves contain an operating system, but nevertheless rely on an operating system to run.

[0081]

[0165] 800 A computing system having one or more compute nodes networked with one or more storage nodes and equipped with data storage capabilities as taught herein.

[0082]

[0166] 802 I / O (input / output) hardware and software. For example, ports, connectors, sockets, network interface cards. Also known as "IO."

[0083]

[0167] 804 Top of memory.

[0168] 806 Aperture Flash Tip.

[0169] Flash trailing edge of 808 aperture.

[0084]

[0170] Unallocated space in 810 aperture.

[0171] 812 A portion of memory within the same memory device but outside the aperture.

[0085]

[0172] 814 A logical mapping between virtual and physical address spaces.

[0173] 900 A computing system including a system having an operating system or container running a virtual machine and equipped with data storage capabilities as taught herein.

[0086]

[0174] 902 Dedicated non-volatile storage 112 to receive the data to be flashed.

[0175] 904 A memory region that is mapped into the dedicated storage 902. The battery backed memory aperture 220 is an example of a region 904.

[0087]

[0176] 906 Operating System.

[0177] 908 An evacuation operation that copies data from volatile memory to dedicated non-volatile storage that backs up the volatile memory. The evacuation can be done intentionally without any immediate threat to the integrity of the data being evacuated, or it can be a flush done in response to an imminent or current threat.

[0088]

[0178] 910 A restore operation which copies data from dedicated non-volatile storage to volatile memory backed by this dedicated non-volatile storage.

[0179] 912 execution of a program. Can also refer to the act of executing a program or an instance of a program operation.

[0089]

[0180] 1000 Flowchart 1000 also refers to a data storage method as illustrated by or according to the flow chart of FIG.

[0090]

[0181] 1002 A request is received to store critical data, for example as a result of a malloc() call, a constructor invocation, evacuation of a file or block or blob, or other operation which requires the storage of data for which a storage location (byte, page, etc.) has not yet been allocated.

[0091]

[0182] 1004 For example, system memory management software may use a free list, a bit vector indicating free chunks, or a similar mechanism to identify unallocated portions of memory.

[0092]

[0183] 1006 A portion of memory may be marked as allocated, for example, by setting a bit in a data structure that tracks memory allocations.

[0184] 1008 For example, a copy of the data is put into memory by performing a block copy or the like function used by memcpy().

[0093]

[0185] 1010 detects threats to data integrity, for example, by receiving a power command on an ACPI compliant system or by receiving a signal from a voltage level monitoring circuit.

[0094]

[0186] 1012 For example, data is flushed from volatile storage to dedicated non-volatile backup storage by copying the data from the DRAM portion of the NVDIMM to the flash portion of the NVDIMM.

[0095]

[0187] 1100 Flowchart 1100 also refers to the storage method illustrated by the flowchart of FIG. 11 (which incorporates the steps of FIG. 9 and the steps of FIG. 10) or to a data storage method according to the flowchart of FIG.

[0096]

[0188] 1102 Preserve the data, i.e., maintain the integrity of at least one copy of the data.

[0189] 1104 A data storage request is received.

[0097]

[0190] 1106 Determine or specify the size or location of the aperture 220.

[0191] 1108 The size of the aperture 220, for example, in bytes or blocks.

[0098]

[0192] 1110 The location of the aperture 220. For example, the offset of both ends of the aperture from the lowest physical address of the memory device that contains the aperture.

[0193] 1112 Send data storage request.

[0099]

[0194] 1114 Re-determine aperture size.

[0195] 1115 Move the aperture.

[0196] 1116 Reserve at least the specified portion of the aperture to hold only non-critical data.

[0100]

[0197] 1118 The reserved portion of the aperture, which can be specified by an absolute or relative memory address, or as a percentage of the total aperture size.

[0198] 1120 Defragment at least a portion of an aperture by copying data or moving data or inserting data into memory not previously allocated, thereby reducing the amount of memory that is unallocated, or increasing the distance of unallocated memory from the flush end of the aperture, or both. For example, this may be referred to as a "flash defrag" to distinguish it from other uses of "defrag" that involve consolidating portions of memory that belong to the same user or the same process or the same file.

[0101]

[0199] 1122 fails to flush some of the data from the aperture, resulting in at least partial loss of data unless another copy of the data is available elsewhere than the aperture, and unless the remainder of the data is available elsewhere than in the flushed non-volatile storage.

[0102]

[0200] 1124 Residing in a physical machine, e.g., stored in memory (volatile or non-volatile) that is part of the physical machine.

[0201] 1126 Keep a copy of the data somewhere outside the aperture and outside the non-volatile storage where the aperture is flushed.

[0103]

[0202] 1128 Move the data.

[0203] 1130 Make space, i.e. change a given piece of memory from allocated to free.

[0104]

[0204] 1132 Designating Data as Critical When data is treated as a unit, designating any part of that unit as critical also designates the unit as critical.

[0105]

[0205] 1134 Reduce the unallocated amount between the allocated portions.

[0206] 1136 Eliminate the unallocated space between the allocated parts.

[0207] 1138 Increase the chances of preserving critical data against threats.

[0106]

[0208] 1140 The likelihood of preserving critical data against a threat. May be a measured probability or a fair assessment.

[0209] 1142 Execute firmware. An example of execution 912.

[0107]

[0210] 1144 Any step discussed in this disclosure that is not assigned any other reference number.

[0211] Operating environment

[0212] With reference to FIG. 1, an operating environment 100 of an embodiment includes at least one computer system 102. The computer system 102 may or may not be a multiprocessor computer system. The operating environment may include one or more machines within a given computer system, which may be clustered, networked in a client-server relationship, and / or networked in a peer-to-peer fashion in a cloud. An individual machine is a computer system, and a group of machines working together is also a computer system. A given computer system 102 may be configured, for example, with applications for end users, as a server for administrators, as a distributed processing node, and / or in other ways.

[0108]

[0213] A human user 104 can interact with the computer system 102 through typed text, touch, voice, movement, computer vision, gestures, and / or other forms of I / O by using a display, keyboard, and other peripherals 106. The screen 126 may be a removable peripheral 106 or may be an integrated part of the system 102. A user interface can facilitate interaction between an embodiment and one or more human users. The user interface may include a command line interface, a graphical user interface (GUI), a natural user interface (NUI), a voice command interface, and / or other user interface (UI) presentations, which may be presented as separate options or may be integrated.

[0109]

[0214] System administrators, network administrators, cloud administrators, security analysts and other security personnel, operational personnel, developers, engineers, and end users are each a particular type of user 104. Instead of one or more humans, automated agents, scripts, playback software, devices, etc. operating to facilitate testing of the system 102 may also be users 104. Depending on their detachability from the processor 110, storage devices and / or networking devices may be considered peripheral equipment in some embodiments and part of the system 102 in other embodiments. Other computer systems not shown in FIG. 1 may also interact with the computer system 102 in a technical manner or with other system embodiments using one or more connections to the network 108, for example, through network interface equipment.

[0110]

[0215] Each computer system 102 includes at least one processor 110. The computer system 102, like other suitable systems, also includes one or more computer-readable storage media 112. The storage media 112 may be of different physical types. The storage media 112 may be volatile memory, non-volatile memory, fixed in place media, removable media, magnetic media, optical media, solid state media, and / or other types of physically durable storage media (as opposed to merely propagating signals or merely energy). In particular, a configured storage medium 114, such as a portable (i.e., external) hard drive, CD, DVD, memory stick, or other removable non-volatile memory medium, when inserted or otherwise installed, can functionally become a technical part of the computer system and make its contents accessible for interaction with or use by the processor 110. The removable configured storage medium 114 is an example of a computer-readable storage medium 112. Some other examples of computer readable storage medium 112 include built-in RAM, ROM, hard disks, and other memory storage devices that are not easily removable by the user 104. In accordance with current U.S. patent requirements, neither computer readable medium, nor computer readable storage medium, nor computer readable memory, are signals per se or mere energy under any pending or granted claims in the United States.

[0111]

[0216] The storage medium 114 comprises binary instructions 116 executable by the processor 110. "Executable" is used broadly herein to include, for example, machine code, interpretable code, bytecode, and / or code executing on a virtual machine. The storage medium 114 also comprises data 118 that is created, modified, referenced, and / or otherwise used to effect the technical effects of executing the instructions 116. The instructions 116 and data 118 comprise the memory or other storage medium 114 in which they reside. When the memory or other computer-readable storage medium is a functional part of a given computer system, the instructions 116 and data 118 also comprise the computer system. In some embodiments, some of the data 118 represent real-world items, such as product characteristics, inventory, physical measurements, settings, images, readings, targets, quantities, etc. Also, such data may be transformed by backing up, restoring, committing, aborting, reformatting, and / or other technical processes.

[0112]

[0217] Although embodiments may be described as being implemented as software instructions executed by one or more processors in a computing device (e.g., a general purpose computer, a server, or a cluster), such description is not meant to be exhaustive of all possible embodiments. Those skilled in the art will appreciate that the same or similar functionality can often also be implemented, in whole or in part, directly in hardware logic to achieve the same or similar technical effect. Instead of or in addition to a software implementation, the technical functionality described herein may be performed, at least in part, by one or more hardware logic components. The hardware logic components may include firmware, be controlled by firmware, or both. For example, embodiments may include hardware logic components 110, 128 such as field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on chip components (SOCs), complex programmable logic devices (CPLDs), and similar components, and do not exclude other implementations. The components of the embodiments may also be grouped into interacting functional modules, for example, based on their inputs, outputs, and / or their technical effect.

[0113]

[0218] In addition to the processor 110 (e.g., CPU, ALU, FPU, and / or GPU), memory / storage media 112, and display 126, the operating environment may also include other hardware 128, such as batteries, buses, power supplies, wired and wireless network interface cards, to name a few. The nouns "screen" and "display" are used interchangeably herein. The display 126 may include one or more touch screens, screens that respond to input from a pen or tablet, or screens that operate only for output. In some embodiments, peripherals 106, such as human user I / O devices (screens, keyboards, mice, tablets, microphones, speakers, motion sensors, etc.), may also be present in operative communication with one or more processors 110 and memory.

[0114]

[0219] In one embodiment, the system includes multiple computers connected by wired and / or wireless network 108. Networking interface equipment 128 can provide access to network 108 using network components such as, for example, packet-switched network interface cards, wireless transceivers, or telephone network interfaces. These components can exist within a given computer system. Virtualization of networking interface equipment and other network components such as switches or routers or firewalls can also exist, for example, in software-defined networks, or sandboxed or other secure cloud computing environments. A given embodiment can also communicate technical data and / or technical instructions by direct memory access, removable non-volatile storage media, or other information storage, retrieval, and / or transmission techniques.

[0115]

[0220] Those skilled in the art will recognize that these aspects, as well as others introduced herein under "Operating Environment," may form part of a given embodiment. The headings in this document are not intended to rigidly classify features into embodiment and non-embodiment feature sets.

[0116]

[0221] One or more items are shown in outline form or in parentheses in the figures to emphasize that they are not necessarily part of some or all embodiments of the illustrated operating environment, but may interoperate with items in the operating environment or in some embodiments as discussed herein. Items not in outline form or without parentheses are not necessarily required to be present in any figure or in any embodiment. In particular, Figure 1 is shown for convenience, and the inclusion of an item in Figure 1 does not imply that the item, or a description of the item's use, was known prior to the current technological innovation.

[0117]

[0222] More about the system

[0223] 1-11, an embodiment uses or provides a feature augmentation system 800 or 900. The feature augmentation helps promote data preservation by implementing a combination of operational steps and device characteristics to preserve data against integrity threats 314. Data 118 is divided into critical data 412 and non-critical data 416, the critical data is stored at physical addresses 424 within apertures 220 of one or more battery-backed memories 218, and the critical data is flushed 1012 before the non-critical data when a threat is detected 1010. The battery-backed memory apertures 220 are sized according to battery characteristics 406 and can be resized as conditions change. A defragger 408 optimizes the use of the apertures to store the critical data 412.

[0118]

[0224] As illustrated by the example architecture of FIG. 2, data 118 may be generated by sources 204, such as virtual machines 206 or containers 706 executing (912) on compute nodes 202. These data sources call or rely on system code 208 to send generated data 118 to storage nodes 212. In this example of FIG. 2, storage nodes 212 have high-speed volatile memory 402, at least a portion of which is battery-backed memory 218. At least a portion of battery-backed memory 218 is designated for use as aperture 220 to help preserve critical data, as taught herein. Storage nodes 212 and compute nodes 202 communicate through requests 222 (e.g., here is data to store. Please return the data stored under identifier Z) and responses 224 (e.g., storage successful, no room in aperture, here is the requested data, or other status or error code). The requests 222 and responses 224 may travel, for example, over a TCP / IP network 108, an Ethernet network 108, a storage area network 108, or a combination thereof.

[0119]

[0225] In general, the compute node-storage node relationship 316 is not necessarily one-to-one. A compute node may process data sent to or received from a storage node, or may communicate with multiple storage nodes. A storage node may store data on behalf of one or more compute nodes. Figure 8, for example, shows a compute node 202 using four storage nodes 212.

[0120]

[0226] 2 and 8, in one embodiment, critical data 412 is not replicated across storage nodes, while non-critical data 416 is replicated across storage nodes. That is, critical data may be preserved against threats primarily or only by flushing the critical data to non-volatile storage, while non-critical data may be preserved against threats primarily or only by replicating it across nodes.

[0121]

[0227] 8, critical data 412 and corresponding allocation areas 414 in aperture 220 belong to virtual machines A, C, and D and are shown with rounded corners in the figure, while non-critical data 416 and corresponding allocation areas 418 in aperture 220 belong to virtual machines B and X and are shown with right angled corners in the figure. Data replicas 226 for data belonging to virtual machines B and X are shown, with storage nodes S1, S2, S3 having replicas of non-critical data 416 for virtual machine B, and storage nodes S1, S2, S4 having replicas of non-critical data 416 for virtual machine X.

[0122]

[0228] 8 and 9 show some examples of system code 208, namely kernel 120 (e.g., hypervisor 308, operating system 906) and firmware (e.g., UEFI firmware 310). The compute node and storage node are computing systems 102 and therefore include hardware. Note that for clarity, hardware other than memory 112 is not specifically called out in FIG. 2, but some examples of hardware, e.g., IO 802 and CPU 110, are shown in FIG. 8 along with memory 112, 218. Memory 218 includes aperture 220, memory outside aperture 812, allocated memory 418 and 414 inside aperture, and unallocated memory 810 inside aperture.

[0123]

[0229] The physical addresses 424 of the battery backed memory 218, or other aperture-containing mapped volatile region 904, may be mapped to virtual addresses by a mapping 814. The mapping 814 provides a level of indirection, which allows the same save 908 and restore 910 firmware to be used with apertures that may have different physical start addresses 806 and physical end addresses 808. The non-volatile memory 112 to which the aperture 220 data is flushed may include flash or other non-volatile storage in the NVDIMM 404, or other dedicated non-volatile storage 902, or both.

[0124]

[0230] 3 illustrates various aspects 300 of a computing environment 100, which are described as appropriate at various points in this disclosure.

[0231] FIG. 4 illustrates an example data storage subsystem 306 that can add innovative data storage capabilities to the computing system 102. The memory of the subsystem 306 includes a battery backed-up memory 218, e.g., an aperture 220 in an NVDIMM 404. As noted in FIGS. 4 and 5, a battery can have characteristics 406 that are tracked or indicated, such as one or more of capacity 502, reliability 504, and age 506. These battery characteristics 406 may overlap or affect each other. For example, reliability may decrease with age. The subsystem 306 may also include volatile memory 402 that is not backed up by a battery 216.

[0125]

[0232] In operation, an aperture 220 has one or more regions 414 that house critical data 412, and an aperture 220 may also have one or more regions 418 that house non-critical data 416. An aperture 220 may house only non-critical data, at least at a given time, such as storage nodes S2, S3, and S4 shown in FIG. 8. However, it is apertures 220 that house critical data that are of interest here. In particular, this disclosure provides information about how critical data may be entered (by initial allocation, or by defragmentation, or both), i.e., stored (1102) in memory 112, to increase the likelihood 1140 of successfully flushing the critical data in response to a data threat.

[0126]

[0233] FIG. 6 illustrates several example criteria 600 that may be applied to distinguish critical data 412 from non-critical data 416. For example, data may be designated critical if it was generated by a virtual machine 206 or container 706 or other digital artifact, which has a high priority 604. Data 118 may be designated 1132 as critical if it was generated by or on behalf of a particular customer 606 who has paid for it, or if it otherwise has a status 608 indicating critical data. Similarly, data may be designated 1132 as critical if it was generated by or on behalf of, or processed by, a particular workload 610 that has a priority or status 612 indicating critical data. These example criteria may be combined in various ways, modified by other criteria 602, or both. By way of example, (a) data may be designated as critical when generated by a high priority virtual machine of a high status customer, but may otherwise be designated as non-critical, (b) data may be designated as critical when generated by any virtual machine of a high status customer executing on a designated secure server blade 702, but may otherwise be designated as non-critical, (c) data may be designated as critical when generated by any customer during a designated time period, but may otherwise be designated as non-critical, etc.

[0127]

[0234] 7 illustrates various examples 700 of a computing system 102, which are described at various points throughout this disclosure.

[0235] An embodiment provides or uses a data storage subsystem 306 in a computing system 102. The data storage subsystem 306 can include a battery backed-up memory 218 having an aperture 220. The aperture 220 has a flush order 312. The flush order 312 is the order in which data is copied from the aperture to the aperture associated non-volatile storage 902 in response to a data integrity threat 314. The flush order defines a flush leading edge 806 of the aperture and a flush trailing edge 808 of the aperture.

[0128]

[0236] The flush order may also be described using terms such as "top-down" or "bottom-up," with the understanding that a larger physical address 424 is above a smaller physical address. Thus, a top-down flush starts at a higher address and works toward a lower address, whereas a bottom-up flush starts at a lower address and works toward a higher address. The flush leading edge 806 of the aperture has a higher address than the flush trailing edge 808 of the aperture in a top-down flush order, and a lower address than the flush trailing edge 808 of the aperture in a bottom-up flush order. FIG. 8 illustrates a bottom-up (lower address to higher address) flush order 312, since the flush leading edge 806 of the aperture has a lower physical address 424 than the flush trailing edge 808 of the aperture.

[0129]

[0237] In one embodiment, the data storage subsystem 306 includes a data storage circuit 304 in operative communication with the battery back-up memory 218. The data storage circuit 304 is configured to perform data storage steps. The data storage steps may include:

[0130] (a) receiving (1002) a request 222 to store a data set A-data 412 in an aperture, the A-data including data designated as critical data;

[0131] (b) Identifying 1004 a portion A-memory 414 of the aperture's unallocated memory 810, where the A-memory is large enough to hold the A-data, and the A-memory has an address 424 that is closer to the aperture's flash leading edge 806 than any other address of any other unallocated memory of the aperture.

[0132] (c) Mark the A-Memory as allocated (1006) and input a copy of the A-Data into the A-Memory (1008). (d) A request 222 is received to store a data set Z-data 416 in an aperture. The Z-data does not include any data designated as critical data.

[0133] (e) Identifying a portion of the aperture's unallocated memory 810, a Z-memory, the Z-memory being large enough to hold the Z-data, the Z-memory having an address 424 closer to the aperture's flush trailing end 808 than any other address of any other unallocated memory of the aperture.

[0134] (f) Mark the Z-Memory as allocated and enter a copy of the Z-Data into the Z-Memory. Here, labels (a) through (f) merely serve as identifiers and thus do not necessarily themselves specify an order of operations. The data preservation subsystem 306 increases the likelihood of successfully flushing 1012 critical data by inputting the critical data to the battery-backed memory before the non-critical data in the flush-order, thereby preserving the critical data. The likelihood of preserving the critical data is higher than in an architecture that stores data in the battery-backed memory without consideration of the criticality of the data, i.e., lacking criticality.

[0135]

[0238] In one embodiment, the battery backed memory aperture 220 resides in or is controlled by a network node 704, referred to herein as X, and the request 222 to store A-data in the aperture comes from a different network node, referred to herein as Y. In FIG. 8, for example, the aperture resides in storage node 212, 704 S1, and the request 222 to store A-data for virtual machine A in the aperture 220 comes from a different network node other than S1, i.e., a computation node. In one embodiment, network node X comprises a storage node in the cloud 302, and network node Y comprises a computation node in the cloud. However, in other embodiments, the nodes may reside in networks other than the cloud.

[0136]

[0239] In one embodiment, network node X and network node Y are related by a storage relationship 316, referred to herein as R, where R is defined such that two nodes M and N are related by R when M stores data on behalf of N. In some cases, R may not be a one-to-one relationship for particular nodes X and Y. For example, a compute node may send critical data for virtual machine VM7 to storage node S7 and send critical data for virtual machine VM8 to storage node S8. Or, storage node S12 may hold critical data from compute node C10 and also hold critical data from a different compute node C11. Of course, many other examples of relationships 316 are possible.

[0137]

[0240] In one embodiment, the battery-backed memory 218 includes an NVDIMM memory 404. For example, the NVDIMM 404 can include an NVDIMM-F flash memory, an NVDIMM-N byte-addressable memory, an NVDIMM-P memory with dynamic RAM and NAND on the same device, an NVDIMM-SW memory, or an NVRAM non-volatile RAM memory. The NVDIMM-SW can also include memory (e.g., DD4RAM), disk, or flash (e.g., SSD partitions), a battery, and a signaling mechanism.

[0138]

[0241] In one embodiment, computing system 102 includes virtual machine 206, where data set A-DATA 412 is sent from virtual machine VM-A to data store subsystem 306, and data set Z-DATA 416 is sent from virtual machine VM-Z to data store subsystem 306. In the example of Figure 8, illustratively, critical data set 412 is sent from virtual machine A to enhanced storage node S1 (and thus to its data store subsystem 306), and non-critical data set 416 is sent from virtual machine B to S1.

[0139]

[0242] Other system embodiments are also described herein, either directly or as being derivable as system versions of the described processes or configured media, and are informed by the extensive discussion herein of computing hardware.

[0140]

[0243] Although specific architectural examples are shown in the figures, embodiments may deviate from these examples. For illustration, items shown in different figures may be included together in one embodiment, items shown in one figure may be omitted, functionality shown in different items may be combined into fewer items or into one item, names of items may be changed, or items may be connected differently to one another.

[0141]

[0244] Examples are provided herein to help illustrate aspects of the technology, but the examples provided within this document do not describe all possible embodiments. Embodiments are not limited to the specific examples, component names, optimizations, algorithm selections, data, data types, configurations, implementations, sequences, displays, features, techniques, and scenarios presented herein. A given embodiment may include, by way of example, additional or different technical features, mechanisms, sequences, data structures, or functions, or may otherwise deviate from the examples presented herein.

[0142]

[0245] Process (aka method)

[0246] Figure 10 illustrates a method 1000, which is an example of a method having data storage capabilities as taught herein and which may be performed or assisted by an augmentation system as illustrated by one or more of Figures 1 through 9. The receiving step 1002, identifying step 1004, marking step 1006, inputting step 1008, detecting step 1010, and flushing step 1012 illustrated in Figure 10 are described throughout this disclosure with and without explicit reference to these reference numerals. Specifically, the discussion of Figures 2, 8, and 9 pertains to steps 1002 through 1012 illustrated in Figure 10.

[0143]

[0247] FIG. 11 further illustrates an access control method (which may be referred to as a "process" in the legal sense of the world). The access control method is suitable for use during operation of a system with enhanced data storage capabilities and includes any refinement, auxiliary, or contextual actions to the steps illustrated in FIG. 10. FIG. 11 also incorporates the steps illustrated in FIG. 9 or FIG. 10. Unless otherwise specified, the technical processes illustrated in these figures or disclosed elsewhere are performed automatically, for example, by the data storage subsystem 306. The processes may also be performed partly automatically and partly manually, to the extent that action by a human administrator or other person is involved. For example, in an embodiment, a human may specify a minimum or maximum percentage of the aperture that should be reserved (1116) for non-critical data. There is no entirely manual process considered to be an innovation herein. In a given embodiment, zero or more of the steps of the illustrated process may be operated repeatedly, perhaps with different parameters or data. Also, steps in the embodiments may be performed in different orders other than the top-down order depicted in Figures 10 and 11. Steps may be performed sequentially, partially overlapping, or fully parallel. In particular, the order in which the action items of flowchart 1000 or the action items of flowchart 1100 are traversed to illustrate steps performed in a process may vary from one implementation of the process to another. Also, steps may be omitted, combined, renamed, reclassified, performed on one or more machines, or deviated from the illustrated flow, provided that the process performed is operable and in accordance with at least one claim.

[0144]

[0248] An embodiment uses or provides a method for preserving data in a computing system. The method can include receiving 1002 a request to store a set of data A-data in an aperture of a battery backed memory. The aperture has a flush order 312. The flush order is an order in which data is copied 908 from the aperture to aperture associated non-volatile storage in response to a data integrity threat 314. The flush order defines a flush leading edge of the aperture and a flush trailing edge of the aperture. The A-data includes data designated as critical data.

[0145]

[0249] The method may also include a step 1004 of identifying a portion A-memory of the aperture's unallocated memory, the A-memory being large enough to hold the A-data, the A-memory having an address closer to the flash end of the aperture than any other address of any other unallocated memory of the aperture. The method may also include a step 1006 of marking the A-memory as allocated, and a step 1008 of inputting a copy of the A-data into the A-memory.

[0146]

[0250] The method can also include receiving 1104 a request to store a dataset Z-data in the aperture, where the Z-data does not include any data designated as critical data.

[0147]

[0251] The method may also include step 1004 of identifying a portion of the unallocated memory of the aperture, a Z-memory, the Z-memory being large enough to hold the Z-data, the Z-memory having an address closer to the flash tail of the aperture than any other address of any other unallocated memory of the aperture. The method may also include step 1006 of marking the Z-memory as allocated, and step 1008 of inputting a copy of the Z-data into the Z-memory.

[0148]

[0252] In this manner, by inputting the critical data into the battery-backed memory before the non-critical data in the flash-order, the likelihood 1140 of successfully flashing (1012) and therefore preserving (1102) the critical data is increased (1138), as opposed to not inputting the critical data into the battery-backed memory before the non-critical data in the flash-order.

[0149]

[0253] In some embodiments, the method may further include step 1106 of determining a size 1108 of the aperture based at least in part on battery characteristics of the battery backed-up memory. For example, a smaller aperture may be obtained for a smaller capacity battery, an older battery, or a less reliable battery. In some embodiments, the method may further include step 1106 of determining a location 1110 of the aperture based at least in part on battery characteristics of the battery backed-up memory. For example, an aperture may be selected in a memory backed-up by a newer, larger, or more reliable battery instead of an aperture in a memory backed-up by a smaller capacity, older, or less reliable battery. In some embodiments, the method may further include step 1114 of changing the size of the aperture or step 1115 of moving the aperture based at least in part on changes in battery characteristics of the battery backed-up memory, for example, by increasing the aperture size when the battery is replaced with a newer, larger, or more reliable battery. As with other steps disclosed herein, these steps may be combined in a given embodiment.

[0150]

[0254] In an embodiment, the method may further include defragmenting 1120 at least a defragmented portion of the aperture, the defragmented portion including the critical data. As discussed elsewhere herein, defragmenting 1120 refers to flash defragmenting. Integrated defragmenting may have side-effects, but is not inherently sufficient to function as flash defragmenting 1120. Defragmenting 1120 may be proactive. Defragmenting 1120 may also be triggered when a tenant or process is terminated, vocates, or transfers.

[0151]

[0255] In one embodiment, the method may further include step 1132 of designating at least a portion of the A-data as critical data based on at least one of the following criteria 602: workload criticality, virtual machine priority, or customer status.

[0152]

[0256] In one embodiment, the method may include detecting 1010 a data integrity threat, flushing 1012 all of the critical data from the aperture to non-volatile storage, and failing to flush 1122 at least some of the non-critical data from the aperture to non-volatile storage. For example, the power stored in the battery may not be sufficient to flush the entire aperture, but may be sufficient to flush all of the critical data. This example scenario illustrates the benefit of inputting the critical data so that, if there was sufficient battery power, the critical data is flushed before the non-critical data is flushed.

[0153]

[0257] In one embodiment, the method may include detecting 1010 a data integrity threat, flushing 1012 at least all of the critical data from the aperture to non-volatile storage, and then restoring 910 all of the flushed data from the non-volatile storage. The flushed critical data may be restored 910 to at least one of a volatile memory, a battery-backed memory from which the data was flushed, or a battery-backed memory different from the battery-backed memory from which the data was flushed.

[0154]

[0258] In one embodiment, the method may include maintaining 1126 at least one copy of the Z-data outside the aperture and outside the aperture-associated non-volatile storage, i.e., when building a system for preserving data, the duplication of non-critical data is compatible with the careful placement 1008 and flushing 1012 of critical data.

[0155]

[0259] In one embodiment, the battery backed memory aperture resides (1124) in a physical machine, here denoted M, and the request to store A-data in the aperture comes (1112) from a different physical machine, here denoted N. For example, in the configurations shown in Figures 2 or 8, the compute node and the storage node can be different physical machines. In alternative configurations, they can be virtual devices running on the same underlying physical hardware.

[0156]

[0260] In one embodiment, the method may include storing A-data on only one storage node and storing copies of Z-data on multiple storage nodes, i.e., in one example, storage of critical data relies only on input 1008 and flushing 1012, while storage of non-critical data relies at least in part on replication 226.

[0157]

[0261] configured storage medium

[0262] An embodiment includes a configuration computer readable storage medium 112. The storage medium 112 may include disks (magnetic, optical, or otherwise), RAM, EEPROMS, or other ROMs, and / or other configurable memories, specifically including computer readable storage media (not just propagating signals). The configured storage medium may specifically be a removable storage medium 114, such as a CD, DVD, or flash memory. General purpose memory, which may or may not be removable and may or may not be volatile, may be configured into an embodiment in the form of data 118 and instructions 116 using items such as an allocator 422, a defragmenter 408, and a mapping 814. The data 118 and instructions 116 are read from the removable storage medium 114 and / or other sources, such as a network connection, to form the configuration storage medium. The configuration storage medium 112 may enable the computer system 102 to perform the technological process steps for data storage as disclosed herein. Thus, the figures serve to illustrate configuration storage medium embodiments and process (a.k.a., method) embodiments, as well as system and process embodiments. In particular, any of the process steps shown in Figure 9, Figure 10, or Figure 11, or taught herein, may be used to configure a storage medium and form configuration storage medium embodiments.

[0158]

[0263] An embodiment uses or provides a computer readable storage medium 112, 114 having data 118 and instructions 116 configured thereon. The instructions 116, when executed by the processor 110, cause a computing system or a subsystem thereof to perform a method for data preservation. The method includes receiving 1002 a plurality of requests. Each request seeks to store a respective set of data in an aperture 220 of the battery backed memory. The aperture has a flush order 312. The flush order 312 is an order for copying data from the aperture to an aperture associated non-volatile storage in response to a data integrity threat. The flush order defines a flush leading edge 806 of the aperture and a flush trailing edge 808 of the aperture. Each of the sets of data includes data designated as critical data.

[0159]

[0264] The method also includes identifying 1004, for at least two of the requests, a respective portion of unallocated memory of the aperture, where each portion of unallocated memory is large enough to hold a respective data set, and each identified portion of unallocated memory has an address closer to the flush end of the aperture than any other address of any other unallocated memory of the aperture.

[0160]

[0265] The method also includes, for at least one of the requests, step 1006 marking each identified portion of unallocated memory as allocated, and step 1008 inputting a copy of the respective data set therein.

[0161]

[0266] The method also includes detecting a data integrity threat 1010 and flushing all of the critical data copied to the aperture from the aperture to non-volatile storage 1012. Thus, the method preserves 1102 all of the critical data copied to the aperture despite the data integrity threat.

[0162]

[0267] In one embodiment, the method further includes a step 1106 of assigning at least one of an aperture size 1108 or an aperture address 1110 based on at least one of a battery capacity 502, a battery age 506, or a battery reliability value 504.

[0163]

[0268] In one embodiment, the step of flashing 1012 includes a step 1142 of executing firmware 310 having a unified extensible firmware interface.

[0164]

[0269] In one embodiment, the method further includes reserving 1116 at least 10 percent 1118 of the aperture to hold data not designated as critical data. For example, reserving 20 percent or reserving 30 percent, respectively, sufficiently qualifies as reserving 1116 at least 10 percent. In one embodiment, the reserved 1116 portion 1118 is at most 10 percent or at most 20 percent. The reservation 1116 may also be considered or implemented as dividing the aperture or memory partition into subpartitions.

[0165]

[0270] In one embodiment, the method further includes a step 1128 of moving data not designated as critical data to a storage location outside the aperture to make room for the data designated as critical data. In one embodiment, the method includes a step 1120 of defragmenting at least a portion of the aperture, thereby reducing (1134) or eliminating (1136) an unallocated portion that was previously located between two allocated portions of the aperture.

[0166]

[0271] Additional Examples and Observations

[0272] It will be appreciated by those skilled in the art that any part of this disclosure, or any particular details therein, need not necessarily satisfy legal criteria such as enablement, written description, or best mode. Also, the embodiments are not limited to the specific networks, protocols, tools, identifiers, fields, data structures, functions, secrets or other proofs, or other implementation choices described herein. Any apparent discrepancy with any other patent disclosure, even from the owner of this innovation, is of no use in interpreting the claims presented in this patent disclosure. We provide some additional examples and observations with the understanding that they relate to all parts of this disclosure.

[0167]

[0273] In one embodiment, the volatile memory 402 is backed up by a battery 216 and can be quickly flushed 1012 to create a copy of the data 118 in the non-volatile storage 902 when data integrity is threatened 314 (e.g., due to an imminent power loss or reboot). The flushing 1012 is constrained by the flush order 312, so some data may not be flushed to a safe location. For example, a battery failure or depletion may result in an incomplete flush. Flushing is an example of a data lifecycle operation; some other examples are allocate, use, copy, deallocate, move, defrag, and restore.

[0168]

[0274] An embodiment is designed and implemented with the goal of provisioning a sufficiently large battery. The aperture size may therefore depend on the amount of data 118 the battery can protect, as determined by the architect or developer. Thus, larger batteries, newer batteries, and more reliable batteries may provide a larger aperture, and conversely, smaller / older / less reliable batteries, or unknown battery characteristics, may provide a smaller aperture. Also, batteries may become de-rated / degraded over a period of time, and therefore the ability to evacuate large amounts of data may be affected as a function of time. The aperture 220 may be resized accordingly.

[0169]

[0275] In one embodiment, a hybrid placement order is implemented whereby critical data is grouped at or near the flash leading edge 806 of the aperture, and non-critical data is grouped at or near the flash trailing edge 808 of the aperture. Non-critical or critical data (or both) can be replicated to other apertures, or to other memory external to any aperture that is battery backed up.

[0170]

[0276] The existence of the data preservation techniques taught herein may, in some cases, be inferred from documentation, from memory map 814, from memory dumps combined with information about data criticality, from machine probes or communications between them, from Joint Test Action Group (JTAG) debugging, from NVDIMM-SW save / restore tests, firmware or other code listings, from other evidence, or from a combination of these.

[0171]

[0277] An embodiment provides or uses an NVDIMM-SW evacuation algorithm that can factor in workload criticality 612 as a way to prioritize and sort multiple NVDIMM-SW data sets. An embodiment implements a storage class memory (SCM) partition 904 that is described by the virtual machine ACPI address range and maps 222, 224 between virtual machines and to the SCM partition. This guides the NVDIMM-SW evacuation routing in the runtime components of the system firmware, e.g., in UEFI / SMM for Intel / AMD CPUs and in UEFI / Trust Zones for ARM CPUs. An embodiment protects 908 the contents 118 in any graceful or ungraceful shutdown and allows 910 the virtual machine contents 118 to be restored at the next boot. In one embodiment, the NVDIMM-SW helps ensure fast access to data while also helping ensure data is flushed to a non-volatile store in the event of a service disruption, such as a node going offline.

[0172]

[0278] In one embodiment, events that may trigger a flush may include, for example, an orderly shutdown, a hard reboot, or a soft reboot.

[0173]

[0279] An orderly shutdown provides running software with the opportunity to protect data from modification by saving current versions of directories, allocation tables, and other structures that describe the organization of data, and to save the data itself. For example, a shutdown command instructs apps and other processes to shut down, then gives those processes an opportunity to flush data to non-volatile memory, close any open files, and free allocated memory back to the OS. The shutdown command also instructs device drivers to flush IO data and current directory information to attached devices. In ACPI-compliant systems, the shutdown command can also cause a Power command to be issued, which causes the NVDIMM to save data from its volatile portion to its non-volatile portion (908).

[0174]

[0280] A hard reboot begins with no power, other than perhaps a power button. Power is applied to the system. The boot memory code (BIOS / EFI / UEFI) runs POST. The boot memory code then loads a boot loader from the R boot device into the RAM. The boot loader loads the OS into RAM from non-volatile working memory or over a network connection. Whatever was in RAM when the hard reboot began can potentially be overwritten.

[0175]

[0281] A soft reboot begins by applying power to the system. It skips the Power On Self-Test (POST). The boot memory code loads a boot loader into RAM from the boot device. The boot loader loads the OS into RAM from non-volatile working memory or over a network connection. Whatever was in RAM when the soft reboot began is potentially overwritten.

[0176]

[0282] In one embodiment, the OS uses the system memory map to identify memory areas 904 to preserve for UEFI runtime functions after OS boot and to identify areas to reclaim for OS use. In one embodiment, the memory map is deployed throughout all phases of UEFI preboot. The memory map changes as and when memory is allocated to store code and data. The format of memory allocation varies and is based on the type of code and data, e.g., boot services code, boot services data, runtime code, runtime data, ACPINVS memory, SMM, etc.

[0177]

[0283] In one embodiment, the NVDIMM-SW is present and provides firmware assisted means to emulate a non-volatile DIMM through the coupling of a DIMM and a block storage device (such as NVMe or SATA). The goal is to adopt a standards-driven approach, which is to seamlessly publish the NVDIMM-SW device to the operating system. In normal operation mode, the pre-boot firmware exposes the available NVDIMM-SW device to the operating system. All WRITE operations from applications leveraging the NVDIMM-SW target the DIMM region associated with the NVDIMM-SW. SAVE# is initiated during either graceful or non-graceful shutdown scenarios. The SAVE# operation is completed before the system shutdown and subsequent resume. The RESTORE# operation is initiated by the pre-boot firmware before control is returned to the operating system. The NVDIMM-SW can be constructed by coupling a DDR4DIMM and an on-board M.2 NVMe module. The NVDIMM-SW paradigm reuses traditional DIMMs to emulate byte-addressable memory. In one embodiment, the system is populated with all DIMMs of the same size and memory type. The NVDIMM-SW can support two modes of operation: NVDIMM Non-Interleaved or NVDIMM Interleaved. In NVDIMM Non-Interleaved, the DIMM on slot 1 is selected as the non-volatile NVDIMM (SW) NVDIMM depending on the size of the non-volatile memory selected through a configuration option. The maximum non-volatile memory size depends on the power duration during a non-graceful evacuation scenario. The non-interleaved NVDIMM-SW is exactly the same as the NVDIMM-N, except that all I2C (Serial Protocol)_DSM (Device Specification Method) data is emulated instead.With NVDIMM interleaving, all DIMMs within a socket are interleaved together to support NUMA. Also, the top memory of each socket is carved out from the system memory map as NVDIMM-SW based on the selected non-volatile memory size. The goal of event handling is to attempt to evacuate data from volatile memory to the non-volatile memory area. The firmware stack (UEFI, BMC) is also responsible for logging errors for both inband and out-of-band listeners.

[0178]

[0284] Some additional combinations and variations

[0285] Any of these combinations of code, data structures, logic, components, communications, and / or functional equivalents thereof may be combined with any of the systems and variations thereof described above. The process may include any of the steps described herein in any operable subset or combination or sequence. Each variant may be performed alone or in combination with any one or more of the other variants. Each variant may be performed with any of the processes described above, and each process may be combined with any one or more of the other processes. Each process or combination of processes may be combined with any of the configuration storage medium combinations and variations described above, including variants.

[0179]

[0286] conclusion

[0287] In short, the teachings presented herein can be applied to enhance data preservation in computing systems. In one embodiment, a combination of operational steps and device characteristics facilitates preserving data 118 against integrity threats 314. Data 118 is divided into critical data 412 and non-critical data 416 based on criteria 602, such as customer 606 requirements 608, workload 610 criticality 612, or virtual machine 206 criticality 604. Data 118 can be generated at a compute node 202 and sent 222 for storage at, for example, a storage node 212. Critical data 412 is stored at a physical address 424 in an aperture 220 of a battery backed memory 218, from which it is flushed 1012 before non-critical data 416 according to a flush order 312. The flush order 312 may be imposed by the circuitry of the battery-backed memory or may be imposed on the battery-backed memory by an external force, such as the system firmware 420. The flush order 312 may be, for example, the flush order of the bottom-up NVDIMM 404. A redundant copy 226 of the data 118 (particularly non-critical data 416) may also be kept (1126) if it is not possible to flush it (1012) in time to preserve the replicated copy (1102). The aperture 220 of the battery-backed memory is sized (1106) and positioned (1106) according to the battery characteristics 406, and may be rearranged (1115) or resized (1114) as conditions change. Flash defragmentation 1120 is performed to optimize utilization of the aperture 220, particularly in the portion 414 of the aperture that holds critical data 412.

[0180]

[0288] It is also contemplated that embodiments may include or benefit from tested and applicable security and privacy controls, such as those of the General Data Protection Regulation (GDPR). It is implicitly understood that appropriate measures must be taken to help prevent misuse of the computing system, for example, by injection or activation of malware, and to help prevent tampering with any personal or private information that the augmentation system may process during program execution. Use of the tools and techniques taught herein may be compatible with the use of such controls.

[0181]

[0289] Although certain embodiments are explicitly shown and described herein as processes, configuration storage media, or systems, it will be appreciated that the discussion of one type of embodiment extends generally to other types of embodiments. To illustrate, the process description in conjunction with Figures 9, 10, and 11 also serves to explain the configuration storage media, and further serves to explain the technical effects and operation of the systems and manufactures, as well as those discussed in conjunction with the other figures. This does not mean that limitations from one embodiment are necessarily read into the other. In particular, the process is not necessarily limited to the data structures and configurations presented while discussing the system or product such as the configuration memory.

[0182]

[0290] It will be appreciated by those skilled in the art that implementation details may relate to specific APIs, specific fields, specific types of components (hardware and software), and specific code, such as specific sample programs, and therefore may not necessarily appear in all embodiments. It will also be appreciated by those skilled in the art that program identifiers and any other terminology used when discussing the details are implementation specific and therefore may not necessarily be relevant to every embodiment. However, while not necessarily presented here, such details may assist the reader by providing context and / or illustrating some of the many possible implementations of the techniques discussed herein.

[0183]

[0291] Given due attention to the items presented herein, including technical processes, technical effects, technical mechanisms, and technical details that are illustrative but not exhaustive of all claimed or claimable embodiments, it will be appreciated that the present disclosure and the embodiments described herein are not directed to subject matter other than the technical art, and are not directed to any idea of ​​itself, such as a principle or inherent cause or motivation, a mere result in itself, a mental operation or mental processing, a business method or prevailing economic practice, a mere method of organizing human activity, a law of nature in itself, a natural occurrence or process, a living organism or part of a living organism, a mathematical formula in itself, isolated software in itself, a mere conventional computer, anything that is not perceptible in its entirety, or any abstract concept in itself, or any meaningless post-solution activity. Those skilled in the art will recognize that this application does not cover any method, process or activity implemented entirely on unspecified apparatus, any method that fails to produce a useful tangible result, any preemption of any field of application, or any other subject matter that is ineligible for patent protection under the laws of the jurisdiction in which patent protection is sought or granted or enforced.

[0184]

[0292] A reference herein to an embodiment having a feature X and elsewhere herein to an embodiment having a feature Y does not exclude from this disclosure an embodiment having both feature X and feature Y, unless such exclusion is expressly stated herein. All possible negative claim limitations fall within the scope of this disclosure, in the sense that any feature defined to be part of an embodiment may be expressly excluded from inclusion in other embodiments, even if that specific exclusion is not set forth in any example herein. The term "embodiment" is used herein merely as a more convenient form of "of the process, system, product, configuration computer-readable storage medium, and / or other examples of the teachings herein, when applied in a manner consistent with applicable law." Thus, a given "embodiment" may include any combination of features disclosed herein, provided that the embodiment is consistent with at least one claim.

[0185]

[0293] Not all items shown in the figures are necessarily present in every embodiment. Conversely, an embodiment may include an item or items not explicitly shown in the figures. Although certain possibilities are illustrated herein by specific examples in the text and drawings, embodiments may deviate from these examples. To give an example, certain technical effects or technical features of an example may be omitted, renamed, grouped differently, repeated, instantiated differently in hardware and / or software, or may be a mixture of effects or features appearing in two or more of these examples. Those skilled in the art will recognize that functionality shown in one location may be provided in a different location in an embodiment, and that functional modules may be defined in various ways in a given implementation without necessarily precluding the desired technical effect from the overall collection of interacting modules. Although separate steps may be shown together in one box in the figures for space limitations or convenience, they may plausibly be performed separately. For example, in a given execution of the method, one step may be performed and not another.

[0186]

[0294] Reference numbers have been used to refer to the drawings throughout. Any apparent discrepancy in language associated with a given reference number, either in the figures or in the text, should be understood as merely broadening the scope of what is referenced by that reference number. Even if the same reference number is used, different instances of a given reference number may refer to different embodiments. Similarly, a given reference number may be used to refer to verbs, nouns, and / or their respective corresponding instances. For example, a processor 110 may process 110 instructions by executing them.

[0187]

[0295] As used herein, terms such as "a," "an," and "the" are intended to include one or more of the indicated item or step. Specifically, in the claims, a reference to an item ordinarily means that at least one such item is present, and a reference to a step generally means that at least one instance of the step is performed. Similarly, "is" and other singular verb forms should be understood to encompass "are" and other plural possibilities, where the context permits, to avoid grammatical errors or misunderstandings.

[0188]

[0296] Headings are for convenience only: information on a given topic may be found in a chapter other than the one whose heading identifies that topic.

[0297] All claims and the abstract as filed are incorporated herein by reference.

[0189]

[0298] To the extent that any term used herein implies or refers to an industry standard, and to the extent applicable law requires identification of a specific version of such a standard, it should be understood that the present disclosure refers to the most recent version of that standard that has been published, at least in draft form (if there is a newer version, the final form shall take precedence), as of the earliest priority date of this disclosure under the applicable patent law.

[0190]

[0299] Although exemplary embodiments have been shown in the drawings and described above, it will be apparent to those skilled in the art that numerous modifications can be made without departing from the principles and concepts set forth in the claims, and that such modifications do not necessarily encompass the entire abstract concept. Although the subject matter has been described in specific language of structural features and / or procedural acts, it will be understood that the subject matter defined in the appended claims is not necessarily limited to the specific technical features and acts described above in the claims. It is not necessary that every means or aspect or technical effect identified in a given definition or example be present or utilized in every embodiment. On the contrary, the specific features and acts and effects described are merely disclosed as examples for consideration when implementing the claims.

[0191]

[0300] All changes which fall short of embracing the entire abstract idea but which come within the meaning and range of equivalency of the claims are to be embraced within their scope to the maximum extent permitted by law.

Claims

1. 1. A data storage subsystem in a computing system, comprising: a battery backed up memory having an aperture, the memory including digital data storage locations addressed in a physical address space, the aperture including at least a portion of the digital data storage locations, the aperture having a physical address flush order, the physical address flush order being an order for copying data from the aperture to an aperture associated non-volatile storage in response to a data integrity threat, the flush order defining a first-to-flush physical address end of the aperture and a last-to-flush physical address end of the aperture; a data storage circuit in operative communication with the battery backup memory and configured to perform a data storage step, the data storage step comprising: (a) receiving a request to store a data set A-data in said aperture, said A-data including data designated as critical data; (b) identifying a portion A-memory of unallocated memory of said aperture, said A-memory being large enough to hold said A-data, said A-memory having a physical address closer to the flash tip physical address of said aperture than any other physical address of any other unallocated memory of said aperture; (c) marking the A-Memory as allocated and inputting a copy of the A-Data into the A-Memory; (d) receiving a request to store a data set Z-data in said aperture, said Z-data not including any data designated as critical data; (e) identifying a portion of unallocated memory of said aperture, a Z-memory, said Z-memory being large enough to hold said Z-data, said Z-memory having a physical address closer to the flash back end physical address of said aperture than any other physical address of any other unallocated memory of said aperture; (f) marking said Z-Memory as allocated and inputting a copy of said Z-Data into said Z-Memory; A data storage circuit including: Equipped with increasing the likelihood of successfully flushing and thereby preserving critical data, rather than by storing the data in the battery-backed memory without regard to the data's criticality; Data storage subsystem.

2. 2. The data storage subsystem of claim 1, wherein the battery backed up memory aperture resides in or is controlled by network node X, and the request to store A-data in the aperture comes from a different network node Y.

3. 3. The data storage subsystem of claim 2, wherein the network node X comprises a storage node in a cloud and the network node Y comprises a computation node in the cloud.

4. 3. The data storage subsystem of claim 2, wherein said network node X and said network node Y are related by a storage relationship R, R being defined such that two nodes M and N are related by R when M stores data on behalf of N, and R is not a one-to-one relationship of said network node X and said network node Y.

5. 2. The data storage subsystem of claim 1, wherein the battery backed-up memory comprises NVDIMM memory.

6. 2. The data storage subsystem of claim 1, wherein said computing system comprises a virtual machine, said data set A-data being sent to said data storage subsystem from a virtual machine VM-A, and said data set Z-data being sent to said data storage subsystem from a virtual machine VM-Z.

7. 1. A method for storing data in a computing system, comprising: receiving a request to store a set of data A-data in an aperture of a battery backed memory, the memory including digital data storage locations addressed in a physical address space, the aperture including at least a portion of the digital data storage locations, the aperture having a physical address flush order, the physical address flush order being an order for copying data from the aperture to an aperture associated non-volatile storage in response to a data integrity threat, the flush order defining a first-to-flush physical address end of the aperture and a last-to-flush physical address end of the aperture, the A-data including data designated as critical data; identifying a portion of unallocated memory of said aperture, an A-memory, said A-memory being large enough to hold said A-data, said A-memory having a physical address closer to the flash tip physical address of said aperture than any other physical address of any other unallocated memory of said aperture; marking said A-memory as allocated and inputting a copy of said A-data into said A-memory; receiving a request to store a data set Z-data in said aperture, said Z-data not including any data designated as critical data; identifying a portion of unallocated memory of said aperture, a Z-memory, said Z-memory being large enough to hold said Z-data, said Z-memory having a physical address closer to the flash tail physical address of said aperture than any other physical address of any other unallocated memory of said aperture; marking said Z-memory as allocated and inputting a copy of said Z-data into said Z-memory; Including, The method of claim 1, further comprising: inputting the critical data into the battery backed memory before non-critical data in the flash order, thereby increasing the likelihood of successfully flashing and thereby preserving critical data.

8. 8. The method of claim 7, further comprising: determining a size of the aperture based at least in part on battery characteristics of the battery backed up memory; determining a position of the aperture based at least in part on battery characteristics of the battery backed up memory; resizing the aperture based at least in part on changes in battery characteristics of the battery backed up memory; moving the aperture based at least in part on changes in battery characteristics of the battery backed up memory; The method further comprising at least one of:

9. 8. The method of claim 7, further comprising the step of flash defragmenting at least a defragmented portion of the aperture, the defragmented portion including critical data.

10. 8. The method of claim 7, further comprising the step of designating at least a portion of the A-data as critical data based on at least one of the following criteria: workload criticality, virtual machine priority, or customer status.

11. 8. The method of claim 7, further comprising: detecting a threat to said data integrity; flushing all of the critical data from the aperture to the aperture associated non-volatile storage; failing to flush at least some non-critical data from the aperture to the aperture-associated non-volatile storage; The method further comprises:

12. 8. The method of claim 7, further comprising: detecting a threat to said data integrity; flushing at least all of the critical data from the aperture to the aperture-associated non-volatile storage; thereafter, restoring all of the flushed data from the aperture-associated non-volatile storage to at least one of a volatile memory, the battery-backed memory from which the data was flushed, and a battery-backed memory different from the battery-backed memory from which the data was flushed; The method further comprises:

13. 8. The method of claim 7, further comprising the step of maintaining at least one copy of said Z-data external to said aperture and external to said aperture associated non-volatile storage.

14. 8. The method of claim 7, wherein the aperture of the battery backed up memory resides in a physical machine, here denoted M, and the request to store the A-data in the aperture comes from a different physical machine, here denoted N.

15. 8. The method of claim 7, further comprising storing the A-data on only one storage node and storing copies of the Z-data on multiple storage nodes.

16. A computer readable storage medium configured to contain data and instructions that, when executed, cause a data storage subsystem to perform a method for data storage in a computing system, the method comprising: receiving a plurality of requests, each request calling for storage of a respective set of data in an aperture of a battery backed memory, the memory including digital data storage locations addressed in a physical address space, the aperture including at least a portion of the digital data storage locations, the aperture having a physical address flush order, the physical address flush order being an order for copying data from the aperture to an aperture associated non-volatile storage in response to a data integrity threat, the physical address flush order defining a first-to-flush physical address end of the aperture and a last-to-flush physical address end of the aperture, each of the respective sets of data including data designated as critical data; identifying, for at least two of the requests, a respective portion of unallocated memory of the aperture, each portion of unallocated memory being large enough to hold the respective set of data, the respective identified portion of unallocated memory having a physical address that is closer to a flash tip physical address of the aperture than any other physical address of any other unallocated memory of the aperture; for at least one of said requests, marking said respective identified portion of unallocated memory as allocated and inputting therein a copy of said respective data set; detecting a threat to said data integrity; flushing all of the critical data copied to the aperture from the aperture to an aperture-associated non-volatile storage; whereby the method preserves all of the critical data copied to the aperture despite a threat to the data integrity.

17. 20. The computer-readable storage medium of claim 16, wherein the method further includes determining at least one of an aperture size and an aperture address, the determining step being based on at least one of a battery capacity, an age of the battery, and a reliability value of the battery.

18. 20. The computer readable storage medium of claim 16, wherein the step of flashing includes executing firmware having a unified extensible firmware interface.

19. 20. The computer-readable storage medium of claim 16, wherein the method further comprises reserving at least 10 percent of the aperture for holding data that is not designated as critical data.

20. 17. The computer readable storage medium of claim 16, wherein the method further comprises: moving data not designated as critical to a storage location outside said aperture to make room for data designated as critical; defragmenting at least a portion of the aperture, thereby reducing or eliminating any unallocated portion of the aperture located between two allocated portions; The computer-readable storage medium further comprising at least one of:

Citation Information

Patent Citations

  • Digital camera

    JP2004080405A

  • Processor, memory, computer system and data transfer method

    JP2006277411A

  • Memory power supply management device and memory power supply management method

    JP2008262451A

  • Image processing apparatus and program

    JP2011005683A

  • Performance improvement of non-volatile memory cache

    JP2013047979A