Tail of the log in persistent main storage

Abstract

A system uses persistent main memory to retain events waiting to be recorded in a persistent storage device. Each event is written to the persistent main memory so that it can be recorded in the event of recovery. For example, the event can be written to the log cache structure along with other states that identify the event in a specific log cache structure, the location of the event within the specific log cache structure, and the order of the events. For recovery, the log in the persistent storage device is evaluated to identify the end of the stored log. By identifying any log cache structure that is after the end of the stored log and can be effectively recovered, the tail of the log can be identified in the persistent main memory. The log cache structure contents are then serialized one log cache at a time, with the oldest first.

Description

[0001] This application is a divisional application of the invention patent application with application number 201680071669.7, application date December 2, 2016, and invention name “Tail of a log in persistent main memory”. Background Art

[0002] For a variety of reasons, computing systems often keep logs of certain events. These logs may help diagnose problems that occur or exist, recover from a computing system failure, perform incremental backups of a computing system, track computing performance, assess security, and so on.

[0003] Some logs are saved on disk to survive system failures. After all, the purpose of some logs is to be used after a system failure (such as when used to recover or diagnose the cause of the failure). When logs are saved on disk, events are usually first written to a volatile log cache (usually called "LC") in volatile memory. The size of each log cache is usually such that it can be serialized and written to disk in a single I / O operation, thereby preserving the use of the disk's I / O channel. Using such a log cache helps reduce the number of I / O operations to the disk.

[0004] Once the log cache is filled (or after a certain amount of time has passed), the contents of the log cache are serialized and written to disk, and another empty log cache is available to capture (at least in volatile memory) additional events. This process is repeated, cycling through several log caches. Once the contents of a particular log cache are confirmed to be written to the log on disk, one or more threads are initiated to write events to that log cache, and then these threads are released back to one or more event providers. While threads are waiting to be released, they may often be in a sleeping state to reduce the overhead associated with the thread.

[0005] The subject matter claimed herein is not limited to embodiments that solve any disadvantages or that operate only in environments such as those described above.Rather, this background is only provided to illustrate one exemplary technology area in which some embodiments described herein may be implemented. Summary of the invention

[0006] At least some embodiments described herein relate to a system for using a persistent main memory to save events that have not yet been written to a log of a persistent storage device. When an event is to be written to a log of a persistent storage device, the event is written to the persistent main memory so that it can be interpreted in the case of recovery. For example, the event can be written to the log cache structure in the persistent main memory together with other states that identify the event in a specific log cache structure, the position of the event in the specific log cache structure, and the order of events other than the events written to the specific log cache structure.

[0007] Once the event and corresponding state are written to the persistent main memory, the thread that initiated the write can return immediately. This is because even if the event is not flushed in the log in the persistent storage device (for example, together with the other contents of the log cache structure), the event can be recovered based on the contents of the persistent main memory. This greatly reduces the delay between the time when the thread initiates the write and the time when the thread is released for other work by multiple orders of magnitude. Typically, any thread that initiates writing an event to the log cache must wait for confirmation that the contents of the log cache have been written to the persistent storage device. This not only causes a delay from the I / O operation to the persistent storage device itself, but also causes a delay from waiting for the log cache to be fully filled to initially initiating the I / O operation.

[0008] Although not required, in some embodiments, a corresponding log cache may also exist in volatile memory, which mirrors the log cache structure in persistent main memory. This allows the contents of the log cache to be preserialized and serialized as they would normally be from a volatile log cache (regardless of the exact mechanism used for preserialization and serialization), still allowing the contents of persistent main memory to be used in the event of a failure that results in loss of volatile memory contents. Therefore, these embodiments minimize the impact on existing event logging infrastructure that uses a volatile memory log cache. Further, since persistent main memory does not need to track every possible preserialization state, these embodiments reduce the number of changes required to persistent main memory.

[0009] For recovery, the log in the persistent storage device is evaluated to identify the end of the stored log. The tail of the log is then identified in the persistent main memory by identifying one or more log cache structures that have events after the end of the stored log and can be effectively used for recovery (e.g., they have states between atomic event write operations). The contents of each of these identified log cache structures are then serialized one at a time using one log cache structure, with the earliest first. For example, if a volatile log cache is also used, the contents can be copied to the volatile log cache, which is then pre-serialized and serialized as normal. This alleviates the need for the persistent main memory itself to track the state of the log cache during pre-serialization.

[0010] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to describe the manner in which the above and other advantages and features of the present invention can be obtained, a more particular description of the invention briefly described above will be reproduced by reference to specific embodiments thereof which are illustrated in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the invention and are not therefore to be considered limiting of its scope, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings, wherein

[0012] Figure 1 illustrates an example computing system in which the principles described herein may be employed;

[0013] Figure 2 illustrates a system according to the principles described herein in which a log management component can log events with low latency in a manner that a recovery component can successfully recover if a failure occurs;

[0014] Figure 3 A flow chart illustrating a method for recording events of a computing system so as to enable the computing system to recover from a possible failure;

[0015] Figure 4 A flow chart illustrating a method for serializing a log cache into a log of a persistent storage device;

[0016] Figure 5 A flow chart illustrating a method for successfully writing an event to persistent storage is shown;

[0017] FIG. 6A to FIG. 6C illustrates the sequential state of portions of persistent main memory, including a log cache data structure, a log cache descriptor, and one or more slot structures, which are associated with each other in a manner that describes the state of the log cache structure and preserves the order in which events were written; and

[0018] Figure 7 A flow chart of a method for recovering a computing system using a log persisted in persistent storage and persistent main memory is illustrated. DETAILED DESCRIPTION

[0019] At least some embodiments described herein relate to a system for using a persistent main memory to save events that have not yet been written to a log of a persistent storage device. When an event is to be written to a log of a persistent storage device, the event is written to the persistent main memory so that it can be interpreted in the case of recovery. For example, the event can be written to the log cache structure in the persistent main memory together with other states that identify the event in a specific log cache structure, the position of the event in the specific log cache structure, and the order of the events other than other events written to the specific log cache structure.

[0020] Once the event and corresponding state are written to the persistent main memory, the thread that initiated the write can return immediately. This is because even if the event is not flushed in the log in the persistent storage device (for example, together with the other contents of the log cache structure), recovery can be performed for the event based on the contents of the persistent main memory. This greatly reduces the delay between the time when the thread initiates the write and the time when the thread is released for other work by multiple orders of magnitude. Typically, any thread that initiates writing an event to the log cache must wait for confirmation that the contents of the log cache have been written to the persistent storage device. This not only causes a delay from the I / O operation to the persistent storage device itself, but also causes a delay from waiting for the log cache to be fully filled to initially initiating the I / O operation.

[0021] Although not required, in some embodiments, a corresponding log cache may also exist in volatile memory, which mirrors the log cache structure in persistent main memory. This allows the contents of the log cache to be preserialized and serialized as they would normally be from a volatile log cache (regardless of the exact mechanism used for preserialization and serialization), still allowing the contents of persistent main memory to be used in the event of a failure that results in loss of volatile memory contents. Therefore, these embodiments minimize the impact on existing event logging infrastructure that uses a volatile memory log cache. Further, since persistent main memory does not need to track every possible preserialization state, these embodiments reduce the number of changes required to persistent main memory.

[0022] For recovery, the log in the persistent storage device is evaluated to identify the end of the stored log. The tail of the log is then identified in the persistent main memory by identifying one or more log cache structures that have events after the end of the stored log and can be effectively used for recovery (e.g., they have state between atomic event write operations). The contents of each of these identified log cache structures are then serialized one at a time using one log cache structure, with the earliest first. For example, if a volatile log cache is also used, the contents can be copied to the volatile log cache, which is then pre-serialized and serialized as normal. This alleviates the need for the persistent main memory itself to track the state of the log cache during pre-serialization.

[0023] Reference Figure 1 Some introductory discussions of computing systems are described. Figures 2 to 7 Describes the recording and recovery operations.

[0024] Computing systems now increasingly take a variety of forms. For example, a computing system may be a handheld device, a home appliance, a laptop computer, a desktop computer, a mainframe, a distributed computing system, a data center, or even a device that is not traditionally considered to be a computing system, such as a wearable object (e.g., glasses). In this specification and claims, the term "computing system" is broadly defined to include any device or system (or combination thereof) including at least one physical and tangible processor and a physical and tangible memory capable of having computer executable instructions thereon, which can be executed by the processor. The memory may take any form and may depend on the nature and form of the computing system. The computing system may be distributed over a network environment and may include multiple constituent computing systems.

[0025] like Figure 1 As illustrated, in its most basic configuration, computing system 100 generally includes at least one hardware processing unit 102 and memory 104. Memory 104 may be physical system memory, which may be volatile, non-volatile, or some combination of the two. The term "memory" may also be used herein to refer to non-volatile mass storage devices, such as physical storage media. If the computing system is distributed, then processing, memory, and / or storage capabilities may also be distributed.

[0026] The computing system 100 also has a number of structures, generally referred to as "executable components". For example, the memory 104 of the computing system 100 is illustrated as including an executable component 106. The term "executable component" is a name for a structure that is readily understood by those skilled in the art of computing, and that can be software, hardware, or a combination thereof. For example, when implemented in software, it should be understood by those of ordinary skill in the art that the structure of the executable component can include software objects, routines, methods that can be executed on the computing system, whether such executable component is present in the stack of the computing system or the executable component is present on a computer-readable storage medium.

[0027] In such cases, it will be appreciated by those of ordinary skill in the art that the structure of the executable component exists on a computer-readable medium such that when interpreted by one or more processors of a computing system (e.g., by a processor thread), the computing system is caused to perform a function. Such a structure may be directly computer-readable by the processor (when the executable component is a binary). Alternatively, the structure may be constructed to be interpretable and / or compiled (whether in a single stage or in multiple stages) to generate such a binary that is directly interpretable by the processor. When the term "executable component" is used, such an understanding of the example structure of an executable component is well within the understanding of those of ordinary skill in the computing arts.

[0028] The term "executable component" is also well understood by those of ordinary skill in the art to include structures that are implemented exclusively or nearly exclusively in hardware, such as in a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or any other special circuit. Thus, whether implemented in software, hardware, or a combination, the term "executable component" is a well-understood term for structures by those of ordinary skill in the computing arts. In this description, the terms "component," "service," "engine," "module," "virtual machine," etc. may also be used. As used in this specification and in this context, these terms (whether or not expressed using modifiers) are also intended to be synonymous with the term "executable component," and therefore also have structures that are well understood by those of ordinary skill in the computing arts.

[0029] In the following description, embodiments are described with reference to actions performed by one or more computing systems. If the actions are implemented in software, one or more processors (of the associated computing system performing the actions) direct the operation of the computing system in response to having executed the computer-executable instructions constituting the executable components. For example, such computer-executable instructions may be embodied on one or more computer-readable media forming a computer program product. Examples of such operations involve manipulation of data.

[0030] Computer-executable instructions (and manipulated data) may be stored in memory 104 of computing system 100. Computing system 100 may also contain communication channels 108 that allow computing system 100 to communicate with other computing systems, such as over network 110.

[0031] Although not all computing systems require a user interface, in some embodiments, the computing system 100 includes a user interface 112 for interacting with a user. The user interface 112 may include an output mechanism 112A and an input mechanism 112B. The principles described herein are not limited to a precise output mechanism 112A or input mechanism 112B, as this will depend on the nature of the device. However, the output mechanism 112A may include, for example, a speaker, a display, a tactile output, a hologram, etc. Examples of the input mechanism 112B may include, for example, a microphone, a touch screen, a hologram, a camera, a keyboard, a mouse for other pointing input, any type of sensor, etc.

[0032] As discussed in more detail below, the embodiments described herein may include or utilize a dedicated or general-purpose computing system including computer hardware (such as, for example, one or more processors and system memory). The embodiments described herein also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media can be any available media that can be accessed by a general or dedicated computing system. The computer-readable medium that stores computer-executable instructions is a physical storage medium. The computer-readable medium that carries computer-executable instructions is a transmission medium. Therefore, by way of example and not limitation, embodiments of the present invention may include at least two distinct computer-readable media: a storage medium and a transmission medium.

[0033] Computer-readable storage media includes RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other physical and tangible storage media that can be used to store desired program code means in the form of computer-executable instructions or data structures and that can be accessed by a general purpose or special purpose computing system.

[0034] A "network" is defined as one or more data links that enable electronic data to be transmitted between computing systems and / or modules and / or other electronic devices. When information is transmitted or provided to a computing system via a network or another communication connection (hardwired connection, wireless connection, or a combination of hardwired connection or wireless connection), the computing system appropriately treats the connection as a transmission medium. Transmission media may include networks and / or data links, which may be used to carry desired program code means in the form of computer-executable instructions or data structures and may be accessed by general-purpose or special-purpose computing systems. Combinations of the above should also be included within the scope of computer-readable media.

[0035] Further, program code means in the form of computer executable instructions or data structures may be automatically transferred from transmission media to storage media (or vice versa) upon reaching various computing system components. For example, computer executable instructions or data structures received over a network or data link may be cached in RAM within a network interface module (e.g., a "NIC") and then ultimately transferred to computing system RAM and / or to a less volatile storage medium of the computing system. Thus, it should be understood that storage media may be included in computing system components that also (or even primarily) utilize transmission media.

[0036] Computer executable instructions include, for example, instructions and data that, when executed at a processor, cause a general-purpose computing system, a special-purpose computing system, or a special-purpose processing device to perform a specific function or group of functions. Alternatively or in addition, computer executable instructions can configure a computing system to perform a certain function or group of functions. Computer executable instructions can be, for example, binary files or even instructions that undergo some translation (such as compilation) before being directly executed by a processor, intermediate format instructions such as assembly language, or even source code.

[0037] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the features or acts described above. Rather, the described features and acts are disclosed as example forms of implementing the claims.

[0038] Those skilled in the art will appreciate that the present invention can be implemented in a network computing environment with many types of computing system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, pagers, routers, switches, data centers, wearables (such as glasses), etc. The present invention can also be implemented in a distributed system environment, where local computing systems and remote computing systems linked by a network (by a hardwired data link, a wireless data link, or a combination of a hardwired data link and a wireless data link) all perform tasks. In a distributed system environment, program modules may be located in local memory storage devices and remote memory storage devices.

[0039] Those skilled in the art will also appreciate that the present invention may be implemented in a cloud computing environment. A cloud computing environment may be distributed, although this is not required. When distributed, a cloud computing environment may be distributed internationally within an organization and / or have components processed across multiple organizations. In this specification and the following claims, "cloud computing" is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage devices, applications, and services). The definition of "cloud computing" is not limited to any other advantages that may be obtained from such a model when properly deployed.

[0040] Figure 2 A system 200 according to the principles described herein is illustrated. The system includes a log management component 201 and a log recovery component 202, which can be described above with respect to Figure 1 Further, the system 200 can be constructed as described above for the executable component 106 of Figure 1 The computing system 100 is constructed as described. The log management component 201 causes events to be written to the persistent main storage 230 so that they will be written to the log 241 through normal operation or recovery operation (in other words, the recording of such events is almost certain, if not absolutely certain).

[0041] System 200 includes a set of event providers 210 that provide events to log management component 201 for logging. For instance, in the example, event provider 210A first provides an event as indicated by arrow 211A (hereinafter referred to as "event A"); event provider 210B then provides an event as indicated by arrow 211B (hereinafter referred to as "event B"); event provider 210C then provides an event as indicated by arrow 211C (hereinafter referred to as "event C"); and event provider 210D then provides an event as indicated by arrow 211D (hereinafter referred to as "event D"). Thus, in this example, log management component 201 receives the following events for logging in the following order: event A, event B, event C, and event D. The rate at which log management component 201 receives events for logging can vary over time.

[0042] Event providers 210 are illustrated as including four event providers 210A, 210B, 210C and 210D. However, the principles described herein are independent of how many event providers are providing events to the log. Thus, ellipse 210E is provided to symbolize the flexibility of the number of event providers that provide events to be recorded. The number of event providers that actively provide events for recording may even fluctuate significantly over time. However, four event providers 210A to 210D are illustrated only for the purpose of example and ease of reference. An event provider can be any software or hardware component or system, whether locally resident on system 200 or remotely located.

[0043] As each event is received, the log management component 201 writes the event to the persistent main storage 230 in the following manner: If a failure occurs before the event is written to the log, the recovery component 202 can provide the event to the log 241 stored in the persistent storage 240 during recovery. However, if recovery is not required, then during normal operation, the event will eventually end up marked in the log 241 stored in the persistent storage 240. If a failure occurs, then during a subsequent recovery, the recovery component 202 is triggered to use the log 241 in the persistent storage 240 and the events written to the persistent main storage 230 to perform recovery.

[0044] The event successfully written to the persistent main memory 230 is written in a manner that the recovery component 202 will finalize the operation so that the event is included in the log 241 of the persistent storage device 240. Thus, the successful writing to the persistent main memory 230 ensures that the event will be recorded even if a failure occurs after the successful writing. Thus, after the event is successfully written to the persistent main memory 230, the success of the event writing can be determined immediately, rather than waiting until it is confirmed that the event has actually been written to the log 241 of the persistent storage device 240. Thus, the thread that initiated the write can be returned almost immediately, and the write to the persistent main memory 230 has been successfully completed once.

[0045] This results in a reduction of several orders of magnitude in latency between the time a thread initiates writing an event and the time the thread is released. Typically, any thread that initiates writing an event to the log cache must collectively wait for confirmation that the contents of the log cache have been written to persistent storage. This results in latency not only from the I / O operation to the persistent storage itself, but also from waiting for the log cache to be fully filled to initiate the I / O operation in the first place.

[0046] More specifically, the persistent main memory 230 includes a plurality of log cache structures 231 into which events are written, i.e., one log cache structure is ready to receive an event at a time. Figure 2 Four log cache structures 231A, 231B, 231C and 231D are illustrated in FIG. However, ellipses 231E and 231F indicate that other numbers are possible. In one example, there may be more than one hundred such log cache structures. The log cache structure is filled with events until the log cache structure is full or until a certain time has passed, at which time the events corresponding to the log cache structure are serialized and placed in the log 241 via I / O operations. In an example in which events A to D are to be recorded, it is assumed that events A and B are written to a log cache structure 231A (as indicated by corresponding arrows 213A and 213B), so the log cache structure 231A is determined to be full (no more events are accepted). At this point, the log cache structure 231B can become active, and as indicated by corresponding arrows 213C and 213D, the next events C and D are provided to the log cache structure 231B. Each log cache structure may be rotated in a round-robin manner.

[0047] In one embodiment, there is also a volatile memory 220 in which the same events are written, wherein the volatile memory also includes a log cache 221 that mirrors the log cache structure of the persistent main memory 230. For example, a volatile log cache 221A may correspond to a persistent log cache structure 231A, both of which contain the same content, and both may even be mirrored when filled with events occupying the same relative space within the log cache. Similarly, volatile log caches 221B to 221D correspond to persistent log cache structures 231B to 231D, respectively. Because there may be other persistent log cache structures as represented by ellipses 231E and 231F, there may be other volatile log caches as represented by ellipses 221E and 221F.

[0048] In embodiments where both the volatile log cache and the persistent log cache are mirrored, events are written to both the volatile log cache and the corresponding persistent log cache structures. For example, event A may be written to the volatile log cache 221A (as represented by dashed arrow 212A) at approximately the same time that event A is written to the persistent log cache structure 231A (as represented by arrow 213A). Similarly, event B may be written to the volatile log cache 221A (as represented by dashed arrow 212B) at approximately the same time that event B is written to the persistent log cache structure 231A (as represented by arrow 213B).

[0049] Further, at this point, assume that the two corresponding log caches 221A and 231A are considered full (either by virtue of actually being full or by virtue of the passage of time). At this point, the active mirrored log cache becomes the mirrored log caches 221B and 231B. Thereafter, in the example of events A to D, event C may be written to the volatile log cache 221B (as represented by dashed arrow 212C) at approximately the same time that event C is written to the persistent log cache structure 231B (as represented by arrow 213C). Subsequently, event D may be written to the volatile log cache 221B (as represented by dashed arrow 212D) at approximately the same time that event D is written to the persistent log cache structure 231B (as represented by arrow 213D).

[0050] While this double writing may seem wasteful, it actually has several advantages. For example, once the log cache is filled as much as it will ever be, there is typically a complex pre-serialization process where the log cache data is moved around for efficient serialization and compact representation in the log. When designing pre-serialization, there are several decision calls that produce different sequences of log cache states in the log cache. Thus, pre-serialization occurs in different ways for different products in different environments, and may even change over time in the same environment or version or the same product.

[0051] If there is only a single log cache in persistent main memory, these numerous pre-serialization changes can be tracked within the log cache structure. Otherwise, if a failure occurs in the middle of pre-serialization, recovery will not be possible with certainty. This tracking can significantly complicate the pre-serialization process. The recovery process can also be complicated because if a failure occurs during the pre-serialization process, the recovery process may have to have additional logic to figure out the position of the log cache structure in the pre-serialization process so that recovery can continue from there.

[0052] Instead, all manipulations involved in pre-serialization can be confined to the volatile log cache. Existing products already interact with the volatile log cache to perform pre-serialization manipulations on the cache contents, so this may allow pre-serialization and serialization to proceed normally without requiring changes to existing products that perform pre-serialization. Further, if a failure were to occur during pre-serialization, resulting in the loss of the contents of the volatile log cache, the volatile log cache can simply be refilled using a persistent log cache structure to provide the volatile log cache with the original state it was in at the start of the previously attempted pre-serialization. Pre-serialization may then begin again, resulting in events in the volatile log cache being serialized into the log. Again, no changes need to be made to the existing pre-serialization process to allow this recovery to occur; and the complexity of tracking state changes during pre-serialization is avoided.

[0053] Figure 3 A flow chart of a method 300 for recording events of a computing system so as to enable the computing system to recover from a possible failure is illustrated. The method 300 may be performed by, for example Figure 2 The log management component 201 executes. Therefore, now frequently referred to Figure 2 200 pairs of systems Figure 3 The method 300 is described.

[0054] Whenever the log management component detects that a thread is writing an event to the log (action 301), method 300 is executed. In response, the event is written to persistent main memory (action 302). If a failure occurs, the write occurs in the following manner: the event will be written to the log, whether relatively immediately during normal operation or later during recovery (as further described below). This will be referred to as a "successful write" to persistent main memory hereinafter. Further, if volatile memory is also used for log cache, the event is written to volatile memory (action 303). In the case of also using volatile memory, when the event has been successfully written to persistent main memory and the event has been written to volatile memory, the "successful" write of the event occurs. As mentioned above, after successful writing, the thread can return to the event provider (action 304) with very low latency.

[0055] Although actions 302 and 303 are illustrated as occurring in parallel, in one embodiment, writing to the volatile log cache in volatile memory (action 303) actually occurs before writing to the persistent log cache structure in persistent main memory (action 302). This is useful in embodiments where the write to the volatile cache first determines the location. For example, the log sequence number (or LSN) indicates the location of the write in the volatile log cache. In this case, the log management component 201 generates the LSN to indicate the location of the event so that appropriate mirroring can occur. In this embodiment, the log management component 201 will perform the following operations in order: 1) look up the LSN based on the current volatile log cache, 2) copy the event content to the current volatile log cache, 3) look up the persistent main memory location based on the LSN, and 4) also copy the event content to that location in the persistent main memory.

[0056] for example, Figure 2 Essentially shows Figure 3 The method of 201 occurs four times. First, the log management component 201 detects that event A is to be written (action 301). As indicated by arrow 213A, the log management component 201 then successfully writes event A to any one of the persistent log cache structures currently in the active persistent log cache structure (in the example, log cache structure 231A) (action 302). Further, as indicated by arrow 212A, the log management component 201 also writes event A to any one of the volatile log caches currently in the active volatile log cache (in the example, log cache 221A) (action 303). At this point, the thread returns (as indicated by arrow 214A) to the event provider 210A (action 304).

[0057] Next, the log management component 201 detects that event B is to be written (action 301). Then, as indicated by arrow 213B, the log management component 201 successfully writes event B to the active log cache structure 231A (action 302). Further, as indicated by arrow 212B, the log management component 201 also writes event B to the active volatile log cache 221A (action 303). At this point, the thread returns (as indicated by arrow 214B) to the event provider 210B (action 304). Next, the log management component 201 detects a change from log caches 221A and 231A to the active log cache of log caches 221B and 231B. As will be described with reference to Figure 4 As described, in response, log management component 201 pre-serializes events of active log caches 221A and 221B to memory and serializes the events to log 241 on persistent storage 240 .

[0058] Third, the log management component 201 detects that event C is to be written (action 301). Then, as indicated by arrow 213C, the log management component 201 successfully writes event C to any one of the currently active persistent log cache structures (in this example, log cache structure 231B, due to detection of changes in the active log cache) (action 302). Further, as indicated by arrow 212C, the log management component 201 also writes event C to any one of the currently active volatile log caches (in this example, log cache 221B, due to detection of changes in the active log cache) (action 303). At this point, the thread returns (as indicated by arrow 214C) to the event provider 210C (action 304).

[0059] Finally, the log management component 201 detects that event D is to be written (action 301). Then, as indicated by arrow 213D, the log management component 201 successfully writes event D to the active log cache structure 231B (action 302). Further, as indicated by arrow 212D, the log management component 201 also writes event D to the active volatile log cache 221B (action 303). At this point, the thread returns (as indicated by arrow 214D) to the event provider 210D (action 304). At this point, the example of events A to D ends. However, this process may continue indefinitely, with one thread and event followed by another thread and event, and the return of the thread occurring immediately.

[0060] Figure 4 A flow chart of a method 400 for serializing a log cache into a log is illustrated. The method 400 may be performed at Figure 2 200. Thus, reference is now frequently made to Figure 2 200 pairs of systems Figure 4 The method 400 is described in detail. The method 400 may be performed by, for example Figure 2 The log management component 201 executes.

[0061] Method 400 is initiated when it is detected that the active log cache is to be changed (act 401). This may occur, for example, if the log cache has been completely filled. Alternatively, this may occur after an earlier version of the log cache has been completely filled or after a certain amount of time has passed. For example, Figure 2 In the example of events A to D, the log management component 201 detects that the active log cache is about to change between the time when event B is written to the log cache and the time when event C is written to the log cache.

[0062] After detecting that the active log cache is to change, the previously active log cache changes its state to full (action 402), and the previously inactive log cache changes its state to active (action 403). The active state means that the log cache can no longer receive events and will be written to the log. The active state means that the log cache can receive events. For example, in Figure 2 In the example of events A to D of FIG. 4 , after event B is written to the log cache but before event C is written to the log cache, persistent log cache structure 231A has filled (act 402) and persistent log cache structure 231B becomes active (act 403). In an embodiment that also uses volatile memory 220, after event B is written to the log cache but before event C is written to the log cache, volatile log cache 221A has filled (act 402) and volatile log cache 221B becomes active (act 403).

[0063] The contents of the log cache are then pre-serialized (action 404), serialized (action 405), and then written to the log 241 (action 406). Pre-serialization (action 404) is optional, but helps to efficiently serialize events for storage and / or efficiently store events in the log. Examples of pre-serialization include moves to more efficiently remove empty space (such as slot array moves), checksum generation and insertion, encryption, byte padding, compression, etc.

[0064] The resulting pre-serialized content is then serialized (act 405) for transmission to a persistent storage device via an I / O channel. Figure 2, the optional pre-serialization and serialization of the contents of the persistent log cache structure 231A is represented by arrow 215A. However, if a volatile log cache is employed, the contents of the volatile log cache 221A will be pre-serialized and serialized as represented by arrow 215B.

[0065] Once the content is securely stored in the log (action 406), the log management component confirms the storage (action 407) and then releases the log cache to an available state (action 408). The available state means that the log cache is not currently being written to, but is available for the log management component to change to an active state when the log management component so instructs. For example, in Figure 2 In the example, once write 215A or (in the case of a volatile log cache) write 215B is confirmed to have been completed to the log (action 407), once the log management component 201 decides to use them again, the log caches 221A and 231A each become available to become active. In this case, because the contents of the previously filled log caches are no longer needed, they will be invalidated and may be safely overwritten in the future. Alternatively, the log caches can be reinitialized with default values ​​and / or modes for future use.

[0066] Typically at this point, all threads that have written events to the log cache are finally released. However, rather than waiting until the log cache is flushed before releasing all trapped threads in batches, threads are released one by one each time method 300 is completed. In current technology, this would reduce latency from milliseconds to potentially a few microseconds.

[0067] Figure 5 A flow chart of a method 500 for successfully writing an event to a persistent storage device is illustrated. The method 500 is Figure 3 An example of action 302 is provided and used to explain Figure 2 Some remaining structures within the persistent main memory 230 as shown in FIG. Method 500 includes: writing an event into a log cache structure of the persistent main memory (act 501), and also assigning a slot structure to the event (act 502). Thus, Figure 2 The persistent main memory 230 is illustrated as including a plurality of slot structures 232. Because the slot structures are not complex, they can be relatively small. The slot structures are then modified (action 503) to indicate the location where the log is written in the log cache structure and to preserve order information indicating the order in which the event is written in the context of other events also being written to the log cache structure.

[0068] In one embodiment, this preservation of the order and position of the slots is preserved using a linked list of slot structures having a log cache descriptor at its root. For example, a log cache descriptor in persistent main memory may be associated with each persistent log cache structure in persistent main memory. For example, Figure 2 It is shown that the persistent main memory 230 also includes a log cache descriptor 233. One log cache descriptor corresponds to each persistent log cache structure. For example, log cache descriptor 233A corresponds to persistent log cache 231A. Similarly, log cache descriptors 233B to 233D correspond to corresponding persistent log cache structures 231B to 231D.

[0069] Now refer to FIG. 6A to FIG. 6C An example of using such a linked list to track the location and order of events within a persistent log cache structure is described. FIG. 6A to FIG. 6C An example subsequent state of the linked list is shown when two event writes to the log cache are performed.

[0070] Fig. 6A An initial state 600A is illustrated in which the log cache structure 610 is empty, but the corresponding log cache descriptor 620 indicates an active state. As an example, the log cache structure 610 may represent immediately after the persistent log cache structure becomes active (before event C is written to it): Figure 2 The persistent log cache structure 231B is not yet identified and is therefore still empty. The log cache descriptor 620 includes a log cache descriptor identifier field 621, which represents a logical association with the log cache structure (as symbolically represented by line 615). The log cache descriptor 620 also includes a state field 622 having an "active" value.

[0071] Now assume that an event is to be written to the log cache structure 610. For example, event C Figure 2 In the example of is first written to the log cache structure 231B. The event is written to the log cache structure (action 501) and the slot structure is allocated. For example, in Figure 6B In the example, state 600B shows that event C has been written to location 611. Furthermore, slot structure 631 has been allocated. For example, slot structure 631 may be Figure 2 A slot structure in the slot structure 232.

[0072] The slot structure is modified (act 503) to identify the location where events are written to the log cache structure and to preserve the order of events. Figure 6B, by having a pointer 623 within a log cache descriptor 620 of the log cache structure 610 point to a slot structure 631, which is associated with the log cache structure 610. The slot structure 631 also includes a location description 632, which points (as represented by arrow 651) to the location 611 of the event C within the log cache structure 610. The description may include, for example, a pointer to the starting address of the location 611 and the size of the location 611. At this point, a recovery component (such as recovery component 202) can use state 600B to restore the volatile log cache 221B to the state it was in after event C was written but before event D was written.

[0073] Now assume that another event is about to be written to the log cache structure 610. For example, Figure 2 In the example of , event D is then written to the log cache structure 231B. The event is written to the log cache structure (action 501) and the slot structure is allocated. For example, in Figure 6C In FIG. 600C , state 600C shows that event D has been written to location 612. Further, slot structure 641 has been allocated. For example, slot structure 641 may be Figure 2 A slot structure in the slot structure 232.

[0074] The slot structure is modified (act 503) to identify where events are written to the log cache structure and to preserve the order of events. Figure 6C , slot structure 641 includes a location description 642 that points to location 612 of event D within log cache structure 610 (as represented by arrow 652). The description may include, for example, a pointer to the starting address of location 612 and the size of location 612. Slot structure 641 is first modified to have a pointer 643 that points to the previous first slot structure 631 in the linked list. Then, pointer 623 within log cache descriptor 620 is modified to point to the new slot structure 641. Thus, at this point, a recovery component (such as recovery component 202) can use state 600C to restore volatile log cache 221B to the state it was in after events C and D were written, and can also know (based on the order in the linked list) that event C occurred before event D.

[0075] Figure 7 A flowchart of a method 700 for recovering a computing system using a log stored in a persistent storage device and in a persistent main memory is illustrated. For example, the method 700 may be performed by Figure 2 Thus, reference will now be made frequently to Figure 2 System 200 pairs Figure 7700. In this example, a failure occurs before the contents of persistent log cache 231A are written to log 241 of the persistent storage device, but after events C and D have been written to persistent log cache 231B.

[0076] The recovery component evaluates the log in the persistent storage device to identify the end of the stored log (act 701). The recovery component then identifies the tail of the log in the persistent main storage by identifying any log cache structures that have events after the end of the stored log (act 702). Figure 2 In the example of , assume that the contents of either of persistent log cache structures 231A and 231B have not been written to log 241 of persistent storage 240. In this case, both persistent log cache structures 231A and 231B are after the end of log 241 and therefore include the tail of the log.

[0077] It is also confirmed that the content of the log cache structure is valid (action 703). If the state of the content is confirmed to be consistent between event writes, the content of the log cache structure is valid. In this case, it is assumed that the persistent log cache structure is all valid.

[0078] For each of the identified valid log cache structures that are not serialized in the log, the recovery component then captures the serialized contents of the log cache structure into the log in the persistent storage device (action 704). This operation is performed one log cache structure at a time and occurs in an appropriate time order so that earlier log cache structures will be serialized into the log earlier than later log cache structures. In the case of using a volatile log cache, the capture can use existing pre-serialization and serialization processes that are already available for volatile log caches. For example, the contents of the log cache structure can first be copied to a corresponding log cache in volatile memory, and then the contents (using optional pre-serialization and serialization) are written from the volatile log cache to the log of the persistent storage device, which is consistent with the previously described actions 404 to 406.

[0079] Thus, an efficient mechanism for logging has been described in which the latency between the time a thread initiates writing an event to a log and the time the thread is returned is significantly reduced. The thread no longer needs to wait until the entire log cache is serialized into the log before performing other work. Further, in some embodiments where volatile memory is used to have a log cache that mirrors a log cache in persistent main memory, the pre-serialization process does not need to be changed, and thus recovery is possible even if a failure occurs during pre-serialization.

[0080] The present invention may be embodied in other specific forms without departing from its spirit or essential features. The described embodiments are considered in all respects to be illustrative only and not restrictive. Therefore, the scope of the present invention is indicated by the appended claims rather than by the preceding description. All changes falling within the meaning and scope of the equivalents of the claims will be included within their scope.

Claims

1. A computing system comprising: one or more processors; as well as one or more hardware storage devices having computer executable instructions stored thereon, which when executable by the one or more processors cause the computing system to operate within an architecture for performing a method for recording event content to support recovery when a failure event occurs, the method comprising: Make the thread initiate writing the event content to the log cache; After the event content is written to the log cache and before the event content is recorded to a log in persistent storage, releasing the thread so that the thread is available to perform other work before the event content is recorded to the log; detecting a change in a status of the log cache; responsive to the detecting, changing the state of the log cache from an active state state to a filling state state, the filling state state rendering the log cache incapable of receiving additional event content; serializing the event content contained in the log cache; Writing the serialized event content to the log in the persistent storage; and The status of the log cache is changed to an available status status, the available status status indicating that the log cache is available to receive new event content. 2 . The computing system of claim 1 , wherein the log cache is maintained in a persistent memory of the computing system. 3 . The computing system of claim 1 , wherein the log cache is maintained in volatile memory of the computing system. 4 . The computing system of claim 1 , wherein the detecting is performed in response to determining that the log cache is full. 5 . The computing system of claim 1 , wherein the detecting is performed in response to determining that an amount of time has elapsed after an event was written to the log cache.

6. The computing system of claim 1, wherein the method further comprises: In response to the detecting, a state of a previously inactive log cache is changed to an active state, the active state enabling the previously inactive log cache to receive new events.

7. The computing system of claim 1, wherein the method further comprises: The event content contained in the log cache is pre-serialized.

8. The computing system of claim 7, wherein the pre-serialization comprises removing empty space in the log cache.

9. The computing system of claim 7, wherein the pre-serializing comprises performing a checksum of the log cache and inserting the generated checksum into the log cache.

10. The computing system of claim 7, wherein the pre-serializing comprises encrypting the log cache.

11. The computing system of claim 7, wherein the pre-serializing comprises compressing the log cache.

12. The computing system of claim 1, wherein the method further comprises: The state of the log cache is changed from the available state state to the active state state, and then at least one event is written to the log cache.

13. The computing system of claim 1, wherein the method further comprises: Reinitializes the log cache using default values ​​for future use.

14. A method for operating a computing architecture that records event content of a computing system to support recovery of the computing system when a failure event occurs, the method comprising: Make the thread initiate writing the event content to the log cache; After the event content is written to the log cache and before the event content is recorded to a log in persistent storage, releasing the thread so that the thread is available to perform other work before the event content is recorded to the log; detecting a change in a status of the log cache; responsive to the detecting, changing the state of the log cache from an active state state to a filling state state, the filling state state rendering the log cache incapable of receiving additional event content; serializing the event content contained in the log cache; Writing the serialized event content to the log in the persistent storage; as well as The status of the log cache is changed to an available status status, the available status status indicating that the log cache is available to receive new event content.

15. The method of claim 14, wherein the detecting is performed in response to determining that the log cache is full.

16. The method of claim 14, wherein the detecting is performed in response to determining that an amount of time has elapsed after an event was written to the log cache.

17. The method according to claim 14, wherein the method further comprises: In response to the detecting, a state of a previously inactive log cache is changed to an active state, the active state enabling the previously inactive log cache to receive new events.

18. The method according to claim 14, wherein the method further comprises: pre-serializing the event content contained in the log cache, and The pre-serialization includes at least one of the following: removing blank space in the log cache, performing a checksum of the log cache, inserting the generated checksum into the log cache, encrypting the log cache, or compressing the log cache.

19. The method according to claim 14, wherein the method further comprises: The state of the log cache is changed from the available state state to the active state state, and then at least one event is written to the log cache.

20. The method according to claim 14, wherein the method further comprises: Reinitializes the log cache using default values ​​for future use.

Images