Record memory value traces for use with independent cache coherence protocol traces.

Abstract

A computer system records a replayable execution trace based on logging Cache Coherence Protocol (CCP) messages to a first trace and logging memory snapshots to a second trace. Based on determining that tracing of execution on a first execution context will be enabled, the computer system begins logging one or more memory snapshots of the first execution context's memory space to the second trace and enables the processor's hardware tracing capabilities. Enabling tracing causes the processor to log CCP messages generated in response to one or more memory accesses to the first execution context's memory space to the first trace. After enabling the processor's hardware tracing capabilities, the computer system also logs or otherwise processes writes from the second execution context to the first execution context's memory space.

Description

Technical Field

[0001] This disclosure relates to systems, methods, and apparatus for protecting sensitive information when recording replayable execution traces of computational context. Background Technology

[0002] Tracking and correcting undesirable software behavior is a core activity in software development. Undesirable software behavior can include many things, such as execution crashes, runtime exceptions, slow performance, incorrect data results, and data corruption. Undesirable software behavior is triggered by a variety of factors, such as data input, user input, and race conditions (e.g., when accessing shared resources). Due to the diversity of triggering factors, undesirable software behavior is often rare, appears random, and is extremely difficult to reproduce. Therefore, identifying a given undesirable software behavior is often very time-consuming and difficult for developers. Once an undesirable software behavior is identified, determining its (multiple) root causes is also often very time-consuming and difficult.

[0003] Developers have used various methods to identify unwanted software behaviors and then pinpoint multiple locations within the application's code that lead to these behaviors. For example, developers frequently test different parts of the application's code against various inputs (e.g., unit tests). Another example is that developers often consider executing the application's code in a debugger (e.g., setting breakpoints / watchpoints, stepping through lines of code during execution, etc.). Yet another example is that developers frequently observe code execution behavior in a profiler (e.g., timing, coverage). And yet another example is that developers often insert diagnostic code (e.g., tracing statements) into the application's code.

[0004] While traditional diagnostic tools (such as debuggers, profilers, etc.) operate on “real-time” forward execution of code, emerging diagnostic tools support “historical” debugging (also known as “time travel” or “reverse” debugging), where the execution of at least a portion of the execution context is recorded in one or more trace files (i.e., execution traces). Using some tracing techniques, execution traces can contain “bit-accurate” historical execution trace data, allowing the recorded portion of the traced execution context to be virtually “replayed” (e.g., through emulation) down to the granularity of individual instructions (e.g., machine code instructions, intermediate language code instructions, etc.). Therefore, using “bit-accurate” trace data, diagnostic tools enable developers to infer previous execution of the recorded subject context, rather than traditional debugging, which is limited to “real-time” forward execution. For example, using replayable execution traces, some history debuggers offer a user experience that supports forward and backward breakpoints / watchpoints, allowing code to be stepped forward and backward, etc. On the other hand, some history profilers are able to derive code execution behavior (e.g., timing, coverage) from previously executed code.

[0005] Some execution tracing techniques are at least partially based on the microprocessor (processor) logging at least a portion of the incoming (i.e., cache misses) data into the processor cache during execution within the processor's execution context. Compared to non-cache-based logging techniques such as software emulation, cache-based logging techniques offer numerous opportunities to reduce recording overhead and / or the amount of data logged into the execution trace. However, if the execution context reads sensitive memory values ​​from memory, such as personally identifiable information (PII), encryption keys, etc., these sensitive memory values ​​may become cache inflows, and traditional cache-based logging techniques can log these sensitive values ​​into the execution trace. Therefore, cache-based logging techniques present potential security issues when tracing execution contexts that interact with memory storing sensitive data values. Summary of the Invention

[0006] At least some of the embodiments described herein mitigate security issues associated with cache-based recording techniques by separating execution tracing into two independent and distinct component traces. The component trace includes a first component trace recorded by the processor, which records at least a portion of Cache Coherence Protocol (CCP) messages communicated between multiple processing units when these processing units perform memory accesses to the execution context. Notably, the first component trace lacks memory values ​​associated with these memory accesses. The component trace also includes a second component trace recorded by the software operating environment, which records one or more snapshots of at least a portion of the system memory. In embodiments, the first component trace contains sufficient information for analyzing and / or debugging the execution context (e.g., analyzing the processor cache used by the execution context), but lacks sufficient information to actually replay the recorded execution of the execution context. Instead, in embodiments, replay of the recorded execution of the execution context is achieved by combining information included in the first component trace (e.g., CCP messages) with information included in the second component trace (e.g., memory snapshots).

[0007] Given the disclosure herein, it can be understood that separating execution traces into different component traces in the manner described herein supports the ability to separate data and store component traces in different security domains, as well as the ability to apply different data security policies to component traces and apply different data protections to component traces. In embodiments, a second component trace (e.g., containing memory data) is stored in a more privileged security domain (e.g., compared to the security domain storing the first component trace), a stricter data security policy is applied to the second component trace (e.g., compared to the data security policy applied to the first component trace), and / or a higher level of data protection is applied to the second component trace (e.g., compared to the data projection scheme applied to the first component trace). Therefore, sensitive memory values ​​(such as PII, encryption keys, etc.) are protected in the second component trace. Meanwhile, separating execution traces into component traces in the manner described herein still provides many of the same opportunities as more traditional cache-based recording techniques to reduce recording overhead and / or reduce the amount of data recorded into the execution trace.

[0008] Furthermore, separating execution tracing into distinct component traces, as described herein, reduces memory bandwidth usage compared to more conventional cache-based recording techniques. Specifically, processors recording component traces that include at least a portion of the CCP messages from memory access communications push these CCP messages onto the memory bus and into the execution trace (e.g., in system memory or persistent storage). However, unlike conventional cache-based recording techniques, this processor avoids pushing the corresponding memory values ​​onto the memory bus. Since the CCP message associated with a given memory access is significantly smaller than the memory value associated with that access, a processor recording component traces using the techniques described herein will push a much smaller amount of data onto the memory bus than a processor recording traces using conventional cache-based recording techniques.

[0009] The embodiments relate to methods, systems, and computer program products for recording replayable execution records based on logging CCP messages to a first trace and logging one or more memory snapshots to a second trace, wherein CCP messages can be used to retrieve memory values ​​from one or more memory snapshots. In these embodiments, a computer system determines to enable tracing of the execution of a first execution context by a plurality of processing units of a processor. Based on the determination to enable tracing of the execution of the first execution context, the computer system (i) begins logging one or more memory snapshots of the memory space of the first execution context to the second trace, and (ii) enables the processor's hardware tracing capabilities. Enabling the processor's hardware tracing capabilities causes the processor to log one or more CCP messages generated in response to one or more memory accesses by one of the plurality of processing units to the memory space of the first execution context. After enabling the processor's hardware tracing function, the computer system performs at least one of the following operations when executing the second execution context: (i) writing the second execution context into the memory space of the first execution context and logging it into one or more of the first or second trace; (ii) logging the identifier of the file mapped from the second execution context to the memory space of the first execution context into the second trace; (iii) writing cache lines in the cache that overlap with memory locations in the memory space of the first execution context, at least based on the second execution context, removing the cache lines from the cache or marking them as invalid; or (iv) writing a storage region within the memory space of the first execution context, at least based on the second execution context, marking the storage region as requiring logging in conjunction with the execution of the first execution context.

[0010] The embodiments also relate to methods, systems, and computer program products for a processor based on logging one or more CCP messages to a first trace, participating in the recording of replayable execution records, wherein the CCP messages can be used to retrieve memory values ​​from one or more memory snapshots stored in a second trace. In these embodiments, a processor including one or more processing units and a cache executes control logic that causes the processor to detect memory access by a first processing unit among a plurality of processing units. During execution context, the target of the memory access is a specific memory address. Based on the detection of the memory access, the processor logs information available for retrieving memory values ​​corresponding to the specific memory address from one or more memory snapshots stored in the second trace to the first trace. This includes logging the specific memory address and at least one of the following: (i) a first CCP message indicating that none of the plurality of processing units has a first cache line overlapping with the specific memory address in the cache; (ii) a second CCP message indicating that the first processing unit initiates a cache miss for the specific memory address; or (iii) a third CCP message indicating that the first processing unit obtains a second cache line overlapping with the specific memory address from a second processing unit of the plurality of processing units.

[0011] This summary is provided to introduce some concepts in a simplified form, which will be further described in the detailed description below. This summary is not intended to identify key or essential features of the claimed object, nor is it intended to help define the scope of the claimed subject matter. Attached Figure Description

[0012] To describe how the above-described and other advantages and features of this disclosure are obtained, the disclosure will be described in more detail with reference to specific embodiments shown in the accompanying drawings. It is understood that these drawings only illustrate typical embodiments of the disclosure and should not be considered as limiting its scope. A more specific and detailed description and explanation of the disclosure will be provided using the drawings, in which:

[0013] Figure 1A An example computing environment is shown that supports recording CCP messages related to memory accesses in the trace execution context to a first trace based on the processor, without recording memory values ​​associated with these memory accesses to the first trace, and recording one or more memory snapshots to a separate second trace based on the operating environment;

[0014] Figure 1BAn example of processor control logic is shown, which is based on the processor recording CCP messages related to memory accesses of the trace execution context into a first trace, while participating in recording a replayable execution trace without recording memory values ​​associated with these memory accesses into the first trace;

[0015] Figure 1C An example of an operating environment is shown, which participates in recording a replayable execution trace, at least based on recording one or more memory snapshots into a second trace;

[0016] Figure 2 An example processor cache is shown;

[0017] Figure 3 An example of performing a tracing is shown;

[0018] Figures 4A-4C Example memory access activities of multiple processing units are shown;

[0019] Figure 5A A flowchart illustrating an example method for recording a replayable execution trace in a computer system is shown. This method is based on recording CCP messages to a first trace and recording one or more memory snapshots to a second trace. The CCP messages can be used to retrieve memory values ​​from the one or more memory snapshots.

[0020] Figure 5B A flowchart is shown illustrating an example method by which a processor logs one or more CCP messages to a first trace. These CCP messages can be used to retrieve memory values ​​from one or more memory snapshots stored in a second trace. Detailed Implementation

[0021] At least some of the embodiments described herein separate execution traces into component traces. In these embodiments, separating execution traces into component traces facilitates data separation and the ability to store component traces in different security domains with different data security policies and different protection measures. This can protect sensitive memory values, such as PIIs, encryption keys, etc., while providing many of the same opportunities to reduce logging overhead and / or the amount of data logged into the execution trace, as with more traditional cache-based logging techniques. In these embodiments, separating execution traces into component traces also helps reduce memory bandwidth usage compared to more traditional cache-based logging techniques. Specifically, at least some of the embodiments described herein log at least a portion of CCP messages communicating between multiple processing units as a first component trace because these processing units perform memory accesses for the execution context, and log one or more snapshots of at least a portion of the system memory as a second component trace. In these embodiments, the first component trace contains sufficient information for profiling and / or debugging the execution context (e.g., analyzing processor cache usage of the execution context), but the first component trace lacks sufficient information to actually replay the recorded execution of the execution context. Nevertheless, by combining the information contained in the first component trace with the information contained in the second component trace, it is possible to replay the recorded execution of the execution environment.

[0022] To achieve these and other embodiments, Figure 1A An example computing environment 100 is illustrated, which supports an operating environment that records CCP messages associated with memory accesses of the trace execution context to a first trace based on the processor, without recording the memory values ​​associated with these memory accesses to the first trace, and based on an operating environment that records one or more memory snapshots to a separate second trace. Specifically, computing environment 100 includes a dedicated or general-purpose computer system 101, which includes an operating environment 110 configured to record a replayable execution trace of the execution context, at least based on recording one or more memory snapshots of the execution context memory to a component trace. Accordingly, computer system 101 includes one or more processors 102 configured to record one or more CCP messages during execution to a separate component trace associated with memory accesses of the trace execution context. The recorded CCP messages can be used to reconstruct the memory values ​​of these memory accesses using the memory snapshots recorded by operating environment 110. As shown, computer system 101 also includes at least system memory 103 (main memory) and persistent memory 104, which are communicatively coupled to each other via one or more communication buses 105 and connected to processor 102.

[0023] Embodiments within the scope of this disclosure may include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media may be any available media accessible to general-purpose or special-purpose computer systems. A computer-readable medium storing computer-executable instructions and / or data structures is a computer storage medium. A computer-readable medium carrying computer-executable instructions and / or data structures is a transmission medium. Therefore, by way of example and not limitation, embodiments of this disclosure may include at least two distinct types of computer-readable media: computer storage media and transmission media.

[0024] Computer storage media are physical storage media (e.g., system memory 103 and / or persistent memory 104) that store computer-executable instructions and / or data structures. Physical storage media include computer hardware such as RAM, ROM, EEPROM, solid-state drives (“SSDs”), flash memory, phase-change memory (“PCM”), optical disc storage, disk storage, or other magnetic storage devices, or any other hardware storage device that can be used to store program code in the form of computer-executable instructions or data structures, accessible and executed by general-purpose or special-purpose computer systems to achieve the functions disclosed herein.

[0025] Transmission media may include networks and / or data links, which can be used to transmit program code in the form of computer-executable instructions or data structures, and are accessible by general-purpose or special-purpose computer systems. A “network” is defined as one or more data links capable of transmitting electronic data between computer systems and / or modules and / or other electronic devices. When information is transmitted to or provided to a computer system via a network or other communication connection (hardwired, wireless, or a combination of hardwired and wireless), the computer system may consider that connection as a transmission medium. A combination of the foregoing should also be included within the scope of computer-readable media.

[0026] Furthermore, upon arrival at various computer system components, program code in the form of computer-executable instructions or data structures can be automatically transferred from the transmission medium to the computer storage medium (and vice versa). For example, computer-executable instructions or data structures received via a network or data link can be buffered in RAM within a network interface module (not shown) and then eventually transferred to the computer system RAM (e.g., system memory 103) and / or a less volatile computer storage medium (e.g., persistent memory 104) within the computer system. Therefore, it should be understood that computer storage media can be included in computer system components that also (even primarily) utilize the transmission medium.

[0027] For example, computer-executable instructions include instructions and data that, when executed on one or more processors, cause a general-purpose computer system, a special-purpose computer system, or a special-purpose processing device to perform a function or a set of functions. For example, computer-executable instructions can be machine code instructions (e.g., binary files), intermediate format instructions (e.g., assembly language), or even source code.

[0028] Those skilled in the art will understand that this disclosure can be implemented in network computing environments with many types of computer system configurations, including personal computers, desktop computers, laptops, information processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, microcomputers, mainframes, mobile phones, PDAs, tablets, pagers, routers, switches, etc. This disclosure can also be implemented in distributed system environments, where local and remote computer systems perform tasks via network connections (via hardwired data links, wireless data links, or a combination of hardwired and wireless data links). Therefore, in a distributed system environment, a computer system can include multiple constituent computer systems. In a distributed system environment, program modules can reside in local and remote memory storage devices.

[0029] Those skilled in the art will also understand that this disclosure can be implemented in a cloud computing environment. A cloud computing environment can be distributed, although this is not required. When distributed, a cloud computing environment can be internationally distributed within an organization and / or have components in multiple organizations. In this description and the following statements, “cloud computing” is defined as a model for enabling on-demand networked access to a shared pool of configurable computing resources, such as networks, servers, storage, applications, and services. The definition of “cloud computing” is not limited to any of the many other advantages that can be obtained from such a model when properly deployed.

[0030] Cloud computing models can be composed of various characteristics, such as on-demand self-service, broad network access, resource pooling, rapid elasticity, and measurable services. Cloud computing models can also appear in various service models, such as Software as a Service (“SaaS”), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS). Cloud computing models can also be deployed using different deployment models, such as private clouds, community clouds, public clouds, hybrid clouds, and so on.

[0031] Some embodiments (e.g., cloud computing environments) may include a system comprising one or more hosts, each capable of running one or more virtual machines. During operation, the virtual machines emulate an operating computing system, supporting an operating system and perhaps one or more other applications. In some embodiments, each host includes a hypervisor that uses physical resources extracted from the virtual machine view to simulate the virtual resources of the virtual machines. The hypervisor also provides appropriate isolation between virtual machines. Thus, from the perspective of any given virtual machine, the hypervisor provides the illusion that the virtual machine interacts with physical resources, even if the virtual machine only interacts with the appearance of physical resources (e.g., virtual resources). Examples of physical resources include processing capacity, memory, disk space, network bandwidth, media drives, etc.

[0032] like Figure 1A As shown, in this embodiment, each processor 102 includes one or more processing units 106, one or more caches 107, control logic 109, and in some embodiments, one or more translation query buffers (TLBs) 108. Typically, each processing unit 106 (e.g., a processor core) executes machine code instructions at execution unit 106b. During the execution of these machine code instructions, at least some instructions use internal processor registers 106a as temporary storage locations and / or read from and write to various locations in system memory 103 via cache 107. Each processing unit 106 in a given processor 102 executes machine code instructions selected from a predefined processor instruction set architecture (ISA) of the corresponding processor 102. The specific ISA for each processor 102 may vary depending on the processor manufacturer and processor model. Common ISAs include the IA-64 and IA-32 architectures of Intel, Inc., the AMD64 architecture of Advanced Micro Devices, Inc., and various advanced RISC machine (“ARM”) architectures of ARM Holdings, PLC, although a large number of other ISAs exist and this disclosure may use these ISAs. Typically, machine code instructions are the smallest externally visible (i.e., external to the processor) code units that a processor can execute.

[0033] Register 106a is a hardware-based storage location defined according to the ISA of its corresponding processor 102. Typically, register 106a is read from and / or written to by machine code instructions or processing unit 106 during the execution of these instructions in execution unit 106b. Register 106a is typically used to store values ​​retrieved from cache 107 for use as input machine code instructions, store the results of executing machine code instructions, store program instruction counts, support thread stack maintenance, etc. In some embodiments, register 106a may include "flags" to indicate certain state changes caused by the execution of machine code instructions (e.g., indicating whether an arithmetic operation resulted in a carry, a zero result, etc.). In some embodiments, register 106a includes one or more control registers (e.g., for controlling different aspects of processor operation) and / or other processor-specific registers (MSRs).

[0034] During the execution of machine code instructions by processing unit 106, cache(s) 107 temporarily cache portions of system memory 103. In some embodiments, cache 107 includes one or more “code” portions of the system memory 103 portion that caches application code, and one or more “data” portions of the system memory 102 portion that caches application runtime data. If processing unit 106 requires data (e.g., code or application runtime data) that is not already stored in cache 107, processing unit 106 initiates a “cache miss”, causing the required data to be fetched from system memory 103 and entered into cache 107, while potentially replacing and “evicting” some other data already stored in cache 107, returning it to system memory 103.

[0035] In an embodiment, each cache 107 includes multiple "cache lines," each cache line storing a memory block from backing memory (e.g., system memory 103). For example, Figure 2An example of a processor cache 200 comprising multiple cache lines 203 is shown. In this example, each cache line 203 includes at least an address portion 201 and a value portion 202. In an embodiment, the address portion 201 in each cache line 203 is configured to store at least one memory address in the system memory 103 corresponding to the cache line, and the value portion 202 initially stores a data value obtained from that address in the system memory 103. Depending on the size of the value portion 202, each cache line 203 may store data spanning multiple consecutive individually addressable locations in the system memory 103. The value portion 202 of each cache line 202 may be modified by the processing unit 106 and eventually evicted from the system memory 103. As indicated by ellipses within the processor cache 200, cache 107 may include a large number of cache lines. For example, a modern 64-bit Intel processor may include a single L1 cache for each processing unit 106 containing 512 or more cache lines. In such a cache, each cache line can typically be used to store a 64-byte (512-bit) value by referencing a 6-byte (48-bit) to 8-byte (64-bit) memory address.

[0036] In some cases, the address portion of the cache line of cache(s) 107 stores a physical memory address, such as the actual corresponding memory address in system memory 103. In other cases, the address portion of the cache line of cache(s) 107 stores a virtual memory address. In an embodiment, a virtual memory address is an address in a virtual address space exposed by the operating system to processes executing on processor(s) 102. This virtual address space provides one or more abstractions, such as processes having their own dedicated memory space and / or processes having more available memory than in system memory 103. For example, such abstractions can be used to facilitate memory isolation between different processes executing on processor(s) 102, including isolation between user-mode processes and kernel-mode processes. In an embodiment, the virtual-to-physical memory address mapping is maintained in memory page tables stored in system memory 103 and managed by the operating system and / or a hypervisor (e.g., operating environment 110). Typically, these memory page tables include multiple page table entries (PTEs) that map a range of virtual memory addresses (i.e., pages) to regions of physical memory addresses (e.g., pages). In embodiments, each PTE stores additional attributes or flags about its corresponding memory page, such as memory page permissions (e.g., read-only, writable, etc.), page status (e.g., dirty, clean, etc.). In embodiments, the TLB 108 within each processor 102 supports virtual addressing and includes a dedicated form of cache for storing recently acquired PTEs that map virtual and physical memory pages, as obtained from memory page tables stored in system memory 103. In some embodiments, the PTE is part of a multi-level hierarchy that includes one or more Page Directory Entries (PDEs) supporting the discovery of individual PTEs. In embodiments where processor 102 lacks TLB 108, the processor may lack support for virtual memory addressing.

[0037] In embodiments, cache 107 is divided into separate layers, tiers, or levels, such as tier 1 (L1), tier 2 (L2), and tier 3 (L3), etc. Depending on the processor implementation, one of these layers may be part of processor 102, itself (e.g., L1 and L2), and / or may be separate from processor 102 (e.g., L3). Thus, in some embodiments, cache 107 includes one of these layers (L1), while in other embodiments cache 107 includes multiple such layers (e.g., L1 and L2, or even L3). In some embodiments, each of the plurality of processing units 106 includes or is associated with its own L1 cache, while each L2 cache is a shared cache used as backing storage for two or more L1 caches (and thus shared by two or more processing units 106). In embodiments, the individual L1 caches are also considered shared because they coordinate with each other via Cache Coherence Protocol (CCP) messages. This coordination ensures data consistency when multiple processing units access the same memory location (e.g., by ensuring consistent viewing of the memory location of each cache across all L1 caches). In this embodiment, multiple L2 and / or L3 caches are similarly coordinated via the processor's CCP. If the processor 102 supports Hyper-Threading, each individual L1 cache shared by two or more logical processing units can also be viewed.

[0038] The processor's CCP defines how to maintain consistency in cache 107 when the various processing units 106 of processor 102 read and write data to cache 107, and how to ensure that the various processing units 106 always read valid data from a given location in cache 107. CCPs are typically associated with the memory model defined by the processor ISA. Examples of common CCPs include the MSI protocol (i.e., Modify, Share, Invalidate), the MESI protocol (i.e., Modify, Exclusive, Share, Invalidate), and the MOESI protocol (i.e., Modify, Own, Exclusive, Share, Invalidate). Each of these protocols defines a state for various locations (e.g., lines) in a shared cache. A "modified" cache location contains data that has been modified in the cache and is therefore inconsistent with the corresponding data in backup memory (e.g., system memory 103 or another cache). When a location with a "modified" state is evicted from the cache, a typical CCP requires the cache to guarantee that its data is written back to backup memory, or another cache to take over this responsibility. A "shared" cache location contains data that has not been modified from backup storage, exists in a read-only state, and is shared by processing unit 106. This data can be evicted from the cache without being written to backup storage. An "invalid" cache location contains no valid data and can be considered empty; it can be used to store cache misses. An "exclusive" cache location contains data that matches the backup storage and is used only by a single processing unit 106. It can be changed to a "shared" state at any time (i.e., in response to a read request) or to a "modified" state upon writing. An "owned" cache location is shared by two or more processing units 106, but one processing unit has exclusive rights to modify it. When this change occurs, it notifies other processing units, as the notified units may need to invalidate or update the cache based on the CCP implementation. In this embodiment, there is an inherent overall ordering of CCP messages.

[0039] In embodiments, the control logic 109 of each processor 102 includes microcode (i.e., executable instructions) and / or on-die gate logic that controls the operation of the processor 102. Typically, the control logic 109 acts as an interpreter between the hardware of the processor 102 and the processor ISA exposed by the processor 102 to execute applications (e.g., operating environment 110 and application 111), and controls the internal operations of the processor 102. In embodiments, the control logic 109 is embodied in processor memory, such as ROM, EEPROM, etc. In some embodiments, the memory on the processor is writable (in which case, the control logic 109 is updatable), while in other embodiments, the memory on the processor is read-only (in which case, the control logic 109 cannot be updated).

[0040] Although not shown, in some embodiments, processor 102 supports the creation of more secure memory enclaves for storing data and / or executable instructions. For example, in one embodiment, processor 102 provides machine code instructions (e.g., Intel Software Daemon Extensions (SGX)) that allow user-level and operating system code to define private regions of memory, called enclaves, whose contents are protected and cannot be read or saved by any process outside the enclave itself, including processes running at higher privilege levels. In one embodiment, the shell-supporting processor 102 encrypts a portion of system memory 103; this portion of memory is then decrypted only within processor 102 itself, and even then, only for code and data running from within the enclave.

[0041] Persistent memory 104 stores computer-executable instructions and / or data structures representing executable software components. Accordingly, during the execution of these software components at processor 102, one or more portions of these computer-executable instructions and / or data structures are loaded into system memory 103. For example, persistent memory 104 is shown storing computer-executable instructions and / or data structures corresponding to operating environment 110 and one or more applications 111. Accordingly, system memory 103 is shown storing one or more operating environment runtimes 110' (e.g., machine code instructions and / or runtime data supporting the execution of operating environment 110), and storing one or more application runtimes 111' (e.g., machine code instructions and / or runtime data supporting the execution of one or more applications 111).

[0042] exist Figure 1A In this embodiment, operating environment 110 is shown as potentially including a hypervisor 110a and one or more operating systems 110b. Correspondingly, operating environment runtime 110' is shown as possibly including a hypervisor runtime 110a' and one or more operating system runtimes 110b'. For example, in some embodiments, operating environment 110 includes a hypervisor 110a that executes directly on the hardware of computer system 101 (e.g., processor 102, system memory 103, and persistent memory 104), and one or more operating systems 110b that execute on the virtual machine monitor 110a. However, in other embodiments, operating environment 110 includes an operating system 110b that executes directly on the hardware of computer system 101 (e.g., processor 102, system memory 103, and persistent memory 104).

[0043] In embodiments, system memory 103 and persistent memory 104 also store other data, such as one or more replayable execution traces (i.e., execution trace 112' stored in system memory 103 and / or execution trace 112 stored in persistent memory 104). In some embodiments, the replayable execution trace is initially stored in system memory 103 during generation (i.e., execution trace 112') and subsequently persisted to persistent memory 104 (i.e., execution trace 112). In other embodiments, the replayable execution trace is initially stored in persistent memory 104 during generation (i.e., execution trace 112). As shown, in the embodiments herein, each execution trace 112 / 112' consists of two component traces (or component trace data streams), component traces 112a / 112a' and component traces 112b / 112b'. In embodiments, component traces 112a / 112a' are generated directly by processor 102 (e.g., under the guidance of control logic 109), while component traces 112b / 112b' are generated by operating environment 110. The specific content of each component trace 112a / 112a' and 112b / 112b' will be discussed subsequently. For brevity, this disclosure relates to execution traces using reference numeral 112, and component traces using reference numerals 112a and 112b. Unless expressly stated otherwise or the context otherwise requires, reference numeral 112 is additionally or alternatively used to apply to reference numeral 112'; additionally or alternatively, any use of reference numeral 112a applies to reference numeral 112a'; and additionally or alternatively, any use of reference numeral 112b applies to reference numeral 112b'. It is worth noting that some embodiments store component traces 112a and 112b together (e.g., as a data stream in the same file, as files in the same directory, etc.), while other embodiments store component traces 112b and 112a separately—e.g., on different storage devices, on different computer systems, etc. In these embodiments, storing component traces 112a and 112b separately allows them to be stored in different security domains, under different policies, using different security protections, etc.

[0044] In embodiments, each replayable execution trace 112 contains a “bit-accurate” execution record of the corresponding context (e.g., process, operating system, virtual machine, enclave, hypervisor, etc.) as the context executed at processor 102. Thus, as used herein, a replayable execution trace is a “bit-accurate” record of the execution activity of that context. This bit-accurate record allows machine code instructions previously executed as part of the context of processing unit 106 to be replayed subsequently so that, during replay, these machine code instructions are re-executed in the same order and using the same data as during the trace recording. The cache-based tracing technique used in the embodiments herein is based on the observation that each processor 102 (including its cache 107) comprises a semi-closed or quasi-closed system. For example, once a portion of the data for the execution context (i.e., machine code instructions and runtime data) is loaded into the processor’s cache 107, processing unit 106 can continue executing that context without any other external input, handling burst events as a semi-closed or quasi-closed system. In particular, once cache 107 is loaded with machine code instructions and runtime data, execution unit 106b can use the runtime data stored in the data portion of cache 107 as input to these machine code instructions, and use register 106a to load and execute these machine code instructions from the code portion of cache 107. As long as the data required for processor 102 to execute (i.e., machine code instructions and runtime data) exists in cache 107, processor 102 can continue executing that context without further external input.

[0045] When processing unit 106 requires some data inflow (e.g., because machine code instructions it is executing, will execute, or may execute access code or runtime data not in cache 107), processor 102 performs a "cache miss" and imports appropriate data from system memory 103 into cache 107. For example, if a data cache miss occurs when processing unit 106 executes machine code instructions that perform memory operations on a memory address within application runtime 111' storing runtime data, processor 102 imports runtime data from that memory address in system memory 103 into one of the cache lines in the data portion of cache 107. Similarly, if a code cache miss occurs when processing unit 106 attempts to fetch machine code instructions from a memory address within application runtime 111' storing application code, processor 102 imports code data from that memory address in system memory 103 into one of the cache lines in the code portion of cache 107. Processing unit 106 then continues execution with the newly imported data until new data is needed.

[0046] In the implementation, as a general overview, the control logic 109 of each processor 102 is configured to participate in recording a bit-precise representation of the execution context associated with memory accesses made via cache 107 by recording at least a subset of CCP messages communicated between processing units 106 of the processor and / or with other processors 102 into a component trace 112a corresponding to the execution context. In the implementation, for memory accesses made by the execution context via cache 107, the control logic 109 records the accessed memory address in component trace 112a, but avoids recording any corresponding memory values. On the other hand, the operating environment 110 is configured to participate in recording a bit-precise representation of the execution environment by recording at least one or more memory values ​​obtained from system memory 103 (e.g., in the form of one or more memory snapshots) into another component trace 112b corresponding to the execution environment. As detailed later, component trace 112a can be used to determine which memory addresses the context accessed during execution (including which processing unit made each access and in what order), while component trace 112b can be used to obtain and / or calculate the accessed memory values. Therefore, component traces 112a and 112b can be combined to reproduce the situation where information flows into the processor's cache 107 when the processor's processing unit 106 executes the code of the context.

[0047] In an embodiment, the control logic 109 of each processor 102 is also configured to record any uncached reads performed by the execution context (e.g., reads from direct memory access (DMA) hardware devices, reads from portions of system memory 103 marked as non-cached, etc.) into component trace 112a, as well as one or more side effects of executing any nondeterministic instructions (e.g., register values ​​changed due to the execution of nondeterministic instructions). In an embodiment, as will be explained in more detail later, one or both of the control logic 109 or operating environment 110 also indicate the activity of other execution contexts that modify the memory space of the execution context being recorded.

[0048] Figure 3 An example of an execution trace is shown (e.g., execution trace 112). Specifically, Figure 3 An execution trace 300 is shown, which includes a first component trace 301 (e.g., corresponding to component trace 112a) and a separate and distinct second component trace 305 (e.g., corresponding to component trace 112b). Typically, component trace 301 stores information directly recorded by processor 102 (e.g., under the instruction of tracer 117 in control logic 109), while component trace 305 stores information recorded by operating environment 110.

[0049] As shown in the figure, component tracing 301 includes one or more data streams 302 (i.e., data streams 302a-302n). In embodiments, different data streams 302 correspond to different threads, different types of tracing data, different tracing data security domains, etc. As shown in data stream 302a, in an embodiment, each data stream 302s includes multiple data packets 304. Because the specific data logged in each data packet 304 varies depending on the specific data start represented by the data packet, these data packets 304 are shown to have different sizes. In embodiments, data packets 304 are used to log information such as CCP messages (or their flags) related to memory accesses processed by cache 107, memory address information related to memory accesses processed by cache 107, hashes of the values ​​of one or more registers 106a, timing information (e.g., instruction counts), code tracing information (e.g., from branch or call traces), logs not cached (i.e., read memory accesses bypassing cache 107), references to executed instructions (e.g., using addresses or offsets in the application binary image), etc. In one embodiment, data stream 302a also includes data packets consisting of one or more keyframes 303 (e.g., keyframes 303a and 303b), each keyframe recording information sufficient to restore the initial processor state to initiate trace replay, such as a snapshot of register values. On the other hand, component trace 305 includes one or more memory snapshots 306 storing memory values ​​obtained from system memory 103. In some embodiments, component trace 302 also includes data stream 307, which includes data packets storing memory writes, timing information, counting information, sorting information, etc. Given this description, it can be understood that component trace 301 primarily includes records of instruction execution and cache activity of processor 102, but typically lacks memory values ​​read by processor 102. On the other hand, component trace 305 primarily includes "bulk data," such as in the form of memory snapshots from which read values ​​can be obtained.

[0050] In one embodiment, execution trace 112 also includes the actual code executed by processing unit 106, such as a record of instructions executed in trace data stream 302 of component trace 301, or a copy of the application binary stored in component trace 305. However, in other embodiments, execution trace 112 omits the actual code executed and instead relies on separate access to application code (e.g., application 111).

[0051] To illustrate how the operating environment 110 and control logic 109 record the precise bit representation of the execution context in the different component traces 112a and 112b, Figure 1BThe control logic 109 is shown, including embodiments of the components used when recording the execution of the execution context to the first component trace 112a (e.g., trace enabler 115, cache refresher 116, tracer 117, etc.). Furthermore, Figure 1C Further details of the operating environment 110 are shown, including components of the tracker 118 used by embodiments of the operating environment 110 when recording the execution of the execution context to the second component tracker 112b (e.g., cache refresh initiator 118a, memory snapshot manager 118b, processor track enabler 118c, external active processor 118d, track encryptor 118e, etc.). The components shown for the control logic 109 and the operating environment 110, as well as any sub-components, represent various functionalities that the control logic 109 and / or the operating environment 110 can implement or utilize according to the various embodiments described herein. However, it should be understood that the described components (including their identifiers, sub-components, and arrangements) are only intended to aid in describing the various embodiments of the control logic 109 and / or the operating environment 110 described herein, and these components do not limit how the software and / or hardware implement the various embodiments of the control logic 109 and / or the operating environment 110 described herein, or their specific functionalities.

[0052] As a general description of the operating environment 110, the hypervisor 110a and / or operating system 110b include a tracer 118 that supports the initiation of one or more execution contexts on the tracer processor 102 and the component tracer 112b that creates these contexts. In embodiments, the tracer 118 executes at least partially as a component of the hypervisor 110a and / or as a kernel-mode component of at least one of the operating systems 110b. In other embodiments, the tracer 118 executes at least partially as a user-mode component of at least one of the operating systems 110b. In embodiments, the tracer 118 has multiple parts that execute in two or more contexts: (i) user mode of at least one of the operating systems 110b, (ii) kernel mode of at least one of the operating systems 110, or (iii) hypervisor 110a. In some embodiments, tracer 118 is an integrated component of at least one of hypervisor 110a and / or operating system 110b, while in other embodiments (e.g., by application 111), tracer 118 is provided as an extension of hypervisor 110a and / or operating system 110b (e.g., as an operating system driver, kernel extension, kernel module, hypervisor module, or extension, etc.). Therefore, even though tracer 118 is shown to be executed (at least partially) as a component of operating environment 110, tracer 118 can actually be provided to operating environment 110 by a third party (e.g., via one of applications 111).

[0053] Turning now to the components of tracker 118, in some embodiments, before starting to trace a given execution context, cache refresh initiator 118a ensures that cache 107 is in a known state by refreshing and / or invalidating one or more portions of cache 107, or ensures that cache 107 is in a reproducible state by capturing one or more portions of cache 107 (e.g., entering one of component traces 112a or 112b). In some embodiments, cache refresh initiator 118a only refreshes, invalidates, or snapshots the cache lines corresponding to memory pages within the memory space of the execution context that enables tracing. Cache refresh initiator 118a is shown in dashed lines because in some embodiments, it is control logic 109 that refreshes, invalidates, and / or snapshots cache 107 (e.g., in response to a request to enable tracing for an execution context).

[0054] Similarly, before initiating tracing of a given execution context, the memory snapshot manager 118b initiates the storage of one or more snapshots of system memory 103 into component trace 112b (e.g., as memory snapshot 306). In some embodiments, the memory snapshot manager 118b snapshots only the portion of system memory 103 corresponding to the memory space of the execution context for which tracing is enabled. In embodiments, the memory snapshot manager 118b associates the memory snapshots with timestamps, counts, etc. There are various embodiments of how the memory snapshot manager 118b initiates memory snapshots. In some embodiments, the memory snapshot manager 118b stores at least one memory snapshot into component trace 112b before starting tracing of the execution context. In some embodiments, the memory snapshot includes each physical memory page mapped to the memory space of the execution context for which tracing is enabled. In variations of this embodiment, the memory snapshot includes only the physical memory pages that actually exist in system memory 103, i.e., only the physical memory pages that have not yet been swapped out to a swap file or swap partition (e.g., on persistent storage 104). In this variation, the memory snapshot manager 118b only snapshots the retrieved physical memory pages to component trace 112b (if the page is subsequently retrieved back to system memory). In another embodiment, the memory snapshot manager 118b initiates memory snapshots by tracing the memory pages actually accessed by the traced execution context, and the memory snapshot manager 118b snapshots these pages as needed while tracing the execution context's actual access. In some embodiments, in addition to initiating memory snapshots before starting to trace the execution context, the memory snapshot manager 118b also initiates and stores occasional memory snapshots during subsequent execution of the traced execution context. In various embodiments, these subsequent memory snapshots are either independent memory snapshots or differential memory snapshots that depend on one or more previous memory snapshots.

[0055] Processor tracing enabler 118c initiates hardware tracing functionality at processor 102 for a given execution context, for example, by executing appropriate instructions at processor 102. In an embodiment, initiating hardware tracing functionality at processor 102 includes processor tracing enabler 118c communicating an identifier of the execution context to be enabled for tracing to processor 102, for example, by setting a value in system memory 103 or register 106a. In some embodiments, the identifier of the execution context is communicated via a Virtual Processor Identifier (VPID).

[0056] In many cases, a trace execution context can be executed in parallel or sequentially with one or more other execution contexts, such as an operating system kernel. In some cases, one or more of these "external" execution contexts may write to the memory space of the trace execution context. In one example, in response to an interrupt from a hardware device, the operating system kernel processes a direct memory access (DMA) request that writes to the memory space of the trace context. In embodiments, the external active processor 118d ensures that any writes to the memory space of the trace context are logged to execution trace 112, or takes some other action to ensure that the writes are logged. In some embodiments, the external active processor 118d invokes a memory snapshot manager 118b to create one or more snapshots of the modified memory (e.g., within memory snapshot 306). In some embodiments, the external active processor 118d logs one or more writes to the data stream of component trace 112b (e.g., in data stream 306). In some embodiments, the external active processor 118d instructs the processor 102 to log one or more writes to component trace 112b (e.g., through communication with external active processor 117e). In some embodiments, the external activity processor 118d associates the writes to be logged with timestamps, counts, etc. In some embodiments, the external activity processor 118d causes any cache lines in cache 107 that have been modified by the writes to be evicted or invalidated (thus ensuring a subsequent cache miss if the tracking context reads the modified memory location). In some embodiments, the external activity processor 118d marks the memory page corresponding to the write for subsequent logging (e.g., via a trap to tracker 118 associated with a subsequent execution of the tracking execution context).

[0057] Because component trace 112b contains a memory snapshot and may contain information related to writing to the trace context memory space, component trace 112b may contain sensitive information (e.g., PII, encryption keys, etc.). Therefore, in an embodiment, trace encryptor 118e encrypts all or part of component trace 112b to protect this sensitive information.

[0058] Turning now to control logic 109, tracing enabler 115 enables hardware-based tracing functionality for processor 102. In an embodiment, tracing enabler 115 operates in response to a request from operating environment 110 (e.g., processor tracing enabler 118c). Prior to enabling tracing, cache refresher 116 ensures cache 107 is in a known state by flushing and / or invalidating one or more portions of cache 107, or ensures cache 107 is in a replayable state by taking a snapshot of one or more portions of cache 107 (e.g., to one of component tracing 112a or 112b). In some embodiments, cache refresher 116 operates in response to instructions from cache refresh initiator 118a, while in other embodiments cache refresher 116 operates in response to instructions from tracing enabler 115. Given the foregoing discussion of cache refresh initiator 118a, it can be understood that in some embodiments cache flusher 116 refreshes, invalidates, or simply snapshots cache lines corresponding to memory pages within the memory space of the execution context where tracing is enabled.

[0059] Tracker 117 is responsible for logging certain types of processor activity (such as CCP messages) to component trace 112a. In an embodiment, memory access detector 117a detects when processing unit 106 of the execution trace execution context accesses a target memory address in system memory 103. In some cases, memory access is directed to a target memory location cached by cache 107. In these cases, based at least on the memory access detected by memory access detector 117a, CCP logger 117b determines whether to log one or more CCP messages associated with the memory access and initiates any appropriate logging to component trace 112a. In an embodiment, CCP logger 117b associates the logged CCP messages with timestamps, counts, etc.

[0060] Now for reference Figures 4A-4C An example of CCP message logging is provided, illustrating memory access activity across multiple processing units. Specifically, Figure 4AAn example 400a is shown, which includes processing units 401 (i.e., processing units 401a-401d) and may include two shared caches 402 (i.e., shared cache 402a shared by processing units 401a and 401b, and shared cache 402b shared by processing units 401c and 401d) and system memory 403. Example 400a shows that system memory 403 has multiple storage locations, each storing a corresponding stored value (i.e., storage location 403a for stored value 7, storage location 603b for stored value 22, etc.). In example 400a, processing unit 401a executes instructions to access (e.g., read or write) memory location 403a. In an embodiment, processing unit 401a sends a CCP message to other processing units 401 to inquire whether the other processing units have a cache copy of memory location 403a. In example 400a, none of the processing units 401b, 401c, or 401d have a cache copy of memory location 403a. Therefore, processing unit 401a initiates a cache miss at memory location 403a (e.g., based on processing unit 401a receiving a negative response from processing units 401b, 401c, and 401d; or based on processing unit 401a not receiving a positive response from processing units 401b, 401c, and 401d). Therefore, as shown, processing unit 401a obtains (e.g., into its L1 cache) a cache copy of memory location 403a, which includes the value 7. As illustrated, if included, shared cache 402a (e.g., L2 cache) may also obtain a copy of memory location 403a, including the value 7. In an embodiment, CCP logger 117b logs one or more first CCP messages to component trace 112a, which provides one or more indications that processing unit 401a has obtained memory location 403a from system memory 403. For example, one or more CCP messages indicate that no other processing unit has a copy of memory location 403a (e.g., a copy of a negative response from processing units 401b, 401c, and 401d; one or more indications that processing unit 401a has not received a positive response from processing units 401b, 401c, and 401d, etc.), or indicate that processing unit 401a initiates one or more CCP messages (possibly rounded from the cache line number) at the address of memory location 403a.

[0061] Go to Figure 4BIn example 400b, processing unit 401a has already written the value 11 to memory location 403a. Subsequently, processing unit 401b executes an instruction to access memory location 403a. In this embodiment, processing unit 401b sends a CCP message to other processing units 401 to inquire whether the other processing units have a cache copy of memory location 403a. In example 400b, processing unit 401a has a cache copy of memory location 403a; therefore, processing unit 401a sends a CCP message to processing unit 401b indicating this CCP message. Thus, as shown, processing unit 401b obtains (e.g., into its L1 cache) the cache copy of memory location 403a from processing unit 401a, including the value 11 (e.g., from processing unit 401a's L1 cache). In an embodiment, CCP logger 117b logs one or more second CCP message logs to component trace 112a, provides processing unit 401b with one or more indications of memory location 403a obtained from the LI cache of processing unit 401a, and sends a copy of CCP message processing unit 401 to processing unit 401b.

[0062] Go to Figure 4CFor example, processing units 401a and / or 401b have evicted memory location 403a from their cache, so system memory 403 now stores the value 11 of memory location 403a. In some embodiments, CCP logger 117b logs one or more third CCP messages to component trace 112a to provide one or more indications of these evictations. Processing unit 401c then executes instructions to access memory location 403a. In an embodiment, processing unit 401c sends CCP messages to other processing units 401 to inquire whether the other processing units have a cached copy of memory location 403a. In example 400c, processing units 401a, 401b, or 401d do not have a cached copy of memory location 403a. Therefore, processing unit 401c enables a cache miss at memory location 403a (e.g., based on processing unit 401c receiving a negative response from processing units 401a, 401b, and 401d; or based on processing unit 401c not receiving a positive response from processing units 401b, 401c, and 401d). Thus, as shown, processing unit 401c obtains (e.g., into its L1 cache) a cache copy of memory location 403a, including the value 11. As shown, if included, shared cache 402b (e.g., L2 cache) can also obtain a copy of memory location 403a, including the value 11. In an embodiment, CCP logger 117b logs one or more fourth CCP messages to component trace 112a. These CCP messages provide one or more indications that processing unit 401c has obtained memory location 403a from system memory 403. For example, one or more CCP messages may indicate that no other processing unit has a copy of memory location 403a (e.g., a copy of a negative response from processing units 401a, 401b, and 401d); that processing unit 401c has not received one or more indications of a positive response from processing units 401a, 401b, and 401d, etc.; or that processing unit 401c has initiated one or more CCP messages (possibly rounded from the cache line number) at the address of memory location 403a.

[0063] It is worth noting that the memory accesses in Examples 400a-400c are based on an initial memory snapshot of system memory 403 (e.g., within component trace 112b) and are reproducible at least based on CCP messages logged by CCP logger 117b (within component trace 112a). For example, the access to memory location 403a by processing unit 401a in Example 400a (i.e., reading value 7) can be reproduced by replaying a cache miss to system memory 403 using one or more first CCP messages (and obtaining value 7 using a memory snapshot). In Example 400b, the access to memory location 403a by processing unit 401b (i.e., reading value 11) is reproducible by replaying the instruction to write value 11 to memory location 403a by processing unit 401a and replaying the value of memory location 403+403a obtained from processing unit 401a by processing unit 401b using one or more second CCP messages. In Example 400c, an access to memory location 403a by processing unit 401c (i.e., reading value 11) can be replayed by replaying a cache miss to system memory 403 using one or more fourth CCP messages. It is worth noting that one or more third CCP messages (evict) can be used, or, if one or more third CCP messages are not logged, it is inferred that the write to processing unit 401a occurred before the read by processing unit 401c. Therefore, all accesses to main memory in Examples 400a-400c are replayable without logging any memory values ​​to component trace 112a.

[0064] In other cases, memory accesses are directed to an "uncached" target memory address, thus bypassing cache 107. In these cases, the uncached read logger 117c logs the memory access to execution trace 112, at least based on the memory access detected by memory access detector 117a. In some embodiments, the uncached read logger 117c logs one or both of the target memory address of the memory access or the value read by the uncached memory access in execution trace 112. When recording a value read by an uncached memory access, some embodiments of the uncached read logger 117c log the value to component trace 112a. In some of these embodiments, the uncached read logger 117c encrypts the value within component trace 112a. In one example, the uncached read logger 117c encrypts packet 304 or a portion thereof within component trace 112a where the value is stored. In another example, the uncached read logger 117c stores the value in an encrypted data stream within component trace 112a. Other embodiments of the uncached read logger 117c log the value to component trace 112b, for example, via a trap to tracer 118. In embodiments, the uncached read logger 117c associates the logged uncached reads with timestamps, counts, etc.

[0065] Sometimes, the trace context may execute indeterminate processor instructions, defined as instructions whose output does not entirely depend on their input. For example, the RDTSC instruction in the Intel IA-64 and IA-32 architectures does not accept input; upon invocation, it returns the value of a processor timestamp counter as output to a register. Therefore, the instruction's output is not entirely dependent on its input. In an embodiment, the nondeterministic instruction logger 117d logs the execution of a nondeterministic instruction based on recording one or more side effects of executing the instruction. In an embodiment, the side effects of instruction execution include modifying one or more registers 106a. Therefore, in an embodiment, the nondeterministic instruction logger 117d logs the execution of a nondeterministic instruction by at least re-encoding the values ​​of one or more registers 106a modified by the executed instruction into the component trace 112a. In an embodiment, the nondeterministic instruction logger 117d associates the logged nondeterministic instructions with timestamps, counts, etc.

[0066] As described by the external activity processor 118d of operating environment 110, the trace execution context can execute concurrently or sequentially with one or more other execution contexts, and one (or more) of these "external" execution contexts can write to the memory space of the trace execution context. In embodiments, the external activity processor 117e cooperates with or operates in place of the external activity processor 118d to ensure that any writes to the memory space of the trace context are logged to execution trace 112, or takes other actions to ensure that writes are logged. In some embodiments, the external activity processor 117e logs at least one of the memory address or write value to component trace 112a (e.g., writing to an encrypted data stream or packet) or component trace 112b. In embodiments, the external activity processor 117e associates the logged writes with timestamps, counts, etc. In some embodiments, the external activity processor 117e evicts or invalidates cache lines that overlap with the written memory address. In some embodiments, the external activity processor 117e marks the memory page corresponding to the written memory address as needing to be logged. In some embodiments, the external active processor 117e initiates a trapped processor (e.g., via a software interrupt) that logs memory pages corresponding to memory addresses.

[0067] As previously described, some embodiments associate events with timestamps, counts, etc. As mentioned earlier, there is an inherent overall ordering of CCP messages. Therefore, in some embodiments, based on one or two inherent orders of logged CCP events (e.g., CCP messages logged in component trace 112a), or timestamps, counts, the order of related events across component trace 112b can be identified in relation to other logged events (e.g., writes, indeterminate instructions, memory snapshots, etc., logged in and / or in component trace 112a). Note that in these embodiments, only the general order of events needs to be determined, not the exact time of the events. For example, if a user-mode execution context reads data from a given memory location, a kernel execution context subsequently writes to that memory location, and a user-space execution context subsequently reads from that memory location again, some embodiments log enough information to order the writes after the first read and before the second read, but may not log the exact time of the write. In these embodiments, all that is needed for proper playback is the ability to apply writes to any time within the window between the first and second reads. In one example, the first and second reads are logged as CCP messages by CCP logger 117b (e.g., logged to component trace 112a), and an instruction count written to the reference external activity processor 118d is also logged (e.g., logged to component trace 112b). Since CCP messages are themselves orderable, in this example, if the instruction count is orderable in the CCP protocol, then the instruction count is sufficient to order the writes between the first and second reads.

[0068] The further operation of control logic 109 and operating environment 110 is now described in more detail, mainly related to... Figure 5A and 5B related. Figure 5A A flowchart of an example method 500a for recording a replayable execution trace in a computer system is shown. This method is based on recording CCP messages to a first trace and recording one or more memory snapshots to a second trace. The CCP messages can be used to obtain memory values ​​from one or more memory snapshots. Figure 5B The flowchart shows an example method 500b in which the processor logs one or more CCP messages to a first trace. These CCP messages can be used to retrieve memory values ​​from one or more memory snapshots stored in a second record. The following discussion involves many methods and method behaviors. While method behaviors may be discussed in a specific order, or illustrated in the flowchart as occurring in a specific order, a specific order is not required unless specifically stated otherwise, or because the behavior is specifically described as depending on another behavior that completes prior to its execution.

[0069] refer to Figure 5A and 5B Method 500a includes actions 501-506 performed by the operating environment (e.g., hypervisor 110a and / or operating system 110b), while method 500b includes actions 507-513 performed by the processor (e.g., processor 102 based on control logic 109). Therefore, in some embodiments, methods 500a and 500b are independent and distinct methods (i.e., one implemented and executed independently under the guidance of operating environment 110, and the other implemented and executed separately under the guidance of control logic 109), while in other embodiments, methods 500a and 500b are components of a single method executed by computer system 101 using cooperation between operating environment 110 and control logic 109.

[0070] Initially referring to operating environment 110, method 500a (e.g., operational execution via operating environment 110) includes action 501, determining to enable hardware-based first execution context tracing. In some embodiments, action 501 includes determining to enable tracing of execution of the first execution context by multiple processing units. In one example, tracer 118 determines that a context configured for execution on processor 102 (e.g., process, operating system, virtual machine, enclave, hypervisor, etc.) will be traced during its execution, and therefore, enables hardware-assisted tracing of that execution context on processor 102. In embodiments, this determination is based on some indication (e.g., in system memory 103, in persistent memory 104, in register 106a, etc.) that a tracing or logging state has been assigned to the execution context (e.g., the state has been assigned based on user input or a policy).

[0071] As shown in the figure, based at least on the executed action 501, method 500a enters one or more of actions 502 or 503, where action 502 initiates a memory snapshot for the first execution context, and action 503 initiates a cache refresh. Figure 5A In this document, actions 502 and 503 are described as possibly being performed in parallel, although it will be understood that the embodiments may perform these actions 502 or 503 alternately in sequence (in any order).

[0072] In some embodiments, action 502 (i.e., initiating a memory snapshot for the first execution context) includes, based on determining that execution tracing of the first execution context should be enabled, initiating logging of one or more memory snapshots of the first execution context's memory space to a second trace. In one example, memory snapshot manager 118b initiates one or more snapshots of system memory 103 to component trace 112b (e.g., as one or more of memory snapshots 306). In some embodiments, memory snapshot manager 118b actually stores at least one memory snapshot to component trace 112b prior to action 504 in method 500a. In other embodiments, memory snapshot manager 118b initiates memory snapshot processing only prior to action 504 in method 500a (e.g., by tracing memory locations accessed by the traced execution context). In an example of a later embodiment, action 502 includes initiating tracing of one or more storage regions (e.g., cache lines, storage pages, etc.) within the memory space of the first execution context, which are accessed by at least one of a plurality of processing units. In an embodiment, the tracking memory region is stored in a snapshot as needed, either during a read or subsequently (e.g., when processor 102 has available cycles, just before committing a write to the tracking memory region, etc.).

[0073] Regardless of whether the memory snapshot manager 118b stores a memory snapshot or merely initiates a memory snapshot process, in various embodiments, prior to method 500a proceeding to action 504, the memory snapshot manager 118b stores at least one complete memory snapshot (e.g., all of system memory 103) or at least one partial memory snapshot (e.g., only a portion of system memory 103). Therefore, in some embodiments of action 502, initiating logging of one or more memory snapshots includes initiating logging of at least one of a complete memory snapshot or a partial memory snapshot. In embodiments, when action 502 initiates logging of a partial memory snapshot, that partial memory snapshot does not include at least one of the following: (i) paged memory pages within the memory space of the first execution context (e.g., memory pages paged to a swap file or swap partition on durable storage 104), or (ii) memory pages within the memory space of the first execution context that are not accessed by the first execution context (i.e., based on tracking which memory regions are accessed by the first execution context).

[0074] In some embodiments, action 503 (i.e., initiating a cache refresh) includes initiating eviction or invalidating one or more cache lines that overlap with the memory space of the first execution context, based on determining that tracing of execution of the first execution context should be enabled. In one example, prior to action 504 in method 500a, cache refresh initiator 118a actively instructs processor 102 to evict all or part of its cache 107 or invalidate it. As indicated by the arrow between actions 503 and 508, this causes processor 102 to perform eviction / use (e.g., using cache refresher 116). In an embodiment, cache refresh initiator 118a instructs processor 102 to evict / invalidate only cache lines that overlap with memory addresses within the memory space of the first execution context.

[0075] Following one or both of actions 502 and 503, method 500a proceeds to action 504 to enable hardware-based tracing of the first execution context. In some embodiments, action 504 is based on determining that execution tracing of the first execution context should be enabled, including enabling the processor's hardware tracing capabilities. In one example, processor tracing enabler 118c communicates with tracing enabler 115 indicating that tracing of the first execution context should be enabled. As indicated by the arrow between actions 504 and 507, this causes the processor to enable hardware-based tracing of the first execution context. In some embodiments, enabling the processor's hardware tracing capabilities causes the processor to log one or more CCP messages generated in response to one or more memory accesses from multiple processing units to the memory space of the first execution context. These embodiments are described in more detail in conjunction with method 500b.

[0076] As previously described, some embodiments of method 500a omit action 503 (i.e., initiating cache refresh). In these embodiments, enabling hardware-based first execution context tracing in action 504 instead triggers a cache refresh. For example, as indicated by the arrow between actions 504 and 508, enabling hardware-based first execution context tracing in action 504 causes processor 102 to perform eviction / invalidation (e.g., using cache refresher 116). Therefore, in some embodiments of method 500a, enabling the processor's hardware tracing functionality also causes the processor to refresh at least one cache line from the cache that overlaps with the memory space of the first execution context.

[0077] As shown in the figure, based at least on the executed action 504, method 500a proceeds to one or more of actions 505 or 503, where action 505 is used to initiate an additional memory snapshot of the first execution context, and action 503 is used to handle modifications to the memory space of the first execution context by the second execution context. Figure 5AIn this embodiment, actions 505 and 506 are shown as possibly being executed in parallel, although it will be understood that these actions 505 or 506 may be executed alternately in sequence (in any order).

[0078] In some embodiments, action 505 (i.e., initiating additional memory snapshots for the first execution context) includes initiating one or more additional memory snapshots during tracing the first execution context after enabling the processor's hardware tracing capabilities. In one example, memory snapshot manager 118b initiates a temporary memory snapshot during the execution of the first context and stores it in memory snapshot 306. In some embodiments, the additional memory snapshots are independent memory snapshots (either full or partial). In other embodiments, the additional memory snapshots are differential memory snapshots that depend on one or more previous memory snapshots (e.g., in the form of a diagram illustrating how previous memory snapshots are converted into subsequent memory snapshots). Notably, these additional memory snapshots can be used to initiate replays at various points in the tracing record.

[0079] In some embodiments, action 506 (i.e., processing the modification of the memory space of the first execution context via the second execution context) includes processing the write via the second execution context after enabling the processor's hardware tracing function. In one example, the first (i.e., the traced) execution context executes in parallel or sequentially with the second execution context (such as an operating system kernel), and the second execution context writes to the memory space of the first execution context. Therefore, the external active processor 118d takes one or more actions to log or otherwise record this write. In an embodiment, action 506 includes an external activity processor 118d that, when executing a second execution context, performs one or more of the following operations: (i) writing the second execution context into the memory space of the first execution context and logging it to one or more of a first or second trace; (ii) logging the identifiers of files mapped from the second execution context to the memory space of the first execution context into the second trace; (iii) writing cache lines in the cache that overlap with memory locations in the memory space of the first execution context, at least based on the second execution context, removing the cache lines from the cache, or marking them as invalid; or (iv) writing a storage region within the memory space of the first execution context, at least based on the second execution context, and logging the storage region as requiring execution in conjunction with the first execution context. In some embodiments, multiple external execution contexts are written into the memory space of the first execution context. In these embodiments, the term "second execution context" refers to these multiple external contexts.

[0080] In some embodiments, the external active processor 118d writes the second execution context into the memory space of the first execution context, logging this to one or more of the first or second trace. In one example, the external active processor 118d logs the write by invoking the memory snapshot manager 118b to create one or more snapshots in component trace 112b (e.g., memory snapshot 306), capturing any memory locations that are written to and modified. In another example, the external active processor 118d writes the log to the data stream of component trace 112b (e.g., data stream 307). In some embodiments, writing via the second execution context includes logging the results of DMA operations (e.g., processed by the operating kernel on behalf of a hardware device) into the second trace.

[0081] In some embodiments, the external activity processor 118d logs the identification information of the file mapped by the second execution context into the second trace, and into the memory space of the first execution context. In one example, the content written to the memory space of the first execution context includes the contents of a file (e.g., an application binary, a data file, etc.) that can be accessed during playback. Therefore, instead of logging the data of the file itself (e.g., one or more writes) into the component trace 112b, the external activity processor 118d logs information identifying the file into the component trace 112b (e.g., referencing one or more filenames, file identifiers, hashes, checksums, etc.).

[0082] In some embodiments, at least based on a cache line written to a cache by a second execution context that overlaps with a memory location in the memory space of a first execution context, an external active processor 118d evicts the cache line from the cache or marks it invalid. In one example, the external active processor 118d instructs a cache refresher 116 to remove or invalidate any cache lines written to cache 107 and move them into system memory 103. By doing so, if the first execution context subsequently accesses a memory address that overlaps with the evicted / invalidated cache line, a CCP logger 117b records a corresponding CCP message associated with the accessed cache miss. In some embodiments, the external active processor 118d also invokes a memory snapshot manager 118b to snapshot (e.g., into component trace 112b) at least a portion of the data corresponding to the evicted data in system memory 103 and / or log writes (e.g., into component trace 112b).

[0083] In some embodiments, at least based on a second execution context writing to a memory region within the memory space of a first execution context, the external active processor 118d marks that memory region as requiring logging associated with the execution of the first execution context. In one example, the external active processor 118d delays logging until the actual time consumed by the write by marking the memory page corresponding to the write (e.g., within a data structure stored in system memory 103 via a PTE or PDE, etc.) as requiring subsequent logging (e.g., if subsequently accessed by the first execution context). To this end, the external active processor 118d reduces the logging overhead imposed on the second execution context and may avoid logging the entire write operation (e.g., if the first execution context will never access the marked memory page again). In an embodiment, if the first execution context subsequently accesses the marked memory page, logging of the marked memory page is triggered by the external active processor 117e in control logic 109. In some embodiments, subsequent accesses to the marked memory page are logged to component trace 112b by initiating a trap in tracer 118. In one embodiment, tracker 118s performs this logging by storing memory snapshots (full or differential) into component trace 112b using memory snapshot manager 118b. Therefore, in this embodiment, in conjunction with subsequent execution of the first execution context, the computer system logs at least a portion of the memory region into the second trace. In other embodiments, subsequent accesses to marked memory pages are logged into component trace 112a based on uncached read logger 117c, as reads are logged as uncached reads. Therefore, in this embodiment, in conjunction with subsequent execution of the first execution context, the computer system logs at least a portion of the memory region as uncached reads into the first trace.

[0084] In some embodiments, the external activity processor 118d always logs detected writes or always delays logging, while in other embodiments, the external activity processor 118d includes logic for deciding whether to log writes or delay logging. In various embodiments, this logic operates based on the amount of data written, the nature of the application programming interface (API) that generated the write, the attributes of the target memory region (e.g., based on the target memory region, such as the stack or heap), etc. In one example, some operating system kernels share one or more memory pages with a user-mode process. In this example, the external activity processor 118d delays logging kernel writes to this shared memory page until the user-mode process uses it. In another example, this shared memory page is read-only for user mode, so it is known that the kernel will make any changes to this shared page. Therefore, in this further example, the external activity processor 118d only logs reads from the user-mode process pointing to memory addresses modified by the kernel, and avoids logging read operations to other memory addresses in the shared memory page. In another example, the external activity processor 118d immediately logs a return value that is written to the memory space of the user-mode process via some kernel call.

[0085] As previously described, in some embodiments, different security protections are used to store component traces 112a and 112b. Therefore, in some embodiments, method 500a includes the action of encrypting a second trace (e.g., component record 112b). Given the foregoing disclosure, it can be understood that the second trace primarily comprises “bulk data,” such as in the form of a memory snapshot, which may contain sensitive memory values. Therefore, in some embodiments, the trace encryptor 118e encrypts the second trace to apply cryptographic security protection to these sensitive memory values. In some embodiments, the trace encryptor 118e encrypts the second record “dynamically” during the creation of the second trace, while in other embodiments, the trace encryptor 118e encrypts the second trace after the first trace has been created.

[0086] Now go to Figure 5B Method 500b is executed at least based on the operation of control logic 109 on processor 102, which includes a plurality of processing units 106 and a cache 107. As shown, method 500b includes an action 507 that enables hardware-based tracing for a first execution context. In some embodiments, action 507 is based on instructions from the operating environment, including enabling hardware tracing capabilities of the processor, including enabling tracing of the execution context. In one example, tracing enabler 115 enables tracing of the first execution context discussed in method 500a based on instructions received from processor tracing enabler 118c in action 504.

[0087] Method 500b further includes an action 508 of performing a cache refresh. In some embodiments, action 508 includes refreshing at least one cache line overlapping the memory space of the execution context to enable trace recording of the execution context. In one example, cache refresher 116 refreshes all caches 107, or refreshes the identifier portion of cache 107 corresponding to the memory space of the first execution context. As discussed in conjunction with actions 503 and 504, in various embodiments, action 508 is performed based on an explicit instruction from cache refresh initiator 118a (i.e., action 503) or based on an instruction from processor trace enabler 118c to enable trace functionality (i.e., action 504). Thus, although action 508 is shown as part of action 507, in some embodiments, action 508 is performed separately from action 507 and precedes action 507 (therefore, action 508 is indicated by dashed lines).

[0088] Following actions 507 and 508, method 500b proceeds to one or more of the following: action 509, detecting a memory access to a specific memory address by the first execution context; action 512, processing a write from the second execution context to the memory space of the first execution context; or action 513, logging the side effects of nondeterministic instructions. The specific action performed by method 500b depends on the nature of the instructions being executed by the tracking execution context.

[0089] In some embodiments, action 509 (i.e., detecting a memory access for a specific memory address via a first execution context) includes detecting a memory access via a first processing unit of a plurality of processing units, wherein the memory access is for a specific memory address during the execution context. In one example, memory access detector 117a detects when machine code instructions of the first execution context access a location in system memory 103. The target specific memory address of the memory access may be an address configured to be cached by cache 107 or may not be configured to be cached by cache 107.

[0090] Based on the nature of the memory access, after action 509, some embodiments of method 500b continue with action 510 to log uncached reads, or action 511 to log CCP messages that can be used to retrieve values ​​from memory snapshots. In embodiments, if the memory access is a read for a memory address not configured to be cached by cache 107, method 500b performs action 510; if the memory access is a read for a memory address configured to be cached by fast cache 107, action 511 is performed.

[0091] In some embodiments, action 510 (i.e., logging uncached reads) includes determining that the memory access is a read for an uncached memory location, and based on the memory access being a read pointing to an uncached memory location, logging at least one value read by the memory access. In one example, memory access detector 117a detects that the memory access is an uncached read and therefore invokes uncached read logger 117c. Uncached read logger 117c, in turn, logs the uncached read to component trace 112a. As previously described, in embodiments, uncached read logger 117c logs one or both target memory addresses of the memory access or the value read by the uncached memory access. When logging a value read by an uncached memory access, some embodiments of uncached read logger 117c encrypt the value within component trace 112a, for example by encrypting packet 304 or a portion thereof, or by storing the value in an encrypted data stream within component trace 112a. Therefore, in some embodiments of action 510, the first trace includes a first trace data stream, and logging the value read by the second memory access includes at least one of the following: (i) logging the value to an encrypted second trace data stream, or encrypting the value in the first trace data stream.

[0092] In some embodiments, action 511 (i.e., logging CCP messages that can be used to retrieve values ​​from memory snapshots) includes logging information that can be used to retrieve memory values ​​corresponding to a specific memory address from one or more memory snapshots stored in a second trace to a first trace based on detected memory access. In one example, CCP logger 117b logs one or more CCP activity indicators resulting from the processing of detected memory access by cache 107 to component trace 112a. In embodiments, action 512 includes logging a specific memory address and at least one of the following: (i) a first CCP message indicating that none of the plurality of processing units has a first cache line in the cache that overlaps with the specific memory address; (ii) a second CCP message indicating that the first processing unit initiates a cache miss for the specific memory address; or (iii) a third CCP message indicating that the first processing unit obtains a second cache line in the cache that overlaps with the specific memory address from a second processing unit of the plurality of processing units. Figures 4A-4C Examples of these CCP messages are provided.

[0093] In some embodiments, CCP logger 117b causes the processor logger to record a first CCP message indicating that none of the plurality of processing units has a first cache line within the cache that overlaps with a specific memory address. For example... Figure 4AAn example 400a is provided, in which processing unit 401a initiates a cache miss at memory location 403a after determining that no other processing unit has a cache copy of memory location 403a. Figure 4C An example 400c is provided, in which processing unit 401c initiates a cache miss at memory location 403a after determining that no other processing unit has a cache copy of memory location 403a. In both examples 400a and 400c, CCP logger 117b logs one or more CCP messages to component trace 112a, providing one or more indications to processing unit 401a / 401c that the processing unit has obtained memory location 403a from system memory 403, such as one or more CCP messages indicating that no other processing unit has a copy of memory location 403a, and / or some other sorting information, such as timestamps, instruction counts, etc.

[0094] In some embodiments, CCP logger 117b causes the processor to log a second CCP message indicating that the first processing unit initiated a cache miss for a specific memory address. For example... Figure 4A An example 400a is provided, in which processing unit 401a initiates a cache miss at memory location 403a after determining that no other processing unit has a cache copy of memory location 403a. Figure 4C An example 400c is provided, in which processing unit 401c initiates a cache miss at memory location 403a after determining that no other processing unit has a cache copy of memory location 403a. In both examples 400a and 400c, CCP logger 117b logs one or more CCP messages to component trace 112a, providing one or more indications to processing unit 401a / 401c that it has obtained memory location 403a from system memory 403, for example, or one or more CCP messages indicating that processing unit 401a / 401c initiated a cache miss at the address of memory location 403a.

[0095] In some embodiments, CCP logger 117b causes the processor to log a third CCP message instructing the first processing unit to obtain a second cache line within the cache that overlaps with a specific memory address from the second processing unit. For example... Figure 4BAn example 400b is provided, in which processing unit 401b obtains a cache copy of memory location 403a from processing unit 401a. In example 400b, CCP logger 117b logs one or more CCP message logs to component trace 112a, providing one or more indications to processing unit 401b that it has obtained memory location 403a from processing unit 401a, and that a copy of the CCP message processing unit 401 is sent to processing unit 401b.

[0096] In some embodiments, CCP logger 117b causes the processor to log one or more CCP messages to a first trace, the messages instructing one or more of a plurality of processing units to fetch them from a cache. For example, Figure 4C An example 400c is provided, in which processing units 401a and / or 401b evict a copy of their memory location 403a to system memory 403. In example 400b, an embodiment of CCP logger 117b logs one or more CCP message messages to component trace 112a, providing one or more instructions for these evictments.

[0097] In some embodiments, the execution context includes a first execution context, and a second execution context initiates a write to the memory space of the first execution context. In these embodiments, action 512 (i.e., processing the write from the second execution context to the memory space of the first execution context) includes an external active processor 117e, which performs at least one of the following to process the write: (i) logging at least one memory address or write value, (ii) evicting cache lines that overlap with the written memory address, (iii) marking the memory page corresponding to the written memory address as requiring logging, or (iv) starting a trap processor that logs the memory page corresponding to the memory address.

[0098] In some embodiments, the external activity processor 117e logs at least one of a memory address or a write value. In one example, the external activity processor 117e logs write operations to a data stream of component trace 112a. In an embodiment, when writing is logged to the data stream of component trace 112a, the first trace includes a first trace data stream, and logging the write value includes at least one of the following: (i) logging the write value to an encrypted second trace data stream, or encrypting the write value within the first trace data stream. In another example, the external activity processor 117e coordinates with the external activity processor 118d to log write operations by invoking a memory snapshot manager 118b to create one or more snapshots in component trace 112b to capture any memory locations modified by the write operation.

[0099] In some embodiments, external activity processor 117e evicts cache lines that overlap with written memory addresses or marks them as invalid. In one example, external activity processor 117e instructs cache refresher 116 to remove or invalidate any written cache lines from cache 107 and place them into system memory 103. By doing so, if the first execution context subsequently accesses a memory address that overlaps with the evicted / invalidated cache line, CCP logger 117b will log the appropriate CCP message associated with the access. In some embodiments, external activity processor 117e also coordinates with external activity processor 118d to invoke memory snapshot manager 118b to take a snapshot (e.g., into component trace 112b) and / or log write operations (e.g., write to component trace 112b) of at least a portion of system memory 103 corresponding to the evicted data.

[0100] In some embodiments, external active processor 117e marks the memory page corresponding to the written memory address as requiring logging. In one example, external active processor 117e coordinates with external active processor 118d to mark the memory page corresponding to the write (e.g., via PTE or PDE, within a data structure stored in system memory 103, etc.) as requiring subsequent logging (e.g., if subsequently accessed by a first execution context).

[0101] In some embodiments, external activity processor 118d initiates a trap processor that logs memory pages corresponding to memory addresses. In one example, external activity processor 117e triggers an interrupt that causes tracer 118 to execute a trap processor that logs memory pages corresponding to memory addresses to component trace 112b.

[0102] In some embodiments, action 513 (i.e., logging the side effects of nondeterministic instructions) includes logging the execution side effects of at least one nondeterministic processor instruction of the execution context to a first trace. In one example, the nondeterministic instruction logger 117d detects the execution of a nondeterministic instruction upon its first execution and logs the side effects of that instruction's execution to component trace 112a. In some embodiments, these side effects are logged by recording the values ​​of one or more registers 106a.

[0103] In embodiments, a first trace (e.g., component trace 112a primarily recorded by tracker 117 during operation of method 500b) and a second trace (e.g., main component trace 112b recorded by tracker 118 during operation of method 500a) may be combined to replay the execution of the first execution context. In some embodiments, the first and second traces are composable, at least in part based on a comparison of a first timestamp and / or count that provides at least a partial ordering of events within the first trace with a second timestamp and / or count that provides at least a partial ordering of events within the second trace.

[0104] In an embodiment, the first trace and the second trace can be combined to replay the execution of the first execution context, based on identifying memory values ​​that are logged in one or more memory snapshots within the second trace and consumed by at least one of a plurality of processing units, using one or more CCP messages logged in the first trace. For example, as in combination Figures 4A-4C The memory accesses discussed, based on the initial memory snapshot of system memory 403 (e.g., within component trace 112b) and at least based on CCP messages logged by CCP logger 117b (within component trace 112a), are reproducible in examples 400a-400c. For example, CCP messages in component trace 112a can be used to identify a memory address (which may have an execution time range when the memory address is requested), and that memory address can then be used to determine a value in component trace 112b (e.g., a value in memory snapshot 306).

[0105] In an embodiment, the first trace and the second trace can be combined to replay the execution of the first execution context, based on identifying memory values ​​consumed by a first of a plurality of processing units using one or more CCP messages logged to the first trace, memory values ​​previously written by a second of the plurality of processing units. For example, as... Figure 4B In the example described, processing unit 401b obtains a cache copy of memory location 403a from processing unit 401a (a cache copy previously written to its memory location 403a), and CCP logger 117b logs CCP messages to component trace 112a to provide an indication that processing unit 401b has obtained memory location 403a from processing unit 401a. Therefore, these CCP messages can be used to reproduce the activity (e.g., based on the ordering of logged CCP messages, which in some embodiments is based on timestamps and / or counts associated with these messages, based on the inherent ordering of CCP messages, etc.).

[0106] In embodiments, tracker 117 operates only when logging is typically enabled for processor 102 and / or for the currently executing context. Therefore, in some embodiments, the processor determines that logging is enabled for at least one processor or execution context, and the processor is at least based on the fact that logging is enabled for at least one of the processors or execution contexts, and that the logging is available to retrieve memory values ​​corresponding to specific memory addresses from one or more memory snapshots stored in the second trace.

[0107] In some embodiments, tracker 117 determines whether a memory region associated with a target memory address for memory access has been marked as logged or unlogged (e.g., by one or more flags in the PTE or PDE, by a mapping in a data structure stored in system memory 103, by a mapping stored in register 106a, etc.). If the memory region associated with the target memory address is marked as unlogged, tracker 117 prohibits logging of memory access. Therefore, in some embodiments, method 500b includes determining whether logging is enabled for a memory region corresponding to a specific memory address, and, at least based on whether logging is enabled for the memory region, determining whether logging can be used to obtain information about the memory value corresponding to the specific memory address from one or more memory snapshots stored in a second track.

[0108] In some embodiments, CCP logger 117b avoids logging CCP messages for memory access, or uncached read logger 117c avoids logging memory access if the target memory address is flagged as an access that is otherwise reconfigurable or predictable (e.g., by accessing a memory-mapped file, by accessing a stored memory snapshot, by simulating a memory-mapped hardware device, by reproducing known behavior, by combining available data in a known manner, etc.). Therefore, in some embodiments, memory access detector 117a determines that the target of the memory access is a read-only memory location, and based at least on memory accesses to read-only memory locations, CCP logger 117b avoids logging CCP messages for memory access, or uncached read logger 117c avoids logging uncached reads for memory access.

[0109] As previously described, processor 102 and its cache 107 form a quasi-closed system. Therefore, by reconstructing the data flowing into these caches 107 from execution trace 112 (plus nondeterministic instructions, cache misses, etc.), the replay of execution trace 112 (i.e., including component traces 112a and 112b) described herein can be achieved. CCP data in component trace 112a provides information about when data flowed into cache 107 via cache misses, while the data flowing in during cache misses can be obtained from memory snapshots in component trace 112b and / or from the replay of execution trace 112 up to the cache miss point.

[0110] Therefore, at least some embodiments described herein separate execution tracing into component tracing. In these embodiments, separating execution tracing into component tracing supports data separation and allows component traces to be stored in different security domains using different data security policies, different protections, etc. This protects sensitive memory values, such as PII, encryption keys, etc., while providing many of the same opportunities as more conventional cache-based recording techniques to reduce recording overhead and / or the amount of data recorded into the execution trace. In these embodiments, separating execution tracing into component tracing also supports reduced memory bandwidth usage compared to more conventional cache-based recording techniques. Specifically, at least some embodiments described herein record at least a portion of CCP messages communicated between multiple processing units as these processing units perform memory accesses to the execution context to enter a first component trace and record one or more snapshots of at least a portion of the system memory into a second component trace. While the first component trace contains sufficient information to support analysis and / or debugging of the execution context (e.g., analyzing the processor cache used by the execution context), the first component trace lacks sufficient information to actually replay the recorded execution of the execution context. Nevertheless, by combining the information contained in the first component trace with the information contained in the second component trace, the recorded execution of the execution context can be replayed.

[0111] Although the subject matter has been described in language specific to structural features and / or methodological behavior, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the aforementioned features or behaviors, or the order of such behaviors. Rather, the features and behaviors are disclosed as exemplary forms for implementing the claims.

[0112] This disclosure may be embodied in other specific forms without departing from its essential characteristics. The described embodiments should be considered in all respects as illustrative rather than restrictive. Therefore, the scope of this disclosure is defined by the appended claims rather than by the foregoing description. All variations within the meaning and equivalence of the claims are to be included within their scope. When elements are introduced in the appended claims, the words “a,” “an,” “the,” and “the” mean one or more elements. The terms “comprising,” “including,” and “having” are intended to encompass and mean that there may be other elements besides those listed.

Claims

1. A computer system (101), comprising: The processor (102) includes multiple processing units (106) and a cache (107). Memory (103); as well as A computer-readable medium (104) having stored thereon computer-executable instructions executable by the processor to cause the computer system to record a replayable execution trace based on recording a cache coherence protocol CCP message to a first trace and based on recording one or more memory snapshots to a second trace, the CCP message being usable to obtain memory values ​​from one or more memory snapshots, characterized in that: The computer-executable instructions include instructions that the processor can execute to cause the computer system to perform at least the following: Determine (118) that tracing of the execution of the first execution context by the plurality of processing units will be enabled; and Based on the determination that tracing of execution in the first execution context will be enabled, at least the following will be performed: Initiate (118b) one or more memory snapshots of the memory space of the first execution context to log recording in the second trace; and Enable (118c) the hardware tracing feature of the processor, which causes the processor to log one or more CCP messages to the first tracing log, the one or more CCP messages being generated in response to one or more memory accesses to the memory space of the first execution context by one of the plurality of processing units; and After enabling the hardware tracing features of the processor, and in conjunction with the execution of the second execution context, at least one of the following is executed (118d): Writes made by the second execution context to the memory space of the first execution context are logged to one or more of the first trace or the second trace; The identity of the mapped file from the second execution context to the memory space of the first execution context is logged in the second trace; At least based on the second execution context, write cache lines in the cache that overlap with memory locations in the memory space of the first execution context, evicting or marking the cache lines as invalid from the cache; or At least based on the second execution context, a memory region within the memory space of the first execution context is written, and the memory region is marked as needing to be logged in conjunction with the execution of the first execution context.

2. The computer system of claim 1, wherein the computer executable instructions further include processor executable instructions for causing the computer system to encrypt the second tracking.

3. The computer system of claim 1, wherein enabling the hardware tracing function of the processor further causes the processor to refresh (116) at least one cache line that overlaps with the memory space of the first execution context from the cache.

4. The computer system of claim 1, wherein the first trace and the second trace are combinable to replay the execution of the first execution context.

5. The computer system of claim 4, wherein the first trace and the second trace are combinable to replay the execution of the first execution context based on identifying memory values ​​in the one or more memory snapshots logged in the second trace and consumed by at least one of the plurality of processing units, using the one or more CCP messages logged in the first trace.

6. The computer system of claim 4, wherein the first trace and the second trace are combinable to replay the execution of the first execution context based on the memory value consumed by the first processing unit of the plurality of processing units and previously consumed by the second processing unit of the plurality of processing units, identified by using one or more CCP messages logged in the first trace.

7. The computer system of claim 1, wherein initiating logging of the one or more memory snapshots comprises: Start logging of partial memory snapshots.

8. The computer system of claim 7, wherein the partial memory snapshot does not include at least one of: (i) a paged memory page in the memory space of the first execution context, or (ii) a memory page in the memory space of the first execution context that is not accessed by the first execution context.

9. The computer system of claim 1, wherein initiating logging of the one or more memory snapshots comprises: Initiate tracing of one or more memory regions within the memory space of the first execution context, wherein the one or more memory regions are accessed by at least one of the plurality of processing units.

10. The computer system of claim 1, wherein the computer system log records the writes made by the second execution context to the storage area within the storage space of the first execution context.

11. The computer system of claim 10, wherein the writing of the log recording of the second execution context comprises: The results of the direct memory access (DMA) operation are logged in the second trace.

12. The computer system of claim 1, wherein the computer system log records the identity of the mapping file from the second execution context to the memory space of the first execution context.

13. The computer system of claim 1, wherein the computer system evicts (118a) the cache line that overlaps with the memory location in the memory space of the first execution context and is written by the second execution context.

14. The computer system according to claim 1, wherein, At least based on the fact that the second execution context has written to the memory area within the memory space of the first execution context, the computer system marks the memory area as needing to be logged in conjunction with the execution of the first execution context.

15. The computer system according to claim 14, wherein, In conjunction with subsequent execution of the first execution context, the computer system logs at least a portion of the memory region into the second trace.

Images