Treating main memory as a collection of tagged cache lines for trace logging
Patent Information
- Application Number
- IN202317049025
- Authority / Receiving Office
- IN · IN
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-02-22
- Filing Date
- 2023-07-20
- Publication Date
- 2026-10-22
- Estimated Expiration
- 2042-01-23
AI Technical Summary
Existing diagnostic tools face challenges in efficiently managing the large size of replayable execution traces due to the vast amount of data generated by modern processors, making it difficult to identify and reproduce undesired software behaviors.
The implementation of cache-based trace logging using tags in a higher memory tier, which tracks whether a cache line's value is already captured by an execution trace, allowing for reduced recording of cache influxes and treating main memory as a collection of tagged cache lines to further minimize trace size.
This approach significantly reduces the size of execution traces and processor utilization, achieving a 3-4x reduction in trace file size when using tags in a higher memory tier and an additional 10-100x reduction when treating main memory as tagged cache lines, compared to prior tracing techniques.
Abstract
Description
BACKGROUND
[001] Tracking down and correcting undesired software behaviors is a core activity insoftware development. Undesired software behaviors can include many things, such asexecution crashes, runtime exceptions, slow execution performance, incorrect data results,data corruption, and the like. Undesired software behaviors are triggered by a vast varietyof factors such as data inputs, user inputs, race conditions (e.g., when accessing sharedresources), etc. Given the variety of triggers, undesired software behaviors are often rareand seemingly random, and extremely difficult to reproduce. As such, it is often very time-consuming and difficult for a developer to identify a given undesired software behavior.Once an undesired software behavior has been identified, it is again often time-consumingand difficult to determine its root cause (or causes).
[002] Developers use a variety of approaches to identify undesired software behaviors, andto then identify one or more locations in an application's code that cause the undesiredsoftware behavior. For example, developers often test different portions of an application'scode against different inputs (e.g., unit testing). As another example, developers oftenreason about execution of an application's code in a debugger (e.g., by settingbreakpoints / watchpoints, by stepping through lines of code, etc. as the code executes). Asanother example, developers often observe code execution behaviors (e.g., timing,coverage) in a profiler. As another example, developers often insert diagnostic code (e.g.,trace statements) into the application's code.
[003] While conventional diagnostic tools (e.g., debuggers, profilers, etc.) have operatedon "live" forward-executing code, an emerging form of diagnostic tools enable "historic"debugging (also referred to as "time travel" or "reverse" debugging), in which the executionof at least a portion of an execution context is recorded into one or more trace files (i.e., anexecution trace). Using some tracing techniques, an execution trace can contain "bit-accurate" historic execution trace data, which enables any recorded portion the tracedexecution context to be virtually "replayed" (e.g., via emulation) down to the granularity ofindividual instructions (e.g., machine code instructions, intermediate language codeinstructions, etc.). Thus, using "bit-accurate" trace data, diagnostic tools enable developersto reason about a recorded prior execution of subject context, as opposed to conventionaldebugging which is limited to a "live" forward execution. For example, using replayableexecution traces, some historic debuggers provide user experiences that enable both forwardand reverse breakpoints / watchpoints, that enable code to be stepped through both forwardsand backwards, etc. Some historic profilers, on the other hand, are able to derive codeexecution behaviors (e.g., timing, coverage) from prior-executed code.
[004] Some techniques for recording execution traces operate based largely on recordinginfluxes to a microprocessor's (processor's) memory cache. However, since modemprocessors commonly execute at the rate of tens- to hundreds- of thousands of MIPS(millions ofinstructions per second), replayable execution traces of a program's thread cancapture vast amounts of information, even if mere fractions of a second of the thread'sexecution are captured. As such, replayable execution traces quickly grow very large in sizein memory and / or on disk.SUMMARY
[005] At least some embodiments described herein reduce the size of replayable executiontraces by performing cache-based trace logging using tags in a higher memory tier. Ingeneral, these embodiments operate to log influxes to a recording first cache level, butleverage tags within a higher memory tier to track whether a value of a given cache lineinflux is already captured by an execution trace (or is otherwise recoverable). In someembodiments, during an influx of a cache line to the first cache level, tracing logic consultsa tag in the higher memory tier to determine if the value that is being influxed can bereconstructed from prior trace logging, such as trace logging performed in connection witha prior influx of the cache line to the first cache level. Ifso, these embodiments refrain fromcapturing a current value of the cache line into the execution trace when influxing the cacheline to the first cache level. Additionally, during evictions from the first cache level, theseembodiments determine whether the cache line being evicted is in a "logged state" withinthe first cache level (i.e., a current value of the cache line can be obtained from a prior-recorded trace, and / or can be constructed by replaying the prior-recorded trace) and sets atag in the higher memory tier as appropriate to indicate whether or not the cache line that isbeing evicted is logged. In embodiments, performing cache-based trace logging to trackwhether a value of a given cache line influx is already captured by an execution trace has atechnical effect ofreducing a number of cache influxes that are recorded into an executiontrace, which in turn has additional technical effects ofreducing a size of the execution traceas compared to prior tracing techniques, and of reducing processor utilization for carryingout the recording of cache influxes as compared to prior tracing techniques.
[006] In some embodiments, the tags within the higher memory tier track whether a valueof a cache line that is being influxed from an upper-level cache is already captured by anexecution trace. In embodiments, one or more of the tags within the higher memory tier arestored in an upper second cache level, and each ofthose tags indicate whether a value of acorresponding cache line in the second cache level is already captured by an execution trace.Additionally, or alternatively, in embodiments, one or more of the tags within the highermemory tier are stored in main memory, and each of those tags indicate whether a value ofa corresponding cache line in the second cache level is already captured by an executiontrace. Experimentally, using tags (whether stored in the second cache level or in mainmemory) to indicate whether a value of a corresponding cache line in the second cache levelis already captured by an execution trace have been shown to reduce trace file size by 3-4x,when compared to prior tracing techniques that lack use of tags in a higher memory tier.
[007] Additionally, or alternatively, in some embodiments one or more of the tags withinthe higher memory tier are stored in main memory, and each ofthose tag(s) indicate whethera value of a block ofmemory in the main memory is already captured by an execution trace(or is otherwise recoverable). Thus, these embodiments operatively treat main memory as acollection oftagged cache lines for trace logging. In these embodiments, a computer systemallocates a plurality of memory blocks within the main memory, along with a correspondingplurality of tags-each of which indicates whether data stored in its associated memoryblock has been captured by an execution trace (or is otherwise recoverable). The computersystem synchronizes these tags with a memory cache (such that the foregoing taggingtechniques remain operable) and manages those tags in light ofmemory operations affectingthe memory blocks-such as direct memory access (DMA) operations, memory pagingoperations, memory initialization operations, and / or file-mapping operations. Inembodiments, treating main memory as a collection oftagged cache lines for trace logginghas a further technical effect of using memory to increase a number of values that can betracked as being previously captured, or not. This, in turn, further reduces the number ofcache influxes that are recorded into an execution trace (further reducing a size of theexecution trace, and further reducing processor utilization for carrying out the recording ofcache influxes). Experimentally, treating main memory as a collection of tagged cache lines for trace logging has been shown to reduce trace file size by 10-100x, when compared tousing tags to indicate whether a value of a corresponding cache line in the second cachelevel is already captured by an execution trace.
[008] In accordance with the foregoing, some embodiments are directed to methods,systems, and computer program products fortreating main memory as a collection of taggedcache lines for trace logging. In these embodiments, a computer system allocates a pluralityof memory blocks within a main memory, including allocating an amount of memory foreach ofthe plurality of memory blocks that is a size of each of a plurality of cache lines ina memory cache. The computer system allocates also allocates a plurality oftags within themain memory. Each tag is associated with one of the plurality of memory blocks andindicates whether data stored in its associated memory block has been captured by anexecution trace. The computer system also synchronizes at least one ofthe plurality oftagswithin the main memory with at least one cache line tag within the memory cache andmanages a traced status of at least one memory block. In some embodiments, managing thetraced status of at least one memory block comprises, based at least on identifying a DMAoperation modifying a particular memory block ofthe plurality ofmemory blocks, setting aparticular tag of the plurality oftags that is associated with the particular memory block toa value indicating that data stored in the particular memory block has not been captured bythe execution trace. In some embodiments, managing the traced status of at least onememory block comprises, based at least on identifying a memory page-in operationaffecting the particular memory block, setting the particular tag based on whether a paged-in value has been captured by the execution trace. In some embodiments, managing thetraced status of at least one memory block comprises, based at least on initializing the datastored in the particular memory block, performing one or more of (i) setting an indicationthat a memory page containing the particular memory block is not logged, or (ii) setting theparticular tag to a value indicating that data stored in the particular memory block has beencaptured by the execution trace. In some embodiments, managing the traced status of at leastone memory block comprises, based at least on associating the particular memory blockwith a file, performing one or more of (i) setting an indication that the memory pagecontaining the particular memory block is not logged, or (ii) setting the particular tag to avalue indicating that data stored in the particular memory block has been captured by theexecution trace.
[009] This summary is provided to introduce a selection of concepts in a simplified formthat are further described below in the Detailed Description. This Summary is not intendedto identify key features or essential features of the claimed subject matter, nor is it intendedto be used as an aid in determining the scope of the claimed subject matter.BRIEF DESCRIPTION OF THE DRAWINGS
[010] In order to describe the manner in which the above-recited and other advantages andfeatures of the invention can be obtained, a more particular description of the inventionbriefly described above will be rendered by reference to specific embodiments thereofwhichare illustrated in the appended drawings. Understanding that these drawings depict onlytypical embodiments of the invention and are not therefore to be considered to be limitingof its scope, the invention will be described and explained with additional specificity and detail through the use ofthe accompanying drawings in which:[Oil] Figure 1A illustrates an example computing environmentthat facilitates cache-basedtrace logging using tags in a higher memory tier;
[012] Figure 1B illustrates additional detail of control logic for cache-based trace loggingusing tags in a higher memory tier;
[013] Figure 1C illustrates additional detail of a memory tagging component that treatsmain memory as a collection oftagged cache lines while trace logging using tags in a highermemory tier;
[014] Figure 2 illustrates an example environment demonstrating multi-level caches;
[015] Figure 3 illustrates an example of a processor cache that includes a plurality ofentries;
[016] Figure 4 illustrates an example of an execution trace;
[017] Figure 5A illustrates a flow chart of an example method for using tags in a highermemory tier to determine whether or not to log a cache line influx;
[018] Figure 5B illustrates a flow chart of an example method for setting tags in a highermemory tier during a cache line eviction; and
[019] Figure 6 illustrates a flow chart of an example method for treating main memory asa collection oftagged cache lines for trace logging.DETAILED DESCRIPTION
[020] At least some embodiments described herein reduce the size ofreplayable executiontraces by performing cache-based trace logging using tags in a higher memory tier. Ingeneral, these embodiments operate to log influxes to a recording first cache level, butleverage tags within a higher memory tier to track whether a value of a given cache lineinflux is already captured by an execution trace (or is otherwise recoverable). In someembodiments, during an influx of a cache line to the first cache level, tracing logic consultsa tag in the higher memory tier to determine if the value that is being influxed can bereconstructed from prior trace logging, such as trace logging performed in connection witha prior influx of the cache line to the first cache level. Ifso, these embodiments refrain fromcapturing a current value ofthe cache line into the execution trace when influxing the cacheline to the first cache level. Additionally, during evictions from the first cache level, theseembodiments determine whether the cache line being evicted is in a "logged state" withinthe first cache level (i.e., a current value of the cache line can be obtained from a prior-recorded trace, and / or can be constructed by replaying the prior-recorded trace) and sets atag in the higher memory tier as appropriate to indicate whether or not the cache line that isbeing evicted is logged.
[021] In some embodiments, the tags within the higher memory tier track whether a valueof a cache line that is being influxed from an upper-level cache is already captured by anexecution trace. In embodiments, one or more of the tags within the higher memory tier arestored in an upper second cache level, and each of those tags indicate whether a value of acorresponding cache line in the second cache level is already captured by an execution trace.Additionally, or alternatively, in embodiments, one or more of the tags within the highermemory tier are stored in main memory, and each ofthose tags indicate whether a value ofa corresponding cache line in the second cache level is already captured by an executiontrace.
[022] Additionally, or alternatively, in some embodiments one or more of the tags withinthe higher memory tier are stored in main memory, and each ofthose tag(s) indicate whethera value of a block ofmemory in the main memory is already captured by an execution trace(or is otherwise recoverable). Thus, these embodiments operatively treat main memory as acollection oftagged cache lines for trace logging.
[023] To the accomplishment of these (and other) embodiments, Figure 1A illustrates anexample computing environment 100a that facilitates cache-based trace logging using tagsin a higher memory tier. In particular, computing environment 100a includes a special-purpose or general-purpose computer system 101, which includes at least one processor 102that is configured to perform a hardware-based execution trace logging, based on recordinginfluxes to at least one level of a cache. As shown, in addition to processor(s) 102, computersystem 101 also includes at least main memory 103 (often referred to as system memory orprimary memory) and durable storage 104, which are communicatively coupled to eachother, and to the processor(s) 102, using at least one communications bus 105.
[024] Embodiments within the scope of the present invention can include physical andother computer-readable media for carrying or having stored thereon computer-executableinstructions and / or data structures. Such computer-readable media can be any availablemedia that can be accessed by a general-purpose or special-purpose computer system.Computer-readable media that store computer-executable instructions and / or data structuresare computer storage media. Computer-readable media that carry computer-executableinstructions and / or data structures are transmission media. Thus, by way of example, andnot limitation, embodiments of the invention can comprise at least two distinctly differentkinds of computer-readable media: computer storage media and transmission media.
[025] Computer storage media are physical storage media (e.g., main memory 103 and / ordurable storage 104) that store computer-executable instructions and / or data structures.Physical storage media include computer hardware, such as RAM, ROM, EEPROM, solidstate drives ("SSDs"), flash memory, phase-change memory ("PCM"), optical disk storage, magnetic disk storage or other magnetic storage devices, or any other hardware storagedevice(s) which can be used to store program code in the form of computer-executableinstructions or data structures, which can be accessed and executed by a general-purpose orspecial-purpose computer system to implement the disclosed functionality ofthe invention.
[026] Transmission media can include a network and / or data links which can be used tocarry program code in the form of computer-executable instructions or data structures, andwhich can be accessed by a general-purpose or special-purpose computer system. A "network" is defined as one or more data links that enable the transport of electronic databetween computer systems and / or modules and / or other electronic devices. Wheninformation is transferred or provided over a network or another communicationsconnection (either hardwired, wireless, or a combination of hardwired or wireless) to acomputer system, the computer system may view the connection as transmission media.Combinations ofthe above should also be included within the scope of computer-readablemedia.
[027] Further, upon reaching various computer system components, program code in theform of computer-executable instructions or data structures can be transferred automaticallyfrom transmission media to computerstorage media (or vice versa). For example, computer-executable instructions or data structures received over a network or data link can bebuffered in RAM within a network interface module (not shown), and then eventuallytransferred to computer system RAM (e.g., main memory 103) and / or to less volatilecomputer storage media (e.g., durable storage 104) at the computer system. Thus, it shouldbe understood that computer storage media can be included in computer system componentsthat also (or even primarily) utilize transmission media.
[028] Computer-executable instructions comprise, for example, instructions and datawhich, when executed at one or more processors, cause a general-purpose computer system,special-purpose computer system, or special-purpose processing device to perform a certainfunction or group of functions. Computer-executable instructions may be, for example,machine code instructions (e.g., binaries), intermediate format instructions such as assemblylanguage, or even source code.
[029] Those skilled in the art will appreciate that the invention may be practiced in networkcomputing environments with many types of computer system configurations, including,personal computers, desktop computers, laptop computers, message processors, hand-helddevices, multi-processor systems, microprocessor-based or programmable consumerelectronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs,tablets, pagers, routers, switches, and the like. The invention may also be practiced indistributed system environments where local and remote computer systems, which arelinked (either by hardwired data links, wireless data links, or by a combination ofhardwiredand wireless data links) through a network, both perform tasks. As such, in a distributedsystem environment, a computer system may include a plurality of constituent computersystems. In a distributed system environment, programmodules may be located in both localand remote memory storage devices.
[030] Those skilled in the art will also appreciate that the invention may be practiced in acloud computing environment. Cloud computing environments may be distributed, althoughthis is not required. When distributed, cloud computing environments may be distributedinternationally within an organization and / or have components possessed across multipleorganizations. In this description and the following claims, "cloud computing" is defined asa model for enabling on-demand network access to a shared pool of configurable computingresources (e.g., networks, servers, storage, applications, and services). The definition of"cloud computing" is not limited to any of the other numerous advantages that can beobtained from such a model when properly deployed.
[031] A cloud computing model can be composed of various characteristics, such as on-demand self-service, broad network access, resource pooling, rapid elasticity, measuredservice, and so forth. A cloud computing model may also come in the form ofvarious servicemodels such as, for example, Software as a Service ("SaaS"), Platform as a Service("PaaS"), and Infrastructure as a Service ("laaS"). The cloud computing model may also bedeployed using different deployment models such as private cloud, community cloud, publiccloud, hybrid cloud, and so forth.
[032] Some embodiments, such as a cloud computing environment, may comprise asystem that includes one or more hosts that are each capable ofrunning one or more virtualmachines. During operation, virtual machines emulate an operational computing system,supporting an operating system and perhaps one or more other applications as well. In someembodiments, each host includes a hypervisor that emulates virtual resources for the virtualmachines using physical resources that are abstracted from view of the virtual machines.The hypervisor also provides proper isolation between the virtual machines. Thus, from theperspective of any given virtual machine, the hypervisor provides the illusion that the virtualmachine is interfacing with a physical resource, even though the virtual machine onlyinterfaces with the appearance (e.g., a virtual resource) of a physical resource. Examples ofphysical resources including processing capacity, memory, disk space, network bandwidth,media drives, and so forth.
[033] As shown in Figure 1A, in embodiments each processor 102 includes at least oneprocessing unit 106, at least one cache 107 (memory cache), and control logic 108 (e.g.,gate logic, executable microcode, etc.). Each processing unit 106 (e.g., processor core) loadsand executes machine code instructions at one or more execution units 106b. Duringexecution of these machine code instructions, the instructions can use internal processorregisters 106a as temporary storage locations, and can read and write to various locations inmain memory 103 via the cache 107. Each processing unit 106 in a given processor 102executes machine code instructions that are defined by a processor instruction setarchitecture (ISA). The particular ISA of each processor 102 can vary based on processormanufacturer and processor model. Common ISAs include the IA-64 and IA-32architectures from INTEL, INC., the AMD64 architecture from ADVANCED MICRODEVICES, INC., and various Advanced RISC Machine ("ARM") architectures from ARMHOLDINGS, PLC, although a great number of other ISAs exist and can be used by thepresent invention. In general, a machine code instruction is the smallest extemally-visible(i.e., external to the processor) unit of code that is executable by a processor.
[034] Registers 106a are hardware storage locations that are defined based on the ISA ofthe processor 102. In general, registers 106a are read from and / or written to by machinecode instructions, or a processing unit 106, as those instructions execute at execution units106b. Registers 106a are commonly used to store values fetched from the cache 107 for useas inputs to executing machine code instructions, to store the results of executing machinecode instructions, to store a program instruction count, to support maintenance of a threadstack, etc. In some embodiments, registers 106a include "flags" that are used to signal somestate change caused by executing machine code instructions (e.g., to indicate if an arithmeticoperation cased a carry, a zero result, etc.). In some embodiments, registers 106a includeone or more control registers (e.g., which are used to control different aspects of processoroperation), and / or other processor model-specific registers (MSRs).
[035] The cache 107 temporarily caches blocks of main memory 103 during execution ofmachine code instructions by one or more of processing units 106. In embodiments, thecache 107 includes one or more "code" portions that cache portions of main memory 103storing application code, as well as one or more "data" portions that cache portions ofmainmemory 103 storing application runtime data. If a processing unit 106 requests data (e.g.,code or application runtime data) not already stored in the cache 107, then the processingunit 106 initiates a "cache miss," causing block(s) of data to be fetched from main memory103 and influxed into the cache 107-while potentially replacing and "evicting" some otherdata already stored in the cache 107 back to main memory 103.
[036] In the embodiments herein, the cache 107 comprises multiple cache levels (sometimes referred to cache tiers or cache layers)-such as a level 1 (LI) cache, a level 2(L2) cache, a level 3 (L3) cache, etc. For example, Figure 2 illustrates an example computingenvironment 200 demonstrating multi-level caches. In Figure 2, the example computingenvironment 200 comprises two processors 201-processor 201a and processor 201b (e.g.,processor(s) 102 of Figure 1A) and a main memory 202 (e.g., main memory 103 of Figure1A). In the example computing environment 200, each processor 201 comprises fourprocessing units (e.g., processing unit(s) 106 of Figure 1A), including processing units A1-A4 forprocessor 201a and processing units B1-B4 for processor 201b.
[037] In example computing environment 200, each processor 201 also includes a three-level cache hierarchy. Computing environment 200 illustrates one example cache layoutonly, and it is not limiting to the cache hierarchies in which the embodiments herein mayoperate. In computing environment 200, each processing unit includes its own dedicated L1cache (e.g., L1 cache "L1 -A1" in processor 201afor unit A1, L1 cache "L1-A2" in processor201a for unit A2, etc.). Relative to the LI caches, each processor 201 also includes twoupper-level L2 caches (e.g., L2 cache "L2-A1" in processor 201a that serves as a backingstore for L1 caches L1-A1 and L1-A2, L2 cache "L1-A2" in processor 201a that serves asa backing store for L1 caches L1-A3 and L1-A4, etc.). Finally, relative to the L2 caches,each processor 201 also includes a single L3 cache (e.g., L3 cache "L3-A" in processor 201athat serves as a backing store for L2 caches L2-A1 and L2-A2, and L3 cache "L3-B" inprocessor 201b that serves as a backing store for L2 caches L2-B1 and L2-B2).
[038] As shown, main memory 202 serves as a backing store for the L3 caches L3-A andL3-B. In this arrangement, and depending on cache implementation, cache misses in an L1cache might be served by its corresponding L2 cache, its corresponding L3 cache, and / ormain memory 202; cache misses in an L2 cache might be served by its corresponding L3cache and / or main memory 202; and cache misses in an L3 cache might be served by mainmemory 202.
[039] In some environments, some cache levels exist separate from a processor; forinstance, in computing environment 200 one or both of the L3 caches could alternativelyexist separate from processors 201, and / or computing environment 200 could include oneor more additional caches (e.g., L4, L5, etc.) that exist separate from processors 201.
[040] As demonstrated by the arrows within each processor 201, when multiple cachelevels exist, each processing unit typically interacts directly with the lowest level (e.g., L1).In many implementations, data flows between the levels (e.g., an L3 cache interacts withthe main memory 202 and serves data to an L2 cache, and the L2 cache in turn serves datato the L1 cache). However, as will be appreciated by one of ordinary skill in the art, theparticular manner in which processing units interact with a cache, and the particular mannerin which data flows between cache levels, may vary (e.g., depending on whether the cacheis inclusive, exclusive, or some hybrid).
[041] Given their arrangement, the caches in computing environment 200 may be viewedas "shared" caches. For example, each L2 and L3 cache serves multiple processing unitswithin a given processor 201 and are thus shared by these processing units. The L1 cacheswithin a given processor 201, collectively, can also be considered shared-even though eachone corresponds to a single processing unit-because the individual L1 caches maycoordinate with each other via a cache coherency protocol (CCP) to ensure consistency (i.e.,so that each cached memory location is viewed consistently across all the LI caches). TheL2 caches within each processor 201 similarly may coordinate via a CCP. Additionally,each individual LI cache may be shared by two or more physical or logical processing units,such as where the processor 201 supports hyper-threading, and are thus "shared" even at anindividual level.
[042] In embodiments, each level of cache(s) 107 comprises a plurality of entries that storecache lines (also commonly referred to as cache blocks). Each cache line / block correspondsto a contiguous block of main memory 103. For example, Figure 3 illustrates an example300 of a memory cache 301 (e.g., an L1 cache, an L2 cache, etc.) that includes a pluralityof entries 303. In example 300, each entry 303 comprises at least an address portion 302athat stores a memory address and a cache line portion 302b that stores a block of datacorresponding to that memory address. During an influx into a given entry 303 of thememory cache 301, the entry's cache line portion 302b is generally filled with a block ofdata obtained from an upper-level cache, or from the main memory 103. Depending on asize of the cache line portion 302b, each entry 303 may potentially store data spanning aplurality of consecutive individually addressable locations within the main memory 103.The cache line portion 302b of each entry 303 can be modified by one or more ofprocessingunits 106, and eventually be evicted back to an upper-level cache, or to the main memory103. As indicated by the ellipses within the memory cache 301, each cache can include alarge number of entries. For example, a contemporary 64-bit INTEL processor may containindividual L1 caches for each processing unit 106 that each comprises 512, or more, entries.In such a cache, each entry is typically used to store a 64-byte (512-bit) value in referenceto a 6-byte (48-bit) to 8-byte (64-bit) memory address. As shown in computing environment200, caches are generally larger in size (i.e., more cache entries) as their cache levelincreases. For example, an L2 cache is generally larger than an L1 cache, an L3 cache isgenerally larger than an L2 cache, and so on.
[043] In some situations, the address portion 302a of each entry 303 stores a physicalmemory address, such as the actual corresponding physical memory address in the mainmemory 103. In other situations, the address portion 302a of each entry 303 stores a virtualmemory address. In embodiments, a virtual memory address is an address within a virtualaddress space that is exposed by an operating system to a process executing at theprocessor(s) 102. This virtual address space provides one or more abstractions to theprocess, such as that the process has its own exclusive memory space and / or that the processhas more memory available to it than actually exists within the main memory 103. Suchabstractions can be used, for example, to facilitate memory isolation between differentprocesses executing at the processor(s) 102, including isolation between user-modeprocesses and kernel-mode processes. In embodiments, virtual to physical memory address mappings are maintained within memory page tables that are stored in the main memory103, and that are managed by an operating system and / or hypervisor (e.g., operatingenvironment 109, described infra). In general, these memory page tables comprise aplurality of page table entries (PTEs) that map ranges (i.e., pages) of virtual memoryaddresses to ranges (i.e., pages) of physical memory addresses. In embodiments, each PTEstores additional attributes, or flags, about its corresponding memory pages, such as memorypage permissions (e.g., read-only, writeable, etc.), page state (e.g., dirty, clean, etc.), and thelike. In embodiments, one or more translation lookaside buffers (TLBs, not shown) withineach processor 102 facilitates virtual addressing, and comprises a dedicated form of cachethat stores recently obtained PTEs mapping virtual and physical memory pages, as obtainedfrom the memory page tables stored in the main memory 103. In some implementations,PTEs are part of a multi-level hierarchy, which includes one or more page directory entries(PDEs) that support discovery of individual PTEs. If a processor 102 lacks a TLB, then it may lack support for virtual memory addressing.
[044] As mentioned, caches coordinate using a CCP. In general, a CCP defines howconsistency is maintained between various caches as various processing units read from and write data to those caches, and how to ensure that the processing units always read consistentdata for a given cache line. CCPs are typically related to, and enable, a memory modeldefined by the processor's instruction set architecture (ISA). Examples of popular ISA'sinclude the x86 and x86_64 families of architectures from INTEL, and the ARMarchitecture from ARM HOLDINGS. Examples of common CCPs include the MSI protocol(i.e., Modified, Shared, and Invalid), the MESI protocol (i.e., Modified, Exclusive, Shared,and Invalid), and the MOESI protocol (i.e., Modified, Owned, Exclusive, Shared, andInvalid). Each of these protocols define a state for individual cache line stored in a sharedcache. A "modified" cache line contains data that has been modified in the shared cache andis therefore inconsistent with the corresponding data in the backing store (e.g., main memory103 or another cache). When a cache line having the "modified" state is evicted from theshared cache, common CCPs require the cache to guarantee that its data is written back thebacking store, or that another cache take over this responsibility. A "shared" cache line isnot permitted to be modified, and may exist in a shared or owned state in another cache. The shared cache can evict this data without writing it to the backing store. An "invalid" cacheline contains no valid data and can be considered empty and usable to store data from cachemiss. An "exclusive" cache line contains data that matches the backing store and is used byonly a single processing unit. It may be changed to the "shared" state at any time (i.e., inresponse to a read request) or may be changed to the "modified" state when writing to it.An "owned" cache location contains data that that is inconsistent with the correspondingdata in the backing store. When a processing unit makes changes to an owned cachelocation, it notifies the other processing units-since the notified processing units may needto invalidate or update based on the CCP implementation.
[045] As shown, each entry in the memory cache 301 may include one or more additionalportions 302c. In some embodiments, one additional portion 302c comprises one or moretracking bits used to track whether a cache line stored in a corresponding entry 303 has beenlogged to a trace or not, as described infra. In some embodiments, an additional portion302c stores a tag that comprises one or more data fields for storing information relevant toits corresponding entry 303. In embodiments, the entries of at least one cache levelcomprises the additional portion 302c for storing tags, and those embodiments use thosetags to improve trace logging, as described infra.
[046] Returning to Figure 1A, in embodiments, control logic 108 comprises microcode(i.e., executable instructions) and / or physical logic gates that control operation of theprocessor 102. In general, control logic 108 functions as an interpreter between the hardwareof the processor 102 and the processor ISA exposed by the processor 102 to executingapplications (e.g., operating environment 109 and application(s) 113) and controls internaloperation ofthe processor 102. In embodiments, the control logic 108 is embodied on on-processor storage, such as ROM, EEPROM, etc. In some embodiments, this on-processorstorage is writable (in which case the control logic 108 is updatable), while in otherembodiments this on-processor storage is read-only (in which case the control logic 108cannot be updated).
[047] The durable storage 104 stores computer-executable instructions and / or datastructures representing executable software components. Correspondingly, duringexecution of these software components at the processor(s) 102, one or more portions ofthese computer-executable instructions and / or data structures are loaded into main memory103.
[048] For example, the durable storage 104 is illustrated as storing computer-executableinstructions and / or data structures corresponding to an operating environment 109 and one or more application(s) 113. Correspondingly, the main memory 103 is shown as storing oneor more operating environment runtime(s) 109' (e.g., machine code instructions and / orruntime data supporting execution ofthe operating environment 109), and as storing one ormore application runtime(s) 113' (e.g., machine code instructions and / or runtime datasupporting execution of one or more of application(s) 113).
[049] The main memory 103 and durable storage 104 can also store other data, such as oneor more replayable execution trace(s) (i.e., execution trace(s) 114' stored in main memory103 and / or execution trace(s) 114 stored in durable storage 104), and one or more ofregioncategorizations 115, memory blocks 116, ortags 117, each described infra. In embodiments,one or more of tags 117 indicates ifthe value of a corresponding memory block in memoryblocks 116 is captured by an execution trace 114. Additionally, or alternatively, one or moreoftags 117 indicates if the value of a corresponding cache line in cache 107 is captured byan execution trace 114.
[050] In embodiments, the operating environment 109 includes a hypervisor 109a and / orone or more operating system(s) 109b. Correspondingly, in embodiments, the operatingenvironment runtime(s) 109' include a hypervisor runtime 109a', and / or one or moreoperating system runtime(s) 109b'. For example, in some embodiments, the operatingenvironment 109 comprises the hypervisor 109a executing directly on the hardware (e.g.,processor(s) 102, main memory 103, and durable storage 104) of computer system 101, andone or more of the operating system(s) 109b executing on top of the hypervisor 109a. Inother embodiments, however, the operating environment 109 comprises an operating system109b executing directly on the hardware (e.g., processor(s) 102, main memory 103, and durable storage 104) of computer system 101.
[051] Additionally, in embodiments, the operating environment 109 includes a debuggingcomponent 110. In various examples, the debugging component 110 is part ofthe hypervisor109a, part of the operating system 109b, or a standalone component that is separate fromboth of the hypervisor 109a and the operating system 109b. As generally indicated by arrowsbetween the debugging component 110 and the region categorizations 115, the memoryblocks 116, and the tags 117, in embodiments the debugging component 110 manages eachof the region categorizations 115 (using the categorization component 111), the memoryblocks 116 (using memory tagging component 112), and the tags 117 (using memorytagging component 112).
[052] In embodiments, the debugging component 110 and the control logic 108 cooperateto record one or more replayable execution trace(s) 114 / 114' of code execution at theprocessor(s) 102. In embodiments, tracing techniques utilized by the operating environment109 and control logic 108 to record replayable execution traces 114 / 114' are based at leaston the processor(s) 102 recording influxes to at least a portion of their cache(s) 107 duringcode execution. In embodiments, each replayable execution trace 114 / 114' comprises a "bit-accurate" record of execution of a corresponding context (e.g., process, operating system,virtual machine, enclave, hypervisor, etc.) as that context executed at the processor(s) 102.As used herein, a replayable execution trace is a "bit accurate" record of that context'sexecution activity. This bit-accurate record enables machine code instructions that werepreviously executed as part ofthe context at the processing unit(s) 106 to be replayed later,such that, during replay, these machine code instructions are re-executed in the same order,and consume the same data that they did during trace recording. While a variety of bit-accurate tracing approaches are possible, as mentioned, the embodiments herein record abit-accurate execution trace based on logging at least some ofthe influxes to cache(s) 107during execution of a traced context (e.g., process, virtual machine, etc.). By logging at leastsome of these influxes during execution of the context, a replayable execution trace114 / 114' of that context captures at least some of the memory reads that were performed bythe machine code instructions that executed as part ofthe context.
[053] The cache-based tracing techniques used by the embodiments herein are built uponan observation that each processor 102 (including its the cache(s) 107) form a semi- orquasi-closed system. For example, once portions of data for an executing context (i.e.,machine code instructions and runtime data) are loaded into a processor's cache(s) 107, aprocessing unit 106 can continue executing that context-without any other external input-as a semi- or quasi-closed system for bursts oftime. In particular, once the cache(s) 107 areloaded with machine code instructions and runtime data, the execution unit 106b can loadand execute those machine code instructions from the cache(s) 107, using runtime datastored in the cache(s) 107 as input to those machine code instructions, and using the registers106a. So long as the data (i.e., machine code instructions and runtime data) that are neededfor the processor 102 to execute that thread exists within the cache(s) 107, the processor 102can continue executing that context without further external input.
[054] When a processing unit 106 needs some influx of data (e.g., because a machine codeinstruction it is executing, will execute, or may execute accesses code or runtime data notalready in the cache(s) 107), the processor 102 may execute a "cache miss," importing datainto the cache(s) 107 from the main memory 103. For example, if a data cache miss occurswhen a processing unit 106 executes a machine code instruction that performs a memoryoperation on a memory address within application runtime 113' storing runtime data, theprocessor 102 imports runtime data from that memory address in the main memory 103 toone of the cache lines of the data portion ofthe cache(s) 107. Similarly, if a code cache missoccurs when a processing unit 106 tries to fetch a machine code instruction from a memoryaddress within application runtime 113' storing application code, the processor 102 importscode data from that memory address in main memory 103 to one of the cache lines of thecode portion of the cache(s) 107. The processing unit 106 then continues execution usingthe newly imported data, until new data is needed.
[055] In embodiments, each processor 102 is enabled to record a bit-accuraterepresentation of execution of a context executing at the processor 102, by recording, into atrace data stream corresponding to the context, sufficient data to be able to reproduce theinfluxes of information into the processor's cache(s) 107 as the processor's processing units106 execute that context's code. For example, some approaches to recording these influxesoperate on a per-processing-unit basis. These approaches involve recording, for eachprocessing unit that is being traced, at least a subset of cache misses within the cache(s) 107,along with a time during execution at which each piece of data was brought into the cache(s)107 (e.g., using a count of instructions executed or some other counter). In someembodiments, these approaches involve also recording, for each processing unit that is beingtraced, any un-cached reads (i.e., reads from hardware components and un-cacheablememory that bypass the cache(s) 107) caused by that processing unit's activity, as well asthe side-effects of having executed any non-deterministic processor instructions (e.g., oneor more values of register(s) 106a after having executed a non-deterministic processorinstruction).
[056] Figure 4 illustrates an example of an execution trace (e.g., one of execution trace(s)114 / 114'). In particular, Figure 4 illustrates an execution trace 400 that includes a pluralityof data streams 401 (i.e., data streams 401a-401n). In embodiments, each data stream 401represents execution of a different context, such as a different thread that executed from thecode of one of application(s) 113. In an example, data stream 401a records execution of afirst thread of an application 113, while data stream 401n records an nth thread of theapplication 113. As shown, data stream 401a comprises a plurality of data packets 402.Since the particular data logged in each data packet 402 can vary, these data packets areshown as having varying sizes. In embodiments, when using time-travel debuggingtechnologies, a data packet 402 records the inputs (e.g., register values, memory values, etc.)to one or more executable instructions that executed as part of this first thread of theapplication 113. In embodiments, memory values are obtained as influxes to cache(s) 107and / or as uncached reads. In embodiments, data stream 401a also includes one or more keyframes 403 (e.g., key frames 403a and 403b) that each records sufficient information, suchas a snapshot ofregister and / or memory values, that enables the prior execution ofthe threadto be replayed, starting at the point ofthe key frame and proceeding forward.
[057] In embodiments, an execution trace also includes the actual code that was executedas part of an application 113. Thus, in Figure 4, each data packet 402 is shown as includinga data inputs portion 404 (non-shaded) and a code portion 405 (shaded). In embodiments,the code portion 405 of each data packet 402, ifpresent, includes the executable instructionsthat executed based on the corresponding data inputs. In other embodiments, however, anexecution trace omits the actual code that was executed, instead relying on having separateaccess to the code of the application 113 (e.g., from durable storage 104). In these otherembodiments, each data packet specifies an address or of fset to the appropriate executableinstruction(s) in an application binary image. Although not shown, it is possible that theexecution trace 400 includes a data stream 401 that stores one or more ofthe outputs of codeexecution. It is noted that used of different data input and code portions of a data packet isfor illustrative purposes only, and that the same data could be stored in a variety of manners,such as by the use of multiple data packets.
[058] If there are multiple data streams 401, in embodiments these data streams can includesequencing events. Each sequencing event records the occurrence of an event that isorderable across different execution contexts, such as threads. In one example, sequencingevents correspond to interactions between the threads, such as accesses to memory that isshared by the threads. Thus, for instance, if a first thread that is traced into a first data stream(e.g., 401a) writes to a synchronization variable, a first sequencing event is recorded intothat data stream (e.g., 401a). Later, if a second thread that is traced into a second data stream(e.g., 401b) reads from that synchronization variable, a second sequencing event is recordedinto that data stream (e.g., 401b). These sequencing events are inherently ordered. Forexample, in some embodiments each sequencing event is associated with a monotonicallyincrementing value, with the monotonically incrementing values defining a total orderamong the sequencing events. In one example, a first sequencing event recorded into a firstdata stream is given a value of one, a second sequencing event recorded into a second datastream is given a value of two, etc.
[059] Some bit-accurate tracing approaches leverage extensions to a processor cache thattrack whether the value of a given cache line can be considered to have been captured intoan execution trace 114 on behalfof at least one processing unit. In various implementations,these cache modifications extend the entries of one or more of processor's caches to includeadditional "logging" bits (e.g., portion 302c), or reserve one or more entries for logging bit use. These logging bits enable a processor's control logic 108 to identify, for each cacheline, one or more processing units that consumed / logged the cache line. Use oflogging bitscan enable the processor's control logic 108 to avoid re-logging cache line influxes for oneexecution context after a processing unit transitions to another execution context (e.g.,another thread, another virtual machine, kernel mode, etc.) if that other context did notmodify the cache line. Additionally, use of logging bits can enable a trace entry for onecontext to reference data already logged on behalf of another context.
[060] Additional, or alternative, bit-accurate tracing approaches use memory markings aslogging cues. More particularly, in embodiments, the debugging component 110 and thecontrol logic 108 cooperate to record replayable execution trace(s) 114 / 114' based on thecategorization component 111 categorizing different memory regions, such as physicalmemory pages in main memory 103, as logged or not logged. These categorizations arerepresented by region categorizations 115 within main memory 103. In embodiments, anexecution context corresponds to at least one of a process executing on top of an operatingsystem 109b, an operating system 109b, a virtual machine / memory partition created by thehypervisor 109a, an enclave, a nested hypervisor, and the like. In embodiments, usingmemory markings as logging cues for processor-based execution tracing is based at least on(i) the categorization component 111 maintaining region categorizations 115 (whichcategorize different memory regions as being logged and not logged), and on (ii) the controllogic 108 using these region categorizations 115 to make logging decisions during tracing.
[061] Additional, or alternative, bit-accurate tracing approaches utilize associative caches,coupled with processor cache way-locking features of some processors to reserve a subsetofthe cache for an entity that being traced, and then logs cache misses relating to that entityinto a reserved subset of the cache. In particular, some bit-accurate tracing approachesutilize way-locking to reserve one or more cache "ways" for an entity that is being traced,such that the locked / reserved ways are used exclusively for storing cache misses relating toexecution of that entity. Thus, by virtue of which way(s) to which a cache entry belongs,embodiments can determine whether or not a corresponding cache line has been logged.
[062] Regardless of which tracking technique(s) are used, in embodiments the controllogic 108 (and / or a co-processor cooperating with control logic 108) logs based on influxesat a particular level in a multi-level cache. For example, in embodiments the control logic108 logs influxes at an L2 cache level, even if one or more higher cache levels are present.In general, logging influxes to a cache with relatively more cache entries results in smallertraces than logging influxes to a cache with relatively fewer cache entries. This is becausea larger cache generally has fewer evictions than a smaller cache, and thus the larger cachehas fewer influxes of the same cache data (and thus, there is less duplicate logging of thesame cache line data). As such, from a trace size perspective, it is often desirable to log at ahigher (upper) cache level (which is generally larger in size than a lower cache level).However, from an implementation and cost perspective, it is often desirable to implementlogging at a lower cache level. For example, it may be less costly (e.g., in terms of processordie size) to implement tracking mechanisms at a lower cache level than it is a higher cachelevel.I. Trace Logging Using Tags in a Higher Memory Tier
[063] The embodiments herein strike a balance between these competing goals byimplementing control logic 108 (and / or a co-processor) that intelligently determineswhether or not to log an influx of a cache line into a recording first cache level based onusing a tag in a higher memory tier (e.g., an upper second cache level, or main memory 103)to determine if a value of the cache line that is being influxed has been previously capturedinto a trace, such as in connection with a prior influx ofthe cache line to the first cache level.
[064] To demonstrate some embodiments of how the control logic 108 (and / or a co-processor) accomplishes the foregoing, Figure 1B illustrates an example computingenvironment 100b showing additional detail of control logic 108, including components thatembodiments of the control logic 108 uses when interacting with the cache 107. The depicted components of control logic 108, together with any sub-components, representvarious functions that the control logic 108 might implement or utilize in accordance withvarious embodiments described herein. It will be appreciated, however, that the depictedcomponents-including their identity, sub-components, and arrangement-are presentedmerely as an aid in describing various embodiments of the control logic 108 describedherein, and that these components are non-limiting to how software and / or hardware mightimplement various embodiments of the control logic 108 described herein, or of theparticular functionality thereof.
[065] As shown, the control logic 108 comprises cache influx logic 118 that operates toinflux a cache line ofdata into a recording cache level (i.e., a cache level into which influxesare being logged), and cache eviction logic 124 that operates to evict a cache line of datafrom the recording cache level. In some embodiments, the control logic 108 supports theenabling and disabling of recording features of a processor, which may be supportedglobally, per-processing unit, per execution context, etc. Thus, the cache influx logic 118 isshown as comprising recording influx logic 119 that operates when infuxing a cache line toa cache that is currently recording, and as potentially comprising non-recording influx logic123 that operates when infuxing a cache line to a cache that is not currently recording.Similarly, the cache eviction logic 124 is shown as comprising recording eviction logic 125that operates when evicting a cache line from a cache that is recording, and as potentiallycomprising non-recording eviction logic 128 that operates when evicting a cache line froma cache that is not recording.
[066] Turning to the cache influx logic 118, the recording influx logic 119 comprises a tagdetermination component 120, logged value logic 121, and non-logged value logic 122. Ingeneral, when a cache line is being influxed from an upper-level cache (e.g., an L3 cache inFigure 2) into a cache level that is being logged (e.g., an L2 cache in Figure 2), the tagdetermination component 120 reads a tag in the higher memory tier (e.g., an additionalportion 302c in an upper-level cache, or one oftags 117 in main memory 202) to determineifthere is an indicium within the tag that a value of the cache line being influxed from theupper-level cache has been previously captured by an execution trace 114. In someembodiments, a value of the cache line is considered to have been previously captured byan execution trace 114 if the present value of the cache line in an upper-level cache isrecorded in the execution trace 114. An additional, or alternative, embodiments a value ofthe cache line is considered to have been previously captured by an execution trace 114 ifthe present value of the cache line in the upper-level cache can be reconstructed based onthe execution trace 114-such as by obtaining a prior value of the cache line from theexecution trace 114, and then replaying one or more executable instructions based on theexecution trace 114 to transform that cache line to arrive at the present value of the cacheline in the upper-level cache. The indicium can vary by implementation and comprisesvalue(s) stored within one or more fields of the tag. For example, in embodiments the tagcomprises at least one of a field for storing an indication of whether or not the cache linehas been logged (e.g., a single bit "logged" flag), a field for storing an address spaceidentifier (ASID) for which the cache line has been logged, a field for storing a virtualmachine identifier (VMID) for which the cache line has been logged, a field for storing anexception level (e.g., ARM processors), a field for storing a ring (e.g., x86 processors), or afield for storing a security state (e.g., ARM processors).
[067] In embodiments, ifthe tag determination component 120 identifies, within the tag,one or more indicia that the cache line was logged (e.g., a logged flag being set, the presenceof an ASID, the presence of a VMID, etc.), then the tag determination component 120further determines ifthe cache line has definitely not been modified after a most recent prioreviction from any recording cache level (e.g., based on the tag having been modified, basedon CCP state stored in the tag or elsewhere, etc.). When the tag determination component120 identifies an indicium that the cache line was logged, and when the tag determinationcomponent 120 further determines that the cache line has definitely not been modified aftera most recent prior eviction from any recording cache level, then the tag determinationcomponent 120 concludes that the value of the cache line has been previously captured byan execution trace 114. In this case, the cache influx logic 118 follows a logic path definedby the logged value logic 121. In general, the logged value logic 121 handles an influx ofthe cache line while refraining from logging a value ofthe cache line into an execution trace114. Even though the logged value logic 121 does not log the value of the cache line, inembodiments the logged value logic 121 does take appropriate action to indicate that thecache line has been logged, such as by appropriately setting tracking bits associated with anentry into which the cache line was stored, by influxing the cache line into a logged way,etc. In embodiments, the logged value logic 121 may store, into an execution trace 114, areference to prior-logged value ofthe cache line.
[068] On the other hand, if the tag determination component 120 cannot identify anindicium that the cache line was logged, or ifthe tag determination component 120 cannotdefinitively determine that the cache line has not been modified after a prior eviction fromthe recording cache level (e.g., the cache line was definitely not logged, or it is indeterminateas to whether the cache line was logged), then the tag determination component 120concludes that the value ofthe cache line has not been previously captured by an executiontrace 114. In this case, the cache influx logic 118 follows a logic path defined by the non-loggedvalue logic 122. In general, the non-logged value logic 122 handles an influx ofthecache line as appropriate for a cache line that has not been previously logged. Inembodiments, the non-logged value logic 122 operates in substantially the same manner asprior bit-accurate tracing approaches that lacked a consideration oftags in a higher memorytier. Thus, the particular action (or inaction) of the non-logged value logic 122 can varydepending on the tracing approach being used, such as logging bits, memory page marking,way locking, etc. In some embodiments, the non-logged value logic 122 captures a value ofthe cache line into an execution trace 114 in connection with performing the influx and takesappropriate action to indicate that the cache line has been logged (e.g., by appropriatelysetting tracking bits associated with an entry into which the cache line was stored, byinfluxing the cache line into a logged way, etc.). In other embodiments, there is separatecontrol logic 108 that will capture the value ofthe cache line based on a subsequent trigger,so the non-logged value logic 122 influxes the cache line without capturing a value of thecache line into an execution trace 114 and / or without indicating that the cache line is logged.At times, the non-logged value logic 122 may refrain from logging the cache line altogether.
[069] Turning to the cache eviction logic 124, the recording eviction logic 125 comprisesa logged determination component 126 and a cache line tagging component 127. In general,when a cache line is being evicted from a cache level that is being logged (e.g., an L2 cachein Figure 2) to an upper-level cache (e.g., an L3 cache in Figure 2), the logged determinationcomponent 126 determines if a current value of that cache line has been captured into anexecution trace 114 at the recording cache level. In some embodiments, the loggeddetermination component 126 determines that a current value of the cache line has beencaptured by the execution trace 114 ifthe current value of the cache line in the recordingcache level is recorded in the execution trace 114 (e.g., in cases where the cache line waslogged during an influx, and the cache line was not modified prior to the eviction). Inadditional, or alternative, embodiments the logged determination component 126determines that a current value of the cache line has been captured by the execution trace114 if the current value of the cache line in the recording cache level can be reconstructedbased on the execution trace 114 (e.g., in cases where the cache line was logged during aninflux, but the cache line was modified by logged instructions prior to the eviction).
[070] In embodiments, if the current value ofthat cache line has been captured, then thecache eviction logic 124 may choose to set a tag in a higher memory tier (e.g., an additionalportion 302c in an upper-level cache, or one of tags 117 in main memory 202) with anindicium that a value ofthe cache line in the upper-level cache to which the cache line wasevicted has been previously captured by an execution trace 114. Otherwise, the cacheeviction logic 124 ensures that the tag in the higher memory tier indicates that the value ofthe cache line in the upper-level cache has not been logged. Thus, depending on thedetermination by the logged determination component 126, the cache line taggingcomponent 127 may set one or more fields within a tag to indicate whether or not the valueof the cache line in the upper-level cache has been logged. In embodiments, this includessetting at least one of a field for storing an indication of whether or not the cache line hasbeen logged (e.g., a single-bit "logged" flag), a field for storing an ASID for which the cacheline has been logged, a field for storing a VMID for which the cache line has been logged,a field for storing an exception level (e.g., ARM processors), a field for storing a ring (e.g.,x86 processors), or a field for storing a security state (e.g., ARM processors). Inembodiments, a lack of an ASID or a VMID in the second and / or third fields indicates thatthe cache line has not been logged. In embodiments, when tag data is stored in mainmemory, the data stored is sufficient to determine whether a cache line was captured or not(e.g., a single bit flag). In embodiments, tag data in main memory (e.g., tags 117) is storedin one or more data structures, such as one or more bitmaps, one or more tree structures(e.g., similar to page table structures), and the like.
[071] In some alternative embodiments, the cache influx logic 118, rather than the cacheeviction logic 124 handles updating oftags to indicate when the values of cache lines in anupper-level cache have been logged. For example, in some embodiments, in connection withlogging a cache line, the non-logged value logic 122 operates much like the cache linetagging component 127 to set one or more fields within a tag in a higher memory tier toindicate that the cache line has been logged. In some embodiments, there is separate loggingand influx logic, such that there are independent logging and influx operations. In theseembodiments, the act of logging and setting tracking information (e.g., logging bits) mayalso set a tag in a higher memory tier (or trigger and eventual update of the tag). Inembodiments, logging actions ensure that there is consistency between a cache line'slogging status and a tag in the higher memory tier, even if those logging actions are notmade in connection with a cache influx. For example, if a cache line's "logged" status iscleared (e.g., due to a write by a non-logged context) while it is in a recording cache level,then a corresponding tag is also cleared (or eventually cleared) in the higher memory tier;later, if the cache line's "logged" status is set, then the corresponding tag is also set (oreventually set) in the higher memory tier.
[072] Operation ofthe control logic 108 is now described in greater detail in connectionwith Figures 5A and 5B which illustrate methods of cache-based trace logging using tags ina higher memory tier. The following discussion now refers to a number of methods andmethod acts. Although the method acts may be discussed in certain orders, or may beillustrated in a flow chart as occurring in a particular order, no particular ordering is requiredunless specifically stated, or required because an act is dependent on another act beingcompleted prior to the act being performed.
[073] Figure 5A illustrates a flow chart of an example method 500a for using tags in ahigher memory tier to determine whether or not to log a cache line influx. Figure 5Billustrates a flow chart of an example method 500b for setting tags in a higher memory tierduring a cache line eviction. In some embodiments, method 500a and method 500b aretreated as entirely separate methods. In other embodiments, method 500a and method 500bare treated as being part of a single combined method. In embodiments, logic forimplementing method 500a and / or 500b are encoded as control logic 108 that configures aprocessor (e.g., processor 102 comprising processing unit 106 and cache 107,or a co-processor) to perform the method.
[074] Methods 500a and 500b refer to a first cache level that is a recording cache level,and a higher memory tier. In various embodiments, the higher memory tier is a second cachelevel arranged as an upper cache level to the first cache level, or main memory (e.g., mainmemory 103). In one example, and referring to Figure 2, the first cache level is an L1 cacheand the higher memory tier is an L2 cache, an L3 cache, or higher (e.g., an L4 cache or mainmemory 202). In another example, the first cache level is an L2 cache and the highermemory tier is an L3 cache, or higher. In another example, the first cache level is an L3cache and the higher memory tier is an L4 cache, or higher. Methods 500a / 500b will nowbe described with respect to the components and data of computing environments 100a and 100b and the example computing environment 200 of Figure 2.
[075] Turning initially to Figure 5A, method 500a begins at an act 501 where there is acache line to be influxed to a first cache level. In general, act 501 comprises influxing acache line into a first cache level. In some embodiments, act 501 comprises influxing acache line into a first cache level from a second cache level arranged as an upper cache levelto the first cache level. For example, based on activity by processing unit A1, a cache lineis influxed from cache L3-A to cache L2-A1; in this example, the first cache level is an L2cache, and the higher memory tier is an L3 cache.
[076] As mentioned, some embodiments enable processor recording features to be enabledor disabled, such as globally, per-processing unit, per execution context, etc. In theseembodiments, method 500a proceeds to an act 502 of determining if a recording feature isenabled. In an example, the cache influx logic 118 determines iftrace recording is enabledor disabled, such as by checking a register value or some other toggleable value. Whenmethod 500a comprises act 502, it will be appreciated that the first cache level referred toin act 501 is a recording cache level only when a recording feature of the processor isenabled.
[077] Ifmethod 500a comprises act 502, and if the recording feature is determined to notbe enabled in act 502, then in embodiments method 500a proceeds to an act 503 of influxingwith non-recording logic (i.e., using non-recording influx logic 123), which in embodimentsignores any tags associated with the influxed cache line (at least for recording purposes).Conversely, ifmethod 500a comprises act 502, and ifthe recording feature is determined tobe enabled in act 502, or if method 500a lacks act 502 (i.e., a recording feature is alwaysactive), then in embodiments method 500a proceeds to an act 504 of influxing withrecording logic (i.e., using recording influx logic 119).
[078] As shown, act 504 comprises an act 505 of reading a tag in a higher memory tier. Ingeneral, act 505 comprises, based at least on the first cache level being a recording cachelevel, reading a tag that is stored in a higher memory tier and that is associated with thecache line. In some embodiments, act 502 comprises reading a tag that is stored in the secondcache level and that is associated with the cache line in the second cache level. For example,the tag determination component 120 reads a tag within cache L3-A, and which is associatedwith the cache line in cache L3-A that is being influxed into cache L2-A1. In otherembodiments, act 502 comprises reading a tag that is stored in main memory and that isassociated with the cache line in the second cache level. In an example, the tag determinationcomponent 120 reads a tag within main memory 202 (e.g., tags 117), and which is associatedwith the cache line in cache L3-A that is being influxed into cache L2-A1.
[079] Act 504 proceeds to an act 506 of determining ifthe cache line is indicated in thehigher memory tier as being logged. In general act 506 comprises, based at least on readingthe tag stored in the higher memory tier, determining whether a first value of the cache linehas been previously captured by a trace. In some embodiments, act 506 comprises, based atleast on reading the tag, determining whether a first value ofthe cache line within the secondcache level has been previously captured by a trace. For example, the tag determinationcomponent 120 determines, from the tag read in act 505, ifthere is an indicium within thetag that a value ofthe cache line, as influxed from cache L3-A into cache L2-A1 in act 501,has been previously captured to an execution trace 114 in connection with a prior influx tocache L2-A1. If so, the tag determination component 120 also determines if the cache linehas definitely not been modified in an upper cache level (e.g., cache L3-A) after a prioreviction from cache L2-A1. Ifthere is an indicium that a value of the cache line has beenpreviously captured, and ifthe cache line has definitely not been modified in an upper cachelevel after a prior eviction from cache L2-A1, then the tag determination component 120concludes that the cache line is indicated in the higher memory tier as being logged (i.e.,"Yes" from act 506). Otherwise, the tag determination component 120 concludes that thecache line is not indicated in the higher memory tier as being logged (i.e., "No" from act 506). In embodiments, the first value of the cache line is determined to have been previouslycaptured by the trace when CCP data indicates that the cache line has not been modifiedwithin an upper second cache level, and the first value of the cache line is determined tohave not been previously captured by the trace when the CCP data indicates that the cacheline could have been modified within the upper second cache level.
[080] Depending on the determination of act 506, act 504 either comprises an act 507 ofinfluxing with the cache line value not certainly known to have been already captured by atrace (i.e., following the "No" path from act 506), or an act 508 ofinfluxing with the cacheline value known to have been already captured by a trace (i.e., following the "Yes" pathfrom act 506).
[081] In some embodiments act 507 comprises, when the first value of the cache line isdetermined to have not been previously captured by the trace, following a non-logged valuelogic path when influxing the cache line into the first cache level. In an example, cacheinflux logic 118 follows a logic path defined by the non-logged value logic 122 wheninfluxing the cache line, and thus cache line is influxed to cache L2-A1 while taking anappropriate logging action (if any). Accordingly, in act 507, the non-logged value logic pathstores the cache line into an entry within the first cache level while initiating logging of thefirst value of the cache line into the trace. In some embodiments, the non-logged value logic122 updates the tag in the higher memory tier to indicate that the cache line has been logged.Thus, in some embodiments, the non-logged value logic path ensures that the tag stored inthe higher memory tier indicates that the cache line has been logged. In an example, thenon-logged value logic 122 sets one or more fields within a tag in cache L3-A, or within atag (e.g., tags 117) in main memory 202, to indicate that a value of the cache line withincache L3-A has not been logged, such as by appropriately setting or clearing a "logged"flag, ensuring that an ASID field is clear or has changed, or ensuring that a VMID field isclear or has changed.
[082] In some embodiments act 508 comprises, when the first value of the cache line isdetermined to have been previously captured by the trace, following a logged value logicpath when influxing the cache line into the first cache level. In an example, cache influxlogic 118 follows a logic path defined by the logged value logic 121 when influxing thecache line, and thus cache line is influxed to cache L2-A1 while refraining from logging avalue of the cache line into an execution trace 114. Accordingly, in act 507, the logged valuelogic path stores the cache line into an entry within the first cache level without initiatinglogging of the first value of the cache line into the trace. Notably, it is possible that thelogged value logic 121 stores some record ofthe influx, such as by storing a reference to aprior-logged value of the cache line (e.g., a prior logged influx by processing unit A2, forinstance). Accordingly, in some embodiments of act 508, the logged value logic path storesthe cache line into an entry within the first cache level while initiating logging, into thetrace, a reference to the first value ofthe cache line previously captured by the trace.
[083] Whether following the logged value logic 121 or the non-logged value logic 122, inembodiments the cache influx logic 118 may take appropriate action to indicate that thecache line has been logged, such as by appropriately setting tracking bits associated with anentry into which the cache line was stored, by influxing the cache line into a logged way,etc. Thus, in embodiments, influxing the cache line into the first cache level also includes,based at least on the first cache level being a recording cache level, at least one of storingthe cache line within a logging way ofthe first cache level, or setting one or more trackingbits associated with an entry in the first cache level that stores the cache line to indicate thatthe cache line has been logged.
[084] Regardless of whether method 500a influxed with non-recording logic in act 503 orinfluxed with recording logic in act 504, in embodiments method 500a comprises act 509,which proceeds to either act 501 (i.e., to process an influx of an additional cache line), or anact 510 of method 500b (i.e., to process an eviction of a cache line).
[085] Turning now to Figure 5B, method 500b begins at act 510 where there is a cacheline to be evicted from the first cache level. In some embodiments, act 510 comprisesdetermining that a cache line in a first cache level is to be evicted. In an example, the controllogic 108 determines that a cache line is evicted from cache L2-A1 into cache L3-A; in thisexample, the first cache level is an L2 cache, and the second cache level is an L3 cache.
[086] As mentioned, some embodiments enable processor recording features to be enabledor disabled, such as globally, per-processing unit, per execution context, etc. In theseembodiments, method 500b proceeds to an act 511 of determining if a recording feature isenabled. In an example, the cache influx logic 118 determines iftrace recording is enabledor disabled, such as by checking a register value or some other toggleable value. Whenmethod 500b comprises act 511, it will be appreciated that the first cache level referred toin act 510 is a recording cache level only when a recording feature of the processor isenabled.
[087] If method 500b comprises act 511, and ifthe recording feature is determined to notbe enabled in act 511, then method 500b proceeds to an act 512 of evicting with non-recording logic (i.e., using non-recording eviction logic 128). Conversely, if method 500bcomprises act 511, and if the recording feature is determined to be enabled in act 511, or ifmethod 500b lacks act 511 (i.e., a recording feature is always active), then method 500bproceeds to an act 513 of evicting with recording logic (i.e., using recording eviction logic125).
[088] As shown, act 513 comprises an act 514 of determining if the cache line is logged inthe first cache level. In some embodiments act 514 comprises, based at least on the firstcache level being a recording cache level, determining whether a second value of the cacheline within the first cache level has been captured by the trace. In an example, the loggeddetermination component 126 determines if a current value of the cache line being evictedfrom cache L2-A1 to cache L3-A has been captured by an execution trace 114, such as bychecking logging status (e.g., logging bits, cache ways, etc.) to determine if the cache linewas logged at influx, and by checking CCP data to determine if the cache line was modifiedwhile in cache L2-A1. If the cache line has been logged and its value has not changed, thenthe logged determination component 126 concludes that the current value ofthe cache linehas been logged (i.e., "Yes" from act 514). Conversely, ifthe cache line has not been loggedor its value has changed, then the logged determination component 126 concludes that thecurrent value ofthe cache line has not been logged (i.e., "No" from act 514).
[089] Depending on the determination of act 514, act 513 either proceeds to an act 515 ofensuring that a tag in the higher memory tier indicates the cache line as not logged (i.e.,following the "No" path from act 514), or an act 516 of ensuring that the tag in the highermemory tier indicates that cache line as logged (i.e., following the "Yes" path from act 514).
[090] In some embodiments act 515 comprises ensuring that the tag stored in the secondcache level indicates that the cache line has not been logged. In an example, the cache linetagging component 127 sets one or more fields within a tag in cache L3-A to indicate thatthe cache line stored in cache L3-A has not been logged, such as by appropriately setting orclearing a "logged" flag, ensuring that an ASID field is clear or has changed, or ensuringthat a VMID field is clear or has changed. In other embodiments act 515 comprises ensuringthat the tag stored in the main memory indicates that the cache line has not been logged. Inan example, the cache line tagging component 127 sets one or more fields within a tag (e.g.,tag 117) in main memory 202 to indicate that the cache line stored in cache L3-A has notbeen logged, such as by appropriately setting or clearing a "logged" flag, ensuring that anASID field is clear or has changed, or ensuring that a VMID field is clear or has changed.
[091] In some embodiments act 516 comprises, based at least on the second value of thecache line having been captured by the trace, ensuring that the tag stored in the highermemory tier indicates that the cache line has been logged. In an example, the cache linetagging component 127 sets one or more fields within a tag in cache L3-A, or in mainmemory 202, to indicate that the value of cache line within cache L3-A has been logged,such as by appropriately setting or clearing a "logged" flag, setting an ASID field to anappropriate address space, or setting a VMID field to an appropriate virtual machineidentifier. Thus, in embodiments, ensuring that the tag in the higher memory tier indicatesthat the cache line has been logged comprises at least one of setting a first field in the tag toindicate that the cache line has been logged, setting a second field in the tag to an address space identifier associated with the cache line, or setting a third field in the tag to virtualmachine identifier associated with the cache line.
[092] Notably, act 516 is shown in broken lines, indicating that, for a given cache line, thecache eviction logic 124 could choose not to set the tag in the higher memory tierto indicatethat the cache line is logged-even when act 515 reaches a "Yes" determination. In thesecases, the cache eviction logic 124 instead sets the tag indicate that the cache line has notbeen logged (i.e., act 515). It will be appreciated that, even though a cache line could bemarked as logged, doing so is not necessary for correct logging (even though this could leadto increased trace size).
[093] Regardless of whether method 500b evicted with non-recording logic in act 512, orevicted with recording logic in act 513, in embodiments method 500b comprises act 517,which proceeds to either act 510 (i.e., to process an eviction of another cache line), or act 501 of method 500a (i.e., to process an influx of a cache line).
[094] As will be appreciated by one of ordinary skill in that art, methods 500a / 500binteract for proper handling of a given cache line. For instance, if a particular cache line isbeing freshly imported from main memory 202, then when influxing the cache line to thefirst cache level method 500a would not find any indication in the higher memory tier thatthe cache line has been logged (act 506), and thus method 500a would influx the cache lineusing the non-logged logic (act 507) and log the cache line. However, if that cache line islater evicted from the first cache level to the second cache level, and its value has beencaptured by an execution trace 114, then method 500b could ensure that a tag in the highermemory tier indicates the cache line as logged (act 516). Then, the next time the cache lineis influxed from the second cache level to the first cache level without having been modifiedin the second cache level, method 500a can influx the cache line using the logged logic (act508) which avoids re-logging the cache line.
[095] Thus, some embodiments comprise following the non-logged value logic path whenin fluxing the cache line into the first cache level, and subsequently ensuring that the tagstored in the higher memory tier indicates that the cache line has been logged when evictingthe cache line from the first cache level (ifits value has been captured by an execution trace114). Additionally, some embodiments comprise ensuring that the tag stored in the highermemory tier indicates that the cache line has been logged when evicting the cache line fromthe first cache level, and subsequently following the logged value logic path when influxingthe cache line into the first cache level.
[096] Accordingly, at least some embodiments described herein perform cache-based tracelogging using tags in a higher memory tier. These embodiments operate to log influxes to afirst cache level that is logging, but leverage tags within a higher memory tier to trackwhether a value of a given cache line influx has been previously captured. In particular,during an influx of a cache line from an upper second cache level to the first cache level,embodiments consult a tag in a higher memory tier (e.g., the second cache level or mainmemory) to determine if a value of the cache line within the second cache level waspreviously captured. Ifso, embodiments refrain from re-logging the cache line in connectionwith the influx to the first cache level. Additionally, during evictions from the first cachelevel to the second cache level, embodiments determine whether a value the cache line beingevicted to the second cache level has been previously captured, and sets a tag in the highermemory tier as appropriate. Thus, the embodiments herein can leverage a potentially largerupper-level cache, or even main memory, to decrease trace size, while limitingimplementation details and complication to a generally smaller lower cache level.II. Treating Main Memory as a Collection of Tagged Cache Lines While TraceLogging Using Tags in a Higher Memory Tier
[097] The foregoing techniques use tags in an upper memory tier (e.g., an upper cachelevel or main memory 103) to track whether a value being influxed from an upper cachelevel (e.g., cache L3-A) to a lower cache level (e.g., L2-A1) has been previously capturedby an execution trace 114. Thus, whether using tags stored in the cache 107 or in the mainmemory 103, these techniques use tags to track whether the value of a cache line in thecache 107 has been previously captured by the execution trace 114.
[098] In embodiments, these tagging techniques are extended to treat main memory as acollection of tagged cache lines, in order to track whether the value of a memory blockwithin main memory 103 has been captured by an execution trace 114, or can be otherwisereconstructed (e.g., because the memory block is initialized memory, because the memoryblock is backed by a file, etc.). These embodiments can be used to avoid logging someinfluxes from main memory 103 to a recording cache level within cache 107, even if thoseinfluxes pass through one or more intermediary cache levels, which further reduces a sizeof execution traces 114. While using tags (whether stored in cache 107 or in main memory103) to indicate whether a value of a corresponding cache line in the cache 107 is already captured by an execution trace have been shown to reduce trace file size by 3-4x (whencompared to prior tracing techniques that lack use oftags in a higher memory tier), treatingmain memory as a collection of tagged cache lines for trace logging has been shown tofurther reduce trace file size by an additional 10-100x.
[099] In general, embodiments that treat main memory as a collection oftagged cache lines for trace logging involve the memory tagging component 112 allocating a plurality ofmemory blocks 116 within main memory 103, along with a corresponding plurality of tags117-one or more of which indicates whether data stored in its associated memory blockhas been captured by an execution trace (or is otherwise recoverable). The memory taggingcomponent 112 then synchronizes appropriate tags within tags 117 with corresponding tagsin cache 107 (such that the foregoing tagging techniques remain operable) and managesthose tags 117 in light of memory operations affecting the memory blocks 116-such asDMA operations, memory paging operations, memory initialization operations, and file-mapping operations.
[100] To further demonstrate these concepts, Figure 1C illustrates an example computingenvironment 100c showing additional detail of a memory tagging component 112 that isconfigured to treat main memory as a collection of tagged cache lines while trace loggingusing tags in a higher memory tier. The depicted components of the memory taggingcomponent 112 represent various functions that the memory tagging component 112 mightimplement or utilize in accordance with various embodiments described herein. It will beappreciated, however, that the depicted components-including their identity andarrangement-are presented merely as an aid in describing various embodiments ofmemorytagging component 112 described herein, and that these components are non-limiting to howsoftware and / or hardware might implement various embodiments of memory taggingcomponent 112 described herein, or ofthe particular functionality thereof.
[101] The memory tagging component 112 is now described in connection with Figure 6,which illustrates a flowchart of a method 600 for treating main memory as a collection oftagged cache lines for trace logging. Thus, the following discussion now refers to a methodand method acts. Although the method acts may be discussed in certain orders, or may beillustrated in a flow chart as occurring in a particular order, no particular ordering is requiredunless specifically stated, or required because an act is dependent on another act beingcompleted prior to the act being performed. In embodiments, instructions for implementingmethod 600 are encoded as executable instructions (e.g., memory tagging component 112and / or control logic 108) stored on one or more hardware storage devices (e.g., durablestorage 104 and / or control logic 108) that are executable by aprocessor (e.g., processor 102)to cause a computer system (e.g., computer system 101) to perform method 600.
[102] The memory tagging component 112 is illustrated as including a memory blockallocation component 130. In embodiments, the memory block allocation component 130allocates a plurality of memory blocks (memory blocks 116) that functionally operate ascache lines stored within main memory 103. Thus, in embodiments, the memory blockallocation component 130 allocates memory blocks that are sufficient in size to store copiesof cache lines in cache 107.
[103] Turning to Figure 6, method 600 comprises an act 601 of allocating, in mainmemory, a collection of cache line sized memory blocks. In some embodiments, act 601comprises allocating a plurality of memory blocks within the main memory, includingallocating an amount ofmemory for each ofthe plurality ofmemory blocks that is a size ofeach ofthe plurality of cache lines.
[104] The memory tagging component 112 is also illustrated as including a tag allocationcomponent 131. In embodiments, the tag allocation component 131 allocates a set of tags117, one or more ofwhich corresponds to one ofthe memory blocks in memory blocks 116.One or more of these tags 117 are used to track whether the data stored in its correspondingmemory block has been captured by an execution trace 114 (or is otherwise recoverable).
[105] Turning to Figure 6, method 600 comprises an act 602 of allocating, in mainmemory, a plurality of tags associated with the collection memory blocks. In someembodiments, act 602 comprises allocating a plurality oftags within the main memory, eachtag being associated with one of the plurality ofmemory blocks, each tag indicating whetherdata stored in its associated memory block has been captured by an execution trace.
[106] In some embodiments, each of the tags 117 is structured similar to the tags discussedabove as being stored in a cache 107, such as a tag including one or more data fields, whichinclude (as examples) at least one of a field for storing an indication of whether or not thememory block has been captured (e.g., a single bit "logged" flag), a field for storing anASID for which the memory block has been logged, a field for storing a VMID for whichthe memory block has been logged, etc. Thus, in some embodiments of act 602, each oftheplurality of tags comprises a plurality of fields, including one or more of a first field toindicate that the memory block has been captured, a second field to store an ASID associatedwith the memory block, or a third field to store a VMID associated with the memory block.In other embodiments, each ofthe tags 117 is a single bit which operates as a flag that is setor cleared to indicate whether or not a corresponding memory block has been captured.Thus, in some embodiments of act 602, each ofthe plurality of tags is a single bit that storeseither a first value (e.g., a zero) indicating that data stored in a corresponding memory blockhas not been captured by the execution trace or a second value (e.g., a one) indicating thatdata stored in the corresponding memory block has been captured by the execution trace.
[107] The memory tagging component 112 is also illustrated as including a cache tagsynchronization component 132. Correspondingly, the control logic 108 (Figure IB) isillustrated as including main memory tag synchronization logic 129. In embodiments, thecache tag synchronization component 132 and the main memory tag synchronization logic129 cooperate to synchronize any tags stored in the cache 107 and the tags 117.
[108] Turning to Figure 6, method 600 comprises an act 603 of synchronizing one or moreof the tags with a memory cache. In some embodiments, act 602 comprises synchronize atleast one of the plurality of tags within the main memory with at least one cache line tagwithin the memory cache.
[109] In embodiments, when evicting a cache line from the cache 107 to the main memory103, one or both of the cache tag synchronization component 132 or the main memory tagsynchronization logic 129 operate to update a tag in tags 117 to reflect a logged status ofthat cache line as was stored in a tag in the cache 107. Thus, in some embodiments of act 603, synchronizing the at least one tag within the main memory with the at least one cacheline tag within the memory cache comprises the processor initiating an eviction of datastored in a particular cache line of the plurality of cache lines to the particular memory block.Then, based at least on initiating the eviction of the data stored in the particular cache lineto the particular memory block, the processor sets the particular tag to indicate whether datastored in the particular memory block has been captured by the execution trace. Here, theprocessor sets the particular tag based on a value of a cache line tag in the memory cache.
[110] Similarly, in embodiments, when influxing a cache line from the main memory 103to the cache 107, one or both of the cache tag synchronization component 132 or the mainmemory tag synchronization logic 129 operate to update a tag in the cache 107 to reflect alogged status of the data being stored into the cache, based on the value of a correspondingtag in tags 117. Thus, in some embodiments of act 603, synchronizing the at least one tagwithin the main memory with the at least one cache line tag within the memory cachecomprises the processor initiating an influx of data stored in the particular memory blockinto a particular cache line of the plurality of cache lines. Then, based at least on initiatingthe influx of the data stored in the particular memory block into the particular cache line,the processor sets a cache line tag in the memory cache to indicate whether data stored inthe particular cache line has been captured by the execution trace. Here, the processor setsthe cache line tag based on a value ofthe particular tag.[11l] In some embodiments, as part of act 603, the main memory tag synchronization logic129 directly accesses (e.g., reads or updates) the tags 117 stored in main memory 103. Inthese embodiments, the main memory tag synchronization logic 129 is aware of a locationof the tags 117 because the cache tag synchronization component 132 has previouslyprovided a base address to the tags 117 in main memory 103 to the memory tagsynchronization logic 129 by writing that base address to one or more of registers 106a.Thus, in some embodiments, the cache tag synchronization component 132 provides themain memory tag synchronization logic 129 the base address of the tags 117, such thatmethod 600 comprises storing, into one or more processor registers a location in the mainmemory of the plurality of tags. In some embodiments, the cache tag synchronizationcomponent 132 may similarly provide the memory tag synchronization logic 129 with alocation of the memory blocks 116, such that method 600 comprises storing, into one ormore processor registers a location in the main memory ofthe plurality of memory blocks.
[112] In other embodiments act 603 operates based on communications between the mainmemory tag synchronization logic 129 and the cache tag synchronization component 132.For example, during an eviction, the main memory tag synchronization logic 129 providesthe cache tag synchronization component 132 with an appropriate value to write to a tag 117in main memory 103, and during an influx the cache tag synchronization component 132provides the main memory tag synchronization logic 129 with an appropriate value to writeto a tag in the cache 107.
[113] Notably, by keeping tags stored in the cache 107 (e.g., an upper second cache level)in sync with tags in main memory 103, act 603 enables full interoperability between method600 and one or more of methods 500a or method 500b.
[114] For example, in embodiments, when cache L2-A1 causes an influx from mainmemory 103 (and, as an intermediary, cache L3-A) that did not already exist in the cache107, the main memory tag synchronization logic 129 synchronizes a tag in cache L3-A withone of tags 117. Thus, when the tag determination component 120 reads that tag from cacheL3-A, it reads a value originating from tags 117. In embodiments, during a later evictionfrom cache L3-A, the main memory tag synchronization logic 129 synchronizes the valueof the tag from cache L3-A to tags 117 in main memory 103. As such method 500a andmethod 500b can operate as described above, while leveraging memory blocks 116 and tags117.
[115] In some embodiments, the cache line is influxed directly to cache L2-A1 (bypassingany intermediary cache), and method 500a is extended to leverage the memory blocks 116and tags 117 in this situation. For instance, referring to method 500a, in some embodimentsact 501 comprises influxing a cache line into the first cache level from main memory. Forexample, based on activity by processing unit Al, a cache line is influxed from mainmemory 202 cache L2-A1 (e.g., where cache L3-A is not present). In these embodiments,the tag determination component 120 may read the value of one of tags 117 directly, in orderto determine if the value being influxed (i.e., one of memory blocks 116) has beenpreviously captured. Thus, in some embodiments, act 506 comprises, based at least onreading the tag, determining whether a first value of the cache line within system memoryhas been previously captured by a trace. In embodiments, during a later eviction from cacheL2-A1 to main memory 103, in act 510 the control logic 108 determines that a cache line isevicted from cache L2-A1 into main memory 202 (e.g., where cache L3-A is not present).Here, the cache line tagging component 127 is extended to update tags 117 as part of acts515 and 516.
[116] Notably, it is possible that one or more of the memory blocks 116 could be modifiedwhile in main memory 103 (e.g., due to a DMA operation on the memory block), or for oneor more of the memory blocks 116 to be paged out (e.g., to durable storage 104).Additionally, it is possible that a value of one or more of the memory blocks 116 is knowableor reconstructable even if it is not stored in an execution trace 114 (e.g., due to that memoryblock being initialized by operating system 109b, or due to that memory block being mappedto a file). Thus, the memory tagging component 112 is illustrated as including a plurality ofhandlers (i.e., DMA handler 133, paging handler 134, memory initialization handler 135,and known memory handler 136) which are configured to managing a traced status ofmemory blocks 116 in each of these situations.
[117] Turning to Figure 6, method 600 comprises an act 604 of managing a traced statusof at least one memory block. In embodiments, act 604 is carried out by one or more of theDMA handler 133, the paging handler 134, the memory initialization handler 135, or theknown memory handler 136.
[118] The DMA handler 133 detects when any of memory blocks 116 are modified by aDMA hardware device. When a memory block is modified by DMA, the DMA handler 133ensures that any corresponding tag in tags 117 is set to indicate that the correspondingmemory block is not captured in an execution trace 114.
[119] Turning to Figure 6, in embodiments act 604 comprises an act 604a of managingtrace status in connection with DMA operation(s) on allocated memory block(s). Inembodiments, act 604a comprises, based at least on identifying a DMA operation modifyinga particular memory block of the plurality of memory blocks, setting a particular tag of theplurality of tags that is associated with the particular memory block to a value (e.g., a zero)indicating that data stored in the particular memory block has not been captured by theexecution trace.
[120] While act 604a is shown in broken lines to indicate that it may not occur with eachinstance of act 604, in some embodiments act 604a does occur, such that managing thetraced status of the plurality of memory blocks in act 604 comprises identifying the DMAoperation modifying the particular memory block. Then, based at least on identifying theDMA operation, act 604 comprises setting the particular tag to a value (e.g., a zero)indicating that data stored in the particular memory block has not been captured by theexecution trace.
[121] When a DMA operation modifies one of memory blocks 116, it is a hardwaredevice-such as a network device-that modifies that memory block. Thus, in someembodiments, it is that hardware device that is responsible for setting any relevant tag(s). Inembodiments, the hardware device is configured to set any relevant tag(s) based on thehardware device being a CCP participant and based on the main memory tagsynchronization logic 129 being made aware of the location of the tags 117 in main memory103. Thus, in some embodiments of act 604a, a hardware device that caused the DMAoperation sets the particular tag to a value (e.g., a zero) indicating that data stored in theparticular memory block has not been captured by the execution trace.
[122] In other embodiments, the operating environment 109, itself, is responsible forsetting any relevant tag(s), such as via the DMA handler 133. Thus, in these otherembodiments of act 604a, an operating system sets the particular tag to a value (e.g., a zero)indicating that data stored in the particular memory block has not been captured by theexecution trace. It is noted that, if the operating environment 109 is responsible for settingany relevant tag(s), then in embodiments the DMA handler 133 is configured to handle asituation in which the operating environment 109 is made aware that a DMA operation isforthcoming, but in which the exact timing of the DMA operation is unknown. Inembodiments, the DMA handler 133 does this by causing the memory tagging component112 to cease tracking the logged status of relevant memory blocks when a DMA operationon those memory blocks is expected, and by resuming tracking those memory blocks onlywhen the DMA operation is known to have occurred. Thus, for a time period from when theDMA operation is expected to when the DMA operation completes, trace recordingproceeds without use of tags 117 for those memory blocks.
[123] The paging handler 134 detects when any of memory blocks 116 are affected by oneor more memory paging operations, such as being paged-out from main memory 103 todurable storage 104, or being paged-in from durable storage 104 to main memory 103.Based on the nature of memory paging operation, and on additional configuration of thepaging handler 134 (discussed infra), the paging handler 134 sets any relevant tags in tags117 accordingly.
[124] Turning to Figure 6, in embodiments act 604 comprises an act 604b of managingtrace status in connection with memory paging involving allocated memory block(s). Inembodiments, act 604b comprises, based at least on identifying a memory page-in operationaffecting the particular memory block, setting the particular tag based on whether a in value has been captured by the execution trace.
[125] While act 604b is shown in broken lines to indicate that it may not occur with eachinstance of act 604, in some embodiments act 604b does occur, such that managing thetraced status ofthe plurality of memory blocks in act 604 comprises identifying the memorypage-in operation affecting the particular memory block.
[126] In some embodiments, the paging handler 134 does not maintain tagging state whenone of memory blocks 116 is paged-out, so the paging handler 134 is configured to detectwhen one of memory blocks 116 is paged-in and set a corresponding tag in tags 117 toindicate that a value ofthe memory block is not captured by an execution trace 114. Thus,in some embodiments of act 604b, based at least on identifying the memory page-inoperation, the paging handler 134 sets the particular tag to a value (e.g., a zero) indicatingthat data stored in the particular memory block has not been captured by the execution tracebased at least on it being unknown whether the paged-in value has been captured by theexecution trace.
[127] In other embodiments, the paging handler 134 is configured to maintain (e.g., inmain memory 103 or in durable storage 104) tagging state when any of memory blocks 116are paged-out. In these embodiments, the paging handler 134 is also configured to detectwhen any of memory blocks 116 are paged-in, and to set corresponding tag(s) in tags 117based on this prior-saved state. Thus, in these embodiments, priorto identifying the memorypage-in operation affecting the particular memory block in act 604b, the paging handler 134identifies a memory page-out operation affecting the particular memory block. Based atleast on identifying the memory page-out operation, the paging handler 134 saves a valueofthe particular tag. Then, after detecting the memory page-out operation in act 604b, thepaging handler 134 identifiesthe memory page-in operation affecting the particular memoryblock. Based at least on identifying the memory page-in operation, the paging handler 134sets the particular tag to the saved value. In some embodiments, the operating system 109bhas full discretion of where tagging stated is stored. In other embodiments, hardwareindicates allowable (or exclusion) areas in main memory 103 for storing the tagging state.In some embodiments, hardware may require specific associations between physicalmemory pages and tagging state indicators.
[128] In yet other embodiments, the paging handler 134 is configured to always record thevalues ofmemory blocks 116 into execution trace 114 when those blocks are paged-out (at least when logging features are enabled / active). In these embodiments, the paging handler134 is also configured to detect when any of memory blocks 116 are paged-in, and to setcorresponding tag(s) in tags 117 to indicate that the value(s) ofthese memory block(s) havebeen captured by execution trace 114. Thus, in these embodiments, prior to identifying thememory page-in operation affecting the particular memory block in act 604b, the paginghandler 134 identifies a memory page-out operation affecting the particular memory block.Based at least on identifying the memory page-out operation, the paging handler 134initiates logging ofthe data stored in the particular memory block. Then, after detecting thememory page-out operation in act 604b, the paging handler 134 identifies the memory page-in operation affecting the particular memory block. Based at least on identifying a memorypage-in operation, the paging handler 134 sets the particular tag to a value (e.g., a zero)indicating that data stored in the particular memory block has not been captured by theexecution trace.
[129] In some embodiments, the operating system 109b initializes memory (e.g., to nullbytes) when allocating that memory to a thread or process. In these cases, the contents ofthis memory are known (or at least reconstructable with knowledge of the memoryinitialization routine), even ifthose contents are not captured by an execution trace 114. Inembodiments, the memory initialization handler 135 identifies block(s) of memory inmemory blocks 116 that have been initialized. In some embodiments, the memoryinitialization handler 135 then sets any corresponding tag(s) in tags 117 in indicate that thevalues of those memory block(s) are captured by an execution trace 114 (e.g., because theyare constructible). In additional, or alternative embodiments, the memory initializationhandler 135 leverages bit-accurate tracing approaches that use memory markings as loggingcues, and sets one or more of region categorizations 115 to indicate that a memory pagecontaining those memory block(s) is not logged.
[130] Turning to Figure 6, in embodiments act 604 comprises an act 604c of managingtrace status in connection with initialization of allocated memory block(s). In embodiments,act 604c comprises, based at least on initializing the data stored in the particular memoryblock, perform one or more of (i) setting an indication that a memory page containing theparticular memory block is not logged, or (ii) setting the particular tag to a value (e.g., aone) indicating that data stored in the particular memory block has been captured by theexecution trace.
[131] While act 604c is shown in broken lines to indicate that it may not occur with eachinstance of act 604, in some embodiments act 604c does occur, such that managing thetraced status of the plurality of memory blocks in act 604 comprises initializing the datastored in the particular memory block. In some embodiments, based at least on initializingthe data stored in the particular memory block, the memory initialization handler 135 setsthe indication that the memory page containing the particular memory block is not logged.In other embodiments, based at least on initializing the data stored in the particular memoryblock, memory initialization handler 135 sets the particular tag to a value (e.g., a one)indicating that data stored in the particularmemory block has been captured by the executiontrace.
[132] In some embodiments, the operating system 109b maps one or more of memoryblocks 116 to a file. Similar to the memory initialization situation just described, in thesecases, the contents of this memory are also known (i.e., by access to the file), even ifthosecontents are not captured by an execution trace 114. In embodiments, the known memoryhandler 136 identifies block(s) of memory in memory blocks 116 that have been mapped toa file. In some embodiments, the known memory handler 136 then sets any correspondingtag(s) in tags 117 in indicate that the values of those memory block(s) are captured by anexecution trace 114 (e.g., because they can be obtained from the file). In additional, oralternative embodiments, the known memory handler 136 leverages bit-accurate tracingapproaches that use memory markings as logging cues, and sets one or more of regioncategorizations 115 to indicate that a memory page containing those memory block(s) is notlogged.
[133] Turning to Figure 6, in embodiments act 604 comprises an act 604d of managingtrace status in connection mapping file(s) to allocated memory block(s). In embodiments act 604d comprises, based at least on associating the particular memory block with a file,perform one or more of (i) setting an indication that the memory page containing theparticular memory block is not logged, or (ii) setting the particular tag to a value (e.g., aone) indicating that data stored in the particular memory block has been captured by theexecution trace.
[134] While act 604d is shown in broken lines to indicate that it may not occur with eachinstance of act 604, in some embodiments act 604d does occur, such that managing thetraced status of the plurality of memory blocks in act 604 comprises associating theparticular memory block with the file; and based at least on associating the particularmemory block with the file. In some embodiments, the known memory handler 136 sets theindication that the memory page containing the particular memory block is not logged. Inother embodiments, the known memory handler 136 sets the particular tag to a value (e.g.,a one) indicating that data stored in the particular memory block has been captured by theexecution trace.
[135] Accordingly, in addition to using tags to indicate whether a value of a correspondingcache line in the second cache level is already captured by an execution trace, at least someembodiments described herein also treat main memory as a collection of tagged cache linesfor trace logging. These embodiments allocate a plurality of memory blocks within mainmemory, along with a corresponding plurality of tags-each of which indicates whetherdata stored in its associated memory block has been captured by an execution trace (or isotherwise recoverable). These embodiments also synchronize these tags with a memorycache and manage those tags in light ofmemory operations affecting the memory blocks-such as DMA operations, memory paging operations, memory initialization operations, andfile-mapping operations. When compared to using tags to indicate whether a value of acorresponding cache line in the second cache level is already captured by an execution trace,treating main memory as a collection of tagged cache lines for trace logging increases anumber of values that can be tracked as being previously captured (or not), which furtherreduces the number of cache influxes that are recorded into an execution trace. This furtherreduces a size of the execution trace, and further reduces processor utilization for carryingout the recording of cache influxes.
[136] The present invention may be embodied in other specific forms without departingfrom its essential characteristics. The described embodiments are to be considered in allrespects only as illustrative and not restrictive. The scope of the invention is, therefore,indicated by the appended claims rather than by the foregoing description. All changeswhich come within the meaning and range of equivalency ofthe claims are to be embracedwithin their scope. When introducing elements in the appended claims, the articles "a," "an,""the," and "said" are intended to mean there are one or more of the elements. The terms"comprising," "including," and "having" are intended to be inclusive and mean that theremay be additional elements other than the listed elements.
Claims
1. A computer system for treating main memory as a collection of tagged cache lines for trace logging, comprising: a processor; a main memory; a memory cache comprising a plurality of cache lines; and one or more storage devices having stored thereon executable instructions that, when executed, cause the computer system to at least: allocate a plurality of memory blocks within the main memory, including allocating an amount of memory for each of the plurality of memory blocks that is a size of each of the plurality of cache lines; allocate a plurality of tags within the main memory, each tag being associated with one of the plurality of memory blocks, each tag indicating whether data stored in its associated memory block has been captured by an execution trace; synchronize at least one of the plurality of tags within the main memory with at least one cache line tag within the memory cache; and manage a traced status of at least one memory block.
2. The computer system of claim 1, wherein, synchronizing the at least one tag within the main memory with the at least one cache line tag within the memory cache comprises: the processor initiating an influx of data stored in a particular memory block into a particular cache line of the plurality of cache lines; and based at least on initiating the influx of the data stored in the particular memory block into the particular cache line, the processor setting a cache line tag in the memory cache to indicate whether data stored in the particular cache line has been captured by the execution trace, wherein the processor sets the cache line tag based on a value of a particular tag of the plurality of tags that is associated with the particular memory block.
3. The computer system of claim 1, wherein, synchronizing the at least one tag within the main memory with the at least one cache line tag within the memory cache comprises: the processor initiating an eviction of data stored in a particular cache line of the plurality of cache lines to a particular memory block; and based at least on initiating the eviction of the data stored in the particular cache line to the particular memory block, the processor setting a particular tag of the plurality of tags that is associated with the particular memory block to indicate whether data stored in the particular memory block has been captured by the execution trace, wherein the processor sets the particular tag based on a value of a cache line tag in the memory cache.
4. The computer system of claim 1, wherein each of the plurality of tags is a single bit that stores either a first value indicating that data stored in a particular memory block has not been captured by the execution trace, or a second value indicating that data stored in the particular memory block has been captured by the execution trace.
5. The computer system of claim 1, wherein managing the traced status of the plurality of memory blocks comprises, based at least on identifying a direct memory access (DMA) operation modifying a particular memory block of the plurality of memory blocks, setting a particular tag of the plurality of tags that is associated with the particular memory block to a value indicating that data stored in the particular memory block has not been captured by the execution trace.
6. The computer system of claim 5, wherein a hardware device that caused the DMA operation sets the particular tag.
7. The computer system of claim 5, wherein an operating system sets the particular tag.
8. The computer system of claim 1, wherein managing the traced status of the plurality of memory blocks comprises, based at least on identifying a memory page-in operation affecting a particular memory block, setting a particular tag of the plurality of tags that is associated with the particular memory block to a value indicating that data stored in the particular memory block has not been captured by the execution trace, based at least on it being unknown whether a paged-in value has been captured by the execution trace.
9. The computer system of claim 1, wherein managing the traced status of the plurality of memory blocks comprises, identifying a memory page-out operation affecting a particular memory block; based at least on identifying the memory page-out operation, saving a value of a particular tag of the plurality of tags that is associated with the particular memory block; after the memory page-out operation, identifying a memory page-in operation affecting the particular memory block; and based at least on identifying the memory page-in operation, setting the particular tag to the saved value.
10. The computer system of claim 1, wherein managing the traced status of the plurality of memory blocks comprises, identifying a memory page-out operation affecting a particular memory block; based at least on identifying the memory page-out operation, initiating logging of the data stored in the particular memory block; after the memory page-out operation, identifying a memory page-in operation affecting the particular memory block; and based at least on identifying the memory page-in operation, setting a particular tag of the plurality of tags that is associated with the particular memory block to a value indicating that data stored in the particular memory block has not been captured by the execution trace.
11. The computer system of claim 1, wherein managing the traced status of the plurality of memory blocks comprises, based at least on initializing the data stored in a particular memory block, setting an indication that a memory page containing the particular memory block is not logged.
12. The computer system of claim 1, wherein managing the traced status of the plurality of memory blocks comprises, based at least on initializing the data stored in a particular memory block, setting a particular tag of the plurality of tags that is associated with the particular memory block to a value indicating that data stored in the particular memory block has been captured by the execution trace.
13. The computer system of claim 1, wherein managing the traced status of the plurality of memory blocks comprises, based at least on associating a particular memory block with a file, setting an indication that a memory page containing the particular memory block is not logged.
14. The computer system of claim 1, wherein managing the traced status of the plurality of memory blocks comprises, based at least on associating a particular memory block with a file, performing one or more of (i) setting an indication that a memory page containing the particular memory block is not logged, or (ii) setting a particular tag of the plurality of tags that is associated with the particular memory block to a value indicating that data stored in the particular memory block has been captured by the execution trace.
15. The computer system of claim 1, the one or more storage devices also having stored thereon executable instructions that, when executed, cause the computer system to store, into one or more processor registers, one or more of, a first location in the main memory of the plurality of memory blocks; or a second location in the main memory of the plurality of tags.
