Track inflows to lower caches by logging entries based on those in upper caches

Abstract

The tracking record is based on the following: based on the knowledge of the upper level cache, the flow of data to the lower level cache is recorded by referring to the previous log data. The computing device includes multiple processing units, multiple N-level caches, and an (N+i)-level cache, which serves as a backup repository for the multiple N-level caches. Based on the activity of the first processing unit, the computing device detects the flow of data to the first N-level cache. The computing device checks the (N+i)-level cache to determine whether the data has been logged on behalf of the second processing unit. Based on the check, the computing device (i) causes the data to be logged for the first processing unit by referring to the log data (i.e., when the data has already been logged), or causes the data to be logged by value for the first processing unit (i.e., when the data has not yet been logged).

Description

Technical Field

[0001] The present application relates generally to the field of computers, and more particularly to methods, devices, and computer program products for tracking records. Background Art

[0002] When writing code during the development of a software application, developers often spend a significant amount of time "debugging" the code to find runtime and other source code errors. To do this, developers can employ several methods to reproduce and locate source code defects (bugs), such as observing the program's behavior based on different inputs, inserting debugging code (e.g., printing variable values, tracing execution branches, etc.), and temporarily deleting sections of code. Tracking down runtime errors to pinpoint code errors can consume a significant portion of application development time.

[0003] In order to assist developers in the code debugging process, many types of debugging applications ("debuggers") have been developed. These tools provide developers with the ability to track the execution of computer code, visualize it, and change it. For example, debuggers can visualize the execution of code instructions, present the values ​​of code variables at different times during code execution, enable developers to change the code execution path, and / or enable developers to set "breakpoints" and / or "watchpoints" on code elements of interest ("breakpoints" and / or "watchpoints" that, when reached during execution, cause the execution of the code to be paused), etc.

[0004] Emerging forms of debugging applications enable "time travel," "reverse," or "historical" debugging. With "time travel" debugging, the execution of a program (e.g., an executable entity such as a thread) is recorded / traced by a tracing application into one or more trace files. These trace files can then be used to later replay the program's execution for forward and backward analysis. For example, a "time travel" debugger can enable developers to set forward breakpoints / watchpoints (as in conventional debuggers) as well as backward breakpoints / watchpoints.

[0005] Several considerations should be taken into account when recording trace files. Most notably, there are inherent tradeoffs between the robustness of the recorded trace data and the overhead incurred by tracing the program. These tradeoffs primarily manifest themselves in trace file size and the performance impact on the execution of the traced program. Furthermore, since tracing may be done with hardware assistance (or entirely in software), there may also be hardware design and other hardware cost considerations. Summary of the Invention

[0006] Embodiments described herein relate to mechanisms for creating bit-accurate "time travel" traces by a processor using hardware assistance. These mechanisms are based on using at least two tiers or layers of processor cache to track execution effects across multiple processing units. Specifically, these mechanisms can modify the processor's hardware and / or microcode so that it assists in (i) detecting influx (i.e., cache misses) into an internal or "lower tier" processor cache based on the activity of the traced processing unit, and (ii) using an external or "upper tier" shared processor cache to determine whether data from the influx has already been logged on behalf of another traced processing unit. If the data has already been logged, the influx can be logged by referencing previous log entries, and these techniques can be extended to "N" levels of cache. Recording trace files in this manner may require only modest processor modifications and can reduce the performance impact of trace recording and the size of trace files by several orders of magnitude compared to previous trace recording methods.

[0007] Embodiments relate to (multiple) computing devices comprising a plurality of processing units, a plurality of N-level caches, and an (N+i)-level cache. The (N+i)-level cache is associated with two or more of the plurality of N-level caches and is configured as a backup repository for the plurality of N-level caches. In these embodiments, the (multiple) computing devices include control logic that configures the (multiple) computing devices to detect an influx into a first N-level cache of the plurality of N-level caches, and wherein the influx includes data stored at a memory location. The control logic further configures the (multiple) computing devices to check the (N+i)-level cache to determine whether data for the memory location has been previously logged on behalf of a second processing unit. The control logic also configures the computing device(s) based on the inspection to perform one of: (i) causing data for the memory location to be logged on behalf of the first processing unit by reference to log data that was previously logged on behalf of the second processing unit (i.e., when data for the memory location was previously logged on behalf of the second processing unit), or (ii) causing data for the memory location to be logged by value on behalf of the first processing unit (i.e., when data for the memory location has not previously been logged on behalf of the second processing unit).

[0008] Embodiments also relate to methods for tracking and recording, wherein the tracking and recording is based on the following: based on the knowledge of one or more upper-level caches, recording the flow of data to the lower-level cache by referencing previous log data. These methods are implemented at a computing device, the computing device including (i) multiple processing units, (ii) multiple N-level caches, and (iii) an (N+i)-level cache, the (N+i)-level cache being associated with two or more of the multiple N-level caches and configured as a backup repository for the multiple N-level caches. The method includes: detecting the flow of data to a first N-level cache in the multiple N-level caches, the flow of data including data stored at a memory location. The method also includes: based on detecting the flow of data to the first N-level cache, checking the (N+i)-level cache to determine whether the data for the memory location has been previously logged on behalf of the second processing unit. The method also includes: based on the inspection, performing one of the following: (i) when the data for the memory location has been previously logged on behalf of the second processing unit, causing the data for the memory location to be logged on behalf of the first processing unit by reference to the log data previously logged on behalf of the second processing unit, or (ii) when the data for the memory location has not been previously logged on behalf of the second processing unit, causing the data for the memory location to be logged by value on behalf of the first processing unit.

[0009] Embodiments may also be embodied as computer-executable instructions (eg, processor microcode) stored on a hardware storage device and executable to perform the aforementioned methods.

[0010] This summary is provided to introduce some concepts in a simplified form that will be further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to illustrate the manner in which the above-recited and other advantages and features of the present invention can be obtained, a more particular description of the invention, briefly described above, will be rendered by reference to specific embodiments of the invention as illustrated in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the invention and are therefore not to be considered limiting of its scope, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0012] Figure 1An example computing environment is illustrated that facilitates recording a "bit-accurate" execution trace across multiple processing units using at least two levels or tiers of processor caches, including detecting influx into an inner or "lower level" processor cache and determining whether the influx can be logged by reference to a previously logged value using an outer or "upper level" shared processor cache;

[0013] Figure 2A An example computing environment including multiple layers of cache is illustrated;

[0014] Figure 2B An example of caching is shown;

[0015] Figure 3 A flow chart illustrating an example method for tracking and recording is provided, wherein the tracking and recording is performed based on: recording inflows to a lower-level cache by referencing previous log data based on knowledge of one or more upper-level caches;

[0016] Figure 4A An example shared cache is illustrated that extends each of its cache lines with one or more additional bookkeeping bits;

[0017] Figure 4B An example of a shared cache is illustrated, the shared cache including one or more cache lines reserved for storing bookkeeping bits applied to regular cache lines; and

[0018] Figure 5 An example of a set-associative mapping between system memory and cache is illustrated. DETAILED DESCRIPTION

[0019] Embodiments described herein relate to mechanisms for creating bit-accurate "time travel" traces by a processor using hardware assistance. These mechanisms are based on using at least two levels or layers of processor cache to track execution effects across multiple processing units. In particular, these mechanisms can modify the processor's hardware and / or microcode so that it assists in (i) detecting inflows (i.e., cache misses) to an internal or "lower level" processor cache based on the activity of the traced processing unit, and (ii) using an external or "upper level" shared processor cache to determine whether the data of the inflow has already been logged by another traced processing unit. If the data has already been logged, the inflow can be logged by referencing previous log entries. These techniques can be extended to "N" levels of cache. Recording trace files in this manner may require only modest processor modifications and can reduce the performance impact of trace recording and trace file size by several orders of magnitude when compared to previous trace recording methods.

[0020] Figure 1 An example computing environment 100 is illustrated that uses at least two levels or layers of processor caches to record "bit-accurate" execution traces across multiple processing units, including detecting influxes into an inner or "lower level" processor cache and using an outer or "upper level" shared processor cache to determine whether the influxes can be logged by reference to previously logged values. As depicted, an embodiment may include or utilize a special purpose or general purpose computer system 101 that includes computer hardware such as, for example, one or more processors 102, a system memory 103, one or more data stores 104, and / or input / output hardware 105.

[0021] Embodiments within the scope of the present invention include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media can be any available media that can be accessed by computer system 101. Computer-readable media that store computer-executable instructions and / or data structures are computer storage devices. Computer-readable media that carry computer-executable instructions and / or data structures are transmission media. Thus, by way of example and not limitation, embodiments of the present invention may include at least two distinct categories of computer-readable media: computer storage devices and transmission media.

[0022] Computer storage is a physical hardware device that stores computer-executable instructions and / or data structures. Computer storage includes various computer hardware such as RAM, ROM, EEPROM, solid-state drives ("SSDs"), flash memory, phase-change memory ("PCM"), optical disk storage, magnetic disk storage or other magnetic storage devices, or any other hardware device(s) that can be used to store program code in the form of computer-executable instructions or data structures, and that can be accessed and executed by computer system 101 to implement the functionality disclosed herein. Thus, for example, computer storage can include the depicted system memory 103, the depicted data repository 104, or other storage devices (such as on-processor storage) as discussed later, which can store computer-executable instructions and / or data structures.

[0023] Transmission medium can comprise the network and / or data link that can be used for carrying program code in the form of computer executable instruction or data structure, and this program code can be accessed by computer system 101." network " is defined as one or more data links that can transmit electronic data between computer system and / or module and / or other electronic equipment.When information is passed through network or another communication connection (hardwire, wireless or hardwire or wireless combination) or is provided to computer system, computer system can be considered as transmission medium by connection. Above-mentioned combination also should be included in the scope of computer readable medium. For example, input / output hardware 105 can comprise the hardware (for example, network interface module (for example, " NIC ")) that connects the network and / or data link that can be used for carrying program code in the form of computer executable instruction or data structure.

[0024] Furthermore, upon reaching various computer system components, program code in the form of computer-executable instructions or data structures may be automatically transferred from the transmission medium to the computer storage device (or vice versa). For example, computer-executable instructions or data structures received over a network or data link may be buffered in RAM within the NIC (e.g., input / output hardware 105) and then ultimately transferred to the system memory 103 and / or a less volatile data storage device (e.g., data repository 104) at the computer system 101. Thus, it should be understood that computer storage devices may be included in computer system components that also (or even primarily) utilize transmission media.

[0025] Computer-executable instructions include, for example, instructions and data that cause computer system 101 to perform a specific function or group of functions when executed at processor(s) 102. Computer-executable instructions may be, for example, binary, intermediate format instructions (e.g., assembly language), or even source code.

[0026] Those skilled in the art will appreciate that the present invention can be put into practice in a network computing environment with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, tablet computers, pagers, routers, switches, etc. The present invention can also be put into practice in a distributed system environment, in which local and remote computer systems with network links (by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links) all perform tasks. Like this, in a distributed system environment, a computer system can include multiple component computer systems. In a distributed system environment, program modules can be located in local and remote memory storage devices.

[0027] Those skilled in the art will also understand that the present invention can be practiced in a cloud computing environment. A cloud computing environment can be distributed, but this is not required. When distributed, a cloud computing environment can be distributed internationally within an organization and / or have components owned across multiple organizations. In this specification and the appended claims, "cloud computing" is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage devices, applications, and services). The definition of "cloud computing" is not limited to any of the other numerous advantages that can be obtained from such a model when properly deployed.

[0028] Cloud computing models can be composed of various characteristics, such as on-demand self-service, broad network access, resource pooling, rapid elasticity, measured services, etc. Cloud computing models can also take the form of various service models, such as, for example, software as a service ("SaaS"), platform as a service ("PaaS"), and infrastructure as a service ("IaaS"). Cloud computing models can also be deployed using different deployment models, such as private cloud, community cloud, public cloud, hybrid cloud, etc.

[0029] Some embodiments (e.g., cloud computing environments) may include a system having one or more host computers, each capable of running one or more virtual machines. During operation, the virtual machines will emulate an operating computing system, thereby supporting an operating system and perhaps one or more other applications. In some embodiments, each host computer includes a hypervisor that emulates the virtual resources of the virtual machine using physical resources that are abstracted from the perspective of the virtual machine. The hypervisor also provides appropriate isolation between the virtual machines. Thus, from the perspective of any given virtual machine, the hypervisor provides the illusion that the virtual machine is interfacing with physical resources, even though the virtual machine is only interfacing with the appearance of physical resources (e.g., virtual resources). Examples of physical resources include processing power, memory, disk space, network bandwidth, media drives, etc.

[0030] As illustrated, data repository 104 may store computer-executable instructions and / or data structures representing application programs, such as, for example, tracer 104a, operating system kernel 104b, and application 104c (e.g., as the subject of tracer 104a and one or more trace files 104d). When these programs are executing (e.g., using processor(s) 102), system memory 103 may store corresponding runtime data, such as runtime data structures, computer-executable instructions, etc. Thus, Figure 1 System memory 103 is illustrated as including runtime application code 103a and application runtime data 103b (eg, each corresponding to application 104c ).

[0031] The tracer 104a can be used to record a bit-accurate trace of the execution of an application, such as application 104c, and store the trace data in trace file(s) 104d. In some embodiments, the tracer 104a is a standalone application, while in other embodiments, the tracer 104a is integrated into another software component, such as an operating system kernel 104b, a hypervisor, a cloud fabric, etc. Although the trace file(s) 104d are depicted as being stored in the data repository 104, the trace file(s) 104d can also be recorded exclusively or temporarily in the system memory 103, or at some other storage device. As explained later, the tracer 104a can interoperate with a specific feature of the processor(s) 102 that implements tracing using the processor's cache coherence protocol (CCP).

[0032] Figure 1 A simplified representation of the internal hardware components comprising processor(s) 102. As shown, each processor 102 includes multiple processing units 102a. Each processing unit can be physical (i.e., a physical processor core) and / or logical (i.e., a logical core provided by a physical core supporting hyperthreading, where more than one application thread executes at the physical core). Thus, for example, even though processor 102 in some embodiments includes only a single physical processing unit (core), it may include two or more logical processing units 102a presented by that single physical processing unit.

[0033] Each processing unit 102a executes processor instructions defined by an application (e.g., tracer 104a, operating system kernel 104b, application 104c, etc.), and the instructions are selected from a predetermined processor instruction set architecture (ISA). The specific ISA of each processor 102 varies based on the processor manufacturer and processor model. Common ISAs include the IA-64 and IA-32 architectures from INTEL, the AMD64 architecture from ADVANCED MICRO DEVICES, and various Advanced RISC Machine ("ARM") architectures from ARMHOLDINGS, PLC, but a large number of other ISAs exist and can be used by the present invention. Generally, an "instruction" is the smallest externally visible (i.e., outside the processor) unit of code that can be executed by a processor.

[0034] Each processing unit 102a obtains processor instructions from one or more processor caches 102b and executes the processor instructions based on data in cache(s) 102b, based on data in registers 102d, and / or in the absence of input data. Generally speaking, each cache 102b is a small amount (i.e., small relative to the typical amount of system memory 103) of random access memory that stores on-processor copies of portions of a backup repository (such as system memory 103 and / or another cache in cache(s) 102b). For example, when executing application code 103a, one or more caches 102b may contain portions of the application runtime data 103b. If processing unit(s) 102a requests data that is not already stored in a particular cache 102b, a "cache miss" occurs, and that data is retrieved from system memory 103 or another cache, potentially "evicting" some other data from that cache 102b.

[0035] Typically, the processor cache(s) 102b are divided into separate tiers, layers, or levels, such as Level 1 (L1), Level 2 (L2), Level 3 (L3), etc. Depending on the processor implementation, the tiers may be part of the processor 102 itself (e.g., L1 and L2) and / or may be separate from the processor 102 (e.g., L3). Figure 1 The cache(s) 102b may include one of these layers (L1), or may include multiple of these layers (eg, L1 and L2, or even L3). Figure 2A An example environment 200 illustrating multiple layers of caching is shown. Figure 2A In the example, there are two processors 201a and 201b (eg, each corresponding to Figure 1 different processors 102) and system memory 202 (e.g., corresponding to Figure 1 In the example environment 200, each processor 201a or 201b includes four physical processing units (ie, units A1-A4 for processor 201a and units B1-B4 for processor 201b).

[0036] The example environment 200 also includes three levels of cache within each processing unit. Environment 200 is merely an example layout and is not limited to a cache hierarchy in which the embodiments herein may operate. In environment 200, at the lowest or innermost level, each processing unit is associated with its own dedicated L1 cache (e.g., L1 cache "L1-A1" in processor 201a for unit A1, L1 cache "L1-A2" in processor 201a for unit A2, etc.). Moving up one level, each processing unit includes two L2 caches (e.g., L2 cache "L2-A1" in processor 201a that serves as a backing store for L1 caches L1-A1 and L1-A2, L2 cache "L1-A2" in processor 201a that serves as a backing store for L1 caches L1-A3 and L1-A4, etc.). Finally, at the highest or outermost level, each processing unit includes a single L3 cache (e.g., L3 cache "L3-A" in processor 201a, which serves as a backing store for L2 caches L2-A1 and L2-A2, and L3 cache "L3-B" in processor 201b, which serves as a backing store for L2 caches L2-B1 and L2-B2). As shown, system memory 202 serves as a backing store for L3 caches L3-A and L3-B.

[0037] like Figure 2A As shown, when multiple cache layers are used, processing unit(s) 102a typically interact directly with the lowest layer (L1). In most cases, data flows between the layers (e.g., on reads, the L3 cache interacts with system memory 103 and passes data to the L2 cache, which in turn passes data to the L1 cache). When processing unit 102a performs a write operation, the caches coordinate to ensure that caches that have affected data shared between processing unit(s) 102a no longer have the data. This coordination is performed using the CCP.

[0038] The caches in environment 200 can therefore be viewed as "shared" caches. For example, each L2 and L3 cache serves multiple processing units within a given processor 201a or 201b and is therefore shared by the processing units. Even though each cache corresponds to a single processing unit, the L1 cache in a given processor 201a or 201b can be viewed as shared overall because the individual L1 caches can coordinate with each other (i.e., via CCP) to ensure consistency (i.e., so that the memory locations of each cache are viewed consistently across all L1 caches). Similarly, the L2 caches within each processor 201a or 201b can be coordinated via CCP. Additionally, if the processor 201a or 201b supports hyperthreading, each individual L1 cache can be viewed as shared by two or more logical processing units and can therefore be "shared" even at the individual level.

[0039] Typically, each cache includes multiple “cache lines.” Each cache line stores a memory block from its backing store (e.g., system memory 202 or a higher-level cache). For example, Figure 2B An example of at least a portion of a cache 203 is illustrated, the cache 203 including a plurality of cache lines 206, each cache line 206 including at least an address portion 204 and a value portion 205. The address portion 204 of each cache line 206 is configured to store an address in the system memory 202 to which the cache line corresponds, and the value portion 205 initially stores a value received from the system memory 202. The value portion 205 may be modified by the processing unit and eventually evicted back to the backup storage library. As indicated by the ellipsis, the cache 203 may include a large number of cache lines. For example, a contemporary 64-bit INTEL processor may include a separate L1 cache with 512 or more cache lines. In such a cache, each cache line may typically be used to store a 64-byte (512-bit) value for a 6-byte (48-bit) to 8-byte (64-bit) memory address. As shown in FIG. Figure 2A As intuitively shown, cache size generally increases with each level (i.e., L2 cache is generally larger than L1 cache, L3 cache is generally larger than L2 cache, etc.).

[0040] The addresses stored in the address portion 204 of each cache line 206 may be physical addresses, such as actual memory addresses in the system memory 202. Alternatively, the addresses stored in the address portion 204 may be virtual addresses, which are addresses that are mapped to physical addresses to provide an abstraction (e.g., using page tables managed by an operating system). Such abstractions may be used, for example, to facilitate memory isolation between different processes executing at the processor(s) 102, including isolation between user-mode processes and kernel-mode processes associated with the operating system kernel 104b. When virtual addresses are used, the processor 102 may include a translation lookaside buffer (TLB) 102f (typically part of a memory management unit (MMU)) that maintains recently used memory address mappings between physical addresses and virtual addresses.

[0041] (Multiple) caches 102b may include a code cache portion and a data cache portion. When the application code 103a is executed, the (multiple) code portion of the (multiple) caches 102b may store at least a portion of the processor instructions stored in the application code 103a and the (multiple) data portion of the (multiple) caches 102b may store at least a portion of the data structure of the application runtime data 103b. In addition, caches may be inclusive, exclusive, or may include both inclusive and exclusive behaviors. For example, in an inclusive cache, the L3 layer stores a superset of the data in the L2 layer below the L3 layer, and the L2 layer stores a superset of the L1 layer below them. In an exclusive cache, the layers may not intersect, for example, if the L3 cache has data that the L1 cache needs, they may exchange information, such as data, addresses, etc.

[0042] Return to Figure 1 Each processor 102 also includes microcode 102c, which includes control logic (i.e., executable instructions) that controls the operation of the processor 102 and typically acts as an interpreter between the processor's hardware and the processor ISA exposed by the processor 102 to executing applications. The microcode 102c is typically embodied in on-processor storage such as ROM, EEPROM, or the like.

[0043] Registers 102d are hardware-based storage locations that are defined based on the ISA of the processor(s) 102 and are read and / or written to by processor instructions. For example, registers 102d are typically used to store values ​​obtained from cache(s) 102b for use by instructions, store the results of executing instructions, and / or store status or state, such as some side effect of executing instructions (e.g., the sign of a value changing, a value reaching zero, the occurrence of a carry, etc.), processor cycle counts, etc. Therefore, some registers 102d may include "flags" for signaling certain state changes caused by the execution of processor instructions. In some embodiments, the processor 102 may also include control registers for controlling different aspects of the processor's operation. Although Figure 1 Registers 102d are depicted as a single block, but it will be understood that each processing unit 102a typically includes one or more corresponding register sets dedicated to that processing unit.

[0044] In some embodiments, the processor(s) 102 may include one or more buffers 102e. As will be discussed below, the buffer(s) 102e may serve as a temporary storage location for trace data. Thus, for example, the processor(s) 102 may store portions of trace data in the buffer(s) 102e and, at appropriate times (e.g., when there is available memory bus bandwidth and / or idle processor cycles), flush the data to the trace file(s) 104d.

[0045] As described above, the processor operates on cache(s) 102b according to one or more CCPs. Generally, a CCP defines how data consistency between the various caches 102b is maintained when the various processing units 102a read and write data from the various caches 102b, and how to ensure that the various processing units 102a always read valid data from a given location in the cache(s) 102b. The CCP is associated with the enabled memory model defined by the ISA of the processor 102.

[0046] Examples of general-purpose CCPs include the MSI protocol (i.e., modified, shared, and invalid), the MESI protocol (i.e., modified, exclusive, shared, and invalid), and the MOESI protocol (i.e., modified, owned, exclusive, shared, and invalid). Each of these protocols defines a state for an individual location (e.g., a line) in cache(s) 102b. A "modified" cache location contains data that has been modified in cache(s) 102b and, therefore, may be inconsistent with corresponding data in a backing store (e.g., system memory 103 or another cache). When a location with a "modified" state is evicted from cache(s) 102b, the general-purpose CCP requires the cache to ensure that its data is written back to the backing store, or that another cache takes over this responsibility. A "shared" cache location contains data that has not been modified from the data in the backing store, exists in a read-only state, and is shared by processing unit(s) 102a. Cache(s) 102b can evict this data without having to write it to the backing store. An "invalid" cache location contains no valid data and can be considered empty and can be used to store data for cache misses. An "exclusive" cache location contains data that matches a backing store and is used only by a single processing unit 102a. This data can be changed to a "shared" state at any time (i.e., in response to a read request), or can be changed to a "modified" state when a write is made to it. An "owned" cache location is shared by two or more processing units 102a, but one of the processing units has exclusive rights to make changes to it. When the process makes a change, the notified processing unit notifies the other processing units because it may need to be invalidated or updated based on the CCP implementation.

[0047] As previously described, embodiments utilize the cache(s) 102b of the processor 102 to efficiently record bit-accurate execution traces of the application 104c and / or the operating system kernel 104b. These embodiments are based on the inventors' observation that the processor 102 (including the cache(s) 102b) forms a semi-closed or quasi-closed system. For example, once portions of data for a process (i.e., code data and runtime application data) are loaded into the cache(s) 102b, the processor 102 can operate autonomously in a semi-closed or quasi-closed system for bursts of time without any input. In particular, once the cache(s) 102b are loaded with data, one or more processing units 102a use the runtime data stored in the data portion(s) of the cache(s) 102b and use registers 102d to execute instructions from the code portion(s) of the cache(s) 102b.

[0048] When processing unit 102a requires some information influx (e.g., because an instruction it is currently executing, about to execute, or may execute accesses code or runtime data that is not yet in cache(s) 102b), a "cache miss" occurs and the information is brought from system memory 103 into cache(s) 102b. For example, if a data cache miss occurs when an executed instruction performs a memory operation at a memory address within application runtime data 103b, the data from that memory address is brought into one of the cache lines of the data section of cache(s) 102b. Similarly, if a code cache miss occurs when an instruction performs a memory operation at a memory address stored in system memory 103, application code 103a, the code from that memory address is brought into one of the cache lines of the code section of cache(s) 102b. Processing unit 102a then continues execution using the new information in cache(s) 102b until new information is again brought into cache(s) 102b (e.g., due to another cache miss or an uncached read).

[0049] The inventors have also observed that in order to record a bit-accurate representation of the execution of an application, the tracer 104a can record sufficient data to reproduce the influx of information into the cache(s) 102b as the processing units execute the thread(s) of the application. For example, one method of recording these influxes operates on a per-processing unit basis and at the innermost cache layer (e.g., L1). The method can involve, for each processing unit being traced, recording all cache misses and uncached reads (i.e., reads from hardware components and uncacheable memory) associated with the processing unit's L1 cache, as well as the time at which each data fragment was introduced into the processing unit's L1 cache during execution (e.g., using an executed instruction count or some other counter). If there are events that can be ordered across processing units (e.g., accesses to shared memory), these events can be logged across the resulting data streams (e.g., by using a monotonically increasing number (MIN) across the data streams).

[0050] However, since the L1 cache layer may include multiple different L1 caches, each of the L1 caches is associated with a different physical processing unit (e.g., Figure 2A ), so logging in this manner may record duplicate data, and therefore more data is strictly required for a "full-fidelity" trace. For example, if multiple physical processing units read from the same memory location (which may often happen in multi-threaded applications), this approach can log cache misses for the same memory location as well as data for each of the multiple physical processing units. It is worth noting that, as used herein, a "full-fidelity" trace is any trace that contains enough information to enable a complete replay of the traced entity (even though a particular "full-fidelity" trace may actually contain less data that encapsulates the same information than would be recorded using alternative tracing techniques).

[0051] To further reduce trace file size, the inventors have developed improved logging techniques that utilize one or more upper level caches to avoid logging at least a portion of this duplicate data. Instead, these improved techniques can log by reference to previously logged data. In particular, embodiments detect influxes (i.e., cache misses) to an internal or "lower level" processor cache (e.g., L1) based on activity by one processing unit, but then log that influx using one or more external or "upper level" shared processor caches when possible, by reference to influxes already logged on behalf of another traced processing unit.

[0052] To understand these techniques, note that in most environments, the upper cache is larger than the lower caches below it, and it is often the backing store for multiple lower caches. For example, in Figure 2AIn the example environment of , each L2 cache is a backing store for two L1 caches, and each L3 cache is a backing store for two L2 caches (and by extension, four L1 caches). Thus, an upper cache can retain knowledge about multiple lower caches (e.g., Figure 2A (e.g., L2 cache L1-A1 may retain knowledge about L1 caches L1-A1 and L1-A2, L2 cache L1-A2 may retain knowledge about L1 caches L1-A3 and L1-A4, and L3 cache L3-A may retain knowledge about L2 caches L2-A1 and L2-A1 as well as L1 caches L1-A1, L1-A2, L1-A3, and L1-A4.) By leveraging knowledge of one or more upper cache layers, embodiments herein enable multiple opportunities to log influx caused by one processing unit by referencing influx that has already been logged on behalf of other processing units.

[0053] According to these embodiments, Figure 3 An example of a method 300 for tracking is shown, which is performed by recording the inflow to a lower level cache by referencing previous log data based on the knowledge of one or more upper level caches. Figure 3 exist Figure 1 and Figure 2A and Figure 2B is described in the context of .

[0054] In particular, Figure 3 Operating in an environment such as processor 102 or 201a, processor 102 or 201a includes a plurality of processing units, a plurality of N-level caches, and an (N+i)-level cache associated with two or more of the plurality of N-level caches and configured as a backing store for the plurality of N-level caches. In method 300 (and in the claims), N and i are positive integers, i.e., N ≥ 1, such that N is equal to 1, 2, 3, etc.; and i ≥ 1, such that i is equal to 1, 2, 3, etc. For example, referring to Figure 2AProcessor 201a includes a plurality of processing units A1, A2, etc. Processor 201a also includes a plurality of N-level caches L1-A1, L1-A2, etc. (i.e., where N is equal to 1). Processor 201a also includes an (N+i)-level cache associated with two or more of the plurality of N-level caches and configured as a backup repository for the plurality of N-level caches. For example, processor 201a includes an (N+i)-level cache L2-A1 (i.e., where N is equal to 1 and i is equal to 1) as a backup repository for N-level caches L1-A1 and L1-A2. In another example, processor 201a includes an (N+i)-level cache L3-A (i.e., where N is equal to 1 and i is equal to 2) as a backup repository for N-level caches L1-A1, L1-A2, etc. Processor 102 / 201a operates method 300 based on control logic such as microcode 102c and / or circuit logic.

[0055] As shown, method 300 includes act 301 of detecting an influx of data into a level N cache during execution at a first processing unit. In some embodiments, act 301 includes detecting an influx of data into a first level N cache of a plurality of level N caches, the influx of data comprising data stored at a memory location. For example, based on the activity of processing unit A1, such as a requested memory access to system memory 202 (e.g., due to normal or speculative execution of a first thread of application 104c), a cache miss may occur in cache L1-A1 (i.e., when N is equal to 1). Thus, a line of cache L1-A1 receives an influx of data (including a then-current value of the requested memory location). Depending on the cache attributes (e.g., which higher-level layers exist, whether the cache architecture is inclusive or exclusive, etc.) and the current cache state, the influx of data may originate from system memory 202 or a higher-level cache (e.g., L2-A1 and / or L3-A).

[0056] Method 300 also includes an action 302 of checking the (N+i) level cache based on execution at the second processing unit to determine whether the data of the influx has been logged. In some embodiments, action 302 includes: based on detecting the influx to the first N level cache, checking the (N+i) level cache to determine whether the data for the memory location has been previously logged by the second processing unit. For example, if i is equal to 1, so that the (N+i) level cache includes the (N+1) level cache, then the processor 201a or 201b can check the L2 cache (such as, L2-A1) (which has knowledge of cache L1-A2 and processing unit A2). This check can be used to determine whether the data for the memory location has been previously logged on behalf of processing unit A2. For example, based on a previous execution of the second thread of application 104c at processing unit A2 that caused a cache miss in cache L1-A2, the data has been previously logged. In an alternative example, if i is equal to 2, such that the (N+i) level cache includes the (N+2) level cache, then the processor 201a or 201b may check an L2 cache, such as cache L3-A (which has knowledge of all other caches in the processor 201a or 201b). This check may be used to determine whether data for the memory location has previously been logged on behalf of any of the processing units A2-A4 (e.g., based on previous execution of one or more other threads of the application 104c at one or more processing units A2-A4 that caused a cache miss in caches L1-A2, L1-A3, and / or L1-A4). Note that in this second example, the L2 cache may be skipped in the check.

[0057] As shown, action 302 can be repeated any number of times, incrementing the value of i each time. Although i is typically incremented by 1 each time, there may be embodiments in which i is incremented by a positive integer greater than 1. The effect of repeating action 302 is that multiple upper-level caches will be checked when i is incremented. For example, if i=1, when action 302 is initially executed, processor 201a or 201b may check the L2 cache layer (e.g., L2-A1 and / or L2-A2). If sufficient knowledge about applicable memory locations is not found in the L2 cache, processor 201a or 201b may repeat action 302 with i=2, thereby checking the L3 cache layer (e.g., L3-A). This can continue for as many cache layers as the computing environment provides. If the value of i incremented is greater than 1, one or more cache layers along the way can be skipped. It will be understood that checking multiple cache layers may be beneficial in an architecture that provides exclusive caches or provides caches that exhibit mixed inclusive / exclusive behavior. This is because in these architectures, it may not be possible to guarantee that the outer cache layer contains a complete superset of the data in the (multiple) inner cache layers.

[0058] In view of the foregoing, it will be appreciated that method 300 may operate in environments such as processor 102 or 201a where i is equal to 1, such that the (N+i) level cache includes the (N+1) level cache, and the processor further includes an (N+2) level cache configured as a backing store for the (N+1) level cache. In these environments, checking the (N+1) level cache to determine whether data for the memory location has previously been logged on behalf of the second processing unit (i.e., action 302) may include determining that no cache line in the (N+1) level cache corresponds to the memory location. Additionally, checking the (N+2) level cache to determine whether data for the memory location has previously been logged on behalf of the second processing unit.

[0059] As shown, based on the result of act 302 , the method includes act 303 : logging the inflow by reference when the data has been logged; or act 304 : logging the inflow by value when the data has not been logged.

[0060] In some embodiments, action 303 includes: when data for the memory location was previously logged on behalf of the second processing unit, causing the data for the memory location to be logged on behalf of the first processing unit by referencing log data that was previously logged on behalf of the second processing unit. Continuing with the above example, for example, if checking (N+1) level cache L2-A1 and / or checking (N+2) level cache L3-A results in the following determination: the data / memory location was already logged on behalf of processing unit A2 (based on inflows to cache L1-A2), then processor 201a may cause inflows to cache L1-A1 to be logged on behalf of processing unit A1 by referencing log entries created for processing unit A2. An example of how logging by reference may be accomplished will be given later.

[0061] Turning to an alternative outcome of action 302, in some embodiments, action 304 includes causing data for the memory location to be logged by value on behalf of the first processing unit when the data for the memory location has not yet been logged on behalf of the second processing unit. For example, if checking the (N+1) level cache L2-A1 and / or checking the (N+2) level cache L3-A results in a determination that the data / memory location has not yet been logged on behalf of another processing unit, the processor 201a causes inflows to cache L1-A1 to be logged by value on behalf of processing unit A1. Logging by value can include, for example, logging memory addresses and memory values ​​in data packets for processing unit A1. Note that logging by value can include any number of compression techniques to reduce the number of bits required to perform the actual logging.

[0062] If combined Figure 1As described, the processor(s) 102 may include buffer(s) 102e that may be used to temporarily store trace data. Thus, in method 300, "causing" different types of data to be logged may include the processor 102 storing such data in the buffer(s) 102e. Additionally or alternatively, it may include the processor 102 transmitting such data to the tracer 104a, writing such data to the trace file(s) 104d, and / or notifying the tracer 104a that data is available in the buffer(s) 102e. In some embodiments, the buffer(s) 102e may include one or more reserved portions of the cache(s) 102b. Thus, using the buffers 102e, in act 304 / 304, causing data for a memory location to be logged on behalf of the first processing unit (by reference or by value) may include delaying logging based on the availability of resources such as processor cycles, memory locations, bus bandwidth, etc. In embodiments where buffer(s) 102 e comprise one or more retained portions of cache(s) 102 b , the delayed logging may include invalidating a cache line (in the N-level cache and / or the (N+i)-level cache) rather than evicting it in order to retain data for the memory location for purposes of delayed logging.

[0063] The description of method 300 has been directed to an upper cache having "knowledge" about lower caches. The specific form of "knowledge" retained by the upper cache about lower caches may vary, as exemplified below.

[0064] In a basic form, this "knowledge" can simply be that there are cache lines in the upper level cache that correspond to cache lines in the lower level caches (i.e., cache lines corresponding to the same memory location and memory data). As mentioned above, in an inclusive cache, the upper level(s) store a superset of the data in the level(s) below them. For example, suppose Figure 2AThe caches in are inclusive. In this case, when the activity of processing unit A2 causes a location from system memory 202 to be imported into cache L1-A2, the same memory location is also cached in caches L2-A1 and L3-A. If the activity of processing unit A2 is being tracked, an embodiment may cause the memory location and its value to be logged on behalf of processing unit A2. Later, if the activity of processing unit A1 causes the same location from system memory 202 to be imported into cache L1-A1, and the location still stores the same data, the location is provided from cache L2-A1 because cache L2-A1 already has the data. The prior art may log the data again for processing unit A1 based on the influx into cache L2-A1. However, embodiments herein may instead recognize that the memory location and its value already exist in cache L2-A1, and therefore already exist in cache L1-A2. Because processing unit A2 is being logged, an embodiment may recognize that the memory location and its value are already being logged on behalf of processing unit A2 and therefore cause this new activity of processing unit A1 to be logged with reference to the log data previously logged on behalf of processing unit A2.

[0065] More refined forms of "knowledge" of upper-level caches are also possible. For example, embodiments may extend cache lines in one or more cache layers with additional "accounting" (or logging) bits that enable processor 102 to identify, for each cache line that implements the accounting bits, whether the cache line has been logged (possibly with the identity of the processing unit(s) that logged the cache line). To understand these concepts, Figure 4A The diagram is similar to Figure 2B 4. An example shared cache 400a of the shared cache 203 of FIG. 4 extends each of its cache lines 404 with one or more additional accounting bits 401. Thus, each cache line 404 includes accounting bit(s) 401, conventional address 402 bits, and value bits 403.

[0066] Alternatively, Figure 4B An example of a shared cache 400b is illustrated, which includes regular cache lines 405 storing memory addresses (e.g., regular addresses 402) and values ​​403, and one or more reserved cache lines 406 for storing accounting bits applicable to the regular cache lines 405. The bits of the reserved cache line(s) 405 are assigned to different accounting bit groups, with each accounting bit corresponding to a different one of the plurality of regular cache lines 405.

[0067] In the example Figure 4BIn a variation of , (multiple) reserved cache lines 406 can be reserved as one (or more) ways in each index of a set-associative cache (which will be discussed in detail later). For example, in an 8-way set-associative cache, one way in a set can be reserved for bookkeeping bits applicable to the other seven ways in the set. Since all ways in a given set are typically read in parallel by most processors, this can reduce the complexity of implementing reserved cache lines and can speed up access to reserved cache lines.

[0068] Regardless of how the accounting bits are actually stored, the accounting bits (multiple) 401 for each cache line may include one or more bits used as a flag (i.e., on or off) that is used by the processor (s) 102 to indicate whether the current value in the cache line represents a processing unit being logged (or, alternatively, whether it is consumed by a processing unit participating in logging). Therefore, the check in action 302 may include using this flag to determine whether the cache line has been logged by a processing unit participating in logging.

[0069] Alternatively, the accounting bits 401 for each cache line may include multiple bits. The multiple bits can be used in several ways. Using an approach referred to herein as "unit bits," the accounting bits 401 for each cache line may include a number of unit bits equal to the number of processing units 102a of the processor 102 (e.g., the number of logical processing units if the processor 102 supports hyperthreading, or the number of physical processing units if hyperthreading is not supported). These unit bits can be used by the processor 102 to track which particular processing unit or units have logged a cache line (if any). Thus, for example, a cache shared by two processing units 102a may associate two unit bits with each cache line.

[0070] In another approach using multiple accounting bits 401 (referred to herein as "index bits"), the accounting bits 401 for each cache line can include multiple index bits sufficient to represent the index of each processing unit 102a of the processor 102 of the computer system 101 that participates in logging, as well as a possible "reserved" value (e.g., -1). For example, if the processor 102 includes 128 processing units 102a, these processing units can be identified per cache line by an index value (e.g., 0-127) using only seven index bits. In some embodiments, one index value is reserved (e.g., "invalid") to indicate that no processor has logged the cache line. Thus, this would mean that seven index bits would actually be able to represent 127 processing units 102a, plus the reserved value. For example, binary values ​​0000000-1111110 can correspond to index positions 0-126 (decimal), and binary value 1111111 (e.g., -1 or 127 in decimal, depending on the interpretation) can correspond to "invalid" to indicate that no processor has logged the corresponding cache line, but this notation can vary depending on the implementation. Thus, the unit bit can be used by processor 102 to indicate whether a cache line has been logged (e.g., a value different from -1) and as an index to the particular processing unit that logged the cache line (e.g., the processing unit that most recently consumed the cache line). This second approach of using multiple accounting bits 401 has the advantage of supporting a large number of processing units in cache 102b with little overhead, but has the disadvantage of having less granularity than the first approach (i.e., only one processing unit is identified at a time).

[0071] In view of the foregoing, it will be understood that in action 302, checking the (N+i) level cache to determine whether data for the memory location has previously been logged on behalf of the second processing unit includes: determining whether the cache line corresponding to the memory location in the (N+i) level cache has one or more accounting bits set.

[0072] Another mechanism that can be used to determine whether a cache line has been logged is to utilize set associative caching and way-locking. Since the processor's cache 102b is typically much smaller than the system memory 103 (typically by several orders of magnitude), there are typically many more memory locations in the system memory 103 than there are lines in any given layer of the cache 102b. As such, some processors define a mechanism for mapping multiple memory locations of the system memory to (multiple) lines of one or more cache layers. Processors typically employ one of two general techniques: direct mapping and associative (or set associative) mapping. Using direct mapping, different memory locations in the system memory 103 are mapped to only one line in the cache layer, so that each memory location can only be cached in a specific line in that layer.

[0073] On the other hand, using a set associative mapping, different locations in system memory 103 may be cached to one of multiple lines in the cache layer. Figure 5 An example 500 of a set-associative mapping between system memory and cache is illustrated. Here, cache lines 504 of a cache layer 502 are logically divided into two different groups of cache lines, including a first group (identified as index 0) of two first cache lines 504a and 504b, and a second group (identified as index 1) of two cache lines 504c and 504d. Each cache line in a group is identified as a different "way", such that cache line 504a is identified by index 0, way 0, cache line 504b is identified by index 0, way 1, and so on. As further depicted, memory locations 503a, 503c, 503e, and 503g (memory indices 0, 2, 4, and 6) are mapped to index 0. Thus, each of these locations in system memory can be cached to any cache line at index 0 within the group (i.e., cache lines 504a and 504b). The specific pattern of mapping depicted is for illustrative and conceptual purposes only and should not be construed as the only way that memory indices can be mapped to cache lines.

[0074] A set-associative cache is often referred to as an N-way set-associative cache, where N is the number of "ways" in each set. Figure 5 Cache 502 will be referred to as a 2-way set associative cache. Processors typically implement N-way caches, where N is a power of 2 (e.g., 2, 4, 8, etc.), with the values ​​of N typically being selected to be 4 and 8 (although the embodiments herein are not limited to any particular value of N or subset of N values). Notably, a 1-way set associative cache is generally equivalent to a direct-mapped cache in that each set contains only one cache line. Additionally, if N is equal to the number of lines in the cache, it is referred to as a fully associative cache in that it includes a single set containing all lines in the cache. In a fully associative cache, any memory location can be cached to any line in the cache.

[0075] Notice, Figure 5 A simplified view of system memory and cache is shown to illustrate the general principles. For example, although Figure 5 Individual memory locations are mapped to cache lines, but it should be understood that each line in the cache may store data associated with multiple addressable locations in system memory. Figure 5 In the example, each location (503a-503h) in the system memory (501) can actually represent multiple addressable memory locations. Additionally, the mapping can be between the actual physical address in the system memory 501 and the line in the cache 502, or an intermediate layer of virtual addresses can be used.

[0076] A set associative cache can be used to determine whether a cache line has been logged by using way locking. Way locking locks or reserves one or more ways in the cache for a certain purpose. In particular, embodiments herein utilize way locking to reserve one or more ways for a processing unit being tracked, such that the locked / reserved ways are dedicated to storing cache misses associated with the execution of that unit. Therefore, referring again to Figure 5 , if "way 0" is locked for the tracked processing unit, then cache lines 504a and 504c (i.e., index 0, way 0 and index 1, way 0) will be dedicated to cache misses related to execution of that unit, and the remaining cache lines will be used for all other cache misses. Therefore, to determine whether a particular cache line has been logged, processor 102 only needs to determine whether the cache line stored in the "N+1" cache layer is part of a way that has been reserved for the tracked processing unit.

[0077] In view of the foregoing, it will be understood that in action 302, checking the (N+i) level cache to determine whether data for the memory location has previously been logged on behalf of the second processing unit includes: determining whether the cache line corresponding to the memory location in the (N+i) level cache is stored in the way corresponding to the logged processing unit.

[0078] As previously explained, caches operate according to a CCP, which defines how coherence is maintained between the various caches when processing units read from and write to cached data, and how to ensure that a processing unit always reads valid data from a given location in the cache. Thus, in conjunction with operating the cache, processor 102 maintains and stores CCP state data. The granularity with which different processors and / or different CCPs track cache coherence state and make this cache coherence data available to tracker 104a can vary. For example, on the one hand, some processors / CCPs track cache coherence for each cache line and each processing unit. Thus, these processors / CCPs can track the state of each cache line associated with each processing unit. This means that a single cache line can have information about its state, as it is associated with each processing unit 102a. Other processors / CCPs have a smaller granularity and only track cache coherence at the cache line level (and lack per-processing unit information). On the other hand, because only one processor can have exclusive access to a line at a time (exclusive, modified, etc.), processor manufacturers may choose to track cache coherence at the cache line level solely for efficiency reasons. As an example of an intermediate granularity, the processor / CCP may track the cache coherency of each cache line, along with the index of the processing unit with the current cache line state (e.g., indices 0, 1, 2, 3 for a four processing unit processor).

[0079] Regardless of the granularity at which CCP state data is maintained at a given processor, the CCP state data may be included in the "knowledge" that the (N+i)-level cache has about cached data. In particular, the CCP state data associated with a given cache line in the (N+i)-level cache may be used to determine whether the cache line has been logged by one of the processing units. For example, if the CCP state data indicates that a particular processing unit has considered a given cache line to be "shared," that data may in turn be used to determine that the processing unit has logged a read from the cache line. Thus, it will be understood that in act 302, checking the (N+i)-level cache to determine whether data for a memory location has previously been logged on behalf of a second processing unit may include determining whether a cache line in the (N+i)-level cache corresponding to the memory location has associated CCP state data that can be used to determine that the cache line has been logged.

[0080] In action 303, the data inflow may be logged by reference to previously logged data (typically data logged by a different processing unit that caused the current inflow). Logging by reference may be accomplished using one or more of a variety of methods (including combinations thereof), some of which are described below.

[0081] The first method logs by referencing the memory address of the previous log record. For example, suppose Figure 2A Processing unit A2 in system memory 202 has logged data representing a specific memory address (i.e., in system memory 202) and specific data stored at that memory address. Later, if the specific memory address / specific data is an inflow for processing unit A1, processing unit A1 may store a log entry that identifies (i) the specific memory address and (ii) processing unit A2. In this way, processing unit A1 avoids re-logging the actual data stored at the memory address (which may be of considerable size). Some variations of this first method may also store sorting data, such as a series of MINs that increment from the data flow across processing units A1 and A2. The MINs may later be used to sort the inflow of processing unit A1 for one or more events at processing unit A2 (e.g., those events also associated with MINs from the same series). Accordingly, in act 303, causing data for a memory location to be logged on behalf of the first processing unit by reference to log data that was previously logged on behalf of the second processing unit may include one or more of the following: logging the address of the memory location, or logging the address of the memory location and sorting data (such as the MIN).

[0082] The second method logs by referring to the previous owner of the cache line where the data was stored. For example, suppose Figure 2AProcessing unit A2 in has logged a first influx of data. Assume also that the first influx causes the data to be cached in a cache line of the (N+i) level cache (e.g., cache L2-A1), where processing unit A2 is identified as the owner of the cache line. Later, if processing unit A1 causes a second influx of the same data, processing unit A1 may become the owner of the cache line in the (N+i) level cache. Processing unit A1 may then store a log entry identifying the previous owner of the cache line (i.e., processing unit A2) so that A2's log entry can be used later to obtain the data. This means that logging by reference can involve recording the identity of the cache line as well as the previous owner of the cache line (e.g., possibly avoiding recording memory addresses and memory values). Therefore, in act 303, causing data for a memory location to be logged on behalf of the first processing unit by reference to log data that has previously been logged on behalf of the second processing unit may include logging the second processing unit as the previous owner of the cache line corresponding to the memory location.

[0083] The third method is to log by reference to CCP data. For example, as mentioned above, the CCP can store cache coherence states for each cache line used by different processing units for reading and writing. The granularity of this data can vary depending on the implementation of the processor, but for example, the cache coherence state of each cache line associated with each processing unit can be tracked, the cache coherence state of each cache line can be tracked, and the index of the processing unit that owns the current cache line state (e.g., 0, 1, 2, 3, etc.) can be tracked. The third method uses available CCP data to track which processing unit(s) previously owned the cache coherence state of the cache line, which cache coherence state can then be used to identify which processing unit(s) have logged the value of the cache line. This means that logging by reference can involve recording CCP data for the cache line (e.g., again potentially avoiding recording memory addresses and memory values). Therefore, in act 303, logging data for a memory location on behalf of a first processing unit by reference to log data that has previously been logged on behalf of a second processing unit can include logging reference to the CCP data of the second processing unit.

[0084] A fourth approach is to log by reference to cache ways. As mentioned, set associative caches can be used to determine whether a cache line has been logged by using way locking. For example, assume that way locking is used to reserve one or more ways for processing unit P2, and P2 logs a first influx of data. The first influx also causes the (N+i) level cache (e.g., cache L2-A1) to store the data of the first influx in a cache line associated with the way. When another processing unit (e.g., P1) has a second influx of the same data, the presence of the cache line in the (N+i) level cache indicates that P2 has logged the data. An embodiment can log references to P2's log data based on annotating the ways in which the cache lines are stored, and can again potentially avoid logging memory addresses and memory values. This embodiment can also be used in conjunction with record sorting information (e.g., MIN) to sort events between P1 and P2. Thus, in act 303, causing data for a memory location to be logged on behalf of the first processing unit by reference to log data that has previously been logged on behalf of the second processing unit may include one or more of: a logged reference to a cache way, or a logged reference to a cache way and sort data.

[0085] In addition to logging the inflow of a first processing unit based on the previous inflow of a second processing unit, embodiments also include optimizations for reducing (or even eliminating) logging by a single processing unit when there are multiple inflows of the same data. For example, referring to Figure 2A , processing unit A1 may cause a cache miss in an N-level cache (e.g., an L1-A1 cache) for specific data at a memory location. In response, the cache hierarchy may import the data into the L1-A1 cache, and potentially also into the (N+i)-level cache (e.g., an L2-A1 cache and / or an L3-A cache). In addition, the influx to processing unit A1 may be logged by value. Later, the data may be evicted from the L1-A1 cache. In a typical cache environment, this may cause the data to be actively evicted from the L2-A1 cache and / or the L3-A cache as well. However, rather than causing (multiple) evictions in the L2-A1 and / or L3-A caches, various embodiments may retain appropriate (multiple) cache lines in one or more of these (N+i)-level caches. Thus, method 300 may include evicting a first cache line in a first N-level cache corresponding to the memory location, while retaining a second cache line in the (N+i)-level cache corresponding to the memory location.

[0086] Later, if processing unit A1 causes a subsequent cache miss in the L1-A1 cache for the same data, the retained cache line(s) in the (N+i) level cache (e.g., L2-A1 and / or L3-A cache) can be used to determine that the data has been logged on behalf of processing unit A1. Thus, in some embodiments, the subsequent cache miss is logged by processing unit A1 with reference to the previous log entry. In other embodiments, the log entry may be omitted entirely for the subsequent cache miss because processing unit A1 already has the data in its tracking. Thus, method 300 may include: based on detecting a subsequent influx into the first N level cache, based on the presence of the second cache line, causing the subsequent influx to be logged by reference, the subsequent influx also including the data stored at the memory location. Additionally or alternatively, method 300 may include (i) detecting a subsequent influx into the first level N cache based on additional code executed at the first processing unit, the subsequent influx also including data stored at the memory location, and (ii) determining that the subsequent influx does not need to be logged based at least on detecting the subsequent influx into the first level N cache and at least on the presence of the second cache line.

[0087] Thus, embodiments herein are based on using at least two levels or tiers of processor caches to track execution effects across multiple processing units to create bit-accurate "time travel" trace records. This may include modifications to processor hardware and / or microcode that assist in (i) detecting influxes into an internal or "lower level" processor cache (i.e., cache misses) based on the activity of the traced processing unit, and (ii) using an external or "upper level" shared processor cache to determine whether data for that influx has already been logged on behalf of another traced processing unit. If the data has already been logged, the influx can be logged by reference to previous log entries, and these techniques can be extended to "N" levels of cache. Recording trace files in this manner may require only modest processor modifications and can reduce the performance impact of trace recording and the size of trace files by orders of magnitude when compared to existing trace recording methods.

[0088] The present invention may be embodied in other specific forms without departing from the spirit or essential characteristics of the present invention. The described embodiments are to be considered in all respects as illustrative only and not restrictive. The scope of the present invention is therefore indicated by the appended claims rather than the foregoing description. All changes that come within the meaning and range of equivalence of the claims are intended to be included within their scope.

Claims

1. A computing device comprising: multiple processing units; Multiple N-level caches; an (N+i)-level cache associated with two or more of the plurality of N-level caches and configured as a backing store for the plurality of N-level caches; as well as Control logic that configures the computing device to perform at least the following: detecting an influx into a first N-level cache of a plurality of N-level caches, the influx comprising data stored at a memory location; as well as Based on detecting the influx to the first N-level cache, checking the (N+i)-level cache to determine whether the data for the memory location has been previously logged on behalf of the second processing unit, and performing one of the following: when the data for the memory location has been previously logged on behalf of the second processing unit, causing the data for the memory location to be logged on behalf of the first processing unit by referencing the log data previously logged on behalf of the second processing unit; as well as The data for the memory location is caused to be logged by value on behalf of the first processing unit when the data for the memory location has not been previously logged on behalf of the second processing unit.

2. The computing device of claim 1 , wherein checking the (N+i) level cache to determine whether the data for the memory location has been previously logged on behalf of the second processing unit comprises one or more of: determining whether a cache line in the (N+i) level cache corresponding to the memory location has one or more accounting bits set; determining whether a cache line corresponding to the memory location in the (N+i)-level cache is stored in a way corresponding to the logged processing unit; and A determination is made as to whether a cache line in the (N+i) level cache corresponding to the memory location has associated cache coherence protocol (CCP) state data that can be used to determine that the cache line has been logged. 3 . The computing device of claim 1 , wherein i is equal to 1, such that the (N+i) level cache comprises an (N+1) level cache. 4 . The computing device of claim 1 , wherein i is equal to 2, such that the (N+i) level cache comprises an (N+2) level cache.

5. The computing device of claim 1 , wherein: i is equal to 1, so that the (N+i)-level cache includes the (N+1)-level cache; The computing device further includes an (N+2) level cache, the (N+2) level cache being configured as a backing store for the (N+1) level cache; and Checking the (N+1) level cache to determine whether the data for the memory location has been previously logged on behalf of the second processing unit includes: determining that a cache line corresponding to the memory location does not exist in the (N+1) level cache; as well as The (N+2) level cache is checked to determine whether the data for the memory location has been previously logged on behalf of the second processing unit.

6. The computing device of claim 1 , wherein causing the data for the memory location to be logged on behalf of the first processing unit by referencing log data previously logged on behalf of the second processing unit comprises one or more of: Log recording the address of the memory location; Log recording the address and sort data of the memory location; Log records references to cache ways; Logs record references to cache ways and sort data; the second processing unit being logged as a previous owner of a cache line corresponding to the memory location; as well as The log record references cache coherence protocol (CCP) data of the second processing unit.

7. The computing device of claim 1 , wherein causing the data for the memory location to be logged on behalf of the first processing unit comprises: The logging is delayed based on availability of one or both of processor resources and memory resources.

8. The computing device of claim 1 , wherein delaying the logging comprises: A cache line is invalidated to retain the data for the memory location for the logging of delays.

9. The computing device of claim 1 , wherein the control logic further configures the computing device to perform at least the following: evicting a first cache line corresponding to the memory location in the first level N cache while retaining a second cache line corresponding to the memory location in the (N+i) level cache; and Based on detecting a subsequent influx into the first Level-N cache, the subsequent influx is caused to be logged by reference based on the existence of the second cache line, the subsequent influx including the data stored at the memory location.

10. The computing device of claim 1 , wherein the control logic further configures the computing device to perform at least the following: evicting a first cache line corresponding to the memory location in the first N-level cache while retaining a second cache line corresponding to the memory location in the (N+i)-level cache; detecting a subsequent influx into the first Level N cache based on additional code execution at the first processing unit, the subsequent influx including the data stored at the memory location; as well as Based at least on detecting the subsequent influx into the first N-level cache and at least on the existence of the second cache line, it is determined that the subsequent influx does not need to be logged.

11. A method for tracking and recording, the tracking and recording being performed based on: recording inflows to a lower-level cache by referencing previous log data based on knowledge of one or more upper-level caches, the method being implemented at a computing device, the computing device comprising: (i) a plurality of processing units, (ii) a plurality of N-level caches, and (iii) an (N+i)-level cache, the (N+i)-level cache being associated with two or more of the plurality of N-level caches and configured as a backing store for the plurality of N-level caches, the method comprising: detecting an influx into a first N-level cache of a plurality of N-level caches, the influx comprising data stored at a memory location; as well as Based on detecting the influx to the first N-level cache, checking the (N+i)-level cache to determine whether the data for the memory location has been previously logged on behalf of the second processing unit, and performing one of the following: When the data for the memory location has been previously logged on behalf of the second processing unit, causing the data for the memory location to be logged on behalf of the first processing unit by referencing the log data previously logged on behalf of the second processing unit, and The data for the memory location is caused to be logged by value on behalf of the first processing unit when the data for the memory location has not been previously logged on behalf of the second processing unit.

12. The method of claim 11 , wherein checking the (N+i) level cache to determine whether the data for the memory location has been previously logged on behalf of the second processing unit comprises one or more of: determining whether a cache line in the (N+i) level cache corresponding to the memory location has one or more accounting bits set; determining whether a cache line corresponding to the memory location in the (N+i)-level cache is stored in a way corresponding to the logged processing unit; and A determination is made as to whether a cache line in the (N+i) level cache corresponding to the memory location has associated cache coherence protocol (CCP) state data that can be used to determine that the cache line has been logged.

13. The method of claim 11, wherein i is equal to 1, such that the (N+i)-level cache comprises an (N+1)-level cache.

14. The method of claim 11, wherein i is equal to 2, such that the (N+i) level cache comprises an (N+2) level cache.

15. The method according to claim 11, wherein: i is equal to 1, so that the (N+i)-level cache includes the (N+1)-level cache; The computing device further includes an (N+2) level cache, the (N+2) level cache being configured as a backing store for the (N+1) level cache; and Checking the (N+1) level cache to determine whether the data for the memory location has been previously logged on behalf of the second processing unit includes: determining that a cache line corresponding to the memory location does not exist in the (N+1) level cache; as well as The (N+2) level cache is checked to determine whether the data for the memory location has been previously logged on behalf of the second processing unit.

16. The method of claim 11, wherein causing the data for the memory location to be logged on behalf of the first processing unit by referencing log data previously logged on behalf of the second processing unit comprises one or more of: Log recording the address of the memory location; Log recording the address and sort data of the memory location; Log records references to cache ways; Logs record references to cache ways and sort data; the second processing unit being logged as a previous owner of a cache line corresponding to the memory location; as well as The log record references cache coherence protocol (CCP) data of the second processing unit.

17. The method of claim 11, wherein causing the data for the memory location to be logged on behalf of the first processing unit comprises delaying the logging, delaying the logging comprising invalidating a cache line to retain the data for the memory location for the delayed logging.

18. The method according to claim 11, further comprising: evicting a first cache line corresponding to the memory location in the first N-level cache while retaining a second cache line corresponding to the memory location in the (N+i)-level cache; as well as Based on detecting a subsequent influx into the first Level-N cache, the subsequent influx is caused to be logged by reference based on the existence of the second cache line, the subsequent influx including the data stored at the memory location.

19. The method according to claim 11, further comprising: evicting a first cache line corresponding to the memory location in the first N-level cache while retaining a second cache line corresponding to the memory location in the (N+i)-level cache; detecting a subsequent influx into the first Level N cache based on additional code execution at the first processing unit, the subsequent influx including the data stored at the memory location; as well as Based at least on detecting the subsequent influx into the first N-level cache and at least on the existence of the second cache line, it is determined that the subsequent influx does not need to be logged.

20. A computer program product comprising one or more hardware storage devices having computer-executable instructions stored thereon, the computer-executable instructions configuring a computing device comprising (i) a plurality of processing units, (ii) a plurality of N-level caches, and (iii) an (N+i)-level cache, the (N+i)-level cache being associated with two or more of the plurality of N-level caches and configured as a backing store for the plurality of N-level caches, to perform at least the following: detecting an influx into a first level N cache of a plurality of level N caches, the influx comprising data stored at a memory location; and Based on detecting the influx to the first N-level cache, checking the (N+i)-level cache to determine whether the data for the memory location has been previously logged on behalf of the second processing unit, and performing one of the following: When the data for the memory location has been previously logged on behalf of the second processing unit, causing the data for the memory location to be logged on behalf of the first processing unit by referencing the log data previously logged on behalf of the second processing unit, and The data for the memory location is caused to be logged by value on behalf of the first processing unit when the data for the memory location has not been previously logged on behalf of the second processing unit.

Images