Tracking by logging the inflow of upper-level shared caches and cache coherence protocol conversions between lower-level caches
By recording cache inflows and cache consistency protocol conversion at the processor cache level, the problems of tracking recording performance and file size in the prior art are solved, and efficient bit-precision time travel tracking records are achieved.
Patent Information
- Application Number
- CN201980013653.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-02-16
- Filing Date
- 2019-01-25
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2039-01-25
AI Technical Summary
When recording tracking files, existing debugging technologies have a trade-off between the robustness of tracking data and performance overhead, resulting in a large size of tracking files, a large impact on program execution performance, and a high hardware cost.
Through a hardware assist mechanism, execution effects are recorded across multiple processing units at least two processor cache levels, and processor hardware and microcode are modified to record inflows to external shared processor caches and cache consistency between internal caches Sex protocol conversion reduces performance impact and file size of track records.
It realizes bit-accurate time travel tracking records, reduces the performance impact and file size of tracking records, reduces hardware costs, and improves the efficiency of tracking records.
Smart Images

Figure CN111727425B_ABST
Abstract
Description
Background Art
[0001] When writing code during the development of a software application, developers typically spend a significant amount of time "debugging" the code to find runtime and other source code errors. In doing so, developers can employ several methods to reproduce and locate source code defects (bugs), such as observing the behavior of the program based on different inputs, inserting debugging code (e.g., printing variable values, tracing execution branches, etc.), temporarily removing code sections, etc. Tracking down runtime errors to pinpoint code defects can take up a significant portion of application development time.
[0002] In order to assist developers in the code debugging process, many types of debugging applications ("debuggers") have been developed. These tools provide developers with the ability to track, visualize, and change the execution of computer code. For example, a debugger can visualize the execution of code instructions, can present code variable values at different times during code execution, can enable developers to change the code execution path, and / or can enable developers to set "breakpoints" and / or "watchpoints" on code elements of interest ("breakpoints" and / or "watchpoints" are reached during execution, causing the execution of the code to be suspended), etc.
[0003] An emerging form of debugging application enables "time travel," "reverse," or "historical" debugging. With "time travel" debugging, the execution of a program (e.g., an executable entity such as a thread) is recorded / traced by a tracing application into one or more trace files. These trace files can then be used to replay the execution of the program at a later time for forward and backward analysis. For example, a "time travel" debugger can enable a developer to set forward breakpoints / watchpoints (like a conventional debugger) as well as reverse breakpoints / watchpoints.
[0004] Several considerations may be taken into account when recording trace files. Most notably, there are inherent tradeoffs between the robustness of the recorded trace data and the overhead incurred by tracing the program. These tradeoffs are primarily in the size of the trace file and the performance impact on the execution of the traced program. Furthermore, since tracing may be done with hardware assistance (or entirely in software), there may also be hardware design and other hardware cost considerations. Summary of the invention
[0005] Embodiments described herein relate to mechanisms for creating bit-accurate "time travel" trace records by a processor using hardware assistance. These mechanisms are based on tracking execution effects across multiple processing units on at least two processor cache tiers or layers. In particular, these mechanisms can modify the hardware and / or microcode of the processor so that it helps record (i) influx (i.e., cache misses) to an external or "upper" shared processor cache, and (ii) cache consistency protocol (CCP) transitions between multiple internal or "lower" processor caches supported by the external / upper shared processor cache. These techniques can be extended to "N" level caches. Recording trace files in this manner can require only modest processor modifications and can reduce the performance impact of trace recording and trace file size by several orders of magnitude compared to existing trace recording methods.
[0006] Embodiments relate to (multiple) computing devices, the computing devices comprising a plurality of processing units, a plurality of (N-1) level caches, and an N-level cache associated with two or more (N-1) level caches of the plurality of (N-1) level caches. The N-level cache is configured as a backup storage repository for the two or more (N-1) level caches. The (multiple) computing devices include control logic, which configures the (multiple) computing devices to cause one or more data inflows to a particular location in the N-level cache to be logged based at least on detecting one or more data inflows to a particular location in the N-level cache during execution across the plurality of processing units. The control logic also configures the (multiple) computing devices to cause one or more (N-1) level CCP conversions between the two or more (N-1) level caches to be logged. The (N-1) level CCP conversion is caused by a particular location accessed by two or more processing units of the plurality of processing units.
[0007] Embodiments also relate to methods for tracking and recording data inflows to an external level cache and CCP transitions between internal caches. These methods are implemented at a computing device, the computing device comprising (i) a plurality of processing units, (ii) a plurality of (N-1) level caches, and (iii) a level N cache, the level N cache being associated with two or more (N-1) level caches of the plurality of (N-1) level caches and configured as a backup repository for the two or more (N-1) level caches. The method comprises: causing one or more data inflows to a specific location in the level N cache to be logged based at least on detecting one or more data inflows to a specific location in the level N cache during execution across the plurality of processing units. The method further comprises: causing one or more (N-1) level cache consistency protocol (CCP) transitions between the two or more (N-1) level caches to be logged, the (N-1) level CCP transitions being caused by the specific location being accessed by two or more processing units of the plurality of processing units.
[0008] This summary is provided to introduce some concepts in a simplified form, which are further described in the detailed description below. This summary is neither intended to identify key features or essential features of the claimed subject matter nor to be used as an aid in determining the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] In order to describe the manner in which the above and other advantages and features of the present invention can be obtained, a more particular description of the invention briefly described above will be made by reference to specific embodiments of the invention illustrated in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the invention and are therefore not to be considered limiting of the scope of the invention, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:
[0010] Figure 1 An example computing environment is illustrated that records a "bit accurate" trace of multiple threads of an application by recording the effects on two or more processor cache layers as those threads are executed by multiple processing units, including recording flows into an upper shared processor cache and cache coherence protocol (CCP) transitions between multiple lower processor caches;
[0011] Figure 2A An example computing environment including multiple layers of cache is illustrated;
[0012] Figure 2B An example of caching is shown;
[0013] Figure 3 illustrates an example of transitions between load and store states when a cache line is read and written by multiple processing units;
[0014] Figure 4 A flow chart illustrating an example method for tracking data inflow to an outer level cache and CCP transitions between inner caches is illustrated;
[0015] Figure 5A An example computing environment including an N-level cache layer and multiple (N-1)-level cache layers is illustrated;
[0016] Figure 5B An example computing environment including an N-level cache layer, a plurality of (N-1)-level cache layers, and a plurality of (N-2)-level cache layers is illustrated;
[0017] Fig. 6A illustrates an example shared cache that extends each of its cache lines with one or more additional bookkeeping bits;
[0018] Figure 6B illustrates an example of a shared cache including regular cache lines storing memory addresses and values and one or more reserved cache lines for storing bookkeeping bits applied to the regular cache lines;
[0019] Fig. 7A illustrates an example of a trace record in which only a subset of lower level caches are logged; and
[0020] Figure 7B An example of a trace record in which only a subset of lower-level caches are logged is illustrated, in an environment with at least two outer N-level caches, each associated with multiple inner (N-1)-level caches. DETAILED DESCRIPTION
[0021] Embodiments described herein relate to mechanisms for creating bit-accurate "time travel" trace records by a processor using hardware assistance. These mechanisms are based on tracking the effects of execution across multiple processing units on at least two processor cache levels or layers. In particular, these mechanisms can modify the hardware and / or microcode of the processor so that it helps record (i) inflows (i.e., cache misses) to an external or "upper level" shared processor cache, and (ii) cache consistency protocol (CCP) conversions between multiple internal or "lower level" processor caches supported by the external / upper level shared processor cache. These techniques can be extended to "N" level caches. Recording trace files in this manner can require only moderate processor modifications, and it can reduce the performance impact of trace recording and the trace file size by several orders of magnitude compared to existing trace recording methods.
[0022] Figure 1An example computing environment 100 is illustrated that facilitates recording "bit accurate" tracking of multiple threads of an application by recording the effects on two or more processor cache layers as those threads are executed by multiple processing units, including recording influxes into an upper shared processor cache and CCP transitions between multiple lower processor caches. As shown, an embodiment may include or utilize a special purpose or general purpose computer system 101 that includes computer hardware such as, for example, one or more processors 102, a system memory 103, one or more data repositories 104, and / or input / output hardware 105.
[0023] Embodiments within the scope of the present invention include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media can be any available media that can be accessed by the computer system 101. Computer-readable media that store computer-executable instructions and / or data structures are computer storage devices. Computer-readable media that carry computer-executable instructions and / or data structures are transmission media. Therefore, by way of example and not limitation, embodiments of the present invention may include at least two distinct types of computer-readable media: computer storage devices and transmission media.
[0024] Computer storage is a physical hardware device that stores computer-executable instructions and / or data structures. Computer storage includes various computer hardware such as RAM, ROM, EEPROM, solid-state drives ("SSD"), flash memory, phase-change memory ("PCM"), optical disk storage, magnetic disk storage or other magnetic storage, or any other hardware device that can be used to store program code in the form of computer-executable instructions or data structures, and that can be accessed and executed by computer system 101 to implement the functionality disclosed herein. Thus, for example, computer storage can include the depicted system memory 103, the depicted data repository 104, or other storage (e.g., on-processor storage) that can store computer-executable instructions and / or data structures.
[0025] Transmission medium can include network and / or data link that can be used for carrying program code in the form of computer executable instruction or data structure, and this program code can be visited by computer system 101." network " is defined as one or more data link that can transport electronic data between computer system and / or module and / or other electronic equipment.When information is transmitted or provided to computer system by network or another communication connection (combination of hardwire, wireless or hardwire or wireless), computer system can be considered as transmission medium by connection.Above-mentioned combination also should be included in the scope of computer readable medium.For example, input / output hardware 105 can include the hardware (for example, network interface module (for example, " NIC ")) that can be used for network and / or data link that can be used for carrying program code in the form of computer executable instruction or data structure.
[0026] In addition, upon reaching various computer system components, program code in the form of computer executable instructions or data structures may be automatically transferred from transmission media to computer storage devices (or vice versa). For example, computer executable instructions or data structures received over a network or data link may be buffered in RAM within a NIC (e.g., input / output hardware 105) and then ultimately transferred to system memory 103 and / or less volatile computer storage devices (e.g., data repository 104) at computer system 101. Thus, it should be understood that computer storage devices may be included in computer system components that also (or even primarily) utilize transmission media.
[0027] Computer executable instructions include, for example, instructions and data that cause computer system 101 to perform a specific function or group of functions when executed at processor(s) 102. Computer executable instructions may be, for example, binary files, intermediate format instructions such as assembly language, or even source code.
[0028] Those skilled in the art will appreciate that the present invention can be practiced in a network computing environment with many types of computer system configurations, including: personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, tablet computers, pagers, routers, switches, etc. The present invention can also be practiced in a distributed system environment, in which local and remote computer systems with network links (through a hardwired data link, a wireless data link, or through a combination of a hardwired and wireless data link) perform tasks. Like this, in a distributed system environment, a computer system can include multiple component computer systems. In a distributed system environment, program modules can be located in local and remote memory storage devices.
[0029] Those skilled in the art will also appreciate that the present invention can be practiced in a cloud computing environment. A cloud computing environment can be distributed, but this is not required. When distributed, a cloud computing environment can be distributed internationally within an organization and / or have components owned across multiple organizations. In this specification and the appended claims, "cloud computing" is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage devices, applications, and services). The definition of "cloud computing" is not limited to any of the other numerous advantages that can be obtained from such a model when properly deployed.
[0030] The cloud computing model can consist of various characteristics, such as on-demand self-service, broad network access, resource pooling, rapid elasticity, measured services, etc. The cloud computing model can also take the form of various service models, such as, for example, software as a service ("SaaS"), platform as a service ("PaaS"), and infrastructure as a service ("IaaS"). The cloud computing model can also be deployed using different deployment models, such as private cloud, community cloud, public cloud, hybrid cloud, etc.
[0031] Some embodiments (such as, cloud computing environments) may include a system with one or more hosts, each capable of running one or more virtual machines. During operation, the virtual machine will simulate an operating computing system, thereby supporting an operating system and perhaps one or more other applications. In some embodiments, each host includes a hypervisor that simulates the virtual resources of the virtual machine using physical resources abstracted from the perspective of the virtual machine. The hypervisor also provides appropriate isolation between virtual machines. Therefore, from the perspective of any given virtual machine, even if the virtual machine is only interfaced with the appearance of physical resources (e.g., virtual resources), the hypervisor provides the illusion that the virtual machine is interfacing with physical resources. Examples of physical resources include processing power, memory, disk space, network bandwidth, media drives, etc.
[0032] As shown, data repository 104 may store computer executable instructions and / or data structures representing application programs, such as, for example, tracer 104a, operating system kernel 104b, and application 104c (e.g., the application that is the subject of tracer 104a tracing and one or more trace files 104d). When these programs are executing (e.g., using (multiple) processors 102), system memory 103 may store corresponding runtime data, such as runtime data structures, computer executable instructions, etc. Thus, Figure 1 System memory 103 is illustrated as including runtime application code 103a and application runtime data 103b (eg, each corresponding to application 104c ).
[0033] The tracer 104a may be used to record a bit accurate trace of an executing application, such as application 104c, and store the trace data in trace file(s) 104d. In some embodiments, the tracer 104a is a standalone application, while in other embodiments, the tracer 104a is integrated into another software component, such as an operating system kernel 104b, a hypervisor, a cloud fabric, etc. Although the trace file(s) 104d are depicted as being stored in the data repository 104, the trace file(s) 104d may also be recorded exclusively or temporarily in the system memory 103 or at some other storage device. As explained later, the tracer 104a may interoperate with specific features of the processor(s) 102 that enable tracing using the CCP of the processor.
[0034] Figure 1 A simplified representation of the internal hardware components that comprise processor(s) 102. As shown, each processor 102 comprises multiple processing units 102a. Each processing unit may be physical (i.e., a physical processor core) and / or logical (i.e., a logical core provided by a physical core supporting hyperthreading, where more than one application thread executes at a physical core). Thus, for example, even though processor 102 in some embodiments comprises only a single physical processing unit (core), it may comprise two or more logical processing units 102a presented by the single physical processing unit.
[0035] Each processing unit 102a executes processor instructions defined by an application (e.g., tracker 104a, operating kernel 104b, application 104c, etc.), and the instructions are selected from a predefined processor instruction set architecture (ISA). The specific ISA of each processor 102 varies based on the processor manufacturer and processor model. Common ISAs include IA-64 and IA-32 architectures from INTEL, AMD64 architecture from ADVANCED MICRO DEVICES, and various Advanced RISC Machine ("ARM") architectures from ARMHOLDINGS, PLC, but a large number of other ISAs exist and can be used by the present invention. Typically, an "instruction" is the smallest externally visible (i.e., outside the processor) unit of code that can be executed by a processor.
[0036] Each processing unit 102a obtains processor instructions from one or more processor caches 102b and executes the processor instructions based on data in the cache(s) 102b, based on data in registers 102d, and / or in the absence of input data. In general, each cache 102b is a small amount (i.e., small relative to the typical amount of system memory 103) of random access memory that stores on-processor copies of portions of a backup storage repository (such as system memory 103 and / or another cache in the cache(s) 102b). For example, when executing application code 103a, one or more caches 102b contain certain portions of application runtime data 103b. If the processing unit(s) 102a requires data that is not yet stored in a particular cache 102b, a "cache miss" occurs, and the data is obtained from the system memory 103 or another cache, possibly "evicting" some other data from the cache 102b.
[0037] Typically, the processor cache(s) 102b are divided into separate hierarchies, layers or levels, such as level 1 (L1), level 2 (L2), level 3 (L3), etc. Depending on the processor implementation, the levels may be part of the processor 102 itself (e.g., L1 and L2) and / or may be separate from the processor 102 (e.g., L3). Thus, Figure 1 The cache(s) 102b may include one of these levels (L1), or may include multiple of these levels (eg, L1 and L2, or even L3). Figure 2A An example environment 200 illustrating multiple layers of caching is shown. Figure 2A In the example, there are two processors 201a and 201b (eg, each corresponding to Figure 1 different processors 102) and system memory 202 (e.g., corresponding to Figure 1 In the example environment 200, each processor 201 includes four physical processing units (ie, units A1-A4 for processor 201a and units B1-B4 for processor 201b).
[0038] The example environment 200 also includes three layers of cache within each processing unit 201. Environment 200 is merely an example layout and is not limited to a cache hierarchy in which the embodiments herein may operate. In environment 200, at the bottom or innermost level, each processing unit is associated with its own dedicated L1 cache (e.g., L1 cache "L1-A1" in processor 201a is used for unit A1, L1 cache "L1-A2" in processor 201a is used for unit A2, etc.). Moving up one level, each processing unit 201 includes two L2 caches (e.g., L2 cache "L2-A1" in processor 201a used as a backup repository for L1 caches L1-A1 and L1-A2, L2 cache "L1-A2" in processor 201a used as a backup repository for L1 caches L1-A3 and L1-A4, etc.). Finally, at the highest or outermost level, each processing unit 201 includes a single L3 cache (e.g., L3 cache "L3-A" in processor 201a that serves as a backing store for L2 caches L2-A1 and L2-A2, and L3 cache "L3-B" in processor 201b that serves as a backing store for L2 caches L2-B1 and L2-B2). As shown, system memory 202 serves as a backing store for L3 caches L3-A and L3-B.
[0039] like Figure 2A As shown, when multiple cache layers are used, processing unit(s) 102a interact directly with the lowest layer (L1). In most cases, data flows between the layers (e.g., on reads, the L3 cache interacts with system memory 103 and serves data to the L2 cache, which in turn serves data to the L1 cache). When processing unit 102a performs a write operation, the caches coordinate to ensure that those caches that have affected data shared between processing unit(s) 102a no longer have the data. This coordination is performed using the CCP.
[0040] The caches in environment 200 may therefore be viewed as "shared" caches. For example, each L2 and L3 cache serves multiple processing units within a given processor 201 and is therefore shared by the processing units. Even though each cache corresponds to a single processing unit, the L1 cache in a given processor 201 may be viewed as shared overall because the individual L1 caches may be coordinated with each other (i.e., via the CCP) to ensure consistency (i.e., so that the memory locations of each cache are viewed consistently across all L1 caches). Similarly, the L2 caches within each processor 201 may be coordinated via the CCP. Additionally, if processor 201 supports hyperthreading, each individual L1 cache may be viewed as shared by two or more logical processing units and may therefore be "shared" even at an individual level.
[0041] Typically, each cache includes multiple “cache lines.” Each cache line stores a memory block from its backing store (e.g., system memory 202 or a higher-level cache). For example, Figure 2B An example of at least a portion of a cache 203 is illustrated, the cache 203 comprising a plurality of cache lines 206, each cache line 206 comprising at least an address portion 204 and a value portion 205. The address portion 204 of each cache line 206 is configured to store an address in the system memory 202 to which the cache line corresponds, and the value portion 205 initially stores a value received from the system memory 202. The value portion 205 may be modified by the processing unit and eventually evicted back to a backup storage repository. As indicated by the ellipsis, the cache 203 may include a large number of cache lines. For example, a contemporary INTEL processor may include a separate L1 cache having 512 or more cache lines. In such a cache, each cache line may typically be used to store a 64-byte (512-bit) value for an 8-byte (64-bit) memory address. As shown in FIG. Figure 2A As intuitively shown, cache size generally increases with each level (i.e., L2 cache is generally larger than L1 cache, L3 cache is generally larger than L2 cache, etc.).
[0042] The addresses stored in the address portion 204 of each cache line 206 may be physical addresses, such as actual memory addresses in the system memory 202. Alternatively, the addresses stored in the address portion 204 may be virtual addresses, which are addresses that are mapped to physical addresses to provide an abstraction (e.g., using a page table managed by an operating system). Such an abstraction may be used, for example, to facilitate memory isolation between different processes executing at the processor(s) 102, including isolation between user mode processes and kernel mode processes associated with the operating system kernel 104b. When virtual addresses are used, the processor 102 may include a translation lookaside buffer (TLB) 102f (typically part of a memory management unit (MMU)) that maintains recently used memory address mappings between physical addresses and virtual addresses.
[0043] The (multiple) caches 102b may include a code cache portion and a data cache portion. When the application code 103a is executed, the (multiple) code portions of the (multiple) caches 102b may store at least a portion of the processor instructions stored in the application code 103a, and the (multiple) data portions of the (multiple) caches 102b may store at least a portion of the data structure of the application runtime data 103b. In addition, the cache may be inclusive, exclusive, or may include both inclusive and exclusive behaviors. For example, in an inclusive cache, the L3 layer stores a superset of the data in the L2 layer below the L3 layer, and the L2 layer stores a superset of the L1 layer below them. In an exclusive cache, the layers may not intersect, for example, if the L3 cache has data needed by the L1 cache, they may exchange information, such as data, addresses, etc.
[0044] Return to Figure 1 Each processor 102 also includes microcode 102c, which includes control logic (i.e., executable instructions) that controls the operation of the processor 102, and which typically acts as an interpreter between the processor's hardware and the processor ISA exposed by the processor 102 to executing applications. The microcode 102 is typically embodied in an on-processor storage device such as a ROM, EEPROM, or the like.
[0045] Registers 102d are hardware-based storage locations that are defined based on the ISA of the processor(s) 102 and are read and / or written to by processor instructions. For example, registers 102d are typically used to store values obtained from cache(s) 102b for use by instructions, store results of executing instructions, and / or store status or conditions (such as some side effects of executing instructions (e.g., the sign of a value changing, a value reaching zero, the occurrence of a carry, etc.)), processor cycle counts, etc. Therefore, some registers 102d may include a "flag" for signaling a certain state change caused by executing a processor instruction. In some embodiments, the processor 102 may also include control registers for controlling different aspects of the processor's operation. Although Figure 1 Registers 102d are depicted as a single block, but it will be understood that each processing unit 102a typically includes one or more corresponding sets of registers 102d that are specific to that processing unit.
[0046] In some embodiments, the processor(s) 102 may include one or more buffers 102e. As will be discussed below, the buffer(s) 102e may be used as a temporary storage location for trace data. Thus, for example, the processor(s) 102 may store portions of trace data in the buffer(s) 102e and, at appropriate times (such as when there is available memory bus bandwidth and / or idle processor cycles), flush the data to the trace file(s) 104d.
[0047] As described above, the processor operates on the cache(s) 102b according to one or more CCPs. Typically, the CCP defines how the consistency between data between the various caches 102b is maintained when the various processing units 102a read and write data from the various caches 102b, and how to ensure that the various processing units 102a always read valid data from a given location in the cache(s) 102b. The CCP is typically associated with and enables the memory model defined by the ISA of the processor 102.
[0048] Examples of general CCPs include the MSI protocol (i.e., modified, shared, and invalid), the MESI protocol (i.e., modified, exclusive, shared, and invalid), and the MOESI protocol (i.e., modified, owned, exclusive, shared, and invalid). Each of these protocols defines a state for a single location (e.g., a line) in the cache(s) 102b. A "modified" cache location contains data that has been modified in the cache(s) 102b and is therefore inconsistent with corresponding data in a backing store (e.g., system memory 103 or another cache). When a location with a "modified" state is evicted from the cache(s) 102b, the general CCP requires the cache to ensure that its data is written back to the backing store, or that another cache takes over the responsibility. A "shared" cache location contains data that has not been modified from the data in the backing store, exists in a read-only state, and is shared by the processing unit(s) 102a. The cache(s) 102b can evict the data without having to write it to the backing store. "Invalid" cache locations do not contain valid data and can be considered empty and can be used to store data from cache misses. "Private" cache locations contain data that matches a backup storage repository and is used only by a single processing unit 102a. The data can be changed to a "shared" state at any time (i.e., in response to a read request), or can be changed to a "modified" state when it is written. An "owned" cache location is shared by two or more processing units 102a, but one of the processing units has exclusive rights to change it. When the process changes, it notifies the other processing units because the notified processing unit may need to be invalidated or updated based on the CCP implementation.
[0049] Different processors and / or different CCPs track cache coherence states and make the cache coherence data available to the tracker 104a at varying granularities. For example, on the one hand, some processors / CCPs track the cache coherence of each cache line and each processing unit. These processors / CCPs can therefore track the state of each cache line associated with each processing unit. Because a single cache line is associated with each processing unit 102a, this means that a single cache line can have information about its state. Other processors / CCPs have smaller granularities and only track cache coherence at the cache line level (and lack information about each processing unit). On the other hand, because only one processor can occupy a (exclusive, modified, etc.) line at a time, the processor manufacturer can choose to track cache coherence at the cache line level only for efficiency reasons. As an example of an intermediate granularity, a processor / CCP can track the cache coherence of each cache line, as well as the index of the processing unit with the current cache line state (e.g., the indexes of four processing unit processors are 0, 1, 2, 3).
[0050] As implied, embodiments utilize the cache(s) 102b of the processor 102 to effectively record a bit-accurate trace of the execution of the application 104c and / or the operating system kernel 104b. These embodiments are established based on the following observation of the inventors: the processor 102 (including the cache(s) 102b) forms a semi-closed or quasi-closed system. For example, once the portions of data for a process (i.e., code data and runtime application data) are loaded into the cache(s) 102b, the processor 102 can operate on its own in a semi-closed or quasi-closed system in a time burst without any input. In particular, once the cache(s) 102b are loaded with data, one or more processing units 102a use the runtime data stored in the data portion(s) of the cache(s) 102b and use registers 102d to execute instructions from the code portion(s) of the cache(s) 102b.
[0051] When processing unit 102a requires some influx of information (e.g., because an instruction it is executing, about to execute, or may execute accesses code or runtime data that is not yet in cache(s) 102b), a "cache miss" occurs and the information is brought into cache(s) 102b from system memory 103. For example, if a data cache miss occurs when an executed instruction performs a memory operation at a memory address within application runtime data 103b, the data from that memory address is brought into one of the cache lines of the data portion of cache(s) 102b. Similarly, if a code cache miss occurs when an instruction performs a memory operation at a memory address application code 103a stored in system memory 103, the code from that memory address is brought into one of the cache lines of the code portion of cache(s) 102b. Processing unit 102a then continues execution using the new information in cache(s) 102b until new information is again brought into cache(s) 102b (e.g., due to another cache miss or an uncached read).
[0052] The inventors have also observed that, in order to record a bit-accurate representation of an executing application, the tracer 104a may record enough data to reproduce the influx of information into the cache(s) 102b as the processing unit executes the thread(s) of the application. The inventors have developed several methods for recording such data, each with various advantages and disadvantages.
[0053] For example, a first approach records all data introduced into cache(s) 102b by logging all cache misses and uncached reads (i.e., reads from hardware components and non-cacheable memory) and the time each data fragment was introduced into cache(s) 102b during execution (e.g., using an executed instruction count or some other counter). Although this approach is easier to implement than the following approach; however, while it typically records a much smaller trace file than the prior art, it may still record more data than is strictly necessary for bit-accurate tracing.
[0054] A second approach (which typically results in a much smaller trace file than the first approach) is to track and record only the cache lines that are "consumed" by each processing unit 102a, rather than recording all cache misses. As used herein, a processing unit has "consumed" a cache line when the processing unit knows its current value. This may be because the processing unit is the unit that wrote the current value of the cache line, or because the processing unit performed a read on the cache line. Some embodiments of this second approach involve extensions to one or more caches 102b (e.g., additional "logging" or "accounting" bits) that enable the processor 102 to identify, for each cache line, the one or more processing units 102a that consumed the cache line.
[0055] A third approach (which typically results in a smaller trace file than the first and second approaches) is to utilize the processor's CCP to determine a subset of "consumed" cache lines for recording in file(s) 104d, and which will still enable the activity of cache(s) 102b to be reproduced. This approach operates at a single cache level (e.g., L1) and logs the influx of data into that cache level, as well as a log of CCP operations at the granularity at which a processing unit causes a given CCP operation. This includes logging which processing unit(s) previously read and / or write accesses to a cache line. However, robust support for this third approach may result in extensive modifications to the processor's CCP logic because it operates at the granularity at which a processing unit causes a CCP operation.
[0056] According to an embodiment of the present invention, the fourth method operates at two or more cache levels. In particular, it logs the cache shared by the "upper level" (e.g., Figure 2A It also uses the CCP of at least one "lower" cache (e.g., if the shared cache is L2, then Figure 2A The CCP of the L1 cache in the shared cache; or if the shared cache is L3, Figure 2A The method may further include using a CCP (i.e., a CCP of the L2 cache in the L2 cache) to log a subset of CCP state transitions (i.e., between portions of a "load" operation and portions of a "store" operation) for each cache memory location. The effect is that less CCP data is logged compared to the third approach (i.e., because it is logged based on load / store transitions rather than per-processing unit activity, it logs much less CCP state data than the third approach). In implementations, such a log may be post-processed and augmented to achieve the level of detail logged in the third approach, but may be implemented using less costly hardware modifications than the third approach (e.g., because less CCP data needs to be tracked and logged by processor 102).
[0057] As mentioned, this fourth method records the transitions between the portions of a "load" operation and the portions of a "store" operation. For ease of description, the examples herein map the cache line states (i.e., modified, owned, exclusive, shared, and invalid) used in the CCPs discussed above (i.e., MSI, MESI, and MOESI) to one of the "load" states (i.e., the cache line has been read) or the "store" states (i.e., the cache line has been written). Table 1 below shows a non-limiting example mapping. It will be understood that there may be CCPs and states other than those discussed herein, and in view of the disclosure herein, one of ordinary skill in the art will recognize that similar mappings may be performed using many different CCPs. Alternatively, an implementation may record transitions between actual CCP states (e.g., modified, owned, exclusive, and shared) rather than transitions between mapped states.
[0058] Agreement Status Mapped state Modified storage Owned load exclusive storage Shared load Invalid Unmapped – the cache line is considered empty and no translations are tracked
[0059] Table 1
[0060] Figure 3 An example 300 of transitions between the load state and store state of Table 1 is illustrated as a cache line is read and written by multiple processing units. In particular, Figure 3 Illustrated is a first timeline 301 showing transitions between load portions and store portions based on load and store operations shown on a second timeline 302 showing cache line activity.
[0061] As shown on the far left of timeline 302, processing unit P1 stores value A to a cache line, and then stores values B and C to the cache line. For example, during this period of storage 304a, P1 may cause the cache line to be in an "exclusive" CCP state. Next, timeline 302 shows that processing unit P2 performs a load, reading value C. Correspondingly, timeline 301 shows this as a transition point 303a from storage period 304a to load period 304b. During period 304b, there are multiple loads from P1 and P2, each of which reads value C. Period 304b may correspond to, for example, a period in which the cache line is in a "shared" CCP state. The next store is performed by P3, and it writes value D. This is shown in timeline 301 as another transition point 303b between load period 304b and new storage period 304c. The next activity is a load by P1 (reading value D), which is shown as a new transition point 303c between the store period 304c and the new load period 304d. This period 304d includes loads of P1, P2, and P3, each reading value D. Finally, timeline 302 shows a store of P1, writing value E. This is shown as the final transition point 303d between the load period 304d and the new store period 304e.
[0062] According to the embodiments of this article, based on Figure 3 The CCP-based data logged for the cache line may include the identity of each transition point 303a-303d. This data can be used for replay to determine the order in which the processing units (i.e., P1, P2, and P3) were executed. Then, by replaying the code executed at each processing unit (in the order identified by the logged CCP data), the values read from and / or written to the cache line can be reconstructed. Although these values can be reconstructed upon replay, some embodiments can record a more robust trace by also logging the values of the cache line at one or more transition points. Other CCP-based data may be logged, such as which processing unit(s) accessed the cache between each transition point.
[0063] exist Figure 3 , each load / store transition also corresponds to a transition between processing units. However, an embodiment may log one or more load / store transitions by a separate processing unit. For example, if at transition point 303a, the load had been performed by P1 instead of P2, then that transition point 303a would be a transition for a separate processing unit (i.e., P1). In addition to logging load / store transitions corresponding to transitions between processing units, or even as an alternative thereto, an embodiment may also log such transitions.
[0064] Of course, other log data may be recorded to help reconstruct the order in which the processing units were executed. For example, an embodiment may occasionally (e.g., every N instructions, where N may be on the order of millions of instructions or more) log a hash of the register values (or similar values) of each processor 102 to help the replay engine choose among multiple potential execution paths.
[0065] Figure 4 An example of a method 400 for tracking data inflow to an outer level cache and CCP transitions between inner caches is illustrated. Figure 4 exist Figure 1-Figure 3 is described in the context of Figure 5A and Figure 5B Additional example computing environments 500a and 500b are described.
[0066] Typically, method 400 operates in an environment that includes multiple processing units and two or more cache layers, where an outer or higher level cache layer (i.e., level N) acts as a backing store for an inner or lower level cache layer (i.e., level (N-1)). The inner or lower level cache layer can in turn act as a backing store for another inner or lower level cache layer (i.e., level (N-2)). These layers can continue inward for any number of layers. Although not required, in many environments, each individual cache in the innermost or lowest cache layer corresponds to a single physical processing unit.
[0067] Figure 5A and Figure 5B The concepts are illustrated. Figure 5A An example 500a of a computing environment 503 including two cache layers (i.e., level N and level (N-1)) is illustrated. For example, Figure 5A At least one N-level cache 501a is depicted. Figure 5A In FIG. 5 , the N-level cache 501a is depicted as an L2 cache layer. Below the N-level cache 501a, Figure 5A Two (N-1) level caches 502a and 502b are shown, but as indicated by ellipsis 505a, there may be more than two (N-1) level caches 502. Figure 5A , (N-1) level cache 502 is depicted as the L1 cache layer. Figure 5A 502, each (N-1) level cache 502 may correspond to a single physical processing unit. Thus, in one example, the N-level cache 501a may correspond to a processor socket containing two or more physical cores, each physical core being associated with its own (N-1) level cache 502.
[0068] As shown by ellipsis 505c and N-level cache 501b, there may be any number of N-level caches 501 (and corresponding lower-level caches). For example, in some processing environments, each N-level cache 501 may correspond to a different processor socket. However, in other processing environments (such as a non-uniform memory access (NUMA) processing environment), these N-level caches 501 may be "groups" within the same processor socket, each group being supported by a different memory or memory segment.
[0069] on the other hand, Figure 5BAn example 500b of a computing environment 504 is illustrated that includes three cache layers (i.e., level N, level (N-1), and level (N-2). Here, example 500b includes at least one level N cache 503a (e.g., an L3 cache), which level N cache 503a includes two or more level (N-1) caches 501a (e.g., an L2 cache), and the (N-1) level caches 501a in turn each include two or more level (N-2) caches 502 (e.g., an L1 cache). Again, as indicated by ellipsis 505d and level N caches 503b, there may be any number of level N caches 503 (and corresponding lower level caches). Although the processing units are Figure 5B Not explicitly shown, but in many processor environments, each (N-2) level cache 502 may correspond to a single physical processing unit, and each (N-1) level cache 501 and (N) level cache 503 may correspond to multiple processing units.
[0070] Return now Figure 4 , method 400 operates across multiple processing units during execution. For example, Figure 4 Operations may be performed to facilitate tracking the execution of multiple threads. As shown, method 400 includes action 401, which is logging inflows to the N-level cache during parallel execution across multiple processing units. In some embodiments, action 401 may include causing one or more data inflows to specific locations in the N-level cache to be logged based at least on detecting one or more data inflows to specific locations in the N-level cache during execution across multiple processing units. For example, during execution across processing units (e.g., by one or more of multiple threads), there may be one or more cache misses, which results in data for a specific memory location in system memory 103 being imported into an external cache. Figure 5A In the context of 503, this may mean that for each cache miss, a particular memory location is cached in a cache line of the N-level cache 501a (e.g., L2 cache). Figure 5B In an environment 504 of , this may mean that for each cache miss, a particular memory location is cached in a cache line of a Level N cache 503a (e.g., an L3 cache). Due to these cache misses, circuitry and / or microcode 102c of processor(s) 102 may initiate logging of data imported into the external cache to be logged to trace file(s) 104d. Alternatively, action 401 may be initiated by a tracer 104a including an operating system kernel 104b and / or a hypervisor.
[0071] Method 400 also includes an act 402 of logging (N-1)-level cache coherence protocol transitions (e.g., between multiple (N-1)-level caches) during parallel execution. In some embodiments, act 402 may include causing one or more (N-1)-level CCP transitions between two or more (N-1)-level caches to be logged, the (N-1)-level CCP transitions resulting from a particular location being accessed by two or more of the multiple processing units. For example, although a cache miss may initially populate an outer / higher level cache (e.g., Figure 5A N-level cache 501a or Figure 5B 503a), but particularly when inclusive caching is used, data is ultimately passed between lower cache layers and consumed by one or more processing units. Embodiments log the CCP transformations for cached memory locations (which can then be used to reconstruct how the data is used) rather than logging the data again as it is passed through the layers and consumed. Thus, for example, in Figure 5A In the context of 503, this may mean logging CCP transitions between the (N-1) level caches (i.e., caches 502a and 502b) as their corresponding processing units read from or write to the (N-1) level cache lines corresponding to the buffered memory locations. Figure 5B In the environment 504 of FIG. 504, since these caches are level N-1 caches in this environment, this may mean logging a CCP switch between caches 501a and 501b. Again, logging may be initiated by circuitry and / or microcode 102c of processor(s) 102. Alternatively, logging may be initiated by a tracer 104a including an operating system kernel 104b and / or a hypervisor.
[0072] As combined Figure 3 As discussed, the CCP logging may include logging one or more transition points between store periods and load periods when the cache memory location is accessed by the (N-1) level cache (caches 502a and 502b). This may also potentially include logging the values stored in the cached memory location at one or more of these transition points. This may also include causing one or more CCP transitions based on the activity of a single processing unit (e.g., load / store transitions of a single processing unit) to be logged.
[0073] exist Figure 5AIn environment 503, there are no additional lower level caches, so method 400 can end. However, environment 504 includes another layer. Thus, method 400 can also include action 403, which is to log (N-2) level cache consistency protocol conversions (for example, between multiple (N-2) level caches) during parallel execution. Action 403 can be performed in an environment that also includes multiple (N-2) level caches, and wherein each (N-1) level cache is associated with two or more of the multiple (N-2) level caches, and the (N-1) level cache is configured as a backup repository for two or more (N-2) level caches. In some embodiments, action 403 can include: causing one or more (N-2) level CCP conversions between two or more (N-2) level caches to be logged, the (N-2) level CCP conversions being caused by a particular location being accessed by two or more processing units of the multiple processing units. Thus, for example, in Figure 5B In environment 504, this may mean logging CCP transitions between the (N-2) level caches (i.e., caches 502a and 502b) as their corresponding processing units read from or write to the (N-2) level cache lines corresponding to the cached memory locations.
[0074] Similar to act 402, the CCP logging may include logging one or more transition points between store periods and load periods when the cache memory location is accessed by the (N-2) level cache (i.e., caches 502a and 502b). This may also potentially include logging the values stored in the cached memory location at one or more of these transition points.
[0075] Figure 4 Indicating act 403 may be performed any number of times, each time decreasing the level of N. Thus, method 400 may operate in an environment including two or more cache layers (ie, N≥2) without an upper limit on the number of cache layers that may be used.
[0076] Additionally, method 400 may be repeated (typically in parallel) for each external N-level cache layer. Thus, for example, in Figure 5A In , method 400 may be performed for cache layer 501a and also for cache layer 501b. Figure 5B, method 400 can be performed for cache layer 503a and also for cache layer 503b. When method 400 is performed on two or more N-level cache layers, log data generated based on each external N-level cache layer can be logged to different data streams in trace file 104d (e.g., one data stream per processing unit), or can be logged to a single data stream while being marked in a manner that enables data from different N-level caches to be separated. The order between the two data sets can be at least partially sorted by logging certain events (e.g., access to shared memory) with reference to a monotonically increasing number of uses (MIN) across data sets, CCP data indicating cache communication between data sets, etc.
[0077] As combined Figure 1 As described, the processor(s) 102 may include buffer(s) 102d, which may be used to temporarily store trace data. Thus, in the method 400, "causing" different types of data to be logged may include the processor 102 storing such data in the buffer(s) 102d. Additionally or alternatively, this may include the processor 102 transmitting such data to the tracer 104a, writing such data to the trace file(s) 104d, and / or notifying the tracer 104a that the data is available in the buffer(s) 102d. In some embodiments, the buffer(s) 102d may include one or more reserved portions of the cache(s) 102b.
[0078] If combined Figure 1 As described, the processor 102 may also include control registers for controlling different aspects of the processor operation. In some embodiments, one or more control registers include one or more bits that control whether logging is enabled. This may be at the granularity of a processing unit group (e.g., a socket, a NUMA group, etc.), or even on a per-processing unit basis. Therefore, using such logging control bits, only a subset of processing units (including the zero set) may actually participate in logging at any given time. In this way, the tracker 104a may control in fine granularity which processing units / threads are being tracked and when the tracking is active. This may enable the tracker 104a to control which application(s) is being tracked, which specific thread(s) in the application is being tracked, whether kernel mode code is being tracked, etc.
[0079] As previously mentioned, some tracing methods utilize extensions to one or more of the cache(s) 102b (i.e., additional "logging" or "accounting" bits) that enable the processor 102 to identify, for each cache line, the one or more processing units 102a that consumed the cache line. Similar techniques can be used in conjunction with method 400 to reduce the amount of data that needs to be logged while still maintaining full fidelity tracing. For example, the use of a "logging" or "accounting" bit can prevent a cache line from being re-logged in the following circumstances: after the processing unit switches to another context (e.g., another thread, kernel mode, etc.), if the other context does not modify the cache line. Additionally or alternatively, the use of a "logging" or "accounting" bit can enable a trace entry for one thread to reference data that has been logged on behalf of another thread. Therefore, as used herein, a "full fidelity" trace is any trace that contains enough information to enable a complete replay of the traced entity, even though a particular "full fidelity" trace may actually contain less data that encapsulates the same information that may be recorded using an alternative tracing technique.
[0080] To understand these concepts, Fig. 6A An example shared cache 600a (similar to Figure 2B 203), the shared cache 600a extends each of its cache lines 604 with one or more additional accounting bits 601. Thus, each cache line 604 includes (multiple) accounting bits 601, conventional address bits 602, and value bits 603. Alternatively, Figure 6B An example of a shared cache 600b is illustrated, which includes a regular cache line 605 storing a memory address 602 and a value 603, and one or more reserved cache lines 606 for storing accounting bits applied to the regular cache line 605. The bits of the (multiple) reserved cache line 606 are allocated to different accounting bit groups, each group corresponding to a different regular cache line 605. Figure 6B In a variation of the example of , (multiple) reserved cache lines 606 can be reserved as one (or more) ways in each index of a set-associative cache. For example, in an 8-way set-associative cache, one way in a set can be reserved for accounting bits applicable to the other seven ways in the set. Since all ways in a given set are typically read in parallel by most processors, this can reduce the complexity of implementing reserved cache lines and can speed up access to reserved cache lines.
[0081] Regardless of how the accounting bits are actually stored, the accounting bits 601 for each cache line of the external (N-level) cache may include one or more bits used as a flag (i.e., turned on or off) used by the (multiple) processors 102 to indicate whether the current value in the cache line is actually consumed by the processing unit participating in the logging. This is useful because if the value imported into the N-level is not actually used by the processing unit being tracked, the value does not need to be logged. For example, if the value imported into the N-level is actually used by the processing unit being tracked, some implementations only log the current value in the cache line (i.e., in action 401). Values can be imported into the N-level cache, but not actually used, because the processor 201 may be involved in speculative execution. The processor 201 can participate in speculative execution by executing instructions that it predicts may be executed in the future to keep its execution pipeline intact. The execution of these speculative instructions may cause N-level cache misses. If it turns out that the speculative instruction(s) that caused these cache misses were not retired (e.g., they were discarded because a different execution path than the predicted / speculated execution path was taken), some implementations may choose not to log these values. Thus, the processor 102 / tracker 104a may log the value of a cache miss only if one of these bits is set for a given cache line. Additionally or alternatively, if the same value is re-imported into the N-level cache, these bits may be used to prevent the same value from being logged again. For example, a speculative cache miss may cause the logged value to be evicted from the N-level cache. Later, a non-speculative instruction may access the value again, resulting in another cache miss. These bits may be used to track this situation, which may be used to avoid re-logging the value.
[0082] In some embodiments, lower-level caches may be configured to "push" consumption data to higher-level caches to assist in logging. For example, when accounting bits are used on an external N-level cache, one or more lower-level caches may transmit an indication of when a particular value was consumed (perhaps along with information about the processing unit that consumed the particular value) to the external N-level cache. This provides the external N-level cache with information that can be used to set / clear bits.
[0083] As mentioned, some embodiments may include a control register with one or more logging control bits that control whether a processing unit participates in logging. As mentioned, while this may be at the granularity of each processing unit, it may also be at the granularity of a group of processing units. Some implementations may lack logging control bits entirely. Either of these situations may present challenges when logging is desired for only a subset of processing units. For example, Fig. 7AAn example 700a of using the trace logging of method 400 is illustrated, but in which only a subset of the lower level cache is logged. In particular, Fig. 7A An external N-level cache 701 (e.g., L2 cache) and four internal (N-1)-level caches 702a-702d are shown. As indicated by the asterisks, only the processing units associated with the (N-1)-level caches 702a, 702b, and 702d are the units that the tracker 104a is interested in logging. However, if the hardware environment lacks logging control bits or has logging control bits that generally operate only at the granularity of the N-level cache 701, then the (N-1)-level cache 702c will also be logged.
[0084] This situation may cause some problems. First, the resulting trace may include unnecessary trace data related to the code executed at cache 702c. Second, security issues may arise when the trust level of the code executed at cache 702c is different from the trust level of the code executed using caches 702a, 702b, and 702d. This may be because the (multiple) processes associated with the code executed using cache 702c have different trust levels than the (multiple) processes associated with the code executed using caches 702a, 702b, and 702d (for example, they may be from different users and / or the code executed using caches 702a, 702b, and 702d may operate in user mode, while the code executed using cache 702c may operate in kernel mode). Alternatively, this may be because the core that manages code executed using cache 702c is different from the core that manages code executed using caches 702a, 702b, and 702d (for example, if both cores are executed on top of a hypervisor and the hypervisor assigns the processing unit(s) associated with cache 702c to one core and the processing unit(s) associated with cache 702a, 702b, and 702d to another core).
[0085] To overcome these problems, some embodiments may post-process the resulting trace file to separate the trace data streams and generate a trace file containing only a subset of the data streams that need to be traced (i.e., to remove data streams for processing units other than the processing unit for which recording is desired). To maintain security, this operation will typically be performed in a hardware and software environment that has a trust level at least equal to the trust level used for trace recording. For example, if trace recording is performed at the kernel level, the post-processing environment will typically have a trust level at least equivalent to the trust level of the kernel, or if trace recording is performed at the hypervisor level, the post-processing environment will typically have a trust level at least equivalent to the trust level of the hypervisor. Post-processing can be performed at the same computer system that performed the recording, or it can be performed at one or more other computer systems with the necessary trust level. Additionally, post-processing can potentially be performed in real time or near real time (by the computer system that is performing the recording or by one (some) other computer systems), or it can be postponed to a later time. Note that as used herein, "trust level" may refer to user-level trust (as opposed to software mandatory licensing), such as the trust level of a system and / or infrastructure administrator. For example, if a hypervisor is part of a cloud infrastructure, other components of the infrastructure may be at the same trust level, even if they are located in different machines, as long as they ensure the same level of isolation, access restrictions, etc. as the hypervisor.
[0086] As mentioned, method 400 can be performed in parallel for multiple external N-level cache layers (e.g., different sockets or NUMA groups). As explained, circumstances may cause the data flow to be different for each external layer. The post-processing described herein also applies to these circumstances. For example, Figure 7B An example 700b is illustrated having two external N-level caches 701a and 701b (e.g., L2 caches), each with four internal (N-1)-level caches 702a-702h (e.g., L1 caches). As indicated by the asterisks, the processing units associated with the (N-1)-level caches 702a, 702b, and 702d within the N-level cache 701a and the processing units associated with the (N-1)-level caches 702e and 702f within the N-level cache 701b are the units that the tracker 104a is interested in recording. However, the (N-1)-level cache 702c within the N-level cache 701a and 702g and 702h within the N-level cache 701b may also be recorded. Here, similar post-processing can be performed on the data recorded by the N-level cache 701a with an appropriate trust level, and separate post-processing can be performed on the data recorded by the N-level cache 701b with an appropriate trust level.
[0087] These principles are not limited to separating out trace data that has not been requested to be recorded. Instead, they can also be used to separate traces of threads that all require recording but have different trust levels with each other. For example, returning to Fig. 7A , threads executing on processing units associated with caches 702a and 702b have different trust levels than threads (multiple) executing on processing units associated with cache 702d, even though they are all executed on the same core. Here, post-processing at least equivalent to the trust level of the core can be used to separate these traces into different trace files.
[0088] This can even be extended to other trust levels. For example, further assume that the kernel that executes threads using caches 702a, 702b, and 702d is different from the kernel that executes (multiple) threads using cache 702c (all executing on the same hypervisor). Here, post-processing at least equivalent to the trust level of the hypervisor can separate the trace data of the threads using caches 702a, 702b, and 702d from the trace data of the (multiple) threads using cache 702c. Then, the trace data of the threads using caches 702a, 702b can be separated from the trace data of the (multiple) threads using cache 702d with post-processing at least equivalent to the trust level of the hypervisor or kernel that uses caches 702a, 702b, and 702d.
[0089] Thus, embodiments herein are based on tracking execution effects across multiple processing units on at least two processor cache hierarchies or layers to create bit-accurate "time travel" trace records. This can include modifications to processor hardware and / or microcode to help record (i) inflows (i.e., cache misses) to an external or "upper level" shared processor cache, and (ii) cache consistency protocol (CCP) conversions between multiple internal or "lower level" processor caches supported by the external / upper level shared processor cache. These techniques can be extended to "N" level caches. Recording trace files in this manner can require only modest processor modifications, and when compared to existing trace recording methods, they can reduce the performance impact of trace recording and trace file size by several orders of magnitude.
[0090] The present invention may be embodied in other specific forms without departing from the spirit or essential features of the present invention. The described embodiments should be considered in all respects only as illustrative and not restrictive. Therefore, the scope of the present invention is indicated by the appended claims rather than the preceding description. All changes falling within the equivalent meaning and scope of the claims should be included within their scope.
Claims
1. A computing device, comprising: Multiple processing units; Multiple N-1 level caches; an N-level cache associated with two or more of the plurality of N-1 level caches and configured as a backing store for the two or more N-1 level caches; as well as control logic comprising computer executable instructions to configure the computing device to record execution of an entity that executes parallel threads across two or more of the plurality of processing units, recording the execution of the entity based on performing at least the following: causing the memory data inflow to the specific cache location in the N-level cache to be logged in a replayable trace of the entity based at least on detecting a memory data inflow from a specific memory location in the system memory to a specific cache location in the N-level cache, the memory data inflow being caused by execution of a first thread of the entity by a first processing unit of the plurality of processing units; as well as at least based on having caused the memory data inflow to the particular cache location in the N-level cache to be logged in the replayable trace of the entity, subsequently causing one or more N-1 level cache coherence protocol (CCP) transitions between the two or more N-1 level caches to be logged in the replayable trace of the entity, based on execution of a second thread of the entity by a second processing unit among the plurality of processing units, the N-1 level CCP transitions being caused by access of the particular cache location by the second processing unit among the plurality of processing units.
2. The computing device of claim 1, wherein: The computing device further includes a plurality of N-2 level caches, and wherein each N-1 level cache is associated with two or more N-2 level caches of the plurality of N-2 level caches, and the N-1 level cache is configured as a backing store for the two or more N-2 level caches; and The stored control logic also configures the computing device to cause one or more N-2 level CCP transitions between the two or more N-2 level caches to be logged into the replayable trace of the entity, the N-2 level CCP transitions being caused by the particular cache location being accessed by the two or more processing units of the plurality of processing units. 3 . The computing device of claim 1 , wherein the memory data influx is caused by one or more cache misses on the N-level cache due to execution of the first thread of the entity. 4 . The computing device of claim 1 , wherein the one or more N-1 level CCP conversions include one or more conversion points between a store period and a load period.
5. The computing device of claim 4, wherein causing the one or more N-1 level CCP transformations to be logged comprises: At one or more of the transition points, a log records a value stored in the particular cache location.
6. The computing device of claim 1, wherein: The computing device also includes a buffer; and The control logic further configures the computing device to perform deferred logging into the replayable trace of the entity based on one or more of: storing the memory data inflow from the particular memory location into the buffer, or The one or more N-1 level CCP translations between the two or more N-1 level caches are stored to the buffer.
7. The computing device of claim 1, wherein the computing device further comprises one or more logging control bits that control whether one or more of the plurality of processing units participate in logging.
8. The computing device of claim 1, wherein the control logic comprises one or more of circuitry or stored microcode.
9. A computing device according to claim 1, wherein at least the N-level cache includes one or more accounting bits, and the one or more accounting bits identify one or more of the following: (i) whether the value at the specific cache location has been logged, or (ii) whether the value at the specific cache location should be logged.
10. The computing device of claim 9, wherein the control logic further configures the computing device to set at least one accounting bit in the level N cache based on communications from one or more of the level N-1 caches.
11. The computing device of claim 9, wherein the control logic further configures the computing device to use the one or more accounting bits to avoid logging memory data influx caused by speculative execution of instructions.
12. The computing device of claim 1 , wherein causing the one or more Level N-1 cache coherence protocol (CCP) transitions between the two or more Level N-1 caches to be logged into the replayable trace of the entity comprises: Causing the identity of a point of each N-1 level CCP transition to be logged into the replayable trace of the entity, including identifying the point in each N-1 level CCP transition that corresponds to one of: (i) a first transition from a load CCP state for the particular cache location to a store CCP state for the particular cache location, or (ii) a second transition from the store CCP state for the particular cache location to the load CCP state for the particular cache location.
13. The computing device of claim 12, wherein the loaded CCP state comprises at least one of an owned CCP state or a shared CCP state, and wherein the stored CCP state comprises at least one of a modified CCP state or an exclusive CCP state.
14. The computing device of claim 12, wherein the first transition corresponds to a transition from a first period of one or more loads to a second period of one or more stores, and the second transition corresponds to a transition from a third period of one or more stores to a fourth period of one or more loads.
15. A method for recording execution of an entity executing parallel threads across two or more processing units of a plurality of processing units based on memory data inflow to an outer level cache and CCP transitions between an inner cache, the method being implemented at a computing device, the computing device comprising (i) the plurality of processing units, (ii) a plurality of N-1 level caches, and (iii) a level N cache associated with two or more of the plurality of N-1 level caches and configured as a backing store for the two or more level N-1 level caches, the method comprising: causing the memory data inflow to the specific cache location in the N-level cache to be logged in a replayable trace of the entity based at least on detecting a memory data inflow from a specific memory location in the system memory to a specific cache location in the N-level cache, the memory data inflow being caused by execution of a first thread of the entity by a first processing unit of the plurality of processing units; as well as at least based on having caused the memory data inflow to the particular cache location in the N-level cache to be logged in the replayable trace of the entity, subsequently causing one or more N-1 level cache coherence protocol (CCP) transitions between the two or more N-1 level caches to be logged in the replayable trace of the entity, based on execution of a second thread of the entity by a second processing unit among the plurality of processing units, the N-1 level CCP transitions being caused by access of the particular cache location by the second processing unit among the plurality of processing units.
16. The method of claim 15, wherein the computing device further comprises: (iv) a plurality of N-2 level caches, and wherein each N-1 level cache is associated with two or more N-2 level caches of the plurality of N-2 level caches, and the N-1 level cache is configured as a backup storage repository for the two or more N-2 level caches, the method further comprising: One or more N-2 level CCP transitions between the two or more N-2 level caches caused by access of the particular cache location by the two or more processing units of the plurality of processing units are caused to be logged into the replayable trace of the entity.
17. The method according to claim 15, further comprising: Causes one or more CCP transformations based on the activity of a single processing unit to be logged.
18. A method according to claim 15, wherein at least the N-level cache includes one or more accounting bits, and the one or more accounting bits identify one or more of the following: (i) whether the value at the particular cache location has been logged, or (ii) whether the value at the particular cache location should be logged, and wherein the method also includes setting at least one accounting bit in the N-level cache based on communications from one or more N-1 level caches in the N-1 level cache.
19. The method of claim 15, wherein at least the N-level cache includes one or more accounting bits, the one or more accounting bits identifying one or more of: (i) whether the value at the particular cache location has been logged, or (ii) whether the value at the particular cache location should be logged, and wherein the method further comprises using the one or more accounting bits to avoid logging memory data influx caused by speculative execution of instructions.
20. A computer program product comprising one or more hardware storage devices having computer executable instructions stored thereon, the computer executable instructions being executed by a computing device comprising (i) a plurality of processing units, (ii) a plurality of N-1 level caches, and (iii) a level N cache associated with two or more of the plurality of N-1 level caches and configured as a backing store for the two or more N-1 level caches, the computer executable instructions being executed to cause the computing device to at least perform the following as part of recording execution of an entity executing parallel threads across two or more of the plurality of processing units: causing the memory data inflow to the specific cache location in the N-level cache to be logged in a replayable trace of the entity based at least on detecting a memory data inflow from a specific memory location in system memory to a specific cache location in the N-level cache, the memory data inflow caused by execution of a first thread of the entity by a first processing unit of the plurality of processing units; and at least based on having caused the memory data inflow to the particular cache location in the N-level cache to be logged in the replayable trace of the entity, subsequently causing one or more N-1 level cache coherence protocol (CCP) transitions between the two or more N-1 level caches to be logged in the replayable trace of the entity, based on execution of a second thread of the entity by a second processing unit among the plurality of processing units, the N-1 level CCP transitions being caused by access of the particular cache location by the second processing unit among the plurality of processing units.
