INVALIDATION OF AN IMPLEMENTATION ENTRY IN A MULTITHREAD DATA PROCESSING SYSTEM

Abstract

A method for invalidating translation entries without deadlock in a multithreaded data processing system including a plurality of processor cores (200), the method comprising: Generating a request to invalidate implementations by executing a corresponding instruction to invalidate implementations by an initiating hardware thread from a plurality of concurrently executing hardware threads of an initiating processor core from a plurality of processor cores, Stopping (502) a propagation of instructions in the initiating hardware thread that follow the request to invalidate translations in the program order until an acknowledgment signal is received (508) that confirms completion of processing of the request to invalidate translations at the initiating processor core, Receiving, in a shared queue (320), memory-modifying requests from the plurality of concurrently executing hardware threads of the initiating processor core, wherein the plurality of memory-modifying requests includes a request to invalidate translations from an initiating hardware thread of the plurality of hardware threads, in response to receiving the request to invalidate translations in the shared queue, removing the request to invalidate translations from the shared queue and buffering (604) the request to invalidate translations in sidecar logic (322), the sidecar logic including a plurality of sidecars, each of which is connected to one of the plurality of concurrently executing hardware threads and configured to buffer only a single request to invalidate translations generated by the corresponding hardware thread; while the request to invalidate translations is buffered in the sidecar logic, broadcasting the request to invalidate translations through the sidecar logic such that the request to invalidate translations is received and processed by the plurality of processor cores; in response to an acknowledgment of completion of processing of the invalidate translation request by the initiating processor core, removing the invalidate translation request from the sidecar by the sidecar logic; and Ensuring completion of processing of the request to invalidate translations on all of the plurality of processor cores by a broadcast synchronization request.

Description

BACKGROUND OF THE INVENTION

[0001] The present invention relates generally to data processing and, more particularly, to invalidating translation entries in a multithreaded data processing system.

[0002] A conventional multiprocessor (MP) computer system comprises multiple processing units (each of which may contain one or more processor cores and their various caches), input / output (I / O) units, and data storage, which may include both system memory (which may be volatile or non-volatile) and non-volatile mass storage. To provide sufficient addresses for memory-oriented I / O operations and the data and instructions used by the operating system and application software, MP computer systems typically point to an effective address space that contains a much larger number of effective addresses than the number of physical memory locations in the memory-oriented I / O units and system memory.Therefore, performing memory-oriented I / O or accessing system memory requires a processor core in a computer system that uses effective addressing to translate an effective address into a real address assigned to a specific I / O unit or physical location in system memory.

[0003] In the POWER™ RISC architecture, the effective address space is partitioned into a number of uniformly sized memory pages, each with a corresponding address descriptor called a page table entry (PTE). The PTE corresponding to a particular memory page contains the basic effective address of the memory page and the corresponding basic real page of the page frame, allowing a processor core to translate any effective address in the memory page into a real page in system memory. The PTEs created in system memory by the operating system and / or hypervisor software are collected in a page frame table.

[0004] To accelerate the translation of effective addresses to real addresses during the processing of memory-oriented I / O and memory access instructions (hereinafter referred to collectively as "memory-referenced instructions"), a conventional processor core often uses, among other translation structures, a cache called a translation buffer (TLB) to buffer recently accessed PTEs within the processor core. As data is moved into and out of physical locations in system memory (e.g., in response to the invocation of a new process or a context switch), the entries in the TLB must naturally be updated to reflect the presence of the new data, and the TLB entries associated with data that has been removed from system memory (e.g.,have been swapped out to non-volatile mass storage) must be invalidated. In many conventional processors, such as those in the POWER™ processor line from IBM Corporation, TLB entry invalidation is the responsibility of software and is implemented through the execution of an explicit TLB entry invalidation instruction (e.g., TLBIE in the POWER™ instruction set architecture (ISA)).

[0005] In MP computer systems, invalidating a PTE cached in a processor core's TLB is complicated by the fact that every other processor core has its own respective TLB, which may also cache a copy of the target PTE. To maintain a consistent view of system memory across all processor cores, invalidating a PTE in one processor core requires invalidating the same PTE, if present, in the TLBs of all other processor cores. In many conventional MP computer systems, invalidating a PTE in all processor cores in the system is implemented by executing a TLB invalidate instruction on an initiating processor core and broadcasting a TLB invalidate request from the initiating processor core to every other processor core in the system. The instruction (orThe instructions (if multiple PTEs need to be invalidated) for invalidating a TLB entry may be followed in the instruction sequence of the initiating processor core by one or more synchronization instructions that guarantee that the invalidation of the TLB entry has been executed by all processor cores.

[0006] In conventional MP computer systems, the TLB invalidate instruction and associated synchronization instructions are strictly serialized. This means that the hardware thread of the initiating processor core containing the TLB invalidate instruction must complete processing of each instruction (e.g., by broadcasting the TLB invalidate request to other processor cores) before continuing execution with the next hardware thread instruction. As a result of this serialization, a significant performance penalty occurs, at least for the hardware thread of the initiating processor core containing the TLB invalidate instruction, especially when the hardware thread contains multiple TLB invalidate instructions.

[0007] In multithreaded processing units, it is common for at least some of the processing unit's queues, buffers, and other memory resources to be shared by multiple hardware threads. The strict serialization of the TLB invalidate instruction and the associated synchronization instructions can cause certain requests associated with the TLB invalidate sequence to block in these shared resources, for example, while waiting for confirmation of request processing by other processor cores. If not handled properly, such blocking can cause high latency and / or deadlocks for other hardware threads sharing the memory resources.

[0008] In view of the foregoing, the present invention recognizes that it would be useful and desirable to provide an improved method for maintaining coherence of PTEs in a multithreaded computer system.

[0009] US 6 119 204 A relates to a method for maintaining the coherence of the translation lookaside buffer (TLB) in a data processing system having at least a first processor and a second processor, the method comprising: in response to detection by the second processor of a request to invalidate a TLB entry, marking at least one memory reference instruction being processed by the second processor and invalidating a TLB entry in the second processor; in response to receipt of a synchronization request at the second processor, indicating to the first processor that the second processor has invalidated the TLB entry when the second processor has completed processing of the marked instruction;and continuing to process instructions in the second processor during an interval between receiving the synchronization request and indicating to the first processor that the second processor has invalidated the TLB entry, wherein continuing to process instructions includes fetching instructions for processing.; SUMMARY

[0010] The invention relates to a method according to claim 1, a processing unit according to claim 7, a data processing system according to claim 8 and a design structure according to claim 9. The embodiments of the invention are specified in dependent claims.

[0011] In a multithreaded computing system with a plurality of processor cores, memory-modifying requests from a plurality of concurrently executing hardware threads are received in a shared queue. The memory-modifying requests include a request to invalidate translations from an initiating hardware thread. The request to invalidate translations is removed from the shared queue and buffered in sidecar logic in one of a plurality of sidecars, each associated with a respective one of the plurality of hardware threads. While the request to invalidate translations is buffered in the sidecar, the sidecar logic broadcasts the request to invalidate translations so that it is received and processed by the plurality of processor cores.In response to an acknowledgement of completion of processing of the invalidate translation request by the initiating processor core, the sidecar logic removes the invalidate translation request from the sidecar. Completion of processing of the invalidate translation request on all of the plurality of processor cores is ensured by a broadcast synchronization request.

[0012] In one example, the order of subsequent memory-referencing instructions is specified with reference to the broadcast synchronization request by executing an additional synchronization instruction (e.g., HWSYNC). BRIEF DESCRIPTION OF THE DIFFERENT VIEWS OF THE DRAWINGS Fig. 1 is a high-level block diagram of an exemplary data processing system; Fig. 2 is a more detailed block diagram of an exemplary processing unit; Fig. Figure 3 is a detailed block diagram of a processor core and a lower-level cache memory; Fig. 4A is a first example instruction sequence for invalidating a translation entry; Fig. 4B is a second example instruction sequence for invalidating a translation entry; Fig. 5 is a high-level logic flow diagram of an example method by which a processor core of a multiprocessor data processing system processes an instruction to invalidate a translation entry; Fig. 6 is a high-level logic flow diagram of an example method by which sidecar logic of a processing unit processes a request to invalidate a translation entry; Fig. 7 is an overview-level logic flow diagram for an example method by which a processing unit snooper services requests to invalidate a translation entry and requests for translation synchronization; Fig. 8 is a high-level logic flow diagram of an exemplary method by which an arbiter of a processing unit processes a request to invalidate a translation entry, according to one embodiment; Fig. 9 is a high-level logic flow diagram of an example method by which a processor core translation sequencer processes a request to invalidate a translation entry; Fig. 10 is a high-level logic flow diagram for an example method by which a processing unit's memory queue processes a request to stop invalidating translations; Fig. 11 is a high-level logic flow diagram of an example method by which a processor core processes a translation synchronization instruction; Fig. 12 is a high-level logic flow diagram of an example method by which sidecar logic of a processing unit processes a translation synchronization request; Fig. 13 is a high-level logic flow diagram of an example method by which a processor core processes a page table synchronization instruction; Fig. 14 is an overview-level logic flow diagram of an example method by which a processor core processes a page table synchronization request; Fig. 15 is a high-level logic flow diagram for an example method by which a processing unit's snooper logic services requests to invalidate translations, requests to stop invalidating translations, and page table synchronization requests; and Fig. Figure 16 is a data flow diagram illustrating a development process. DETAILED DESCRIPTION

[0013] With reference to the following figures, in which like reference numerals refer to like and corresponding parts throughout, and in particular with reference to Fig. Figure 1 illustrates a high-level flowchart of an exemplary data processing system 100 according to one embodiment. In the illustrated embodiment, the data processing system 100 is a cache-coherent symmetric multiprocessor (SMP) data processing system having multiple processing nodes 102 for processing data and instructions. The processing nodes 102 are connected to a system interconnect 110 for communicating address, data, and control information. The system interconnect 110 may be implemented, for example, as a bus interconnect, a switch interconnect, or a hybrid interconnect.

[0014] In the illustrated embodiment, each processing node 102 is implemented as a multi-chip module (MCM) containing four processing units 104a through 104d, each preferably implemented as a single integrated circuit. The processing units 104 in each processing node 102 are interconnected for data exchange with each other and with the system interconnect 110 by a local interconnect 114, which, like the system interconnect 110, may be implemented, for example, with one or more buses and / or switches. The system interconnect 110 and the local interconnects 114 together form a system structure.

[0015] As explained in more detail below with reference to Fig. 2, the processing units 104 each include a memory controller 106 connected to the local interconnect 114 to provide an interface to a respective system memory 108. Access to, caching, and modification of data and instructions in the system memories 108 may generally be performed by a processor core in each processing unit 104 of each processing node 102 in the data processing system 100. The system memories 108 thus form the lowest level of memory in the distributed shared memory system of the data processing system 100. In alternative embodiments, one or more memory controllers 106 (and system memories 108) may be connected to the system interconnect 110 instead of to a local interconnect 114.

[0016] It will be clear to those skilled in the art that the SMP data processing system 100 of Fig. 1 may contain many additional components not illustrated, such as connecting bridges, non-volatile memory, connectors for connecting to networks or connected units, etc. Since such additional components are not necessary for understanding the described embodiments, they are not shown in Fig. 1 is not illustrated or further discussed herein. It should also be understood that the extensions described herein are applicable to data processing systems of diverse architectures and are in no way limited to the Fig. 1 illustrated generalized data processing system architecture.

[0017] With reference to Fig. Figure 2 shows a more detailed block diagram of an exemplary processing unit 104 according to an illustrative embodiment. In the illustrated embodiment, each processing unit 104 is an integrated circuit having one or more processor cores 200 for processing instructions and data. In a preferred embodiment, each processor core 200 supports simultaneous multithreading (SMT) and is thus capable of independently executing multiple hardware threads for simultaneous execution.

[0018] The operation of each processor core 200 is supported by a multi-level memory hierarchy having, at the lowest level, a shared system memory 108 accessed via an integrated memory controller 106. As illustrated, the shared system memory 108 stores a page frame table 220 containing a plurality of page table entries (PTEs) 222 for performing effective-to-real address translation to enable access to memory locations in the system memory 108. At the upper levels, the multi-level memory hierarchy includes one or more levels of cache memory, which in the illustrated embodiment includes a level 1 (L1) write-through cache 302 (see Fig. 3) in and specific to each processor core 200 and a level 2 (L2) write-back cache 230 for each processor core 200. Although the illustrated cache hierarchies include only two levels of cache, those skilled in the art will appreciate that alternative embodiments may include additional levels (L3, L4, etc.) of on-chip or off-chip, dedicated or shared, integrated or lookaside cache, which may be fully inclusive, partially inclusive, or non-inclusive of the contents of the upper levels of cache.

[0019] Each processing unit 104 further includes an integrated and distributed fabric controller 216, which is responsible for controlling the flow of operations on the system fabric comprising the local interconnect 114 and the system interconnect 110, and for implementing the coherency data exchange required to implement the selected cache coherency protocol. The processing unit 104 further includes an integrated I / O (input / output) controller 214, which supports the connection of one or more I / O devices (not shown).

[0020] With reference to Fig. 3 illustrates a detailed block diagram of an exemplary embodiment of a processor core 200 and its associated L2 cache 230, according to one embodiment.

[0021] In the illustrated embodiment, processor core 200 includes one or more execution units 300 that execute instructions from multiple hardware threads for simultaneous execution. The instructions may include, for example, arithmetic instructions, logical instructions, and memory reference instructions, as well as translation lookaside buffer invalidate entry (hereinafter referred to by the POWER™ ISA mnemonic TLBIE (Translation Lookaside Buffer Invalidate Entry) instructions and associated synchronization instructions. The execution unit(s) 300 may generally execute instructions from a hardware thread in any order, as long as data dependencies and any explicit ordering requirements required by the synchronization instructions are respected.

[0022] The processor core 200 also includes a memory management unit (MMU) 308, which is responsible for translating effective target addresses determined by the execution of memory-referenced instructions in the execution unit(s) 300 into real addresses. The MMU 308 performs effective-to-real address translation by referencing one or more translation structures 310, such as a translation address buffer (TLB), a block address table (BAT), a segment translation buffer (SLB), etc. The number and type of these translation structures vary depending on the implementation and architecture. When present, the TLB reduces the latency associated with effective-to-real address translation by caching the PTEs 222 retrieved from the page frame table 220. One of theThe translation sequencer 312 associated with the translation structures 310 processes entries for invalidating effective-to-real address translations contained in the translation structure(s) 310 and manages such invalidation relative to memory-referenced instructions currently executing in the processor core 200.

[0023] Processor core 200 additionally includes various memory devices shared by the multiple hardware threads supported by processor core 200. The memory devices shared by the multiple hardware threads include an L1 store queue 304 that temporarily buffers store and synchronization requests generated by the execution of corresponding store and synchronization instructions from execution unit(s) 300. Because L1 cache 302 is a write-through cache, meaning that coherency is entirely determined at a lower level of the cache hierarchy (e.g., at L2 cache 230), requests flow through L1 STQ 304 and are then passed to L2 cache 230 for processing via bus 318.The memory devices of processor core 200, which are shared by multiple hardware threads, additionally include a load miss queue (LMQ) 306 that temporarily buffers load requests that have failed in the L1 cache 302. Because such load requests have not yet been satisfied, they tend to hit the wrong memory page if the address translation entry used to obtain the real target addresses of the load requests is invalidated before the load requests are satisfied. Consequently, if a PTE or other translation entry is to be invalidated, all load requests in the LMQ 306 that depend on that translation entry must be evicted from the LMQ 306 and satisfied before the effective address translated by the relevant translation entry can be reassigned.

[0024] With further reference to Fig. 3, the L2 cache 230 includes a cache array 332 and an L2 directory 334 of the contents of the cache array 332. Assuming that the cache array 332 and the L2 directory 334 are configured associatively as usual, memory locations in the system memories 108 are mapped to particular congruence classes in the cache array 332 using predetermined index bits in the (real) addresses of the system memory. The specific memory blocks stored in the cache lines of the cache array 332 are recorded in the L2 directory 334, which contains a directory entry for each cache line. Although this Fig. 3, it will be apparent to those skilled in the art that each directory entry in cache directory 334 includes various fields, for example, a tag field that identifies the physical address of the memory block contained in the corresponding cache line of cache array 332, a state field that indicates the coherency state of the cache line, an LRU (Least Recently Used) field that indicates a replacement order for the cache line with respect to other cache lines in the same congruence class, and inclusivity bits that indicate whether the memory block is included in the associated L1 cache 302.

[0025] The L2 cache 230 also includes an L2 STQ 320, which receives memory-modifying requests and synchronization requests from the L1 STQ 304 via an interface 318 and buffers such requests. Note that the L2 STQ 320 is a unified memory queue that buffers requests for all hardware threads of the associated processor core 200. As a result, all thread memory requests, TLBIE requests, and associated synchronization requests flow through the L2 STQ 320. Although the L2 STQ 320 contains multiple entries in most embodiments, the L2 STQ 320 must operate deadlock-free regardless of depth (i.e., even when implemented as a single-entry queue).To this end, the L2 STQ 320 is connected via an interface 321 to an associated sidecar logic 322 that includes one request-buffering entry 324 (referred to herein as a "sidecar") per hardware thread, supported by the associated processor core 200. As such, the number of sidecars 324 is unrelated to the number of entries in the L2 STQ 320. As further described herein, the use of sidecars 324 allows for the potential creation of a deadlock for requests to be removed from the L2 STQ 320, so that no deadlocks occur during the invalidation of a translation entry.

[0026] The L2 cache 230 further includes forwarding / response logic 336 that receives local load and store requests initiated by the associated processor core 200 via buses 327 and 328, respectively, and remote requests monitored on the local interconnect 114 via bus 329. Such requests, including local and remote load requests, store requests, TLBIE requests, and associated synchronization requests, are processed by the forwarding / response logic 336 and then forwarded to the appropriate state machines for servicing.

[0027] In the illustrated embodiment, the state machines implemented in the L2 cache 230 to service requests include a plurality of read-claim (RC) machines 342 that independently and concurrently service load (LD) and store (ST) requests received from the associated processor core 200. To service remote memory access requests originating from processor cores 200 other than the associated processor core 200, the L2 cache 230 also includes a plurality of snoop (SN) machines 344. Each snoop machine 344 can independently and concurrently service a remote memory access request monitored by the local interconnect 114.As will be appreciated, servicing memory access requests by RC engines 342 may require replacing or invalidating memory blocks in cache array 332 (and L1 cache 203). Accordingly, L2 cache 230 also includes CO (castout) engines 340 that manage the eviction and write-back of memory blocks from cache array 332.

[0028] In the illustrated embodiment, L2 cache 230 additionally includes multiple translation snoop (TSN) engines 346 used to service TLBIE requests and associated synchronization requests. It should be understood that, in some embodiments, TSN engines 346 may be implemented in a different subunit of a processing unit 104, for example, a non-cacheable unit (NCU) (not illustrated) that services non-cacheable memory access operations. In at least one embodiment, the same number of TSN engines 346 are implemented at each L2 cache 230 to facilitate implementation of a consensus protocol (as discussed below) that coordinates the processing of multiple concurrent TLBIE requests in data processing system 100.

[0029] The TSN engines 346 are all connected to an arbiter 348, which selects requests processed by the TSN engines 346 for transmission to the translation sequencer 312 in the processor core 200 via a bus 350. In at least some embodiments, the bus 350 is implemented as a unified bus that not only transmits requests from the TSN engines 346, but also returns data from the L2 cache 230, as well as other operations, to the processor core 200. It should be noted that the translation sequencer 312 must accept requests from the arbiter 348 in a non-blocking manner to avoid deadlock.

[0030] With reference to Fig. 4A illustrates a first exemplary instruction sequence 400 for invalidating a translation entry that may be executed by a processor core 200 of a data processing system 100 according to one embodiment. The purpose of the instruction sequence 400 is to: (a) disable a translation entry (e.g., PTE 222) in the page frame table 220 so that the translation entry cannot be reloaded by any MMU 308 of the data processing system 100, (b) invalidate any copy of the translation entry (or another translation entry that translates the same effective address as the translation entry) cached by any processor core 200 in the data processing system 100, and (c) eliminate any outstanding memory access requests that depend on the old translation entry before reallocating the effective address.If the translation was updated before the memory requests that depend on the old translation entry were removed, the memory requests may corrupt the memory page identified by the old translation entry. Likewise, if load requests that depend on the old translation entry and failed in the L1 cache 302 were not satisfied before the translation was reallocated, the load requests would read data from a different memory page than the intended one, thus observing data that should not be visible to the load requests.

[0031] The instruction sequence 400, which may be preceded or followed by any number of instructions, begins with one or more store (ST) instructions 402. Each store instruction 402, when executed, causes a store request to be generated which, when passed to the relevant system memory 108, invalidates a target PTE 222 in the page frame table 220. If the store request invalidates the PTE 222 in the page frame table 220, the MMUs 308 no longer load the invalidated translation from the page frame table 220.

[0032] The one or more store instructions 402 in the instruction sequence 400 are followed by a heavyweight synchronization (i.e., HWSYNC) instruction 404, which is a boundary that ensures that a following TLBIE instruction 406 is not reordered by the processor core 200 so that it executes before any of the store instructions 402. The HWSYNC instruction 404 thus ensures that when a processor core 200 reloads a PTE 222 from the page frame table 220 after the TLBIE instruction 406 has invalidated copies of the PTE 222, the processor core 200 is guaranteed to have honored the invalidation due to a store instruction 402 and therefore will neither use the destination PTE 222 nor reload it into the translation structure(s) 310 until the effective address translated by the destination PTE 222 has been reassigned and set to "valid."

[0033] Following the HWSYNC instruction 404 in the instruction sequence 400 is at least one TLBIE instruction 406 which, when executed, generates a corresponding TLBIE request that invalidates all translation entries that translate the effective destination address of the TLBIE request in all translation structures 310 throughout the data processing system 100. Following the one or more TLBIE instructions 406 in the instruction sequence 400 is a translation synchronization (i.e., TSYNC) instruction 408 which ensures that, before executing thread continuation with subsequent instructions, the TLBIE request generated by the execution of the TLBIE instruction 406 has finished invalidating all translations of the effective destination addresses in all translation structures 310 throughout the data processing system 100, and all previous memory access requests that depend on the now invalidated translation have been eliminated.

[0034] The instruction sequence 400 ends with a second HWSYNC instruction 410, which enforces a boundary that prevents any memory-referencing instructions after the HWSYNC instruction 410 in program order from executing until the processing of the TSYNC instruction 406 is complete. In this way, all more recent memory-referencing instructions that require a translation of the effective destination address of the TLBIE request will receive a new translation in place of the old translation invalidated by the TLBIE request. Note that the HWSYNC instruction 410 has no function directly related to invalidating the destination PTE 222 in the page frame table, invalidating translation entries in the translation structures 310, or eliminating memory-referencing instructions that depend on the old translation.

[0035] To assist in understanding the inventions disclosed herein, the history of a TLBIE instruction 406 and the TLBIE request generated therefrom is described from beginning to end with reference to the Fig. 5 to 10. The Fig. 11 and Fig. 12 additionally illustrate the history of the TSYNC instruction 408 and the corresponding TSYNC request, which ensure that the invalidation requested by the TLBIE request has been completed on all monitoring processor cores 200.

[0036] With first reference to Fig. 5 illustrates a high-level logical flow diagram for an exemplary method by which an initiating processor core 200 of a multiprocessor computing system 100 processes an instruction to invalidate a translation entry (e.g., TLBIE) according to one embodiment. The illustrated process represents processing executed in a single hardware thread, which means that multiple such processes may be executed concurrently (i.e., in parallel) on a single processor core 200, and further, that multiple such processes may be executed concurrently on various different processor cores 200 throughout the computing system 100. As a result, multiple different address translation entries in the various processor cores 200 of the computing system 100 may be invalidated concurrently by different initiating hardware threads.

[0037] The illustrated process begins at block 500 and continues with block 501, which illustrates the execution of a TLBIE instruction 406 in an instruction sequence 400 by the execution unit(s) 300 of a processor core 200. The execution of the TLBIE instruction 406 determines an effective target address for which all translation entries buffered in the translation structure(s) 310 throughout the data processing system 100 must be invalidated. In response to the execution of the TLBIE instruction 406, the processor core 200 stops the propagation of all further instructions in the initiating hardware thread because, in the exemplary embodiment of Fig. 3, the sidecar logic 322 includes only a single sidecar 324 per thread, meaning that a maximum of one TLBIE request per thread can be active at a time. In further embodiments that have multiple sidecars 324 per thread, multiple concurrently active TLBIE requests per thread can be supported.

[0038] At block 504, a TLBIE request corresponding to TLBIE instruction 406 is generated and issued to L1 STQ 304. The TLBIE request may include, for example, a translation type that specifies the type of request (i.e., TLBIE), the effective address for which cached translations must be invalidated, and an indication of the initiating processor core 200 and the hardware thread that issued the TLBIE request. Processing of requests in L1 STQ 304 continues, and the TLBIE request is finally moved from L1 STQ 304 to L2 STQ 320 via bus 318 at block 506.The process then continues to block 508, which illustrates that the initiating processor core 200 continues to withhold the dispatch of instructions in the initiating hardware thread until it receives a TLBCMPLT_ACK signal from the memory subsystem over bus 325, indicating that the initiating processor core 200 has completed processing of the TLBIE request. (The generation of the TLBCMPLT_ACK signal is described below with reference to block 1010 of FIG. Fig. 10.) It should also be noted that because the initiating thread pauses instruction propagation, there can be no conflict for the initiating thread's sidecar 324 from a TSYNC request corresponding to the TSYNC instruction 408, since only one of the two request types can be present in the L2 STQ 320 and the sidecar logic 322 for any given thread.

[0039] In response to a determination at block 508 that a TLBCMPLT_ACK signal has been received, the process continues from block 508 to block 510, which illustrates that the processor core 200 resumes propagating instructions in the initiating thread; thus, the release of the thread at block 510 enables processing of the TSYNC instruction 408 (which is the next instruction in the instruction sequence 400) to continue, as described below with reference to Fig. 11 begins. The process of Fig. 5 then ends at block 512.

[0040] With reference to Fig. 6 illustrates a high-level logic flow diagram for an exemplary method by which the sidecar logic 322 of an L2 cache 230 processes a request to invalidate a translation entry (e.g., TLBIE) from a hardware thread of the associated processor core 200, according to one embodiment. The process of Fig. 6 is based on processing per thread.

[0041] The process of Fig. 6 begins at block 600 and then continues to block 602, which illustrates sidecar logic 322 determining whether a TLBIE request from a hardware thread of the associated processor core 200 has been loaded into the L2 STQ 320. If not, the process begins again at block 602. However, in response to a determination that a TLBIE request from a hardware thread of the associated processor core 200 has been loaded into the L2 STQ 320, the sidecar logic 322 removes the TLBIE request from the L2 STQ 320 and moves the TLBIE request via interface 321 to the sidecar 324 corresponding to the initiating thread (block 604). Removing the TLBIE request from the L2 STQ 320 ensures that no deadlock occurs due to an inability of the L2 STQ 320 to receive incoming requests from the associated processor core 200 and allows such requests to flow through the L2 STQ 320.

[0042] At block 606, sidecar 324 participates in a consensus protocol (which may be conventional) via interface 326 and local interconnect 114 to ensure that one (and only one) TSN engine 346 in each and all L2 caches 230 receives the TLBIE request. Furthermore, the consensus protocol ensures that the various TSN engines 346 do not take action to service the TLBIE request until all corresponding TSN engines 346 have received the TLBIE request. Afterward, the process returns to block 602, as described.

[0043] With reference to Fig. Figure 7 illustrates an overview-level logical flowchart for an exemplary method by which TSN engines 346 process TLBIE requests and TSYNC requests, according to one embodiment. The illustrated process is executed independently and concurrently for each TSN engine 346.

[0044] The process begins at block 700 and then continues with blocks 702 and 720. Block 702 and a subsequent block 704 illustrate that, in response to receiving a TLBIE request via the consensus protocol, a TSN engine 346 buffers the TLBIE request and enters a TLBIE_active state. The TLBIE request received at block 606 by Fig. 6 broadcast via system structure 110, 114 to the L2 cache 230 of the initiating processor core 200 and those of all other processor cores 200 of data processing system 100 is received at the L2 cache 230 via interface 329, processed by forward / response logic 336, and then assigned to the TSN engine 346. As noted above, in a preferred embodiment, the consensus protocol enforces the condition that the TLBIE request is assigned to a TSN engine 346 in an L2 cache 230 only if a TSN engine 346 is equally assigned to the TLBIE request by all other L2 caches 230. The TSN engine 346, which assumes the TLBIE_active state, informs the associated arbiter 348 that a TLBIE request is ready for processing, as further described below with reference to block 802 of Fig. 8 described.

[0045] Block 706 illustrates that the TSN engine 346 remains in the TLBIE_active state until the processing of the TLBIE request by the associated processor core 200 (i.e., invalidating the relevant translation entries in the translation structure(s) 310 and removing relevant memory-referencing requests from the processor core 200) is completed, as indicated by receiving a TLBCMPLT_ACK signal over a signal line 330. In response to receiving the TLBCMPLT_ACK signal, the TLBIE_active state is reset, and the TSN engine 346 is enabled for reallocation (block 708). Thereafter, the process returns from Fig. 7 from block 708 back to block 702, which has been described.

[0046] Referring now to blocks 720 through 724, a TSN engine 346 determines at block 720 whether it is in the TLBIE_active state created at block 704. If not, the process begins again at block 720. However, if the TSN engine 346 is in the TLBIE_active state created at block 704, the TSN engine 346 monitors to determine whether a TSYNC request has been detected for the initiating hardware thread of the TLBIE request (block 722). If no TSYNC request has been detected, the process continues repeating at blocks 720 through 722. However, if the TSN engine 346 is in the TLBIE_active state in response to detecting a TSYNC request from the initiating hardware thread in response to its TLBIE request, the TSN engine 346 provides a coherency "Retry" response via the system structure 110, 114, as indicated at block 724.As described below with reference to block 1208 of . Fig. As discussed in Figure 12, a coherency "retry" response from any TSN snooper 346 handling the TLBIE request for the initiating hardware thread forces the L2 source cache 230 to reissue the TSYNC request and prevents the initiating hardware thread from proceeding with the HWSYNC instruction 410 until the TSYNC request completes without a coherency "retry" response. The TSYNC request completes without a coherency "retry" response when all other processor cores 200 besides the initiating processor core 200 have completed processing the TLBIE request. (The TSYNC request is not issued by the initiating processor core 200 until the processing of the TLBIE request is complete, as described above with reference to block 508 of Fig. 5.)

[0047] With reference to Fig. 8 illustrates a high-level logic flow diagram for an exemplary method by which an arbiter 348 of the L2 cache 230 processes a TLBIE request, according to one embodiment. The process begins at block 800 and then continues to block 802, which illustrates the arbiter 348 determining whether one of the TSN engines 346 is in the TLBIE_active state. If not, the process is Fig. 8 is repeated again at block 802. However, in response to determining that one or more of the TSN engines 346 are in the TLBIE_active state, the arbiter 348 selects one of the TSN engines in the TLBIE_active state whose TLBIE request was not recently forwarded and transmits the TLBIE request via the interface 350 to the translation sequencer 312 of the associated processor core 200 (block 804). To avoid deadlock, the translation sequencer 312 is configured to receive TLBIE requests within a specified time and not to indefinitely delay accepting a TLBIE request.

[0048] The process continues from block 804 to block 806, which represents the arbiter 348 waiting to receive a TLBCMPLT_ACK message indicating that the associated processor core 200, in response to the TLBIE request, has invalidated the relevant translation entry(s) in the translation structure(s) 310 and eliminated the relevant memory-pointing requests whose target addresses may have been translated by the invalidated translation entries. Thus, at block 806, the arbiter 348 waits for a TLBCMPLT_ACK message, just as the initiating thread (block 508) and a TSN engine 346 in each of the L2 caches 230 (block 706) do. In response to receiving a TLBCMPLT_ACK message at block 806, the process returns to block 802, which has been described.It should be noted that by the time the process returns to block 802, the previously selected TSN engine 346 is no longer in the TLBIE_active state for the already processed TLBIE request because the TLBIE_active state has been reset, as illustrated at blocks 706 through 708, before the process returns to block 802.

[0049] The process of Fig. 8 (and in particular blocks 802 and 806) ensures that only one TLBIE request at a time is processed by the processor core 200. The serial processing of TLBIE requests by the processor core 200 eliminates the need to tag TLBCMPLT_ACK messages to associate them with TLBIE requests and simplifies instruction marking mechanisms, as described below with reference to Fig. 9. However, those skilled in the art will recognize that in other embodiments, processor core 200 may be configured to serve multiple TLBIE requests simultaneously with some additional complexity.

[0050] With reference to Fig. 9 illustrates a high-level logic flow diagram for an exemplary method by which a translation sequencer 312 of an initiating or monitoring processor core 200 processes a TLBIE request according to one embodiment. Fig. 9 begins at block 900 and then continues to block 902, which illustrates the translation sequencer 312 waiting for the arbiter 348 to receive a TLBIE request forward, as described above with reference to block 804 of Fig. 8. In response to receiving a TLBIE request, the translation sequencer 312 invalidates one or more translation entries (e.g., PTEs or other translation entries) in the translation structure(s) 310 that translate the effective target address of the TLBIE request (block 904). In addition, at block 906, the translation sequencer 312 marks any memory-referenced requests that need to be removed from the processor core 200.

[0051] In a less precise embodiment, at block 906, the translation sequencer 312 marks all memory-referencing requests from all hardware threads in the processor core 200 whose target addresses have been translated, assuming that for all such memory-referencing requests, the target addresses may have been translated by a translation entry(s) invalidated by the TLBIE request received at block 902. Thus, in this embodiment, the marked memory-referencing requests would include all store requests in the L1 STQ 304 and all load requests in the LMQ 306. This embodiment advantageously eliminates the need to translate comparators for all entries of the L1 STQ 302 and the LMQ 306, but may result in higher latency due to long clear times.

[0052] A more precise embodiment implements comparators for all entries of L1 STQ 304 and LMQ 306. In this embodiment, each comparator compares a subset of effective address bits specified by the TLBIE request (and which are not translated by MMU 308) with corresponding real address bits of the real target address specified in the associated entry of L1 STQ 304 or LMQ 306. Only memory-pointing requests for which a match is detected by the comparators are marked by translation sequencer 312. Thus, this more precise embodiment reduces the number of marked memory-pointing requests at the expense of additional comparators.

[0053] In some implementations of the less precise and more precise marking embodiments, the marking applied by the translation sequencer 312 is applied only to requests in the processor core 200 and persists only until the marked results are eliminated from the processor core 200. In such implementations, the L2 cache 230 may revert to the pessimistic assumption that for all store requests currently executing in the L2 cache 230, their addresses may have been translated by the translation entry invalidated by the TLBIE request and require that all such store requests be eliminated before processing store requests using a new translation of the effective target address of the TLBIE request.In further implementations, the more precise marking applied by the translation sequencer 312 may also extend to memory requests currently executing in the L2 cache 230.

[0054] The process of Fig. 9 continues from block 906 to block 908, which illustrates the translation sequencer 312 waiting for the requests marked in block 906 to be eliminated from the processor core 200. In particular, the translation sequencer 312 waits until all load requests marked at block 906 have had the requested data returned to the processor core 200 and all store requests marked at block 906 have been issued to the L2 STQ 320. In response to all marked requests being eliminated from the processor core 200, the translation sequencer 312 inserts a TLBCMPLT request into the L2 STQ 320 to indicate that the servicing of the TLBIE request by the translation sequencer 312 is complete (block 910). The process of Fig. 9 then ends at block 912.

[0055] With reference to Fig. 10 illustrates an overview-level logical flow diagram for an example method by which an L2 STQ 320 processes a TLBCMPLT request according to one embodiment.

[0056] The process of Fig. 10 begins at block 1000 and then continues with block 1002, which illustrates the L2 STQ 320 receiving and queuing in one of its entries a TLBCMPLT request issued by the associated processor core 200, as described above with reference to block 910 of Fig. 9. As illustrated at block 1004, after receiving the TLBCMPLT request, the L2 STQ 320 waits until all older memory requests from all hardware threads are eliminated from the L2 STQ 320. Once all of the older memory requests are eliminated from the L2 STQ 320, the process continues from block 1004 to block 1006, which illustrates the L2 STQ 320 transmitting a TLBCMPLT_ACK signal over bus 330 to the TSN engine 346 that issued the TLBIE request and to the arbiter 348, which, as described above with reference to blocks 706 and 806, await acknowledgment of completion of processing of the TLBIE request.

[0057] At block 1008, the L2 STQ 320 determines whether the associated processor core 200 is the initiating processor core of the TLBIE request whose completion is signaled by the TLBCMPLT request, for example, by examining the thread-identifying information in the TLBCMPLT request. If not (meaning that the process is executing at the L2 cache 320 associated with a monitoring processor core 200), processing of the TLBIE request is complete, and the L2 STQ 320 removes the TLBCMPLT request from the L2 STQ 320 (block 1014). The process then terminates at block 1016.

[0058] On the other hand, if the L2 cache 320 determines at block 1008 that the associated processor core 200 is the initiating processor core 200 of a TLBIE request buffered in the sidecar logic 322, the process continues from block 1008 to block 1009, which illustrates the L2 STQ 320 asserting the TLBCMPLT_ACK signal over bus 330 to the sidecar logic 322. In response to receiving the TLBCMPLT_ACK signal, the sidecar logic 322 asserts a TLBCMPLT_ACK signal over bus 325 to the associated processor core 200. As described above with reference to block 508 of Fig. 5, receipt of the TLBCMPLT_ACK signal enables the initiating thread of the processor core 200 to resume propagation of new instructions (i.e., TSYNC instruction 408, the behavior of which is described with reference to Fig. 11). The relevant sidecar 324 then removes the completed TLBIE request (block 1012), and the process proceeds to blocks 1014 and 1016, which have been described.

[0059] With reference to Fig. 11 illustrates a high-level logic flow diagram for an example method by which a processor core 200 processes a translation synchronization (e.g., TSYNC) instruction, according to one embodiment.

[0060] The illustrated process begins at block 1100 and continues with block 1101, which illustrates the execution of a TSYNC instruction 408 in an instruction sequence 400 by the execution unit(s) 300 of a processor core 200. In response to the execution of a TSYNC instruction 408, the processor core 200 stops the propagation of all subsequent instructions in the hardware thread (block 1102). As noted above, the propagation is stopped because, in the exemplary embodiment of Fig. 3 the sidecar logic 322 contains only a single sidecar 324 per hardware thread of the processor core 200, which means that a maximum of one TLBIE or TSYNC request per thread can be active at any one time.

[0061] At block 1104, a TSYNC request corresponding to TSYNC instruction 408 is generated and issued to L1 STQ 304. The TSYNC request may, for example, include a translation type that includes the type of request (i.e., TSYNC) and an indication of the initiating processor core 200 and the hardware thread that issued the TSYNC request. Processing of requests in L1 STQ 304 continues, and the TSYNC request is eventually moved from L1 STQ 304 to L2 STQ 320 via bus 318, as indicated at block 1106. The process then continues to block 1108, which illustrates that the initiating processor core 200 continues to withhold the propagation of instructions in the initiating hardware thread until it receives a TSYNC_ACK signal from the memory subsystem over bus 325 indicating that the initiating processor core 200 has completed processing of the TSYNC request.(The generation of the TSYNC_ACK signal is described below with reference to block 1210 of . Fig. 12.) Again, it should be noted that because the initiating thread pauses instruction propagation, the initiating thread's sidecar 324 cannot be conflicted by another TLBIE request because, for any given thread, only one of the two request types can be present in the L2 STQ 320 and the sidecar logic 322.

[0062] In response to a determination at block 1108 that a TSYNC_ACK signal has been received, the process continues to block 1110, which illustrates that the processor core 200 resumes propagating instructions in the initiating thread; thus, releasing the thread at block 1110 allows processing of the HWSYNC instruction 410 (which is the next instruction in the instruction sequence 400) to begin. The process of Fig. 11 then ends at block 1112.

[0063] With reference to Fig. 12 illustrates an overview-level logic flow diagram for an exemplary method by which sidecar logic 324 processes a TSYNC request, according to one embodiment. The process begins at block 1200 and then proceeds to block 1202, which illustrates sidecar logic 324 monitoring for a notification via interface 321 that a TSYNC request has been enqueued into L2 STQ 320. In response to receiving a notification via interface 321 that a TSYNC request has been enqueued into L2 STW 320, sidecar logic 322 moves the TSYNC request via interface 321 to the sidecar 324 of the initiating hardware thread (block 1204).In response to receiving the TSYNC request, the sidecar 324 issues the TSYNC request on the system fabric 110, 114 via the interface 326 (block 1206), and thereafter monitors the coherency response to the TSYNC request to determine whether any TSN engine 346 has issued a "retry" coherency response, as previously described with reference to block 724 of FIG. Fig. 7 (block 1208). As noted above, a TSN engine 346 provides a "retry" coherency response if the TSN engine is still in the TLBIE_active state and is waiting for the monitoring processor core 200 to complete processing the previous TLBIE request from the same initiating processor core 200 and hardware threads. It can be noted that by the time a TSYNC request is issued, the TSN engine 346 of the issuing processing unit is no longer in the TLBIE_active state and will not issue a "retry" coherency response because the TLBCMPLT_ACK signal resets the TSN engine 346 of the issuing processor core to an inactive state at block 1006 before the TLBCMPLT_ACK is issued to the initiating processor core 200 at block 1010.Receipt of the TLBCMPLT_ACK signal by processor core 200 causes the initiating processor core 200 to resume propagating instructions after the TLBIE instruction 406, thus executing the TSYNC instruction 408 to generate the TSYNC request. However, the initiating processor core 200 may complete processing the TLBIE request long before the monitoring processor cores 200 have completed invalidating their translation entries and eliminated the memory-referencing instructions that were marked as dependent or potentially dependent on the invalidated translation entries.Accordingly, the TSYNC request ensures that the invalidation of the translation entries and the removal of the memory-pointing instructions that depend on the translation entries invalidated at the monitoring processor cores 200 are completed before the initiating processor core executes the HWSYNC instruction 410.

[0064] Finally, once all monitoring processor cores 200 have completed their processing of the TLBIE request, the TSYNC request completes without a coherency "retry" response. In response to the TSYNC request completing without a coherency "retry" response at block 1208, the sidecar 324 asserts a TSYNC_ACK signal to the initiating processor core 200 over bus 325 (block 1210). As described above with reference to block 1210, in response to receiving the TSYNC_ACK signal, the initiating processor core 200 executes the HWSYNC instruction 410, which completes the initiating thread's de-sequencing requirements with respect to younger memory-referenced instructions. Then, at block 1210, the sidecar 324 removes the TSYNC request (block 1212), and the process returns to block 1202, which has been described.

[0065] After the detailed description of the instruction sequence 400 of Fig. 4A and the associated processing with reference to the Fig. 5 to 12 will be referred to below Fig. 4B, which illustrates an alternative code sequence 420 that reduces the number of instructions, and in particular synchronization instructions, in the sequence for invalidating translations. As shown, the instruction sequence 420 includes one or more store instructions 422 for invalidating PTEs 222 in the page frame table 220, an HWSYNC instruction 424, and one or more TLBIE instructions 426 that invalidate cached translation entries for specified effective addresses in all processor cores 200. Instructions 422 through 426 thus correspond to instructions 402 through 406 of the instruction sequence 400 of Fig. 4A. The instruction sequence 420 additionally contains a PTESYNC instruction 430, which directly follows the TLBIE instruction 426. The PTESYNC instruction 430 combines the work performed by the TSYNC instruction 408 and the HWSYNC instruction 410 of the instruction sequence 400 of Fig. 4A, in a single instruction. That is, the execution of the PTESYNC instruction 430 generates a PTESYNC request that is broadcast to all processing units 104 of the data processing system 100 to ensure system-wide completion of the TLBIE request generated by the TLBIE instruction 426 (as is also done by the TSYNC request generated by the execution of the TSYNC instruction) and to enforce instruction ordering with respect to younger memory-referencing instructions (as is done by the HWSYNC request generated by the execution of the HWSYNC instruction 410).

[0066] Given the similarities between instruction sequences 420 and 400, the processing of instruction sequence 420 is the same as for instruction sequence 400 described in the Fig. 5 to 12, except for the processing related to the PTESYNC request generated by the execution of the PTESYNC instruction 430, which is described below with reference to the Fig. 13 to 15.

[0067] With reference to Fig. 13 illustrates a high-level logic flow diagram for an exemplary method by which a processor core 200 processes a page table synchronization (e.g., PTESYNC) instruction 430, according to one embodiment. As noted above, the PTESYNC instruction 430 and the PTESYNC request generated by its execution serve two functions: to ensure system-wide completion of the TLBIE request(s) generated by the TLBIE instruction(s) 426 and to enforce instruction ordering with respect to younger memory-referenced instructions.

[0068] The illustrated process begins at block 1300 and continues with block 1301, which illustrates a processor core 200 generating a PTESYNC request by executing a PTESYNC instruction 430 in an instruction stream 420 in the execution unit(s) 300. The PTESYNC request may, for example, include a translation type that includes the type of request (i.e., PTESYNC) and an indication of the initiating processor core 200 and the hardware thread that issued the PTESYNC request. In response to the execution of a PTESYNC instruction 430, the processor core 200 stops the propagation of all younger instructions in the initiating hardware thread (block 1302). As noted above, the propagation is stopped because, in the exemplary embodiment of Fig. 3, the sidecar logic 322 includes only a single sidecar 324 per hardware thread of the processor core 200, which means that in this embodiment, a maximum of one TLBIE or PTESYNC request per thread can be active at any one time.

[0069] Following block 1302, the process continues from Fig. 13 continues in parallel with block 1303 and blocks 1304 through 1312. Block 1303 represents the initiating processor core 200 performing the load sequencing function for the PTESYNC request by waiting for all corresponding older load requests from all hardware threads to be eliminated from the LMQ 306 (i.e., those that would be architecturally required for an HWSYNC to receive the requested data before completing processing of the HWSYNC request). Waiting for these load requests to be satisfied at block 1303 guarantees that the group of load requests identified at block 906 will receive data from the correct memory page (even if the target address was on the memory page that was reallocated) rather than from a newly reallocated memory page.

[0070] In parallel with block 1303, processor core 200 also issues the PTESYNC request corresponding to PTESYNC instruction 430 to L1 STQ 304 (block 1304). The process continues from block 1304 to block 1308, which illustrates processor core 200 performing the memory de-ordering function for the PTESYNC request by waiting for all corresponding older memory requests from all hardware threads to be evicted from L1 STQ 304 (i.e., those that would be architecturally required for an HWSYNC to be evicted from L1 STQ 304). When the memory ordering performed at block 1308 is complete, the PTESYNC request is issued over bus 318 from L1 STQ 304 to L2 STQ 320, as indicated at block 1310.

[0071] The process then continues from block 1310 to block 1312, which illustrates that the initiating processor core 200 monitors to detect receipt of a PTESYNC_ACK signal from the memory subsystem over bus 325, indicating that the initiating processor core 200 has completed processing of the PTESYNC request. (The generation of the PTESYNC_ACK signal is described below with reference to block 1410 of Fig. 14.) Again, it should be noted that because the initiating hardware thread continues to pause instruction propagation, there can be no conflict for the initiating hardware thread's sidecar 324 by another TLBIE request, since for any given thread, there can only be either one TLBIE request or one PTESYNC request in the L2 STQ 320 and the sidecar logic 322.

[0072] Only in response to positive determinations at both blocks 1303 and 1312 does the process of Fig. 13 continues with block 1314, which illustrates that the processor core 200 resumes the passing of instructions in the initiating thread; thus, the release of the thread at block 1314 allows the processing of instructions after the PTESYNC instruction 430 to begin. The process of Fig. 13 then ends at block 1316.

[0073] With reference to Fig. 14 illustrates a high-level logic flow diagram for an exemplary method by which the L2 STQ 320 and the sidecar logic 322 of a processing unit 104 process a PTESYNC request, according to one embodiment. The process of Fig. 14 begins at block 1400 and then continues with block 1402, which illustrates the L2 STQ 320 monitoring for receipt of a PTESYNC request from the L1 STQ 304, as described above with reference to block 1310 of Fig. 13. In the second embodiment of Fig. 4B, in response to receiving the PTESYNC request, the L2 STW 320 and the sidecar logic 322 cooperate to perform two functions, namely (1) storing the ordering of memory requests in the L2 STQ 320 and (2) ensuring completion of the TLBIE request on all of the other processing cores 200. In the embodiment of Fig. 14, these two functions are executed in parallel along the two paths illustrated at blocks 1403, 1405 and blocks 1404, 1406, and 1408, respectively. In alternative embodiments, these functions could instead be serialized by first executing the ordering function, illustrated at blocks 1403 and 1405, and then ensuring completion of the TLBIE request at blocks 1404, 1406, and 1408. (Note that attempting to serialize the ordering for these functions while ensuring completion of the TLBIE request before executing the memory ordering may result in a deadlock.)) Referring now to blocks 1403 through 1405, the L2 STQ 320 performs memory ordering for the PTESYNC request by ensuring that all corresponding older memory requests in the L2 STQ 320 have been eliminated from the L2 STQ 320. The group of memory requests being ordered at block 140 includes a first subgroup whose destination addresses may have been translated by the translation entry invalidated by the earlier TLBIE request. This first subgroup corresponds to the one marked at block 906. In addition, the group of memory requests being ordered at block 1403 includes a second subgroup containing the architecturally defined memory requests whose ordering would be assigned by an HWSYNC.When all such memory requests have been removed from the L2 STQ 320, the L2 STQ 320 removes the PTESYNC request from the L2 STQ 320 (block 1405). Removing the PTESYNC request allows memory requests younger than the PTESYNC request to flow through the L2 STQ 320.

[0074] Referring now to block 1404, the sidecar logic 322 detects the presence of the PTESYNC request in the L2 STQ 320 and copies the PTESYNC request to the appropriate sidecar 324 via the interface 321 before removing the PTESYNC request from the L2 STQ 320 at block 320. The process then continues with the loop illustrated at blocks 1406 and 1408, in which the sidecar logic 322 continues to issue PTESYNC requests on the system fabric 110, 114 until no processor core 200 responds with a coherency "retry" response (i.e., until the previous TLBIE request on the same processor core and hardware thread has been completed by all monitoring processor cores 200).

[0075] Only in response to the completion of both functions illustrated at blocks 1403, 1405 and blocks 1404, 1406, and 1408 does the process proceed to block 1410, which illustrates that sidecar logic 322 issues a PTESYNC_ACK signal to the associated processor core via bus 325. Sidecar logic 322 then removes the PTESYNC request from sidecar 324 (block 1412), and the process returns to block 1402, which has been described.

[0076] With reference to Fig. 15 illustrates an overview-level logic flow diagram for an exemplary method by which TSN engines 346 process TLBIE requests, TLBCMPT_ACK signals, and PTESYNC requests according to one embodiment. As indicated by like reference numerals, the Fig. 15 the same as the previously described Fig. 7, except for block 1522. Block 1522 illustrates that while the TSN engine 346 is in the TLBIE_active state established at block 704, it monitors to determine if a PTESYNC request has been detected that specifies an initiating processor core and a hardware thread that matches the TLBIE request. If not, the process continues to repeat the loop with blocks 720 and 1522. However, if in response to a TSN engine 346 detecting a PTESYNC request that specifies a processor core and an initiating hardware thread that match the TLBIE request while in the TLBIE_active state, the TSN engine 346 provides a "retry" coherency response, as indicated at block 724.As discussed above, a coherency "retry" response from any TSN snooper 346 servicing the TLBIE request for the initiating hardware thread forces the PTESYNC request to be retried and prevents the initiating hardware thread from executing any memory-referencing instructions younger than the PTESYNC instruction 430 until the PTESYNC request completes without a coherency "retry" response.

[0077] With reference to Fig. 16, a block diagram of an exemplary development flow 1600 is shown, for example, used for the development, simulation, test, layout, and fabrication of semiconductor IC logic. The development flow 1600 includes processes, machines, and / or mechanisms for processing design structures or units to logically or otherwise generate functionally equivalent representations of the design structures described above and in the Fig. 1 to 3. The design structures processed and / or generated by the development flow 1600 may be encoded on machine-readable transmission or storage media to contain data and / or instructions that, when executed or otherwise processed on a data processing system, generate a logically, structurally, mechanically, or otherwise functionally equivalent representation of hardware components, circuits, units, or systems. Machines include, but are not limited to, any machines used in an IC development process such as developing, manufacturing, or simulating a circuit, component, unit, or system. Machines may include, for example: lithography machines, machines and / or equipment for generating masks (e.g.,electron beam writer), computer or equipment for simulating design structures, any device used in the manufacturing or test process, or any machine for functionally programming equivalent representations of the design structures in any medium (e.g., a machine for programming a programmable gate array).

[0078] The development flow 1600 may vary depending on the type of representation being developed. For example, a development flow 1600 for creating an application-specific integrated circuit (ASIC) may differ from a development flow 1600 for developing a standard component or from a development flow 1600 for instantiating the design into a programmable array, such as a programmable gate array (PGA) or a field-programmable gate array (FPGA) offered by Altera® Inc. or Xilinx® Inc.

[0079] Fig. 16 illustrates several such design structures, including an input design structure 1620, which is preferably processed by a development process 1616. The design structure 1620 may be a logical simulation design structure generated and processed by the development process 1616 to produce a logically equivalent functional representation of a hardware device. The design structure 1620 may also or alternatively include data and / or program instructions that, when processed by the development process 1616, generate a functional representation of the physical structure of a hardware device. The design structure 1620 may be generated using electronic computer-aided design (ECAD), such as implemented by a core developer / designer, whether representing functional and / or structural design features.When the design structure 1620 is encoded on a machine-readable data transmission, gate array, or storage medium, it may be accessed and processed by one or more hardware and / or software modules in the development process 1616 to simulate or otherwise represent an electronic component, circuit, electronic or logic module, device, unit, or system, as described in the . Fig. 1 through 3. Therefore, the design structure 1620 may include files or other data structures, including human- and / or machine-readable source code, compiled structures, and computer-executable code structures that, when processed by a data processing system for development or simulation, functionally simulate or otherwise represent circuits or other levels of hardware logic designs. Such data structures may include development entities or other hardware description language (HDL) data structures that correspond to and / or are compatible with lower-level HDL design languages ​​such as Verilog and VHDL and / or higher-level design languages ​​such as C or C++.

[0080] The development process 1616 preferably uses and integrates hardware and / or software modules to synthetically create, implement, or otherwise process a functional equivalent of a design / simulation of the components, circuits, units, or logical structures included in the Fig. 1 through 3 to generate a netlist 1680 that may include design structures such as design structure 1620. For example, netlist 1680 may include compiled or otherwise processed data structures representing a list of wires, discrete components, logic gates, control circuits, I / O units, models, etc., that describe the connections to other elements and circuits in the design of an integrated circuit. Netlist 1680 may be synthesized using an iterative process in which netlist 1680 is re-synthesized one or more times, depending on design specifications and parameters for the device. As with other design structure types described herein, netlist 1680 may be recorded on a machine-readable storage medium or programmed into a programmable gate array.The storage medium may be a non-volatile storage medium such as a magnetic or optical disk drive, a programmable gate array, a compact flash, or other flash memory. Additionally, or alternatively, the storage medium may be system or cache memory or buffer storage space.

[0081] The development process 1616 may include hardware and software modules for processing a variety of input data structure types, including the netlist 1680. Such data structure types may be located, for example, in library elements 1630 and may contain a group of commonly used elements, circuits, and units, including models, layouts, and symbolic representations for a particular manufacturing technology (e.g., various technology nodes, 32 nm, 45 nm, 90 nm, etc.). The data structure types may further include design specifications 1640, characterization data 1650, test data 1660, design rules 1670, and test data files 1685 containing input test patterns, output test patterns, and other test information.The development process 1616 may further include, for example, standard mechanical development processes such as stress analysis, thermal analysis, simulation of mechanical events, process simulation for operations such as casting, molding, and compression molding, etc. A mechanical design expert can estimate the scope of possible mechanical design tools and applications used in the development process 1616. The development process 1616 may also include modules for performing standard circuit development processes such as timing analysis, testing, design rule verification, placement and routing of operations, etc.

[0082] The development process 1616 uses and integrates logic and physical design tools such as HDL, compilers, and simulation modeling tools to process the design structure 1620 along with some or all of the illustrated supporting data structures in conjunction with any additional mechanical design or data (if applicable) to generate a second design structure 1690. The design structure 1690 is located on a storage medium or programmable gate array in a data format used for data exchange of mechanical units and structures (e.g., information stored in an IGES, DXF, Parasolid XT, JT, DRG, or any suitable format for storing or reproducing such mechanical design structures).Similar to design structure 1620, design structure 1690 preferably comprises one or more files, data structures, or other computer-encoded data or instructions residing in transmission or data storage media that, when processed by an ECAD system, generates a logical or otherwise functional equivalent of one or more of the embodiments as shown in FIGS. Fig. 1 to 3. In one embodiment, the design structure 1690 may comprise a compiled executable HDL simulation model that implements the Fig. The units shown in 1 to 3 are functionally simulated.

[0083] The design structure 1690 may also use a data format used for the exchange of integrated circuit layout data and / or a symbolic data format (e.g., information stored in a GDSII (GDS1), GL1, OASIS, map files, or any other suitable format for storing such design data structures). The design structure 1690 may include information such as symbolic data, map files, test data files, design content files, manufacturing data, layout parameters, wires, metal levels, holes, shapes, data for routing through the production line, and any other data required for a manufacturer or other designer / developer to create a device or structure as described above and in the Fig.1 to 3. The design structure 1690 may then proceed to a phase 1695 in which the design structure 1690, for example, proceeds with tapeout, is released to manufacturing, is released to a mask house, is sent to another design house, is returned to the customer, etc.

[0084] As described, in at least one embodiment of a multithreaded computing system having a plurality of processor cores, memory-modifying requests from a plurality of concurrently executing hardware threads are received in a shared queue. The memory-modifying requests include a request to invalidate translations from an initiating hardware thread. The request to invalidate translations is removed from the shared queue and buffered in sidecar logic in one of a plurality of sidecars, each of which is associated with a respective one of the plurality of hardware threads. While the request to invalidate translations is buffered in the sidecar, the sidecar logic transmits the request to invalidate translations to be received and processed by the plurality of processor cores.In response to an acknowledgement of completion of processing of the invalidate translation request by the initiating processor core, the sidecar logic removes the invalidate translation request from the sidecar. Completion of processing of the invalidate translation request on all of the plurality of processor cores is ensured by a broadcast synchronization request.

[0085] In one embodiment, the order of subsequent memory-referencing instructions is specified with reference to the broadcast synchronization request by executing an additional synchronization instruction (e.g., HWSYNC).

[0086] While various embodiments have been specifically shown and described, it should be apparent to one skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the appended claims, and that these alternative implementations are all within the scope of the appended claims. Although aspects have been described with respect to a computer system executing, for example, program code that controls the functions of the present invention, it should be understood that the present invention may alternatively be implemented as a program product having a computer-readable storage device storing program code that can be processed by a processor of a data processing system to cause the data processing system to perform the described functions.The computer-readable storage device may include volatile or non-volatile memory, an optical or magnetic disk, or the like, but excludes free content such as broadcast signals per se, transmission media per se, and forms of energy per se.

[0087] For example, the program product may contain data and / or instructions that, when executed or otherwise processed on a data processing system, generate a logical, structural, or otherwise functionally equivalent representation (including a simulation model) of hardware components, circuits, units, or systems disclosed herein. Such data and / or instructions may include development entities or other data structures in Hardware Description Language (HDL) that correspond to and / or are compatible with lower-level HDL design languages ​​such as Verilog and VHDL and / or higher-level design languages ​​such as C or C++. Furthermore, the data and / or instructions may also use a data format used for exchanging integrated circuit layout data and / or a symbolic data format (e.g.,Information stored in a GDSII (GDS2), GL1, OASIS, map files, or any other suitable format for storing such design data structures).

Claims

[1] A method for invalidating translation entries without deadlock in a multithreaded data processing system including a plurality of processor cores (200), the method comprising: Generating a request to invalidate implementations by executing a corresponding instruction to invalidate implementations by an initiating hardware thread from a plurality of concurrently executing hardware threads of an initiating processor core from a plurality of processor cores, Stopping (502) a propagation of instructions in the initiating hardware thread that follow the request to invalidate translations in the program order until an acknowledgment signal is received (508) that confirms completion of processing of the request to invalidate translations at the initiating processor core, Receiving, in a shared queue (320), memory-modifying requests from the plurality of concurrently executing hardware threads of the initiating processor core, wherein the plurality of memory-modifying requests includes a request to invalidate translations from an initiating hardware thread of the plurality of hardware threads, in response to receiving the request to invalidate translations in the shared queue, removing the request to invalidate translations from the shared queue and buffering (604) the request to invalidate translations in sidecar logic (322), the sidecar logic including a plurality of sidecars, each of which is connected to one of the plurality of concurrently executing hardware threads and configured to buffer only a single request to invalidate translations generated by the corresponding hardware thread; while the request to invalidate translations is buffered in the sidecar logic, broadcasting the request to invalidate translations through the sidecar logic such that the request to invalidate translations is received and processed by the plurality of processor cores; in response to an acknowledgment of completion of processing of the invalidate translation request by the initiating processor core, removing the invalidate translation request from the sidecar by the sidecar logic; and Ensuring completion of processing of the request to invalidate translations on all of the plurality of processor cores by a broadcast synchronization request. [2] The method of claim 1, and further comprising: in response to a supervisory broadcast of the request to invalidate translations on a system fabric of the data processing system, a translation snoop engine (346) remaining in an active state until a signal is received confirming completion of processing of the request to invalidate translations at a supervisory processor core coupled to the translation snoop engine, and then returning to an inactive state. [3] The method according to claim 1, wherein the request to invalidate implementations specifies an effective address; and in response to receiving the request to invalidate implementations at the initiating processor core: the processor core invalidates one or more translation entries that translate the effective address; the processor core is waiting for one or more memory-referenced requests that depend on the one or more translation entries being removed from the initiating processor core; and the processor core then sends a termination request to the shared queue providing acknowledgment of completion of processing of the invalidate translation request by the initiating processor core. [4] The method of claim 3, and further comprising: in response to the shared queue receiving the termination request, ensuring, by the shared queue, that all older memory requests in the shared queue have been removed from the shared queue prior to removing the request to invalidate transformations from the sidecar. [5] The method of claim 1, wherein ensuring completion includes the sidecar logic broadcasting a translation synchronization request to all of the plurality of processor cores. [6] The method of claim 1, and further comprising: Generating the synchronization request by the initiating processor core by executing a first synchronization instruction; and Prescribing, by the initiating processor core, the order for execution of subsequent memory-referencing instructions with respect to the translation synchronization instruction by executing a second synchronization instruction. [7] A processing unit for a multi-threaded data processing system having a plurality of processor cores and for invalidating translation entries without deadlock in a multi-threaded data processing system, the processing unit comprising: a processor core (200) with: a translation structure (310) that temporarily stores address translations; a memory management unit (308) that translates effective addresses into real addresses by referring to the translation structure; an execution unit (300) concurrently executing a plurality of hardware threads, wherein an initiating thread of the plurality of hardware threads generates a request to invalidate translations by executing a corresponding invalidate translation instruction, and halts propagation of instructions in the initiating hardware thread following the request to invalidate translations in program order until an acknowledgement signal is received confirming completion of processing of the request to invalidate translations at the processor core; a shared queue (320) receiving memory modifying requests from the plurality of concurrently executing hardware threads, the plurality of memory modifying requests including the request to invalidate translations; Sidecar logic (322) connected to the shared queue, the sidecar logic including a plurality of sidecars, each connected to one of the plurality of hardware threads and configured to buffer only a single request to invalidate translations generated by the corresponding hardware thread, and the processing unit configured to perform: in response to receiving the request to invalidate implementations in the shared queue, removing the request to invalidate implementations from the shared queue and buffering the request to invalidate implementations in a sidecar associated with the initiating thread; while the request to invalidate translations is buffered in the sidecar, broadcasting the request to invalidate translations such that the request to invalidate translations is received and processed by the plurality of processor cores; in response to an acknowledgment of completion of processing of the invalidate translation request by the initiating processor core, removing the invalidate translation request from the sidecar; and Ensuring completion of processing of the request to invalidate translations on all of the plurality of processor cores by a broadcast synchronization request. [8] A data processing system including a plurality of processing units according to claim 7 and a system structure connecting the plurality of processing units. [9] Design structure embodied in a machine-readable storage unit for developing, manufacturing or testing an integrated circuit, the design structure comprising: a processing unit for a multi-threaded data processing system having a plurality of processor cores and for invalidating translation entries without deadlock in a multi-threaded data processing system, the processing unit comprising: a processor core (200) with: a translation structure (310) that temporarily stores address translations; a memory management unit (308) that translates effective addresses into real addresses by referring to the translation structure; an execution unit (300) concurrently executing a plurality of hardware threads, wherein an initiating thread of the plurality of hardware threads generates a request to invalidate translations by executing a corresponding invalidate translation instruction, and halts propagation of instructions in the initiating hardware thread following the request to invalidate translations in program order until an acknowledgement signal is received confirming completion of processing of the request to invalidate translations at the processor core; a shared queue (320) receiving memory modifying requests from the plurality of concurrently executing hardware threads, the plurality of memory modifying requests including the request to invalidate translations; Sidecar logic (322) connected to the shared queue, the sidecar logic including a plurality of sidecars, each connected to one of the plurality of hardware threads and configured to buffer only a single request to invalidate translations generated by the corresponding hardware thread, and the processing unit configured to perform: in response to receiving the request to invalidate implementations in the shared queue, removing the request to invalidate implementations from the shared queue and buffering the request to invalidate implementations in a sidecar associated with the initiating thread; while the request to invalidate translations is buffered in the sidecar, broadcasting the request to invalidate translations such that the request to invalidate translations is received and processed by the plurality of processor cores; in response to an acknowledgment of completion of processing of the invalidate translation request by the initiating processor core, removing the invalidate translation request from the sidecar; and Ensuring completion of processing of the request to invalidate translations on all of the plurality of processor cores by a broadcast synchronization request.

Images