Method and apparatus for a coherent shared memory system

Abstract

The present invention relates to a method for coordinating an instruction sequence executed by a processor device of a coherent shared memory system. An instruction is executed and causes the processor device to fill a copy of a memory line into a processor cache. Based on detecting first flag information indicating that the coherent propagation across the shared memory system with respect to the memory line is not confirmed, the processor device marks the memory line. Based on detecting second flag information indicating that the coherent propagation across the shared memory system with respect to the memory line is confirmed, the processor device clears the mark of the memory line. Based on executing a memory barrier instruction, the execution of the memory barrier instruction is prevented from being completed while marking the memory line.

Description

Background Art

[0001] The present disclosure relates to the field of computer systems. More specifically, it relates to a method for coordinating the execution of an instruction sequence including multiple instructions by a processor device of a consistent shared memory system, a processor device of a consistent shared memory system, a method for managing a consistent directory for implementing memory consistency, a consistent directory for implementing memory consistency of a shared memory system, and a consistent shared memory system including a consistent directory for implementing memory consistency of a shared memory system.

[0002] A shared memory system includes multiple processors that are all provided with memory access, thereby improving the efficiency of data processing. The shared memory can provide multiple processors with simultaneous access to the same data, avoid storing redundant copies of data, and enable data transfer between processors. A shared memory system, for example, provides a main memory that can be accessed by multiple processors. In order to enable the processor to quickly access the memory lines of the main memory, the memory architecture of such a shared memory system can further include multiple processor cache memories for caching data to be processed by the processor. For example, each processor can be operably connected to one or more processor cache memories.

[0003] However, in a shared memory system with multiple processor cache memories, there is a risk that complications may arise due to lack of data coherence. Whenever data contained in one of the processor cache memories is updated, it must be ensured that all remaining processor cache memories will use the updated data when processing the same data, in order to avoid different processors using inconsistent data. In other words, it is necessary to ensure that coherence propagates throughout the shared memory system in order to establish coherence throughout the system.

[0004] To prevent the processor from using outdated data that has been modified, ordering constraints on memory operations are required. A strictly ordered memory model stipulates that changes to the value of a shared variable must always be visible to the other threads of the executing multithreaded program. In other words, the order of storage operations must be defined in advance, and no exceptions can be allowed to avoid race conditions. However, a strictly ordered model can be inflexible, resulting in limited processing efficiency. In contrast, processing efficiency can be improved by allowing relaxation of the strong order. There may be instructions that can be executed in parallel or even in any order without causing problems or causing race conditions. Memory models that allow for relaxation of the order are called weakly ordered memory models. In order to still maintain some degree of ordering, weakly ordered memory models specify memory barriers established by special, well-defined memory barrier instructions. Memory barrier instructions define a point in an instruction sequence where consistency must be ensured and synchronization performed if necessary. This weak memory model stipulates that changes to the value of a shared variable only need to be made visible to other threads when such a memory barrier is reached. Memory barrier instructions define a point in an instruction sequence where consistency must be ensured and synchronization performed if necessary. This weak memory model dictates that changes to the value of a shared variable need only be made visible to other threads when this memory barrier is reached. A memory barrier instruction defines a point in a sequence of instructions at which consistency must be ensured and synchronization performed if necessary. This weak memory model dictates that changes to the value of a shared variable need only be made visible to other threads when this memory barrier is reached.

[0005] The weakly ordered memory model can provide more freedom when optimizing instruction sequences. It only needs to ensure that the values ​​of potential shared variables at memory barriers are guaranteed to be the same in both optimized and unoptimized instruction sequences. In particular, reordering statements within an instruction block that does not contain any memory barriers can be considered safe.

[0006] To ensure memory consistency and improve performance of shared memory systems, there is a continuing need to improve the implementation and management of memory barriers. Summary of the invention

[0007] Various embodiments provide a method for coordinating an instruction sequence comprising a plurality of instructions executed by a processor device of a coherent shared memory system, a processor device of a coherent shared memory system, a method for managing a coherence directory to achieve memory coherence of a shared memory system, a coherence directory to achieve memory coherence of a shared memory system, and a coherent shared memory system comprising a coherence directory to achieve memory coherence of a shared memory system. Advantageous embodiments are described in the dependent claims. The embodiments of the invention can be freely combined with each other if they are not mutually exclusive.

[0008] In one aspect, the present invention relates to a method for coordinating an instruction sequence including a plurality of instructions executed by a processor device of a consistent shared memory system. The processor device is operably connected to a processor cache memory. The instructions of the instruction sequence are executed by the processor device. The execution of the instructions causes the processor device to fill a memory line copy of a main memory of the shared memory system to the processor cache memory. The processor device marks the memory line upon detecting first flag information, the first flag information indicating that propagation of memory consistency across the shared memory system regarding the filling of the memory line copy is not confirmed. The processor device does not mark the marked memory line upon detecting second flag information, the second flag information indicating that propagation of memory consistency on the shared memory system regarding the filling of the memory line copy is confirmed. When the processor device executes a memory barrier instruction of the instruction sequence, the processor device blocks completion of the execution while marking the memory line.

[0009] Embodiments may have the beneficial effect of enabling the processor device to effectively ensure that the execution of the instruction sequence stops once the memory barrier instruction is reached, i.e. executed, in the case where the propagation of consistency has not been confirmed. The marking of the memory line indicates that the propagation of memory consistency across the shared memory system has not been confirmed with respect to the filling of the copy of the memory line. In other words, the marking indicates that no information is available, i.e., whether the memory consistency of the filling of the copy of the memory line across the shared memory system has been established. In the case where the memory line is not marked, there is no problem with consistency, but the propagation of consistency has been clearly confirmed and no pause is required. In the case where the memory line has been marked, it must be waited for to be clearly confirmed and must be paused. Such confirmation may, for example, be provided by a directory that implements the directory-based consistency of the shared memory system or by the remaining processor devices of the shared memory system to a reply to a request for establishing consistency broadcasted by the processor device that performs the filling of the memory line. Since only the memory lines for which the consistency propagation has not been confirmed are marked, i.e., the indicator is registered as a mark, local storage space of the processor device may be saved.

[0010] The execution may be continued only if the memory line is untagged, i.e. if memory coherence is confirmed to propagate in the shared memory system. The propagation of the memory coherence throughout the shared memory system means that the coherence of the shared memory system is established with respect to the filling of the copies of the memory lines to the processor cache memory. The coordination of the execution of the instruction sequences is implemented by separate processor devices (e.g., processor chips or processor cores). Thus, the method can be used for directory-based coherence as well as broadcast-based coherence.

[0011] According to an embodiment, execution of a memory barrier instruction is prevented from being completed until all memory lines previously marked by the processor device are not marked. An embodiment may have the beneficial effect that the clearing of a single memory line is a necessary condition for continuing to execute the memory barrier instruction. As a sufficient condition, it may be achieved that all memory lines not previously marked by the processor device are marked, i.e., the processor device includes neither the marked memory lines nor the marks of the memory lines. Thus, instructions that result in the filling of copies of memory lines to the processor cache may be freely executed without regard to memory consistency until the memory barrier instruction is reached. Upon reaching the memory barrier instruction, memory consistency must be established and positively confirmed before the execution of the memory barrier instruction can continue. Memory consistency may be propagated on the shared memory system before the execution of the memory barrier instruction. However, the memory barrier instruction constitutes a point in the instruction sequence at which consistency propagation of the filling of all copies of the memory line to the processor cache memory on the shared memory system is required to have been completed. The respective completion may be ensured by requiring an explicit confirmation of successful propagation. Before executing the memory barrier, the processor may establish knowledge about the consistency propagation, but does not necessarily have to establish knowledge.

[0012] According to an embodiment, the marking of the memory line comprises adding an identifier of the memory line to a cache of the processor device. Unmarking the marked memory line comprises removing the identifier of the memory line from the cache of the processor. Embodiments may have the beneficial effect of whether the cache can provide a direct overview regardless of whether there are memory lines that still lack the necessary confirmation of consistency propagation across the shared memory system. The cache may provide a flag, for example, in the form of an unordered set of indicators. The indicator indicates that the memory cache includes the line, so the consistency propagation of the individual memory lines has not yet been confirmed. If, for a particular memory line, the cache does not include an indicator, the consistency propagation of the individual memory lines has been confirmed. The flag may be set by adding an indicator indicating the memory line to be marked to the cache area. Therefore, the cache area is a cache area for memory lines that have not been propagated.

[0013] According to an embodiment, the memory barrier instruction is prevented from being executed until the cache of the processor device is empty. Embodiments may have the beneficial effect that they provide a fast and efficient implementation of a consistency check in the case where the memory cache instruction that requires a consistency check is executed. In the case where the cache does not include indicators, consistency has been established and confirmed for each memory line previously filled into the processor cache memory by the processor device. In the case where the cache still includes indicators, the processor device will wait until the consistency of those remaining indicators is confirmed and the cache is empty. Therefore, it can be ensured that for instructions of the instruction sequence executed after the memory barrier instruction,

[0014] According to an embodiment, a message related to modification of the content of a processor cache memory and processed by a processor device includes a consistency propagation field. The consistency propagation field includes one of the following: tag information indicating that propagation of the memory consistency modification across the shared memory system is not confirmed; tag information indicating that propagation of the memory consistency modification across the shared memory system is confirmed, or tag information indicating that neither a tag nor a clear tag action is required.

[0015] Embodiments may have the beneficial effect of enabling efficient communication between the processor device and the rest of the shared memory system, thereby providing feedback on the coherence propagation across the shared memory system of filling the copy of the memory line into the processor cache and filling the copy of the memory line into the processor cache memory. For example, the processor device may request exclusive write access to the memory line of the main memory. In the absence of the request in the processor cache memory, the request may be sent to the coherence directory or broadcast in the shared memory system. The request may include tag information indicating no tag or clear tag action. So far, only one request has been sent without actually starting the filling of the processor cache. The reply to the request received by the processor device may include a copy of the requested data, such as a write-exclusive copy, which the receiving processor fills into the processor cache. The reply may include flag information indicating that the propagation of memory coherence across the shared memory system regarding the modification has not been confirmed. After establishing memory coherence on the shared memory system, such as by invalidating shared read copies of the requested memory line included by processor cache memories of other processor devices, the invalidation may be confirmed by other processor devices performing the same operation.

[0016] In another aspect, the present invention relates to a processor device of a consistent shared memory system. The processor device is operably connected to a processor cache memory and is configured to coordinate the execution of an instruction sequence including a plurality of instructions. The coordination includes executing the instructions of the instruction sequence. The execution of the instructions causes the processor device to fill the processor cache memory with a copy of a memory line of a main memory of the shared memory system. The coordination also includes marking the corresponding memory line upon detecting flag information, the flag information indicating that the propagation of the memory consistency of the filling of the copy of the memory line across the shared memory system has not been confirmed. Upon detecting that the flag information indicating that the propagation of the memory consistency of the filling of the copy of the memory line across the shared memory system has been confirmed, the processor device clears the mark of the memory line. When executing a memory barrier instruction of the instruction sequence, the processor device prevents the execution of the memory barrier instruction from being completed while marking the memory line.

[0017] The embodiments may have the beneficial effect of providing a processor device of a coherent shared memory system, the processor device being configured to effectively implement the above method to coordinate the execution of an instruction sequence including a plurality of instructions.

[0018] According to an embodiment, execution of the memory barrier instruction is prevented from completing until all memory lines previously marked by the processor device are cleared. Embodiments may have the beneficial effect that the processor device ensures that the memory barrier instruction is prevented from completing until the consistency of all populations of the shared memory system relative to memory line copies previously executed by the processor device is confirmed.

[0019] According to an embodiment, the processor device comprises a cache for registering memory lines, copies of which are populated to the processor cache memory, and for which propagation of memory consistency regarding the population of the respective copies on the shared memory system is not confirmed. Marking a memory line comprises adding an identifier of the memory line to the cache of the processor device. The tag clearing of the marked memory line comprises removing the identifier of the memory line from the cache of the processor.

[0020] Embodiments may have the beneficial effect that the infrastructure for registering memory lines for which propagation of memory consistency across the shared memory system in terms of population of various copies is not yet confirmed.

[0021] According to an embodiment, execution of the memory barrier instruction is prevented from completing until the cache of the processor device is empty. Embodiments may have the beneficial effect that an effective overview may be provided by the cache whether there are any memory lines whose copies have been filled into the processor cache memory before completing the memory barrier instruction and which still need to be confirmed. If the cache is empty, all necessary confirmations have been received. In the case where the cache is not empty, there are still outstanding confirmations that the processor device must wait for.

[0022] In another aspect, the present invention relates to a method for managing a consistency directory for implementing memory consistency of a shared memory system. The consistency directory maintains flag information of a memory line assigned to a main memory of the shared memory system. A processor device of the shared memory system initiates filling a copy of the memory line into a processor cache memory operably connected to the processor device. The flag information indicates whether propagation of memory consistency across the shared memory system regarding the filling of the copy of the memory line is confirmed. The consistency directory includes the flag information in a message and sends the message to the processor device.

[0023] Embodiments may have the beneficial effect of providing an efficient method for managing a consistency directory for implementing memory consistency of a shared memory system. The consistency directory may check the consistency of the shared memory system and, if necessary, initiate an action for establishing consistency across the shared memory system in response to the filling of copies of the memory lines into the processor cache memory. For example, in the case of filling an exclusive write copy and / or a revised copy of an exclusive write copy of the memory line into the processor cache memory, the consistency directory may initiate invalidation of copies of other memory lines in the shared memory system, particularly in processor caches of other processor devices of the shared memory system. For example, outdated copies, i.e., stale copies,

[0024] According to an embodiment, in order to fill the copy of the memory line into the consistency of the remote cache memory, when it is determined that the content of the remote processor cache memory of another processor device of the shared storage system needs to be modified in order to propagate the memory, the consistency directory initiates maintenance of the flag information assigned to the memory line.

[0025] Embodiments may have the following beneficial effects: to ensure memory consistency, processing power is conserved and the tag information is generated and maintained only when the consistency directory determines that the contents of the remote processor cache memory of another processor device must be modified in order to modify the following. For example, in the case of a shared read request, it is not necessary to modify the previously loaded shared read copies of the request memory line in the remote processor cache of the other processor device, because the additional shared read request does not change the consistency of the memory. However, in the case of an exclusive write request, the remaining shared read copies may need to be modified (i.e., invalidated) to establish memory consistency.

[0026] According to an embodiment, the coherency directory sends a request to the remote processor cache memory. The request requests modification of the contents of the remote processor cache memory in order to propagate memory coherency regarding populating the copy of the memory line to the remote processor cache memory.

[0027] Embodiments may have the beneficial effect of achieving effective control of memory consistency through a consistency directory.By sending a request and receiving a corresponding reply, the consistency directory may confirm the completion of the propagation of consistency to the processor device once all modifications are initiated and / or completed.

[0028] According to an embodiment, the coherence directory comprises registers for tracking modifications of the contents of processor cache memories of other processor devices of the shared memory system with respect to the filling of the memory line copies due to propagation of memory coherence throughout the shared memory system.

[0029] Embodiments may have the beneficial effect of efficiently implementing registers for tracking the modifications and determinations when the consistency propagation is complete.

[0030] According to an embodiment, when the memory consistency propagation across the shared memory system regarding the population of the copy of the memory line is confirmed, the register includes a consistency propagation notification indicator, which identifies a processor cache memory of a processor device of the shared memory system to be notified by the consistency directory.

[0031] Embodiments may have the beneficial effect of ensuring that the coherence directory is able to effectively track which processor cache memories of the shared memory system must be notified of the coherence, i.e. which of the processor cache memories have been filled with memory line copies, which may need to be positively checked in the future to confirm coherence propagation in the event of execution of a memory barrier instruction.

[0032] In another aspect, the present invention relates to a consistency directory for implementing memory consistency of a shared memory system. The consistency directory is configured to maintain flag information of a memory line assigned to a main memory of the shared memory system. A processor device of the shared memory system initiates an operation of populating a copy of the memory line into a processor cache memory operably connected to the processor device. The flag information indicates whether the propagation of memory consistency across the shared memory system is confirmed with respect to the populating of the copy of the memory line. The consistency directory is further configured to include the flag information in a message and send the message to the processor device.

[0033] Embodiments may have the advantageous effect of providing a coherence directory for implementing memory coherence of a shared memory system, the coherence directory being configured to effectively maintain tag information assigned to a memory line of a main memory maintaining the shared memory system.

[0034] In another aspect, the present invention relates to a coherent shared memory system comprising a coherent directory for implementing memory coherence of the shared memory system, a main memory, and a processor device operatively connected to a processor cache memory.

[0035] The processor device is configured to coordinate the execution of an instruction sequence including a plurality of instructions. The coordination includes executing instructions of the instruction sequence. The execution of the instructions causes the processor device to populate a copy of a memory line of the main memory into the processor cache memory. Based on detecting that flag information indicating that memory consistency propagation across the shared memory system of the filling of the copy of the memory line is not confirmed, the processor device marks the respective memory lines. Based on detecting that flag information indicating that memory consistency propagation across the shared memory system of the filling of the copy of the memory line is confirmed, the processor device clears the mark for the respective memory lines. Based on executing a memory barrier instruction of the instruction sequence, the processor device prevents completion of the execution while marking the memory line.

[0036] The consistency directory is configured to maintain the flag information of the memory line assigned to the main memory based on confirmation of initiating the population of the copy of the memory line. The flag information indicates whether the memory consistency propagation across the shared storage system regarding the population of the copy of the memory line has been determined. First, the consistency directory includes the flag information, which indicates that the memory consistency propagation across the shared storage system regarding the population of the copy of the memory line has not been confirmed as a first message, and sends the first message to the processor device. Based on the propagation obtaining confirmation, the consistency directory updates the flag information after obtaining the propagation confirmation, and includes the updated flag information in a second message. The updated flag information indicates that the memory consistency propagation across the shared storage system regarding the population of the copy of the storage message is confirmed as a second message. The second message with the updated flag information is sent by the consistency directory to the processor device.

[0037] The embodiment can have the beneficial effect of providing a consistent shared memory system, the consistent shared memory system includes a consistency directory for implementing memory consistency of the shared memory system, a main memory, and a processor device operably connected to a processor cache memory. The shared memory system can effectively implement the coordination of the execution of instruction sequences by various processor devices, and can maintain control of consistency propagation across the shared memory through the consistency directory. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 depicts a schematic diagram of an exemplary computer system suitable for implementing a coherent shared memory system;

[0039] Figure 2 depicts a schematic diagram of an exemplary coherent shared storage system;

[0040] Figure 3 depicts a schematic diagram of an exemplary coherent shared storage system;

[0041] Figure 4 depicts a schematic diagram of an exemplary coherent shared storage system;

[0042] Figure 5 depicts a schematic diagram of an exemplary coherent shared storage system;

[0043] Figure 6 A schematic diagram depicting an exemplary method of execution of a sequence of instructions;

[0044] Figure 7A schematic diagram depicting an exemplary consistency directory for implementing consistency of a shared storage system;

[0045] Figure 8 A schematic diagram depicting an exemplary method for managing a coherence directory for implementing memory coherence of a shared memory system;

[0046] Fig. 9 A schematic diagram depicting an exemplary method for managing a coherence directory for implementing memory coherence of a shared memory system;

[0047] Fig. 10A and Fig. 10B A flowchart depicting an exemplary method of executing a sequence of instructions and achieving memory coherence for a shared memory system; and

[0048] Fig.11 A schematic diagram illustrating an exemplary multi-processor architecture implementing a shared memory system is depicted.

[0049] Detailed description

[0050] Descriptions of various embodiments of the present invention are given for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, practical applications or technical improvements of technologies found in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

[0051] Processor cache memory temporarily stores data so that future requests for that data can be serviced more quickly. Data stored in a processor cache may be a copy (i.e., a duplicate) of data stored elsewhere (e.g., in main memory). A cache hit occurs when the requested data can be found in the cache, and a cache miss occurs when the requested data cannot be found in the cache. A cache hit is serviced by reading the data from the processor cache memory, which is faster than reading the data from a slower data storage area (e.g., main memory). Thus, the use of processor cache memory can speed up system performance of a computer system.

[0052] To provide data via the processor cache memory, data is transferred between the backing memory (e.g., main memory) and the processor cache memory in blocks of fixed size. Such blocks of data are called memory lines or cache lines. When a memory line is copied from the main memory to the processor cache, a cache entry is created. The cache entry may include the copied data, i.e., the memory line, and the requested location in the backing memory, i.e., the address tag providing the address of the respective memory line in the main memory.

[0053] When a cache client, such as a processor, for example, needs to read or write from a location in a backing store, i.e., read or write a memory line, the cache client may first check the corresponding entry in the cache memory. The processor cache memory checks the addresses of the individual memory lines in all cache entries. In the case of a cache hit, i.e., finding a cache entry containing the sought address tag, the processor may immediately read data from or write data to the processor cache entry with the required address tag, thereby modifying the memory line stored therein. In the case of a cache miss, the processor cache memory may allocate a new cache entry and copy the memory line assigned the sought address tag from another cache memory, such as another cache client of the processor, or from the backing store, to the new cache entry. The request may then be satisfied from the contents of the new cache entry and the new memory line of the cache memory.

[0054] Cache memories may achieve performance improvements for data transfers, particularly data transfers that are repeatedly transferred. For example, a processor cache memory may achieve performance improvements on the initial transfer of a memory line due to the caching that occurs within the cache memory. Furthermore, a processor cache memory may achieve performance improvements because the memory line is retrieved from the main memory and stored in the faster intermediate processor cache memory, so that in the case of repeated requests the corresponding memory line may be retrieved from the faster intermediate processor cache memory instead of the main memory. Furthermore, a processor cache memory may achieve performance improvements for writing modified memory lines by storing the memory line immediately in the intermediate processor cache memory and deferring the transfer of the corresponding memory line to the main memory at a later stage. Alternatively, the transfer of the respective memory line to the main memory may occur as a background process.

[0055] A processor cache memory may include a plurality of entries. Each entry may include cache data, i.e. also referred to as a cache memory line, which is a copy of data of a memory line (i.e. a memory block) in some backing memory such as a main memory. Each entry may further include an address tag, the address tag identifying the data, i.e. a memory line, included in the entry. More precisely, the address tag may identify data stored in the backing memory of which the corresponding data in the cache memory entry is a copy. In the event of a write access to the corresponding cache memory entry, the memory line may be modified, thereby representing a modified copy of certain data stored in the backing memory. After modifying the memory line in the processor cache memory, in order to ensure consistency between the memory line stored in the processor cache memory and the corresponding data stored in the backing memory, a memory consistency protocol may be used to update the corresponding memory line in the backing memory, the copy of the data in the backing memory. In addition, each entry may include state information, the state information identifying the state of the data stored in the corresponding entry (i.e. a memory line).

[0056] If a copy of a memory line is modified by a single cache client, all other copies of the memory line with the same address may become out-of-date. Therefore, it may be necessary to manage the handling of those out-of-date memory lines. To ensure memory consistency, it may be necessary to ensure that out-of-date copies of memory lines are no longer used as valid copies. Out-of-date copies may be invalidated or updated.

[0057] When a cache client, such as a processor core or processor chip, needs to access data that is supposed to be stored in a backing store, it may first check one or more processor cache memories. If an entry can be found in one of these processor cache memories and its address tag matches the address tag of the requested memory line, the memory line provided by the corresponding processor cache memory entry may be used in place of the corresponding data storage stored in the backing store. This situation is called a cache memory hit. In the case of a cache memory miss, i.e., when no entry with the corresponding address tag of the required memory line is found in one or more processor cache memories, the required uncached data is retrieved from the backing store. During miss handling, the uncached data retrieved from the backing store may, for example, be copied to the processor cache memory, thereby enabling cache memory hits in the event of further access attempts.

[0058] In order to maintain coherency between one or more processor cache memories and main memory, as well as between different cache memories, the caching process may have to adhere to a cache coherence protocol.

[0059] In computer architecture, memory coherence refers to the consistency of shared resource data stored in multiple processor caches of a shared memory system. In a shared memory system, maintaining memory coherence can be particularly important.

[0060] In a shared memory system, such as in the form of a shared memory multiprocessor system with at least one separate processor cache memory per processor, there may be many copies of the same shared memory line: for example, one copy of "1" in main memory and one or more local caches of each processor requesting a corresponding copy of the memory line. When one of the copies is changed, the other copies must reflect the change. Memory consistency is the discipline of ensuring that changes to the copies of the memory line in the processor caches are propagated throughout the system in a timely manner.

[0061] The shared memory system may include multiple processor cache memories. According to an embodiment, the shared memory system may be provided in the form of an electronic data processing system such as a computer. For example, each electronic data processing system includes one or more multi-core processors. Each multi-core processor includes two or more independent processing units, i.e., cores, which read and execute program instructions. Each of these cores may be provided with a local cache memory, i.e., a separate cache memory for caching data used by each core. According to an embodiment, the local cache memory may only serve the core to which it is assigned.

[0062] According to another example, the electronic data processing system may include multiple processor chips. Two or more processor chips may each include a local processor cache memory, ie, a single cache memory serves only the processor included by the processor chip assigned to the respective cache memory.

[0063] Memory consistency may be implemented, for example, using an invalidation-based memory consistency protocol, such as MESI or modifications such as H-MESI, MOESI, MESIF, or MERSI.

[0064] According to MESI, each cache entry is assigned one of the following four possible states: modified, exclusive, shared, and invalid. For example, two additional bits are used to indicate the state. The "modified" identification only exists in the current cache memory and is dirty, that is, a memory line that has been modified compared to the corresponding data line stored in the main memory. Therefore, before allowing the data line that is no longer valid stored in the main memory state to be read in other ways, the cache memory is required to write the modified memory line back to the main memory. The writeback of the corresponding memory line can change the state of the memory line from the modified state to the shared state.

[0065] "Exclusive" means a memory line that is present only in the current cache memory and is clean, i.e., it still matches the corresponding data line stored in the main memory. The exclusive state can be changed to the shared state, for example, in response to receiving a read request from another cache memory. When such a read request is received, a copy of the requested memory line can be sent to the requesting cache memory as a shared cache memory line, and the state of the copied cache memory line retained in the cache memory can be changed to shared. If the state of the corresponding memory line is written to the corresponding memory line, it can, for example, be changed from exclusive to modified.

[0066] A "shared" flag may be stored in multiple cache memories and clean memory lines.

[0067] "Invalid" identifies a cache entry that is not used and is not used to satisfy a memory access. The state of a memory line in a cache may be changed, for example, from shared to invalid if a copy of the corresponding memory line is requested as an exclusive memory line by another cache memory. For example, a shared copy of the respective memory line may be forwarded as an exclusive copy, while the remaining duplicate memory lines are invalidated to avoid inconsistencies, i.e., different cache memories are using different copies of the same memory line. If an invalid memory line is to be read, its valid copy, i.e., shared or exclusive, must be read from another cache memory or from the main memory.

[0068] In addition to the four common MESI protocol states, the MOESI protocol includes a fifth "owned" state, representing data that is modified and shared. This avoids the need to write modified data back to main memory before sharing. Although the data must still be written back eventually, the writeback can be postponed. "Modified" identifies a memory line that is the only valid copy of said memory line and has been modified.

[0069] "Owned" identifies the copy of the memory line that is the only copy that can be modified, while other processor cache memories may also contain valid copies, but modifications are not allowed. Modifications to the owned memory line must be broadcast to all other processor caches that share the memory line. The introduction of the owned state allows dirty sharing.

[0070] ) data, i.e., a modified memory line can be moved around the various processor cache memories without updating main memory. The memory line can be changed to the "modified" state after invalidating all shared copies, or it can be changed to the "shared" state by writing the revision back to main memory.

[0071] The memory barrier instruction provides an ordering function that ensures that all memory instructions initiated before the memory barrier instruction are completed, and that subsequent memory instructions are not initiated until after the memory barrier instruction is completed. When the memory barrier instruction is completed, all memory accesses initiated before the memory barrier instruction have completed.

[0072] Memory barrier instructions may be necessary because modern processors may employ performance optimizations that result in out-of-order execution. This reordering of memory operations such as loads and stores may go unnoticed within a single thread of execution. However, in the case of concurrent programs and device drivers, such performance optimizations may result in unpredictable behavior unless adequately controlled.

[0073] A shared memory system is configured to allow weakly ordered execution of load and store memory access instructions between processor devices. An instruction sequence of a processor includes a memory barrier instruction that enforces execution of a prior access instruction of the sequence by a processor device and coherency propagation across the shared memory system prior to execution of a subsequent instruction of the sequence. Embodiments may have the beneficial effect of implementing memory barrier instructions with fast execution while using a small amount of computer resources.

[0074] Typical coherency propagation may invalidate one or more cached copies of the previous value of a memory line.

[0075] According to an embodiment, a shared memory system that allows weakly ordered execution of load and store memory access instructions across processor devices includes a plurality of processors. Each processor includes an unpropagated buffer for storing unpropagated memory lines. The unpropagated buffer tracks memory line addresses previously accessed by the processor for which propagation consistency to other processors is unknown. In addition, each processor includes a memory barrier instruction that is implemented by stopping the processor from executing subsequent memory access instructions of its instruction sequence until the unpropagated buffer is empty.

[0076] According to an embodiment, a message relating to a memory line to be sent by a coherence directory to a processor cache memory N includes a propagate field indicating whether memory coherency for the memory line has been propagated. The coherence directory sets the propagate field of the message to no if a temporary entry of the coherence directory for the memory line expects the coherence directory to receive an invalidation confirmation message from one or more processor caches that cached an earlier copy of the memory line.

[0077] If the directory entry for the memory line or the temporary entry for the directory expects that the coherence directory has not received invalidation acknowledgment messages from one or more processor caches that cached an earlier copy of the memory line, the coherence directory sets the propagate field of the message to "yes". If the line has a temporary entry and the propagate field of the message is set to "yes", the coherence directory sets the propagate notification field of cache N in the temporary entry for the coherence directory for the line to no.

[0078] If the Propagate field of the message is set to No, then the coherency directory sets the Propagate Notification field for cache N to Yes in the transient entry of the coherence directory for the memory line.

[0079] If the coherence directory updates the transient entry for a memory line, and if the updated transient entry expects that the coherence directory has not received an invalidation acknowledgment message from a processor cache memory that cached an earlier copy of the memory line, the coherence directory sends a message of the propagated memory line to each processor cache with its propagate notification field set to "yes" in the transient entry for the memory line.

[0080] For each such propagate message sent by the coherence directory to the processor cache memory, the coherence directory sets the corresponding propagate notify field for the processor cache memory to No in the transient entry.

[0081] According to an embodiment, a coherence directory updates a transient entry for a memory line as part of a coherence directory action caused by the coherence directory receiving a message related to the memory line from a processor cache memory.

[0082] According to an embodiment, the identifier of the tag memory line stored in the cache includes the storage address of the corresponding memory line in the main memory. The embodiment can have the beneficial effect of providing an effective identifier for identifying the memory line. Since the memory address is already used to identify the memory line in the main memory and in the processor cache, no additional logic for translating or interpreting the identifier is required.

[0083] According to an embodiment, memory consistency of a shared storage system is achieved using a consistency directory.Embodiments may have the beneficial effect of providing a central unit in the form of a consistency directory that may control, check and validate the propagation of consistency across a shared storage system.

[0084] According to an embodiment, at least one of the first and second flag information is provided by a message received by a processor device from a consistency directory. An embodiment may have the beneficial effect that the consistency directory confirms the propagation of consistency on the shared storage system. According to an embodiment, at least one of the first and second flag information is provided by a message sent by a processor device to the consistency directory. An embodiment may have the beneficial effect of announcing a requirement to confirm the propagation of consistency on the shared storage system.

[0085] According to an embodiment, broadcasting is used to achieve memory consistency of a shared memory system. An embodiment may have the beneficial effect that a broadcast-based protocol for achieving consistency of a shared memory system does not require a central consistency directory. In addition, a central consistency directory does not have to be checked and maintained in order to achieve memory consistency. Therefore, if sufficient bandwidth is available, a broadcast-based protocol for achieving memory consistency may be faster than a directory-based protocol because all transactions of a memory line include requests / requests visible to all processors. A disadvantage of a broadcast-based protocol is that it lacks scalability. Each request must be broadcast to all nodes in the system, i.e., all processor caches, which means that as the system becomes larger, the bandwidth required to implement the method continues to grow.

[0086] According to an embodiment, the instruction to cause the processor device to populate the copy of the memory line to the processor cache memory is an instruction specifying a write access to the corresponding memory line. According to an embodiment, the propagation of memory coherence on the shared memory system includes invalidating other copies of the memory line used by remote processor cache memories of other processor devices of the shared memory system. Embodiments can have the beneficial effect of ensuring that no other processor device uses an outdated shared read copy of the memory line, i.e., a stale copy, while the processor device writes a new copy of the corresponding memory line.

[0087] According to an embodiment, the identifier of the tag memory line stored in the cache includes the storage address of the corresponding memory line in the main memory. Embodiments may have the beneficial effect of using a common identifier to identify memory lines in the main memory, in the processor cache memory, and in the cache of the processor device.

[0088] According to an embodiment, the modification requested by the coherence directory includes invalidating another copy of a memory line included in a remote processor cache memory. Embodiments may have the beneficial effect of ensuring that any of the remote processor cache memories do not use an outdated copy (i.e., a stale copy) of a memory line while the processor device fills a new copy of the corresponding memory line into the processor cache memory. Thus, using incoherent copies of memory lines and introducing race conditions may be effectively avoided.

[0089] Figure 1 A computer system 100 suitable for implementing a shared storage system is depicted. It will be appreciated that the methods described herein are automated by a computer system. The methods may further be implemented in software 112, 122 (including firmware 122), hardware (processor) 105, or a combination thereof. In an exemplary embodiment, the methods described herein are implemented in the form of software as executable instructions and may be used by a special or general purpose digital computer such as a personal computer, workstation, minicomputer, or mainframe computer. Thus, the most general system 100 includes a general purpose computer 101.

[0090] In an exemplary embodiment, in terms of hardware architecture, Figure 1 As shown, the computer 101 includes a processor 105, a memory (main memory) 110 coupled to a storage controller 115, and one or more input and / or output (I / O) devices (or peripheral devices) 10, 145 which are communicatively coupled via a local input / output controller 135. The input / output controller 135 can be, but is not limited to, one or more buses or other wired or wireless connections, such as are known in the art. The input / output controller 135 can have additional elements to simplify communication, which are omitted for simplicity, such as controllers, buffers, caches, drivers, repeaters, and receivers to enable communication. In addition, the local interface can include address, control, and / or data connections to enable appropriate communication between the above components. As described herein, the I / O devices 10, 145 can generally include any universal cryptographic card or smart card known in the art.

[0091] Processor 105 is a hardware device for executing software, in particular, a processor for executing software stored in memory 110. Processor 105 can be any custom or commercially available processor, a central processing unit (CPU), an auxiliary processor in a plurality of processors associated therewith. Computer 101, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, or any device generally used to execute software instructions. Processor 105 may include one or more local cache memories and / or be communicatively connected thereto. Processor 105 may include one or more processor cores. Each processor core may be operably connected to one or more local processor cache memories. Processor 105 may be implemented in the form of multiple processor chips, each of which may include one or more local processor cache memories. Processor cache memories may each include a cache memory controller for controlling the corresponding cache memory. In the case of multiple processor cache memories, memory consistency may be implemented in the form of directory-based memory consistency or broadcast-based memory consistency.

[0092] The memory 110 may include a volatile storage element (e.g., a random access memory (RAM, such as DRAM, SRAM, SDRAM, etc.)) and a non-volatile storage element (e.g., any one or a combination of ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and programmable read-only memory (PROM). Note that the memory 110 may have a distributed architecture in which the various components are remote from each other but can be accessed by the processor 105.

[0093] The software in the memory 110 may include one or more separate programs, each of which includes an ordered list of executable instructions for implementing logical functions (especially the functions involved in the embodiments of the present invention). Figure 1 In the example of , the software in memory 110 includes instructions or software 112 .

[0094] The software in the memory 110 will typically also include a suitable operating system (OS) 111. The OS 111 essentially controls the execution of other computer programs such as possible software 112.

[0095] The methods described herein may be provided in the form of a source program 112, an executable program 112 (object code), a script, or any other entity including a set of instructions 112 to be executed. When provided as a source program, the corresponding program needs to be translated via a compiler, assembler, interpreter, etc., which may or may not be included in the memory 110, in order to operate correctly in association with the OS 111. In addition, the methods may be written in an object-oriented programming language with data and method classes, or a procedural programming language with routines, subroutines, and / or functions.

[0096] In an exemplary embodiment, a conventional keyboard 150 and a mouse 155 may be coupled to the input / output controller 135. Other output devices such as I / O devices 145 may include input devices, such as but not limited to printers, scanners, microphones, and the like. Finally, the I / O devices 10, 145 may also include devices that communicate both input and output, such as but not limited to network interface cards (NICs) or modulators / demodulators (for accessing other files, devices, systems, or networks), radio frequency (RF) or other transceivers, telephone interfaces, bridges, routers, and the like. The I / O devices 10, 145 may be any common cryptographic card or smart card known in the art. The system 100 may further include a display controller 125 coupled to the display 130. In an exemplary embodiment, the system 100 may further include a network interface for coupling to a network 165. The network 165 may be an IP-based network for communicating between the computer 101 and any external servers, clients, and the like via a broadband connection. The network 165 sends and receives data between the computer 101 and the external system 30, which may involve performing some or all of the steps of the methods discussed herein. In an exemplary embodiment, network 165 may be a managed IP network managed by a service provider. Network 165 may be implemented wirelessly, for example, using wireless protocols and technologies such as WiFi, WiMax, etc. Network 165 may also be a packet-switched network, such as a local area network, a wide area network, a metropolitan area network, an Internet network, or other similar types of network environments. Network 165 may be a fixed wireless network, a wireless local area network (LAN), a wireless wide area network (WAN), a personal area network (PAN), a virtual private network (VPN), an intranet, or other suitable network system, and include devices for receiving and sending signals.

[0097] If the computer 101 is a PC, a workstation, an intelligent device, etc., the software in the memory 110 may further include a basic input output system (BIOS) 122. The BIOS is used to initialize and test the hardware at startup, start the OS 111, and support data transfer between hardware devices. The BIOS is stored in the ROM so that the BIOS can be executed when the computer 101 is activated.

[0098] When the computer 101 is in operation, the processor 105 is configured to execute software 112 stored in the memory 110, to communicate data with the memory 110, and to generally control the operation of the computer 101 according to the software. The processor 105 reads all or part of the methods and OS 111 described herein, but usually the latter, which may be cached within the processor 105, and then executes. The memory 120 may include disk storage such as HDD storage.

[0099] Figure 2An exemplary consistent shared memory system 200 in the form of a data processing system is shown that uses a consistent directory 210 to implement directory-based consistency. The consistent shared memory system 200 includes a plurality of N processor cores 202, 252, each core connected to a local processor cache memory 204, 254, and data is exchanged between the processor cache memories 204, 254 via an interconnect structure 206. The interconnect structure 206 further connects the processor cache memories 204, 254 to the consistent directory 210. The consistent directory 210 keeps track of the contents of the processor cache memories 204, 254, i.e., the memory lines included therein, thereby achieving consistency. In addition, the interconnect structure 206 connects the cache memories 204, 254 to a memory storage interface 212. Via the memory storage interface 212, data can be received from a backing memory in the form of a shared memory such as a main memory and sent to the backing memory. For example, processor core 1 attempts to retrieve a memory line from processor cache memory 1, resulting in a cache miss because processor cache memory 1 does not contain a valid copy of the requested memory line, or processor core 1 may, for example, request a write copy, but processor cache memory 1 can only contain a read copy. Processor core 1 may send a request for a valid copy of the desired memory line to coherence directory 210 via interconnect structure 206. Coherence directory 210 checks whether any of the other processor cache memories include the requested memory line. If one of the other processor caches contains the required memory line, coherence directory 210 may forward the request to the respective other processor cache memory that includes the desired memory line. If no other processor cache memory in the cache memory includes the required memory line, coherence directory 210 may forward the request to the backing memory via memory storage interface 212. Thus, a valid copy of the required memory line may be transferred to the requesting processor core 1, which populates the corresponding memory line in cache memory 1.

[0100] Figure 3 An exemplary consistent shared memory system 200 in the form of a data processing system is shown, which includes a plurality of processor cores 202, 252, each processor core having a local processor cache memory 204, 254. Memory consistency is achieved based on broadcast. Figure 2 In contrast to the shared storage system, Figure 3The coherent shared memory system 200 does not include a coherence directory. In the case of broadcast-based coherence, in the case of a cache miss, the processor core 1 broadcasts a request for a valid copy of the desired memory line to the other processor cores 2 to N of the associated shared memory system 200 via the interconnect structure 206. If the processor core 1 does not receive a reply including the requested valid copy of the desired memory line, the processor core 1 may request the copy from the backing memory via the interconnect 206 and the memory storage interface 212.

[0101] Figure 4 An exemplary consistent shared memory system 200 in the form of a data processing system including a plurality of processor chips 201, 251 is shown. Each processor chip has a local processor cache memory 204, 254. The cache memory 204 is implemented based on broadcast. An interconnect structure 206 enables communication between the cache memory 204, 254 and a memory storage interface 212. The memory storage interface 212 may be provided, for example, by a memory subsystem chip. It may be similar to Figure 3 Method to retrieve a valid copy of the desired memory line.

[0102] Figure 5 An exemplary coherent shared memory system 200 in the form of a data processing system including a plurality of processor chips 201, 251 is shown. Each processor chip has a local processor cache memory 204, 254. The cache memory 204 is implemented using a coherent directory 210 provided by a memory subsystem chip 203. The memory subsystem chip 203 also provides an interconnect structure 206 that enables communication between the cache memories 204 and 254, the coherent directory 210, and a memory storage interface 212. The memory storage interface 212 may be provided, for example, by the memory subsystem chip 203. It may be similar to Figure 2 Method to perform retrieval of a valid copy of the desired memory line.

[0103] Figure 6An exemplary method of using barrier instructions to coordinate instructions executed by a processor device (e.g., processor chip 201) is shown. Processor chip 201 includes an instruction execution unit 246 that executes instructions. The executed instruction may require a copy of a memory line. The instruction may be, for example, a read or write instruction. The instruction execution unit 246 checks whether the processor cache memory 204 includes the required memory line. In the event of a cache memory miss, a request is sent to the shared memory 214 to request a copy of the required memory line so that the copy is filled into the processor cache memory 204. Messages sent to the shared memory 214, such as requests or commands, and messages received from the shared memory 214 are checked in step 302 for propagation of the memory consistency across the shared memory system 200 for the filling of the copy of the memory line. For example, a reply including the requested copy of the memory line may be received from the shared memory 214. The copy of the memory line provided by the reply is filled into the processor cache memory 204, for example, in order to complete the execution of the above-mentioned instruction using prefetching or to accelerate the execution of potential subsequent instructions. In step 302, the message itself or a copy of the message generated for the check may be used. The messages may each include a consistency propagation field that is checked in step 304. In the case where the consistency propagation field includes flag information indicating that the memory consistency propagation of the filling of the copy of the memory line is not confirmed, an indicator of the corresponding memory line, for example, its memory address, is added to the buffer 248 of the processor chip 201 in step 306. The buffer 248, also called an unpropagated buffer, may include an unordered set of addresses of memory lines for which the propagation of memory consistency across the shared memory system 200 has not been confirmed. In the case where the consistency propagation field includes flag information indicating that the propagation of memory consistency of the filling of the copy of the memory line is confirmed, the indicator of the corresponding memory line (if present) is removed from the buffer 248 of the processor chip 201 in step 308. The consistency propagation field includes information indicating that no action is required with respect to the buffer 248, and the action 310 is not performed.

[0104] The instruction execution unit 246 based on the processor chip 201 executes the memory barrier instruction, and the barrier request is sent to the buffer 248, and the buffer 248 is checked whether it includes any indicator indicating a memory line, and the indicator indicates that the memory line with consistency propagation on the consistent shared memory system 200 has not been confirmed. If the buffer 248 includes any indicator, then in step 300, wait until the buffer 248 is empty. As long as the buffer 248 is not empty, the execution of the barrier instruction is blocked. In the case of determining that the buffer 248 is empty, the barrier response is sent to the instruction execution unit 246 to notify the instruction execution unit 246 that the buffer 248 is empty. Once the buffer 248 is empty, the instruction execution unit 246 completes the execution of the barrier instruction.

[0105] Figure 7 An exemplary consistency directory 210 of a consistency shared memory system 200 is shown. The consistency shared memory system 200 includes a plurality of processor cache memories 204, 244, which are communicatively connected to a shared memory 214 via an interconnect structure 204. The shared memory 214 may include a plurality of memory hubs 270, 280. In the case of a cache miss, a request from the processor cache memories 204, 244 is sent to the consistency directory 210 via the interconnect structure 206. The consistency directory 210 includes a directory controller 272 that controls the consistency directory 210. The consistency directory 210 may further include a directory table 274. The directory table 274 may include a plurality of entries, each entry having a memory line. In other words, each entry of the directory table 274 may be assigned to a memory line, and one or more processor cache memories 204, 244 use a copy thereof. The entry may include an address tag with a storage address of the memory line to which the corresponding entry is assigned, state information, access type information, and ID information. The state information may indicate whether the state of the copy of the memory line in the processor cache memory 204, 244 is stable, i.e., T=0, or whether a change is currently being made so that the state is transient, i.e., T=1. In a stable state, nothing is currently changing. The entry may also identify the type of access of the processor cache memory 204, 244 to the memory line, such as shared read access or exclusive write access. In addition, the processor cache memory 204, 244 currently using the copy of the memory line may be identified by ID information, for example, the ID information may include an identifier of a processor operably connected to the processor cache memory 204, 244 to be identified and / or an identifier of the respective processor cache memory 204, 244 itself.

[0106] In the case where the state of the copy of the memory line in the processor cache memory 204, 244 is transient, i.e., the copy of the memory line is currently being changed in order to establish consistency through the processor cache memory 204, 244, the directory table entry may include a pointer to an entry in the transient table 276. For example, in the case where one of the processor cache memories 204, 244 requests an exclusive write copy of a memory line, the shared read copy used by the other processor cache memories 204, 244 must be invalidated. The consistency directory 210 may include a transient table 276 to track the consistency propagation across the shared memory system for each memory line. If the copy of a memory line is in a transient state, an entry may be generated for the corresponding memory line in the transient table 276. The transient table 276 may include multiple entries, each entry having a memory line. In other words, each entry of the transient table 276 may be assigned to a memory line where one or more processor cache memories 204, 244 use a transient copy. The entries of the transient table 276 may identify all processor cache memories 204, 244 that are currently using a copy of the corresponding memory line. For each of these processor cache memories 204, 244, the entry may track the latest state of the copy of the memory line used by the corresponding processor cache memory 204, 244, such as shared read, exclusive write, or invalid. In addition, the entry may track requests received from the respective processor cache memory 204, 244 and outstanding requests sent by the consistency directory to the respective processor cache memory 204, 244, i.e., requests that require a reply. Finally, for each of these processor cache memories 204, 244, the transient table entry may include an indicator indicating whether the respective processor cache memory 204, 244 must be notified once the propagation of memory consistency on the shared memory system is confirmed. Information about memory lines is obtained through the consistency directory 210. The transient table 276 may be accompanied by a bit vector 278, which indicates which entries of the transient table are currently free and which are currently used to track the propagation of consistency. A memory line. A request that requires a reply. Finally, for each of these processor cache memories 204, 244, the transient table entry may include an indicator indicating whether the respective processor cache memory 204, 244 must be notified once the propagation of memory coherency on the shared memory system is confirmed. Information about memory lines is obtained through the coherency directory 210. The transient table 276 may be accompanied by a bit vector 278 indicating which entries in the transient table are currently free and which are currently used to track the propagation of memory line coherency.

[0107] Upon receiving a request for a copy of a memory line from one of the processor cache memories 204, 244, the directory controller 272 of the coherence directory 210 may use the directory table 274 to determine whether one of the remaining processor cache memories 204, 244 includes a valid copy of the requested memory line. In the event that one of the remaining processor cache memories 204, 244 includes a valid copy of the requested memory line, the directory controller 272 may forward the request to the corresponding processor cache memory 204, 244 to retrieve a copy of the requested memory line. If neither of the remaining processor cache memories 204, 244 contains a valid copy of the requested memory line,

[0108] Figure 8 An exemplary method for managing a coherence directory 210 for implementing coherence of a shared memory system is shown. In step 400, the coherence directory 210 receives a message from a processor cache memory (e.g., C0) of the shared memory system via the interconnect structure 206. The message may be related to a memory line L. For example, the message may be memory line L. A directory controller 272 of the coherence directory 210 may check in step 402 whether there is a need to send a message M to the processor cache memory C0, such as in response to the received message. For example, DC may need to send a message M to the processor cache memory C0 in the form of a reply containing the requested memory line L. If there is no need to send the message M, the directory controller 272 may continue to process the received message according to its content. In the event that the message M needs to be sent, the method may continue with step 404. In step 404, the directory controller 272 checks whether the transient table 276 includes a transient entry for the memory line L. To do this, the directory controller 272 may first check the directory table 274. If the directory table 274 has an entry for the memory line L, the directory controller 272 may check whether the respective entries of the directory table 274 include a pointer to the transient table 276 of entries assigned to the memory line L. 276 does not contain a transient entry for the memory line L, and the method may continue to execute step 410. In step 410, a marking information may be added to the message M to indicate that the propagation of the memory consistency on the shared memory system regarding the memory line L has been confirmed, that is, Prop=Yes. Since the shared memory system does not include the transient state of the copy of the memory line L, the memory consistency of the memory line L is stable and does not need to be propagated. In step 416, the message M is sent to the processor cache memory C0 via the interconnect structure 206. In step 418, the processor cache memory C0 receives the message M from the consistency directory 210.

[0109] If it is determined in step 404 that the transient table 276 includes a transient entry for memory line L, the method may continue to step 406. The transient entry for memory line L means that coherency regarding memory line L must be propagated across the shared memory system on memory line L. In step 406, the transient table 276 is used to check whether there are actions to be performed by one or more processor cache memories of the shared memory system to establish coherency. For example, it may be expected that one or more valid copies of memory line L must be invalidated in order to establish coherency. If there are no remaining actions to be performed, the method may continue in step 408. Since the memory line L is in a transient state, i.e., coherency has not yet been confirmed, the coherency propagation notification indicator of the processor cache memory C0 in the transient entry of the memory line L in the transient table 276 may indicate that once the directory controller 272 of the coherency directory 210 obtains such confirmation, confirmation of the coherency propagation must be provided to the processor cache C0. No remaining actions indicate that the propagation of the coherency has been successfully completed, i.e., confirmation of the propagation is provided. In step 408, if the acknowledgement is notified to the processor cache memory C0, the propagation notification indicator for the processor cache memory C0 may be modified so that it indicates that an acknowledgement does not need to be provided to the processor cache memory C0. In step 410, the corresponding acknowledgement is added to the message M. The message M is then sent to the processor cache memory C0, as shown in steps 416 and 418.

[0110] In the event that the check in step 406 indicates that there are still actions to be performed, the method may continue with step 412. Such actions to be performed may be indicated by an outstanding request recorded in the transient entry of the memory line L, i.e., an identification request that has not yet been performed or confirmed. In step 412, it is ensured that the consistency propagation notification indicator of the processor cache memory C0 in the transient entry of the memory line L in the transient table 276 indicates that confirmation of the propagation of consistency must be provided to the processor cache memory C0 as soon as possible. Because such confirmation is obtained by the directory controller 272 of the consistency directory 210. In step 414, tag information may be added to the message M to indicate that the propagation of memory consistency across the shared memory system regarding the memory line L has not yet been determined, i.e., Prop=No. Then, as shown in steps 416 and 418, the message M is sent to the processor cache memory C0.

[0111] Fig. 9 shows the use of Figure 8. In step 500, the coherence directory 210 receives a message from a processor cache memory (e.g., C0) of the shared memory system via the interconnect structure 206. The message may be related to a memory line L. For example, the message may be a request for a copy of the memory line L. In step 502, the directory controller 272 of the coherence directory 210 checks whether the directory controller 272 has updated the transient entry of the memory line L in the transient table 276 using the directory table 274 and the transient table 276. If the directory controller 272 determines that there is no update of the transient entry of the memory line L in the transient table 276, the directory controller 272 continues to process the received message according to its content. If the directory controller 272 has updated the transient entry of the memory line L in the transient table 276, the method may continue to step 504. In step 504, the transient table 276 is used to check whether one or more actions to be performed in order to establish coherence are present in one or more of the memory areas of the processor cache memories of the shared memory system. For example, it is checked whether a response is expected based on the transient entry for the memory line L, the response confirming that the copy of the memory line L used by one of the processor cache memories of the shared memory system is invalid. If the directory controller 272 determines that an action to establish coherence is expected, the directory controller 272 may, based on its contents, for example, based on the transient entry for the memory line L. Figure 8 The method continues to process the received message. If the directory controller 272 determines that no action is expected to be performed to establish consistency, the method can continue to step 506. In step 506, the directory controller 272 checks whether the transient entry of the memory line L includes a consistency propagation notification indicator, which is used to identify any processor cache memory CX of the shared memory system, X∈{0,1,…,N} to confirm the situation of being notified when the consistency is propagated on the shared memory system regarding the memory line L. In the case where the directory controller 272 determines one or more processor cache memories CX, a message is sent to each processor cache memory CX in the form of the command "D2Ccmd.Propagated(addr A, Prop=Yes)" in step 508 to indicate that the consistency propagation across the shared memory system regarding the memory line L with memory address A has been confirmed, that is, Prop=Yes. In step 512, the message is received by the processor cache memory CX. In step 510, the coherency propagation notification indicator for the notified processor cache memory CX is changed such that said indicator no longer indicates an information requirement for the processor cache memory CX.

[0112] Fig. 10A and 10BAn exemplary method is shown for coordinating the execution of an instruction sequence using barrier instructions by a processor of a consistent shared memory system. The consistency of the shared memory system can be achieved using a consistency directory. A processor of the shared memory system, namely P0, can execute the instructions of the instruction sequence. The execution of the instructions can include writing a memory line with a memory address A, that is, filling a modified copy of the memory line with the memory address A into the processor cache memory of P0. The memory barrier instruction included in the instruction sequence is executed by P0. Until the propagation of the memory consistency of the modified copy of the memory line with the memory address A on the entire shared memory system is completed and confirmed, the execution of the memory barrier instruction is blocked from completion. For example, the effect of the write by P0 must be propagated to another processor of the shared memory system, such as P1. Initially, no processor cache can hold a copy of the memory line with address A. Therefore, the directory table on the memory hub with the consistency directory may not contain an entry referencing the memory line memory address A.

[0113] In step 600, processor P1 may execute a read instruction "load A" which misses P1's local processor cache C1. Therefore, C1 issues a request to the coherence directory, such as C2Dreq.I2S(Addr=A, Prop=NA), which contains information indicating that no action (NA) needs to be taken on P1's unpropagated cache area UB1. The request C2Dreq.I2S(Addr=A, Prop=NA) is a request (req) sent from the processor cache (C) to the coherence directory (D) to request a shared read copy (S) of the memory line with address A. In step 602, the directory controller of the coherence directory, i.e., DC, receives the request C2Dreq.I2S(Addr=A, Prop=NA). DC cannot find an entry for the memory address A in the directory table of the coherence directory, i.e., DT. Therefore, DC creates a new entry in DT of type stable, i.e., T=0, and a read type in the shared type, i.e., W=0. The resulting entry in DT may be of the form [Addr=A, T=0, W=0, P1 is set in the ID field]. DC uses the memory controller to obtain the requested data from the main memory, i.e., a copy of the memory line at address A. In step 604, DC sends a reply D2Creply.I2S(Addr=A, data, Prop=yes) to P1. In step 606, P1 receives the reply D2Creply.I2S(Addr=A, data, Prop=yes), which removes address A from UB1 if it exists. The reply D2Creply.I2S(Addr=A, data, Prop=yes) causes the local processor cache C1 to establish an entry for a shared read copy of the memory line address A, and satisfies "load A" by filling the received shared read copy of the memory line address A into C1.

[0114] In step 608, it is assumed that after completing the above actions, P0 executes the instruction "store A, value", that is, the instruction to write the value to the memory line with address A, and the instruction is lost in C0 of P0. Therefore, C0 issues a request C2Dreq.I2E (Addr=A, Prop=NA) with no operation (NA) on the unpropagated buffer area UB0 of P0, and sends the request to the consistency directory. In step 610, DC receives the request C2Dreq.I2E (Addr=A, Prop=NA) for the exclusive write copy of the memory line with memory address A. For the memory line with memory address A, DC finds the stable point established above in DT. , the entry of shared read [Addr=AT=0.W=0. P1 is set in the ID field]. In step 612, DC sends a request D2Creq.S2I (addr=A, Prop=Yes in order for P1 to invalidate the shared read copy of the memory line with memory address A in C1). In addition, the DC selects any unused entry (i.e., TT) in the transient table and marks its index as TTindexA, for example. The DC changes the DT entry to a transient type, i.e., T=1, resulting in the entry [Addr=AT=1.Pointer=TTindexA.]. In addition, the DC sets the TT entry at TTindexA to [P0(E, -, -, yes).P1(S, -, S2I, no).P2(I, -, -, no)] so that the transient is encoded for each processor cache according to the syntax (LATEST_STATE, RECEIVED_REQUEST, SENT_REQUEST, PropNotify?). The TT entry indicates that P0's C0 includes an exclusive write copy and that P0's C0 must be notified after confirmation of memory consistency propagation across the shared storage system when exclusive write copies are filled into C0 is completed. P1 contains a shared read copy, but the consistency directory waits for confirmation of the execution of an invalidation request sent to P1's C1. Relative to filling dedicated write copies into C0, there is no need to notify P1's C1 of consistency propagation. Finally, the processor cache memory C2 of another processor P2 of the shared memory system includes an invalid copy of the memory line with address A and does not have to be informed of the propagation of consistency. DC uses the memory controller to obtain the requested data, i.e., a copy of the memory line with address A in the main memory. In step 614, DC sends a reply D2Creply.I2E(Addr=A, data, Prop=No) to processor P0. The reply to P0 includes Prop=No, and the transient entry for P0 includes PropNotify=Yes, because the transient entry for the memory line with address A identifies at least one processor other than P0, i.e., P1, with an outstanding SENT_REQUEST.In this case, P1 includes SENT_REQUEST=S2I.

[0115] In step 616, P1 receives a request D2Creq.S2I (Addr=A, Prop=Yes) which requests to delete address A from UB1 if it exists. The request D2Creq.S2I (A, Prop=Yes) further causes the processor cache C1 to invalidate the shared read copy of the memory line with address A. In step 618, P1 issues and sends a reply C2Dreply.S2I (Addr=A, Prop=NA) to the coherency directory to confirm that the shared read copy of the memory line with address A is invalid. The reply indicates that no action needs to be taken on UB1, i.e., NA.

[0116] In step 620 , P0 receives a reply D2Creply.I2E(Addr=A, Data, Prop=No) which adds an exclusive write copy of the memory line with address A to UB0 .

[0117] ReplyD2Creply.I2E(Addr=A, data, Prop=No) causes C0 to create an entry for a dedicated write copy of the memory line with address A and fill the corresponding memory line to C0. In addition, the instruction "store A, value" is executed by storing the corresponding value in the copy of the memory line with address A. On the one hand, steps 612, 616 and 618 are continued, and on the other hand, steps 614 and 620 are continued, both occurring simultaneously or in any order.

[0118] In step 622, the directory controller DC receives a reply C2Dreply.S2I (Addr=A, Prop=NA) from the processor P1. For the memory address A, DC finds an entry of transient type in DT, namely T=1, [Addr=AT=1.Pointer=TTindexA, which has a pointer to a TT entry [P0(E, -, -, yes). P1(S, -, S2I, no). P2(I, -, -, no)]. The reply C2Dreply.S2I (Addr=A from processor P1, Prop=NA) causes DC to update the P1 value in the TT entry. The updated TT entry becomes [P0(E, -, -, yes). P1(I, -, -, no). P2(I, -, -, no)]. The updated TT entry does not include any outstanding SENT_REQUEST to any processor of the shared storage system, thereby confirming the propagation of consistency on the shared storage system. In addition, the updated TT entry includes a value P0 (E, -, -, yes), which requests notification of P0 when the propagation of consistency is confirmed.

[0119] In step 624, DC sends the command D2Ccmd.Propagated(Addr=A, Prop=Yes) to processor P0. In addition, DC clears P0 of PropNotify, that is, sets the TT entry to [P0(E, -, -, No). P1(I, -, -, No). P2(I, -, -, No)]. The TT entry does not describe any transient state because there is no outstanding SENT_REQUEST for any processor of the shared memory system. In other words, each TT entry describes a stable state. Therefore, DC sets the entry in DT of the memory line with address A to type stable, i.e., T=0, and type to write differently, i.e., W=1. The resulting DT entry is [Addr=A. T=0. W=1. P0 is set in the ID field]. Finally, DC clears the TT input with TTindexA and returns TTindexA to the unused entry 278 in the transient table.

[0120] At step 626, P0 executes a memory barrier instruction. The memory barrier instruction may, for example, require that all write instructions issued previously by the instruction sequence be complete. The memory barrier instruction will be prohibited from completing until UB0 is empty. Since P0 has not yet received

[0121] D2Ccmd.Propagated(Addr=A, Prop=Yes), so UB0 still contains address A, and execution of the memory barrier instruction is prevented. In step 628, C0 receives the reply D2Ccmd.Propagated(Addr=A, Prop=Yes), which removes address A from UB0. Therefore, in step 628, UB0 on ​​P0 is empty, and the memory barrier instruction is completed. In step 630, execution of the instruction sequence on P0 continues.

[0122] An exemplary Directory-to-Cache (D2C) message may for example comprise a request "D2Creq()", a reply "D2Creply()" and a command "D2Ccmd()". A request D2Creq() requires a subsequent reply C2Dreply(). A reply D2Creply() is sent in response to a prior request C2Dreq(). A command D2Ccmd() has no subsequent reply.

[0123] An exemplary cache-to-directory (C2D) message may, for example, include a request "C2Dreq()", a reply "C2Dreply()" and a command "C2Dcmd()". A request C2Dreq() requires a subsequent reply C2Dreply(). A reply C2Dreply() is a response sent to a prior request C2Dreq(). A command C2Dcmd() has no subsequent reply.

[0124] When a processor device sends or receives a memory access type message, i.e., a message related to filling a memory line into a processor cache, a consistency propagation check unit of the processor device may check the consistency propagation notification field of the memory access type message to determine whether an operation needs to be performed on an unpropagated cache area of ​​the processor device, and which operations need to be performed. Memory access type messages may also include messages related to the invalidation of memory lines in the processor cache memory. Possible values ​​of the consistency propagation notification field in the memory access type message may include: "Prop = Yes" causes the memory address of the memory line in the corresponding message to be deleted from the unpropagated cache area (if any). "Prop = No" causes the storage address of the memory line in the corresponding message to be added to the unpropagated cache area. "Prop = NA" does not perform any operation on the unpropagated cache area.

[0125] In the following table, an exemplary directory-to-cache (D2C) request is provided:

[0126]

[0127] In the following table, an exemplary Directory-to-Cache (D2C) reply is provided:

[0128]

[0129]

[0130] In the following table, exemplary directory-to-cache (D2C) commands are provided:

[0131] information prop value Comment D2Ccmd.Propagated(addr,prop) yes

[0132] In the following table, an exemplary cache-to-directory (C2D) request is provided:

[0133]

[0134] In the following table, an exemplary cache-to-directory (C2D) reply is provided:

[0135]

[0136] In the following table, exemplary cache-to-directory (C2D) commands are provided:

[0137]

[0138]

[0139] Fig.11An exemplary coherent shared memory architecture in the form of a multi-processor computer system in the form of a multi-processor server 200 including a plurality of processor chips 201 is shown. The multi-processor server 200 includes a set of memory hub chips 203. The processor chips 201 may be provided with a plurality of ports 224. According to one embodiment, the number of ports 224 provided by each processor chip 201 may be equal to the number of memory hub chips 203. Each processor chip 201 is provided with a local cache memory 204 for caching memory lines, i.e., memory lines processed by the processor chip 201. The processor chips 201 of the server 200 may be configured identically or may be different. Application software may be executed on one or more processor chips 201, so a given application may implicitly or explicitly utilize and benefit from similar or different processor chips 201.

[0140] Each memory hub chip 203 may be provided with a plurality of local memory modules 234, for example, a dual in-line memory module (DIMM) including a series of dynamic random access memory integrated circuits. Thus, each memory hub chip 203 implements a memory hub device. In addition, each memory hub chip 203 may include a plurality of ports 222. For example, the number of ports 222 of each memory hub chip 203 may be equal to the number of processor chips 201. In addition, for the memory lines stored in the memory modules in the local 236 of each memory hub chip 203, each memory hub chip 203 may include a consistency directory 210 for implementing directory-based consistency for the memory lines stored in the cache memory 204 of one or more processor chips 201. The chips 203 of the server 200, all memory hub chips 20 may be configured identically or similarly, and each memory hub chip 203 performs similar functions. Application software may execute on one or more processor chips 201 , so the performance of a given application typically benefits from having memory serviced by many similar memory hub chips 203 , with each specific memory address being serviced by a single predefined memory hub chip 203 .

[0141] Each processor chip 201 may be communicatively coupled to each memory hub chip 203, for example, via a bidirectional point-to-point communication connection 230, such as a serial communication connection. Thus, each processor chip 201 may be provided with memory access to each memory module 234 local to one of the memory hub chips 203. Access to the memory modules 236 may be provided based on a unified memory access (UMA) architecture. A given memory line, i.e., a memory line, may be stored on one or more memory modules 236 local to the same memory hub chip 203 as a backing memory, i.e., a main memory. A given memory page, including, for example, multiple memory lines, may be interleaved across the memory modules 236 of all memory hub chips 203.

[0142] The computer system may, for example, include 16 processor chips 201 and 128 memory hub chips 203. In this case, each processor chip 201 may include 128 ports 224 to communicatively couple to each memory hub chip 203. Yes, each memory hub chip 203 may be provided with 16 ports 222, so that each memory hub chip 203 may be communicatively coupled to each processor chip 201 via a different point-to-point communication connection 230.

[0143] Various aspects of the present invention are described herein with reference to the flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It will be understood that each frame of the flowchart illustration and / or block diagram and the combination of frames in the flowchart illustration and / or block diagram can be implemented by computer-readable program instructions.

[0144] The present invention may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to execute aspects of the present invention.

[0145] The computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punch card or a raised structure in a groove on which instructions are recorded, and any suitable combination of the foregoing. As used herein, a computer readable storage medium should not be understood as being a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable) or an electrical signal transmitted by a wire.

[0146] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device via a network (e.g., the Internet, a local area network, a wide area network), or downloaded to an external computer or external storage device, a regional network, and / or a wireless network. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0147] The computer-readable program instructions for performing the operation of the present invention can be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data or source code or object codes written in any combination of one or more programming languages ​​(including object-oriented programming languages, such as Smalltalk, C++, etc.) and conventional process programming languages ​​(such as "C" programming language or similar programming languages). The computer-readable program instructions can be executed entirely on the computer of the user's computer system, partially on the computer of the user's computer system, partially on a remote computer as an independent software package, or completely on a remote computer or server. In the latter case, the remote computer can be connected to the computer of the user's computer system through any type of network (including a local area network (LAN) or a wide area network (WAN)), or can be connected to an external computer (for example, by using the Internet of an Internet service provider). In some embodiments, electronic circuits including, for example, programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (PLAs) can personalize computer-readable program instructions by utilizing the state information of computer-readable program instructions. Electronic circuits, to perform aspects of the present invention. The remote computer can be connected to the computer of the user's computer system through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, by accessing the Internet using an Internet service provider). In some embodiments, an electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can personalize the computer-readable program instructions by utilizing the state information of the computer-readable program instructions. Electronic circuit to perform aspects of the present invention. The remote computer can be connected to the computer of the user's computer system through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, by accessing the Internet using an Internet service provider). In some embodiments, an electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can personalize the computer-readable program instructions by utilizing the state information of the computer-readable program instructions. Electronic circuit to perform aspects of the present invention. Or a connection can be established with an external computer (for example, by using an Internet service provider through the Internet). In some embodiments, electronic circuits including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may personalize computer readable program instructions by utilizing state information of the computer readable program instructions. The electronic circuits may perform aspects of the present invention.Alternatively, a connection may be established with an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may be personalized to perform aspects of the present invention by utilizing state information of computer-readable program instructions.

[0148] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It will be understood that each frame of the flow chart diagram and / or block diagram and the combination of frames in the flow chart diagram and / or block diagram can be implemented by computer-readable program instructions.

[0149] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions are executed via the processor of the computer or other programmable data processing device, creating a device for implementing the functions / actions specified in the flow chart and / or block diagram blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium, which may direct the computer, programmable data processing device, and / or other device to function in a particular manner.

[0150] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing device, or other device, so that a series of operating steps are performed on the computer, other programmable device, or other device to produce an implemented computer processing process, so that the instructions executed on the computer, other programmable device, or other device implement the functions / actions specified in the flowchart and / or block diagram blocks.

[0151] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention.In this regard, each square frame in the flow chart or block diagram can represent a module, a fragment or a part of an instruction, which includes one or more executable instructions for realizing the specified logical function.In some alternative embodiments, the function pointed out in the square frame may not occur in the order pointed out in the figure.For example, depending on the function involved, two frames shown in succession can actually be performed substantially simultaneously, or these frames can be performed in reverse order sometimes.

[0152] Possible combinations of the above features can be as follows:

[0153] 1. A method for coordinating an instruction sequence comprising a plurality of instructions executed by a processor device of a coherent shared memory system, wherein the processor device is operably connected to a processor cache memory, the method comprising:

[0154] instructions of the sequence of instructions to be executed by the processor device, wherein execution of the instructions causes the processor device to populate the processor cache memory with a copy of a memory line of a main memory of the shared memory system;

[0155] marking, by the processor device, the respective memory line based on detecting that first flag information indicating that memory consistency propagation across the shared memory system regarding the population of the copy of the memory line is not asserted;

[0156] clearing, by the processor device, a flag for each of the memory lines based on detecting that second flag information indicating memory consistency propagation across the shared memory system regarding the population of the copy of the memory line is confirmed;

[0157] Based on executing a memory barrier instruction of the sequence of instructions, the processor device prevents execution of the memory barrier instruction from completing while marking the memory line.

[0158] 2 . The method of claim 1 , wherein completion of execution of the memory barrier instruction is prevented until all memory lines previously marked by the processor device are cleared.

[0159] 3. The method of claim 1 , wherein the marking of the memory line comprises adding an identifier of the memory line to a cache of the processor device, and wherein the clear marking of the marked memory line comprises removing the identifier of the memory line from the cache of the processor.

[0160] 4. The method of claim 3, wherein the identifiers of the tagged memory lines stored in the cache comprise storage addresses of the respective memory lines in the main memory.

[0161] The method of claim 3 , wherein completion of execution of the memory barrier instruction is prevented until the cache of the processor device is empty.

[0162] The method of claim 1 , wherein the memory coherence of the shared memory system is implemented using a coherence directory.

[0163] 7. The method of claim 6, wherein at least one of the first and second flag information is provided by a message received by the processor device from the consistency directory.

[0164] 8. The method of claim 6, wherein at least one of the first and second flag information is provided by a message sent by the processor device to the consistency directory.

[0165] 9. The method of claim 1, wherein broadcasting is used to achieve memory consistency of the shared memory system.

[0166] 10. The method of claim 1, wherein the instructions that cause the processor device to populate the processor cache memory with copies of the memory lines are instructions that specify write access to the respective memory lines.

[0167] 11. The method of claim 1 , wherein the memory coherency propagation across the shared memory system comprises invalidating other copies of the memory line used by remote processor cache memories of other processor devices of the shared memory system.

[0168] 12. A method according to claim 1, wherein a message related to the modification of the contents of the processor cache memory and processed by the processor device includes a consistency propagation field, wherein the consistency propagation field includes one of the following: mark information indicating that the memory consistency propagation of the modification across the shared memory system has not been confirmed; flag information indicating that the memory consistency propagation of the modification across the shared memory system has been confirmed, or flag information indicating that neither a marking operation nor a clearing marking operation is required.

[0169] 13. A processor device of a coherent shared memory system, wherein the processor device is operably connected to a processor cache memory and is configured to coordinate the execution of an instruction sequence including a plurality of instructions, the coordination comprising:

[0170] an instruction to execute an instruction sequence, wherein execution of the instructions causes the processor device to populate the processor cache memory with a copy of a memory line of a main memory of the shared memory system;

[0171] marking, by the processor device, the respective memory line based on detecting that flag information indicating that memory consistency propagation across the shared memory system regarding the population of the copy of the memory line is not asserted;

[0172] clearing, by the processor device, a flag for each memory line based on detecting that flag information indicating memory consistency propagation across the shared memory system for the population of the copy of the memory line is confirmed;

[0173] Based on executing a memory barrier instruction of the sequence of instructions, execution of the memory barrier instruction is prevented from completing while marking the memory line.

[0174] 14. The processor apparatus of claim 13, wherein completion of execution of the memory barrier instruction is prevented until all memory lines previously marked by the processor device are cleared.

[0175] 15. The processor device of claim 13 , wherein the processor device comprises a cache for registering memory lines, copies of the memory lines being populated to the processor cache memory, and memory consistency propagation across the shared memory system for the populating of the respective copies of the memory lines is not confirmed, wherein the marking of the memory lines comprises adding an identifier of the memory lines to the cache of the processor device, and wherein the clear marking of the marked memory lines comprises deleting the identifier of the memory lines from the cache of the processor.

[0176] 16. The processor device of claim 15, wherein the identifiers of the tagged memory lines stored in the cache comprise storage addresses of the respective memory lines in the main memory.

[0177] 17. The processor apparatus of claim 15, wherein completion of execution of the memory barrier instruction is prevented until the cache of the processor device is empty.

[0178] 18. A method for managing a consistency directory for implementing memory consistency of a shared memory system, the method comprising:

[0179] maintaining, by the coherence directory, flag information of a memory line assigned to a main memory of the shared memory system, wherein a processor device of the shared memory system has initiated populating a copy of the memory line into a processor cache memory operatively connected to the processor device, the flag information indicating whether memory coherence propagation across the shared memory system with respect to the populating of the copy of the memory line has been determined;

[0180] The flag information is included in a message sent by the consistency directory to the processor device;

[0181] The message is sent to the processor device via the coherence directory.

[0182] 19. The method of claim 18, wherein the consistency directory initiates the maintenance of the flag information assigned to the memory line based on determining that the contents of a remote processor cache memory of another processor device of the shared memory system need to be modified in order to propagate memory consistency regarding populating the copy of the memory line into the remote cache memory.

[0183] 20. The method of claim 19, wherein the coherency directory sends a request to the remote processor cache memory to request modification of contents of the remote processor cache memory to propagate the memory coherency regarding the population of the copy of the memory line to the remote processor cache memory.

[0184] 21. The method of claim 20, wherein the modification requested by the coherency directory comprises invalidating another copy of the memory line included in the remote processor cache memory.

[0185] 22. The method of claim 19, wherein, based on the propagation of the storage consistency across the shared storage system regarding the population of the replica of the storage line, a consistency directory includes registers for keeping track of the content revisions of the processor caches of the other processor devices of the shared storage system.

[0186] 23. A method according to claim 22, wherein when storage consistency propagation across the shared memory system regarding the population of the copy of the memory line is confirmed, the register includes a consistency propagation notification indicator, the consistency indicator identifying a processor cache memory of a processor device of the shared memory system that is to be notified by the consistency directory.

[0187] 24. A consistency directory for implementing memory consistency of a shared memory system, the consistency directory being configured to:

[0188] maintaining flag information for a memory line assigned to a main memory of the shared memory system, wherein a processor device of the shared memory system has initiated populating a copy of the memory line into a processor cache memory operatively connected to the processor device, the flag information indicating whether memory consistency propagation across the shared memory system has been determined with respect to the populating of the copy of the memory line;

[0189] Including the flag information in a message;

[0190] The message is sent to the processor device.

[0191] 25. A coherent shared memory system comprising a coherent directory for implementing memory coherence of the shared memory system, a main memory, and a processor device operably connected to a processor cache memory, the processor device being configured to coordinate execution of an instruction sequence comprising a plurality of instructions, the coordination comprising:

[0192] an instruction to execute an instruction sequence, wherein execution of the instructions causes the processor device to populate the processor cache memory with a copy of a memory line of a main memory of the shared memory system;

[0193] marking, by the processor device, the respective memory line based on detecting that flag information indicating that memory consistency propagation across the shared memory system regarding the population of the copy of the memory line is not asserted;

[0194] clearing, by the processor device, a flag for each memory line based on detecting that flag information indicating memory consistency propagation across the shared memory system for the population of the copy of the memory line is confirmed;

[0195] based on executing a memory barrier instruction of the sequence of instructions, preventing execution of the memory barrier instruction from completing while marking the memory line;

[0196] and the consistency directory is configured to maintain the flag information assigned to the memory line of the main memory based on initiating the population of the copy of the memory line, the flag information indicating whether the memory consistency propagation across the shared memory system regarding the population of the copy of the memory line has been determined, wherein the maintaining comprises:

[0197] first including flag information indicating that propagation of the memory consistency across the shared memory system regarding the population of the copy of the memory line is not acknowledged as a first message;

[0198] sending the first message to the processor device;

[0199] After obtaining confirmation based on the propagation, updating the flag information and including the updated flag information, the updated flag information indicating that the memory consistency propagation across the shared storage system regarding the population of the copy of the storage message is confirmed as a second message;

[0200] The second message is sent to the processor device.

Claims

1. A method for coordinating an instruction sequence comprising a plurality of instructions executed by a processor device of a coherent shared memory system, wherein: The processor device is operatively connected to a processor cache memory, the method comprising: instructions of the sequence of instructions to be executed by the processor device, wherein execution of the instructions causes the processor device to populate the processor cache memory with a copy of a memory line of a main memory of the shared memory system; receiving, by the processor device, a message from a controller included as part of a shared memory system, the message including flag information indicating whether memory coherency propagation across the shared memory system is confirmed, wherein the controller maintains a transient table and a directory table of memory line entries; marking, by the processor device, respective memory lines based on detecting that the received flag information indicates that memory coherency propagation across the shared memory system with respect to the population of the copies of the memory lines is not acknowledged; clearing, by the processor device, a flag for the respective memory line based on detecting that the received flag information indicates that memory coherency propagation across the shared memory system with respect to the population of the copy of the memory line is confirmed; and Based on executing a memory barrier instruction of the sequence of instructions, the processor device prevents execution of the memory barrier instruction from completing while marking the memory line. 2 . The method of claim 1 , wherein completion of execution of the memory barrier instruction is prevented until all memory lines previously marked by the processor device are cleared.

3. The method according to claim 1, wherein: The marking of the memory line includes adding an identifier of the memory line to a cache of the processor device, and wherein the clear marking of the marked memory line includes removing the identifier of the memory line from the cache of the processor.

4. The method of claim 3, wherein the identifiers of the tagged memory lines stored in the cache comprise storage addresses of the respective memory lines in the main memory. The method of claim 3 , wherein completion of execution of the memory barrier instruction is prevented until the cache of the processor device is empty.

6. The method according to claim 1, wherein: The memory coherence of the shared memory system is achieved using a coherence directory.

7. The method according to claim 6, characterized in that The flag information is provided by a message sent by the processor device to the consistency directory.

8. The method according to claim 1, wherein: Broadcasting is used to achieve memory consistency of the shared memory system.

9. The method according to claim 1, characterized in that: The instructions that cause the processor device to populate the processor cache memory with copies of the memory lines are instructions that specify write access to the respective memory lines.

10. The method according to claim 1, wherein: The memory coherency propagation across the shared memory system includes invalidating other copies of the memory line used by remote processor cache memories of other processor devices of the shared memory system.

11. The method according to claim 1, wherein: A message related to the modification of the contents of the processor cache memory and processed by the processor device includes a consistency propagation field, wherein the consistency propagation field includes one of the following: flag information indicating that the memory consistency propagation of the modification across the shared memory system has not been confirmed; flag information indicating that the memory consistency propagation of the modification across the shared memory system has been confirmed, or flag information indicating that neither a marking operation nor a clearing marking operation is required.

12. A processor device for a consistent shared memory system, wherein: The processor device is operatively connected to a processor cache memory and is configured to coordinate execution of an instruction sequence comprising a plurality of instructions, the coordination comprising: an instruction to execute an instruction sequence, wherein execution of the instructions causes the processor device to populate the processor cache memory with a copy of a memory line of a main memory of the shared memory system; receiving, by the processor device, a message from a controller included as part of a shared memory system, the message including flag information indicating whether memory coherency propagation across the shared memory system is confirmed, wherein the controller maintains a transient table and a directory table of memory line entries; marking, by the processor device, respective memory lines based on detecting that the received flag information indicates that memory coherency propagation across the shared memory system with respect to the population of the copies of the memory lines is not acknowledged; clearing, by the processor device, a flag for the respective memory line based on detecting that the received flag information indicates that memory coherency propagation across the shared memory system with respect to the population of the copy of the memory line is confirmed; and Based on executing a memory barrier instruction of the sequence of instructions, execution of the memory barrier instruction is prevented from completing while marking the memory line.

13. The processor device of claim 12, wherein execution of the memory barrier instruction is blocked from completing until all memory lines previously marked by the processor device are cleared.

14. The processor device of claim 12 , wherein the processor device comprises a cache for registering memory lines, copies of the memory lines being populated to the processor cache memory, and memory consistency propagation across the shared memory system for the population of respective copies of the memory lines is not confirmed, wherein the marking of the memory lines comprises adding an identifier of the memory lines to the cache of the processor device, and wherein the clear marking of the marked memory lines comprises deleting the identifier of the memory lines from the cache of the processor.

15. The processor device of claim 14, wherein the identifiers of the tagged memory lines stored in the cache comprise storage addresses of the respective memory lines in the main memory.

16. The processor device of claim 14, wherein completion of execution of the memory barrier instruction is blocked until the buffer of the processor device is empty.

17. A method for managing a consistency directory for implementing memory consistency of a shared memory system, the method comprising: maintaining, by the coherence directory, flag information of a memory line assigned to a main memory of the shared memory system, wherein a processor device of the shared memory system has initiated populating a copy of the memory line into a processor cache memory operatively connected to the processor device, the flag information indicating whether memory coherence propagation across the shared memory system with respect to the populating of the copy of the memory line has been determined, wherein the coherence directory comprises a directory table and a transient table; The flag information is included in a message sent by the consistency directory to the processor device; as well as The message is sent to the processor device via the coherence directory.

18. The method of claim 17 , wherein the consistency directory initiates the maintenance of the tag information assigned to the memory line based on determining that content of a remote processor cache memory of another processor device of the shared memory system needs to be modified in order to propagate memory consistency regarding populating the copy of the memory line into the remote processor cache memory.

19. The method according to claim 18, wherein: The coherency directory sends a request to the remote processor cache memory to request modification of contents of the remote processor cache memory to propagate the memory coherency regarding the population of the copy of the memory line to the remote processor cache memory.

20. The method of claim 19, wherein the modification requested by the coherency directory comprises invalidating another copy of the memory line included in the remote processor cache memory.

21. The method according to claim 18, wherein: Based on the propagation of the memory coherency across the shared memory system regarding the population of the copy of the memory line, a coherency directory includes a register for keeping track of the content revision of the processor cache of the other processor device of the shared memory system.

22. A method according to claim 21, wherein when storage consistency propagation across the shared memory system regarding the population of the copy of the memory line is confirmed, the register includes a consistency propagation notification indicator, the consistency propagation notification indicator identifying a processor cache memory of a processor device of the shared memory system that is to be notified by the consistency directory.

23. A consistency directory device for implementing memory consistency of a shared memory system, the consistency directory device being configured to: maintaining flag information for a memory line assigned to a main memory of the shared memory system, wherein a processor device of the shared memory system has initiated populating a copy of the memory line into a processor cache memory operatively connected to the processor device, the flag information indicating whether memory consistency propagation across the shared memory system has been determined with respect to the populating of the copy of the memory line, wherein the consistency directory device includes a directory table and a transient table; Including the flag information in a message; as well as The message is sent to the processor device.

24. A coherent shared memory system comprising a coherent directory for implementing memory coherence of the shared memory system, a main memory, and a processor device operably connected to a processor cache memory, the processor device being configured to coordinate execution of an instruction sequence comprising a plurality of instructions, the coordination comprising: an instruction to execute an instruction sequence, wherein execution of the instructions causes the processor device to populate the processor cache memory with a copy of a memory line of a main memory of the shared memory system; receiving, by the processor device, a message from a controller included as part of a shared memory system, the message including flag information indicating whether memory coherency propagation across the shared memory system is confirmed, wherein the controller maintains a directory table of memory line entries; marking, by the processor device, respective memory lines based on detecting that the received flag information indicates that memory coherency propagation across the shared memory system with respect to the population of the copies of the memory lines is not acknowledged; clearing, by the processor device, a flag for the respective memory line based on detecting that the received flag information indicates that memory coherency propagation across the shared memory system with respect to the population of the copy of the memory line is confirmed; based on executing a memory barrier instruction of the sequence of instructions, preventing execution of the memory barrier instruction from completing while marking the memory line; and the coherence directory is configured to maintain the flag information assigned to the memory line of the main memory based on initiating the population of the copy of the memory line, the flag information indicating whether the memory coherence propagation across the shared memory system regarding the population of the copy of the memory line has been determined, wherein the coherence directory includes a transient table, wherein the maintaining includes: first including flag information indicating that propagation of the memory consistency across the shared memory system regarding the population of the copy of the memory line is not acknowledged as a first message; sending the first message to the processor device; Based on the propagating obtaining confirmation, updating the flag information and including the updated flag information indicating that the memory consistency propagation across the shared memory system regarding the population of the copy of the memory line is confirmed as a second message; and The second message is sent to the processor device.

Images