Apparatus and method for speculative vectorization of program code

Abstract

An apparatus and method for speculative vectorization of program code is provided. The apparatus comprises processing circuitry for executing program code, the program code comprising an identified code region comprising at least a plurality of speculative vector memory access instructions. Speculative vectorization is employed with execution of each speculative vector memory access instruction to perform a series of scalar memory access operations using a plurality of processing lanes. A tracking store is used to maintain tracking information for each speculative vector memory access instruction, the tracking information providing an indication of memory addresses accessed within each lane. A checking circuit then references the tracking information during execution of the identified code region by the processing circuitry in order to detect any inter-lane memory hazards resulting from the execution of the plurality of speculative vector memory access instructions. For at least a first type of inter-lane memory hazard, a state storage element is used to maintain an indication of each lane for which the checking circuit has determined that the type of memory hazard exists. A replay determination circuit is then arranged to, when the end of the identified code region is reached, trigger a re-execution of the identified code region for each lane identified by the state storage element as having an inter-lane memory hazard in response to the state storage element identifying at least one lane as having an inter-lane memory hazard. Such a method can significantly increase the ability to vectorize scalar code, resulting in significant performance improvements.

Description

Background Technology

[0001] This technology relates to apparatus and methods for speculative vectorization of program code.

[0002] To improve performance, vector processing circuits have been developed that enable multiple operations to be executed in parallel across multiple processing channels provided by the vector processing circuit. Associated with such vector processing circuits, vector instruction set architectures (ISAs) have been developed that provide new instructions and wider data paths. For example, a Single Instruction Multiple Data (SIMD) vector processing circuit includes multiple processing elements that can simultaneously perform the same operation on multiple different datasets to leverage data-level parallelism. Thus, a single vector instruction can specify one or more vectors of input data values, where the input data values ​​within each vector are processed in a corresponding parallel processing channel within the vector processing circuit, thereby enabling the same operation to be executed in parallel on multiple data values.

[0003] Such techniques can significantly improve performance and enable the vectorization of a series of scalar operations, allowing them to be executed in parallel through corresponding vector operations. However, modern SIMD architectures still rely on programmers or compilers to convert code into vector form, doing so only when it is safe to do so. In fact, the limitations of compiler memory aliasing analysis and the existence of infrequent memory data dependencies mean that entire regions of code cannot be safely vectorized in existing systems without risking changes to the semantics of the application, thus limiting the achievable performance improvements.

[0004] The aim is to provide an improvement technique that can enhance performance in scenarios where vectorizable code is available. Summary of the Invention

[0005] In one exemplary arrangement, an apparatus is provided comprising: processing circuitry for executing program code including an identified code region comprising at least a plurality of speculative vector memory access instructions, wherein execution of each speculative vector memory access instruction is employed to perform speculative vectorization of a series of scalar memory access operations using a plurality of processing channels; a tracking storage device for maintaining tracking information for each speculative vector memory access instruction, the tracking information providing an indication that a memory address is accessed within each channel; an inspection circuitry for referencing the tracking information during execution of the identified code region by the processing circuitry to detect any inter-channel memory hazards arising from the execution of the plurality of speculative vector memory access instructions; a state storage element for maintaining an indication for each channel for which the inspection circuitry determines at least a first type of inter-channel memory hazard; and a replay determination circuitry arranged to trigger re-execution of the identified code region for each channel identified by the state storage element in response to identifying at least one channel as having an inter-channel memory hazard upon reaching the end of the identified code region.

[0006] In another example arrangement, a method for speculatively vectorized program code is provided, the method comprising: executing program code including an identified code region comprising at least a plurality of speculative vector memory access instructions with processing circuitry, wherein execution of each speculative vector memory access instruction is used to perform speculative vectorization of a series of scalar memory access operations using a plurality of processing channels; for each speculative vector memory access instruction, maintaining trace information in a trace storage device, the trace information providing an indication that a memory address is accessed in each channel; employing a checking circuitry to reference the trace information during execution of the identified code region by the processing circuitry to detect any inter-channel memory hazards arising from the execution of the plurality of speculative vector memory access instructions; maintaining an indication in a state storage element that the checking circuitry determines at least one type of inter-channel memory hazard for each channel; and triggering re-execution of the identified code region for each channel identified by the state storage element upon reaching the end of the identified code region and the state storage element identifying at least one channel as having an inter-channel memory hazard.

[0007] In another example arrangement, a computer program is provided for controlling a host data processing device to provide an instruction execution environment, the instruction execution environment comprising: process logic for executing program code including an identified code region comprising at least a plurality of speculative vector memory access instructions, wherein execution of each speculative vector memory access instruction is employed to perform speculative vectorization of a series of scalar memory access operations using a plurality of processing channels; a trace data structure for maintaining trace information for each speculative vector memory access instruction, the trace information providing an indication that a memory address is accessed within each channel; and a checking process. The process logic includes: a checking procedure logic for referencing the trace information during execution of the identified code region by the processing procedure logic to detect any inter-channel memory hazards arising from the execution of the plurality of speculative vector memory access instructions; a state data structure for maintaining an indication for each channel for which the checking procedure logic determines at least a first type of inter-channel memory hazard; and replay determination procedure logic arranged to trigger re-execution of the identified code region for each channel identified by the state storage element at the end of the identified code region, in response to the state data structure identifying at least one channel as having an inter-channel memory hazard.

[0008] In another example arrangement, an apparatus is provided, comprising: a processing means for executing program code including an identified code region, the identified code region including at least a plurality of speculative vector memory access instructions, wherein execution of each speculative vector memory access instruction is employed to perform speculative vectorization of a series of scalar memory access operations using a plurality of processing channels; a tracking storage means for maintaining tracking information for each speculative vector memory access instruction, the tracking information providing an indication that a memory address is accessed within each channel; an inspection means for referencing the tracking information during execution of the identified code region by processing logic to detect any inter-channel memory hazards arising from the execution of the plurality of speculative vector memory access instructions; a state storage element means for maintaining an indication for each channel for which the inspection logic determines at least a first type of inter-channel memory hazard; and a replay determination means for triggering re-execution of the identified code region for each channel identified by the state storage element in response to a state data structure identifying at least one channel as having an inter-channel memory hazard upon reaching the end of the identified code region. Attached Figure Description

[0009] The present technology will be further described by way of illustration only, with reference to examples of the present technology shown in the accompanying drawings, wherein:

[0010] Figure 1 A data processing apparatus according to an example is illustrated schematically;

[0011] Figure 2 This illustrates a specific implementation of mergeability based on an example. Figure 1 A block diagram of components within the device to facilitate the execution of speculative vector memory access instructions;

[0012] Figure 3 A more detailed illustration is shown in an example arrangement. Figure 2 Information captured within the tracking storage device;

[0013] Figure 4 This illustrates how entries in the tracking storage device can be populated for multiple example use cases based on a specific example implementation;

[0014] Figure 5 The illustration schematically demonstrates how tracking memory devices can be implemented using an enhanced version of the memory buffer and a memory hazard buffer already provided in an out-of-order system, according to an example arrangement.

[0015] Figure 6A and Figure 6B A flowchart is provided illustrating how the identified code regions can be executed based on an example layout;

[0016] Figure 7 It shows that alternative implementations are possible instead of Figure 6B A flowchart of some of the steps performed in step 385;

[0017] Figure 8 This illustrates how tag information can be associated with vector memory access instructions, based on an example layout; and

[0018] Figure 9 A specific implementation of the simulator that can be used in an example setup is shown. Detailed Implementation

[0019] The techniques described herein relate to the vectorization of scalar memory access operations. When seeking to vectorize such scalar memory access operations so that those operations can be executed in parallel across multiple processing channels, there is a possibility of through-the-path memory dependencies (TMDs), which are dependencies caused by memory aliasing. Specifically, memory aliasing problems can occur when multiple such scalar memory access operations are vectorized because an address accessed in one channel for a vectorized scalar memory access operation may have memory aliasing issues relative to an address accessed in a different channel for another vectorized scalar memory access operation. By way of example, if a scalar load operation is followed by a scalar store operation, and multiple iterations of those load and store operations are vectorized such that, for example, a vector load operation is performed after a vector store operation, it should be understood that each iteration of the original scalar code is represented by a different channel within the vector processing arrangement. In the case where a given address is used for both the higher channel of a speculative vector load instruction and the lower channel of a speculative vector store instruction, a read-after-write (RAW) inter-channel memory hazard may arise because the load operation will obtain the old value at the given address when it should receive the new value.

[0020] When considering the vectorization of raw scalar code, it is often unknown whether such TMDs will be materialized, or whether such TMDs occur only periodically. Typically, prior to the techniques described herein, code segments including such scalar memory access operations would not be vectorized due to potential memory aliasing issues. However, according to the techniques described herein, a mechanism is provided that enables speculative vectorization of such code regions by allowing memory address conflicts to be detected in hardware and automatically triggering selective re-execution of only the affected specific channels.

[0021] Specifically, an apparatus is provided having processing circuitry for executing program code, wherein the program code includes an identified code region comprising at least a plurality of speculative vector memory access instructions. Execution of each speculative vector memory access instruction is employed to perform a series of scalar memory access operations using multiple processing channels through speculative vectorization. Specifically, the vectorization is speculative because it can occur in situations where memory hazards are unknown or periodic, without requiring prior determination of whether any inter-channel memory hazard is actually introduced by the speculative vectorization.

[0022] The device also includes a tracking memory to maintain tracking information for each speculative vector memory access instruction, the tracking information providing an indication of the memory address accessed within each channel. The tracking memory may be a dedicated memory provided solely for tracking speculative vector memory access instructions. However, as will be discussed in more detail later, in some embodiments, the tracking memory may be formed by enhancing the memory structure already present in the device for other reasons, and thus in those cases, the tracking memory may also maintain information about other instructions being executed. However, the interest of the present invention lies in how the tracking memory can be used to track information about speculative vector memory access instructions.

[0023] The device also includes a checking circuit to reference trace information during the execution of identified code regions by the processing circuitry to detect any inter-channel memory hazards arising from the execution of multiple speculative vector memory access instructions. Specifically, although the scalar memory access operation may be speculatively vectorized by specifying speculative vector memory access instructions, the checking circuitry can determine whether any inter-channel memory hazards exist or not by referring to trace information maintained by the tracking memory device when those speculative vector memory access instructions are executed by the processing circuitry.

[0024] Furthermore, a state storage element is provided to maintain an indication for each channel in which the checking circuitry has determined at least one type of inter-channel memory hazard. While the state storage element can be used to maintain an indication for each channel in which the checking circuitry has determined any form of inter-channel memory hazard, as will be discussed in more detail later, in some cases, certain types of memory hazards can be resolved by other mechanisms, and therefore it may not be necessary to capture information about those inter-channel memory hazards within the state storage element. However, there will be at least one type of inter-channel memory hazard where the state storage element needs to maintain an indication for each channel in which the checking circuitry has determined that such a type of inter-channel memory hazard exists.

[0025] The device also provides a replay determination circuit, which is arranged such that, when the end of the identified code region is reached, in response to the state storage element identifying at least one channel as having an inter-channel memory hazard, re-execution of the identified code region is triggered for each channel identified by the state storage element.

[0026] This method allows for the safe vectorization of scalar memory access operations that might not have been previously vectorized, because the device can detect and track the presence of any inter-channel memory hazards arising from speculative vectorization of such scalar memory access operations during execution. Furthermore, any re-execution of the identified code region can be limited to the affected channel, resulting in a particularly effective implementation.

[0027] Therefore, by using the above techniques, TMD can be processed in hardware, enabling code regions exhibiting such dependencies to be speculatively vectorized, thereby increasing the degree to which scalar operations can be vectorized, and thus realizing the performance benefits associated with the use of vector code.

[0028] In one example implementation, the apparatus further includes control storage elements to store inter-channel dependency predicates referenced by the processing circuitry during execution of the identified code region. During the initial execution of the identified code region, the inter-channel dependency predicates can be initialized to identify all channels as active, such that the execution of each speculative vector memory access instruction (and indeed any vector instructions within the identified code region) is not constrained by the inter-channel dependency predicates. Therefore, during the initial execution of the identified code region, the execution of each instruction is only constrained by any predicate that can be explicitly specified for that instruction. Specifically, it should be understood that in some instruction set architectures, individual instructions may have predicates associated with them to identify the channels active for that particular instruction. In contrast, the inter-channel dependency predicate is an implicit predicate because it is not visible to the programmer and therefore is not specified in the individual instructions. Instead, it is a predicate maintained by hardware to control selective re-execution in a specified channel when determined by the aforementioned replay determination circuitry.

[0029] Specifically, while the inter-channel dependency predicate identifies all channels as active during initial execution, during the re-execution of the identified code regions, the inter-channel dependency predicate is set to constrain the re-execution of the identified code regions to the channels identified in the state storage element. As previously discussed, the state storage element is used to maintain an indication of each channel for which one or more types of inter-channel memory hazards are detected during the execution of the program code, and thus the information maintained in the state storage element when the identified code regions have been executed can be used to identify any channels for which the identified code regions need to be re-executed, whereby the inter-channel dependency predicate is then set.

[0030] There are multiple ways to identify code regions. For example, specific instructions can be used to identify those instructions within regions that require inspection circuitry. However, in one example implementation, the start of the identified code region is identified by speculative vectorization (SV) instruction identification. By using such specific instructions, instructions within the code region that are not needed for identification become different variants of instructions used outside the identified code region, which is advantageous given that instruction coding space is often valuable.

[0031] In a specific implementation using such an SV start instruction, the processing circuitry may, in response to the execution of the SV start instruction, record the address indication of the first instruction following the SV start instruction. Then, the replay determination circuitry, in response to determining that the identified code region needs to be re-executed, outputs the address indication of that first instruction so that re-execution begins from that first instruction. It should be noted that re-execution does not return to the SV start instruction, but rather to the first instruction following it, because this is the instruction that initiates the function specified within the identified code region, and the SV start instruction essentially serves to annotate the beginning of the identified code region.

[0032] There are several ways to utilize the address indication output by the replay determination circuitry. For example, the address indication can be routed back to the fetch circuitry within the device to begin fetching instructions from that address. It may also be necessary to refresh existing instructions within the device's processing pipeline so that the next instruction to be executed after the end of the identified code region is the instruction at the address indicated by the replay determination circuitry output.

[0033] In one example arrangement, the processing circuitry may further respond to the execution of the SV start instruction to initialize the previously mentioned inter-channel dependency predicate, that is, to identify all channels as active, such that during the initial execution of the identified code region, the execution of each vector instruction in the identified code region is not constrained by the inter-channel dependency predicate.

[0034] There are multiple ways to identify the end of a identifiable code region. For example, the identified code region can end with a backward branch instruction whose target is the first instruction in the identified code region (i.e., the instruction immediately following the SV start instruction in a specific implementation using such an instruction). In some cases, branch instructions can be used to identify the end of an identified code region and cause the playback determination circuitry to determine whether one or more channels require...

[0035] Any re-execution of the identified code region. However, in an alternative implementation, the end of the identified code region is identified by a speculative vectorization (SV) end instruction (i.e., there exists a specific instruction for identifying the end of the identified code region). In one such implementation, the processing circuitry responds to the execution of the SV end instruction such that the replay determination circuitry determines whether any channel requires re-execution of the identified code region. This provides a simple and efficient mechanism for selectively re-executing the identified code region for any channel that has been identified as having at least a first type of inter-channel memory hazard.

[0036] Speculative vector memory access instructions can be introduced into program code in a variety of ways. For example, a programmer might directly specify such speculative vector memory access instructions. However, in one example implementation, speculative vector memory access instructions are introduced by a compiler seeking to speculatively vectorize scalar code, and the compiler is configured to annotate the program code to identify instructions forming the identified code regions (e.g., by adding the previously discussed SV start and SV end instructions). Therefore, vectorizing scalar code is unsafe because it is impossible to assess whether memory aliasing problems will occur, or because such memory aliasing problems are known to occur periodically. The mechanism described herein enables the compiler to speculatively vectorize scalar code in situations previously considered impossible. However, because the hardware is able to identify such inter-channel memory hazards during execution and selectively re-execute the identified code regions for any affected channels according to the techniques described herein, it is safe for the compiler to speculatively vectorize such scalar code.

[0037] The identified code region can take many forms, but in one example implementation, it forms at least a portion of a vector loop used for vectorizing scalar loops. It has been found that the techniques described herein can be particularly advantageous for the vectorization of scalar loops.

[0038] The trace information maintained by the trace storage device for each speculative vector memory access instruction can take various forms. In one example implementation, in addition to providing an indication that a memory address is accessed within each channel, the trace information is arranged to also provide a type indication to each speculative vector memory access instruction, indicating whether the speculative vector memory access instruction is a speculative vector load instruction for loading data from memory or a speculative vector store instruction for storing data into memory. The checking circuit is then arranged to refer to the type indication when determining whether an inter-channel memory hazard has been detected. By maintaining information about the type of memory access instruction involved in the trace information, this allows the checking circuit to check for various different inter-channel memory hazards, such as read-after-write inter-channel memory hazards, read-after-write inter-channel memory hazards, or write-after-write inter-channel memory hazards.

[0039] If needed, the type indicator can also provide additional information. For example, in some embodiments, the type indicator is also configured to identify whether a speculative vector memory access instruction performs an access to a contiguous or non-contiguous memory address. This additional information can be used to interpret the trace information provided to a particular speculative vector memory access instruction.

[0040] Tracing memory devices can be organized in various ways, but in one example implementation, multiple entries are included, each associated with a speculative vector memory access instruction for storing trace information of at least a subset of scalar memory access operations performed by executing that speculative vector memory access instruction. While in some cases a single entry may be used to store trace information of all scalar memory access operations performed by executing that speculative vector memory access instruction, it can simplify lookup procedures within the tracing memory device and / or aid in identifying and / or resolving inter-channel memory hazards if certain constraints are placed on addresses that can be traced within a single entry.

[0041] In one example arrangement, each entry in the tracking memory is constrained to store tracking information for one or more memory addresses aligned to a given memory block granularity address. By constraining the memory addresses that can be tracked within a single entry in this way, CAM-based (Content-Addressable Memory) lookups are supported. Specifically, the memory address does not need to be considered during the lookup operation.

[0042] Instead of using a certain number of least significant bits, only a certain number of most significant bits captured for an entry need to be compared with the corresponding most significant bits to determine if the entry involves a problematic address. The size of a given memory block granularity can vary depending on the specific implementation, but in one example implementation, this granularity can be set to be equal to the width of the data path between the processing circuitry and the cache, for example, equal to the width of the L1 data cache data path, which could be 128 or 256 bits.

[0043] When the above method is used, only a certain number of the most significant bits need to be captured to represent the memory address covered by the entry, which can be supplemented with additional information describing the specific address range encoded within the entry. This can be, for example, in the form of per-byte significant bits.

[0044] For speculative vector memory access instructions used to perform accesses to contiguous memory addresses, only a single entry is required if those contiguous memory addresses are aligned with the aforementioned memory block granularity. However, when contiguous memory addresses intersect with multiple memory blocks aligned to the same address, more than one entry can be allocated in the trace memory to store the associated trace information.

[0045] In one example implementation where the address traced by each entry needs to be aligned with the address of a given memory block, then, in the case of speculative vector memory access instructions used to perform accesses to non-contiguous memory addresses, separate entries in the trace memory can be allocated to store the associated trace information for each memory address accessed. In the case of load instructions, vector memory access instructions used to perform accesses to non-contiguous memory addresses can be called aggregation instructions because information from multiple discrete addresses is aggregated together for storage in the destination vector register. Similarly, when speculative vector memory access instructions are used to perform write accesses to non-contiguous memory addresses, the speculative vector memory access instructions can be called scattering instructions because different data elements in the source vector register are output to non-contiguous memory addresses in memory for storage.

[0046] In one example implementation, for each memory address identified within an entry in the tracking memory device, the tracking information maintained within that entry is arranged to provide channel identification information sufficient to determine the channel involved for each memory address. When an entry is associated with a series of consecutive accesses, this information about the channel involved for each memory address can be directly derived from the address information, for example, by referring to the previously mentioned per-byte significant bits. In such scenarios, the vector element size can be considered when interpreting the per-byte significant bits to identify the vector element associated with the entry. However, for entries involving the previously mentioned scattering / gathering accesses, a specific channel identifier, which may also be referred to herein as a vector element index, can be provided within each entry. Specifically, because the addresses are not contiguous, the channel involved for a particular address cannot be determined from the address information itself. Therefore, by providing a specific channel identifier for such entries, it becomes possible to determine the memory address associated with any particular channel of the gathering / scattering operation.

[0047] In one example implementation, each speculative vector memory access instruction is either a speculative vector load instruction for loading data from memory, or a speculative vector store instruction for storing data into memory. Multiple processing channels can be viewed as extending from the lowest channel associated with the first scalar memory access operation in a series speculatively vectorized by the speculative vector memory access instructions to the highest channel associated with the final scalar memory access operation in that series. The checking circuitry can then be arranged to detect one or more of the following inter-channel memory hazards: a read-after-write (RAW) inter-channel memory hazard, where a given address is used for both the higher channel of the speculative vector load instruction and the lower channel of the speculative vector store instruction that appears in original program order after the speculative vector load instruction; a read-after-write (WAR) inter-channel memory hazard, where a given address is used for both the higher channel of the speculative vector store instruction and the lower channel of the speculative vector load instruction that appears in original program order after the speculative vector store instruction; and a write-after-write (WAW) inter-channel memory hazard, where a given address is used for both the higher channel of the first speculative vector store instruction and the lower channel of the second speculative vector store instruction.

[0048] It should be noted that the aforementioned inter-channel memory hazards differ from the types of hazards typically identifiable by the out-of-order processing components provided in out-of-order processing systems. Specifically, when instructions are reorderable, there is a possibility that the reordering of the instructions themselves can create memory hazards in any particular processing channel. These can be considered vertical memory hazards because they occur in the reordered instructions.

[0049] Common intra-processing channel identification (i.e., danger arises because the same address is used in channel x of one instruction and in the same channel x of another instruction). However, the above-described type of inter-channel memory danger can occur regardless of instruction reordering. Specifically, these dangers occur when a series of scalar memory access operations are vectorized, due to the temporal implications associated with each processing channel.

[0050] Specifically, each channel is associated with a different iteration of a scalar memory access operation, and no memory hazard occurs due to the use of conflicting addresses within a single channel, and because such addresses are used across multiple speculative memory access instructions for different channels. Therefore, even if two speculative vector memory access instructions are executed in the original program order, and thus there is no memory hazard introduced by reordering those instructions, inter-channel memory hazards can still occur because the underlying scalar memory access operations will be reordered through the speculative vectorization of those scalar memory access operations. To distinguish these from the common memory hazards that may be checked when instructions are reordered (as previously discussed, these can be referred to as vertical memory hazards), the aforementioned inter-channel memory hazards are referred to herein as horizontal memory hazards, as they are hazards that occur across multiple speculative vector memory access instructions across different channels.

[0051] In one example implementation, the device further includes a storage buffer to temporarily store write data generated by the execution of each speculative vector storage instruction, wherein write data for a given speculative vector storage instruction is prohibited from being output from the storage buffer to memory until any inter-channel memory hazard arising from the execution of the given speculative vector storage instruction has been resolved.

[0052] In a specific implementation providing such a memory buffer, when the checking circuit detects a WAR (Inter-Channel Memory Hazard), the checking circuit can be arranged to resolve the WAR by causing the processing circuit to retrieve data associated with the lower channel from memory, rather than from the memory buffer, when executing a speculative vector load instruction. The state storage element then does not update in response to the identified WAR. Specifically, a WAR occurs because of the fact that the given address is used for both the higher channel of the speculative vector store instruction and the lower channel of the speculative vector load instruction that appears in the original program order after the speculative vector store instruction. However, considering the underlying family of scalar memory access operations speculatively vectorized by these two vector store and vector load instructions, the associated load operation can receive a new value at the given address when the underlying intent of the scalar memory access operation is that it will receive an older version.

[0053] However, since the new write data is temporarily buffered in the storage buffer, it is still possible to obtain the correct data for the load instruction from memory. Therefore, by retrieving data from memory instead of from the storage buffer, the WAR inter-channel memory hazard can be effectively resolved without re-executing the identified code region of the problematic channel. Thus, the state storage element does not need to be updated in response to an identified WAR inter-channel memory hazard.

[0054] In one example implementation, the state storage element is updated in response to a memory hazard detected by the checking circuitry for at least each RAW channel, and the channel identified in the state storage element is the higher channel of the speculative vector load instruction. Specifically, the higher channel of the speculative vector load instruction will retrieve the old value at a given address, while the intent of the underlying series of scalar memory access operations is that it will retrieve the new value. Therefore, the channel to be identified in the state storage element is the higher channel of the speculative vector load instruction, and it will be connected to that channel for which the replay determination circuitry will later trigger the re-execution of the identified code region.

[0055] In one example implementation, the state storage element is further updated in response to each inter-WAW channel memory hazard detected by the checking circuitry, and the channel identified in the state storage element is the higher channel of the first speculative vector store instruction. Specifically, in the presence of an inter-WAW channel memory hazard, a given memory address will not be updated to reflect the latest version of the data expected by the underlying series of scalar memory access operations. However, by identifying the channel as the higher channel of the first speculative vector store instruction in the state storage element, the identified code region can then be re-executed for that channel, resulting in the correct write data being written to memory.

[0056] However, in alternative implementations, it may be possible to resolve inter-channel memory hazards (ICMs) without re-executing the identified code regions. Specifically, in one example implementation, when the checking circuitry detects an ICM, it is configured to resolve the ICM by causing a memory buffer to be marked such that write data generated in the lower channel during the execution of a second speculative vector store instruction is prevented from being output to memory. The state storage element then does not update in response to the identified ICM. Therefore, by appropriately marking data in the memory buffer, incorrect write data output to memory can be suppressed by the memory buffer, causing the memory contents to terminate as expected by the underlying scalar memory access operations. Thus, for ICMs identified in such implementations, it is not necessary to update the state storage element.

[0057] The techniques described herein can be used with both sequential and out-of-order systems. However, in one example implementation, the processing circuitry is arranged to execute out-of-order (OOO) processing of instructions. In such cases, the device will include at least one OOO trace structure having trace entries for tracking memory hazards introduced by instruction reordering. As previously discussed, these are referred to herein as vertical memory hazards, and those skilled in the art will understand that several existing trace structures exist for seeking to detect such vertical memory hazards.

[0058] In such OOO systems, the previously described tracking memory devices, which are then used by the checking circuitry to detect horizontal memory hazards, can be implemented by enhancing the existing OOO tracking structure. Specifically, the tracking entries in one or more OOO tracking structures can be enhanced such that the tracking memory devices are incorporated into such tracking structures, thereby allowing the checking circuitry to detect inter-channel memory hazards arising from the allocation of scalar memory access operations within a series to different processing channels. This can provide a particularly cost-effective implementation in terms of additional transistor count, area, and / or power consumption.

[0059] The OOO tracing structure enhanced in the manner described above can take various forms. In one example implementation, an OOO tracing structure includes a storage buffer to temporarily store write data resulting from the execution of a store instruction until it is determined that the write data can be committed to memory, and the storage buffer is enhanced to allow tracing information to be maintained therein for use in speculative vector store instructions within the code region to be identified.

[0060] Additionally or alternatively, an enhanced OOO tracing structure may include a memory hazard buffer to track the execution of out-of-order memory access instructions, and an enhanced memory hazard buffer to allow tracing information to be maintained therein for use in identifying speculative vector load instructions within code regions.

[0061] There are several ways in which vector memory access instructions can be identified as speculative vector memory access instructions, as discussed earlier, and therefore require monitoring by checking circuitry. In one example implementation, it is assumed that every vector memory access instruction in the identified code region is a speculative vector memory access instruction, such that the checking circuitry is arranged to check for inter-channel memory hazards associated with the execution of each vector memory access instruction in the identified code region. Thus, by way of example, in the case of using the previously mentioned SV start and SV end instructions, any vector memory access instruction appearing between the SV start and SV end instructions can be considered a speculative vector memory access instruction. This provides a particularly efficient implementation because it does not require modification of the instruction code to distinguish between speculative and non-speculative versions of vector memory access instructions.

[0062] In alternative implementations, vector memory access instructions can be tagged to distinguish those that are speculative vector memory access instructions and therefore should be monitored by the inspection circuitry, from those that are known not to cause inter-channel memory hazards and therefore can be ignored by the inspection circuitry. Therefore, in such implementations, it is possible to identify the code region...

[0063] Vector memory access instructions that do not require checking can be used, thus reducing unnecessary checks by the checking circuitry. Tagging information may also be sufficient to group speculative vector memory access instructions that need to be checked relative to each other. Therefore, identified code regions can include different groups of speculative vector memory access instructions, where each speculative vector memory access instruction within a group needs to be checked relative to every other speculative vector memory access instruction in that group, but not relative to vector memory access instructions in another group. Therefore, it should be understood that tagging information can be used to isolate disjoint regions of memory, for example, under compiler control. Where the independence between program objects (e.g., arrays) can be proven, the compiler can assign a unique tag to each object accessed in a speculative region, which is then used to reduce the amount of checking required by the hardware.

[0064] By using such a marking scheme, this can result in fewer checks and smaller space requirements, or allow for larger loops to be vectorized, since the address information of any "no-check" instructions does not need to be recorded in the tracking memory at runtime.

[0065] A specific example will now be described with reference to the accompanying drawings.

[0066] Figure 1An example of a data processing apparatus 2 having a processing pipeline comprising multiple pipeline stages is schematically shown. The pipeline includes a branch predictor 4, which predicts the outcome of branch instructions and generates a series of fetch addresses for the instructions to be fetched. A fetch stage 6 fetches the instructions identified by the fetch addresses from an instruction cache 8. A decode stage 10 decodes the fetched instructions to generate control information for controlling subsequent stages of the pipeline. An out-of-order processing unit 12 is provided at the next stage to handle the out-of-order execution of instructions. These units can take various forms, including, for example, reorder buffers (ROBs) and renaming circuitry. ROBs are used to keep track of instruction progress and ensure that instructions are submitted in order, even if they are executed out of order. The renaming stage 12 performs register renaming to map the architecture register pointers identified by the instructions to physical register pointers that identify registers 14 provided in the hardware. Register renaming can be used to support out-of-order execution because it eliminates the danger between instructions specifying the same architecture registers by mapping them to different physical registers in the hardware register file. This increases the likelihood that instructions can be executed in a different order than the program fetched from cache 8, which can improve performance by allowing later instructions to execute while earlier instructions wait for operands to become available. The ability to map architecture registers to different physical registers also facilitates rolling back the architecture state in the event of branch misprediction. Release stage 16 queues instructions for execution until the required operands for processing those instructions become available in register 14. Execution stage 18 executes the instructions to perform the corresponding processing operations. Write-back stage 20 writes the results of the executed instructions back to register 14.

[0067] Execution level 18 may include multiple execution units, such as a branch unit 21 for evaluating whether a branch instruction has been correctly predicted, an ALU (Arithmetic Logic Unit) 22 for performing arithmetic or logical operations, a floating-point unit 24 for performing operations using floating-point operands, and a load / store unit 26 for performing load operations to load data from the memory system into register 14 or store operations to store data from register 14 into the memory system. In this example, the memory system includes a Level 1 instruction cache 8, a Level 1 data cache 30, a Level 2 cache 32 shared between data and instructions, and main memory 34. However, it should be understood that this is merely one example of a possible memory hierarchy, and other specific implementations may have further cache tiers or different arrangements. Load / store unit 26 may use translation lookahead buffer 36, and fetch unit 6 may use translation lookahead buffer 37 to map virtual addresses generated by the pipeline to physical addresses that identify locations within the memory system. It should be understood that... Figure 1The pipeline shown is merely an example, and other examples may have different sets of pipeline stages or execution units. For example, an out-of-order processor will typically not have an OOO processing unit 12.

[0068] As previously discussed, the technique described herein allows a series of scalar memory access operations to be speculatively vectorized before it is known whether memory dependency (TMD) will occur. A mechanism is then provided that enables any such memory address conflicts to be detected in hardware and to automatically trigger selective re-execution only for the specific affected channels.

[0069] TMD is referred to as Horizontal Memory Hazard in this paper, and there are several different Horizontal Memory Hazards that can be detected using the techniques described herein. Specifically, the mechanisms described herein can detect RAW inter-channel memory hazards, WAR inter-channel memory hazards, and WAW inter-channel memory hazards. Examples of each of these types of hazards are provided below for vector operations representing consecutive access and two vector operations representing non-consecutive access:

[0070] RAW Hazard – Aggregation / Scattering Pairs:

[0071] LD [A0,A1,A2,A3] / / LD retrieves the old value at address A2 and should retrieve the new value.

[0072] ST [A4,A2,A6,A7]

[0073] RAW – Continuous Visit:

[0074] LD [A0:A0+16] / / LD retrieves the old value at address [A0+8:A0+16].

[0075] ST [A0+8:A0+24]

[0076] WAR Danger – Cluster / Scatter Pairs:

[0077] ST [A0,A1,A2,A3]

[0078] LD [A4,A2,A6,A7] / / LD gets the new value at address A2, but should get the old value.

[0079] WAR Danger – Continuous Access:

[0080] ST [A0:A0+16]

[0081] LD [A0+8:A0+24] / / LD gets a new value at address [A0+8:A0+16], but should get the old value.

[0082] WAW Hazard – Aggregation / Scattering Pairs:

[0083] ST [A0,A1,A2,A3]

[0084] ST [A4,A2,A6,A7] / / The memory location at address A2 will not receive the latest value.

[0085] WAW Danger – Continuous Visit:

[0086] ST [A0:A0+16]

[0087] ST [A0+8:A0+24] / / The memory at address [A0+8:A0+16] will not receive the latest value.

[0088] In the example above, the instructions are written in the original program order.

[0089] According to the techniques described herein, hardware checks can be performed at execution time associated with speculative vector memory access instructions within the identified code region, employing speculative vectorization of those instructions to perform a series of scalar memory access operations using multiple parallel processing channels. For example, the execution phase 18 of the data processing device 2 can be arranged in a SIMD configuration so that operations can be executed in parallel across multiple parallel processing channels provided by the circuitry during execution phase 18.

[0090] There are multiple ways to specify the code regions to be identified, but as discussed in a previous example, we introduce two machine instructions called SV start and SV end instructions. Specifically, these instructions are used to mark the beginning and end of the identified code regions (which may also be referred to as speculative regions in this document). The code for a simple loop utilizing this speculative marking is shown below (specifically, this represents the main loop of the histogram update example):

[0091]

[0092] The first load instruction, ld1w, performs a series of load operations to load a sequence of data elements into the destination register z0. The address is calculated by multiplying the offset in scalar register x2 by a scaling factor of 4 (since the elements in this example are word-sized elements (i.e., 32-bit (4-byte) elements)) and then adding the adjusted offset to the value in scalar register x0. The SV start instruction then identifies the beginning of the identified code region, which in this example can be seen to include both aggregation and scattering instructions. Each of these instructions can be considered a speculative vector memory access instruction. For each load operation required by the aggregation instruction (each load operation is executed in a different parallel processing channel), the address is determined by taking the corresponding element from the vector source register z0 to form a vector offset, multiplying the vector offset by a scaling factor of 4, and adding the adjusted vector offset to the contents of scalar register x1. An access is then performed on the address to retrieve the data value, which is then stored at the corresponding element location within the destination register z1.

[0093] In the two load instructions discussed so far, the predicate register p0 is specified, where the suffix " / z" indicates that the result of any inactive channel should be zero.

[0094] Within the identified code region, a vector addition instruction is executed to add 1 to each data element in vector register z1. Subsequently, a vector scattering instruction st1w is executed to store the data elements in vector source register z1 back into a series of memory addresses calculated using the value in scalar register x1 and the corresponding element in source vector register z0 (again, before the value added to scalar register x1).

[0095] Given the word size element used, the corresponding element in the source vector register z0 is multiplied by 4. For scattering operations, a predicate can be specified so that only the data values ​​in the active channel are stored back to memory, thereby achieving implicit merging operations.

[0096] Then, using the information maintained in the status register as the result of checks performed during the execution of the gather and scatter operations, the SV end instruction identifies the end of the identified code region and triggers a determination on whether the gather, add, and scatter instructions for one or more channels need to be re-executed, as will be discussed in more detail later.

[0097] The two instructions following the SV end instruction then set some values ​​for the next iteration of the loop. The increment instruction increases the value in the scalar register x2 by the number of 32-bit elements in the vector register. Then the while instruction is used to set bits in the predicate p0. Specifically, although the value in x2 is less than the value in x3 (which stores information about the number of scalar iterations of the loop), the bits in the predicate p0 are set to true. Therefore, by way of example, if the original scalar loop has twenty iterations, and each vector loop handles eight scalar iterations, the predicate p0 will be set to true in the first and second iterations, but will take the form "00001111" in the third iteration. If at least the first element of the predicate p0 is set, the final branch instruction returns to the beginning of the loop.

[0098] It should be understood that there is a possibility that a pair of gather and scatter instructions could lead to a memory hazard between RAW channels. Prior to the techniques described herein, this might have prevented the compiler from generating a vectorized version of the original scalar code, but with the techniques described herein, speculative vectorization of those scalar operations is possible, where the hardware checks for the presence of such RAW hazards during execution.

[0099] Figure 2 It shows that it can be used Figure 1 A block diagram of some components is provided within the device to implement the techniques described herein. Any speculative vector memory access instructions will be executed within the load / store pipeline 26, such as... Figure 2 As shown, the load / store pipeline can be fed by the contents of load queue 50 and store queue 55. The load / store pipeline 26 then accesses memory via level 1 data cache 30 to read data from memory for storage in register 14 or to store data from register 14 back to memory.

[0100] A trace memory device 65 is provided to maintain certain trace information for each speculative vector memory access instruction executed by the load / store pipeline 26, which at least provides an indication of whether a memory address is accessed within each channel. (See later...) Figure 3 and Figure 4 Further details describe the trace memory device. While trace memory device 65 could be a dedicated structure purely for providing trace information about speculative vector memory access instructions, as will be discussed later... Figure 5 In more detail, in some cases, the tracking storage device can be implemented by enhancing the existing structure already provided within the data processing device 2 in situations where the device 2 is used to facilitate out-of-order processing of instructions.

[0101] The trigger check circuit 60 performs certain check operations when execution occurs within the previously discussed identified code region (i.e., within the code region identified by the SV start and SV end instructions). Therefore, the SV start and SV end instructions are used to provide trigger information to the check circuit 60 to identify when the check circuit should monitor load and store operations executed by the load / store pipeline 26, and specifically monitor those operations to detect any inter-channel memory hazards arising from the execution of those memory access instructions. When performing such check operations, the check circuit 60 references the trace memory device 65 to understand the memory addresses accessed within each channel and the type of speculative memory access operation performed.

[0102] A status register 70 may be provided to maintain an indication for each channel for which the inspection circuitry determines at least a first-type inter-channel memory hazard. Specifically, as will be discussed in more detail later, the status register is used to keep track of any detected RAW channel hazards, and in some specific implementations, also to keep track of any WAW channel hazards. However, alternative mechanisms may be used to resolve WAR hazards without requiring any replay of instructions in the identified code region, and therefore it is not necessary to keep track of WAR hazards within the status register 70.

[0103] In response to the SV start instruction, a trigger is sent to control register 80 to initialize the inter-channel dependency predicate. This is an implicit predicate, as it is not a predicate that can be specified with respect to any particular instruction and is not visible to the programmer. Instead, it is used to control the selective re-execution of identified code regions within one or more channels. In response to the SV start instruction, the inter-channel dependency predicate is initialized to identify all channels as active, such that execution is not constrained by the inter-channel dependency predicate during the first execution of the identified code regions. Although control register 80 is shown as being referenced by load / store pipeline 26, it should be noted that the control register controls instruction execution by any execution unit within execution phase 18, and is therefore referenced by ALU 22, for example, when executing vector add instructions in the example loop discussed earlier.

[0104] Upon encountering an SV end instruction, a trigger is sent to replay determination circuit 75, which then determines whether status register 70 identifies any channel with an inter-channel memory hazard. If so, the replay determination circuit is configured to update the contents of control register 80 using the information maintained in status register 70 to identify the channel that needs to be re-executed. Furthermore, replay determination circuit 75 issues a replay trigger, for example, by publishing the address of the first instruction following the SV start instruction to fetch stage 6, causing the required instruction to be re-fetched for execution. Any intermediate instructions within the pipeline can then be flushed using standard mechanisms.

[0105] When the SV start instruction is executed, it causes memory element 85 to be updated to identify the address of the first instruction following the SV start instruction, and thus, for example, the address of the ld1w aggregation instruction in the specific example case discussed earlier. This memory element 85 is then referenced by the replay determination circuit 75 when needed to identify the address of the instruction when a replay trigger is issued.

[0106] Figure 3 This is a diagram illustrating in more detail the information that can be maintained within a tracking memory device 65 in one example embodiment. As shown, the tracking memory device 65 may include a plurality of entries 100, each entry being used to maintain tracking information for at least a subset of scalar memory access operations performed by executing speculative vector memory access instructions. An address information field 105 is used to identify each address involved in the entry. In one example embodiment, memory addresses that can be associated with any single entry 100 are constrained such that they are aligned with a given memory block granularity address, and thus the address information 105 can be captured within a subfield 125 that identifies the most significant bit of the address. The number of most significant bits required will depend on the size of the memory block granularity. This facilitates CAM-based lookups, as for any given address, a lookup can be performed within each entry 100 based on the information maintained in field 125 to detect if a hit exists. The address information can then be supplemented by per-byte significant bits 130 used to identify the actual address associated with the information maintained in that entry.

[0107] The type indicator field 110 is used to identify whether an entry involves a speculative vector load instruction or a speculative vector store instruction. Furthermore, in one example implementation, the type indicator also captures an indication of whether a load instruction is a contiguous or discontinuous load instruction, and similarly for store instructions, it can identify whether a store instruction performs a contiguous or discontinuous access. As previously discussed, a discontinuous load instruction may be referred to as a cluster instruction, and a discontinuous store instruction may be referred to as a scatter instruction.

[0108] The channel identification information field 115 can be used to identify one or more specific channels involved in the address information maintained in field 105. For consecutive accesses, specific channel identification information may not be necessary, as the relevant channel may be determined from the address information itself, for example, by referring to the valid bits per byte 130. Specifically, if the size of the data element that can be captured within entry 100 in an example implementation is known, the number of data elements represented by the valid bits per byte is known, and because the accesses are consecutive, this information can be directly mapped to the relevant channel.

[0109] However, for discontinuous accesses, in one example implementation, a separate entry is provided for each address, and it is not possible to identify the channel involved in the address from the address information itself. Therefore, the actual channel indication can be provided within the channel identification information field 115. It should be understood that it is important to know the channel involved in each address in order to be able to determine the presence or absence of inter-channel memory hazards, as previously stated, those inter-channel memory hazards occur because a given address is used in multiple speculative vector memory access instructions across different channels.

[0110] For storage operations, additional field 120 can be used to capture the write data stored by the storage operation. In some implementations, this write data may be stored in a separate structure, but as will be discussed in more detail later in an example implementation, a storage buffer for retaining this write data is also enhanced to capture additional information required by the tracking storage device, and thus the tracking storage device can be viewed as maintaining write data in addition to other information.

[0111] Figure 4 This is a schematic diagram illustrating how entries within the trace memory device 65 can be utilized depending on the type of access performed. Region 150 of the memory address space... Figure 4 As shown, cases A and B involve consecutive addresses accessed by speculative vector memory access instructions. If all consecutive addresses are aligned with memory block boundaries, then as indicated in case A, it may only be necessary to trace one entry in the memory device to capture trace information for all consecutive accesses, which in this example involve eight consecutive accesses.

[0112] However, as indicated in Case B, if consecutive accesses intersect with memory block boundaries, it may be necessary to track multiple entries within the memory device. In the specific example shown, four accesses occur before the memory block boundary and four occur after it, and therefore, in this scenario, two entries will be used to track within the memory device, one for the four accesses and the other for the additional four accesses.

[0113] An alternative representation 155 of the memory address space is also shown, in which a series of discontinuous accesses are performed by speculative vector memory access instructions, and specifically, an example of performing eight discontinuous accesses is shown. In one example implementation, individual entries are maintained in the tracking memory for each access, so there will be eight entries in this example. Specific channel identification information is provided within each entry using a channel identification information field 115.

[0114] The techniques described herein can be used in both sequential and out-of-order systems. However, when employing this technique in an out-of-order system, the existing architecture can be enhanced to address any vertical memory hazards that may arise from reordering instructions. Figure 5 Specific examples of implementations are shown, in which both the memory buffer (SB) 200 and the memory hazard buffer (MHB) 205 are enhanced to maintain the necessary tracking information to detect inter-channel memory hazards (i.e., horizontal memory hazards). A load / store problem queue 210 is shown, from which load and store micro-operations are output for processing by the load / store unit. Memory disambiguation and forwarding logic 215 can be used in a standard manner to detect vertical hazards, wherein a vertical hazard signal is published via path 220 when a vertical memory hazard is detected. As previously discussed, such vertical memory hazards can arise from reordering instructions due to memory aliasing occurring within the same channel across multiple reordered instructions. In the techniques described herein, vertical memory hazards always take precedence over horizontal memory hazards, and therefore, if a vertical memory hazard is detected, there is no need to further check for horizontal memory hazards, and forward processing will be determined by the vertical hazard signal published via path 220.

[0115] However, in the absence of vertical memory hazards, the memory disambiguation and forwarding logic 215 is enhanced to also implement the previously described checking circuitry to seek the detection of any horizontal (i.e., inter-channel) memory hazards. The status register 225 is updated to reflect the results of such checking operations, and specifically, for any channel where at least one type of inter-channel memory hazard is detected, a logic one value may be stored in the corresponding bit of the status register. In this example, the status register 225 has bits for each channel, and therefore a logic zero value indicates that no inter-channel memory hazard has been detected (or at least no inter-channel memory hazard of one or more types monitored by the status register), and a value of 1 indicates that one or more types of inter-channel memory hazards monitored by the status register have been detected. Thus, the status register can be viewed as establishing predicate values ​​indicating vector elements or channels affected by inter-channel memory hazards.

[0116] In addition to the common information output from the publish queue associated with load and store micro-operations, it also outputs indications of vector element size and vector element index information (required only for aggregate or scatter micro-operations), making this information available when populating store buffer entries or memory hazard buffer entries as needed.

[0117] The storage buffer 200 has an address information field 230 along with a per-byte valid bit field 240 to identify the address to which the entry relates. Fields 230 and 240 can be considered to correspond to a previous reference. Figure 3Fields 125 and 130 are discussed. Written data is also captured in field 235 because each entry in the memory buffer involves a write operation. Fields 245, 250, and 255 indicate additional fields used to enhance the memory buffer to provide sufficient information for the checking circuit 215 to detect the presence of horizontal memory hazards. Specifically, the access type field 245 is used to identify whether the access type is normal (i.e., the entry does not involve a speculative vector memory access instruction that needs to be checked by the checking circuit) or whether it involves a speculative vector memory access (i.e., an access to be checked by the checking circuit). Figure 5 As shown, this access type information can also identify whether speculative storage operations are used for sequential or discontinuous (i.e., scattering) access.

[0118] For example Figure 5 As shown, vector element size information can be captured within field 250. As previously discussed, this can be used, for example, to interpret the 240 valid bits per byte. Vector element index field 255 is also used to capture vector element index information, i.e., one or more channels to which the entry pertains. Figure 5 As shown, within the storage buffer, only scattering micro-operations require this field.

[0119] While memory buffer 200 is used to hold the addresses and data of in-flight memory operations that are still in a speculative state, a memory hazard buffer 205 is also provided. This memory hazard buffer is typically used to hold in-flight memory accesses that are being tracked out of order to ensure that their execution preserves the semantics of the original program and memory consistency model of the target ISA. For simplicity, this will focus here on memory disambiguation units targeting a relaxed consistency model that allows newer loads to be freely reordered relative to older memory, as long as there is no memory aliasing. Typically, this structure would need to hold the addresses of loads being tracked out of order and would perform checks when a memory operation is released to ensure that their addresses do not overlap with the address of one of these newer loads, otherwise pipeline squash would be triggered due to the detection of a vertical hazard. However, to support the resolution of horizontal hazards, memory hazard buffer 205 may be supplemented with additional information, such as Figure 5 As shown.

[0120] Therefore, in addition to address 260 and the per-byte valid bits field 265, the access type field 270 is used to capture whether an entry involves normal access (i.e., no checks are required to determine the presence of a horizontal memory hazard) or speculative vector load instructions that do not require reference when performing the aforementioned checks for horizontal memory hazards. Figure 5As shown, the type information can also distinguish between speculative continuous micro-operations and speculative clustering (i.e., discontinuous) micro-operations. It can also be used to distinguish another type of speculative vector load operation, referred to in this paper as speculative load and broadcast operations. Load and broadcast accesses load scalar values ​​from memory and repeat them across destination vector registers.

[0121] Similar to storage buffer 200, vector element size field 275 can be used to identify the size of vector elements, and thus enable the interpretation of each byte's valid bits 265 when determining the number of channels involved in the address information. Additionally, vector element index field 280 can be provided to store vector element index information needed to identify the channels associated with entries used for aggregation micro-operations.

[0122] It's worth noting that while for vertical disambiguation, only out-of-order loads need to be recorded in MHB205, when using horizontal memory hazard checking techniques, it may be necessary to keep track of all vector loads published in speculative regions. The compiler will know the total number of loads and stores in speculative regions and can therefore ensure that vectorization does not cause overflows in the MHB and SB structures.

[0123] Figure 6A and Figure 6B A flowchart illustrating how the identified code region can be executed based on an example layout is provided. In step 300, the process waits for the SV start instruction to be executed. Once the SV start instruction is executed, in step 305 the address of the next instruction following the SV start instruction is recorded, for example, by being written to... Figure 2 In the previously referenced storage element 85. Furthermore, the inter-channel dependency (referred to herein as P_TMD) predicate is initialized, as previously discussed; in one implementation, this predicate is maintained in control register 80.

[0124] In step 315, it is then determined whether a vector memory access instruction should be executed, and if not, the process proceeds to... Figure 6B Step 370 determines whether the next instruction is an SV end instruction. If not, the process returns to step 315. When it is determined in step 315 that a vector memory access instruction is to be executed, step 320 determines whether a vertical danger has been detected, as previously described, which is a danger caused by reordering. For example, refer to... Figure 5 As discussed, the standard memory disambiguation mechanism can be used to detect vertical hazards, and in the presence of a vertical hazard, the vertical hazard signal is activated at step 325, which will typically cause the pipeline to be refreshed.

[0125] However, if no vertical hazard is detected, the process proceeds to step 330, where it is determined whether the vector memory access instruction detected in step 315 is an execution load or store. If an execution load is performed, then in step 335, a check is performed in memory buffer 200 to determine whether a WAR (Horizontal Memory Hazard) exists for the previously executed vector store instruction. Specifically, a WAR (Inter-Channel Memory Hazard) can occur in the case of both a higher channel for a speculative vector store instruction at a given address and a lower channel for a speculative vector load instruction that appears in the original program order after the speculative vector store instruction.

[0126] In step 335, a bit vector is generated to identify the inspection results, and specifically, each channel will have a bit present, and the bit vector will be set to identify each channel for which a WAR-level memory hazard has been detected. The process then proceeds to step 350, where it is determined whether a WAR-level memory hazard has been detected, i.e., whether any bits are set in the bit vector. If not, then in step 355, each channel can obtain the required data from a memory buffer, if available, and otherwise from memory. Specifically, since there is no WAR hazard, it is safe to use the contents of the memory buffer when the required data is stored there. The data remains in the memory buffer until any potential hazard has been identified, and therefore, it is correct to obtain the data from the memory buffer, if there is no hazard, as this will be the most up-to-date data that should be available when processing the load operation.

[0127] However, if a WAR-level memory hazard is detected, in step 360, for each channel associated with a set bit in the bit vector, data is obtained from memory, for example, by accessing the Level 1 data cache 30 instead of from a memory buffer. Specifically, by obtaining data from memory, the data that would otherwise be obtained through a load operation is retrieved, and thus the WAR-level memory hazard is resolved. For any other channels in the bit vector without a set bit, the required data is obtained from a memory buffer if available and otherwise from memory.

[0128] If, in step 330, it is determined that the memory access instruction is a store instruction, the process proceeds to step 340, where a store is performed on the memory buffers of all active channels. The active channels will be determined by P_TMD (i.e., the inter-channel dependency predicate) and any explicit predicates specified for the vector store instruction. As previously discussed, during the initial execution of the identified code region (i.e., the code region between the SV start and SV end instructions), all inter-channel dependency predicates will be set such that they do not constrain the operations performed when instructions within the identified code region are executed.

[0129] In parallel with the storage operation, then in step 345, a check is performed in memory hazard buffer 205 to determine whether any RAW-level memory hazards exist using the previously executed vector load instructions. Additionally, a check may be performed in storage buffer 200 to determine whether any WAW-level memory hazards exist using the previously executed vector store instructions.

[0130] The process then proceeds to step 365, where the status register is updated to identify the result of the check performed in step 345. Specifically, bits can be set for each affected channel where a hazard has been detected. The contents of the status register are arranged to be sticky, so once they are set, they are not cleared until any replay determination is made in response to the execution of the SV end instruction.

[0131] After executing any of steps 355, 360, or 365, or after the "not" path in step 315, the process proceeds to step 370, where it is determined whether the next instruction is an SV end instruction. If not, the process returns to step 315.

[0132] However, once the SV end instruction is encountered, the process proceeds to step 375, where it is determined whether any conflict has been detected, i.e., whether any bits in the status register have been set. If not, the process proceeds to step 380, where it is determined that no replay is needed. Then, the process proceeds to the next instruction after the SV end instruction.

[0133] However, if a conflict is detected in step 375, the inter-channel dependency predicate is set to equal the contents of the status register, and the status register is subsequently cleared in step 390. A replay is initiated at step 395, where the replay determination circuit 75 issues a trigger signal to identify the instruction immediately following the SV start instruction as the next instruction. As previously discussed, the replay determination circuit 75 can obtain this information from storage element 85. This replay triggering procedure may involve sending a signal to fetch stage 6 to cause instructions to begin fetching from the identified instruction, where the remaining instructions in the pipeline are flushed.

[0134] In one example implementation, status register 70 has only a single bit per channel and therefore does not distinguish between the detection of RAW-level memory hazards and WAW-level memory hazards. However, Figure 7 An alternative implementation is shown, in which the status register has multiple bits per channel to enable differentiation between WAW and RAW level memory hazards, and specifically... Figure 7 It shows the executable sequence of steps instead of Figure 6BStep 385. In step 400, it is determined whether any channel has a RAW hazard. If not, the process proceeds to step 420, where the inter-channel dependency predicate is set to identify each channel whose status register indicates a RAW hazard. Then, by proceeding to step 390, the process continues with the previously discussed steps. Figure 6B The rest of the process.

[0135] However, if at least one channel is determined to have a WAW risk in step 400, then in step 405, the relevant storage buffer entry is updated. Specifically, the storage buffer entry storing the write data for each such channel for which a WAW risk has been detected is updated to prevent that data from being output to memory. For the example of the first and second vector store instructions appearing in this order in the original program sequence, this would involve updating the storage buffer entry storing the write data generated by the second vector store instruction to prevent data in one or more of its channels from being output to memory.

[0136] Subsequently, in step 410, it is determined whether any channel is at risk of RAW, and if so, the process proceeds to step 420. However, if no channel is at risk of RAW, it is determined in step 415 that no replay is required. The process can then proceed to the next instruction after the SV end instruction.

[0137] In one example implementation, all vector memory access instructions occurring between the SV start and SV end instructions are considered speculative vector memory access instructions and are therefore monitored by the checking circuitry. However, in an alternative implementation, to account for horizontal memory hazards, vector memory access instructions may be flagged to distinguish those that are speculative and therefore should be monitored by the checking circuitry from those known not to cause inter-channel memory hazards and therefore ignored by the checking circuitry. This is in... Figure 8 The diagram schematically illustrates that the vector memory access instruction 455 has an associated tag 460.

[0138] Tagging information can take several forms. For example, as a first option, it can be a single bit value, where setting a status indicator should perform a check and clearing a status indicator does not require checking for horizontal memory hazards. However, if needed, as an alternative, the tag can be multiple bit fields, which allows vector memory access instructions to be grouped for checking. By identifying different groups using tagging information, each speculative vector memory access instruction can then be checked relative to every other speculative vector memory access instruction in that group, but no checks need to be performed on those instructions relative to vector memory access instructions in another group. Therefore, by using such tagging information, it is possible to reduce the amount of checking required to determine the presence or absence of horizontal memory hazards. This could potentially allow for larger loops to be vectorized, since the address information of any “not checked” instructions does not need to be recorded in the tracking memory at runtime.

[0139] Figure 9 A specific implementation of a usable simulator is illustrated. While the previously described example implementations emulate the invention in terms of apparatus and methods for operating specific processing hardware supporting the technologies involved, it is also possible to provide an instruction execution environment implemented using a computer program according to the examples described herein. Such computer programs are generally referred to as simulators, in part because they provide a software-based implementation of a hardware architecture. Types of simulator computer programs include emulators, virtual machines, models, and binary converters, including dynamic binary converters. Typically, the simulator implementation can run on a host processor 530 that supports the simulator program 510, which optionally runs a host operating system 520. In some arrangements, multiple emulation layers may exist between the hardware and the provided instruction execution environment and / or multiple different instruction execution environments provided on the same host processor. Historically, powerful processors were required to provide simulator implementations that execute at a reasonable speed, but this approach may be reasonable in certain situations, such as when it is desirable to run code native to another processor for compatibility or reuse reasons. For example, the simulator implementation may provide additional functionality to the instruction execution environment that is not supported by the host processor hardware, or provide an instruction execution environment that is typically associated with a different hardware architecture. An overview of simulation is given in “Some Efficient Architecture Simulation Techniques (Robert Bedichek, Winter 1990 USENIX Conference, Pages 53–63)”.

[0140] With respect to the embodiments previously described with reference to specific hardware constructions or features, in simulated embodiments, equivalent functionality may be provided by suitable software constructions or features. For example, specific circuitry may be implemented as computer program logic in simulated embodiments. Similarly, memory hardware such as registers or cache memory may be implemented as software data structures in simulated embodiments. One or more hardware elements referenced in the previously described embodiments are present in an arrangement on host hardware (e.g., host processor 530), and where appropriate, some simulated embodiments may utilize the host hardware.

[0141] The simulator program 510 may be stored on a computer-readable storage medium (which may be a non-transitory medium) and provides a program interface (instruction execution environment) to the target code 500 (which may include an application program, operating system, and management program). This program interface is identical to the application interface of the hardware architecture modeled by the simulator program 510. Therefore, the program instructions of the target code 500 (including the aforementioned SV start and SV end instructions) can be executed from within the instruction execution environment using the simulator program 510, enabling the host computer 530, which does not actually possess the hardware features of the aforementioned device 2, to emulate these features.

[0142] Based on the examples above, it should be understood that the techniques described herein increase the ability to vectorize scalar code by allowing speculative vectorization of code regions that include multiple vector memory access instructions, without needing to determine whether any TMDs are likely to occur before performing such vectorization. Instead, a mechanism is provided that allows memory address conflicts to be detected in hardware during execution and then automatically triggered for selective re-execution only for the specific affected channel. This can lead to a significant performance improvement due to the ability to vectorize code that was previously not vectorizable.

[0143] In this application, the phrase "configured as..." is used to mean that the elements of the device have a configuration capable of performing the defined operations. In this context, "configuration" means the arrangement or manner of interconnection of hardware or software. For example, the device may have dedicated hardware that provides the defined operations, or a processor or other processing device may be programmed to perform the function. "Configured as" does not mean that the elements of the device need to be changed in any way to provide the defined operations.

[0144] While exemplary embodiments of the invention have been described in detail herein with reference to the accompanying drawings, it should be understood that the invention is not limited to those precise embodiments, and various changes, additions, and modifications can be made therein by those skilled in the art without departing from the scope and spirit of the invention as defined by the appended claims. For example, features of the dependent claims may be combined with features of the independent claims in various ways without departing from the scope of the invention.

Claims

1. Apparatus for speculative vectorization of program code, the apparatus comprising: processing circuitry to execute program code, the program code comprising an identified code region comprising at least a plurality of speculative vector memory access instructions, wherein execution of each speculative vector memory access instruction employs speculative vectorization to perform a series of scalar memory access operations using a plurality of processing lanes; wherein the processing circuitry is arranged to perform out-of-order, OOO, processing of instructions; trace storage to maintain, for each speculative vector memory access instruction, trace information providing an indication of memory addresses accessed within each lane; checking circuitry to reference the trace information during execution of the identified code region by the processing circuitry in order to detect any inter-lane memory hazards arising from the execution of the plurality of speculative vector memory access instructions; at least one OOO trace structure having trace entries to track memory hazards introduced by instruction reordering; wherein the trace entries in the at least one OOO trace structure are augmented such that the trace storage is incorporated within the at least one OOO trace structure, thereby allowing the checking circuitry to detect the inter-lane memory hazards occurring as a result of assigning scalar memory access operations within the series to different processing lanes; state storage elements to maintain an indication of each lane determined by the checking circuitry to have at least a first type of inter-lane memory hazard; and replay determination circuitry arranged to, when the end of the identified code region is reached, trigger re-execution of the identified code region for each lane identified by the state storage elements as having an inter-lane memory hazard in response to the state storage elements identifying at least one lane as having an inter-lane memory hazard.

2. The apparatus of claim 1, the apparatus further comprising: control storage elements to store inter-lane dependency predicates referenced by the processing circuitry in execution of the identified code region; wherein: in an initial execution of the identified code region, the inter-lane dependency predicates are initialized to identify all lanes as active such that the execution of each speculative vector memory access instruction is unconstrained by the inter-lane dependency predicates; and in a re-execution of the identified code region, the inter-lane dependency predicates are set to constrain re-execution of the identified code region to the lanes identified in the state storage elements.

3. The apparatus of claim 1 or claim 2, wherein: the start of the identified code region is identified by a speculative vectorization, SV, start instruction; the processing circuitry is responsive to execution of the SV start instruction to record an address indication of a first instruction following the SV start instruction; and the state storage elements are responsive to the address indication to identify a first lane in which the first instruction following the SV start instruction is executed. The replay determination circuitry is responsive to determining that re-execution of the identified code region is required to output the address indication of the first instruction to cause re-execution from the first instruction.

4. The apparatus of claim 3, when dependent on claim 2, wherein the processing circuitry is further responsive to execution of the SV start instruction to initialize the inter lane dependency predicate.

5. The apparatus of claim 1, wherein: the end of the identified code region is identified by a speculative vector (SV) end instruction; and the processing circuitry is responsive to execution of the SV end instruction to cause the replay determination circuitry to determine whether any lane requires re-execution of the identified code region.

6. The apparatus of claim 1, wherein the speculative vector memory access instructions are introduced by a compiler seeking to vectorize scalar code, and the compiler is arranged to annotate the program code to identify the instructions forming the identified code region.

7. The apparatus of claim 1, wherein the identified code region forms at least part of a vector loop for a vectorized scalar loop.

8. The apparatus of claim 1, wherein: the tracking information is arranged to provide, for each speculative vector memory access instruction, a type indication to indicate whether the speculative vector memory access instruction is a speculative vector load instruction to load data from memory, or a speculative vector store instruction to store data to memory; and the checking circuitry is arranged to reference the type indication when determining whether an inter lane memory hazard has been detected.

9. The apparatus of claim 8, wherein the type indication is further arranged to identify whether the speculative vector memory access instruction performs an access to contiguous memory addresses or to non-contiguous memory addresses.

10. The apparatus of claim 1, wherein the tracking storage comprises a plurality of entries, wherein each entry associated with a speculative vector memory access instruction is to store the tracking information for at least a subset of the scalar memory access operations performed by execution of the speculative vector memory access instruction.

11. The apparatus of claim 10, wherein each entry in the tracking storage is constrained to store tracking information for one or more memory addresses that are aligned to a given memory block granularity address.

12. The apparatus of claim 11, wherein for a speculative vector memory access instruction to perform an access to contiguous memory addresses, when the contiguous memory addresses cross a plurality of address aligned memory blocks, more than one entry in the tracking storage is allocated to store the associated tracking information.

13. The apparatus of claim 11 or claim 12, wherein for a speculative vector memory access instruction to perform an access to non-contiguous memory addresses, a separate entry in the tracking storage is allocated to store the associated tracking information for each memory address accessed.

14. The apparatus of claim 10, wherein for each memory address identified within an entry in the tracking storage, the tracking information maintained in the entry is arranged to provide lane identification information sufficient to determine the lane with which each memory address is involved.

15. The apparatus of claim 1, wherein: each speculative vector memory access instruction is a speculative vector load instruction for loading data from memory, or a speculative vector store instruction for storing data to memory; the plurality of processing lanes extends from a lowest lane associated with a first scalar memory access operation in the series vectorized by the speculative vector memory access instruction to a highest lane associated with a final scalar memory access operation in the series; the checking circuitry is arranged to detect one or more of the following inter-lane memory hazards: a read after write (RAW) inter-lane memory hazard in which a given address is used by both a higher lane of a speculative vector load instruction and a lower lane of a speculative vector store instruction that appears after the speculative vector load instruction in original program order; a write after read (WAR) inter-lane memory hazard in which a given address is used by both a higher lane of a speculative vector store instruction and a lower lane of a speculative vector load instruction that appears after the speculative vector store instruction in original program order; a write after write (WAW) inter-lane memory hazard in which a given address is used by both a higher lane of a first speculative vector store instruction and a lower lane of a second speculative vector store instruction.

16. The apparatus of claim 15, further comprising: a store buffer for temporarily storing write data generated by execution of each speculative vector store instruction, wherein the write data of the given speculative vector store instruction is inhibited from being output from the store buffer to memory until any inter-lane memory hazard resulting from the execution of the given speculative vector store instruction has been resolved.

17. The apparatus of claim 16, wherein: when the checking circuitry detects that there is a WAR inter-lane memory hazard, the checking circuitry is arranged to resolve the WAR inter-lane memory hazard by causing the processing circuitry to retrieve the data for the given address associated with the lower lane from memory rather than from the store buffer when executing the speculative vector load instruction; and the state storage element is not updated in response to an identified WAR inter-lane memory hazard.

18. The apparatus of claim 16 or claim 17, wherein the state storage element is updated in response to at least each RAW inter-lane memory hazard detected by the checking circuitry, and the lane identified in the state storage element is the higher lane of the speculative vector load instruction.

19. The apparatus of claim 18, wherein the state storage element is further updated in response to each WAW inter- lane memory hazard detected by the checking circuitry, and the lane identified in the state storage element is the higher lane of the first speculative vector store instruction.

20. The apparatus of claim 16 or claim 17, wherein: when the checking circuitry detects that there is a WAW inter-lane memory hazard, the checking circuitry is arranged to resolve the WAW inter-lane memory hazard by causing the store buffer to be marked such that the write data produced in the lower lane during execution of the second speculative vector store instruction is inhibited from being output to memory; and the state storage element is not updated in response to an identified WAW inter-lane memory hazard.

21. The apparatus of claim 1, wherein: the at least one OOO tracking structure comprises a store buffer to temporarily store write data produced by execution of a store instruction until it is determined that the write data can be committed to memory; and the store buffer is augmented to allow the tracking information to persist therein for speculative vector store instructions within the identified code region.

22. The apparatus of claim 1 or claim 21, wherein: the at least one OOO tracking structure comprises a memory hazard buffer to track execution of out-of-order issued memory access instructions; and the memory hazard buffer is augmented to allow the tracking information to persist therein for speculative vector load instructions within the identified code region.

23. The apparatus of claim 1, wherein each vector memory access instruction in the identified code region is assumed to be a speculative vector memory access instruction, such that the checking circuitry is arranged to check for inter-lane memory hazards associated with execution of each vector memory access instruction in the identified code region.

24. The apparatus of claim 1, wherein: the vector memory access instructions are marked to distinguish between those vector memory access instructions that are speculative vector memory access instructions and therefore should be monitored by the checking circuitry, and those vector memory access instructions that are known not to incur an inter-lane memory hazard and therefore can be ignored by the checking circuitry.

25. A method of speculative vectorization of program code, the method comprising: executing program code with processing circuitry, the program code comprising an identified code region, the identified code region comprising at least a plurality of speculative vector memory access instructions, wherein execution of each speculative vector memory access instruction is employed to perform a speculative vectorization of a series of scalar memory access operations using a plurality of processing lanes; wherein the processing circuitry is arranged to perform out-of-order (OOO) processing of instructions; for each speculative vector memory access instruction, maintaining tracking information within a tracking storage, the tracking information providing an indication of memory addresses accessed within each lane; to detect any inter-lane memory hazards arising from the execution of the plurality of speculative vector memory access instructions; maintaining tracking entries with at least one OOO tracking structure to track memory hazards introduced by instruction reordering; wherein the tracking entries in the at least one OOO tracking structure are augmented such that the tracking storage is incorporated within the at least one OOO tracking structure, thereby allowing the checking circuitry to detect the inter-lane memory hazards occurring as a result of the scalar memory access operations within the series being allocated to different processing lanes; within a state storage element, maintaining an indication of each lane for which the checking circuitry determines at least a first type of inter-lane memory hazard; and when the end of the identified code region is reached, and the state storage element identifies at least one lane as having an inter-lane memory hazard, triggering a re-execution of the identified code region for each lane identified by the state storage element.

26. A computer program product for controlling a host data processing apparatus to provide an instruction execution environment, and comprising: processing program logic for executing program code, the program code comprising an identified code region, the identified code region comprising at least a plurality of speculative vector memory access instructions, wherein execution of each speculative vector memory access instruction is employed to perform a speculative vectorization of a series of scalar memory access operations using a plurality of processing lanes; wherein the processing program logic is arranged to perform out-of-order OOO processing of instructions; a tracking data structure for maintaining tracking information for each speculative vector memory access instruction, the tracking information providing an indication of memory addresses being accessed within each lane; checking program logic for referencing the tracking information during execution of the identified code region by the processing program logic, in order to detect any inter-lane memory hazards arising from the execution of the plurality of speculative vector memory access instructions; at least one OOO tracking structure having tracking entries to track memory hazards introduced by instruction reordering; wherein the tracking entries in the at least one OOO tracking structure are augmented such that the tracking data structure is incorporated within the at least one OOO tracking structure, thereby allowing the checking program logic to detect the inter-lane memory hazards occurring as a result of the scalar memory access operations within the series being allocated to different processing lanes; a state data structure for maintaining an indication of each lane for which the checking program logic determines at least a first type of inter-lane memory hazard; and The replay determination program logic is arranged to, on reaching the end of the identified code region, in response to the state data structure identifying at least one lane as having inter-lane memory hazards, trigger re-execution of the identified code region for each lane identified by the state data structure.

Images