Method and apparatus for data processing

Abstract

In response to an instruction decoder decoding a range prefetch instruction specifying first and second address ranges and a step parameter, the prefetch circuitry controls the prefetching of data from a plurality of specified address ranges to at least one cache memory, depending on the first and second address ranges and the step parameter. The initial address and size of each specified range depend on the first and second address ranges. The step parameter specifies an offset between the initial addresses of consecutive specified ranges. The use of this range prefetch instruction helps improve programmability and balances the prefetch coverage with the circuit area of ​​the prefetch circuitry.

Description

[Technical Field]

[0001] This technology relates to the field of data processing. More specifically, it relates to pre-extraction. [Previous Technology]

[0002] Prefetching is used in data processing systems to load data into cache memory before the time the data is requested by the processing circuitry. This removes the latency associated with processing cache misses or translation back buffer misses from the critical timing path during actual access requests generated by the processing circuitry, thereby improving performance. [Summary of the Invention]

[0003] At least some embodiments provide an apparatus comprising: an instruction decoder for decoding instructions; a processing circuitry for performing data processing in response to the decoding of the instructions by the instruction decoder; at least one cache memory for caching data accessed by the processing circuitry; and a prefetch circuitry for prefetching data into the at least one cache memory; wherein: in response to the instruction decoder decoding a range prefetch instruction specifying first and second address range specification parameters and a step parameter, the prefetch circuitry is configured to control the prefetching of data from a plurality of specified address ranges to the at least one cache memory depending on the first and second address range specification parameters and the step parameter, wherein the initial address and size of each specified range are dependent on the first and second address range specification parameters, and the step parameter specifies an offset between the consecutive initial addresses of the plurality of specified ranges.

[0004] At least some instances provide a method comprising: decoding instructions; performing data processing in response to decoding of the instructions by the instruction decoder; caching data in at least one cache memory for access by the processing circuitry system; and prefetching data into the at least one cache memory; wherein: in response to decoding of a range prefetch instruction specifying first and second address range specification parameters and a step parameter, the prefetching of data from a plurality of specified address ranges to the at least one cache memory is controlled depending on the first and second address range specification parameters and the step parameter, wherein the initial address and size of each specified range are dependent on the first and second address range specification parameters, and the step parameter specifies an offset between the plurality of consecutive starting addresses of the specified ranges.

[0005] Further features, characteristics and advantages of this technology will become apparent from the following embodiments, which are read in conjunction with the accompanying drawings.

Implementation Method

[0007] One method of performing prefetching is to use hardware prefetching, wherein a hardware-implemented prefetching circuitry monitors the addresses of demand access requests for data issued by the processing circuitry, enabling the hardware prefetcher to learn the resulting access patterns and subsequently issue prefetch requests to load data at the given address into cache memory before anticipated demand access to the same address. Using a typical hardware prefetcher, the prefetcher can learn from the addresses of demand accesses and from microarchitectural clues (such as cache misses) to determine how to issue future prefetch requests without explicit hints provided by software based on parameters of software prefetch instructions specified by the programmer or compiler. Typical hardware prefetching schemes can be effective, but may have some drawbacks. The hardware prefetcher may need to monitor a reasonable number of demand accesses before the prefetcher can be trained and understand the address stream detected on demand accesses, and therefore may be difficult to perform effective data prefetching on the initial portion of the address stream. If a particular software application operates on a very small stream, the hardware prefetcher may not have enough samples to identify the stream and prefetch in a timely manner, because the stream may have ended by the time the prefetcher has gained sufficient confidence in its predictions. This results in a loss of coverage for the prefetcher and thus a loss of performance. The hardware prefetcher may also need to store large amounts of prefetch prediction state data to track demand access behavior and use it to evaluate the confidence in predicted address access patterns. For example, the training unit of a typical hardware prefetcher may have a hardware budget of tens of kilobytes, consuming a significant amount of power and energy. Another problem with typical hardware prefetchers is that they often do not know when the address stream will end, and even once the prefetcher has been successfully trained to predict future accesses within a certain address stream, the hardware prefetcher often continues to prefetch beyond the end of the stream by a considerable distance after the executing software has moved on from that stream and begun operating on data in other address regions. Such inaccurate prefetching can waste available memory system bandwidth and cause cache pollution (e.g., replacing more useful cache entries with over-prefetched data), which can reduce cache hit rates for other accesses and thus reduce cost-effectiveness.

[0008] Another method of prefetching can be software prefetching, in which the programmer or compiler includes a prefetch instruction specifying a single target address of data expected to be used in the future within the code to be executed, and in response to the single address-specified prefetch instruction, the prefetch circuitry controls the prefetching of the data corresponding to that single address. For example, the prefetch circuitry may issue a request for a single cache line to be loaded into cache memory in response to a single address-specified software prefetch instruction. By providing software prefetch instructions, the programmer or compiler can provide explicit indication of the future intent to access a particular address, thus initiating cache memory ready for later access.

[0009] However, software prefetching using this single-address-specified prefetch instruction still has several drawbacks. Programmers may find it very difficult to add such software prefetch instructions to the code. Since each instruction specifies a single address, in practice, the code will need to include a large number of such software prefetch instructions corresponding to different addresses that the code is expected to access later. The point at which to insert prefetch instructions into the code may not always be obvious to software developers. Furthermore, since the effectiveness of prefetching will depend on how far in advance the corresponding single-address-specified range prefetch instruction is inserted for a certain required access, it is difficult to control the timeliness of prefetching using this method. If the single-address-specified prefetch instruction is included too early, the prefetched data may have already been evicted from cache memory when a subsequent required access occurs, wasting resources associated with performing the prefetch. If the single-address-specified range prefetch instruction is included too late, the prefetched data may not yet reside in cache memory when a subsequent required access occurs, and therefore the required access may still encounter a cache miss. Therefore, using general software prefetch instructions that specify a single address to make timing correct becomes a trial-and-error process, which means that software developers have to spend a lot of development time figuring out the best place to add software prefetch instructions.

[0010] Another problem with traditional software prefetching is that code that uses a single address to specify software prefetch instructions becomes specific to the microarchitecture of the processor core and memory system. Code written according to an instruction set architecture can run on a wide variety of different processing systems with different microarchitecture designs. Depending on factors such as the number of cache levels provided, the size of the cache storage provided at a given cache level, the cache line size (the size of the basic unit of memory that moves between memory and cache), how many other access request sources are competing for memory system bandwidth, and many other specific properties of the particular processor implementation in the hardware, the code can experience different latency when data is fetched from memory to cache and can have different cache hit / miss rates. These factors mean that when code executes on one microarchitecture platform, the optimal timing for prefetching can be earlier than when the same code executes on another microarchitecture platform. This means that it is extremely difficult to write code that can run efficiently across a range of microarchitectures using single-address specified software prefetch instructions. The development cost of producing different code versions for different microarchitectures would be very high.

[0011] In an example discussed below, a device has an instruction decoder for decoding instructions and a processing circuitry system for performing data processing in response to the decoding of the instructions by the instruction decoder. At least one cache memory is provided for caching data accessed by the processing circuitry system. A prefetch circuitry system is provided for prefetching data into the at least one cache memory. The instruction decoder supports a range prefetch instruction specifying first and second address range specification parameters and a step parameter. In response to the instruction decoder decoding the range prefetch instruction, the prefetch circuitry system controls the prefetching of data from a plurality of specified address ranges into the cache memory, depending on the first and second address range specification parameters and the step parameter, wherein the initial address and size of each specified range are dependent on the first and second address range specification parameters, and the step parameter specifies an offset between the plurality of consecutive starting addresses of the specified ranges.

[0012] Therefore, range prefetch instructions can be used by software to signal several specified memory address ranges that will be accessed in the future. The stride parameter allows a single instruction to provide a prefetch hint for a subsequent access pattern of several data blocks of a certain size that are discontinuous in the address space and separated by a constant stride interval. This access pattern can be useful for many machine learning applications, such as in machine learning models that use convolutions of weighted kernels and activation data matrices to train machine learning models. Such convolutions may rely on applying kernels to different locations within the matrix, and for each kernel location, this may require reading several relatively short data blocks representing portions of different columns or rows in the matrix, with gaps between address ranges accessed for activation values ​​used at a given kernel location. By specifying a stride pattern of discontinuous ranges, it is easier to efficiently train the prefetch circuitry to learn to effectively prefetch such access patterns. It should be understood that this type of convolution is only one instance of an access pattern that can benefit from prefetching instructions from the boundaries of a specified address range and the stride range, and other operations can also benefit from it.

[0013] The prefetch circuitry can then respond to these prompts to control how it prefetches data from the specified address range into cache memory. Compared to the previously mentioned single-address prefetch instructions, range prefetch instructions are easier for programmers or compilers to use because they can include a single range prefetch instruction that specifies an address partially associated with the given data structure to be processed, rather than inserting a large number of prefetch instructions corresponding to individual addresses. This also simplifies software development by significantly reducing the number of prefetch instructions that must be included in the code, thus reducing the amount of fetch, decode, and publish bandwidth consumed by prefetch instructions and enabling greater throughput for other types of instructions. Since timeliness can be hardware-controlled by the prefetch circuitry within the boundaries of the specified address range provided as a prompt by the range prefetch instructions, programmers using range prefetch instructions do not need to worry about the insertion timeliness of a given prefetch instruction relative to the required access. The use of first and second address range specification parameters and step parameters in range prefetch instructions also means that, compared to the use of prefetch instructions specifying a single address, applications become less microarchitecture and system architecture dependent, making it simpler to write code that can be executed across a range of different microarchitecture processor implementations.

[0014] On the other hand, compared to general hardware prefetching which does not use any software-specified hints provided by prefetch instructions, the use of range prefetch instructions is advantageous because they explicitly specify parameters that identify the boundaries of the address ranges expected to be accessed in the future. This means that when access begins from an address stream with predictable properties, the prefetching circuitry can quickly start up to prefetch from those ranges, rather than requiring a long initial training period. Therefore, in practice, when accessing data within a range corresponding to a given data structure, in a purely hardware prefetching implementation, it is unlikely that the initial portion of the range will be successfully prefetched because the prefetcher may still be gaining confidence. Conversely, using range prefetch instructions, the first address range specification parameter can be used to identify the start of the range, and thus the warm-up time can be significantly reduced, improving the prefetching success rate and thus improving performance. Range prefetch instructions can be particularly useful when the demand address stream following a predictable pattern is very small, which will be difficult to prefetch effectively using a purely hardware prefetcher.

[0015] Similarly, since the size of a given address range can be identified from the parameters specified in the first and second address ranges, this reduces the problem of over-prefetching beyond the end of the data structure being accessed, avoids unnecessary consumption of prefetch bandwidth, and avoids the problem of other data being evicted by over-prefetched data, which can lead to performance issues in pure hardware prefetchers. Generally speaking, for a given hardware circuit area budget, a prefetcher using hints provided by range prefetch instructions can provide a greater prefetch coverage (a greater percentage of demand accesses that hit on previously prefetched data, otherwise the demand access would miss in cache memory) than an equivalent hardware prefetcher that learns its prefetching from implicit cues inferred from demand access monitoring and microarchitectural information (such as access hit / miss information), and therefore provides better performance. On the other hand, for a given prefetch coverage level, since maintaining an equivalent performance level without using such range prefetch cues would generally require storing a larger training dataset and comparing the training dataset with the required access, a prefetcher that can be controlled based on range prefetch instructions can require less circuit area than a conventional hardware prefetcher that does not support range prefetch cues received from range prefetch instructions.

[0016] Therefore, in summary, by supporting range prefetch instructions, this can provide a better balance between prefetch coverage (performance) and circuit area and power consumption than it might be possible to use traditional software and hardware prefetch solutions, and can also achieve better programmability, making it easier for software developers to write code that includes prefetch instructions.

[0017] The first and second address range specification parameters may be any two software-specified parameters of the range prefetch instruction that can be used to identify the boundary of one of the selected address ranges. For example, the selected range may be the first range of a plurality of address ranges identified using the instruction. The boundaries of the other ranges may be determined by applying an offset indicated by the stride to the boundary of one of the selected ranges.

[0018] The first and second address range specification parameters can be specified in different ways. For example, the first and second address range specification parameters can each specify the memory address as an absolute address, or specify the memory address by referencing a reference address other than the first and second address range specification parameters themselves. For example, the first address range specification parameter can specify the start address of the selected range and the second address range specification parameter can specify the end address of the selected range, taking an absolute term relative to a reference address (such as a program counter indicating the current point reached during execution).

[0019] However, since it is relatively unlikely that the size of the specified range will be larger than a certain amount, fewer bits can be used if one of the start and end addresses is specified relative to the other. In some instances, the start address may be specified relative to the end address. However, it is generally more natural to encode the start address of the specified range as the base address in the operand of the range prefetch instruction, and to express the end address relative to the start address (e.g., by a size parameter or offset value). Thus, a useful example could be that the first address range specification parameter contains the base address (start address) of the specified range and the second address range specification parameter contains a range size parameter specifying the size of each of the specified ranges. This would allow the end address of the specified range to be determined based on applying an offset corresponding to the specified size to the base address. The specified base and size can specify the boundary of the first specified range, and the stride can then be used to identify the start / end of subsequent ranges in the step range group. The ranges in the step range group can have the same size.

[0020] The first and second address range specification parameters can be encoded in the range prefetch instruction in different ways. For example, each of the first and second address range specification parameters can use an immediate value directly encoded in the instruction code of the range prefetch instruction or a register code identified by the corresponding register specifier in the code of the range prefetch instruction.

[0021] In the case where the first address range specifies the base address of the parameter encoding and the second address range specifies the size of the parameter encoding range, a useful example is to encode the first address range specified parameter in the first register specified by the range prefetch instruction and encode the second address range specified parameter in the second register specified by the range prefetch instruction, or by directly specifying the immediate value encoding in the encoding of the range prefetch instruction.

[0022] Regardless of the encoding method of the address range specification parameter, in some instances, the encoding of the address range specification parameter can support indicating a given specified address range as an address block spanning several bytes beyond a power of exactly two. For example, the size of the specified range can be any chosen number of bytes up to a maximum size. The maximum size of a given specified address range can be greater than the size of a single cache line, where the cache line is the size of a data unit that can be moved between cache memories or between cache memories in a single transfer. Therefore, the address range that can be specified by a range prefetch instruction is not limited to a single cache line and can be any arbitrary size range, making it not limited to blocks of power-of-two sizes. This allows the software to select a range that matches the boundary of a portion of a data structure that the software will process. It is generally impossible for a single-address prefetch instruction specifying a single address to result in bringing a single cache line into a byte block corresponding to a power-of-two size (thus requiring further single-address prefetch instructions to prefetch other portions of the data structure spanning multiple cache lines) or a cache memory with a size other than a power of two. Similarly, the encoding of the stride parameter can support indicating offsets as several bytes other than powers of two.

[0023] Alternatively, other instances may restrict the encoding of size and / or stride to a power of 2 number of bytes, thus not supporting specifying size or stride at a single-byte granularity. For example, size or stride parameters may be specified as multiples of a base cell size, such as 2, 4, or 8 bytes.

[0024] It should be understood that other techniques may also be used for the size or stride of the encoding range.

[0025] In some instances, the range prefetch instruction may also specify a count parameter indicating how many ranges separated by the stride offset will be prefetched. This allows prefetching to determine when to stop prefetching from multiple ranges. The stride parameter and the count parameter can be encoded in different ways, such as using immediate values ​​or values ​​in registers. It may be possible to use a different register encoding for the stride or count value than the register used to specify the parameter for the first or second address range.

[0026] However, in an instance where the first address range specification parameter specifies a base address and the second address range specification parameter is specified as a size or an offset relative to the base address indicated by the first address range specification parameter, fewer bits may be required for the second address range specification parameter than for the first parameter, and therefore some spare bits may be available in the register used to provide the second address range specification parameter. Thus, in some cases, at least one of the step parameter and the count parameter may be encoded in the same register as the second address range specification parameter. This reduces the number of register specifiers required in the range prefetch instruction encoding and saves encoding space that can be reused for other purposes.

[0027] It should be understood that while the range prefetch instruction supports the stride parameter and count parameter described earlier, the range prefetch instruction can also support encoding that indicates only a single address range to be prefetched, wherein the boundaries of the range are determined based on parameters specified for the first and second address ranges. For example, when the count parameter or stride parameter has a certain preset value, this can indicate that only a single range will be prefetched. Therefore, all instances of the range prefetch instruction do not need to trigger prefetching from multiple ranges separated by stride instances, but nevertheless, the range prefetch instruction has encoding that enables the stride parameter (and optionally the count parameter) to be determined, giving programmers the option to use a single instruction to start the prefetcher to prefetch data from multiple ranges separated by constant stride intervals.

[0028] The first and second address range specification parameters and step parameters provided by the range prefetch instruction may not be the only information used by the prefetch circuitry to control the prefetching of data from the specified address range to at least one cache memory. Prefetching may also depend on at least one of the following: the address associated with a demand access request issued by the processing circuitry to request access to data or monitoring of loaded data; and microarchitectural information associated with the result of the prefetch request or demand access request. Therefore, while the indication of the boundaries of the address range required for demand access is expected to be useful for starting the prefetcher, reducing prefetch warm-up time, avoiding over-prefetching, and simplifying software development for reasons given earlier, the prefetch hardware may have the flexibility to determine which addresses within the specified range should be prefetched and the timing of such prefetchings, within the specified range. By monitoring the address and / or data of a demand access request and the microarchitectural information indicating the result of the prefetch request or demand access request, the prefetch circuitry can make microarchitectural-dependent decisions to achieve improved prefetch coverage for a given microarchitectural implementation. However, guided by the range specification parameters of the range prefetch instructions, this may require a smaller circuit area budget than alternative hardware prefetching without such range indication. Therefore, prefetching can be dynamically adjusted by the prefetch circuitry in the hardware, but guided by information on the range boundaries provided by the software-defined range prefetch instructions. Loaded data associated with a demand access request can, for example, be used to control prefetching (e.g., in a scenario where loaded data from memory is recognized as an indicator for determining the address of subsequent accesses). Information from the loaded data of a demand access can be used to predict which addresses can be accessed in the future.

[0029] In cases where microarchitectural information is used to adjust prefetching, this microarchitectural information may include any one or more of the following: prefetch line fill latency (indicating the latency between issuing a prefetch request and the return of the corresponding prefetched data to at least one cache memory), demand access frequency (the frequency at which demand access requests are issued by the processing circuitry system), demand access hit / miss information (e.g., providing information indicating the percentage of demand access requests that are hit or missed in a given cache memory level; in some cases, this information may be maintained for different cache memory levels), demand access prefetch hit information (e.g., indicating the percentage of demand access requests that are hit on previously prefetched data, as opposed to hits on previously fetched data in response to demand access), and prefetch usefulness information (e.g., an indication of the percentage of prefetch requests that are subsequently used by demand access before the prefetched data is evicted from the cache). It is not necessary for any given implementation to consider all of these types of microarchitectural information. One or more of these types may be considered.

[0030] The prefetch circuitry can control the timing of prefetch requests issued to a plurality of given addresses within a specified range based on addresses associated with demand access requests for data issued by the processing circuitry, or microarchitectural information associated with the monitoring of loaded data or the results of prefetch requests or demand access requests. This differs from conventional software prefetch instructions, which typically act as direct triggers for specified addresses to be prefetched, rather than leaving the flexibility to determine the exact timing to the hardware. Therefore, prefetch range instructions can be viewed as a mechanism for software to initiate the prefetcher using hints about future address access patterns rather than triggers that prefetching must occur correctly, allowing the hardware to make the final decision on prefetch timing based on microarchitectural clues and the software to be more platform-independent.

[0031] In some instances, the prefetching circuitry may have a sparse access pattern detection circuitry to detect sparse patterns of accessed addresses within a given range of multiple address ranges indicated by a range prefetching instruction, based on monitoring of the addresses of demand access requests, and to select which specific addresses within the range to prefetch based on the detected sparse patterns. Even when the anticipated usage pattern is sparse, such that all addresses not within the given range are potentially accessible, a microarchitectural implementation with a sparse access pattern detection circuitry can thus benefit from the use of range prefetching instructions. For example, the sparse access pattern detection circuitry may have a training circuitry that monitors address sequences to detect alternating sequences of step address patterns or step intervals within the demand address stream and stores predictive state information indicating the detected patterns and associated confidence levels to learn which addresses are most effective for prefetching. However, guided by the address range specification parameters from the range prefetch instruction, this allows the sparse access pattern detection circuitry to start up faster, enabling better prefetching of the initial portion of the specified range. Furthermore, the prompt from the range prefetch instruction also indicates to the sparse access pattern detection circuitry when the detected mode will end, thus avoiding over-prefetching.

[0032] It should be noted that in cases where the range prefetch instruction uses a step parameter to specify a plurality of specified ranges, a sparse access pattern detection circuitry can still be applied to further detect address access step patterns within each of the specified ranges. In this case, the specified ranges identified by the range prefetch instruction can be interpreted as boundaries of regions that may require future access, but the sparse access pattern detection circuitry can learn which specific addresses in these regions are actually accessed and thus train the prefetch circuitry.

[0033] Other embodiments may not have a sparse access pattern detection circuitry system and may instead use a simpler hardware implementation with a prefetch circuitry system, which may assume that the access pattern within a given range is a dense access pattern, such that all addresses within the given range should be prefetched.

[0034] Another form of hardware-controlled prefetch control may be an advance distance, which controls how much the prefetch address stream of the prefetch request issued by the prefetch circuit system is ahead of the demand address stream of the demand access request issued by the processing circuit system to request access to data. Based on the advance distance, the prefetch circuit system can control the timing of the prefetch request issued for the address.

[0035] It is useful to define the advance distance (or any implicit specification of the advance distance based on the position of the range prefetch instruction relative to other code) independently of any explicit specification of the prefetch instruction by the prefetch circuitry system. For example, the advance distance can be set based on a specific microarchitecture implementation of the prefetch circuitry system, which can be independent of the software code because the same software code can be executed on different microarchitecture implementations. By eliminating the need for software control over the advance distance (unlike the method where programmers would need to consider the point of insertion of software prefetch instructions based on an evaluated and useful expected advance distance for prefetch instructions with a single address specification for prefetch instructions), this makes it easier for software developers to produce platform-independent code that can operate efficiently across various microarchitectures. For example, the advance distance of the prefetch circuitry system can be adjusted based on microarchitecture information associated with the result of a prefetch request for a demand access request. This microarchitecture information can be any of the types of microarchitecture information mentioned earlier.

[0036] Different microarchitectural implementations of the prefetch circuitry system can respond to range prefetch instructions in different ways. A simple approach is to directly issue a prefetch request corresponding to an address within the specified range (or at least an address within the first of the specified range) as a response to the range prefetch instruction. However, in practice, this risks wasting memory bandwidth by prefetching later portions of the specified range, as previously prefetched data may have already been evicted from cache memory and thus no longer benefit from prefetching by the time the demand access stream reaches a later portion of the specified range. Another approach is to not directly issue a prefetch request as a response to the range prefetch instruction, but the range specification parameters and step size can be used to store information related to the boundaries of the specified address range, which can then be used to quickly understand the newly detected address stream and trigger further prefetching of subsequent addresses within that range after a demand access request for an address within the specified range is detected. However, while this method still provides better performance than a pure hardware prefetching method that does not use any software prefetching instructions, waiting for demand access to reach a given specified address range may risk that the prefetch coverage is less effective for the initial portion of the given specified address range, because when the first demand access reaches that range, it may be too late to prefetch the demand access data from that initial portion in a timely manner to utilize the prefetched data.

[0037] Therefore, a useful method is to trigger the prefetching of data from an initial portion of a plurality of specified ranges to at least one cache memory in response to a range prefetch instruction decoded by an instruction decoder. In response to the range prefetch instruction, this prefetching of the initial portion can be performed directly, regardless of whether any required access associated with addresses within the specified range has been detected. By prefetching data from the initial portion as a direct response to the range prefetch instruction, this improves prefetch coverage and subsequent cache memory hit rate for required accesses, and reduces the warm-up losses associated with conventional hardware prefetching.

[0038] The size of the initial portion prefetched in response to a range prefetch instruction can be defined in hardware by the prefetch circuitry, independent of any explicit specification of the initial portion size by the range prefetch instruction. Therefore, the software does not need to directly specify how much of the specified range should be prefetched as the initial portion in response to the range prefetch instruction. The size of the initial portion can be fixed for a specific microarchitecture by the hardware designer, or it can be dynamically adjusted (at runtime) based on microarchitecture information as a result of a prefetch request or demand access request, as described earlier. This allows different microarchitectures executing the same code to make their own decisions about how much initial data to prefetch in response to a range prefetch instruction. The size of the initial portion can depend on parameters such as the number of cache levels provided, the size of the cache capacity of different cache levels, the latency of memory access, the frequency or volume of other accesses to the memory system, etc.

[0039] The initial portion of the specified range has been prefetched. In response to the range prefetch instruction, the prefetching of data from the remaining portion of the first specified range at the given address may not occur directly. Instead, the prefetch circuitry may trigger the prefetching of data from the remaining portion of the given address in response to the detection of a demand access request from the processing circuitry specifying a target address leading by a given address at a given distance or less. By deferring the prefetching of data from the given address until the demand access reaches an address within a given distance of that given address, it is highly likely that the prefetched data at the given address will still reside in the given cache memory level when subsequent demand access requests for that address are issued, increasing cache hit rate and improving performance. Furthermore, the size of the given distance may not be software-defined and may be determined in hardware by the prefetch circuitry independently of any explicit specification of the given distance by the range prefetch instruction. Furthermore, the given distance can be a fixed distance set for a specific microarchitecture implementation (which may differ from the given distance set on other microarchitecture implementations supporting the same code), or it can be adjusted at runtime based on microarchitecture information, as described earlier. For one microarchitecture, a larger given distance than another may be useful to account for different latency when bringing data into a given cache level.

[0040] The initial part and the size of the given distance mentioned in the preceding paragraph are examples of the parameters mentioned earlier regarding the advance distance for managing the pre-fetched address stream to request how long the address stream should be published.

[0041] When a range prefetch instruction specifies multiple ranges using a stride parameter, the prefetching of multiple second, third, and further ranges across the specified ranges can be controlled in different ways. For example, the prefetch circuitry can monitor demand access to determine when such streams arrive. For example, prefetching from a given specified range can begin when the demand access has reached a given distance from the end of a previously specified range.

[0042] Furthermore, some implementations of the prefetch circuit system may not always employ the same method for controlling the prefetch timing. For example, a preferred prefetch timing may differ for ranges below a certain threshold compared to ranges above that threshold. It should be understood that the example of the specific prefetch timing control described herein is only one example, and other implementations may choose different methods.

[0043] The prefetching circuitry can be configured such that, after the instruction decoder decodes the range prefetching instruction, when the prefetching circuitry identifies, based on the second address range specification parameter, that the end of the given range has been reached by the prefetched address stream prefetched by the prefetching circuitry or the demand address stream specified by the demand access request issued by the processing circuitry, the prefetching of the given specified range can be stopped. This would be impractical in typical hardware or software prefetching schemes that use a single address specification prefetching instruction. Therefore, by preventing over-prefetching beyond the end of the specified range, this avoids contaminating the cache memory with unnecessary data that is unlikely to be used, and helps reserve cache memory capacity for other data that can be more useful, thus improving performance.

[0044] In some instances, the range prefetch instruction may also specify one or more prefetch hint parameters for providing a hint related to data associated with the specified range, and the prefetch circuitry may control the prefetching of data from the specified address range to at least one cache memory based on the one or more prefetch hint parameters. For example, the prefetch hint parameter may be any information other than information at the address of the specified range, providing clues that can be used to control the evaluated use of the data within the specified range during prefetching.

[0045] For example, one or more prefetch prompt parameters may include any one or more of the following: • A type parameter indicating whether the prefetched data is data expected to be loaded, data expected to be stored, or an instruction expected to be fetched. If the data is expected to be loaded or stored, this may trigger the prefetching of the data into the data cache memory, and if the data is expected to be prefetched, it may be prefetched into the instruction cache memory. Therefore, the target cache memory (which will be the recipient of the prefetched data) may depend on the type parameter selection. Furthermore, the indication of whether the data is expected to be loaded or stored may be useful for selecting the synchronization state of requesting prefetched data. For example, in a system with multiple caches that cache data from a shared memory system, a synchronization scheme may be used to manage synchronization between different cached copies at a given address. When a store operation is expected to cause software using cached data to update the data value associated with the corresponding address, it is preferable to obtain a cached copy of the data in a unique homogeneity state. This invalidates other copies of the data from the same address, preventing those other caches from retaining expired data after the store operation. On the other hand, for data fetched into cache memory that is only subject to a load operation, since the load is not expected to overwrite the data, it is possible to bring the data into cache memory in a shared homogeneity state where the corresponding version of the data from the same address can still reside in other caches. Therefore, it is useful for a prefetch circuit system to control the homogeneity state of data prefetched from a specified address range based on a type parameter that provides a prefetch hint parameter for a range prefetch instruction. • Data usage timing parameters indicate at least one of the following: an indication of how quickly the processing circuitry requests prefetched data associated with a specified range after a range prefetch instruction, and a target level of the cache memory hierarchy to which the data is prefetched. Data usage timing parameters can be used to influence the lead time of a prefetch request relative to the assessed demand access release and / or to which cache memory hierarchy the data is prefetched. Data usage timing parameters can be represented in different ways. In some cases, it may simply be an ambiguous relative indication of whether data may be needed "soon" or "not soon" (or may indicate two or more qualitative levels of expected usage timing), without quantifying any particular usage timing. In some cases, how the prefetching circuitry responds to a usage timing parameter value may be microarchitecture-dependent, and therefore, while software may use data usage timing parameters to express ambiguous intentions regarding relative timing, different microarchitectural implementations may respond to such ambiguous indications in different ways when access to a specified address may be required. Other implementation schemes can more explicitly specify the timing parameters used for data by, for example, specifying that data in a cache memory level will be prefetched to a specific level within it.

[0046] A data reuse timing parameter indicating at least one of the following: an indication of the evaluated interval between a first required access to prefetched data and a further required access to prefetched data, or, if a cache memory hierarchy is replaced from a given level, the cache memory hierarchy will evict the prefetched data to a preferred selected level. The data reuse timing parameter can be used to control cache replacement of items in the cache memory. While the data use timing parameters mentioned above are hints relating to the timing of the first required access to prefetched data after prefetching, the data reuse timing parameters are hints relating to the timing of subsequent accesses after that first required access. When data is prefetched, data reuse timing parameters can be used to set a subsequent data associated with the prefetched data in the cache. This subsequent data can be used to determine whether to evict the prefetched data when another address is to be allocated to the cache (e.g., evictping a cache entry that is less likely to be reused soon may be better than evicting a cache entry that is more likely to be reused soon), or, if evicting the prefetched data, to determine which subsequent level of the cache should be allocated to the evicted data. Furthermore, data reuse timing parameters can be a relatively precise indication of the expected interval between a first required access and a subsequent required access (e.g., an approximate indication of the number of instructions or memory accesses), or an indication of a specific level of the cache to which the data is evicted, or an ambiguous relative indication of whether the data may be needed "soon" or "not soon" after the first required access to the data (or an indication of two or more qualitative levels of expected data reuse timing). Furthermore, the specific microarchitectural implementation of the prefetching circuitry or the specific way in which cache memory uses data reuse timing parameters can vary depending on the implementation. • Access pattern hints, which indicate the relative density / sparseness of access patterns across a specified range of addresses, or indicate a sparse access pattern type. This type of prefetch hint can be useful to allow the prefetching circuitry to determine whether all addresses across a specified range should be prefetched or only some addresses should be prefetched. For example, access pattern hints can be used by the prefetching circuitry to determine whether it is worthwhile to expend resources training the sparse access pattern detection circuitry described earlier to learn a specific sparse access pattern across a given specified range. In some cases, access pattern hints can simply indicate some relative indication of density or sparsity, for example, a single bit indicating whether access is dense (i.e., all or almost all addresses are expected to be accessed) or sparse (fewer addresses are expected to be accessed from a specified range). Other implementations can provide a finer granularity of the relative density / sparseness specification. Some instances may specify a particular sparse access pattern type. For example, the sparse memory access pattern type can indicate that the access pattern may be step-by-step, spatial, temporal, random, etc. This hint about the expected access type can be useful in simplifying the training of a sparse pattern detector.• Reuse probability parameter, which indicates the likelihood that data prefetched from a specified address range is expected to be used once or multiple times. This reuse probability parameter can be used to control how long data prefetched from a specified address range remains in cache memory after prefetching or access by demand access. For example, the reuse probability parameter can indicate whether the expected access stream from the specified address range is a non-time (stream) type access, in which each access to a given address within such range will use the data once and then the address may no longer be needed, or whether the access is a time-mode access, in which data prefetched from a given address within the specified range may be accessed more than once or more than a certain number of times. The reuse probability parameter can therefore be used by the prefetch circuitry to control the storage of some consequent data associated with the prefetched data in cache memory, which can then be used to determine whether to prioritize the eviction of the prefetched data after it has been accessed by demand access. • No prefetch indication parameter, which indicates that data should be suppressed from prefetching from a specified range. In some instances, the prefetch range instruction can also support encoding a non-prefetch cue parameter that instructs the prefetch circuitry not to prefetch any data from the specified range. This can be useful because it avoids the prefetch circuitry expending training resources learning information related to access patterns within the specified range. This can also be useful for other address ranges where reserving prefetch training resources for prefetching may be more beneficial.

[0047] Any given implementation of a range prefetch instruction does not need to support all such types of prefetch prompt parameters. One or more of these prefetch prompt parameters may be supported (or none may be supported at all). The encoding of the prefetch prompt parameters can vary significantly depending on the implementation. In some cases, the encoding of two or more of these prompt parameters may be combined into a single field, where different prefetch prompt code values ​​specified in that field are assigned to different combination patterns of the prefetched prompts. Therefore, each prefetched prompt parameter does not need to be identified using completely separate parameters of the instruction, although this is also possible.

[0048] In some instances, the prefetch circuitry may depend on one or more prefetch hint parameters set with cache memory replacement hint information or cache memory replacement policy information associated with data prefetched from a specified range specified by a range prefetch instruction. The cache memory can then control the replacement of cache memory items based on the cache memory replacement hint information or cache memory replacement policy information set by the prefetch circuitry. Here, the cache memory replacement policy information may be information directly used by the cache memory to control the replacement of cache memory items, for example, used to control which item in a candidate group should be replaced with newly allocated data. For example, the cache memory replacement policy information may be Least Recently Used (LRU) information used to determine which of a group of candidate cache memory items has been least recently accessed, or it may specify a priority scheme indicating the relative priority level for replacement of a given cache memory item. Generally, any known cache replacement strategy can be used. However, by using range prefetch instructions to specify prefetch hint parameters, these hints can be used to set the cache replacement strategy information associated with the item when data is prefetched from a specified range to a given cache memory item. Alternatively, the use of prefetch hint parameters may not directly affect the setting of replacement strategy information associated with a specific cache memory item, but may only be used to set cache replacement hint information that can indirectly control cache replacement, such as hints that prefer or dislike eviction of items associated with addresses within a given address range. Cache replacement hint information may also be information indicating that a given cache memory item can be evicted (or marked as priority eviction) after the first required access hits the item. It should be understood that cache replacement can be implemented in many ways using prefetch hint parameters, but generally, prefetch circuitry systems can use a range of prefetch hint parameters within the prefetch instruction range to influence cache replacement, making it more likely to maintain a higher cache hit rate based on the expected usage patterns indicated by the software using the prefetch hint parameters. (Example data processing equipment)

[0049] Figure 1 schematically illustrates an example of a data processing device 2. The device includes an instruction decoder 4 to decode program instructions from or fetched from an instruction cache. Based on the decoded instructions, the instruction decoder 4 controls a processing circuit system 6 to perform data processing operations corresponding to the instructions. The processing circuit system 6 has access to a register 8 for providing operands for instructions and for storing the results of the processed instructions. The processing circuit system 6 includes several execution units 10, 12 for executing different types of program instructions. As an example, Figure 1 shows a processing circuit system 6 including an arithmetic / logic unit (ALU) 10 for performing arithmetic or logical operations, and a load / store unit 12 for performing load operations to load data from the memory system into the register 8 or store operations to store data from the register 8 into the memory system. It should be understood that these are only some examples of possible execution units, and it is possible to provide other types of execution units or multiple execution units of the same type.

[0050] The memory system in this example includes an instruction cache memory (not shown in Figure 1 for simplicity), several data cache memories 14, 16, and 18, and memory 20, implemented in a cache memory hierarchy. In this example, the cache memory hierarchy includes three cache memory levels: Level 1 data cache memory 14, Level 2 cache memory 16, and Level 3 cache memory 18. Some of these cache memories can also be used for instructions (e.g., Level 2 or Level 3 cache memories can be shared for both data and instructions). Other examples may have different numbers of cache memories in the hierarchy or different data cache organization relative to the instruction cache.

[0051] When a load / store operation is executed in response to a load / store instruction decoded by the instruction decoder 4, the virtual address specified by the operand of the load / store instruction is translated into a physical address by the memory management unit (MMU) 22 based on the address mapping information specified in the page table entry of the page table structure stored in the memory system. The information from the page table structure can be cached in the MMU 22 in the translation lookup buffer (TLB).

[0052] The address for a load / store operation is looked up in the cache memory, and if a hit is found in a given level of cache memory 14, 16, 18, the data becomes available faster than if it had to be retrieved from memory 20. A hit in a higher-level cache memory in the cache memory hierarchy (e.g., in level 1 data cache memory 14) allows the data to be available faster than a hit in a lower-level cache memory closer to memory 20, but higher-level cache memories (such as level 1) have smaller capacity, and therefore there is a trade-off between cache memory access latency and capacity at each level.

[0053] Data access requests issued by the load / store unit 12 are demand access requests issued on demand based on addresses specified by load / store instructions of the executing software program. If data is not loaded into cache memories 14, 16, 18 before the program reaches the execution point of executing the corresponding load / store instruction, there may be additional latency due to cache misses because of the delay in loading data from lower cache memory levels or from memory. Furthermore, if the virtual address to be accessed is not hit in TLB 24, MMU 22 may need to perform page table traversal to traverse the page table structure stored in memory to locate the address mapping information used to translate the virtual address. Such page table traversal can cause significant latency when processing demand access.

[0054] A prefetch circuitry 30 is provided to load data into a given level of cache memory 14, 16, 18 before the processing circuitry 6 issues a corresponding demand access request via a prefetched address. This means that the latency associated with a TLB miss or cache miss can be encountered during prefetching rather than during demand access, and therefore, when prefetching is successful, this reduces the critical timing path for processing demand access that would otherwise miss in the cache.

[0055] The prefetch circuitry 30 can use various information sources to maintain a set of prefetch state information 32, which can be used to predict which addresses may be needed for future access, and based on the prefetch state information 32, can issue a prefetch request 34 to bring data from certain prefetch addresses into a certain level of cache memory 14, 16, 18. For example, the information used to predict prefetch addresses may include a stream of the required address 36 specified by the required access request issued by the load / store unit 12, microarchitectural information 38 indicating information related to the result of the required access request or prefetch request 34 issued by the load / store unit 12, and software-defined hints 40 obtained from the prefetch instructions decoded by the instruction decoder 4. The required address 36 may be a virtual address or a physical address – training the prefetch circuitry 30 on either is possible. Software prefetching versus hardware prefetching

[0056] Software prefetching is used to pre-load cache lines into cache memory before future accesses. This eliminates cache memory filling and TLB filling potential from critical paths during actual accesses, thereby improving performance. Traditional software prefetching loads only one cache line at a time (because the software prefetch instruction only specifies a single address). An alternative relies on a hardware prefetcher to learn access patterns and issue prefetches before required accesses, achieving the same benefits. However, both traditional software prefetching and hardware prefetching have several drawbacks.

[0057] Disadvantages of traditional software prefetching: 1. Programmers find it difficult to incorporate traditional software prefetching instructions that specify a single address. The access pattern is not obvious in the software. Moreover, making it timely and correct becomes a trial-and-error process. It is very difficult to make it work perfectly. 2. The core of traditional software prefetching becomes microarchitecture and system architecture specific. Even the size of the cache line (the basic unit of memory prefetching) varies between implementations. When running on different systems, it will inevitably break. 3. If the system supports a scalable vector architecture, where the instruction set architecture allows microarchitecture designers to choose different sizes for vector registers, and the instruction set architecture is designed to allow the same code to run on different microarchitectures with different vector sizes, the development of code using software prefetching instructions becomes even more difficult due to the unknown vector length. Therefore, even with conservative assumptions related to the cache line length, it is impossible to incorporate prefetching in a generally accurate manner. 4. If it turns out that software prefetching actually harms rather than improves performance (e.g., prefetching offers no performance benefit due to incorrect timeliness, and prefetching causes other data to be evicted, which in turn leads to additional cache misses), there is no way to fix this at runtime, and the code will need to be rewritten. 5. Traditional software prefetching has a single-line goal. The trade-off between code bloat, complexity, and accuracy is tricky. For example, software may eventually over-prefetch to avoid copying loops that omit prefetching.

[0058] Disadvantages of traditional hardware prefetching: 1. Hardware prefetchers use several demand accesses to train and understand the stream. If the stream is small, the hardware prefetcher does not have enough samples to identify the stream and prefetch it in a timely manner. This leads to a loss in coverage. 2. If there are too many parallel streams with interleaved accesses, the hardware prefetcher will find it even more difficult to identify and understand the streams. The training unit of the hardware prefetcher can occupy tens of KB of hardware budget and consume a lot of power and energy. 3. Hardware prefetchers do not know when a particular access stream ends. Therefore, they tend to prefetch beyond the end of the stream. Such inaccurate prefetching can monopolize available bandwidth and cause cache memory pollution. Range prefetching instructions

[0059] Figure 2 illustrates a range prefetch instruction that can help address these shortcomings. The instruction set architecture supported by the instruction decoder 4 and the processing circuitry system 6 may include a range prefetch instruction that specifies a first address range specification parameter 50, a second address range specification parameter 52, and a stride parameter 56. In this example, the instruction also specifies an additional prefetch hint 54 and a counting parameter 58.

[0060] In this example, the first address range specification parameter 50 specifies the start address (also known as the base address) of the range that begins with the specified address range. The second address range specification parameter 52 specifies the size of the specified range and thus implicitly identifies the end address of the range relative to the start address. The end address can be determined by adding the value corresponding to the specified one to the start address. An alternative to specifying the first and second address range specification parameters is to define the start address using a size or offset parameter defined relative to the end address. It is also possible to specify both the start and end addresses as absolute addresses or as separate reference addresses other than the range boundary addresses, instead of specifying one of the start / end addresses relative to the other. The size of the address range can be encoded using parameters of a size that is not a power of two bytes, whichever is chosen by the support system, so the size is not limited to a block size that is a power of two.

[0061] For some codes of range prefetch instructions, the instruction may specify a single address range. For example, this may be when the stride parameter 56 is set to a preset value (such as 0), or the count parameter 58 is set to a preset value (such as 0 or 1), or when other parts of the instruction code indicate that there is only one range of values ​​to be prefetched.

[0062] For other encodings, the range prefetch instruction represents multiple ranges separated by an interval specified by the stride parameter 56, and the count parameter 58 indicates how many ranges will be represented. This will be discussed in more detail below with reference to Figures 6 and 7.

[0063] In the example of Figure 2, both the first and second address range specification parameters are encoded as data stored in a register specified by a range prefetch instruction, wherein the range prefetch instruction includes register specifiers Xn and Xm for identifying the registers storing the first and second address range specification parameters. However, other methods can use immediate values ​​to specify the address range specification parameters; for example, the range size can be defined as an immediate value. The step value 56 and the count value 58 can also be specified in a further register or specified as immediate values, but in this example, they are specified in the same register as the range size 52, as described in Figure 6 discussed below.

[0064] Optionally, the range prefetch instruction may also specify one or more additional prefetch hints useful for controlling prefetching. For example, the range prefetch instruction shown in FIG2 includes a prefetch operation field 54 that can be specified in a register referenced by the instruction or specified as an immediate value, and may have encodings that identify one or more types of prefetch hint information, such as: • Load / store / instruction type information that identifies addresses in the range(s) defined by parameters 50, 52 that are expected to be used by future load operations, store operations, or instruction fetches – this may be useful for controlling prefetching to data cache or instruction cache or for controlling the synchronization state set for prefetched data; • Reuse probability parameters, such as an indication of whether the addresses(s) in the specified range are likely to be used for a streaming access mode of loading, where data from each address may be used only once (or relatively several times), or an access time mode, where data from the given addresses in the specified range may be used more than once (or a relatively large number of times); • Data usage timing parameters, such as indications of the target level of the cache hierarchy to which data will be prefetched, or relative indications of how quickly data associated with a specified range will be requested by the processing circuitry after a range prefetch instruction, which may be useful for controlling the timing of prefetch operations and / or for controlling which level of the cache hierarchy to which prefetched data is prefetched; • Data reuse timing parameters, such as indications of the selected level (e.g., L2 or L3) of the cache to which prefetched data should be evicted when replaced in a given level (e.g., L1) of the cache, or relative indications of how quickly a further demand for access to the same address is expected after a first demand for access to the address of prefetched data; • Access pattern hints, which indicate the relative density / sparseness of access patterns of addresses within a specified range, or indicate a particular sparse access pattern type, such as accesses that may be step-by-step, spatial, temporal, or multi-step; The prompt information is not prefetched, indicating that subsequent code is unlikely to use data from addresses within the specified range defined by the first address range specification parameter 50 and the second address range specification parameter 52.

[0065] Any given implementation of a range prefetch instruction may choose to implement any one or more of these types of prefetches, and it is not necessary to encode all of them. Prefetch hint information may be useful to allow the prefetch circuit system 30 to modify how it controls the issuance of prefetch requests for addresses(multiple) within a specified range.

[0066] Therefore, it is proposed to transmit range prefetch instructions for one or more memory address ranges to be accessed in the future. If the range prefetch instructions are executed on a microarchitecture having prefetch circuitry 30, the range prefetch instructions can program the hardware backend of prefetch circuitry 30, which is responsible for timely prefetching by using microarchitectural clues (such as current activity mode) and tracking progress in the installed range. The range prefetch instructions will allow the hardware backend to bypass the training phase and begin directly issuing accurate prefetches.

[0067] Range prefetch instructions can handle dense or sparse access to coded address ranges. Range prefetch instructions also encode stride and counting parameters to handle more complex access patterns. Range prefetch instructions can also rely on hardware to detect complex access patterns within (multiple) coded address ranges, such as multi-stride patterns, spatial patterns, temporal patterns, etc. The hardware backend can also provide hints to the cache replacement strategy to make smarter replacement decisions (e.g., when new data needs to be allocated to a given cache memory 14, 16, 18, the prefetch circuitry 30 can control the settings of cache replacement information associated with cache memory entries to determine which cache memory entry to replace – for example, setting data to be used after the cache memory entry in Least Recently Used (LRU), Rereference Interval Prediction (RRIP), or any other known cache replacement strategy).

[0068] From an architectural point of view, range prefetch instructions do not need to cause any operation to be performed – it is acceptable for certain microarchitecture processor implementations to ignore range prefetch instructions and process them as no-operation (NOP) instructions. For example, processor implementations that do not have any prefetch circuitry 30 or do not have prefetch circuitry that would support the use of hints 40 available from range prefetch instructions may process range prefetch instructions as NOPs.

[0069] However, there are several advantages to such microarchitectural implementations that do not support the use of prompts from range prefetch instructions in the prefetch circuit system 30.

[0070] Overcoming the shortcomings of traditional software prefetching: A range prefetch instruction can be provided for one or more ranges in a group. This instruction can be placed once outside the inner loop. The hardware backend is responsible for issuing the prefetch in a timely manner. Therefore, applications become more microarchitecturally and system-architecturally independent. The associated backend also allows detection of whether range prefetching is actually detrimental to performance. For example, if prefetching is found to harm performance, the prefetching circuit system 30 has the flexibility to selectively shut down the prefetching of certain ranges.

[0071] Overcoming the shortcomings of traditional hardware prefetching: Because the range prefetch instruction provides precise hints on the start address and range of the address stream, the range prefetch instruction and its simple hardware backend can achieve higher accuracy, and only use a small portion of the hardware burden of traditional hardware prefetchers.

[0072] Programmability Benefits: As can be seen in Figure 5 discussed below, the range pre-extraction instruction can be added to existing applications in a very simple and elegant way. Therefore, in addition to performance benefits, ease of programmability is the main benefit of this instruction. As an example, for a use case with a depthwise convolution kernel, engineers spent half a day trying to add traditional software pre-extraction instructions to the depthwise convolution kernel, ultimately achieving only a slight speedup. In contrast, adding the range pre-extraction instruction took about 10 minutes and the performance improvement was much greater than when using traditional software pre-extraction instructions. Example of range-based pre-extraction control

[0073] Figure 3 illustrates an example of how the pre-extraction circuit system 30 can control pre-extraction based on parameters of a range pre-extraction instruction. Figure 3 shows the control of a single range, for example, the first range of multiple ranges specified by the range pre-extraction instruction if multiple ranges are encoded using a stride parameter.

[0074] In response to the instruction decoder 4 decoding a range prefetch instruction, address parameters 50, 52 (and additional prefetch hints 54, if provided) may be provided to the prefetch circuitry 30. For example, the prefetch circuitry may record information about the start and end addresses of the identified range in the prefetch state storage 32. In response to the decoding of the prefetch instruction, the prefetch circuitry may issue a prefetch request 34 corresponding to an address within the initial portion of the range, wherein the initial portion is of a specific size J.

[0075] If the overall size of the range is very small (e.g., smaller than the size J defined for the initial portion), for example, only two or three cache lines, the entire range can be prefetched as a direct response to the range prefetch instruction, and no further additional prefetching is required to respond to the demand access requests related to that range.

[0076] However, for a large range specified by a range prefetch instruction, prefetching the entire range in response to decoding the range prefetch instruction may be inefficient, as this would risk that the prefetching of later addresses within that range would be useless (because when the demand access issued by the load / store unit 12 actually reaches those later addresses, the prefetched data may have been evicted from the cache in response to an access to other addresses that could trigger a replacement in the cache). Therefore, the initial portion prefetched in response to a range prefetch instruction may not cover the entire range between the start and end addresses of the range.

[0077] For addresses within the remaining portion of the specified range, as shown in Figure 3, when the prefetch circuit system detects from the address 36 of the demand access that the demand access has reached or exceeded a certain address (#A – D), it can prefetch the data at the prefetch address #A, where D is a lead distance parameter that controls the prefetch stream to have a margin of lead over the demand access stream.

[0078] The initial dimensions J and distance D are two instances of advance distance parameters that affect the timeliness of prefetching. These parameters can be defined by the prefetching hardware of the circuit system 30 independently of any specific software-defined parameters specified by the range prefetch instruction. J and D can be fixed or can be dynamically adjusted based on microarchitecture information 38, such as information on cache hit / miss rate, demand access frequency, prefetch latency, prefetch success rate, or prefetch usefulness. This is useful for advance distances J and D that are hardware-controlled rather than software-controlled, because different microarchitecture platforms will experience different latency when accessing memory, so responding to clues from the microarchitecture to determine how far the prefetch stream should be released before the demand access stream can be useful.

[0079] It should be understood that Figure 3 is only one way of controlling prefetching based on the range information defined in the range prefetch instruction, and other methods may also be used.

[0080] If a range prefetch instruction specifies multiple ranges, for subsequent ranges of the stride pattern after the first range, the initial portion of size J can be prefetched in various timing sequences: for example, directly in response to a range prefetch instruction similar to the first range, or when the demand access stream has reached a given distance from the end of the previous range. When the demand stream reaches a distance D from such addresses, addresses in the remaining portion of the subsequent range can be prefetched, similar to that shown for the first range in Figure 3. An easily programmable code example is shown.

[0081] Figures 4 and 5 show code examples illustrating the programmability benefits of using range prefetch instructions instead of individual software prefetch instructions. In this example, for ease of explanation, it is assumed that there is no stepping mode, and therefore a single range is encoded only by individual instances of the instructions. However, it will be seen that similar benefits can be applied to stepping ranges.

[0082] Figure 4 shows an example of a program loop modified by a programmer or compiler to include a traditional software prefetch instruction (PF) for prefetching addresses before the expected access timing of the address. Without the prefetch instruction, the loop would look like this: for (i=0; i<N; i++) { process(buffer1[i], buffer2[i*2]); process(buffer1[i], buffer2[i*2+1]);}

[0083] Therefore, each iteration of the loop performs some arbitrary operation on the data from the first and second buffers (the processing operation is marked as "process" to indicate that no specific operation is intentional, and it can be any arbitrary operation). An element from the first buffer will be combined with two different elements from the second buffer. Without adding a prefetch operation, the loop is reasonably written to be compact and simple.

[0084] If this example uses a single address to specify a software prefetch instruction to control prefetching, this makes the code more complex because the prefetch instruction needs to be included at a certain distance before the instruction that will actually use the corresponding address specified by the prefetch instruction. Therefore, the prefetch instruction "PF" embedded in the processing loop itself must specify the address of the corresponding address that will be accessed by the processing instruction itself at a certain distance from ADV. This in turn means that if the loop count value of i is less than ADV, it is impossible to include a software fetch instruction for the initial part of the prefetch loop iteration in the main program loop. Therefore, as shown in FIG4, an additional preliminary loop is added to include a prefetch instruction for starting the cache memory with the prefetched data to be processed in the initial part of the ADV iteration in the main loop. Furthermore, when the main loop now includes prefetch instructions that are prefetched by the main demand access included in the loop at a certain distance, if this loop is simply executed up to the required number N of multi-master data processing loop iterations, the prefetch specification will exceed the end overfetch of the data structure currently operating on it, which can waste cache memory resources. Therefore, in the example of Figure 4, the main loop can be executed through fewer iterations than the required number N (N-ADV), and without any prefetch instructions included in the tail loop, an additional tail loop is added to execute the remaining number of data processing iterations.

[0085] Therefore, since three loops are now being processed instead of just a single loop, using a single address to specify the prefetch instruction introduces many additional loop control burdens. This burden arises because when compiling the high-order code shown in Figure 4 into a hardware-supported instruction set, additional branch and comparison instructions are needed to compare the loop counter i and the threshold to determine whether to continue further iterations of each loop and control the program flow. Since branches can suffer from prediction errors, including these additional loops can harm performance. An alternative could be to eliminate the tail loop and only accept over-prefetching that occurs when the main loop has completed N iterations. However, this wastes memory system bandwidth when performing unnecessary prefetches, and can subsequently lead to additional cache misses, harming performance, as unnecessary prefetched data can evict other data.

[0086] Another problem is that the optimal value of the advance distance (ADV) can vary across different microarchitecture platforms, and even within the same microarchitecture platform, it can vary based on the volume of other memory access modes that can occur during code execution. Therefore, it is very difficult for software developers or compilers to set the ADV appropriately for a range of microarchitectures. This can make the code of this type shown in Figure 4 highly platform-dependent, significantly increasing software development costs.

[0087] In contrast, as shown in Figure 5, by using range prefetch instructions, software developers do not need to consider the relative timing of prefetchers at individual addresses. Instead, software developers simply include a number of range prefetch instructions before the original processing loop. Each range prefetch instruction specifies first and second address range parameters that define the start and end addresses of the address range corresponding to the data structure to be processed. For example, for the code example shown in Figure 5, one range prefetch instruction specifies the start address and size of the first buffer, and the second range prefetch instruction specifies the start address and size of the second buffer. This allows the prefetcher to provide the useful information prefetched in the initial portions of the two address ranges corresponding to the first and second buffers to prefetcher 30, and then the timeliness of subsequent prefetching of data within the address range can be hardware controlled without explicit software programming. This avoids the need for the start and end loops shown in Figure 4, and also significantly reduces the number of prefetch instructions to be inserted, saving instruction fetching and decoding bandwidth. Stepped Range Encoding

[0088] Figure 6 shows a more detailed example of the instruction encoding for a range prefetch instruction. The instruction encoding includes: • a first register identifier field Rn, which specifies a register providing the range start address 50; • a second register identifier field Rm, which specifies a register providing the range size parameter 52 (indicated as "length" in Figure 6), the stride parameter 56, and the count parameter 58; and • a prefetch operation field Rp, which provides an additional prefetch hint 54 as described earlier. The prefetch operation field Rp can be interpreted as a register identifier field identifying the register providing the prefetch hint, or as an immediate value identifying the prefetch hint.

[0089] The remaining bits of the instruction code other than the fields Rn, Rm, and Rp provide operand bits that identify whether this instruction is a range prefetch instruction. These bits can be used to communicate different variations of the range prefetch instruction or can be reused for other purposes.

[0090] The encoding of the register data stored in the register identified in the Rn field of the instruction is shown in the lower part of Figure 6. The range size (length) parameter 52, the stride parameter 56, and the count parameter 58 are all encoded in the same register. In this example, the length and stride parameters are each specified using 22 bits, and the count is specified using 16 bits, meaning that all three values ​​can be fitted into a single 64-bit register that can be identified using the Rm field. Of course, other sizes of length, stride, and count parameters are also possible. Furthermore, in this example, the length and stride values ​​can be positive or negative with a sign. It is also possible to define the length and stride as always positive without a sign. The range size and stride values ​​are encoded into any number of bytes specified by the software, not limited to bytes that are powers of 2.

[0091] Figure 7 schematically illustrates the use of a first address range specification parameter 50, a second address range specification parameter 52, a step parameter 56, and a counting parameter 58 for controlling the pre-fetching of data from multiple address ranges separated by specified step intervals. The range start address 50 (indicated as B to indicate the base address) identifies the address at the beginning of the first of the range group represented by the range pre-fetch instruction. The range size parameter 52 (or length L) indicates the size of each range in the range group. Therefore, the end address of the first range in the group is at address B+L. The step parameter S indicates the offset between the start addresses of consecutive ranges in the group, and thus the second range in the group extends from address B+S to B+S+L, the third range in the group extends from address B+2S to B+2S+L, and so on. The count value C indicates the total number of ranges controlled by the range prefetch instruction, such that the final range in the group can be extended from address B+(C-1)S to B+(C-1)S+L.

[0092] Alternatively, if it is implicitly assumed that the range prefetching circuit system always indicates at least one range to be prefetched (making a count value of 0 useless), then a count C can be defined such that the total number of ranges it indicates is reduced by 1, and thus in this case, the end of the stream can be at address B+CS+L, such that the total number of ranges is then C+1, taking into account the fact that the first range does not require an explicit encoder, and thus it is possible to encode the maximum number of larger ranges in a certain number of bits.

[0093] However, if the count field 58 encoding includes the total number of ranges C in the first range of the group, it can be more intuitive for software developers to use (C=0 encoding can be reused to encode other information, such as additional prefetch tips).

[0094] Figure 8 illustrates a use case where this step address range group can be useful for controlling pre-extraction. Machine learning algorithms (such as convolutional neural networks) may involve convolution operations of a relatively small weight core matrix and a large input activation matrix, wherein the output matrix produced in the convolution operation depends on various combinations of multiplying the core weights by the input activations for different combinations of core / activation locations, wherein logically the core effectively sweeps across different positions of the input matrix, and when the core is at a given position relative to the matrix, the multiplication performed in the convolution operation represents the product of the core weights and activations at the corresponding location.

[0095] Such convolutions may require numerous load / store operations on memory to load various sets of excitations for different core locations and store the results back into the processing. The data loaded for a given core location may correspond to several relatively small portions of columns or rows (e.g., portions represented by the shaded areas of the matrix in the top part of Figure 8), such that they may follow an access pattern (such as that shown in Figure 7), in which there are several ranges of interest separated by a constant stride interval. It should be understood that this is a simplification of the convolution operation for ease of interpretation, and in practice, the layout of the excitation data in memory may not match the logical configuration of the matrix shown in Figure 8 (e.g., where there are multiple excitation layers, and the elements associated with a given row or column of a given layer may not actually be contiguous in memory). However, access to several ranges that are not contiguous in memory but are of the same size and have a regular offset between the beginning of one range and the beginning of the next range is common in such convolution operations.

[0096] Therefore, by providing range pre-extraction instructions that support specified stride parameters, as shown in Figures 6 and 7, this allows a single instruction to initiate the pre-extraction circuitry system 30 to control the accurate pre-extraction of several address ranges discontinuously located in the address space with a much less training burden than using purely hardware pre-extraction techniques, while being much easier to program than the option of using software pre-extraction instructions that specify individual addresses. Example microarchitecture options for the pre-extraction circuitry system 30

[0097] Figures 9 to 11 show various examples of possible microarchitectural implementations of the pre-extracted circuit system 30.

[0098] Figure 9 shows a first example of a pre-extraction circuit system including a hardware pre-extractor 80, which can pre-extract data of the address predicted for the address 36 to be accessed based on any known scheme, using pre-extraction training. For example, this can be based on stride detection (with a single stride or multiple different strides) within the demand address stream.

[0099] Compared to general hardware prefetching, the hardware prefetcher 80 is modified to supplement its training with additional prefetch hints from range prefetch instructions, without requiring a dedicated prefetch state machine separate from the existing state machine implemented for training the predictive state 32 based on the monitoring of the address 36 of the demand access. A general hardware prefetcher may already have the ability to monitor the address 36 of the demand access, detect patterns (such as stride sequences), and gradually adjust the confidence level in the observed sequence so that once the confidence level reaches a certain level, the hardware prefetcher 80 can then begin to issue prefetch addresses to the prefetch address queue 82 based on the patterns detected in the demand address stream 36, so that the control logic can then issue a prefetch request 34 to the memory system to prefetch data from the indicated prefetch addresses in the queue 82.

[0100] When range prefetching instructions are supported, the range boundary and stride start prefetching state 32 from the range prefetching instructions are used to more quickly select a reliable prediction stream for the prefetched address. For example, the start address from the range prefetching instructions (or the sum of the start address 50 and multiple stride parameters 56) can be used to initialize a new item in the prefetch prediction state 32 to immediately recognize that a predictable address stream will follow that start address. However, using standard hardware prefetching techniques, the hardware prefetcher may consume some time to detect from the demand address stream 36 that the demand access has reached the region of the address space and select a reliable prediction.

[0101] Another method is that when a range prefetch instruction is encountered to identify the range to be prefetched for the given address range, the confidence of the corresponding item in the prefetch prediction state 32 can be increased. Therefore, its confidence is greater than the confidence usually set when the same demand address stream 36 has been encountered without the range prefetch instruction being executed.

[0102] Similarly, the end address of the range identified by the range prefetch instruction can be programmed into an item of the prefetch prediction state 32, so that the hardware prefetcher 80 can stop generating prefetch requests once the prefetch stream has exceeded the end address or the demand address stream 36 has been observed to have moved beyond the end address of the specified range.

[0103] Figure 10 shows a second example, in which, in addition to the hardware prefetcher 80 which can operate based on the monitoring of the demand address stream 36 according to conventional hardware prefetching techniques, the prefetching circuit system 30 also includes a range prefetcher 90 that runs a separate state machine to respond to the prefetching prompt 40 received from the range prefetching instruction. Using this method, the range prefetcher 90 can operate in parallel with the hardware prefetcher 80, and both the hardware prefetcher 80 and the range prefetcher 90 can provide prefetch addresses to the prefetch address queue 82 to trigger the corresponding prefetch request 34 to be issued to the memory system.

[0104] The range pre-extractor 90 can maintain a set of prediction states 92 separate from the pre-extracted prediction states 32 maintained by the hardware pre-extractor, but the range pre-extracted prediction states 92 can be simpler. For example, the range pre-extracted prediction states 92 can simply record the parameters 50, 52, 54, 56, and 58 of the range pre-extracting instruction, but may not need to include other fields included in the pre-extracted prediction states 32, such as a confidence field or a field recording multiple possible candidates for the detected access pattern. Other instances may encode the prediction states 92 in another way – for example, the separate items of the prediction states can be assigned to multiple ranges encoded using the stride parameters of the range pre-extracting instruction.

[0105] Using the method shown in Figure 10, when a range prefetch instruction is decoded by instruction decoder 4, the range prefetcher can control prefetching in the manner shown in Figure 3, wherein, for the first specified range of a set of one or more ranges encoded by the range prefetch instruction, in response to installing a new item in the range prefetch state table 92, the initial portion of size J is initially prefetched, and then subsequent prefetching is controlled in a timing mode that means the prefetching maintains a certain distance D ahead of the required address 36. Microarchitectural information 38 can be used to set the size J of the initial portion and the lead distance D, as discussed earlier.

[0106] In this example, the range pre-extractor 90 can also provide a signal 94 to the hardware pre-extractor 80 to prevent the hardware pre-extractor 80 from performing pre-extraction training within the range in which the range pre-extractor 90 is performing pre-extraction. If the no-pre-extraction cue described in FIG2 is supported, this signal can also be used to suppress pre-extraction by reaching the range indicated by the range pre-extraction instruction, since pre-extraction is not required at all.

[0107] Therefore, by providing a range prefetcher 90 that can perform more accurate predictions using prediction state information that is smaller than that of the hardware prefetcher 80, due to the use of hints available from range prefetch instructions and the suppression of training by the hardware prefetcher 80 within those ranges, resources in the hardware prefetcher 80 that are not yet explicitly indicated as useful for prefetching by software prefetch instructions are reserved for detecting other address patterns. This means that for a given circuit area size of the prefetch circuitry 30, overall performance can be improved because more training resources are available for other addresses outside the range processed by the range prefetcher 90, or alternatively, for a given amount of prefetch coverage, the size of the hardware prefetcher 80 and the prediction state memory 32 can be reduced while maintaining an equivalent performance level. Therefore, this approach provides a better balance between performance and circuit area and power cost.

[0108] Figure 11 shows another possible microarchitecture implementation in which, in this example, a range prefetcher 90 is provided within the prefetch circuitry 30, but there is no dedicated hardware prefetcher 80 for detecting access patterns outside the range indicated by the range prefetch instruction prompt 40. In this case, prefetching may not be performed outside these ranges at all. This approach can be less costly in terms of circuit area. The range prefetcher 90 in this example can operate similarly to that described in Figure 10.

[0109] Optionally, as shown in FIG11, the range prefetcher 90 may include a sparse access pattern detection circuitry 96 to perform further hardware control training on the prediction of sparse access patterns within the range being processed by the range prefetcher 90. This helps to handle situations where, although the software has indicated that address accesses may occur in the overall range(s) within which, not every address will necessarily be accessed, and therefore, if prefetching data from each address in that range results in the replacement of other data in the cache, this can cause performance-impacting over-prefetching. The sparse access pattern detection circuitry 96 may operate in the same manner as the hardware prefetcher 80 shown in a previously illustrated example, wherein it monitors the demand address sequence 36 to maintain a prediction state 32 that expresses information relating to the type and associated confidence level of the detected access pattern, allowing prefetch requests to be generated for predicted addresses based on the detected sparse access pattern predictions. However, compared to the hardware prefetcher 80 shown in Figure 9 or Figure 10, the sparse access pattern detection circuitry 96 and the corresponding predicted state 32 are lower in cost in terms of circuit area and power consumption because the training performed by the sparse access pattern detection circuitry 96 is applied only to the addresses within the range being processed by the range prefetcher 90 rather than the entire address space, and therefore generates less predicted state data and requires less comparison of predicted state data.

[0110] When additional prefetch hints 54 are available from a series of prefetch instructions, this can affect sparse access pattern detection. For example, a dense / sparse indicator can be used to determine whether the sparse access pattern detection circuitry 96 is used for a given range tracked in the range tracking data 92. If the corresponding range prefetch instruction indicates that the pattern will be dense access, the power cost of training the sparse access pattern detection circuitry 96 can be avoided by suppressing the use of the sparse access pattern detection circuitry for that range, and a limited prediction state memory 32 will also be reserved for training on other ranges. On the other hand, when the dense / sparse indicator specifies a sparse access mode, training can then be initiated using sparse access pattern detection 96, and if the sparse access mode type is specified in the prefetch hints 54, this can help the sparse access pattern detection circuitry 96 to more quickly select a confidence prediction or decide which specific form of pattern detection to apply to the given address range.

[0111] It should be understood that the same code can be implemented across any of the microarchitectural implementations of the pre-extractors shown in Figures 9 to 11, and that range information 50, 52 and additional pre-extraction hints 54 can be used by the pre-extractor in many other ways to improve its predictions.

[0112] Although not shown in any of Figures 9 to 11, some instances of the prefetch circuitry may also include address translation circuitry similar to the MMU 22 of the processing circuitry 6 shown in Figure 1. Similar to MMU 22, the address translation circuitry of the prefetch circuitry may include a relatively small TLB 24. By providing some translation logic in the prefetch circuitry 30, this allows addresses to be translated for the purpose of prefetching without consuming bandwidth used for demand address translation, which may be more critical for performance. Which addresses are translated by the address translation circuitry of the prefetch circuitry 30 may depend on whether the prefetching is trained based on virtual addresses or physical addresses. If the training is based on virtual addresses, the translation may be for the prefetch address specified in the prefetch request issued to the memory system. If the training is based on physical addresses, the translation may be based on the range of addresses specified by the operands of the range prefetch instruction.

[0113] Alternatively, in other instances, any address translation required for pre-fetching can be handled by the main MMU 22 associated with the processing circuitry system 6, which also handles the translation of the required access. Example Method

[0114] Figure 12 is a flowchart illustrating the method of performing prefetching. In step 200, instruction decoder 4 decodes a range prefetch instruction specifying the first address range specification parameter 50, the second address range specification parameter 52, and the step parameter 56. In step 202, in response to the decoding of the range prefetch instruction, the prefetch circuitry 30 controls the prefetching of data from one or more specified address ranges to at least one cache memory 14, 16, 18, depending on the first and second address range specification parameters and the step (and, if specified by the range prefetch instruction, also depending on one or more additional prefetch hints). Some instances may also encode counting parameters to control prefetching.

[0115] Figure 13 is a second flowchart illustrating in more detail the method of controlling prefetching based on range prefetching instructions. In step 210, the instruction decoder 4 decodes the range prefetching instruction specifying the first address range specification parameter 50, the second address range specification parameter 52, the step parameter 56, and the count parameter 58 (and optionally the prefetching prompt 54). In step 211, the prefetching circuit system 30 initializes the run count of how many ranges specified by the range prefetching instruction have been processed, and processes the first of the specified ranges as the currently specified range to be processed.

[0116] In response to a range prefetch instruction, in step 212, the prefetch circuitry 30 prefetches data from an initial portion of a first specified address range. The size of the initial portion is defined by the hardware of the prefetch circuitry 30 and may be independent of the information specified in the range prefetch instruction. Optionally, the size of the initial portion may be adjusted based on monitoring of demand access and / or based on microarchitecture information 38 related to the prefetch request or the result of demand access. Although some instances may also use software-provided hints on the size of the initial portion, or use information specifying a link to use a previous range to help prefetch the initial portion of that range to determine the size of the initial portion (however, if the microarchitecture clues indicate that different sizes of initial portions are evaluated to be better for performance, the hardware still has the flexibility to abandon the size of the initial portion indicated by the software, even if software-provided hints are available). Microarchitectural clues (such as cache miss rate, the percentage of prefetch requests considered successful due to subsequent demand accesses hitting the same address, or information related to the latency of processing prefetch requests or demand access requests) can be useful in determining how far in advance the demand access stream should publish a prefetch request for the corresponding address, and therefore, increasing the size of the initial portion of the prefetch in step 212 can be useful if a greater lead time is desired.

[0117] Having pre-fetched the initial portion, the pre-fetch circuitry 30 monitors the address 36 of the requested access. In step 214, it is detected that the requested access has reached the target address located at a given distance or less in the remaining portion of the currently specified range prior to the given address. Again, the given distance is hardware-defined by the pre-fetch circuitry. When the requested access has reached the target address, in step 215, pre-fetching of data up to the given address is performed. Again, the given distance may be adjusted based on the monitoring of the requested access and / or microarchitectural information related to the pre-fetch request or the result of the requested access.

[0118] In step 216, the prefetch circuitry 30 detects whether the requested access address is within a given distance from the end of the current specified range, and the run count maintained by the prefetch circuitry 30 indicates how many ranges are still pending processing, suggesting that at least one specified range remains to be prefetched after the current range. The given distance in step 216 may be the same as or different from the given distance in step 214. If the requested access address is within a given distance from the end of the current range, and at least one range remains to be prefetched, then in step 217, the prefetch circuitry 30 issues a prefetch request for an address in the initial portion of the next specified range following the current specified range in the step range group. If the requested access is not yet within a given distance from the end of the current specified range, or the run count indicates that this is the last range to be prefetched, step 217 is omitted.

[0119] In step 218, the prefetch circuitry 30 also checks whether the requested address 36 or the address for which the prefetch request was issued has reached or exceeded the end of the currently specified range. If the requested address stream or the prefetch address stream has not yet reached the end of the specified range, the method continues looping through steps 214, 216, and 218 until the requested access or prefetch access has exceeded the end of the specified range.

[0120] Once the end of the specified range has been reached, in step 220, the pre-extraction stops for the specified range.

[0121] In step 222, the prefetch circuitry 30 checks the run count (maintained within the prefetch circuitry 30 to track how many discontinuous ranges in the processed step group) to indicate whether there is still at least one range to be processed in the discontinuous range group. If there is still at least one range to be prefetched, then in step 224, the next specified range becomes the currently specified range. Therefore, the start / end address of the next range is offset by the step parameter relative to the start / end address of the range that previously served as the current range. The run count, used to track how many ranges remain to be processed, is updated to indicate how many ranges remain to be prefetched, and then the method returns to step 214 to continue monitoring demand access to determine when to issue a prefetch request for the next range.

[0122] Therefore, when the stride and counting parameters are used to define multiple ranges, each range is then processed in a corresponding manner through steps 214 to 224, and then when in step 222, it is finally determined that there are no more ranges to pre-extract, and then in step 226, pre-extraction stops for the ranges represented by the range pre-extraction instruction.

[0123] Although Figure 13 shows that some steps are performed sequentially in a certain order, it will be understood that other implementations may allow for reordering of steps and for some steps to be performed in parallel. For example, the checks in steps 214, 216, and 218 may be performed in parallel or in a different order.

[0124] Furthermore, it should be understood that the method shown in Figure 13 is only one example, and other implementations may use the pre-extraction instruction parameters provided by the range pre-extraction instruction in different ways.

[0125] In this application, the term "configured to..." is used to mean that an element of a device has a configuration capable of performing the defined operation. In this context, "configuration" means the arrangement or manner of interconnection of hardware or software. For example, the device may have dedicated hardware that provides the defined operation, or a processor or other processing device may be programmed to perform the function. "Configured to" does not mean that the device element needs to be changed in any way to provide the defined operation.

[0126] Although illustrative embodiments of the invention have been described in detail herein with reference to the accompanying drawings, it should be understood that the invention is not limited to these precise embodiments, and that various changes and modifications may be made therein by those skilled in the art without departing from the scope of the invention as defined in the appended claims. [Simplified Explanation of the Diagram]

[0006] [Figure 1] schematically illustrates an example of a data processing system including a prefetch circuit system; [Figure 2] illustrates an example of a range prefetch instruction; [Figure 3] illustrates an example of using first and second address range specification parameters of a range prefetch instruction to control prefetching from a given specified range; [Figure 4] shows an example of an alternative code for using a single address specification prefetch instruction; [Figure 5] illustrates an example of a code for using a range prefetch instruction; [Figure 6] shows a specific example of the encoding of a range prefetch instruction; [Figure 7] illustrates using stride and count parameters to control the prefetching of multiple address ranges at specified stride intervals; [Figure 8] illustrates a convolution operation that can benefit from the use of a range prefetch instruction; [Figures 9],

[10] , and

[11] illustrate different examples of microarchitecture implementations of a prefetch circuit system; [Figure 12] illustrates a flowchart of controlling prefetching based on a range prefetch instruction; and [Figure 13] illustrates a flowchart of controlling prefetching based on parameters specified by a first and a second address range in more detail.

Claims

1. An apparatus for data processing, comprising: an instruction decoder for decoding instructions; a processing circuitry for performing data processing in response to the decoding of the instructions by the instruction decoder; at least one cache memory for caching data accessed by the processing circuitry; and a prefetch circuitry for prefetching data into the at least one cache memory; wherein: In response to the instruction decoder decoding a range prefetch instruction specifying first and second address ranges, specified parameters, and a step parameter, the prefetch circuitry is configured to obtain the first and second address ranges and the step parameter, wherein the starting address and size of each specified range are dependent on the first and second address ranges, and the step parameter specifies an offset between the plurality of consecutive starting addresses within the specified ranges. Based on the first and second address ranges and the step parameter, the prefetch circuitry is configured to determine whether to prefetch specified data from the plurality of addresses within the specified address ranges into the at least one cache memory.

2. The device of claim 1, wherein the first address range specification parameter includes a base address of one of the plurality of specified ranges, and the second address range specification parameter includes a range size parameter specifying a size of each of the plurality of specified ranges.

3. The device as requested in item 1, wherein the range prefetch instruction also specifies a counting parameter indicating a certain range within the plurality of specified ranges.

4. The device of claim 1, wherein the stride parameter is encoded in a register identical to one of the first address range specified parameter and the second address range specified parameter.

5. The device of claim 1, wherein in response to the instruction decoder decoding the range prefetch instruction, the prefetch circuitry is configured to control the prefetching of data from the plurality of specified address ranges to the at least one cache memory depending on: the first and second address range specification parameters and the step parameter; and at least one of: the address associated with a demand access request issued by the processing circuitry to request access to data or monitoring of loaded data; and microarchitectural information associated with the prefetch request or a result of such demand access requests.

6. The device as requested in item 5, wherein the microarchitecture information includes at least one of the following: prefetch line fill latency, demand access frequency, demand access hit / miss information, demand access prefetch hit information, and prefetch usefulness information.

7. The device of claim 1, wherein the prefetch circuitry is configured to control the timing of a prefetch request issued to a plurality of given addresses within a specified range based on the address associated with a demand access request issued by the processing circuitry to request access to data, or via monitoring of loaded data, or via microarchitectural information associated with the prefetch request or a result of such demand access request.

8. The device of claim 1, wherein the prefetching circuitry includes a sparse access pattern detection circuitry to detect a sparse pattern of the plurality of accessed addresses within a given specified range based on monitoring of the addresses of the demand access request, and to select which specific addresses within the given specified range to prefetch based on the detected sparse pattern.

9. The device of claim 1, wherein the prefetch circuitry is configured to control the timing of prefetch requests based on a lead distance defined by the prefetch circuitry, the lead distance indicating how far ahead a prefetch address stream of prefetch requests is relative to a demand address stream of demand access requests issued by the processing circuitry to request access to data.

10. The device of claim 9, wherein the prefetch circuitry is configured to define the advance distance independently of any explicit specification of the advance distance by means of the range prefetch instruction.

11. The device as claimed in claim 9, wherein the prefetch circuitry is configured to adjust the lead distance based on microarchitectural information associated with a result of the prefetch requests or the demand access requests.

12. The device of claim 1, wherein in response to the instruction decoder decoding the range prefetch instruction, the prefetch circuitry is configured to trigger the prefetching of data from a first initial portion of the plurality of specified ranges to the at least one cache memory.

13. The apparatus of claim 12, wherein the prefetch circuitry is configured to define the size of the initial portion independently of any explicit specification of a size of the initial portion by the range prefetch instruction.

14. The device of claim 12, wherein the prefetching circuitry is configured to trigger the prefetching of data from the given address in response to the detection of a demand access request issued by the processing circuitry specifying a target address located at a given address within a remaining portion of the first specified range at a given distance or less.

15. The device of claim 14, wherein the prefetching circuitry is configured to define the given distance independently of any explicit specification of the range prefetching instruction.

16. The device of claim 1, wherein after the decoding of the range prefetch instruction, when the prefetch circuitry recognizes that a given range of a given range has been prefetched by the prefetch circuitry or a demand address stream specified by a demand access request issued by the processing circuitry has arrived, the prefetch circuitry is configured to stop prefetching the given range.

17. The device of claim 1, wherein the range prefetch instruction also specifies one or more prefetch prompt parameters for providing a prompt related to the plurality of data associated with the specified range, and the prefetch circuitry is configured to control the prefetching of data from the plurality of data associated with the specified range to the at least one cache memory depending on the one or more prefetch instruction parameters.

18. The device of claim 17, wherein the one or more prefetch prompt parameters include at least one of the following: a type parameter indicating whether the prefetched data is data expected to be subject to a load operation, data expected to be subject to a store operation, or an instruction expected to be subject to an instruction fetch operation; a data usage timing parameter indicating at least one of the following: an indication of how quickly the prefetched data will be requested by the processing circuitry system after a prefetch instruction in the range, and a target level of a cache memory hierarchy to which the data will be prefetched; a data reuse timing parameter indicating at least one of the following: an indication of an evaluated interval between a first required access to the prefetched data and a further required access to the prefetched data, or a selected level of the cache memory hierarchy to which the prefetched data is preferably evicted if replaced from a given level of the cache memory hierarchy; An access mode hint indicates the relative density / sparseness of an access mode for the plurality of addresses within the specified range, or indicates a sparse access mode type; a reuse probability parameter indicates the probability that the prefetched data is expected to be used once or multiple times; and a no-prefetch hint parameter indicates that data prefetching from the plurality of addresses within the specified range should be suppressed.

19. The device of claim 17, wherein the prefetch circuitry is configured to set cache memory replacement prompt information or cache memory replacement strategy information associated with the plurality of data prefetched from the specified range specified by the range prefetch instruction, depending on the one or more prefetch prompt parameters; and the cache is configured to control the replacement of cache memory items based on the cache memory replacement prompt information or cache memory replacement strategy information set by the prefetch circuitry.

20. A method for data processing, comprising: decoding instructions; performing data processing in response to decoding the instructions by the instruction decoder; caching data in at least one cache memory for access via the processing circuitry system; and prefetching data into the at least one cache memory; wherein: In response to the decoding of a range prefetch instruction specifying first and second address range parameters and a step parameter, the first and second address range parameters and the step parameter are obtained, wherein the starting address and size of each specified range are dependent on the first and second address range parameters, and the step parameter specifies an offset between the plurality of consecutive starting addresses of the specified range, and based on the first and second address range parameters and the step parameter, it is determined whether to prefetch specified data from the plurality of addresses of the specified address range into the at least one cache memory.

Images