Instruction prefetch method, apparatus and artificial intelligence chip

Abstract

The application provides an instruction prefetching method and device and an artificial intelligence chip, relates to the chip design and manufacturing technical field, and comprises the following steps: obtaining a thread bundle identifier of a current thread bundle; determining that the current thread bundle is a memory access thread bundle based on the thread bundle identifier; and adjusting the number of instructions to be prefetched by the current thread bundle. The method and device provided by the application avoid over-prefetching of the memory access thread bundle, save the saved instruction cache space for the calculation thread bundle which needs it more, thereby realizing efficient prefetching of instructions to be executed by the processor, improving the overall utilization efficiency and hit rate of the instruction cache, reducing unnecessary memory bandwidth consumption, and finally improving the overall processing performance of the processor.

Description

Technical Field

[0001] This application relates to the field of chip design and manufacturing technology, and in particular to an instruction prefetching method, apparatus and artificial intelligence chip. Background Technology

[0002] Instruction prefetching is a key technology for processors (especially general-purpose graphics processors) to improve execution efficiency and address memory latency. By predicting and prefetching instructions that will be needed in the future during program execution, these instructions are loaded into the cache in advance, thereby reducing access latency, speeding up program execution, and having a significant impact on processor performance.

[0003] Therefore, how to efficiently prefetch the instructions that the processor is about to execute, improve cache utilization and processor performance has become a technical problem that the industry urgently needs to solve. Summary of the Invention

[0004] This application provides an instruction prefetching method, apparatus, and artificial intelligence chip to solve the technical problem of how to efficiently prefetch instructions that the processor is about to execute, thereby improving cache utilization and processor performance.

[0005] This application provides an instruction prefetching method, including: Get the thread bundle identifier of the current thread bundle; Based on the thread bundle identifier, the current thread bundle is determined to be a memory access thread bundle; Adjust the instruction prefetch count of the current thread bundle.

[0006] In some embodiments, determining that the current thread bundle is a memory access thread bundle based on the thread bundle identifier includes: Compare the thread bundle identifier of the current thread bundle with the preset thread bundle identifier list; In the case of a match, the current thread bundle is determined to be a memory access thread bundle; The preset thread bundle identifier list is used to store the thread bundle identifiers of statically allocated memory access thread bundles.

[0007] In some embodiments, determining that the current thread bundle is a memory access thread bundle based on the thread bundle identifier includes: Receive the thread bundle identifier of the dynamically specified memory access thread bundle sent by the instruction decoder; Compare the thread bundle identifier of the current thread bundle with the thread bundle identifier of the memory access thread bundle; If a match is found, the current thread bundle is determined to be a memory access thread bundle.

[0008] In some embodiments, adjusting the instruction prefetch count of the current thread bundle includes: Reduce the instruction prefetch count of the current thread bundle.

[0009] In some embodiments, adjusting the instruction prefetch count of the current thread bundle includes: Adjust the number of instructions prefetched for the current thread bundle to the first target value; The first target value is determined based on the processor performance parameters of the current thread bundle and / or the resource requirement parameters of the program executed by the current thread bundle.

[0010] In some embodiments, adjusting the instruction prefetch count of the current thread bundle includes: The amount of data read by the memory access thread bundle sent by the instruction decoder; Based on the instruction read data volume and instruction cache line size, a second target value is determined for the instruction prefetch quantity of the memory access thread bundle; Adjust the instruction prefetch count of the current thread bundle to the second target value.

[0011] This application provides an instruction prefetching device, comprising: The acquisition unit is used to obtain the thread bundle identifier of the current thread bundle; The determining unit determines that the current thread bundle is a memory access thread bundle based on the thread bundle identifier; The comparison unit is used to adjust the instruction prefetch count of the current thread bundle.

[0012] This application provides an artificial intelligence chip, including the aforementioned instruction prefetching device.

[0013] This application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the instruction prefetching method when executing the computer program.

[0014] This application provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the instruction prefetching method described above.

[0015] The instruction prefetching method, apparatus, and AI chip provided in this application can obtain thread bundle identifiers and distinguish memory access thread bundles based on these identifiers, thereby enabling targeted adjustment of their instruction prefetching quantity and the use of a more suitable instruction prefetching quantity to meet their instruction requirements. This differentiated processing method avoids over-prefetching of memory access thread bundles, reserving the saved instruction cache space for computation thread bundles that need it more. This achieves efficient prefetching of instructions that the processor is about to execute, improves the overall utilization efficiency and hit rate of the instruction cache, reduces unnecessary memory bandwidth consumption, and ultimately improves the overall processing performance of the processor. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the instruction prefetching method provided in this application.

[0019] Figure 2 This is a flowchart illustrating the static thread bundle identification method provided in this application.

[0020] Figure 3 This is a hardware schematic diagram of the thread bundle dynamic identification method provided in this application.

[0021] Figure 4 This is a flowchart illustrating the thread bundle dynamic identification method provided in this application.

[0022] Figure 5 This is a hardware schematic diagram of the method for dynamically determining the instruction prefetch quantity provided in this application.

[0023] Figure 6 This is a flowchart illustrating the method for dynamically determining the instruction prefetch quantity provided in this application.

[0024] Figure 7 This is a schematic diagram of the instruction prefetching device provided in this application.

[0025] Figure 8 This is a schematic diagram of the structure of the artificial intelligence chip provided in this application.

[0026] Figure 9This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps, units, or modules is not necessarily limited to those explicitly listed, but may include other steps, units, or modules not explicitly listed or inherent to such processes, methods, products, or devices.

[0029] General Purpose Computing on Graphics Processing Units (GPGPUs) commonly employ a Single Instruction Multiple Thread (SIMT) architecture. To provide flexibility in program writing, related technologies have introduced Cooperative Groups, which allow developers to flexibly divide thread groups into functionally distinct subgroups at the software level, thereby achieving complex parallel computing modes that were previously unattainable. However, this flexibility also brings new challenges to the hardware. A typical scenario is that different subgroups within a thread group may execute completely different code segments. This places higher demands on the processor's instruction cache prefetch mechanism.

[0030] If a large number of cache lines are prefetched for all subgroups, it will result in significant cache space waste for subgroups that execute only a few instructions, and will also crowd out the cache resources needed by other subgroups. If the prefetch amount is uniformly reduced, it will decrease the cache hit rate of compute-intensive subgroups, leading to a performance bottleneck.

[0031] A specific subgroup can be designated to handle data read and write tasks. Analysis of the instruction stream of this subgroup reveals that it has the characteristics of a small number of instructions (typically simple code logic with few loops) and high instruction latency (memory access instructions themselves require long waiting times). In other words, it can be determined that the subgroup's bandwidth requirements are actually very low. Based on the above analysis, the technical solution provided in this application optimizes instruction prefetching.

[0032] Figure 1 This is a flowchart illustrating the instruction prefetching method provided in this application, as shown below. Figure 1 As shown, the method includes steps 110, 120 and 130.

[0033] Step 110: Obtain the thread bundle identifier of the current thread bundle.

[0034] Specifically, the execution entity of the instruction prefetching method provided in this application embodiment is an instruction prefetching device. This device can be implemented in software, such as an instruction prefetching program running in the processor; or it can be implemented in hardware, such as a hardware module that executes the instruction prefetching method.

[0035] In one specific implementation, the instruction prefetching device is located in the processor's instruction cache unit (Cache).

[0036] A warp is the basic unit of execution and scheduling in a parallel processor, such as a general-purpose graphics processing unit (GPU). It typically consists of a group (e.g., 32) of threads that execute the same instructions in parallel. To manage and schedule numerous warps, the processor hardware assigns a unique warp ID to each active warp. This warp ID can be used to uniquely identify and track the warp throughout its entire lifecycle.

[0037] When a thread bundle needs to fetch new instructions during execution, for example, when a cache miss occurs, the instruction fetcher generates an instruction fetch request. This request is sent to the instruction cache unit. To indicate which thread bundle needs the instruction, the instruction fetch request must contain the requesting thread bundle identifier. Therefore, in this embodiment, fetching can be understood as the instruction prefetching device parsing or extracting the thread bundle identifier of the thread bundle for which the instruction needs to be prefetched from the received instruction fetch request.

[0038] Step 120: Determine the current thread bundle as a memory access thread bundle based on the thread bundle identifier.

[0039] Specifically, after obtaining the thread bundle identifier, it is necessary to determine the type of the current thread bundle. This application defines a thread bundle that primarily performs high-latency memory access operations, has a relatively simple instruction flow, and low instruction density as a memory access thread bundle. A typical task of a memory access thread bundle is to load data from low-speed, high-latency global memory to high-speed, low-latency shared memory. In contrast, there are computation thread bundles or other types of thread bundles (which can be collectively referred to as non-memory access thread bundles) that perform a large number of arithmetic and logical operations, and have higher instruction density and higher requirements for instruction bandwidth.

[0040] To determine the type of the current thread bundle, you can use the thread bundle identifier itself or related information associated with that identifier to determine whether it belongs to a predefined memory access thread bundle.

[0041] In one specific implementation, the determination process can be as follows: The thread bundle identifier of the current thread bundle is compared with a preset identifier set. If the thread bundle identifier of the current thread bundle exists in the preset identifier set, the current thread bundle is determined to be a memory access thread bundle; otherwise, it is determined to be a non-memory access thread bundle. This preset identifier set can be stored in a specific register of the processor, on-chip memory, or a dedicated hardware lookup table for fast lookup and matching by the instruction prefetching device. For example, in one application scenario, two thread bundles with IDs 0 and 1 can be pre-specified through a programming model or driver to specifically perform memory copy tasks. Then, this preset identifier set would contain {0, 1}. When the instruction prefetching device receives an instruction request with a thread bundle identifier of 0, it can determine that the thread bundle is a memory access thread bundle through lookup and comparison.

[0042] Step 130: Adjust the instruction prefetch count of the current thread bundle.

[0043] Specifically, the instruction prefetch count refers to the amount of instruction data continuously loaded from main memory (or the next-level cache) to the instruction cache unit during a single instruction prefetch operation, typically measured in cache lines. Prefetch mechanisms in related technologies usually employ a fixed, relatively large instruction prefetch count (e.g., prefetching 4 or 8 cache lines at a time) for all thread bundles in the hope of improving the hit rate of subsequent instructions.

[0044] In this embodiment, after determining the current thread bundle as a memory access thread bundle, the number of instructions prefetched for its application is adjusted to distinguish it from the prefetch number of non-memory access thread bundles.

[0045] Adjusting the instruction prefetch count could involve changing it from a default value to a smaller, preset value. Because the instruction sequences of memory access thread bundles are typically short (e.g., containing only a few load, store, and address calculation instructions), and the latency in executing these instructions lies primarily in the memory access itself, rather than instruction fetching, prefetching a large number of instructions not only fails to provide performance gains but also consumes valuable instruction cache space, causing instructions from other computation thread bundles to be prematurely replaced, resulting in more cache misses.

[0046] For example, suppose the system's default instruction prefetch count for a normal computation thread bundle is 4 cache lines. Once it's determined that the current thread bundle is a memory access thread bundle, its instruction prefetch count can be adjusted to 1 cache line. This way, only the most needed instructions and a few adjacent instructions are loaded into the instruction cache unit, satisfying the execution requirements of the memory access thread bundle while maximizing cache space conservation. This smaller adjusted value (such as 1 or 2 cache lines) can be fixed by hardware or configured by software.

[0047] The instruction prefetching method provided in this application obtains thread bundle identifiers and distinguishes memory access thread bundles accordingly. It can then adjust the instruction prefetching quantity in a targeted manner, using a more suitable instruction prefetching quantity to meet the instruction requirements. This differentiated processing method avoids over-prefetching of memory access thread bundles, reserving the saved instruction cache space for computation thread bundles that need it more. This achieves efficient prefetching of instructions that the processor is about to execute, improves the overall utilization efficiency and hit rate of the instruction cache, reduces unnecessary memory bandwidth consumption, and ultimately improves the overall processing performance of the processor.

[0048] It should be noted that each implementation method of this application can be freely combined, rearranged, or executed individually, and does not need to rely on or depend on a fixed execution order.

[0049] In some embodiments, determining that the current thread bundle is a memory access thread bundle based on the thread bundle identifier includes: Compare the current thread bundle's thread bundle identifier with the preset thread bundle identifier list; If a match is found, the current thread bundle is determined to be a memory access thread bundle; The preset thread bundle identifier list is used to store the thread bundle identifiers of statically allocated memory access thread bundles.

[0050] Specifically, the preset thread bundle identifier list is a collection used to store specific thread bundle identifiers. The contents of this list are preset, meaning they can be configured before the kernel in the processor executes the application or at the initial stage of execution. This list can be physically implemented as one or more special-purpose registers, a dedicated on-chip static random access memory, or a hardware lookup table. The choice of storage medium for the list can be based on a trade-off between performance requirements and hardware cost, with the core requirement being that it can be quickly accessed by the instruction prefetching mechanism.

[0051] Static allocation refers to the division of thread bundle tasks by software (e.g., programmers through specific programming interfaces or compilers through code behavior analysis) during the program compilation phase or before runtime. After analyzing the entire computational task, programmers or compilers can pre-plan to dedicate one or more specific thread bundles to performing memory-intensive tasks. For example, when writing a program, a programmer can explicitly specify in the code that the thread bundle with thread bundle ID 0 is specifically responsible for loading data from global memory. Therefore, the pre-defined thread bundle identifier list stores the identifiers of these thread bundles that are statically allocated as memory access thread bundles.

[0052] Figure 2 This is a flowchart illustrating the static thread bundle identification method provided in this application, as shown below. Figure 2 As shown, the comparison operation in this embodiment is performed in the instruction prefetching device. The instruction prefetching device is integrated into the instruction cache unit. The instruction prefetching device includes at least one comparator and at least one selector.

[0053] When the instruction cache unit receives an instruction fetch request from the instruction fetcher, the comparator passes the thread bundle identifier of the current thread bundle carried in the request as input. The comparator then reads all entries in a preset list of thread bundle identifiers and compares the input thread bundle identifier with each identifier in the list. This comparison operation can be performed in parallel to ensure low comparison latency.

[0054] If the comparator finds that the input current thread bundle identifier matches any entry in the preset thread bundle identifier list (i.e., a match has occurred), the comparator will output a valid signal. This signal explicitly indicates that the thread bundle currently requesting the instruction is a memory access thread bundle. If the comparator compares the input thread bundle identifier with all entries in the list and finds no match (i.e., a mismatch), the signal will be invalid, indicating that the current thread bundle is a non-memory access thread bundle.

[0055] Subsequently, the instruction prefetching unit determines the next operation based on this comparison result signal. For example, after determining that the current thread bundle is a memory access thread bundle, it triggers the selector to select the instruction prefetch quantity corresponding to the memory access thread bundle for output; if the current thread bundle is a non-memory access thread bundle, the selector selects the instruction prefetch quantity corresponding to the non-memory access thread bundle (which can be the default value) for output. Generally, the instruction prefetch quantity corresponding to the memory access thread bundle is less than the instruction prefetch quantity corresponding to the non-memory access thread bundle.

[0056] The instruction prefetching method provided in this application utilizes a collaborative approach of software static allocation and hardware fast comparison to achieve a low-overhead, high-efficiency memory access thread bundle identification mechanism. The static allocation mechanism empowers programmers or compilers to directly control and optimize instruction prefetching behavior, making optimization more deterministic and targeted. This allows for precise adjustment of the prefetching strategy for specific memory access thread bundles, achieving fine-grained management of instruction cache resources and improving the overall utilization efficiency and hit rate of the instruction cache.

[0057] In some embodiments, determining that the current thread bundle is a memory access thread bundle based on the thread bundle identifier includes: Receive the thread bundle identifier of the dynamically specified memory access thread bundle sent by the instruction decoder; Compare the thread bundle identifier of the current thread bundle with the thread bundle identifier of the memory access thread bundle; If a match is found, the current thread bundle is determined to be a memory access thread bundle.

[0058] Specifically, unlike the static allocation mechanism in the above embodiments, this embodiment discloses a more automated and intelligent dynamic identification mechanism. This mechanism does not require pre-specification by the programmer or compiler; instead, the processor hardware dynamically identifies memory access thread bundles at runtime.

[0059] The instruction decoder is a key hardware unit in the processor instruction pipeline. Its main responsibility is to parse the instruction code fetched from the instruction cache, identify the operation type (such as arithmetic operation, logical operation, memory load / store, etc.), operands, and addressing mode, and generate corresponding control signals to drive the subsequent execution units.

[0060] In this embodiment, the functionality of the instruction decoder has been extended. When the instruction decoder parses a specific instruction or a class of specific instructions, it can determine that the thread executing the instruction is performing a memory access task.

[0061] Dynamic assignment, in contrast to static allocation in the above embodiments, refers to the identification process occurring during the actual execution of the program. When a thread executes a specific memory access instruction, the hardware automatically "marks" or "assigns" it as a memory access thread. This assignment is immediate, based on the current instruction, and may be temporary.

[0062] Figure 3 This is a hardware schematic diagram of the thread bundle dynamic identification method provided in this application, such as... Figure 3 As shown, an interface or communication link can be added from the instruction decoder to the instruction cache unit to send the thread bundle identifier of the dynamically specified memory access thread bundle. When the instruction decoder decodes a predefined memory access instruction in its pipeline, it not only generates the control signals required to execute the instruction, but also actively sends the thread bundle identifier of the thread bundle currently executing the instruction to the instruction cache unit through this new interface.

[0063] Figure 4 This is a flowchart illustrating the thread bundle dynamic identification method provided in this application, as shown below. Figure 4 As shown, after receiving the thread bundle identifier sent by the instruction decoder, the instruction prefetching unit in the instruction cache unit stores it in an internal list or lookup table. This list, which can be called the "dynamic memory access thread bundle identifier list," records in real time the thread bundles currently identified as performing memory access. The contents of this list change dynamically; when a new memory access thread bundle is identified, its thread bundle identifier is added. Correspondingly, a mechanism can also be designed to remove the thread bundle identifier from the list when the thread bundle completes its memory access task.

[0064] The comparison operation in this embodiment is similar to that in the above embodiments, and will not be repeated here. However, the objects being compared are different. Here, when the instruction cache unit needs to make a prefetch decision for the current thread bundle (i.e., the thread bundle that is requesting instruction prefetching), it compares the thread bundle identifier of the current thread bundle with all the thread bundle identifiers dynamically sent by the instruction decoder stored in the dynamic memory access thread bundle list mentioned above.

[0065] If the comparison matches, the current thread bundle is determined to be a memory access thread bundle. Once determined to be a memory access thread bundle, the instruction prefetching mechanism adjusts the number of instructions prefetched for the current thread bundle. If the comparison does not match, it is considered a non-memory access thread bundle, and the default prefetching strategy is used.

[0066] The instruction prefetching method provided in this application implements a mechanism for automatically identifying memory access thread bundles at runtime. It adaptively captures the dynamic behavior of the program during runtime and adjusts the number of instructions prefetched for dynamically identified memory access thread bundles in real time. This makes instruction cache optimization more intelligent and flexible, capable of handling more diverse application behavior patterns, thereby effectively improving processor performance.

[0067] In some embodiments, adjusting the instruction prefetch count of the current thread bundle includes: Reduce the number of instructions prefetched for the current thread bundle.

[0068] Specifically, in parallel processors such as general-purpose graphics processing units (GPUs), there is typically a default instruction prefetch count. This default value is designed to handle most situations, especially for computationally intensive thread bundles. These thread bundles often have complex instruction flows and high instruction-level parallelism, so prefetching a larger number of instructions at once (e.g., prefetching 2, 4, or more cache lines at a time) can effectively improve the instruction cache hit rate and prevent the processor from stalling due to waiting for instructions.

[0069] Based on the analysis of thread bundle characteristics in this application, the behavior of memory access thread bundles differs significantly from that of non-memory access thread bundles. The instruction sequences of memory access thread bundles are typically very short and have a fixed structure (e.g., short loops containing address calculations, data loading / storage, and loop control), and their main bottleneck in execution speed lies in waiting for high-latency memory operations to complete, rather than the speed of instruction fetching. To address this characteristic, the specific method proposed in this embodiment for adjusting the instruction prefetch count of the current thread bundle is to reduce the instruction prefetch count of the current thread bundle.

[0070] Reduce the instruction prefetch count from the default value to a smaller one. This smaller value should be sufficient to cover the core instructions required by the memory access thread bundle in a single execution iteration, but without consuming excessive cache space.

[0071] The instruction prefetching method provided in this application avoids loading a large number of instructions that will not be executed by the memory access thread bundle into the cache by loading only a small number of necessary instructions. This prevents these useless instructions from replacing the valid instructions needed by other computational thread bundles. Secondly, it improves the overall space utilization and hit rate of the instruction cache unit, allowing valuable cache resources to be allocated more efficiently to the truly computationally intensive thread bundles. Finally, it reduces the memory bandwidth occupied by instruction fetching by avoiding unnecessary data transfer.

[0072] In some embodiments, adjusting the instruction prefetch count of the current thread bundle includes: Adjust the instruction prefetch count of the current thread bundle to the first target value; The first target value is determined based on the processor performance parameters of the current thread bundle and / or the resource requirements parameters of the program executed by the current thread bundle.

[0073] Specifically, this embodiment provides a static configuration method for the number of instruction prefetches, and the configured value can more accurately match the actual needs of the memory access thread bundle.

[0074] The first target value is a specific, quantified number of instructions to prefetch, such as one cache line or two cache lines. It differs from the default prefetch number that may exist in the hardware for non-memory access thread bundles. Once a thread bundle is identified as a memory access thread bundle, the instruction prefetch logic will no longer use the default value, but instead will use the first target value to guide its prefetch operation.

[0075] The primary target value is stored in a location accessible and modifiable by software (such as a dedicated configuration register). This provides the hardware basis for the static configuration of that value.

[0076] The process of determining the first target value is a performance trade-off and optimization process that can be completed during the development and debugging phases.

[0077] Processor performance parameters mainly refer to the inherent attributes of the processor hardware platform, which affect the effectiveness of the instruction prefetching strategy.

[0078] In one specific implementation, processor performance parameters may include parameters such as the size of the instruction cache unit and the size of the cache line.

[0079] In processors with small instruction cache units, cache pollution caused by excessive prefetching is more severe. Therefore, for such processors, a very small first target value (e.g., 1) can be chosen to maximize the protection of valuable cache space.

[0080] In a processor with a large cache line capacity, if the hardware cache line is very large (e.g., 128 bytes), then even if the first target value is 1, it may be possible to load enough instructions at once. Conversely, if the cache line is small (e.g., 32 bytes), it may be necessary to set the first target value to 2 to fully cover the instructions of a short loop.

[0081] The resource requirements parameters of a program refer to the resource requirements of the executed application (kernel) itself, and these characteristics also affect the effectiveness of the instruction prefetching strategy.

[0082] In one specific implementation, the program's resource requirements parameters may include the following parameters: the proportion of memory access thread bundles in all thread bundles and the program's performance data, etc.

[0083] In an application, if the proportion of memory access thread bundles among all thread bundles is small, that is, the vast majority of thread bundles are computationally intensive non-memory access thread bundles and only a very small number are memory access thread bundles, then in order to ensure the performance of the main task, it is very necessary to minimize the prefetching amount of memory access thread bundles, that is, to set a very small first target value.

[0084] Alternatively, the instruction prefetch count can be configured to a range of possible values, from smallest to largest. Then, run the entire program and analyze key performance indicators such as total execution time and instruction cache miss rate. By comparing performance data under different configurations, the primary target value that optimizes overall performance can be determined.

[0085] The instruction prefetching method provided in this application performs fine-grained performance tuning based on specific hardware platform characteristics and application requirements. This hardware-software co-optimization method enables the instruction prefetching strategy to better adapt to diverse application scenarios, achieving targeted deep optimization, thereby maximizing the exploitation of hardware potential and improving the overall performance of the processor.

[0086] In some embodiments, adjusting the instruction prefetch count of the current thread bundle includes: The amount of data read by the memory access thread bundle sent by the instruction decoder; Based on the instruction read data volume and instruction cache line size, a second target value is determined for the instruction prefetch count of the memory access thread bundle; Adjust the instruction prefetch count of the current thread bundle to the second target value.

[0087] Specifically, the instruction copy size refers to the total amount of data that an instruction needs to read, such as the size of the data block copied from global memory (processor storage unit) to shared memory.

[0088] The cache line size is an inherent parameter of the processor hardware that represents the basic unit of data exchange in a single instruction cache operation, such as 64 bytes or 128 bytes. This value is known to the hardware logic.

[0089] Figure 5 This is a hardware schematic diagram of the method for dynamically determining the instruction prefetch quantity provided in this application, as shown below. Figure 5As shown, in this embodiment, the function of the instruction decoder is further enhanced. When it parses a memory operation instruction carrying data volume information, it must not only identify the instruction type but also extract the amount of data to be read. Then, the instruction decoder sends the amount of data to be read and the thread bundle identifier executing the instruction to the instruction cache unit through its hardware interface with the instruction cache unit.

[0090] Figure 6 This is a flowchart illustrating the method for dynamically determining the instruction prefetch quantity provided in this application, as shown below. Figure 6 As shown, in this embodiment, unlike the statically configured first target value in the above embodiments, the second target value is dynamically calculated.

[0091] The instruction prefetching unit in the instruction cache unit receives two inputs: the amount of instruction data to be fetched from the instruction decoder and the hardware's inherent instruction cache line size. The calculation logic is to determine how many instruction cache lines need to be prefetched to fully cover the amount of data involved in the upcoming memory operation task.

[0092] In this embodiment, the method for calculating the number of instructions prefetched is as follows: Step 1: Divide the amount of data read by the instruction decoder by the data channel bandwidth (the number of binary bits that can be transmitted in parallel at one time on the data bus in the processor) to obtain the execution time of the processor for each data read instruction.

[0093] When there are a large number of instructions, the execution time of each data read instruction can be determined by the average execution time of the processor in processing each data read instruction.

[0094] Step 2: Divide the instruction cache line size by the instruction size of each data fetch instruction to obtain the number of data fetch instructions processed by the processor in each instruction cache line.

[0095] Step 3: Multiply the number of data read instructions stored in each instruction cache line processed by the processor by the execution time of each data read instruction processed by the processor to obtain the execution time of all data read instructions processed by the processor for each instruction cache line.

[0096] Step 4: Divide the average instruction fetch time in the system (the average time it takes for the processor to receive the instruction from the time it issues a request to fetch the data) by the execution time of the processor to process all data fetch instructions for each cache line to obtain the instruction prefetch count (the number of cache lines that need to be prefetched).

[0097] The following is a specific example. Assume that the instruction cache line size is 128 bytes, the average instruction fetch time in the system is 256 clock cycles, the data channel bandwidth is 128 bytes, the average instruction size per data fetch instruction is 16 bytes, and the instruction decoder fetches 1 MB of data.

[0098] The execution time of each data read instruction by the processor is obtained by dividing the amount of data read by the instruction decoder (1MB) by the data channel bandwidth (128B). The execution time of each data read instruction is 8 cycles (expressed in terms of processor clock cycles).

[0099] Dividing the instruction cache line size (128 B) by the instruction size of each data fetch instruction (16 B), we obtain the number of data fetch instructions that the processor can process in each instruction cache line, which is 8 instructions.

[0100] The total execution time for all data read instructions in each instruction cache line is calculated by multiplying the number of data read instructions (8) processed by the processor by the execution time of each data read instruction (8 cycles), which is 64 cycles.

[0101] The instruction prefetch count is calculated by dividing the average instruction fetch time in the system (256 cycles) by the execution time of all data fetch instructions per cache line (64 cycles). The result is 4 cache lines.

[0102] After the prefetch quantity calculation module calculates the second target value, this value is immediately used to guide the instruction prefetching operation for the current memory access thread bundle. The instruction cache controller uses this dynamically calculated second target value instead of the default prefetch quantity to issue instruction fetch requests to the next level of memory.

[0103] The instruction prefetching method provided in this application no longer relies on fixed preset values ​​or offline software configuration. Instead, it calculates the optimal number of instructions to prefetch in real time and dynamically at the hardware level based on the actual load of each memory operation instruction. This on-demand prefetching mechanism ensures that prefetching operations are neither insufficient (leading to performance degradation) nor excessive (leading to cache pollution and bandwidth waste), enabling instruction cache and memory bandwidth resources to be utilized in the most optimized and efficient manner in every operation, thereby significantly improving the overall operating efficiency and performance of the processor.

[0104] The apparatus provided in the embodiments of this application is described below. The apparatus described below can be referred to in correspondence with the method described above.

[0105] Figure 7 This is a schematic diagram of the instruction prefetching device provided in this application, as shown below. Figure 7 As shown, the instruction prefetching device 700 includes: The acquisition unit 710 is used to acquire the thread bundle identifier of the current thread bundle; Determining unit 720 determines the current thread bundle as a memory access thread bundle based on the thread bundle identifier; Comparison unit 730 is used to adjust the instruction prefetch count of the current thread bundle.

[0106] The instruction prefetching device provided in this application embodiment can obtain thread bundle identifiers and distinguish memory access thread bundles based on them, thereby adjusting the number of instructions prefetched in a targeted manner and using a more suitable number of instructions prefetched to meet the instruction requirements. The above-mentioned differentiated processing method avoids over-prefetching of memory access thread bundles, and leaves the saved instruction cache space for the computation thread bundles that need it more. This achieves efficient prefetching of instructions that the processor is about to execute, improves the overall utilization efficiency and hit rate of the instruction cache, reduces unnecessary memory bandwidth consumption, and ultimately improves the overall processing performance of the processor.

[0107] Figure 8 This is a schematic diagram of the structure of the artificial intelligence chip provided in this application, such as... Figure 8 As shown, the artificial intelligence chip 800 includes the instruction prefetching device 700 in the above embodiments.

[0108] Specifically, the Artificial Intelligence (AI) chip in this embodiment refers to a hardware module specifically designed to handle the large number of computational tasks in artificial intelligence applications; it is also known as an AI accelerator or computing card. Its core function is to accelerate machine learning, deep learning, and other algorithms through software and hardware optimization, and it is widely used in fields such as visual processing, speech recognition, and natural language processing.

[0109] Artificial intelligence chips can include parallel processors such as general-purpose graphics processors, application-specific integrated circuits, neuromorphic chips, and digital signal processors.

[0110] The artificial intelligence chip provided in this application embodiment, by setting up an instruction prefetching device, achieves efficient prefetching of instructions that the processor is about to execute, improves the overall utilization efficiency and hit rate of the instruction cache, reduces unnecessary memory bandwidth consumption, and ultimately improves the overall processing performance of the processor.

[0111] Figure 9 This is a schematic diagram of the structure of the electronic device provided in this application, such as... Figure 9As shown, the electronic device may include: a processor 910, a communications interface 920, a memory 930, and a communications bus 940, wherein the processor, communications interface, and memory communicate with each other via the communications bus. The processor can invoke logical commands stored in the memory to execute the methods described in the above embodiments, for example: Get the thread bundle identifier of the current thread bundle; determine that the current thread bundle is a memory access thread bundle based on the thread bundle identifier; adjust the instruction prefetch count of the current thread bundle.

[0112] Furthermore, when the logical commands in the aforementioned memory can be implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several commands to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0113] The processor in the electronic device provided in this application embodiment can call logical instructions in the memory to implement the above method. Its specific implementation method is the same as the aforementioned method implementation method and can achieve the same beneficial effects, which will not be repeated here.

[0114] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the methods provided in the above embodiments.

[0115] The specific implementation method is the same as the aforementioned method implementation method and can achieve the same beneficial effects, so it will not be repeated here.

[0116] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0117] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0118] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An instruction prefetching method, characterized in that, include: Get the thread bundle identifier of the current thread bundle; Based on the thread bundle identifier, the current thread bundle is determined to be a memory access thread bundle; Adjust the instruction prefetch count of the current thread bundle; The number of instructions prefetched is determined based on the following steps: The execution time of the processor for each data read instruction is obtained by dividing the amount of data read by the instruction decoder by the data channel bandwidth. The number of data fetch instructions stored in each instruction cache line can be obtained by dividing the instruction cache line size by the instruction size of each data fetch instruction. The execution time of all data read instructions for each instruction cache line is obtained by multiplying the number of data read instructions processed by the processor in each instruction cache line by the execution time of each data read instruction. The instruction prefetch count is obtained by dividing the average instruction fetch time in the system by the execution time of the processor to process all data fetch instructions for each cache line.

2. The instruction prefetching method according to claim 1, characterized in that, The step of determining that the current thread bundle is a memory access thread bundle based on the thread bundle identifier includes: Compare the thread bundle identifier of the current thread bundle with the preset thread bundle identifier list; In the case of a match, the current thread bundle is determined to be a memory access thread bundle; The preset thread bundle identifier list is used to store the thread bundle identifiers of statically allocated memory access thread bundles.

3. The instruction prefetching method according to claim 1, characterized in that, The step of determining that the current thread bundle is a memory access thread bundle based on the thread bundle identifier includes: Receive the thread bundle identifier of the dynamically specified memory access thread bundle sent by the instruction decoder; Compare the thread bundle identifier of the current thread bundle with the thread bundle identifier of the memory access thread bundle; If a match is found, the current thread bundle is determined to be a memory access thread bundle.

4. The instruction prefetching method according to claim 1, characterized in that, Adjusting the instruction prefetch count of the current thread bundle includes: Reduce the instruction prefetch count of the current thread bundle.

5. The instruction prefetching method according to claim 1, characterized in that, Adjusting the instruction prefetch count of the current thread bundle includes: Adjust the number of instructions prefetched for the current thread bundle to the first target value; The first target value is determined based on the processor performance parameters of the current thread bundle and / or the resource requirement parameters of the program executed by the current thread bundle.

6. The instruction prefetching method according to claim 1, characterized in that, Adjusting the instruction prefetch count of the current thread bundle includes: The amount of data read by the memory access thread bundle sent by the instruction decoder; Based on the instruction read data volume and instruction cache line size, a second target value is determined for the instruction prefetch quantity of the memory access thread bundle; Adjust the instruction prefetch count of the current thread bundle to the second target value.

7. An instruction prefetching device, characterized in that, include: The acquisition unit is used to obtain the thread bundle identifier of the current thread bundle; The determining unit determines that the current thread bundle is a memory access thread bundle based on the thread bundle identifier; A comparison unit is used to adjust the instruction prefetch count of the current thread bundle; The number of instructions prefetched is determined based on the following steps: The execution time of the processor for each data read instruction is obtained by dividing the amount of data read by the instruction decoder by the data channel bandwidth. The number of data fetch instructions stored in each instruction cache line can be obtained by dividing the instruction cache line size by the instruction size of each data fetch instruction. The execution time of all data read instructions for each instruction cache line is obtained by multiplying the number of data read instructions processed by the processor in each instruction cache line by the execution time of each data read instruction. The instruction prefetch count is obtained by dividing the average instruction fetch time in the system by the execution time of the processor to process all data fetch instructions for each cache line.

8. An artificial intelligence chip, characterized in that, Includes the instruction prefetching device as described in claim 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the instruction prefetching method according to any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the instruction prefetching method according to any one of claims 1 to 6.

Images