Instruction prefetching method and device and artificial intelligence chip

Abstract

The invention provides an instruction prefetching method and device and an artificial intelligence chip, and relates to the technical field of chip design and manufacturing, and the method comprises the steps: obtaining a thread bundle identifier of a current thread bundle; determining the current thread bundle as a memory access thread bundle based on the thread bundle identifier; and adjusting the instruction prefetch number of the current thread bundle. According to the method and the device provided by the invention, the memory access thread bundle is prevented from being excessively prefetched, and the saved instruction cache space is reserved for the calculation thread bundle which needs the instruction cache space, so that the instruction to be executed by the processor is efficiently prefetched, the overall utilization efficiency and the hit rate of instruction cache are improved, and the operation efficiency of the processor is improved. Unnecessary memory bandwidth consumption is reduced, and finally the overall processing performance of the processor is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of chip design and manufacturing, and particularly relates to an instruction prefetching method, device and artificial intelligence chip. BACKGROUND

[0002] Instruction prefetching technology is one of the key technologies for a processor (especially a general-purpose graphics processor) to improve execution efficiency and cope with memory delay. By predicting instructions that will be needed in the future during program execution and prefetching, the instructions are loaded into a cache in advance, thereby reducing access delay and speeding up program running, which has a great impact on the performance of the processor.

[0003] Therefore, how to efficiently prefetch instructions to be executed by the processor and improve the utilization of the cache and the performance of the processor has become a technical problem to be solved in the industry. SUMMARY

[0004] The present application provides an instruction prefetching method, device and artificial intelligence chip, which are used to solve the technical problem of how to efficiently prefetch instructions to be executed by the processor and improve the utilization of the cache and the performance of the processor.

[0005] The present application provides an instruction prefetching method, which comprises the following steps. Obtaining a thread bundle identifier of a current thread bundle; Determining that the current thread bundle is a memory access thread bundle based on the thread bundle identifier; Adjusting the number of instructions to be prefetched by the current thread bundle.

[0006] In some embodiments, the step of determining that the current thread bundle is a memory access thread bundle based on the thread bundle identifier comprises the following steps. Comparing the thread bundle identifier of the current thread bundle with a preset thread bundle identifier list; In the case of matching comparison, it is determined that the current thread bundle is a memory access thread bundle; The preset thread bundle identifier list is used to store the thread bundle identifiers of statically allocated memory access thread bundles.

[0007] In some embodiments, the step of determining that the current thread bundle is a memory access thread bundle based on the thread bundle identifier comprises the following steps. Receiving a thread bundle identifier of a dynamically specified memory access thread bundle sent by an instruction decoder; Comparing the thread bundle identifier of the current thread bundle with the thread bundle identifier of the memory access thread bundle; In the case of matching comparison, it is determined that the current thread bundle is a memory access thread bundle.

[0008] In some embodiments, the adjusting the instruction prefetch quantity of the current thread bundle comprises: reducing the instruction prefetch quantity of the current thread bundle.

[0009] In some embodiments, the adjusting the instruction prefetch quantity of the current thread bundle comprises: adjusting the instruction prefetch quantity of the current thread bundle to a first target value; wherein the first target value is determined based on a processor performance parameter of running the current thread bundle and / or a resource demand parameter of a program executed by the current thread bundle.

[0010] In some embodiments, the adjusting the instruction prefetch quantity of the current thread bundle comprises: receiving an instruction read data quantity of a memory access thread bundle sent by an instruction decoder; determining a second target value of the instruction prefetch quantity of the memory access thread bundle based on the instruction read data quantity and an instruction cache line size; adjusting the instruction prefetch quantity of the current thread bundle to the second target value.

[0011] The present application provides an instruction prefetch device, comprising: an obtaining unit configured to obtain a thread bundle identifier of a current thread bundle; a determining unit configured to determine, based on the thread bundle identifier, that the current thread bundle is a memory access thread bundle; a comparing unit configured to adjust an instruction prefetch quantity of the current thread bundle.

[0012] The present application provides an artificial intelligence chip comprising the instruction prefetch device.

[0013] The present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the instruction prefetch method when executing the computer program.

[0014] The present application provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the instruction prefetch method.

[0015] The instruction prefetch method, device and artificial intelligence chip provided by the application can obtain a thread bundle identifier and distinguish a memory access thread bundle based on the identifier, can adjust the number of instructions to be prefetched for the memory access thread bundle, and uses a more suitable number of instructions to be prefetched to meet the instruction demand of the memory access thread bundle. The differential processing method avoids over-prefetching of the memory access thread bundle, saves the instruction cache space for a calculation thread bundle that needs the instruction cache space more, thereby realizing efficient prefetching of instructions to be executed by the processor, improving the overall utilization efficiency and hit rate of the instruction cache, reducing unnecessary memory bandwidth consumption, and finally improving the overall processing performance of the processor. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings, which are incorporated into and form a part of the specification, illustrate an embodiment consistent with the present application and, together with the description, serve to explain the principles of the application.

[0017] In order to more clearly illustrate the technical solutions in the application or the prior art, the accompanying drawings needed in the embodiments or the prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0018] Figure 1 is a flowchart of the instruction prefetch method provided by the application.

[0019] Figure 2 is a flowchart of the thread bundle static identification method provided by the application.

[0020] Figure 3 is a hardware schematic diagram of the thread bundle dynamic identification method provided by the application.

[0021] Figure 4 is a flowchart of the thread bundle dynamic identification method provided by the application.

[0022] Figure 5 is a hardware schematic diagram of the instruction prefetch quantity dynamic determination method provided by the application.

[0023] Figure 6 is a flowchart of the instruction prefetch quantity dynamic determination method provided by the application.

[0024] Figure 7 is a structural schematic diagram of the instruction prefetch device provided by the application.

[0025] Figure 8 is a structural schematic diagram of the artificial intelligence chip provided by the application.

[0026] Figure 9FIG. 1 is a structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0027] In order to make the persons skilled in the art better understand the scheme of the present application, the technical scheme in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by the persons skilled in the art without creative labor should belong to the protection scope of the present application.

[0028] It should be noted that the terms "first", "second", and the like in the present application are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units or modules does not have to be limited to only those steps or units or modules clearly listed, but can include other steps or units or modules that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] ‌General Purpose computing on Graphics Processing Unit (GPGPU) generally adopts Single Instruction Multiple Thread (SIMT) architecture. In order to provide flexibility of program writing, the related technology introduces Cooperative Groups technology, which allows developers to flexibly divide a Thread Group into subgroups with different functions at the software level, thereby realizing complex parallel computing modes that were previously difficult to achieve. However, such flexibility also brings new challenges to hardware. A typical scenario is that different subgroups within a Thread Group may execute completely different code segments. This puts higher requirements on the Instruction Cache prefetch mechanism of the processor.

[0030] If a large number of cache lines are prefetched for all subgroups, for those subgroups that only execute a small number of instructions, it will cause serious waste of cache space and squeeze the cache resources required by other subgroups. If the number of prefetches is uniformly reduced, the cache hit rate of the compute-intensive subgroup will be reduced, resulting in a performance bottleneck.

[0031] A specific sub-group can be designated to be responsible for data read-write tasks. By analyzing the instruction stream of this sub-group, it can be determined that the instruction stream has the characteristics of small number of instructions (code logic is usually relatively simple, and the number of loops is small) and high instruction delay (memory access instructions themselves require a long waiting time). That is, it can be determined that the bandwidth demand of the sub-group for instructions is actually very low. According to the above analysis, the technical solution provided by the present application optimizes instruction prefetching.

[0032] Figure 1 is a flowchart of the instruction prefetching method provided by the present application, as shown in Figure 1 The method comprises steps 110, 120 and 130.

[0033] Step 110, obtaining the thread bundle identifier of the current thread bundle.

[0034] Specifically, the execution subject of the instruction prefetching method provided by the present application is an instruction prefetching device. The device can be implemented by software, such as an instruction prefetching program running in a processor; or by hardware, such as a hardware module for executing the instruction prefetching method.

[0035] In a specific embodiment, the instruction prefetching device is arranged in the instruction cache unit (Cache) of the processor.

[0036] A thread bundle (Warp) is a basic unit of execution and scheduling of general-purpose graphics processors and other parallel processors, usually composed of a group (for example, 32) of threads (Thread) executing the same instruction in parallel. In order to manage and schedule a large number of thread bundles, the hardware of the processor will allocate a unique thread bundle identifier (Warp ID) to each active thread bundle. The thread bundle identifier can be used to uniquely identify and track the thread bundle throughout its life cycle.

[0037] When a thread bundle needs to obtain new instructions during execution, for example, when a cache miss occurs, the instruction fetcher will generate an instruction fetch request. The instruction fetch request will be sent to the instruction cache unit. In order to indicate which thread bundle needs the instruction, the thread bundle identifier of the requester must be included in the instruction fetch request. Therefore, the obtaining in the present embodiment can be understood as the instruction prefetching device parsing or extracting the thread bundle identifier of the thread bundle for which instructions are currently needed from the received instruction fetch request.

[0038] Step 120, determining that the current thread bundle is a memory access thread bundle based on the thread bundle identifier.

[0039] Specifically, after obtaining the thread bundle identifier, the type of the current thread bundle needs to be determined. The present application defines a thread bundle that mainly executes high-latency memory access operations, has a simple instruction stream and low instruction density as a memory access thread bundle. The typical task of the memory access thread bundle is to load data from the low-speed, high-latency global memory to the high-speed, low-latency shared memory. In contrast, the calculation thread bundle that performs a large number of arithmetic logic operations or other types of thread bundles (which can be collectively referred to as non-memory access thread bundles) have high instruction density and have higher requirements for instruction bandwidth.

[0040] The type of the current thread bundle can be determined by the thread bundle identifier itself or relevant information associated with the identifier to determine whether it belongs to a pre-defined memory access thread bundle.

[0041] In a specific embodiment, the determination process can be specifically: comparing the obtained thread bundle identifier of the current thread bundle with a preset identifier set. If the thread bundle identifier of the current thread bundle exists in the preset identifier set, it is determined that the current thread bundle is a memory access thread bundle; otherwise, if it does not exist, it is determined that it is a non-memory access thread bundle. The preset identifier set can be stored in a specific register of the processor, on-chip memory or a dedicated hardware lookup table for fast query and matching by the instruction prefetch device. For example, in an application scenario, two thread bundles with ID 0 and ID 1 can be pre-specified by a programming model or a driver program to be used for executing memory copy tasks. Therefore, the preset identifier set contains {0, 1}. When the instruction prefetch device obtains an instruction request with a thread bundle identifier of 0, it can be determined that the thread bundle is a memory access thread bundle by query comparison.

[0042] Step 130, adjusting the instruction prefetch quantity of the current thread bundle.

[0043] Specifically, the instruction prefetch quantity refers to the amount of instruction data continuously loaded from the main memory (or the next level cache) to the instruction cache unit when an instruction prefetch operation occurs, which is usually measured by the number of cache lines. The pre-fetch mechanism in the related art usually adopts a fixed, larger instruction prefetch quantity (such as 4 or 8 cache lines at a time) for all thread bundles in order to expect to improve the hit rate of subsequent instructions.

[0044] The present embodiment adjusts the instruction prefetch quantity applied to the current thread bundle after determining that it is a memory access thread bundle, so as to distinguish it from the prefetch quantity of the non-memory access thread bundle.

[0045] The adjustment can be adjusting the instruction prefetch quantity from a default value to a preset smaller value. Because the instruction sequence of a memory access thread bundle is usually short (e.g., only contains a few load, store and address calculation instructions), and the delay of executing these instructions is mainly in the memory access itself, rather than in instruction fetching. Therefore, prefetching a large number of instructions for it not only cannot bring performance benefits, but also occupies valuable instruction cache space, causing the instructions of other computation thread bundles to be replaced prematurely, thereby causing more cache misses.

[0046] For example, it is assumed that the default instruction prefetch quantity of the system for a common computation thread bundle is 4 cache lines. When it is determined that the current thread bundle is a memory access thread bundle, the instruction prefetch quantity thereof can be adjusted to 1 cache line. In this way, only the most needed instruction and a small number of instructions adjacent thereto are loaded into the instruction cache unit, which not only meets the execution requirement of the memory access thread bundle, but also maximizes the saving of cache space. The adjusted smaller value (e.g., 1 or 2 cache lines) can be fixed by hardware or configured by software.

[0047] The instruction prefetch method provided by the embodiments of the present application can distinguish a memory access thread bundle based on the thread bundle identifier, and can adjust the instruction prefetch quantity of the memory access thread bundle accordingly, so as to meet the instruction requirement of the memory access thread bundle by using a more suitable instruction prefetch quantity. The above differentiated processing manner avoids over-prefetching of the memory access thread bundle, and saves the instruction cache space for computation thread bundles that need it more, so as to realize efficient prefetching of instructions to be executed by the processor, improve the overall utilization efficiency and hit rate of the instruction cache, reduce unnecessary memory bandwidth consumption, and finally improve the overall processing performance of the processor.

[0048] It should be noted that each embodiment of the present application can be freely combined, the order can be changed or the embodiments can be executed independently, and does not need to rely on or depend on a fixed execution order.

[0049] In some embodiments, determining, based on the thread bundle identifier, that the current thread bundle is a memory access thread bundle comprises: comparing the thread bundle identifier of the current thread bundle with a preset thread bundle identifier list; in a case of matching comparison, determining that the current thread bundle is a memory access thread bundle; The preset thread bundle identifier list is used to store the thread bundle identifiers of the statically allocated memory access thread bundles.

[0050] In particular, the preset thread bundle identification list is a collection of thread bundle identifications. The content in the list is preset, meaning that it can be configured before the cores in the processor execute the application or in the initial stage of execution. The list can be physically implemented as one or more dedicated registers, a dedicated piece of on-chip static random access memory, or a hardware lookup table. The choice of storage medium for the list can be a trade-off between performance requirements and hardware costs, with the core requirement being that it can be quickly accessed by the instruction prefetch device.

[0051] Static allocation refers to the division of thread bundle tasks being determined by software (e.g., programmers through a specific programming interface or compilers through code behavior analysis) before program compilation or execution. After analyzing the entire computing task, the programmer or compiler can pre-plan to dedicate one or more specific thread bundles to execute memory-intensive tasks. For example, when writing a program, the programmer can explicitly specify in the code that the thread bundle with thread bundle ID 0 is responsible for loading data from global memory. Therefore, the thread bundle identifications stored in the preset thread bundle identification list are those thread bundle identifications that are statically allocated as memory access thread bundles.

[0052] Figure 2 is a flowchart of the thread bundle static identification method provided by the present application, as shown in Figure 2 The comparison operation in the present embodiment is performed in the instruction prefetch device. The instruction prefetch device is integrated in the instruction cache unit. The instruction prefetch device at least includes a comparator and at least one selector.

[0053] When the instruction cache unit receives an instruction fetch request sent by the fetcher, the comparator passes the thread bundle identification of the current thread bundle carried in the request as input to the comparator. The comparator then reads all entries in the preset thread bundle identification list and compares the input thread bundle identification with each identification in the list. The comparison operation can be parallel to ensure a low comparison delay.

[0054] If the comparator finds that the input current thread bundle identification is the same as any entry in the preset thread bundle identification list, i.e., a comparison match occurs, the comparator outputs a valid signal. This signal explicitly indicates that the thread bundle currently requesting the instruction is a memory access thread bundle. If the comparator compares the input thread bundle identification with all entries in the list and does not find the same item, i.e., a comparison mismatch occurs, the signal will be in an invalid state, indicating that the current thread bundle is a non-memory access thread bundle.

[0055] Subsequently, the instruction prefetch unit determines the subsequent operation according to the comparison result signal. For example, after determining that the current thread bundle is a memory access thread bundle, the selector is triggered to select the instruction prefetch quantity corresponding to the memory access thread bundle for output; if the current thread bundle is a non-memory access thread bundle, the selector selects the instruction prefetch quantity (which can be a default value) corresponding to the non-memory access thread bundle for output. Generally, the instruction prefetch quantity corresponding to the memory access thread bundle is less than the instruction prefetch quantity corresponding to the non-memory access thread bundle.

[0056] The instruction prefetch method provided by the embodiment of the present application uses the cooperative mode of software static allocation and hardware rapid comparison to implement a low-overhead and high-efficiency memory access thread bundle identification mechanism. Through the static allocation mechanism, the programmer or compiler is given the ability to directly control and optimize the instruction prefetch behavior, so that the optimization is more deterministic and targeted, thereby accurately adjusting the prefetch strategy for a specific memory access thread bundle and realizing fine management of instruction cache resources and improving the overall utilization efficiency and hit rate of the instruction cache.

[0057] In some embodiments, determining that the current thread bundle is a memory access thread bundle based on the thread bundle identification includes: receiving the thread bundle identification of the dynamically specified memory access thread bundle sent by the instruction decoder; comparing the thread bundle identification of the current thread bundle with the thread bundle identification of the memory access thread bundle; in the case of matching comparison, determining that the current thread bundle is a memory access thread bundle.

[0058] Specifically, unlike the static allocation mechanism in the above embodiment, the embodiment discloses a more automated and intelligent dynamic identification mechanism. The mechanism does not need to be specified in advance by the programmer or compiler, but is dynamically identified by the processor hardware at runtime.

[0059] The instruction decoder is a key hardware unit in the instruction pipeline of the processor, and its main responsibility is to parse the instruction code taken out from the instruction cache unit, identify the operation type (such as arithmetic operation, logical operation, memory load / storage, etc.), operand, and addressing mode of the instruction, and generate corresponding control signals to drive the subsequent execution unit.

[0060] In the embodiment, the function of the instruction decoder is extended. When the instruction decoder parses a specific instruction or a specific type of instruction, it can determine that the thread bundle executing the instruction is executing a memory access task.

[0061] Dynamic assignment, in contrast to static allocation in the above embodiments, refers to the identification process occurring during the actual execution of the program. When a thread executes a specific memory access instruction, the hardware automatically "marks" or "assigns" it as a memory access thread. This assignment is immediate, based on the current instruction, and may be temporary.

[0062] Figure 3 This is a hardware schematic diagram of the thread bundle dynamic identification method provided in this application, such as... Figure 3 As shown, an interface or communication link can be added from the instruction decoder to the instruction cache unit to send the thread bundle identifier of the dynamically specified memory access thread bundle. When the instruction decoder decodes a predefined memory access instruction in its pipeline, it not only generates the control signals required to execute the instruction, but also actively sends the thread bundle identifier of the thread bundle currently executing the instruction to the instruction cache unit through this new interface.

[0063] Figure 4 This is a flowchart illustrating the thread bundle dynamic identification method provided in this application, as shown below. Figure 4 As shown, after receiving the thread bundle identifier sent by the instruction decoder, the instruction prefetching unit in the instruction cache unit stores it in an internal list or lookup table. This list, which can be called the "dynamic memory access thread bundle identifier list," records in real time the thread bundles currently identified as performing memory access. The contents of this list change dynamically; when a new memory access thread bundle is identified, its thread bundle identifier is added. Correspondingly, a mechanism can also be designed to remove the thread bundle identifier from the list when the thread bundle completes its memory access task.

[0064] The comparison operation in this embodiment is similar to that in the above embodiments, and will not be repeated here. However, the objects being compared are different. Here, when the instruction cache unit needs to make a prefetch decision for the current thread bundle (i.e., the thread bundle that is requesting instruction prefetching), it compares the thread bundle identifier of the current thread bundle with all the thread bundle identifiers dynamically sent by the instruction decoder stored in the dynamic memory access thread bundle list mentioned above.

[0065] If the comparison matches, the current thread bundle is determined to be a memory access thread bundle. Once determined to be a memory access thread bundle, the instruction prefetching mechanism adjusts the number of instructions prefetched for the current thread bundle. If the comparison does not match, it is considered a non-memory access thread bundle, and the default prefetching strategy is used.

[0066] The instruction prefetch method provided by the embodiment of the present application realizes a mechanism of automatically identifying a memory access thread bundle at runtime; can adaptively capture the dynamic behavior of program runtime, and can adjust the instruction prefetch quantity of the dynamically identified memory access thread bundle in real time. This makes the optimization of the instruction cache more intelligent and flexible, can cope with more diverse application program behavior patterns, and thus effectively improves the processor performance.

[0067] In some embodiments, adjusting the instruction prefetch quantity of the current thread bundle comprises: reducing the instruction prefetch quantity of the current thread bundle.

[0068] Specifically, in a parallel processor such as a general-purpose graphics processor, there is usually a default instruction prefetch quantity. The default value is to cope with most cases, especially for computation-intensive thread bundles. Such thread bundles usually have complex instruction streams and high instruction level parallelism, so prefetching more instructions at a time (for example, prefetching 2, 4 or more cache lines at a time) can effectively improve the hit rate of the instruction cache and avoid the processor from stalling due to waiting for instructions.

[0069] According to the analysis of the characteristics of the thread bundle by the present application, the behavior pattern of the memory access thread bundle is significantly different from that of the non-memory access thread bundle. The instruction sequence of the memory access thread bundle is usually very short and fixed in structure (for example, a short loop containing address calculation, data loading / storage and loop control), and the main bottleneck of its execution speed is waiting for the completion of the high-delay memory operation, rather than the speed of instruction acquisition. In view of this feature, the specific way of adjusting the instruction prefetch quantity of the current thread bundle proposed by the present embodiment is to reduce the instruction prefetch quantity of the current thread bundle.

[0070] The instruction prefetch quantity is reduced from the default value to a smaller value. The smaller value should be sufficient to cover the core instructions required by the memory access thread bundle in one execution iteration, but should not occupy too much cache space.

[0071] The instruction prefetch method provided by the embodiment of the present application avoids loading a large number of instructions that will not be executed by the memory access thread bundle into the cache by mistake, thereby preventing these useless instructions from replacing the valid instructions required by other computation thread bundles. Secondly, the overall space utilization and hit rate of the instruction cache unit are improved, so that the valuable cache resources can be more efficiently allocated to the computation-intensive thread bundles that really need them. Finally, the memory bandwidth occupied by instruction acquisition is reduced, because unnecessary data transmission is avoided.

[0072] In some embodiments, adjusting the instruction prefetch quantity of the current thread bundle comprises: adjusting the instruction prefetch quantity of the current thread bundle to a first target value; wherein the first target value is determined based on a processor performance parameter and / or a resource requirement parameter of a program executed by the current thread bundle.

[0073] In particular, the embodiment provides a static configuration method for the instruction prefetch quantity, and the configured value can more accurately match the actual requirement of the memory access thread bundle.

[0074] The first target value is a specific and quantitative instruction prefetch quantity, for example, 1 cache line or 2 cache lines. It is distinguished from the default prefetch quantity that may exist in hardware and is used for non-memory access thread bundles. When a thread bundle is identified as a memory access thread bundle, the instruction prefetch logic will no longer use the default value, but will use the first target value to guide its prefetch operation.

[0075] The first target value is stored in a location that can be accessed and modified by software, for example, a dedicated configuration register. This provides a hardware basis for the static configuration of the value.

[0076] The determination process of the first target value is a performance trade-off and optimization process, which can be completed in the development and debugging phase.

[0077] The processor performance parameter mainly refers to the inherent properties of the processor hardware platform, which will affect the effectiveness of the instruction prefetch strategy.

[0078] In a specific embodiment, the processor performance parameter can include the following parameters: the capacity size of the instruction cache unit and the capacity size of the cache line, etc.

[0079] In a processor with a small capacity of the instruction cache unit, the cache pollution problem caused by excessive prefetching will be more serious. Therefore, for such a processor, a very small first target value (for example, 1) can be selected to maximize the protection of valuable cache space.

[0080] In a processor with a large capacity of the cache line, if the cache line of the hardware is very large (for example, 128 bytes), then even if the first target value is 1, enough instructions can be loaded at a time. On the contrary, if the cache line is small (for example, 32 bytes), the first target value may need to be set to 2 to completely cover the instructions of a short loop.

[0081] The resource requirement parameter of the program refers to the resource requirement characteristics of the executed application (kernel), which will also affect the effectiveness of the instruction prefetch strategy.

[0082] In a specific embodiment, the resource requirement parameter of the program can include the following parameters: the proportion of the memory access thread bundle in all thread bundles and the performance data of the program, etc.

[0083] In an application, if the proportion of memory access thread bundles in all thread bundles is small, that is, most of the thread bundles are compute-intensive non-memory access thread bundles, and only a few are memory access thread bundles, it is necessary to reduce the prefetch amount of memory access thread bundles to the minimum, that is, to set a very small first target value, in order to guarantee the performance of the main task.

[0084] The instruction prefetch amount can also be configured from small to large possible values. Then run the entire program, and count the total execution time of the program, the instruction cache miss rate and other key performance indicators. By comparing the performance data under different configurations, the first target value that can make the overall performance optimal is finally determined.

[0085] The instruction prefetch method provided by the embodiment of the application performs fine performance tuning according to the specific hardware platform characteristics and application program requirements. This kind of software and hardware cooperative optimization method can make the instruction prefetch strategy better adapt to diversified application scenarios, realize targeted deep optimization, and thus maximize the hardware potential and improve the overall performance of the processor.

[0086] In some embodiments, adjusting the instruction prefetch amount of the current thread bundle comprises: receiving the instruction read data amount of the memory access thread bundle sent by the instruction decoder; determining a second target value of the instruction prefetch amount of the memory access thread bundle based on the instruction read data amount and the instruction cache line size; adjusting the instruction prefetch amount of the current thread bundle to the second target value.

[0087] Specifically, the instruction read data amount (memory copy size) refers to the total amount of data that needs to be read by an instruction, for example, the size of the data block copied from the global memory (processor storage unit) to the shared memory.

[0088] The instruction cache line size (Cache Line Size) is a fixed parameter of the processor hardware, which represents the basic unit of single data exchange of the instruction cache, for example, 64 bytes or 128 bytes. The value is known to the hardware logic.

[0089] Figure 5 is a hardware schematic diagram of the instruction prefetch amount dynamic determination method provided by the application, as Figure 5As shown, in this embodiment, the function of the instruction decoder is further enhanced. When it parses such memory operation instruction carrying data amount information, it not only identifies the instruction type, but also can extract the instruction read data amount. Then, the instruction decoder will send the instruction read data amount and the thread bundle identification of the instruction execution to the instruction cache unit through the hardware interface between them.

[0090] Figure 6 is the flowchart of the method for dynamically determining the instruction prefetch amount provided by the present application, as shown in Figure 6 In this embodiment, unlike the first target value statically configured in the above embodiment, the second target value is dynamically calculated.

[0091] The instruction prefetch device in the instruction cache unit receives two inputs: the instruction read data amount from the instruction decoder and the instruction cache line size inherent to the hardware. The logic of the calculation is to determine how many instruction cache lines need to be prefetched to completely cover the data amount involved in the memory operation task to be executed.

[0092] In this embodiment, the calculation method of the instruction prefetch amount is as follows: Step one, based on the instruction read data amount of the instruction decoder divided by the data channel bandwidth (the number of binary bits that can be transmitted in parallel by the data bus in the processor at one time), the execution time of the processor processing each data read instruction is obtained.

[0093] In the case of a large number of instructions, the average execution time of the processor processing each data read instruction can be used to determine the execution time of the processor processing each data read instruction.

[0094] Step two, based on the instruction cache line size divided by the instruction size of each data read instruction, the number of data read instructions stored in each instruction cache line processed by the processor is obtained.

[0095] Step three, based on the number of data read instructions stored in each instruction cache line processed by the processor multiplied by the execution time of the processor processing each data read instruction, the execution time of all data read instructions in each instruction cache line processed by the processor is obtained.

[0096] Step four, based on the average time of instruction reading in the system (the average time taken by the processor to receive the data read instruction after issuing a request for the instruction) divided by the execution time of the processor processing all data read instructions in each cache line, the instruction prefetch amount (the number of cache lines that need to be prefetched) is obtained.

[0097] A specific example is given below. Assume that the instruction cache line size is 128 B (bytes), the average length of instruction reading in the system is 256 cycles (clock periods), the data channel bandwidth is 128 B, the average instruction size of each data read instruction is 16 B, and the instruction reading data amount of the instruction decoder is 1 MB (megabytes).

[0098] According to the instruction reading data amount (1 MB) of the instruction decoder divided by the data channel bandwidth (128 B), the execution length of the processor for processing each data read instruction is obtained, which is 8 cycles (converted into the clock period of the processor for representation).

[0099] According to the instruction cache line size (128 B) divided by the instruction size (16 B) of each data read instruction, the number of data read instructions stored in each instruction cache line and processed by the processor is obtained, which is 8 instructions.

[0100] According to the number of data read instructions stored in each instruction cache line and processed by the processor (8) multiplied by the execution length of the processor for processing each data read instruction (8 cycles), the execution length of the processor for processing all data read instructions in each instruction cache line is obtained, which is 64 cycles.

[0101] According to the average length of instruction reading in the system (256 cycles) divided by the execution length of the processor for processing all data read instructions in each cache line (64 cycles), the number of instructions to be prefetched is obtained, which is 4 cache lines.

[0102] After the second target value is calculated by the prefetch number calculation module, the value is immediately used to guide the instruction prefetch operation of the current memory access thread bundle. The instruction cache controller uses the dynamically calculated second target value to replace the default prefetch number, and sends an instruction acquisition request to the next level memory.

[0103] The instruction prefetch method provided by the embodiments of the present application no longer depends on fixed preset values or offline software configuration, but dynamically calculates the optimal instruction prefetch number in real time at the hardware level according to the actual load of each memory operation instruction. This on-demand prefetch mechanism ensures that the prefetch operation is neither insufficient (leading to performance degradation) nor excessive (leading to cache pollution and bandwidth waste), so that the instruction cache and memory bandwidth resources can be optimally and efficiently utilized in each operation, thereby significantly improving the overall running efficiency and performance of the processor.

[0104] The apparatus provided by the embodiments of the present application is described below, and the apparatus described below can be correspondingly referred to the method described above.

[0105] Figure 7 is a structural schematic diagram of an instruction prefetching apparatus provided by the present application, as shown in Figure 7 The instruction prefetching apparatus 700 comprises: An obtaining unit 710, configured to obtain a thread bundle identifier of a current thread bundle; A determining unit 720, configured to determine, based on the thread bundle identifier, that the current thread bundle is a memory access thread bundle; A comparing unit 730, configured to adjust an instruction prefetching quantity of the current thread bundle.

[0106] The instruction prefetching apparatus provided by the embodiment of the present application can adjust the instruction prefetching quantity of the memory access thread bundle by obtaining the thread bundle identifier and distinguishing the memory access thread bundle based on the thread bundle identifier, and can use a more suitable instruction prefetching quantity to meet the instruction requirement of the memory access thread bundle; the above differentiated processing manner avoids over-prefetching of the memory access thread bundle, and saves the instruction cache space for the computing thread bundle which needs it more, so as to realize efficient prefetching of the instructions to be executed by the processor, improve the overall utilization efficiency and hit rate of the instruction cache, reduce unnecessary memory bandwidth consumption, and finally improve the overall processing performance of the processor.

[0107] Figure 8 is a structural schematic diagram of an artificial intelligence chip provided by the present application, as shown in Figure 8 The artificial intelligence chip 800 comprises the instruction prefetching apparatus 700 in the above embodiment.

[0108] Specifically, the artificial intelligence chip (Artificial Intelligence, AI) in the embodiment refers to a hardware module specially used for processing a large amount of computing tasks in artificial intelligence applications, and is also called an AI accelerator or a computing card. Its core function is to accelerate machine learning, deep learning and other algorithms through software and hardware optimization, and it is widely used in the fields of visual processing, speech recognition, natural language processing and the like.

[0109] The types of the artificial intelligence chip can include general-purpose graphics processors, application-specific integrated circuits, brain-like chips and digital signal processors and other parallel processors.

[0110] The artificial intelligence chip provided by the embodiment of the present application realizes efficient prefetching of the instructions to be executed by the processor, improves the overall utilization efficiency and hit rate of the instruction cache, reduces unnecessary memory bandwidth consumption, and finally improves the overall processing performance of the processor.

[0111] Figure 9 is a structural schematic diagram of an electronic device provided by the present application, as shown in Figure 9As shown, the electronic device can include a processor 910, a communications interface 920, a memory 930, and a communications bus 940, wherein the processor, the communications interface, and the memory complete mutual communication through the communications bus. The processor can invoke a logical command in the memory to execute the method described in the above embodiments, for example: Obtaining a thread bundle identifier of a current thread bundle; determining, based on the thread bundle identifier, that the current thread bundle is a memory access thread bundle; and adjusting an instruction prefetch quantity of the current thread bundle.

[0112] In addition, the logical command in the memory described above can be implemented in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, includes several commands to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0113] The processor in the electronic device provided by the embodiments of the present application can invoke the logical command in the memory to implement the above method, and the specific implementation manners and the same beneficial effects as the foregoing method implementation manners are consistent, and will not be described here.

[0114] The embodiments of the present application also provide a computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method provided by the above embodiments.

[0115] The specific implementation manners and the same beneficial effects as the foregoing method implementation manners are consistent, and will not be described here.

[0116] The embodiments of the present application provide a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the above method.

[0117] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0118] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, and the computer software products can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and include a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0119] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An instruction prefetching method, characterized in that, include: Get the thread bundle identifier of the current thread bundle; Based on the thread bundle identifier, the current thread bundle is determined to be a memory access thread bundle; Adjust the instruction prefetch count of the current thread bundle.

2. The instruction prefetching method according to claim 1, characterized in that, The step of determining that the current thread bundle is a memory access thread bundle based on the thread bundle identifier includes: Compare the thread bundle identifier of the current thread bundle with the preset thread bundle identifier list; In the case of a match, the current thread bundle is determined to be a memory access thread bundle; The preset thread bundle identifier list is used to store the thread bundle identifiers of statically allocated memory access thread bundles.

3. The instruction prefetching method according to claim 1, characterized in that, The step of determining that the current thread bundle is a memory access thread bundle based on the thread bundle identifier includes: Receive the thread bundle identifier of the dynamically specified memory access thread bundle sent by the instruction decoder; Compare the thread bundle identifier of the current thread bundle with the thread bundle identifier of the memory access thread bundle; If a match is found, the current thread bundle is determined to be a memory access thread bundle.

4. The instruction prefetching method according to claim 1, characterized in that, Adjusting the instruction prefetch count of the current thread bundle includes: Reduce the instruction prefetch count of the current thread bundle.

5. The instruction prefetching method according to claim 1, characterized in that, Adjusting the instruction prefetch count of the current thread bundle includes: Adjust the number of instructions prefetched for the current thread bundle to the first target value; The first target value is determined based on the processor performance parameters of the current thread bundle and / or the resource requirement parameters of the program executed by the current thread bundle.

6. The instruction prefetching method according to claim 1, characterized in that, Adjusting the instruction prefetch count of the current thread bundle includes: The amount of data read by the memory access thread bundle sent by the instruction decoder; Based on the instruction read data volume and instruction cache line size, a second target value is determined for the instruction prefetch quantity of the memory access thread bundle; Adjust the instruction prefetch count of the current thread bundle to the second target value.

7. An instruction prefetching device, characterized in that, include: The acquisition unit is used to obtain the thread bundle identifier of the current thread bundle; The determining unit determines that the current thread bundle is a memory access thread bundle based on the thread bundle identifier; The comparison unit is used to adjust the instruction prefetch count of the current thread bundle.

8. An artificial intelligence chip, characterized in that, Includes the instruction prefetching device as described in claim 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the instruction prefetching method according to any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the instruction prefetching method according to any one of claims 1 to 6.

Images