Data prefetching method of GPU, graphics processing unit, device and medium

Abstract

Embodiments of the present application disclose a GPU data overissue prefetch method, a graphics processor, an apparatus and a medium. The method comprises: a GPU execution core continuously issuing a plurality of data read instructions for a same destination address to a sampling controller, each data read instruction carrying a request version number of the destination address; the sampling controller initiating a data read request to a data read-write unit according to the data read instruction; the data read-write unit obtaining prefetch data in a global memory according to the data read request, marking and storing the prefetch data using the request version number; the data read-write unit determining a target request version number according to a synchronization flag of the GPU execution core, and transmitting the prefetch data corresponding to the target request version number to a general register; and the GPU execution core reading the prefetch data in the general register for calculation. Multiple data read instructions for the same destination address are simultaneously issued, multiple data reads are delayed and folded, and the execution core computing power is improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method for over-fetching data over-processing in a GPU, a graphics processor, an apparatus, and a medium. Background Technology

[0002] With the explosive development of artificial intelligence (AI) technology, the computing power of graphics processing units (GPUs) has become a core factor affecting AI inference performance. GPU computing power is directly affected by its internal execution capabilities, as well as by the data supply path.

[0003] To fully utilize the computing power of the GPU's internal execution cores, the data required for their execution needs to be prepared in advance; otherwise, the execution cores will experience idle time while waiting for data. However, in current technologies, when general-purpose register (GPR) resources are limited, a serial approach is typically used to load and consume data sequentially, followed by another round of loading and consuming. This serial approach causes the latency of multiple data loading rounds to accumulate, exacerbating data read latency and resulting in idle time for the GPU execution cores, thus impacting GPU computing power. Summary of the Invention

[0004] This invention provides a GPU data over-fetching method, graphics processor, device, and medium to fold multiple data read delays and improve the computing power of the execution core.

[0005] According to one aspect of the present invention, a data over-fetching method for a GPU is provided. This method is applied in a GPU, the GPU comprising: a GPU execution core, a sampling controller, a data read / write unit, general-purpose registers, and global memory; the method includes:

[0006] The GPU execution core continuously issues multiple data read instructions to the sampling controller for the same destination address, wherein each data read instruction carries a request version number for the destination address;

[0007] The sampling controller initiates a data read request to the data read / write unit according to the data read instruction;

[0008] The data read / write unit retrieves prefetched data from global memory according to the data read request, marks the prefetched data with the request version number, and stores the prefetched data.

[0009] The data read / write unit determines the target request version number based on the synchronization flag of the GPU execution core, and transmits the prefetch data corresponding to the target request version number to the general-purpose register;

[0010] The GPU execution core reads prefetched data from general-purpose registers for computation.

[0011] According to another aspect of the present invention, a graphics processing unit is provided, comprising: a GPU execution core, a sampling controller, a data read / write unit, general-purpose registers, and global memory; wherein,

[0012] The GPU execution core is used to continuously issue multiple data read instructions to the sampling controller for the same destination address, wherein each data read instruction carries a request version number of the destination address;

[0013] A sampling controller is used to initiate a data read request to the data read / write unit according to the data read instruction;

[0014] The data read / write unit is used to obtain prefetched data from global memory according to the data read request, mark the prefetched data with the request version number, and store the prefetched data.

[0015] The data read / write unit is also used to determine the target request version number based on the synchronization flag of the GPU execution core, and to transfer the prefetch data corresponding to the target request version number to the general-purpose register;

[0016] The GPU execution core is also used to read prefetched data from general-purpose registers for computation.

[0017] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0018] At least one processor; and a memory communicatively connected to said at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the GPU data over-fetching method according to any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the GPU data over-fetching method according to any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the GPU data over-fetching method according to any embodiment of the present invention.

[0022] The technical solution of this invention involves the GPU execution core continuously issuing multiple data read instructions to the sampling controller for the same destination address. Each data read instruction carries a request version number for the destination address. The sampling controller initiates a data read request to the data read / write unit based on the data read instructions. The data read / write unit retrieves prefetched data from global memory based on the data read request, marks and stores the prefetched data using the request version number. The data read / write unit determines the target request version number based on the synchronization flag of the GPU execution core and transfers the prefetched data corresponding to the target request version number to a general-purpose register. The GPU execution core reads the prefetched data from the general-purpose register for computation, thus solving the problem of GPU data prefetching. By simultaneously issuing multiple data read instructions for the same destination address, the delay of multiple data reads can be folded, improving the computing power of the execution core.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of a GPU data over-prefetching method according to Embodiment 1 of the present invention;

[0026] Figure 2 This is a schematic diagram of the structure of a graphics processor according to Embodiment 2 of the present invention;

[0027] Figure 3 This is a schematic diagram of the structure of an electronic device that implements the GPU data over-fetching method according to an embodiment of the present invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] Example 1

[0031] Figure 1 This is a flowchart of a GPU data over-fetching method according to Embodiment 1 of the present invention. This embodiment is applicable to situations where data is acquired in advance when the GPU execution core performs calculations. The method can be executed by a graphics processor, which can be configured in an electronic device, such as a mobile phone, computer, server, or controller.

[0032] In this embodiment of the invention, the GPU includes: a GPU execution core, a sampling controller, a data read / write unit, general-purpose registers, and global memory. The GPU execution core is the main computing component, capable of executing parallel threads. It is responsible for performing specific operations such as integer, floating-point addition, subtraction, multiplication, division, logical judgments, or matrix multiplication and addition. The sampling controller is responsible for acquiring data according to data read instructions such as SMP, LD, or LDM instructions. The data read / write unit, based on the data read instructions transmitted by the sampling controller, is responsible for performing data migration operations between the GPR and global memory. The data read / write unit can be implemented using a First-In-First-Out (FIFO) controller. The general-purpose registers provide high-speed data temporary storage space for the execution core to support arithmetic and logical operations. Global memory can be the video memory on the GPU board, such as Graphics Double Data Rate 6 (GDDR6) or High Bandwidth Memory (HBM), with a maximum capacity (up to tens of GB). Global memory, as a large-capacity off-chip storage medium, is used to store input data to be processed, intermediate calculation results, and final output data, thereby realizing a complete calculation process from data loading, operation execution to result writing back.

[0033] For example, a serial data loading instruction stream could be:

[0034] LD_Matrix R0, [R20.xy], t2 / / Load the data from t2 into R0-R7

[0035] LD_Matrix R8, [R22.xy], t2 / / Load the data from t2 into R8-R15

[0036] EXE_Matrix / / Perform the first round of matrix operations using matrix data R0-R15 obtained from t2.

[0037] LD_Matrix R0, [R24.xy], t7 / / Load data from t7 into R0-R7

[0038] LD_Matrix R8, [R26.xy], t7 / / Load data from t7 into R8-R15

[0039] EXE_Matrix / / Performs a second round of matrix operations using matrix data R0-R15 obtained from t7.

[0040] In the above instruction stream for serial data loading, the content after " / / " on the right side of the instruction is the explanation content. Taking "LD_Matrix R0, [R20.xy], t2" as an example, LD_Matrix is ​​the opcode, indicating a matrix loading instruction; R0 is the destination operand, representing register R0; t2 represents the texture resource in the second slot of the texture unit, which is the source operand in the data read instruction; "[R20.xy]" is the texture coordinate of the source operand in the texture shader resource view (texture sharp, t sharp). When the remaining 16 GPRs (R0 to R15) are available for storing matrix data, the data needed for the first round of matrix operations can be loaded first using "LD_MatrixR0, [R20.xy], t2" and "LD_Matrix R8, [R22.xy], t2". After the result of the first round of matrix operations is obtained, the data used for the first round of matrix operations in R0 to R15 is no longer needed. At this time, the second round of matrix data loading instructions "LD_Matrix R0, [R24.xy], t7" and "LD_Matrix R8, [R26.xy], t7" are issued to retrieve the data needed for the second round of matrix operations. The delays of the first and second rounds of matrix data loading are effectively added together, causing idle time in the GPU execution cores and resulting in a reduction in computing power.

[0041] The embodiments of the present invention can improve the existing serial data loading method by folding the delay in multiple rounds of data loading. Taking the above data loading situation as an example, the GPU data over-fetching method provided by the embodiments of the present invention only requires one round of data loading delay to complete multiple rounds of data prefetching, thereby improving GPU computing power.

[0042] like Figure 1 As shown, the method includes:

[0043] Step 110: The GPU execution core continuously sends multiple data read instructions to the sampling controller for the same destination address. Each data read instruction carries the request version number of the destination address.

[0044] The data read instruction can be a scalar data read instruction or a matrix data read instruction. The data read instruction can be a Texture Sample Instruction (SMP) or a Load Instruction (LD). When loading matrix data, the data read instruction can be "SMP_LDM" (representing texture sample matrix data) or "LD_Matrix" (representing loaded matrix data), etc. Continuous delivery allows subsequent rounds of data read instructions to be issued even before the data obtained in the first round has been used.

[0045] For multiple data read instructions to the same destination address, it can be understood that the data obtained from multiple rounds of data reads needs to be stored in the same destination address, i.e., over-issuing data read instructions for the same destination address. When resources are sufficient, the data obtained from multiple rounds of data reads can be stored in different destinations, and the over-issuance prefetching method of this embodiment can be avoided. However, when GPR resources are scarce and there are not enough registers for data storage, the same registers may need to be reused for data storage in different rounds of data processing. In this case, the GPU data over-issuance prefetching method provided in this embodiment can be used to overcome the problem of having to perform data loading serially due to insufficient GPR resources, thus folding the data prefetching delay.

[0046] The request version number can indicate the data fetch round for the same destination address. For example, the instruction stream for GPU data over-fetching is as follows:

[0047] SMP_LDM R0.v0, [R20.xy], t5 / / Read data from R20 and R21 of t5 and store it in R0, version number v0

[0048] SMP_LDM R0.v1, [R22.xy], t5 / / Read data from R22 and R23 of t5 and store it in R0, version number v1

[0049] CHK.LDM / / Data query signal, determines whether data should be written to GPR

[0050] EXE_M R32, R0 / / Perform matrix operations

[0051] SMP_LDM.rls / / Signal triggered by version number update

[0052] CHK.LDM

[0053] EXE_M R26, R0

[0054] SMP_LDM.rls

[0055] In the GPU's data over-fetch instruction stream described above, "SMP_LDM R0.v0, [R20.xy], t5" and "SMP_LDM R0.v1, [R22.xy], t5" are consecutively issued matrix data read instructions targeting the same destination address R0. The matrix data read instruction "SMP_LDM R0.v0, [R20.xy], t5" carries the request version number v0 for the destination address R0. The matrix data read instruction "SMP_LDM R0.v1, [R22.xy], t5" carries the request version number v1 for the destination address R0.

[0056] In the over-issued SMP_LDM instructions, the source operand coordinates of the first data read instruction are R20, R21, indicating that a matrix of a specific size will be read from the top-left corner specified by [R20, R21] in t sharp 5. The source operand coordinates of the second data read instruction are R22, R23, indicating that the matrix data to be read is located at the position specified by [R22, R23] in t sharp 5. Two different sets of matrix data can be read in the two data read instructions. The destination address of both over-issued data read instructions is R0. By adding a 1-bit version field, different rounds of access to R0 are indicated. R0.v0 can be used in the first data read instruction, and R0.v1 can be used in the second data read instruction.

[0057] In this embodiment of the invention, after two rounds of matrix data read requests are sent through the execution core via over-issuing SMP_LDM, the return latency of the two rounds of matrix data is folded. During data reading, the read latency from global memory to the data read / write unit is the most time-consuming part of the entire process. This embodiment of the invention, by over-issuing data read instructions, can overlap the time for multiple rounds of data reading into a single round of data reading time, thus reducing data read latency. In the above GPU data over-fetch instruction stream, taking the over-issuance of two data read instructions as an example, in practical applications, more data read instructions can be over-issued as needed, and are not limited to two.

[0058] Step 120: The sampling controller initiates a data reading request to the data reading and writing unit according to the data reading instruction.

[0059] When the sampling controller (SMP controller) sends a data read request to the data read / write unit, it can simultaneously send the request version number of the destination address.

[0060] Step 130: The data read / write unit retrieves the prefetched data from the global memory according to the data read request, marks it with the request version number, and stores the prefetched data.

[0061] The data reading unit can retrieve data from global memory sequentially according to data reading requests. After the global memory data is returned to the data reading unit, the data reading unit can mark the request version number of the data, such as whether the data belongs to the first round of data reading instructions or the second round of reading instructions.

[0062] Step 140: The data read / write unit determines the target request version number based on the synchronization flag of the GPU execution core, and transfers the prefetch data corresponding to the target request version number to the general-purpose register.

[0063] The synchronization flag of a GPU execution core represents the request version number of the data required for its computation. After each computation, the synchronization flag is updated to correspond to the request version number of the required data. For example, when the GPU first retrieves data from the destination address for computation, the synchronization flag corresponds to the first round of request version number. When the GPU completes its first computation on the data at the destination address, the synchronization flag is updated to correspond to the second round of request version number, allowing it to retrieve the second round of data from the destination address for computation, and so on.

[0064] The synchronization flag of the GPU execution core can be directly sent to the data read / write unit, or it can be sent to the data read / write unit via the sampling controller. The data read / write unit can determine the target request version number based on the synchronization flag and transfer the prefetch data corresponding to the target request version number to the general-purpose register.

[0065] The data read / write unit returns the corresponding version of the prefetched data to the general-purpose register according to the synchronization flag of the GPU execution core, which can ensure the accuracy and reliability of data in data operation and avoid errors in the operation result caused by returning prefetched data out of order.

[0066] This invention provides an improved method for updating the synchronization flag of a GPU execution core. Optionally, the method further includes: when the GPU execution core completes its current execution, it sends a version number update trigger signal to a sampling controller or a data read / write unit; the sampling controller or the data read / write unit updates the synchronization flag according to the version number update trigger signal; and the data read / write unit determines the target request version number based on the synchronization flag of the GPU execution core and transmits the prefetch data corresponding to the target request version number to a general-purpose register.

[0067] In the GPU data over-fetch instruction stream described above, "SMP_LDM.rls" represents the version number update trigger signal. The "SMP_LDM.rls" instruction can set a 1-bit trigger signal (release) field to update the synchronization flag (sync_flag) state. When the sampling controller or data read / write unit receives the update trigger signal, it can update the synchronization flag, such as from 0 to 1, from 1 to 0, incrementing by 1, or resetting it. After updating the synchronization flag, the data read / write unit determines the target request version number based on the GPU execution core's synchronization flag and transfers the prefetch data corresponding to the target request version number to a general-purpose register.

[0068] For example, the data reading unit queries sync_flag. If sync_flag is 1, it allows the prefetched data belonging to v1 returned from global memory to enter the general-purpose register of the execution core; otherwise, it allows the prefetched data of v0 to enter the general-purpose register.

[0069] Step 150: The GPU execution core reads prefetched data from the general-purpose registers for computation.

[0070] In the GPU data over-fetch instruction stream described above, "EXE_M R32, R0" and "EXE_M R26, R0" represent the first and second rounds of prefetch data calculation, respectively. The synchronization flag can have an initial default value corresponding to the request version number of the first round; after the GPU execution core executes "EXE_M R32, R0", the synchronization flag can be updated to correspond to the request version number of the second round. After the last execution, the synchronization flag can be restored to its initial default value. For example, in the GPU data over-fetch instruction stream described above, if only two data read instructions are over-fetched, the synchronization flag can be restored to its initial default value after the GPU completes the execution of "EXE_MR26, R0".

[0071] The technical solution of this invention involves the GPU execution core continuously issuing multiple data read instructions to the sampling controller for the same destination address. Each data read instruction carries a request version number for the destination address. The sampling controller initiates a data read request to the data read / write unit based on the data read instructions. The data read / write unit retrieves prefetched data from global memory based on the data read request, marks the prefetched data with the request version number, and stores the prefetched data. The data read / write unit determines the target request version number based on the synchronization flag of the GPU execution core and transfers the prefetched data corresponding to the target request version number to a general-purpose register. The GPU execution core reads the prefetched data from the general-purpose register for computation, thus solving the problem of GPU data prefetching. By simultaneously issuing multiple data read instructions for the same destination address, the delay of multiple data reads can be folded, improving the computing power of the execution core.

[0072] Based on the above implementation, optionally, the GPU execution core reads prefetched data from the general-purpose register for calculation, including: the GPU execution core sends a data query signal to the general-purpose register to query whether the prefetched data has been written into the general-purpose register; when the GPU execution core determines that the prefetched data has been written into the general-purpose register, it determines that the query is successful and reads the prefetched data from the general-purpose register for calculation.

[0073] In the GPU's data overfetch instruction stream described above, "CHK.LDM" represents the data query signal, used to check whether the matrix data has been successfully returned and written to the general-purpose register. The data query signal can be issued before each calculation by the GPU core to ensure that the data required for the operation is written to the general-purpose register. Multiple data query signals can exist in the GPU's data overfetch instruction stream.

[0074] Optionally, when the GPU execution core determines that the query has passed, the method further includes resetting the data query signal. By resetting the data query signal in a timely manner when the query passes, it is easier to perform data queries again by sending the data query signal to determine the data required for subsequent rounds to be written into the general-purpose register.

[0075] Optionally, after the data read / write unit transfers the prefetch data corresponding to the target request version number to the general-purpose register, it sends a data transfer completion signal to the general-purpose register; when the general-purpose register receives the data query signal sent by the GPU execution core, it sends a data transfer completion signal to the GPU execution core; the GPU execution core determines that the prefetch data is to be written to the general-purpose register based on the data transfer completion signal.

[0076] If the general-purpose register receives the data query signal from the GPU execution core but has not yet received the data transfer completion signal (req_end), instruction execution can pause on the CHK.LDM instruction, waiting for the return of prefetched data from the shared memory side. When the GPU execution core scheduler receives the req_end signal, it indicates that the prefetched data for the corresponding version number is ready. The execution core scheduler can then enable the pass flag required by CHK.LDM, and the execution flow can continue to execute the subsequent matrix operation instruction EXE_M R32, R0, while simultaneously pulling the pass flag of CHK.LDM low, i.e., resetting the data query signal.

[0077] Optionally, the number of synchronization flag bits is related to the number of threads in the GPU execution core; the data read / write unit determines the target request version number based on the synchronization flag of the GPU execution core, and transfers the prefetch data corresponding to the target request version number to the general-purpose register, including: the data read / write unit determines the target request version number of each thread based on the synchronization flag of each thread in the GPU execution core, and transfers the prefetch data corresponding to the target request version number to the general-purpose register; the GPU execution core reads the prefetch data in the general-purpose register for calculation, including: each thread of the GPU execution core reads the prefetch data corresponding to its thread in the general-purpose register for calculation.

[0078] For example, a multi-bit `sync_flag` can be created in the sampling controller or data read / write unit, with one bit corresponding to each thread. The `sync_flag` can be 32, 64, or 128 bits, and can initially be 0. The data read / write unit determines the target request version number for each thread based on the synchronization flags of the GPU execution core, and then transfers the prefetch data corresponding to the target request version number to a general-purpose register.

[0079] For example, when the sampling controller or data read / write unit receives "SMP_LDM.rls" from the GPU execution core, it can update the synchronization flag corresponding to that thread, such as flipping it from 0 to 1 or from 1 to 0. The data read / write unit queries the synchronization flag of the corresponding thread. If the synchronization flag of the corresponding thread is 1, it can allow prefetched data with version number v1 returned from global memory to enter the GPR; otherwise, it can allow prefetched data with version number v0 returned from global memory to enter the GPR. After the data read unit returns all the prefetched data of the corresponding version number to the GPR, it can send req_end to the GPR side, and the GPR sends red_end to the GPU execution core scheduler. In fact, the instruction execution of the current thread is currently stopped at the CHK.LDM instruction, that is, waiting for the prefetched data to return from the global memory side. When the execution core scheduler receives the `req_end` signal, it indicates that the prefetch data for the corresponding version number is ready. The execution core scheduler can then enable the pass flag required by `CHK.LDM`, and the execution flow can then proceed to execute subsequent matrix operation instructions `EXE_M R32, R0`, while simultaneously pulling the pass flag of `CHK.LDM` low. After the first round of matrix operation instructions is completed, the GPU execution core can issue `SMP_LDM.rls`. The sampling controller or data read / write unit toggles the bit in `sync_flag` corresponding to the current thread. Alternatively, a release bit can be added along with the version number bit (v0 or v1) corresponding to the data used in this round of matrix operations. The hardware can directly invert the version bit and assign it to the corresponding bit in the local `sync_flag`. In practical applications, the function of determining the target request version number can be implemented through synchronization flags. After the above synchronization flags are updated, the GPU execution core can issue `CHK.LDM` to wait for the preparation of the second round of matrix data. By over-fetching and reducing the latency of returning data from shared memory during multiple rounds of data prefetching, the second round of data can quickly reach the data reading unit from the global memory side. The data reading unit continues to query the target request version number corresponding to the synchronization flag, and allows the corresponding data to enter the GPR so that the GPU execution core can perform the second round of computation.

[0080] The above-described GPU data over-fetch instruction stream is an exemplary description of the technical solution of this invention, which involves two rounds of data computation. In practical applications, more rounds of data over-fetching can be performed. For example, more rounds of data prefetching instructions can be over-issued by changing the number of bits in the request version number and the synchronization flag in the exemplary description. Alternatively, more rounds of data prefetching instructions can be over-issued by repeatedly toggling the request version number and the synchronization flag.

[0081] Based on the above implementation, when the amount of data required in each round of computation is large and multiple destination addresses such as R0 to R15 are used to store the data, the following method can be used for processing. Optionally, when the GPU execution core issues multiple data read instructions with the same request version number but different destination addresses to the sampling controller, an unlocked flag is carried in the non-last data read instruction of the same request version number; after the data read / write unit transfers the prefetch data corresponding to the target request version number to the general-purpose register, it sends a data transfer completion signal to the general-purpose register, including: after the data read / write unit transfers the prefetch data corresponding to the last data read instruction of the target request version number to the general-purpose register, it sends a data transfer completion signal to the general-purpose register.

[0082] Among these, multiple data read instructions with the same request version number but different destination addresses refer to data read instructions that require multiple destination addresses for data storage when the current round requires a large amount of data. For example, a stream of multiple data read instructions with the same request version number but different destination addresses is as follows:

[0083] SMP_LDM.dnlk R0.v0, [R20.xy], t5

[0084] SMP_LDM R8.v0, [R22.xy], t5

[0085] SMP_LDM.dnlk R0.v1, [R24.xy], t5

[0086] SMP_LDM R8.v1, [R26.xy], t5

[0087] CHK.LDM

[0088] EXE_M R32, R0~R15

[0089] SMP_LDM.rls

[0090] CHK.LDM

[0091] EXE_M R28, R0~R15

[0092] SMP_LDM.rls

[0093] In this stream of multiple data read instructions with the same request version number but different destination addresses, “SMP_LDM.dnlk R0.v0, [R20.xy], t5” and “SMP_LDM R8.v0, [R22.xy], t5” represent two data read instructions for the same request version number v0 but different destination addresses. “SMP_LDM.dnlk R0.v1, [R24.xy], t5” and “SMP_LDM R8.v1, [R26.xy], t5” represent two data read instructions for the same request version number v1 but different destination addresses. Here, “dnlk” indicates no lock flag, specifically meaning that this instruction does not include a req_end signal of 1 when entering the data read unit. For the same request version number but different destination addresses, a data read instruction without “dnlk” at the end indicates that a req_end signal of 1 is included when entering the data read unit. In other words, `req_end` is appended to the end of all data in `SMP_LDM.dnlkR0.v0` and `SMP_LDM R8.v0 version 0` to ensure that all data for version number v0 is ready. The remaining instruction operations are the same as or similar to those explained above, and will not be repeated here.

[0094] The technical solution of this embodiment utilizes a data read / write unit to issue excess data read instructions even when the GPR register is insufficient. When insufficient registers prevent the over-issue request from returning data to the execution core, the returned data of the over-issue request resides in the data read / write unit. The execution core, sampling controller, and data read / write unit use version numbers and synchronization flags to determine whether data can be returned. Even with limited GPR resources, over-issuance of GPU data read instructions is still possible. By over-issuing requests with a request version number carrying the destination address during data prefetching, GPR is cleverly virtually reused under limited GPR resource conditions. Data required for multiple rounds of computation is issued back-to-back, and the return delay of multiple rounds of data is folded, greatly reducing the overall data loading time overhead. This significantly benefits the timely supply of data to the execution core and greatly improves the performance of modern GPUs or AI processors.

[0095] Example 2

[0096] Figure 2 This is a schematic diagram of the structure of a graphics processor according to Embodiment 2 of the present invention. Figure 2 As shown, the graphics processor includes: a GPU execution core, a sampling controller, a data read / write unit, general-purpose registers, and global memory; wherein,

[0097] The GPU execution core is used to continuously issue multiple data read instructions to the sampling controller for the same destination address. Each data read instruction carries the request version number of the destination address.

[0098] The sampling controller is used to initiate a data read request to the data read / write unit according to the data read instruction;

[0099] The data read / write unit is used to retrieve prefetched data from global memory according to data read requests, mark the prefetched data with the request version number, and store the prefetched data.

[0100] The data read / write unit is also used to determine the target request version number based on the synchronization flag of the GPU execution core, and to transfer the prefetch data corresponding to the target request version number to the general-purpose register;

[0101] The GPU execution core is also used to read prefetched data from general-purpose registers for computation.

[0102] Optionally, the GPU execution core is also used to send a version number update trigger signal to the sampling controller or data read / write unit when the current execution is completed;

[0103] The sampling controller or data read / write unit is also used to update the trigger signal and the synchronization flag according to the version number;

[0104] The data read / write unit is also used to return the target request version number based on the synchronization flag of the GPU execution core, and to transfer the prefetch data corresponding to the target request version number to the general-purpose register.

[0105] Optionally, the GPU execution core is specifically used to send a data query signal to the general-purpose register to query whether the prefetched data has been written into the general-purpose register; when it is determined that the prefetched data has been written into the general-purpose register, the query is confirmed to be successful, and the prefetched data is read from the general-purpose register for calculation.

[0106] Optionally, the GPU execution core is also used to reset the data query signal when the query is determined to be successful.

[0107] Optionally, the data read / write unit is also used to send a data transmission completion signal to the general-purpose register after transmitting the prefetch data corresponding to the target request version number to the general-purpose register;

[0108] The general-purpose register is also used to send a data transfer completion signal to the GPU execution core when a data query signal is received from the GPU execution core.

[0109] The GPU execution core is also used to determine the prefetch data to be written to the general-purpose register based on the data transmission completion signal.

[0110] Optionally, the GPU execution core is also used to carry an unlocked flag in the non-last data read instruction with the same request version number but different destination addresses when issuing multiple data read instructions to the sampling controller with the same request version number;

[0111] The data read / write unit is specifically used to send a data transfer completion signal to the general-purpose register after the prefetch data corresponding to the last bit of the target request version number read instruction is transferred to the general-purpose register.

[0112] Optionally, the number of bits for the synchronization flag is related to the number of threads in the GPU execution core;

[0113] The data read / write unit is specifically used to determine the target request version number of each thread based on the synchronization flags of each thread in the GPU execution core, and to transfer the prefetch data corresponding to the target request version number to the general-purpose register.

[0114] The GPU execution core consists of threads that read prefetched data corresponding to the thread from general-purpose registers for computation.

[0115] The graphics processor provided in the embodiments of the present invention can execute the GPU data over-fetching method provided in any embodiment of the present invention, and has the corresponding structure and beneficial effects of the execution method.

[0116] Example 3

[0117] Figure 3 This is a schematic diagram of an electronic device implementing the GPU data over-prefetching method according to embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and do not limit the implementation of the invention described and / or claimed herein.

[0118] like Figure 3As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) or random access memory (RAM), communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from the storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. Input / output (I / O) interfaces are also connected to the bus 14.

[0119] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0120] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the GPU's data over-fetching method.

[0121] In some embodiments, the GPU data overfetching method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the GPU data overfetching method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the GPU data overfetching method by any other suitable means (e.g., by means of firmware).

[0122] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0123] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0124] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0125] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0126] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0127] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0128] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0129] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for over-prefetching data in a GPU, characterized in that, The method is applied to a GPU, which includes: a GPU execution core, a sampling controller, a data read / write unit, general-purpose registers, and global memory; the method includes: The GPU execution core continuously issues multiple data read instructions to the sampling controller for the same destination address, wherein each data read instruction carries a request version number for the destination address; The sampling controller initiates a data read request to the data read / write unit according to the data read instruction; The data read / write unit retrieves prefetched data from global memory according to the data read request, marks the prefetched data with the request version number, and stores the prefetched data. The data read / write unit determines the target request version number based on the synchronization flag of the GPU execution core, and transmits the prefetch data corresponding to the target request version number to the general-purpose register; The GPU execution core reads prefetched data from general-purpose registers for computation.

2. The method according to claim 1, characterized in that, Also includes: When the GPU execution core completes its current execution, it sends a version number update trigger signal to the sampling controller or data read / write unit. The sampling controller or data read / write unit updates the trigger signal and the synchronization flag according to the version number; The return data read / write unit determines the target request version number based on the synchronization flag of the GPU execution core, and transmits the prefetch data corresponding to the target request version number to the general-purpose register.

3. The method according to claim 1, characterized in that, The GPU execution core reads prefetched data from general-purpose registers for computation, including: The GPU execution core sends a data query signal to the general-purpose register to check whether the prefetched data has been written to the general-purpose register. When the GPU execution core determines that prefetched data has been written to a general-purpose register, it confirms that the query has passed and reads the prefetched data from the general-purpose register for computation.

4. The method according to claim 3, characterized in that, When the GPU execution core determines that the query has passed, it also includes resetting the data query signal.

5. The method according to claim 3, characterized in that, After transmitting the prefetch data corresponding to the target request version number to the general-purpose register, the data read / write unit sends a data transmission completion signal to the general-purpose register. When the general-purpose register receives a data query signal from the GPU execution core, it sends a data transfer completion signal to the GPU execution core. The GPU execution core determines the amount of prefetched data to be written to a general-purpose register based on the data transmission completion signal.

6. The method according to claim 5, characterized in that, When the GPU executes multiple data read instructions with the same request version number but different destination addresses issued by the core to the sampling controller, an unlocked flag is carried in the non-last data read instruction with the same request version number. After the data read / write unit transmits the prefetch data corresponding to the target request version number to the general-purpose register, it sends a data transmission completion signal to the general-purpose register, including: After the data read / write unit transmits the prefetch data corresponding to the last bit of the target request version number read instruction to the general-purpose register, it sends a data transmission completion signal to the general-purpose register.

7. The method according to claim 1, characterized in that, The number of bits in the synchronization flag is related to the number of threads in the GPU execution core; The data read / write unit determines the target request version number based on the synchronization flag of the GPU execution core, and transfers the prefetch data corresponding to the target request version number to a general-purpose register, including: The data read / write unit determines the target request version number of each thread based on the synchronization flag of each thread of the GPU execution core, and transmits the prefetch data corresponding to the target request version number to the general-purpose register. The GPU execution core reads prefetched data from general-purpose registers for computation, including: Each thread of the GPU execution core reads prefetched data corresponding to its thread from general-purpose registers for computation.

8. A graphics processor, characterized in that, The graphics processor includes: a GPU execution core, a sampling controller, a data read / write unit, general-purpose registers, and global memory; wherein, The GPU execution core is used to continuously issue multiple data read instructions to the sampling controller for the same destination address, wherein each data read instruction carries a request version number of the destination address; A sampling controller is used to initiate a data read request to the data read / write unit according to the data read instruction; The data read / write unit is used to obtain prefetched data from global memory according to the data read request, mark the prefetched data with the request version number, and store the prefetched data. The data read / write unit is also used to determine the target request version number based on the synchronization flag of the GPU execution core, and to transfer the prefetch data corresponding to the target request version number to the general-purpose register; The GPU execution core is also used to read prefetched data from general-purpose registers for computation.

9. The graphics processor according to claim 8, characterized in that, The GPU execution core is also used to send a version number update trigger signal to the sampling controller or data read / write unit when the current execution is completed; The sampling controller or data read / write unit is also used to update the trigger signal and update the synchronization flag according to the version number; The data read / write unit is also used to return the target request version number based on the synchronization flag of the GPU execution core, and to transfer the prefetch data corresponding to the target request version number to the general-purpose register.

10. The graphics processor according to claim 8, characterized in that, The GPU execution core is specifically used to send a data query signal to the general-purpose register to check whether the prefetched data has been written into the general-purpose register. When it is determined that the prefetched data has been written into the general-purpose register, the query is confirmed to be successful, and the prefetched data is read from the general-purpose register for calculation.

11. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to said at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the GPU data over-prefetching method according to any one of claims 1-7.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause the graphics processor to execute the GPU data over-prefetching method according to any one of claims 1-7.

Images