A method and device for improving matrix multiplication calculation efficiency in GPU

Abstract

The present invention relates to the field of high-performance computing technology, and provides a method and device for improving the computational efficiency of matrix multiplication in a GPU. The present invention determines a storage unit from a first-level cache unit, a second-level cache unit, a third-level cache unit, and a memory in a first access order, and then obtains first input data to complete a first computing task; determines a storage unit from the third-level cache unit and the memory in a second access order, and then obtains second input data to complete a second computing task; the present invention can simultaneously process the first computing task and the second computing task in the same time period, and can share the third-level cache unit, reducing the delay of the two-level cache and the corresponding instruction scheduling time consumption, and can use a GPU instead of an AI chip, with strong universality, solving the problem that the AI ​​chip solution of the prior art is difficult to meet the rapidly growing computing power requirements of the AI ​​model.

Description

Technical Field

[0001] The present invention relates to the field of high performance computing technology, and in particular to a method and device for improving the efficiency of matrix multiplication calculation in a GPU. Background Art

[0002] With the continuous development of artificial intelligence (AI) technology, its application has penetrated into various fields such as autonomous driving, speech recognition and smart cities, and has gradually become the mainstream choice in the industry. In order to meet the efficient parallel computing requirements and low power consumption requirements of AI models, many AI chip manufacturers have proposed various optimization solutions.

[0003] The AI ​​chip solutions in the existing technology have great limitations and cannot meet the rapidly growing computing power requirements of AI models. On the one hand, many AI chips are customized chips, that is, they are designed for specific devices or systems. The hardware architecture of such AI chips is fixed and cannot adapt to other types of tasks or applications, and is insufficient to support the calculation of evolving AI models. Some AI chips contain a large number of computing cores inside, which can process a large number of tensor calculations in parallel, but they are often only suitable for computing scenarios with large matrices corresponding to tensors. When the matrix is ​​small, the effective utilization rate of matrix multiplication is too low and the practicality is poor; some AI chips are designed for smaller matrices, and the single calculation amount is too low for large matrices. On the other hand, in many AI chip solutions, the distance between the computing operator and the external storage unit is too large, resulting in a long overall computing delay process and low computing efficiency.

[0004] In view of this, overcoming the defects of the prior art is an urgent problem to be solved in the field of this technology. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a method and device for improving the efficiency of matrix multiplication calculations in GPUs. The purpose is to add an independent pipeline in the GPU architecture, which is used solely for matrix multiplication calculations. Then, the GPU can be used instead of the AI ​​chip to support the calculation of AI models, support calculations of various scales, and greatly reduce the delay and instruction scheduling time, reduce the distance between the calculation operator and the external storage unit, and have high calculation efficiency, thus solving the problem that the AI ​​chip solutions in the prior art are difficult to meet the rapidly growing computing power requirements of AI models.

[0006] The present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for improving the computational efficiency of matrix multiplication in a GPU, comprising:

[0008] Obtaining a first computing task and / or a second computing task;

[0009] When processing the first computing task, determining a storage unit from a first-level cache unit, a second-level cache unit, a third-level cache unit, and a memory in a first access order, obtaining first input data from the determined storage unit, and completing the first computing task according to the first input data;

[0010] When processing the second computing task, a storage unit is determined from the three-level cache unit and the memory according to a second access order, and second input data is obtained from the determined storage unit; and the second computing task is completed according to the second input data.

[0011] Further, when processing the second computing task, determining a storage unit from the three-level cache unit and the memory according to the second access order, and obtaining the second input data from the determined storage unit includes:

[0012] querying whether the third-level cache unit has the second input data according to the second virtual address carried by the second data request of the second computing task;

[0013] When the third-level cache unit has the second input data, acquiring the second input data to complete processing of the second data request;

[0014] When the second input data is not present in the third-level cache unit, the second virtual address is converted into a second physical address; and according to the second physical address, the second input data is read from the memory to complete the processing of the second data request.

[0015] Further, the second input data includes weight data and data to be processed;

[0016] The completing the second computing task according to the second input data comprises:

[0017] Acquire at least one piece of data to be processed and an operation mode of the second computing task;

[0018] Obtain at least one weight data;

[0019] Based on the operation rules of the operation mode, the acquired weight data and the corresponding data to be processed are calculated to obtain corresponding output data;

[0020] Return at least one output data obtained by calculation to complete the second calculation task.

[0021] Further, when processing the first computing task, determining a storage unit from a first-level cache unit, a second-level cache unit, a third-level cache unit, and a memory according to a first access order, obtaining first input data from the determined storage unit, and completing the first computing task according to the first input data includes:

[0022] querying whether the first input data is in the first-level cache unit according to the first virtual address carried by the first data request of the first computing task;

[0023] When the first-level cache unit has the first input data, the first input data is obtained to complete the processing of the first data request; when the first-level cache unit does not have the first input data, querying whether the second-level cache unit has the first input data according to the first virtual address;

[0024] When the second-level cache unit has the first input data, the first input data is obtained to complete the processing of the first data request; when the second-level cache unit does not have the first input data, querying whether the third-level cache unit has the first input data according to the first virtual address;

[0025] When the first input data is present in the third-level cache unit, the first input data is obtained to complete processing of the first data request; when the first input data is not present in the third-level cache unit, the first virtual address is converted into a first physical address; according to the first physical address, the first input data is read from the memory to complete processing of the first data request.

[0026] In a second aspect, the present invention further provides a device for improving the computational efficiency of matrix multiplication in a GPU, which is used to implement the method for improving the computational efficiency of matrix multiplication in a GPU described in the first aspect, wherein the device for improving the computational efficiency of matrix multiplication in a GPU comprises a firmware unit, a first task processor, a second task processor, a first level cache unit, a second level cache unit, a third level cache unit, and a memory, wherein:

[0027] The first task processor is used to obtain a first computing task from the firmware unit, determine a storage unit from a first-level cache unit, a second-level cache unit, a third-level cache unit and a memory according to a first access order, obtain first input data from the determined storage unit, and complete the first computing task according to the first input data;

[0028] The second task processor is used to obtain a second computing task from the firmware unit, determine a storage unit from the three-level cache unit and the memory according to a second access order, obtain second input data from the determined storage unit; and complete the second computing task according to the second input data.

[0029] Further, the first task processor includes a first task master controller and a memory management unit, wherein:

[0030] The first task master controller is used to query whether the first input data is in the first-level cache unit according to the first virtual address carried by the first data request of the first computing task;

[0031] The first-level cache unit is used to obtain the first input data when the first-level cache unit has the first input data, and send the obtained first input data to the first task master controller, so as to complete the processing of the first data request through the first task master controller; it is also used to query whether the second-level cache unit has the first input data according to the first virtual address when the first-level cache unit does not have the first input data; it is also used to receive the first input data from the second-level cache unit, so as to complete the processing of the first data request through the first task master controller;

[0032] The second-level cache unit is used to obtain the first input data when the second-level cache unit has the first input data, and send the obtained first input data to the first-level cache unit; it is also used to query whether the third-level cache unit has the first input data according to the first virtual address when the second-level cache unit does not have the first input data; it is also used to receive the first input data from the third-level cache unit, and send the received first input data to the first-level cache unit;

[0033] The third-level cache unit is used to obtain the first input data when the third-level cache unit has the first input data, and send the obtained first input data to the second-level cache unit; and is also used to read the first input data from the memory according to the first physical address, receive the first input data from the memory, and send the received first input data to the second-level cache unit when the third-level cache unit does not have the first input data;

[0034] The memory management unit is configured to convert the first virtual address into the first physical address.

[0035] Furthermore, the second task processor includes a second task master controller and a memory management unit, wherein:

[0036] The second task master controller is used to query whether the third-level cache unit has the second input data according to the second virtual address carried by the second data request of the second computing task;

[0037] The three-level cache unit is further used to obtain the second input data when the third-level cache unit has the second input data, and send the obtained second input data to the second task master controller, so as to complete the processing of the second data request through the second task master controller; and is also used to read the second input data from the memory according to the second physical address when the third-level cache unit does not have the second input data, and send the obtained second input data to the second task master controller, so as to complete the processing of the second data request through the second task master controller;

[0038] The memory management unit is used to convert the second virtual address into the second physical address.

[0039] Furthermore, the second task master controller includes a bridge, a built-in function module, a weight input cache unit, a data input cache unit, a calculation unit and an output cache unit, wherein:

[0040] The bridge is used to obtain at least one weight data and send the at least one weight data to the weight input cache unit;

[0041] The built-in function module is used to obtain at least one data to be processed, and send the at least one data to be processed to the data input cache unit; and is also used to receive at least one output data from the output cache unit, and write the output data to the memory to complete the second computing task;

[0042] The computing unit is used to obtain the operation mode of the second computing task; further used to obtain the at least one weight data from the weight input cache unit, and obtain the at least one to-be-processed data from the data input cache unit; further used to calculate the obtained weight data and the corresponding to-be-processed data based on the operation rules of the operation mode, obtain corresponding output data, and send the output data to the output cache unit;

[0043] The output buffer unit is used to send the output data to the built-in function module.

[0044] Furthermore, the second task master controller further includes a scheduler and a request manager, wherein:

[0045] The scheduler is configured to send a third data request to the request manager;

[0046] The request manager is used to forward the third data request to the bridge when the data required by the third data request belongs to weight data; and is also used to forward the third data request to the built-in function module when the data required by the third data request belongs to data to be processed;

[0047] The bridge is further used to send a weight data read request to the third-level cache unit, so as to complete the response to the third data request through the weight data read request;

[0048] The built-in function module is further used to send a pending data read request to the three-level cache unit, so as to complete the response to the third data request through the pending data read request;

[0049] The three-level cache unit is also used to return the weight data required by the third data request according to the weight data read request; and is also used to return the pending data required by the third data request according to the pending data read request.

[0050] The device for improving the computational efficiency of matrix multiplication in the GPU also includes:

[0051] At least one processor; and a storage unit communicatively connected to the at least one processor; wherein the storage unit stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the method for improving the computational efficiency of matrix multiplication in a GPU as described in the first aspect.

[0052] In a third aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, which are executed by one or more processors to complete the method for improving the computational efficiency of matrix multiplication in a GPU as described in the first aspect.

[0053] In a fourth aspect, a chip is provided, comprising: a processor and an interface, for calling and running a computer program stored in a storage unit from a storage unit, and executing the method for improving the computational efficiency of matrix multiplication in a GPU as in the first aspect.

[0054] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer or a processor, enables the computer or the processor to execute a method for improving the computational efficiency of matrix multiplication in a GPU as described in the first to fourth aspects and any one of the aspects.

[0055] In the sixth aspect, a method system for improving the computational efficiency of matrix multiplication in a GPU is provided, comprising an apparatus for improving the computational efficiency of matrix multiplication in a GPU as in the second aspect, and using the method for improving the computational efficiency of matrix multiplication in a GPU as described in the first aspect to complete the interaction of the apparatus for improving the computational efficiency of matrix multiplication in a GPU as in the second aspect.

[0056] Different from the prior art, the present invention has at least the following beneficial effects:

[0057] The GPU architecture of the present invention supports various types of calculations including matrix multiplication, determines a storage unit from a first-level cache unit, a second-level cache unit, a third-level cache unit and a memory according to a first access order, obtains corresponding first input data from the determined storage unit, and completes a first calculation task according to the first input data; by adding an independent matrix multiplication / vector multiplication main controller in the GPU architecture, it is realized that a storage unit is determined from a third-level cache unit and a memory according to a second access order, obtains corresponding second input data from the determined storage unit, and completes the second calculation task according to the second input data; wherein, the present invention can simultaneously process the first calculation task and the second calculation task in the same time period, and can share the third-level cache unit, thereby improving the calculation efficiency and maximizing the use of the calculation resources; for the pipeline for processing the second calculation task, since the delay of the second-level cache and the corresponding instruction scheduling time are reduced, the overall calculation time is greatly reduced, the power consumption is reduced, and the GPU can be used instead of the AI ​​chip; since a pipeline solely for the second calculation task is provided, the data format in the matrix multiplication / vector multiplication main controller can be set separately, and thus various AI models can be adapted, and the universality is strong. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0059] Figure 1 It is a flowchart of a method for improving the computational efficiency of matrix multiplication in a GPU provided by an embodiment of the present invention;

[0060] Figure 2 1 is a schematic diagram of the internal structure of a device for improving the computational efficiency of matrix multiplication in a GPU according to an embodiment of the present invention;

[0061] Figure 3 is a schematic diagram of the internal structure of a GPU in the prior art provided by an embodiment of the present invention;

[0062] Figure 4 is a flow chart of step 20 provided in an embodiment of the present invention;

[0063] Figure 5 is a flow chart of step 30 provided in an embodiment of the present invention;

[0064] Figure 6 is another flowchart of step 30 provided in an embodiment of the present invention;

[0065] Figure 7 is a schematic diagram of the internal structure of a second task master controller provided by an embodiment of the present invention;

[0066] Figure 8 is a schematic diagram of a specific example of completing the second computing task provided by an embodiment of the present invention;

[0067] Fig. 9 It is a schematic diagram of the architecture of a device for improving the computational efficiency of matrix multiplication in a GPU provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0069] In the description of the present invention, the terms "inside", "outside", "longitudinal", "lateral", "upper", "lower", "top", "bottom" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and do not require that the present invention must be constructed and operated in a specific orientation. Therefore, they should not be understood as limitations on the present invention.

[0070] The terms "first", "second", etc. in the present invention are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0071] In this application, unless otherwise specified or limited, the term "connection" should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection or an indirect connection through an intermediate medium. In addition, the term "coupling" can be a way of achieving electrical connection for signal transmission.

[0072] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0073] The calculation of AI models often includes a large number of matrix-vector multiplication operations and matrix-matrix multiplication operations. For example, in deep learning and machine learning algorithms, matrix multiplication is a basic and frequently occurring operation, and many AI models (such as neural networks) involve a large number of matrix multiplication operations to perform tasks such as forward and backward propagation.

[0074] At present, various AI models and various industrial application scenarios are experiencing explosive growth. In the existing technology, AI chips are generally used to support the calculation of various AI models. Among them, the Graphics Processing Unit (GPU) can also be used to accelerate AI calculations, but it is designed for graphics processing tasks, so the degree of optimization in AI model calculations is often not as good as that of AI chips. AI chips are specially optimized for different computing tasks or AI models, and can usually achieve lower power consumption while maintaining high performance, which can significantly reduce energy consumption and operating costs; although GPUs have powerful performance, their power consumption is usually high.

[0075] However, AI models iterate and change very quickly, and there are many variants of transcendental functions. Traditional AI chip solutions are often targeted at certain types of applications or systems and have great limitations.

[0076] Some AI chip solutions are hard-core matrix multiplication, that is, in the hardware design of the AI ​​chip, the matrix multiplication operation is implemented as a fixed, hard-coded function, and the matrix multiplication operation is directly embedded in the hardware design, and the operation is optimized in a targeted manner to significantly improve the performance of the matrix multiplication and improve the overall energy efficiency of the AI ​​chip. However, since the matrix multiplication operation is hard-coded into the hardware, it cannot adapt to all types of AI models. When the AI ​​model is changed or expanded, the hardware has to be redesigned or other programmable hardware platforms have to be selected.

[0077] The specific problems of hard-core matrix multiplication are as follows:

[0078] (1) Due to the popularity of AI technologies such as large models, AI chips have higher computing power requirements and need to be flexible for calculations of various sizes. For example, in some customized AI chip solutions, AI chips can support 256*256 matrix multiplication, but when the matrix is ​​very small, the 256*256 customized design will make the effective utilization rate of matrix multiplication very low; some customized AI chip solutions use 16*1 vector multiplication, but for large matrices, the single calculation amount is too low and the practicality is poor.

[0079] (2) All AI chip architectures used to process tensor operations require importing data from an external double data rate synchronous dynamic random-access memory (DDR) into the L2 cache, from the L2 cache to the L1 cache, and from the L1 cache to the L0 cache before entering the computing unit where the tensor to be calculated is located and performing the corresponding calculation. Moreover, under this architecture, instruction scheduling takes too long, so the overall computing delay process of this architecture is often too long.

[0080] In order to solve the above problems, Figure 1 As shown, an embodiment of the present invention provides a method for improving the efficiency of matrix multiplication calculation in a GPU, comprising:

[0081] Step 10: Obtain the first computing task and / or the second computing task.

[0082] The embodiment of the present invention uses a GPU to replace an AI chip to complete the first computing task and / or the second computing task. The first computing task is a computing task that a GPU is suitable for completing in a general scenario, and the second computing task is a computing task that a traditional AI chip is suitable for completing; the first computing task includes a three-dimensional rendering task and a general data processing task, and the second computing task includes a matrix-vector multiplication task and a matrix-matrix multiplication task.

[0083] Step 20: When processing the first computing task, determine the storage unit from the first-level cache unit, the second-level cache unit, the third-level cache unit and the memory according to the first access order, obtain the first input data from the determined storage unit, and complete the first computing task according to the first input data.

[0084] The first input data is the input data required by the first computing task; when the first computing task requests the first input data from the first-level cache unit, all data requests carry the address of the first input data. The first-level cache unit may be an L0 cache (i.e., cache L0), the second-level cache unit may be an L1 cache (i.e., cache L1), the third-level cache unit may be an L2 cache (i.e., cache L2), and the memory may be DDR.

[0085] It should be noted that the first-level cache unit, the second-level cache unit and the third-level cache unit are all caches with built-in processing logic (the cache is a high-speed cache storage unit), and the relevant built-in processing logic is: in order to implement the method of improving the calculation efficiency of matrix multiplication in the GPU in the embodiment of the present invention, additional design and optimization are implemented in the cache; the built-in processing logic is selected and implemented by technical personnel in this field according to the specific usage scenario and in accordance with the embodiment of the present invention.

[0086] like Figure 2 As shown, the GPU architecture of the embodiment of the present invention includes two different first task master controllers, and the pipeline for processing the 3D rendering task is "firmware unit-first task master controller-shader-level cache unit-level cache unit-level cache unit-level cache unit-level cache unit-memory", wherein the firmware unit is firmware, and the first task master controller is Figure 2 The first task controller on the left side is shown as processing the pipeline of the 3D rendering task as “firmware-rendering data controller-shader-L0 cache-L1 cache-L2 cache-DDR”; the pipeline for processing general data processing tasks is “firmware unit-first task controller-shader-level cache unit-level cache unit-level cache unit-level 3 cache unit-memory”, where the first task controller refers to Figure 2 The first task master in the middle shown, that is, the pipeline for processing general data processing tasks is "firmware-compute data master-shader-L0 cache-L1 cache-L2 cache-DDR", and the architecture will be described in detail below. Among them, the render data master (render Data Master, abbreviated as renderDM) is responsible for receiving three-dimensional rendering tasks from the host central processing unit (Host CentralProcessing Unit, abbreviated as Host CPU), such as rendering and texture mapping of 3D scenes, etc. RenderDM ensures that these three-dimensional rendering tasks are executed efficiently and accurately on the GPU to produce high-quality image output. The compute data master (Compute Data Master, abbreviated as CDM) is responsible for obtaining general data processing tasks from the Host CPU. CDM splits these general data processing tasks into multiple subtasks for further processing and distribution later.

[0087] Step 30: When processing the second computing task, determine a storage unit from the three-level cache unit and the memory according to a second access order, obtain second input data from the determined storage unit; and complete the second computing task based on the second input data.

[0088] Among them, the second computing task is a matrix-vector multiplication task and a matrix-matrix multiplication task; the second input data is the input data required to be requested by the second computing task; when the second computing task requests the second input data from the first-level cache unit, all data requests carry the address of the second input data.

[0089] like Figure 2 and Figure 3As shown, in order to support calculations of various scales and reduce latency and instruction scheduling to a large extent, the embodiment of the present invention uses GPU instead of AI chip to support the calculation of AI model. An independent second task master controller is added to the GPU architecture to achieve the addition of an independent pipeline, that is, Figure 2 The "firmware unit-second task master-third level cache unit-memory" shown is used to complete matrix-vector multiplication tasks and matrix-matrix multiplication tasks separately to make full use of external bandwidth, which will be described in detail below. Among them, the second task master is the matrix multiplication / vector multiplication master (Matmul / Vector Data Master, abbreviated as MVDM), and the added independent pipeline is "firmware-MVDM-L2 cache-DDR".

[0090] It is worth noting that, unlike the prior art, in the embodiment of the present invention, the pipeline for processing the first computing task and the pipeline for processing the second computing task are independent of each other and can be used at the same time; for example, when the pipeline "firmware-MVDM-L2 cache-DDR" is used to perform a matrix-vector multiplication task, the pipeline "firmware-rendering data master-shader-L0 cache-L1 cache-L2 cache-DDR" can be used in parallel to perform a three-dimensional rendering task. The embodiment of the present invention can use the pipeline for processing the first computing task and the pipeline for processing the second computing task at the same time, which can more effectively utilize system resources and greatly break through the computing efficiency bottleneck of the prior art when processing different tasks at the same time.

[0091] The GPU architecture of the present invention supports various types of calculations including matrix multiplication, determines a storage unit from a first-level cache unit, a second-level cache unit, a third-level cache unit and a memory according to a first access order, obtains corresponding first input data from the determined storage unit, and completes a first calculation task according to the first input data; by adding an independent matrix multiplication / vector multiplication main controller in the GPU architecture, it is realized that a storage unit is determined from a third-level cache unit and a memory according to a second access order, obtains corresponding second input data from the determined storage unit, and completes the second calculation task according to the second input data; wherein, the present invention can simultaneously process the first calculation task and the second calculation task in the same time period, and can share the third-level cache unit, thereby improving the calculation efficiency and maximizing the use of the calculation resources; for the pipeline for processing the second calculation task, since the delay of the second-level cache and the corresponding instruction scheduling time are reduced, the overall calculation time is greatly reduced, the power consumption is reduced, and the GPU can be used instead of the AI ​​chip; since a pipeline solely for the second calculation task is provided, the data format in the matrix multiplication / vector multiplication main controller can be set separately, and thus various AI models can be adapted, and the universality is strong.

[0092] Below Figure 2The various unit modules in the GPU architecture shown are described as follows:

[0093] Firmware is software embedded in hardware to control the basic operation of the hardware; shader is an embedded program used to process specific stages in graphics rendering; the first-level cache unit is the cache level closest to the processor core and has the fastest access speed, and the distance between the second-level cache unit and the third-level cache unit and the processor core increases successively. The first-level cache unit, the second-level cache unit and the third-level cache unit are cascaded, and they are all high-speed cache memories. The memory management unit (MMU for short) is mainly responsible for the conversion between virtual addresses and physical addresses, and provides memory access authorization for hardware mechanisms. In the process of executing three-dimensional rendering tasks and general data processing tasks in the embodiments of the present invention, DDR is usually used as video memory to store resources such as graphics data; in the process of executing matrix-vector multiplication tasks and matrix-matrix multiplication tasks in the embodiments of the present invention, DDR is used to store resources such as tensors to be processed.

[0094] The following is based on Figure 2 The GPU architecture shown in the figure illustrates the method of obtaining the first input data in the process of executing the 3D rendering task and the general data processing task in the embodiment of the present invention:

[0095] like Figure 4 As shown, the step 20 includes:

[0096] Step 201: According to the first virtual address carried by the first data request of the first computing task, query whether the first input data exists in the first-level cache unit.

[0097] In the GPU architecture of the embodiment of the present invention, the firmware first obtains the three-dimensional rendering tasks and general data processing tasks from the host CPU. These first computing tasks will call the shader to execute the corresponding instructions. When the shader needs the first input data, it will query whether the first input data is available from the first-level cache unit (i.e., L0 cache). It should be noted that all data requests (such as requests for first data) are marked with the virtual address and corresponding length of the requested data. The shader can only obtain the first input data from the first-level cache unit for the calculation of the subsequent first computing task.

[0098] like Figure 2As shown, for example, when the shader needs the first input data, it will send a first data request to the L0 cache, and the first data request contains the address and length of the required first input data. According to the first virtual address in the first data request, the L0 cache will search its internal storage to check whether there is the required first input data (that is, to find out whether there is data with the same address). If so, the L0 cache will return the queried first input data; if not, the L0 cache will send a request for querying the first input data to the L1 cache.

[0099] Step 202: When the first-level cache unit has the first input data, obtain the first input data to complete the processing of the first data request; when the first-level cache unit does not have the first input data, query whether the second-level cache unit has the first input data according to the first virtual address.

[0100] For example, when there is first input data in the L0 cache, the shader can obtain the first input data from the L0 cache to complete the first computing task; when there is no first input data in the L0 cache, the L0 cache applies for the first input data from the L1 cache according to the first virtual address.

[0101] Step 203: When the second-level cache unit has the first input data, obtain the first input data to complete the processing of the first data request; when the second-level cache unit does not have the first input data, query whether the third-level cache unit has the first input data according to the first virtual address.

[0102] For example, when there is the first input data in the L1 cache, the L1 cache will send the obtained first input data to the L0 cache, and the shader can obtain the first input data from the L0 cache to complete the first computing task; when there is no first input data in the L1 cache, the L1 cache will apply for the first input data from the L2 cache according to the first virtual address.

[0103] Step 204: When the first input data is present in the third-level cache unit, the first input data is obtained to complete the processing of the first data request; when the first input data is not present in the third-level cache unit, the first virtual address is converted into a first physical address; according to the first physical address, the first input data is read from the memory to complete the processing of the first data request.

[0104] For example, when there is first input data in the L2 cache, the L2 cache sends the obtained first input data to the L1 cache, and the L1 cache sends the received first input data to the L0 cache. The shader can obtain the first input data from the L0 cache to complete the first computing task.

[0105] When the first input data is not in the L2 cache, the L2 cache requests the first input data from the DDR. Since the GPU uses virtual addresses to ensure address continuity, and the DDR uses actual physical addresses, when data is to be read from the DDR, the L2 cache uses the MMU page table to convert the first virtual address into the corresponding physical address, and the L2 cache requests the first input data from the DDR according to the physical address, and returns it to the shader step by step to complete the first computing task.

[0106] The following is based on Figure 2 The GPU architecture shown in FIG. 1 illustrates a method for obtaining second input data in the process of executing a matrix-vector multiplication task and a matrix-matrix multiplication task in an embodiment of the present invention:

[0107] like Figure 5 As shown, in step 30, when the task to be processed is a second computing task, determining a storage unit from which second input data of the second computing task can be obtained according to a second access order of the three-level cache unit and the memory, and obtaining the second input data from the determined storage unit includes:

[0108] Step 301a: According to the second virtual address carried by the second data request of the second computing task, query whether the third-level cache unit has the second input data.

[0109] In the GPU architecture of the embodiment of the present invention, the firmware first obtains the matrix-vector multiplication task and the matrix-matrix multiplication task from the host CPU. These second computing tasks will call the second task master controller to execute the corresponding instructions; wherein the second task master controller can run independently of the computing data master controller and the rendering data master controller, so it can run in parallel with the two when running independently. When the second task master controller needs the second input data, it will query whether the second input data is available from the third-level cache unit (i.e., L2 cache). The second task master controller can only obtain the second input data from the third-level cache unit for the calculation of the subsequent second computing task.

[0110] Step 302a: When the third-level cache unit has the second input data, the second input data is acquired to complete the processing of the second data request.

[0111] For example, when there is second input data in the L2 cache, the second task master controller can obtain the second input data to complete the second computing task.

[0112] Step 303a: When the third-level cache unit does not have the second input data, convert the second virtual address into a second physical address; and read the second input data from the memory according to the second physical address to complete the processing of the second data request.

[0113] For example, when there is no second input data in the L2 cache, the L2 cache applies to the DDR for the second input data. Similarly, when data is to be read from the DDR (i.e., memory), the L2 cache uses the MMU page table (i.e., memory management unit) to convert the second virtual address into a corresponding physical address, and the L2 cache applies to the DDR for the second input data according to the physical address, and returns the applied second input data to the MVDM (i.e., the second task master) to complete the second computing task.

[0114] The following is a description of the process of completing the matrix-vector multiplication task and the matrix-matrix multiplication task after obtaining the second input data in an embodiment of the present invention:

[0115] like Figure 6 As shown, in step 30, the second input data includes weight data and data to be processed;

[0116] The completing the second computing task according to the second input data comprises:

[0117] Step 301b: Obtain at least one data to be processed and an operation mode of the second computing task.

[0118] The operation mode is an identifier of a specific operation rule of a matrix-vector multiplication task or a matrix-matrix multiplication task. In an optional embodiment, 1-bit data may be used to notify the MVDM of the specific operation rule to be executed for the current second computing task.

[0119] For example, Figure 7 As shown, at this time, the second task master controller needs to perform a mapping calculation of a linear layer in the AI ​​model, and the data to be processed is data 1 input into the linear layer through the data input cache unit; so it is necessary to obtain data 1, and the acquisition operation mode is matrix-matrix multiplication.

[0120] Step 302b: Obtain at least one weight data.

[0121] For example, the current weight data obtained by the linear layer after learning is weight 1, so weight 1 needs to be obtained through the weight input cache unit.

[0122] Step 303b: Based on the operation rules of the operation mode, the acquired weight data and the corresponding data to be processed are calculated to obtain corresponding output data.

[0123] The operation rules are selected by technicians in this field according to actual usage scenarios and are not limited here.

[0124] For example, the operation rule expression of the mapping calculation of the linear layer is as follows:

[0125]

[0126] in," " represents matrix multiplication operation, Z is the output data, W is the weight data, and X is the data to be processed; Figure 7 As shown, the operation result of weight 1 and data 1 is output 1.

[0127] Step 304b: Return at least one output data obtained by calculation to complete the second calculation task.

[0128] For example, output 1 is returned and written back to the memory through the built-in function module for use in the mapping calculation of the linear layer in the next round of iteration.

[0129] It is worth noting that the first computing task and the second computing task share the third-level cache unit (i.e., L2 cache). In the L2 cache, if the data is not immediately sent or written to its target location after the previous computing task is processed, there is no need to apply to the external storage unit (such as DDR) to read the corresponding data. The data can be directly read from the third-level cache unit, avoiding cache consistency issues and address remapping. Since the embodiment of the present invention uses a GPU instead of an AI chip and executes the first computing task and the second computing task in parallel, it can support various types of GPU computing.

[0130] Since MVDM is directly connected to the L2 cache, and the L2 cache is the fastest module that can obtain corresponding data from DDR, and in the calculation of the AI ​​model, each round of calculation will use the results of the previous round of calculations, so when a certain round of calculations is required, MVDM can efficiently obtain the results of the previous round of calculations; wherein, the results of the previous round of calculations are generally stored in the L2 single-level cell (Single Level Cell, abbreviated as SLC). Ordinary first input data needs to pass through the pipeline from L2 cache to L1 cache, and L1 cache to L0 cache before corresponding operations can be performed in the shader. Each additional level of cache unit has an additional level of delay, which leads to a decrease in computing performance. Therefore, compared with other AI chip solutions, the independent pipeline used to implement matrix multiplication operations in the present invention reduces the delay of the two-level cache and the time it takes for the shader to schedule the two-level cache.

[0131] In order to better illustrate the use process of the second task master controller in the embodiment of the present invention, the embodiment of the present invention provides an example of steps for completing the second computing task, which are as follows:

[0132] Step S1: get batch,height,width,format_length

[0133] if batch>1

[0134] mode=mm

[0135] else

[0136] mode=mv

[0137] n_batch=batch / 16;

[0138] n_height=height*format_length / 256

[0139] n_width =width / 16;

[0140] i_batch=0;

[0141] i_height=0;

[0142] i_width=0;

[0143] Step S2: send weight addr request

[0144] Step S3: send data addr request

[0145] Step S4: check weight and data ready

[0146] Step S5: ready to formulate

[0147] Step S6: finish i_batch

[0148] Step S7: finish iLheight

[0149] Step S8: out data

[0150] Step S9: finish i_width

[0151] Step S10: stop

[0152] In order to better illustrate the specific implementation process of the above step examples, the embodiment of the present invention also provides four specific implementation examples for completing the second computing task, which are as follows:

[0153]

[0154] (1) If Figure 8 As shown, the embodiment of the present invention provides a specific example of completing the matrix-vector multiplication task (A[1,4096] * B[4096, 4096]).

[0155] Step A1. i=0, batch = 1, mode =mv, data mode share; inter = 64;

[0156] number = 256bit / 4bit = 64

[0157] input = 64*32 = 2048bit = 8*256bit

[0158] Among them, step A1 is used to control different parameters, A is the input data to be processed (ie, input), and its format is 32 bits; B is the input weight data (ie, weight), and its format is 4 bits.

[0159] Since bandwidth is the most precious in the system, the embodiment of the present invention fully utilizes 256-bit bandwidth as much as possible, so each beat can process 256 / 4 (bit) = 64 numbers (ie, number). The matching input is 32bit*64 numbers, 32bit*64=2048bit=8*256bit, and 8 beats can be passed into the computing unit.

[0160] Step A2. send weight data request [i+1:i+64,1:16]

[0161] Among them, Figure 8 As shown, step A2 is used to obtain weight data from top to bottom and from left to right; in each calculation, every 16 rows will be obtained, and then the next 16 rows will be obtained.

[0162] Step A3. send data request [1, i+1:i+64]

[0163] Among them, step A3 is used to obtain the data to be processed corresponding to the weight data.

[0164] Step A4. input and weight all return

[0165] Among them, step A4 is used to ensure that the data required for calculation is in place.

[0166] Step A5. formulate:mv each alu data get same data

[0167] Among them, step A5 is used to send the data to be processed and the corresponding weight data to each computing unit under the operation mode of the matrix-vector multiplication task.

[0168] Step A6. save middle value 16 bit16 = 256bit

[0169] i = i + inter;

[0170] if i <input width

[0171] Step A2

[0172] else

[0173] Step A7

[0174] Among them, when it is necessary to calculate the multiplication and addition of 4096 numbers, but only the multiplication and addition of 64 numbers are calculated each time, multiple rounds of calculation are required; except for the last round of calculation, the intermediate calculation results need to be temporarily stored when performing other rounds of calculation.

[0175] Step A7. send result to SLC 256bit need one cycle

[0176] After all the calculations are completed, the output data will be written out. When writing the output data, the embodiment of the present invention will also use all 256 bits as much as possible.

[0177] (2) Step B1. i = 0, batch = 1, mode = mm, data mode each; inter = 64;

[0178] number = 256bit / 4bit = 64

[0179] input = 64*32*16 = 2048*16bit = 16*8*256bit

[0180] Step B2. send weight data request [i+1:i+64,1:16]

[0181] Step B3. send data request [1:16, i+1:i+64]

[0182] Step B4. input and weight all return

[0183] Step B5. formulate:mv each alu data get same data

[0184] Step B6. save middle value 16 bit16 = 256bit

[0185] i = i + inter;

[0186] if i <input width

[0187] Step B2

[0188] else

[0189] Step B7

[0190] Step B7 send result to SLC 256bit need one cycle

[0191] (3) Step C1. i = 0, batch = 1, mode = mv, data mode share; inter = 16;

[0192] number = 256bit / 16bit = 16

[0193] input = 16*32 = 512bit = 2*256bit

[0194] Step C2. send weight data request [i+1:i+16,1:16]

[0195] Step C3. send data request [1, i+1:i+16]

[0196] Step C4. input and weight all return

[0197] Step C5. formulate

[0198] Step C6. save middle value 16 bit16 = 256bit

[0199] i = i + inter;

[0200] if i <input width

[0201] Step C2

[0202] else

[0203] Step C7

[0204] Step C7. send result to SLC 256bit need one cycle

[0205] (4) Step D1. i=0, batch = 1, mode =mv, data mode share; inter = 32;

[0206] number = 256bit / 8bit = 32

[0207] input = 16*32 = 512bit = 2*256bit

[0208] Step D2. send weight data request [i+1:i+32,1:16]

[0209] Step D3. send data request [1, i+1:i+32]

[0210] Step D4. input and weight all return

[0211] Step D5. formulate

[0212] Step D6. save middle value 16 bit16 = 256bit

[0213] i = i + inter;

[0214] if i <input width

[0215] Step D2

[0216] else

[0217] Step D7

[0218] Step D7. send result to SLC 256bit need one cycle

[0219] The above specific implementation examples demonstrate that the second task master controller of an embodiment of the present invention can flexibly and dynamically adjust its internal execution algorithm according to the format length of the input data (such as weight data, data to be processed, etc.), and the embodiment of the present invention, as a universal matrix-vector multiplication solution, can adapt to various AI models. All AI models contain a large number of matrix multiplication operations, matrix-vector multiplication operations and convolution operations, especially the currently popular large language models that require high computing power; for example, the large language model llama2 7B mainly involves matrix multiplication operations in the form of "A[4096,4096]*B[4096,4096]" and matrix-vector multiplication operations in the form of "A[1,4096]*B[4096,4096]"; the large language model llama2 70B mainly involves matrix multiplication operations in the form of "A[8192,8192]*B[8192,8192]" and matrix-vector multiplication operations in the form of "A[1,8192]*B[8192,8192]"; the solution of the embodiment of the present invention can effectively solve the computing power requirement problem of large language models.

[0220] like Figure 2 As shown, the present invention also provides a device for improving the calculation efficiency of matrix multiplication in a GPU, which is used to implement the method for improving the calculation efficiency of matrix multiplication in a GPU described in an embodiment of the present invention. The device for improving the calculation efficiency of matrix multiplication in a GPU includes a firmware unit, a first task processor, a second task processor, a first level cache unit, a second level cache unit, a third level cache unit and a memory, wherein:

[0221] The first task processor is used to obtain the first computing task from the firmware unit, determine the storage unit from the first-level cache unit, the second-level cache unit, the third-level cache unit and the memory according to the first access order, obtain the first input data from the determined storage unit, and complete the first computing task according to the first input data.

[0222] The second task processor is used to obtain a second computing task from the firmware unit, determine a storage unit from the three-level cache unit and the memory according to a second access order, obtain second input data from the determined storage unit; and complete the second computing task according to the second input data.

[0223] The second task processor may be an MVDM. The first computing unit and the second computing unit are independent of each other and can be used simultaneously to more effectively utilize system resources.

[0224] The following specifically describes the process of reading the first input data when completing the three-dimensional rendering task and the general data processing task. The first task processor includes a first task main controller and a memory management unit, wherein:

[0225] The first task master controller is used to query whether the first input data exists in the first-level cache unit according to the first virtual address carried by the first data request of the first computing task.

[0226] Among them, the first task master controller can be CDM or renderDM.

[0227] The first-level cache unit is used to obtain the first input data when the first-level cache unit has the first input data, and send the obtained first input data to the first task master controller, so as to complete the processing of the first data request through the first task master controller; it is also used to query whether the second-level cache unit has the first input data according to the first virtual address when the first-level cache unit does not have the first input data; it is also used to receive the first input data from the second-level cache unit, so as to complete the processing of the first data request through the first task master controller;

[0228] The second-level cache unit is used to obtain the first input data when the second-level cache unit has the first input data, and send the obtained first input data to the first-level cache unit; it is also used to query whether the third-level cache unit has the first input data according to the first virtual address when the second-level cache unit does not have the first input data; it is also used to receive the first input data from the third-level cache unit, and send the received first input data to the first-level cache unit;

[0229] The third-level cache unit is used to obtain the first input data when the third-level cache unit has the first input data, and send the obtained first input data to the second-level cache unit; and is also used to read the first input data from the memory according to the first physical address, receive the first input data from the memory, and send the received first input data to the second-level cache unit when the third-level cache unit does not have the first input data;

[0230] The memory management unit is configured to convert the first virtual address into the first physical address.

[0231] The following specifically describes the process of reading the second input data when completing a matrix-vector multiplication task or a matrix-matrix multiplication task, wherein the second task processor includes a second task master controller and a memory management unit, wherein:

[0232] The second task master controller is used to query whether the third-level cache unit has the second input data according to the second virtual address carried by the second data request of the second computing task.

[0233] The second task master controller is MVDM.

[0234] The three-level cache unit is further used to obtain the second input data when the third-level cache unit has the second input data, and send the obtained second input data to the second task master controller, so as to complete the processing of the second data request through the second task master controller; and is also used to read the second input data from the memory according to the second physical address when the third-level cache unit does not have the second input data, and send the obtained second input data to the second task master controller, so as to complete the processing of the second data request through the second task master controller;

[0235] The memory management unit is used to convert the second virtual address into the second physical address.

[0236] The following is a description of the process of completing the matrix-vector multiplication task and the matrix-matrix multiplication task after obtaining the second input data in an embodiment of the present invention:

[0237] The second task master controller includes a bridge, a built-in function module, a weight input cache unit, a data input cache unit, a calculation unit and an output cache unit, and the second task master controller also includes a scheduler and a request manager, wherein:

[0238] The scheduler is used to send a third data request to the request manager. Figure 3 and Figure 7 As shown, after MVDM obtains the information of the second computing task from the firmware, the scheduler will be responsible for completing the task scheduling in the process of a matrix-vector multiplication task or a matrix-matrix multiplication task; the scheduler sends a third data request for the second input data to the request manager (request).

[0239] The request manager is used to forward the third data request to the bridge when the data required by the third data request belongs to weight data; and is also used to forward the third data request to the built-in function module when the data required by the third data request belongs to data to be processed.

[0240] Among them, according to the different data types requested by the third data request, the request manager forwards the third data request to a bridge or a built-in function module (Built-in Functions, abbreviated as BIF), and then obtains the second input data through the bridge or the built-in function module; the built-in function module is used to perform corresponding operations based on predefined built-in functions.

[0241] The bridge is also used to send a weight data read request to the third-level cache unit to complete the response to the third data request through the weight data read request.

[0242] The built-in function module is further used to send a pending data read request to the three-level cache unit, so as to complete the response to the third data request through the pending data read request.

[0243] The three-level cache unit is also used to return the weight data required by the third data request according to the weight data read request; and is also used to return the pending data required by the third data request according to the pending data read request.

[0244] The bridge is used to obtain at least one weight data and send the at least one weight data to the weight input cache unit.

[0245] like Figure 7 As shown, the weight input cache unit can be a weight input first-in-first-out data cache, that is, a first-in-first-out (FIFO) data cache is used as a buffer for weight data.

[0246] The built-in function module is used to obtain at least one data to be processed and send the at least one data to be processed to the data input cache unit; it is also used to receive at least one output data from the output cache unit and write the output data to the memory to complete the second computing task.

[0247] The data input buffer unit may be a data input first-in first-out data buffer, that is, a first-in first-out FIFO buffer is used as a buffer for data to be processed.

[0248] The computing unit is used to obtain the operation mode of the second computing task; it is also used to obtain the at least one weight data from the weight input cache unit, and to obtain the at least one data to be processed from the data input cache unit; it is also used to calculate the acquired weight data and the corresponding data to be processed based on the operation rules of the operation mode, obtain corresponding output data, and send the output data to the output cache unit.

[0249] The output buffer unit may be an output first-in-first-out data buffer, that is, a first-in-first-out FIFO buffer is used as a buffer for output data.

[0250] After the operation mode is acquired, when the corresponding weight data and the data to be processed are ready, the calculation unit performs the corresponding calculation.

[0251] The output buffer unit is used to send the output data to the built-in function module.

[0252] After the calculation is completed, the output data will be written to the memory through the built-in function module.

[0253] It is worth noting that there are two types of data paths in MVDM: one is a read-only data path via a bridge, in which the bridge can only issue a read request, and the data requested by the read request can also be returned from the read-only path; the other is a readable and writable data path via a built-in function module, in which the built-in function module can issue a readable and writable request, and the requested data can also be returned from the readable and writable path. Since in the relevant applications of AI models, the GPU needs to execute more read requests than write requests, and the path with only read requests is relatively simple, requiring less area and functions, the present invention targets the characteristics of weight data in the AI ​​model, and designs data paths for weight data and data to be processed respectively, so as to further reduce the area and functions required by the GPU chip, thereby realizing the reasonable call of the data path, significantly reducing energy consumption and operating costs while maintaining high performance, so that the GPU architecture is suitable for the calculation of AI models and can replace AI chips.

[0254] It should be noted that the methods not described in detail in the embodiments of the present invention are well known to those skilled in the art and can be implemented with reference to the prior art. They will not be expanded in the subsequent embodiments and should not be regarded as unclear aspects of the present invention.

[0255] like Fig. 9 As shown, the device for improving the computational efficiency of matrix multiplication in a GPU in this embodiment further includes one or more processors 21 and a memory 22. Fig. 9 A processor 21 is taken as an example.

[0256] The processor 21 and the memory 22 may be connected via a bus or other means. Fig. 9 The example of connecting through bus is taken in the following.

[0257] The memory 22 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs and non-volatile computer executable programs, such as the method for improving the efficiency of matrix multiplication calculations in the GPU in this embodiment. The processor 21 executes the method for improving the efficiency of matrix multiplication calculations in the GPU by running the non-volatile software programs and instructions stored in the memory 22.

[0258] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0259] The program instructions / modules are stored in the memory 22, and when executed by the one or more processors 21, the method for improving the computational efficiency of matrix multiplication in the GPU in the above embodiment is executed, for example, the method described above is executed. Figure 1 and Figure 4-Figure 6 The steps shown.

[0260] An embodiment of the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors, such as Fig. 9 A processor 21 can enable the above one or more processors to execute the method for improving the efficiency of matrix multiplication calculation in the GPU in the specific implementation of the present invention, for example, executing the above described Figure 1 and Figure 4-Figure 6 The steps shown can also be implemented Fig. 9 The modules and units described above; or executing the method for improving the computational efficiency of matrix multiplication in GPU in the specific implementation of the present invention, for example, executing the above described Figure 1 and Figure 4-Figure 6 The steps shown can also be implemented Fig. 9 The various modules and units described.

[0261] It is worth noting that the information interaction, execution process, etc. between the modules and units within the above-mentioned devices and systems are based on the same concept as the processing method embodiment of the present invention. The specific contents can be found in the description of the method embodiment of the present invention and will not be repeated here.

[0262] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0263] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for improving the computational efficiency of matrix multiplication in a GPU, characterized in that: include: Obtaining a first computing task and / or a second computing task; When processing the first computing task, determining a storage unit from a first-level cache unit, a second-level cache unit, a third-level cache unit, and a memory in a first access order, obtaining first input data from the determined storage unit, and completing the first computing task according to the first input data; When processing the second computing task, determining a storage unit from the three-level cache unit and the memory according to a second access order, and obtaining second input data from the determined storage unit; completing the second computing task according to the second input data; Wherein, when processing the second computing task, determining a storage unit from the three-level cache unit and the memory according to the second access order, and obtaining the second input data from the determined storage unit includes: querying whether the third-level cache unit has the second input data according to the second virtual address carried by the second data request of the second computing task; When the third-level cache unit has the second input data, acquiring the second input data to complete processing of the second data request; When the third-level cache unit does not have the second input data, convert the second virtual address into a second physical address; and read the second input data from the memory according to the second physical address to complete the processing of the second data request; Wherein, when processing the second computing task, the second computing task calls the second task master controller, and the second task master controller only obtains the second input data from the third-level cache unit; If the data involved in the computing task is not immediately sent or written to the corresponding target location after the previous computing task is processed, the data involved in the computing task is directly read from the third-level cache unit; wherein the previous computing task is the first computing task or the second computing task.

2. The method for improving the computational efficiency of matrix multiplication in a GPU according to claim 1, characterized in that: The second input data includes weight data and data to be processed; The completing the second computing task according to the second input data comprises: Acquire at least one piece of data to be processed and an operation mode of the second computing task; Obtain at least one weight data; Based on the operation rules of the operation mode, the acquired weight data and the corresponding data to be processed are calculated to obtain corresponding output data; Return at least one output data obtained by calculation to complete the second calculation task.

3. The method for improving the computational efficiency of matrix multiplication in a GPU according to claim 1, characterized in that: When processing the first computing task, determining a storage unit from a first-level cache unit, a second-level cache unit, a third-level cache unit, and a memory in a first access order, obtaining first input data from the determined storage unit, and completing the first computing task according to the first input data includes: querying whether the first input data is in the first-level cache unit according to the first virtual address carried by the first data request of the first computing task; When the first-level cache unit has the first input data, the first input data is obtained to complete the processing of the first data request; when the first-level cache unit does not have the first input data, querying whether the second-level cache unit has the first input data according to the first virtual address; When the second-level cache unit has the first input data, the first input data is obtained to complete the processing of the first data request; when the second-level cache unit does not have the first input data, querying whether the third-level cache unit has the first input data according to the first virtual address; When the first input data is present in the third-level cache unit, the first input data is obtained to complete processing of the first data request; when the first input data is not present in the third-level cache unit, the first virtual address is converted into a first physical address; according to the first physical address, the first input data is read from the memory to complete processing of the first data request.

4. A device for improving the computational efficiency of matrix multiplication in a GPU, characterized in that: It includes a firmware unit, a first task processor, a second task processor, a first-level cache unit, a second-level cache unit, a third-level cache unit and a memory, wherein: The first task processor is used to obtain a first computing task from the firmware unit, determine a storage unit from a first-level cache unit, a second-level cache unit, a third-level cache unit and a memory according to a first access order, obtain first input data from the determined storage unit, and complete the first computing task according to the first input data; The second task processor is used to obtain a second computing task from the firmware unit, determine a storage unit from the three-level cache unit and the memory according to a second access order, obtain second input data from the determined storage unit; and complete the second computing task according to the second input data; The second task processor includes a second task master controller and a memory management unit, wherein: The second task master controller is used to query whether the third-level cache unit has the second input data according to the second virtual address carried by the second data request of the second computing task; The three-level cache unit is further used to obtain the second input data when the third-level cache unit has the second input data, and send the obtained second input data to the second task master controller, so as to complete the processing of the second data request through the second task master controller; and is also used to read the second input data from the memory according to the second physical address when the third-level cache unit does not have the second input data, and send the obtained second input data to the second task master controller, so as to complete the processing of the second data request through the second task master controller; The memory management unit is configured to convert the second virtual address into the second physical address; Wherein, when processing the second computing task, the second computing task calls the second task master controller, and the second task master controller only obtains the second input data from the third-level cache unit; If the data involved in the computing task is not immediately sent or written to the corresponding target location after the previous computing task is processed, the data involved in the computing task is directly read from the third-level cache unit; wherein the previous computing task is the first computing task or the second computing task.

5. The device for improving matrix multiplication calculation efficiency in a GPU according to claim 4, characterized in that: The first task processor includes a first task master controller and a memory management unit, wherein: The first task master controller is used to query whether the first input data is in the first-level cache unit according to the first virtual address carried by the first data request of the first computing task; The first-level cache unit is used to obtain the first input data when the first-level cache unit has the first input data, and send the obtained first input data to the first task master controller, so as to complete the processing of the first data request through the first task master controller; it is also used to query whether the second-level cache unit has the first input data according to the first virtual address when the first-level cache unit does not have the first input data; it is also used to receive the first input data from the second-level cache unit, so as to complete the processing of the first data request through the first task master controller; The second-level cache unit is used to obtain the first input data when the second-level cache unit has the first input data, and send the obtained first input data to the first-level cache unit; it is also used to query whether the third-level cache unit has the first input data according to the first virtual address when the second-level cache unit does not have the first input data; it is also used to receive the first input data from the third-level cache unit, and send the received first input data to the first-level cache unit; The third-level cache unit is used to obtain the first input data when the third-level cache unit has the first input data, and send the obtained first input data to the second-level cache unit; and is also used to read the first input data from the memory according to the first physical address, receive the first input data from the memory, and send the received first input data to the second-level cache unit when the third-level cache unit does not have the first input data; The memory management unit is configured to convert the first virtual address into the first physical address.

6. The device for improving matrix multiplication calculation efficiency in a GPU according to claim 4, characterized in that: The second task master controller includes a bridge, a built-in function module, a weight input cache unit, a data input cache unit, a calculation unit and an output cache unit, wherein: The bridge is used to obtain at least one weight data and send the at least one weight data to the weight input cache unit; The built-in function module is used to obtain at least one data to be processed, and send the at least one data to be processed to the data input cache unit; and is also used to receive at least one output data from the output cache unit, and write the output data to the memory to complete the second computing task; The computing unit is used to obtain the operation mode of the second computing task; further used to obtain the at least one weight data from the weight input cache unit, and obtain the at least one to-be-processed data from the data input cache unit; further used to calculate the obtained weight data and the corresponding to-be-processed data based on the operation rules of the operation mode, obtain corresponding output data, and send the output data to the output cache unit; The output buffer unit is used to send the output data to the built-in function module.

7. The device for improving matrix multiplication calculation efficiency in a GPU according to claim 6, characterized in that: The second task master controller also includes a scheduler and a request manager, wherein: The scheduler is configured to send a third data request to the request manager; The request manager is used to forward the third data request to the bridge when the data required by the third data request belongs to weight data; and is also used to forward the third data request to the built-in function module when the data required by the third data request belongs to data to be processed; The bridge is further used to send a weight data read request to the third-level cache unit, so as to complete the response to the third data request through the weight data read request; The built-in function module is further used to send a pending data read request to the three-level cache unit, so as to complete the response to the third data request through the pending data read request; The three-level cache unit is also used to return the weight data required by the third data request according to the weight data read request; and is also used to return the pending data required by the third data request according to the pending data read request.

8. A non-volatile computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the method for improving the computational efficiency of matrix multiplication in a GPU as described in any one of claims 1-3.

Images