Computing resource management method and apparatus, electronic device, and storage medium

Abstract

The application relates to the technical field of computers and provides a kind of computing resource management method, device, electronic equipment and storage medium, wherein the method comprises: in the case of performing matrix product accumulation operation, the working state of the first memory is set to the resource occupation state;The terminal write instruction of performing matrix product accumulation operation is used to write the final operation result of matrix product accumulation operation into the second memory, and the working state of the first memory is changed from the resource occupation state to the normal state.The method couples the state change operation with the write instruction of the final operation result through the terminal write instruction, ensures that the occupied first memory resource will be reliably released when the matrix product accumulation operation task successfully outputs the final operation result.Thereby, the resource occupation risk caused by incomplete recovery is effectively avoided, and the reliability and effectiveness of the computing resource management are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology, and in particular to a computing resource management method and device, electronic equipment and storage medium. BACKGROUND

[0002] In the process of using L2 cache (Level 2 cache) to accelerate calculation, in order to improve the data hit rate of L2 cache, the HBM (High Bandwidth Memory) workspace is set to have a privileged occupation of L2 cache during the calculation process of the MMA (Matrix Multiply Accumulate) instruction, and the data is loaded into the workspace by the subsequent LDM (Load Multiple registers) instruction and the priority of the set L2 cache is degraded to a state of none. However, due to the existence of the Read cache, the LDM instruction cannot completely guarantee that the data is sent to the HBM workspace, so it cannot completely recover to the state of none after the operator ends. A large model is usually composed of many operators, and when the L2 cache priority is set to Privileged (high priority) by the MMA operator and has not been restored to the state of none, in some cases, the operators after the MMA may not be released due to the privileged occupation of the L2 cache, which may cause problems such as mutual blocking between multiple requests in the instruction transmission process of the operator calculation process.

[0003] In various sizes of models, the MMA operator needs to process various regular or irregular shapes. In chip design, the execution granularity of the MMA instruction is usually designed to be large in order to fully utilize the Tensor Core (tensor core) computing power, and the LDM instruction granularity is designed to be relatively small. During the execution of the above-mentioned MMA operator, the LDM usually only considers the real output boundary to avoid invalid calculation, so in the case of irregular shape, there may be a problem that part of the area in the applied HBM workspace does not correctly restore the L2 cache priority, which may cause problems such as instruction blocking of the subsequent operator when the L2 cache is always occupied by the privileged. SUMMARY

[0004] The present application provides a computing resource management method, device, electronic equipment and storage medium to solve the defect that the high priority state of the secondary cache of part of the HBM workspace is not correctly restored due to irregular shape in the prior art, causing resource residue and possibly blocking the instruction execution of the subsequent operator.

[0005] The application provides a kind of computing resource management method, it is applied to computing device, the computing device includes processing unit, first memory and second memory, the method includes the following steps:

[0006] In the case of performing matrix product accumulation operation, the working state of the first memory is set to resource occupation state;

[0007] The terminal write instruction of the matrix product accumulation operation is executed;The terminal write instruction is used to write the final operation result of the matrix product accumulation operation into the second memory, and the working state of the first memory is changed from the resource occupation state to the normal state.

[0008] According to the computing resource management method provided by the application, the matrix product accumulation operation includes performing multiple block matrix multiplication operations on the first matrix and the second matrix to obtain the final operation result;

[0009] The terminal write instruction is the write instruction corresponding to the last block matrix multiplication operation in the multiple block matrix multiplication operations.

[0010] According to the computing resource management method provided by the application, the multiple block matrix multiplication operations on the first matrix and the second matrix to obtain the final operation result include:

[0011] The operation result of each block matrix multiplication operation of the first matrix and the second matrix is accumulated with the operation result in the second memory to obtain the final operation result;

[0012] Wherein, the operation result of the first block matrix multiplication operation of the first matrix and the second matrix is written into the second memory.

[0013] According to the computing resource management method provided by the application, the operation result of each block matrix multiplication operation of the first matrix and the second matrix is accumulated with the operation result in the second memory to obtain the final operation result, including:

[0014] The operation result obtained by the current block matrix multiplication operation is accumulated with the operation result in the second memory to generate an updated operation result;

[0015] The updated operation result is written back to the second memory;

[0016] Wherein, the final operation result is the updated operation result corresponding to the last block matrix multiplication operation.

[0017] According to the computing resource management method provided by the application, the working state of the first memory is set to resource occupation state, including:

[0018] The first memory is configured as a privilege occupation state, so that data associated with a preset workspace in the second memory is preferentially cached in the first memory.

[0019] According to the computing resource management method provided by the application, after the working state of the first memory is changed from the resource occupation state to the normal state, the method further comprises:

[0020] The final operation result is loaded from the second memory to a target register through a batch loading instruction.

[0021] According to the computing resource management method provided by the application, the first memory is a two-level cache, and the second memory is a high-bandwidth memory.

[0022] The application further provides a computing resource management device applied to a computing device, wherein the computing device comprises a processing unit, a first memory and a second memory, and the device comprises:

[0023] The setting unit is configured to set the working state of the first memory as a resource occupation state in the case of performing a matrix product accumulation operation.

[0024] The execution unit is configured to execute a terminal write instruction of the matrix product accumulation operation, and the terminal write instruction is used to write a final operation result of the matrix product accumulation operation to the second memory and change the working state of the first memory from the resource occupation state to a normal state.

[0025] The application further provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the computing resource management method as described above when executing the program.

[0026] The application further provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the computing resource management method as described above.

[0027] The application further provides a computer program product comprising a computer program, and the computer program is executed by a processor to implement the computing resource management method as described above.

[0028] The application provides a computing resource management method, device, electronic equipment and storage medium, in which, in the case of performing a matrix product accumulation operation, the working state of a first memory is set as a resource occupation state; a terminal write instruction is executed, and the terminal write instruction is used for writing a final operation result of the matrix product accumulation operation into a second memory and changing the working state of the first memory from the resource occupation state to a normal state. The method couples the state change operation and the write instruction of the final operation result by using the terminal write instruction, so that the occupied first memory resource is reliably released when the matrix product accumulation operation task successfully outputs the final operation result. Therefore, the resource occupation risk caused by incomplete recovery is effectively avoided, and the reliability and effectiveness of the computing resource management are improved. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0030] Figure 1 is a calculation schematic diagram of an MMA operator in the prior art.

[0031] Figure 2 is a schematic diagram of a region in which a load memory instruction does not cover a working space of a high-bandwidth memory.

[0032] Figure 3 is a flowchart of the computing resource management method provided by the application.

[0033] Figure 4 is a structural schematic diagram of the computing device provided by the application.

[0034] Figure 5 is a schematic diagram in which the matrix product accumulation operation instruction degrades the working space of the high-bandwidth memory for the last time.

[0035] Figure 6 is a schematic diagram in which the matrix product accumulation operation uses a two-level cache to accelerate calculation.

[0036] Figure 7 is a structural schematic diagram of the computing resource management device provided by the application.

[0037] Figure 8 is a structural schematic diagram of the electronic equipment provided by the application.

[0038] Reference signs:

[0039] 41: processing unit; 42: first memory; 43: second memory; 710: setting unit; 720: execution unit; 810: processor; 820: communication interface; 830: memory; 840: communication bus. DETAILED DESCRIPTION

[0040] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0041] The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second" and the like are generally of a kind.

[0042] Figure 1 is a schematic diagram of the calculation of the MMA operator in the prior art, as shown in Figure 1 The MMA operator is used to calculate the multiplication between two matrices, and the calculation formula is where A and B are input matrices, and C is an output matrix. The dimension of the input matrix A is MxK, the dimension of the input matrix B is KxN, and the dimension of the output matrix C is MxN. As one of the most common and basic operators in large model training and inference, the performance of the MMA operator has an important influence on the overall model performance. Therefore, in GPU (Graphics Processing Unit, graphics processor) programming, high-speed cache resources such as L2 cache are often used to improve the throughput of the overall operator and achieve the purpose of improving the performance of the MMA.

[0043] In various sizes of models, the MMA operator needs to process various regular or irregular shapes. In chip design, for performance considerations, the execution granularity of the MMA instruction is usually designed to be large, so as to fully exert the tensor core (TensorCore) computing power, and the LDM instruction granularity is designed to be relatively small. Figure 2 is a schematic diagram of the area of the workspace of the high-bandwidth memory which is not covered by the load memory instruction provided by the present application, as shown in Figure 2 Figure 2 ​The green rectangle represents the area covered by an LDM instruction, while the length and width of the blue area correspond to the actual length and width of the output matrix, respectively. During the execution of the MMA operator described above, LDM typically only considers the actual output boundaries to avoid invalid computations. However, in irregularly shaped cases, some areas (red areas) within the allocated workspace (blue area) may not have their L2 cache priority correctly restored. This could lead to instruction blocking issues for subsequent operators when the L2 cache is consistently occupied by privileged access.

[0044] To address the above problems, this invention provides a computing resource management method. Figure 3 This is a flowchart illustrating the computing resource management method provided by the present invention, as shown below. Figure 3 As shown, the method includes steps 110 and 120.

[0045] Step 110: When performing matrix multiplication and accumulation operations, set the working state of the first memory to resource occupied state.

[0046] Specifically, this method is applied to computing devices. Figure 4 This is a schematic diagram of the structure of the computing device provided by the present invention, as shown below. Figure 4 As shown, the computing device includes a processing unit 41, a first memory 42, and a second memory 43. The computing device can be a general-purpose computing system or a dedicated computing system designed for specific computing tasks, such as artificial intelligence, machine learning, and scientific computing. For example, the computing device can be a server equipped with a graphics processor, a dedicated AI (Artificial Intelligence) accelerator chip, or a system-on-a-chip (SoC) including multiple computing cores, etc. This embodiment of the invention does not specifically limit the specific type of device.

[0047] Here, the processing unit 41 is the core component for performing computational tasks. The processing unit can be a GPU (Graphics Processing Unit), etc. In a preferred embodiment of the present invention, considering the characteristics of matrix multiplication and addition operations, the processing unit 41 can specifically be a streaming multiprocessor (SM) in a GPU or a hardware acceleration unit specifically designed for performing matrix multiplication and addition operations, such as a tensor core. The processing unit 41 is responsible for interpreting and executing various instructions, including matrix multiplication and addition operation instructions and data read / write instructions.

[0048] The first memory 42 is a memory with higher access speed but generally smaller capacity compared with the second memory 43. The first memory 42 is generally used as a cache of the processing unit to reduce the delay caused by the access of the second memory 43 by the processing unit. The first memory 42 can be specifically a level one cache (L1 Cache), a level two cache, a level three cache (L3 Cache), etc., and the embodiments of the present application do not make specific limitation thereon. In the application scenario of the present embodiment, the first memory 42 is preferably a level two cache.

[0049] Here, the second memory 43 is a memory with larger capacity but slower access speed compared with the first memory 42. The second memory 43 is generally used as the main memory of the system to store the original data to be processed, the intermediate results in the calculation process and the final operation result. The second memory 43 can be specifically a dynamic random-access memory (DRAM), a double data rate synchronous dynamic random-access memory (DDR SDRAM), a high bandwidth memory, etc., and the embodiments of the present application do not make specific limitation thereon. In specific practice, one or more preset workspaces can be opened in the second memory 43 to temporarily store the intermediate data or the final result of the matrix multiplication accumulation operation.

[0050] Specifically, in the case of performing the matrix multiplication accumulation operation, the working state of the first memory 42 can be set to the resource occupation state. The matrix multiplication accumulation operation refers to a mathematical operation on one or more matrices. For example, the calculation of where A, B and C are all matrices, which is one of the most common and basic operations in the scenarios of deep learning model training and reasoning. It should be noted that the matrix multiplication accumulation operation in the embodiments of the present application is not limited to strict mathematical matrix multiplication, and the core thereof is a high-efficiency multiply-add fusion tensor calculation paradigm. It can be extended and mapped to a series of complex tensor operations depending on the dense multiplication accumulation mode, such as linear transformation fused with an activation function, a core part of convolution calculation, etc., which can all benefit from the efficient reuse of intermediate results and the low-delay access characteristics of the MMA hardware.

[0051] It should be noted that in the case of performing the matrix product accumulation operation, the working state of the first memory 42 is set to the resource occupation state, that is, the working state of the first memory 42 is set to the resource occupation state directly on the write-out instruction of the matrix product accumulation. More specifically, the process of this configuration and taking effect is: first, it is configured on the write instruction of the matrix product accumulation operation, and after the processing unit 41 completes a block matrix multiplication operation, when the data stream carrying the operation result passes through the first memory 42 to be written out and accumulated in the preset working space in the second memory 43, the working state setting of the first memory 42 takes effect. Here, the working state refers to the cache strategy mode adopted by the first memory 42 for the memory working area associated with the matrix product accumulation operation. The working state can include the resource occupation state and the normal state.

[0052] The resource occupation state is a special configuration state. In the resource occupation state, the first memory will preferentially cache data related to the matrix calculation task. Specifically, when the processing unit 41 needs to access a certain data in the second memory 43, if the first memory 42 is in the resource occupation state and the data belongs to the range designated for preferential caching, the data is loaded into the first memory 42, and the priority of its cache line being replaced by new data is reduced, thereby being retained more persistently. This can greatly improve the repeated access speed of the data.

[0053] Here, the normal state is the default working state of the first memory 42. In the normal state, the first memory 42 caches all data in accordance with the standard replacement algorithm, such as the LRU (Least Recently Used) algorithm, without special priority differentiation. In a specific hardware implementation, the normal state can correspond to a non-priority state.

[0054] Therefore, in the case of performing the matrix product accumulation operation, the working state of the first memory 42 is set to the resource occupation state, the purpose of which is to provide cache support for the intermediate results stored in the working space of the second memory 43 in the subsequent matrix calculation process, thereby speeding up the entire matrix product accumulation operation process.

[0055] Step 120, executing a terminal write instruction of the matrix product accumulation operation; the terminal write instruction is used to write the final operation result of the matrix product accumulation operation into the second memory, and to change the working state of the first memory from the resource occupation state to the normal state.

[0056] Specifically, the terminal write instruction for performing the matrix product accumulation operation is executed, wherein the terminal write instruction is used to write the final operation result of the matrix product accumulation operation to the second memory and change the working state of the first memory from the resource occupation state to the normal state. This step releases the occupied computing resources safely while completing the computing task through a dual-function terminal write instruction.

[0057] The terminal write instruction for performing the matrix product accumulation operation refers to the process in which the processing unit 41, such as a tensor core, completes the operation specified by the instruction code. Specifically, the terminal write instruction in this context specifically refers to the write operation instruction responsible for writing the final operation result to the second memory 43 at the final stage of the matrix product accumulation operation process. It should be noted that a complex matrix product accumulation operation may include multiple computing stages and corresponding multiple write operations, such as writing intermediate results, and the terminal write instruction is the last one in this series of write operations.

[0058] Here, the final operation result refers to the final data obtained after the completion of the entire matrix product accumulation operation task. For example, in the operation of , the final operation result is the complete matrix C.

[0059] It should be understood that the specific function of the terminal write instruction includes a data write function and a state recovery function. The core task of the data write function is to write the final operation result obtained by calculation from the register or internal cache of the processing unit 41 to the address space of the specified second memory 43. And the function of restoring the working state of the first memory 42 is coupled in the terminal write instruction. That is, when the processing unit 41 executes the terminal write instruction, in addition to completing the data write operation, it will also simultaneously issue an internal signal to switch the working state of the first memory 42 from the previously set resource occupation state to the normal state.

[0060] In the prior art, the setting and restoring of the cache priority are usually performed by different instructions, for example, a matrix multiplication instruction is used to set a high priority state, and a subsequent load instruction is used to restore to a normal state. However, due to the presence of a read cache component in the hardware architecture or the uncertainty of the data access path when processing irregular matrix multiplication accumulation operations, the load instruction responsible for state restoration may not completely cover all storage areas that have been set to a high priority state. The incompleteness of this recovery mechanism causes part of the resources of the first memory 42 to remain in an irregular priority state, which cannot be completely released, and thus may cause resource blocking and performance interference for other computing tasks that need to access the memory. The embodiments of the present application accurately identify the terminal write key node that must occur in the matrix multiplication accumulation operation process, and bind the cache priority restoration operation to the terminal write instruction. This design ensures that when the matrix multiplication accumulation operation is successfully executed and the final operation result is output, the terminal write instruction as the end of the process signal will trigger the reset operation of the first memory 42 state, so that it can be reliably restored from the resource occupation state to the normal state. This mechanism fundamentally solves the problem of incomplete priority restoration, ensures that the system resources can be completely released after the operation is completed, thereby significantly improving the overall stability of the system resources under complex computing tasks, effectively avoiding the risk of resource occupation caused by incomplete restoration, and improving the reliability and effectiveness of computing resource management.

[0061] The method provided by the embodiments of the present application sets the working state of the first memory to a resource occupation state in the case of performing matrix multiplication accumulation operation; and executes a terminal write instruction of the matrix multiplication accumulation operation, the terminal write instruction being used to write the final operation result of the matrix multiplication accumulation operation to a second memory and change the working state of the first memory from the resource occupation state to a normal state. The method couples the state change operation with the write instruction of the final operation result through the terminal write instruction, so as to ensure that the occupied first memory resource will be reliably released when the matrix multiplication accumulation operation task successfully outputs the final operation result. Thus, the risk of resource occupation caused by incomplete restoration is effectively avoided, and the reliability and effectiveness of computing resource management are improved.

[0062] Based on the above embodiments, the matrix multiplication accumulation operation includes performing multiple block matrix multiplication operations on the first matrix and the second matrix to obtain the final operation result;

[0063] The terminal write instruction is a write instruction corresponding to the last block matrix multiplication operation in the multiple block matrix multiplication operations.

[0064] Specifically, in one specific application scenario, for example, the matrix multiplication commonly seen in large model training, the dimensions of the matrix A and the matrix B that need to be calculated are usually very large. The matrix size that the processing unit 41 can handle at a time is limited, and it cannot complete the multiplication of the entire large matrix at one time. Therefore, a common and efficient calculation strategy is to use the idea of blocking.

[0065] Specifically, in the embodiment of the present application, the matrix product accumulation operation includes performing multiple block matrix multiplication operations on the first matrix and the second matrix to obtain the final operation result. This means that the original large matrix, for example, the first matrix A and the second matrix B, is logically divided into several smaller sub-matrices (blocks), and then the processing unit performs multiple multiplication operations on these sub-matrices (blocks). Each block matrix multiplication operation produces an intermediate result.

[0066] Correspondingly, the entire matrix product accumulation operation process includes multiple operations of writing the intermediate result into the memory. Among them, the terminal write instruction is the write instruction corresponding to the last block matrix multiplication operation in the multiple block matrix multiplication operations.

[0067] In an optional embodiment, at the program logic level, the timing of the last block operation can be uniquely identified by a loop counter, an iteration termination flag, or a specific program flow control mechanism. For example, in a for loop structure that performs N block operations, the write instruction executed in the Nth iteration process is the terminal write instruction.

[0068] The method provided by the embodiment of the present application specifically divides the complex matrix product accumulation operation into multiple block matrix multiplication operations on the first matrix and the second matrix, and clearly defines the terminal write instruction as the write operation corresponding to the last operation in the multiple block matrix multiplication operations. By dividing the matrix product accumulation operation task into iteratively executable block operations, a reliable technical anchor point is provided for precisely implanting resource recovery operations in complex loop calculation processes, thereby ensuring the reliability of the calculation resource management in complex calculation processes.

[0069] Based on the above embodiment, the performing multiple block matrix multiplication operations on the first matrix and the second matrix to obtain the final operation result includes:

[0070] Step 21, accumulating the operation result of each block matrix multiplication operation of the first matrix and the second matrix and the operation result in the second memory to obtain the final operation result;

[0071] Wherein, the operation result of the first block matrix multiplication operation of the first matrix and the second matrix is written into the second memory.

[0072] Specifically, the final operation result in the matrix multiplication accumulation operation is combined from intermediate results generated by all the block matrix multiplication operations. Specifically, the first matrix and the second matrix are subjected to multiple block matrix multiplication operations to obtain the final operation result, including:

[0073] The operation result of each block matrix multiplication operation of the first matrix and the second matrix is accumulated with the operation result in the second memory to obtain the final operation result.

[0074] Specifically, the first block matrix multiplication operation is first performed, and the block multiplication result of the first matrix and the second matrix is written into the second memory 43. In this process, the processing unit 41 calculates the first partial result and directly stores it into the workspace in the second memory 43. At this time, the initial partial result is saved in the workspace. At the instruction level, this operation can be implemented by a matrix multiplication write-out (MMA out) type instruction.

[0075] Subsequently, in each subsequent block matrix multiplication operation, the processing unit 41 calculates a new partial result, does not directly overwrite the existing data in the workspace, but performs an accumulation operation: sums the newly calculated partial result with the matrix multiplication accumulation operation result currently stored in the workspace of the second memory, and rewrites the updated accumulation result into the workspace. This process corresponds to a matrix multiplication accumulation operation type instruction at the instruction implementation level.

[0076] It should be understood that the above calculation-accumulation process will be performed in a loop until all the matrix blocks are completed. After the last block matrix multiplication and accumulation operation is completed, the data stored in the workspace of the second memory 43 is the final operation result of the entire matrix multiplication.

[0077] Based on the above embodiment, step 21 includes:

[0078] Step 211, accumulating the operation result obtained by the current block matrix multiplication operation with the operation result in the second memory to generate an updated operation result;

[0079] Step 212, writing the updated operation result back to the second memory;

[0080] The final operation result is the updated operation result corresponding to the last block matrix multiplication operation.

[0081] Specifically, the operation result of the current sub-block matrix multiplication is accumulated with the operation result in the second memory to generate an updated operation result. The operation result of the current sub-block matrix multiplication refers to the matrix operation result generated by the processing unit 41 in the current calculation iteration. The operation result in the second memory refers to the matrix multiplication accumulation operation result obtained by all previous sub-accumulation operations and stored in the working space of the second memory 43.

[0082] After obtaining the updated operation result, the updated operation result can be written back to the second memory. By repeatedly performing steps 211 and 212, the values in the second memory are continuously accumulated until the last sub-block operation is completed. Therefore, the final operation result is the updated operation result corresponding to the last sub-block matrix multiplication.

[0083] It can be understood that the terminal write instruction is a write instruction for writing the last updated operation result, i.e., the final operation result, back to the second memory in the last sub-block matrix multiplication.

[0084] The method provided by the embodiment of the application accumulates the operation result of the current sub-block matrix multiplication with the operation result in the second memory to generate an updated operation result, and writes the updated operation result back to the second memory. The final operation result is the updated operation result corresponding to the last sub-block matrix multiplication. The method ensures that the cache resource is continuously resident before the final operation result is generated, thereby guaranteeing the high-speed access performance of all intermediate operation results and accurately releasing the resource at the moment when the operation is completed, so that the best balance between the calculation performance and the resource utilization efficiency is achieved.

[0085] Based on the above embodiment, the step 110 of setting the working state of the first memory to a resource occupation state comprises:

[0086] The step 1101 sets the first memory to a privileged occupation state, so that the data associated with the preset working space in the second memory is preferentially cached in the first memory.

[0087] Specifically, in the embodiment of the application, the first memory 42 can be set to a privileged occupation state (Privileged state), so that the data associated with the preset working space in the second memory 43 is preferentially cached in the first memory 42.

[0088] The privileged occupation state is a specific cache strategy that can be configured by software instructions. When a certain cache line of the first memory, such as the L2Cache, is set to this state, it has the highest priority in the cache replacement decision, i.e., it is least likely to be replaced by a conventional cache replacement algorithm.

[0089] The preset working space is a memory region explicitly applied by a program, such as an MMA operator, in the second memory 43, and is specially used to store intermediate accumulation results and final operation results of the matrix product accumulation operation.

[0090] In a specific embodiment, it is necessary to note that in the case of performing the matrix product accumulation operation, the working state of the first memory 42 is set to the privileged occupation state, that is, the working state of the first memory 42 is set to the privileged occupation state directly on the write-out instruction of the matrix product accumulation. More specifically, the configuration and the process of taking effect are as follows: first, the configuration is performed on the write instruction of the matrix product accumulation operation, and after the processing unit 41 completes a block matrix multiplication operation, when the data stream carrying the operation result passes through the first memory 42 to be written out and accumulated in the preset working space in the second memory 43, the working state setting of the first memory 42 takes effect, so that the data associated with the preset working space in the second memory 43 is preferentially cached in the first memory 42.

[0091] The method provided by the embodiment of the application sets the first memory to the privileged occupation state, so that the data associated with the preset working space in the second memory is preferentially cached in the first memory. The method ensures that all intermediate result data can continuously reside in the cache during the entire matrix product accumulation operation, thereby guaranteeing the high-speed and low-delay access of the processing unit to the intermediate result data. At the same time of completing the matrix product accumulation operation, the privileged occupation state can be accurately released, and the cache resource can be immediately released, thereby realizing the efficient cooperation between the maximization of cache hit rate and the dynamic recovery of storage resources, and further improving the operation energy efficiency of the calculation resource management.

[0092] Based on the above embodiment, after the working state of the first memory is changed from the resource occupation state to the normal state, the method further includes:

[0093] The final operation result is loaded from the second memory to a target register by a batch loading instruction.

[0094] Specifically, after the working state of the first memory 42 is changed from the resource occupation state to the normal state, the final operation result of the matrix product accumulation operation has been completely and safely written into the working space of the second memory 43. At the same time, the cache access priority of the first memory 42 has also been restored to the normal setting accordingly.

[0095] Therefore, the final operation result can be loaded from the second memory 43 to a target register by a batch loading instruction. The batch loading instruction is an instruction specially used to efficiently read a data block (matrix) from the memory to the target register. Here, the batch loading instruction can be an LDM instruction.

[0096] Here, the target register refers to a high-speed storage unit inside the processing unit 41 for temporarily storing data. The target register can be a TLR (Tensor Load Register) or the like, and embodiments of the present application do not make specific limitations thereto.

[0097] It should be understood that the execution timing of this operation step is of critical importance, and the start of the timing control logic must be strictly later than the successful recovery of the state of the first memory 42 to the normal state. This design ensures that the execution process of the subsequent LDM instruction is completely independent and will not be affected by any residual effects of the previous privileged occupation state. This is the core innovation of the present application compared to the prior art. The prior art solution couples the task of downgrading the cache state to the data loading process of the LDM instruction, but this mechanism has inherent defects. When processing irregularly shaped matrices causes the LDM instruction to only load part of the data of the workspace, or the requested data has been hit in the cache, the bulk loading instruction may not access all the memory regions that need to be downgraded, resulting in incomplete state downgrade and leaving resource locking risks.

[0098] The present application fundamentally solves the above problems by introducing a terminal write instruction with explicit timing constraints. The terminal write instruction is used to write the final operation result of the matrix product accumulation operation to the second memory 43, and to change the working state of the first memory 42 from the resource occupation state to the normal state. Thereafter, the LDM instruction only undertakes its data loading function, thereby optimizing the data path performance while ensuring system state consistency.

[0099] Based on the above embodiment, the first memory is a level two cache, and the second memory is a high bandwidth memory.

[0100] Specifically, the first memory is a level two cache. The level two cache is located between the L1 Cache and the main memory in the storage hierarchy, has a larger capacity than the L1 Cache and a faster access speed than the main memory, is an ideal choice for balancing hit rate and access delay, and is very suitable for use as an intermediate result cache for large-scale operations such as matrix product accumulation operations.

[0101] The second memory is a high bandwidth memory. The HBM integrates multiple DRAM (Dynamic Random Access Memory) chips with the processor core on the same package substrate through 3D stacking technology, providing extremely high storage bandwidth and being able to meet the throughput requirements of powerful processing units such as tensor cores for data.

[0102] The method provided by the embodiments of the present application has the first memory as a level two cache and the second memory as a high bandwidth memory, which can maximize the performance of the hardware.

[0103] Based on any of the above embodiments, Figure 5 is a schematic diagram of the matrix product accumulation operation instruction provided by the present application degrading the workspace of the high-bandwidth memory at the last time of writing, as Figure 5 shown, by identifying the last time the matrix product accumulation operation instruction is executed, the accumulation is completed, the L2 cache privilege occupation of the workspace of the high-bandwidth memory (HBM workspace) is restored to nothing, thereby fundamentally avoiding the problem that the workspace of the HBM is not completely degraded by the LDM instruction due to the irregular shape that may occur subsequently.

[0104] First, when performing the first matrix product accumulation operation, the processing unit calculates the first partial result and writes it into the workspace of the high-bandwidth memory. The first memory workspace state associated with this write operation is set to nothing, i.e. the normal state. Then, when performing the second, third and subsequent matrix product accumulation operations, the partial result generated by each operation is accumulated with the data already stored in the workspace of the high-bandwidth memory, and the updated accumulation result is written back. In these intermediate steps, the instruction for writing sets the first memory workspace state to the privileged state to ensure that frequent access to the workspace can obtain cache acceleration.

[0105] Then, when performing the last matrix product accumulation operation, the processing unit writes the final operation result calculated into the workspace of the high-bandwidth memory. The terminal write instruction that performs this write simultaneously restores the first memory workspace state from the privileged state to nothing, i.e. the normal state, thereby reliably releasing the occupied cache resources.

[0106] Finally, after the entire matrix product accumulation operation is completed and the cache state is restored, the final result stored in the workspace of the high-bandwidth memory can be loaded into the target register of the processing unit, such as TLR, through the load multiple registers (LDM) instruction, and finally used for subsequent processing or directly as output.

[0107] The method provided by the embodiment of the present application can ensure that the MMA operator safely and efficiently utilizes the L2 cache to optimize the calculation and memory access, while avoiding potential performance risks to subsequent operators from the mechanism level. The method has good code universality, does not require the introduction of additional steps and overhead, has a controllable impact on operator performance within 1%, and helps to reduce the dependence on L2 cache hardware complexity, thereby reducing hardware power consumption and area.

[0108] Based on any of the above embodiments, Figure 6 is a schematic diagram of the matrix product accumulation operation using two-level cache acceleration calculation provided by the present application, asFigure 6 As shown, matrix A and matrix B are sent to the processing unit for tensor core computation, and the specific operation is matrix multiply-accumulate operation. The operation process includes multiple block calculations. The partial results generated by the calculation are written into the storage system in two ways: the results of the first calculation are directly written by the matrix multiply-accumulate operation write instruction; and the subsequent partial results are written back after being accumulated with the existing intermediate results by the matrix multiply-accumulate operation accumulation instruction. The target storage locations of these write and accumulation operations are the workspaces in the high-bandwidth memory. When accessing the workspaces, the data passes through the level 2 cache for acceleration. In the embodiment of the present application, the working state of the level 2 cache is set to the resource occupation state to improve the access hit rate. When the last matrix multiply-accumulate operation write is performed, the state of the level 2 cache is restored to the normal state.

[0109] When all the matrix multiply-accumulate operations are completed, the final result is stored in the workspace of the high-bandwidth memory, and then the computing device executes a load multiple registers instruction (LDM instruction) to load the final operation result from the workspace to the target register. Finally, the data stored in the target register can be used as the final output for subsequent processing or storage.

[0110] The computing resource management device provided by the present application is described below. The computing resource management device described below can be referred to in correspondence with the computing resource management method described above.

[0111] Based on any of the above embodiments, the present application provides a computing resource management device, Figure 7 is a structural schematic diagram of the computing resource management device provided by the present application, as Figure 7 shown, the device is applied to a computing device, the computing device includes a processing unit, a first memory, and a second memory, and the device includes:

[0112] The setting unit 710 is configured to set the working state of the first memory to the resource occupation state in the case of performing the matrix multiply-accumulate operation.

[0113] The execution unit 720 is configured to execute a terminal write instruction of the matrix multiply-accumulate operation; and the terminal write instruction is configured to write the final operation result of the matrix multiply-accumulate operation to the second memory and change the working state of the first memory from the resource occupation state to the normal state.

[0114] The device provided by the embodiment of the present application sets the working state of the first memory as a resource occupation state in the case of performing the matrix product accumulation operation; a terminal write instruction is executed to write the final operation result of the matrix product accumulation operation into the second memory, and the working state of the first memory is changed from the resource occupation state to a normal state. The method couples the state change operation with the write instruction of the final operation result through the terminal write instruction, so as to ensure that the occupied first memory resource will be reliably released when the matrix product accumulation operation task successfully outputs the final operation result. Therefore, the resource occupation risk caused by incomplete recovery is effectively avoided, and the reliability and effectiveness of the calculation resource management are improved.

[0115] According to any one of the above embodiments, the matrix product accumulation operation includes performing multiple block matrix multiplication operations on the first matrix and the second matrix to obtain the final operation result.

[0116] The terminal write instruction is a write instruction corresponding to the last block matrix multiplication operation in the multiple block matrix multiplication operations.

[0117] According to any one of the above embodiments, the performing multiple block matrix multiplication operations on the first matrix and the second matrix to obtain the final operation result specifically includes:

[0118] An accumulation unit is configured to accumulate the operation result of each block matrix multiplication operation of the first matrix and the second matrix and the operation result in the second memory to obtain the final operation result.

[0119] The operation result of the first block matrix multiplication operation of the first matrix and the second matrix is written into the second memory.

[0120] According to any one of the above embodiments, the accumulation unit is specifically configured to:

[0121] accumulate the operation result of the current block matrix multiplication operation and the operation result in the second memory to generate an updated operation result;

[0122] write the updated operation result back to the second memory;

[0123] The final operation result is the updated operation result corresponding to the last block matrix multiplication operation.

[0124] According to any one of the above embodiments, the setting unit 710 is specifically configured to:

[0125] configure the first memory as a privileged occupation state, so that the data associated with the preset working space in the second memory is preferentially cached in the first memory.

[0126] Based on any of the above embodiments, further comprising a loading unit, in particular for:

[0127] loading the final operation result from the second memory to a target register by a batch loading instruction.

[0128] Based on any of the above embodiments, the first memory is a level two cache, and the second memory is a high bandwidth memory.

[0129] Figure 8 is a structural schematic diagram of an electronic device provided by the present application, as Figure 8 shown, the electronic device can include: a processor (processor) 810, a communications interface (Communications Interface) 820, a memory (memory) 830 and a communication bus 840, wherein the processor 810, the communication interface 820, the memory 830 complete the mutual communication through the communication bus 840. The processor 810 can call the logic instruction in the memory 830 to execute the computing resource management method, the method comprising: in the case of executing a matrix product accumulation operation, setting the working state of the first memory to a resource occupation state; executing a terminal write instruction of the matrix product accumulation operation; the terminal write instruction is used to write the final operation result of the matrix product accumulation operation to the second memory, and change the working state of the first memory from the resource occupation state to the normal state.

[0130] In addition, the logic instruction in the memory 830 described above can be realized in the form of a software function unit and sold or used as an independent product when used, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. And the foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and various program code storage media.

[0131] In another aspect, the present application also provides a computer program product comprising a computer program, which is stored in a non-transitory computer readable storage medium, and the computer program is executable by a processor to cause a computer to perform the computing resource management method provided by any of the above methods, which comprises: setting a working state of a first memory to a resource occupation state in a case of performing a matrix product-sum operation; executing a terminal write instruction of the matrix product-sum operation; the terminal write instruction is used to write a final operation result of the matrix product-sum operation into a second memory, and change the working state of the first memory from the resource occupation state to a normal state.

[0132] In another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executable by a processor to implement the computing resource management method provided by any of the above methods, which comprises: setting a working state of a first memory to a resource occupation state in a case of performing a matrix product-sum operation; executing a terminal write instruction of the matrix product-sum operation; the terminal write instruction is used to write a final operation result of the matrix product-sum operation into a second memory, and change the working state of the first memory from the resource occupation state to a normal state.

[0133] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e., they may be located in one place, or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0134] From the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software plus necessary universal hardware platforms, and of course, can also be realized by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0135] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A computing resource management method applied to a computing device, the computing device comprising a processing unit, a first memory, and a second memory, characterized in that, The method includes: When performing matrix multiplication and accumulation operations, the working state of the first memory is set to resource occupied state; A terminal write instruction is executed to perform the matrix multiplication and accumulation operation; the terminal write instruction is used to write the final result of the matrix multiplication and accumulation operation into the second memory, and to change the working state of the first memory from the resource occupied state to the normal state; the first memory is used as a cache of the processing unit, and the second memory is used as the main memory of the system; Setting the working state of the first memory to a resource-occupied state includes: Configure the first memory to a privileged state so that data associated with a preset workspace in the second memory is preferentially cached in the first memory.

2. The computing resource management method according to claim 1, characterized in that, The matrix multiplication accumulation operation includes performing multiple block matrix multiplication operations on the first matrix and the second matrix to obtain the final operation result; The terminal write instruction is the write instruction corresponding to the last block matrix multiplication operation in the multiple block matrix multiplication operations.

3. The computing resource management method according to claim 2, characterized in that, The step of performing multiple block matrix multiplication operations on the first matrix and the second matrix to obtain the final operation result includes: The results of each block matrix multiplication operation between the first matrix and the second matrix are accumulated with the results in the second memory to obtain the final operation result; Specifically, the result of the first block matrix multiplication operation between the first matrix and the second matrix is ​​written into the second memory.

4. The computing resource management method according to claim 3, characterized in that, The step of accumulating the results of each block matrix multiplication operation between the first matrix and the second matrix with the results in the second memory to obtain the final result includes: The result of the current block matrix multiplication operation is added to the result in the second memory to generate an updated result; Write the update operation result back to the second memory; The final calculation result is the update calculation result corresponding to the last block matrix multiplication operation.

5. The computing resource management method according to any one of claims 1 to 4, characterized in that, After changing the operating state of the first memory from the resource-occupied state to the normal state, the method further includes: The final calculation result is loaded from the second memory to the target register using a batch load instruction.

6. The computing resource management method according to any one of claims 1 to 4, characterized in that, The first memory is a level 2 cache, and the second memory is a high-bandwidth memory.

7. A computing resource management device, applied to a computing device, the computing device comprising a processing unit, a first memory, and a second memory, characterized in that, The device includes: The setting unit is used to set the working state of the first memory to a resource occupied state when performing matrix multiplication and accumulation operations. An execution unit is used to execute a terminal write instruction for the matrix multiplication and accumulation operation; the terminal write instruction is used to write the final result of the matrix multiplication and accumulation operation into the second memory, and change the working state of the first memory from the resource occupied state to the normal state; the first memory is used as a cache of the processing unit, and the second memory is used as the main memory of the system; The setting unit is specifically used for: Configure the first memory to a privileged state so that data associated with a preset workspace in the second memory is preferentially cached in the first memory.

8. The computing resource management device according to claim 7, characterized in that, The matrix multiplication accumulation operation includes performing multiple block matrix multiplication operations on the first matrix and the second matrix to obtain the final operation result; The terminal write instruction is the write instruction corresponding to the last block matrix multiplication operation in the multiple block matrix multiplication operations.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the computing resource management method as described in any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the computing resource management method as described in any one of claims 1 to 6.

Images