Computing resource management method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of computers, and provides a computing resource management method and device, electronic equipment and a storage medium, and the method comprises the steps: setting the working state of a first memory as a resource occupation state under the condition of executing a matrix product accumulation operation; the terminal write-in instruction is used for executing the matrix product accumulation operation and is used for writing a final operation result of the matrix product accumulation operation into the second memory and converting the working state of the first memory from the resource occupation state to the conventional state. According to the method, state transition operation is coupled with a writing instruction of a final operation result through a terminal writing instruction, and it is ensured that when a matrix product accumulation operation task successfully outputs the final operation result, occupied first storage resources are reliably released. Therefore, the resource occupation risk caused by incomplete recovery is effectively avoided, and the reliability and effectiveness of computing resource management are improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a computing resource management method, apparatus, electronic device, and storage medium. Background Technology

[0002] During the MMA (Matrix Multiply Accumulate) computation, the HBM (High Bandwidth Memory) workspace is privilegedly occupied by the L2 cache to improve L2 cache hit rate. This is achieved through subsequent LDM (Load Multiple Registers) instructions that load data into the workspace and downgrade the L2 cache priority to zero. However, due to the existence of the read cache, LDM instructions cannot be guaranteed to be sent to the HBM workspace, thus failing to fully restore the state to zero after the operator finishes. Large models typically consist of many operators. When the L2 cache priority is set to "Privileged" by the MMA operator and not restored to zero, subsequent operators following the MMA may not release the L2 cache due to privileged occupation, potentially causing problems such as mutual blocking between multiple requests during operator computation.

[0003] In various model sizes, MMA operators need to handle both regular and irregular shapes. During chip design, for performance reasons, MMA instructions are typically designed with a larger execution granularity to fully utilize the computing power of Tensor Cores, while LDM instructions are designed with a relatively smaller granularity. During the execution of the MMA operators, LDM usually only considers the actual output boundaries to avoid invalid computations. Therefore, in the case of irregular shapes, there is a possibility that some areas in the allocated HBM workspace may not have their L2 cache priority correctly restored, potentially leading to instruction blocking issues for subsequent operators when the L2 cache is consistently occupied by privileged access. Summary of the Invention

[0004] This invention provides a computing resource management method, apparatus, electronic device, and storage medium to solve the defect in the prior art where irregular shapes cause the high-priority state of the secondary cache of part of the HBM workspace to not be properly restored, resulting in resource residues, which may block the execution of instructions of subsequent operators.

[0005] This invention provides a computing resource management method applied to a computing device, the computing device including a processing unit, a first memory, and a second memory, the method comprising the following steps: When performing matrix multiplication and accumulation operations, the working state of the first memory is set to resource occupied state; A terminal write instruction is executed to perform the matrix multiplication and accumulation operation; the terminal write instruction is used to write the final result of the matrix multiplication and accumulation operation into the second memory, and to change the working state of the first memory from the resource occupied state to the normal state.

[0006] According to a computing resource management method provided by the present invention, the matrix multiplication accumulation operation includes performing multiple block matrix multiplication operations on a first matrix and a second matrix to obtain the final operation result; The terminal write instruction is the write instruction corresponding to the last block matrix multiplication operation in the multiple block matrix multiplication operations.

[0007] According to a computing resource management method provided by the present invention, the step of performing multiple block matrix multiplication operations on a first matrix and a second matrix to obtain the final calculation result includes: The results of each block matrix multiplication operation between the first matrix and the second matrix are accumulated with the results in the second memory to obtain the final operation result; Specifically, the result of the first block matrix multiplication operation between the first matrix and the second matrix is ​​written into the second memory.

[0008] According to a computing resource management method provided by the present invention, the step of accumulating the results of each block matrix multiplication operation between the first matrix and the second matrix with the operation results in the second memory to obtain the final operation result includes: The result of the current block matrix multiplication operation is added to the result in the second memory to generate an updated result; Write the update operation result back to the second memory; The final calculation result is the update calculation result corresponding to the last block matrix multiplication operation.

[0009] According to a computing resource management method provided by the present invention, setting the working state of the first memory to a resource-occupied state includes: Configure the first memory to a privileged state so that data associated with a preset workspace in the second memory is preferentially cached in the first memory.

[0010] According to a computing resource management method provided by the present invention, after changing the working state of the first memory from the resource-occupied state to the normal state, the method further includes: The final calculation result is loaded from the second memory to the target register using a batch load instruction.

[0011] According to a computing resource management method provided by the present invention, the first memory is a level 2 cache and the second memory is a high-bandwidth memory.

[0012] The present invention also provides a computing resource management device applied to a computing device, the computing device including a processing unit, a first memory, and a second memory, the device comprising: The setting unit is used to set the working state of the first memory to a resource occupied state when performing matrix multiplication and accumulation operations. An execution unit is used to execute a terminal write instruction for the matrix multiplication and accumulation operation; the terminal write instruction is used to write the final result of the matrix multiplication and accumulation operation into the second memory, and change the working state of the first memory from the resource occupied state to the normal state.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the computing resource management method described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the computing resource management method as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the computing resource management method as described above.

[0016] The computing resource management method, apparatus, electronic device, and storage medium provided by this invention, when performing matrix multiplication and accumulation operations, set the working state of the first memory to a resource-occupied state; execute a terminal write instruction for the matrix multiplication and accumulation operation, which writes the final operation result of the matrix multiplication and accumulation operation to the second memory, and changes the working state of the first memory from the resource-occupied state to the normal state. This method couples the state transition operation with the write instruction for the final operation result through the terminal write instruction, ensuring that the occupied first memory resources are reliably released when the matrix multiplication and accumulation operation task successfully outputs the final operation result. Therefore, it effectively avoids the risk of resource occupation caused by incomplete recovery, and improves the reliability and effectiveness of computing resource management. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the calculation of the MMA operator in the prior art.

[0019] Figure 2 This is a schematic diagram of the area in the working space of the high-bandwidth memory that is not covered by the memory loading instructions provided by this invention.

[0020] Figure 3 This is a flowchart illustrating the computing resource management method provided by the present invention.

[0021] Figure 4 This is a schematic diagram of the structure of the computing device provided by the present invention.

[0022] Figure 5 This is a schematic diagram illustrating the degradation of the high-bandwidth memory workspace during the final write of the matrix multiplication and accumulation operation instructions provided by this invention.

[0023] Figure 6 This is a schematic diagram illustrating the matrix multiplication and accumulation operation accelerated by using a second-level cache, as provided by the present invention.

[0024] Figure 7 This is a schematic diagram of the computing resource management device provided by the present invention.

[0025] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention.

[0026] Figure label: 41: Processing unit; 42: First memory; 43: Second memory; 710: Setting unit; 720: Execution unit; 810: Processor; 820: Communication interface; 830: Memory; 840: Communication bus. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] The terms "first," "second," etc., used in this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and that the objects distinguished by "first," "second," etc., are generally of the same class.

[0029] Figure 1 This is a schematic diagram of the calculation of the MMA operator in the prior art, such as Figure 1 As shown, the MMA operator is used to calculate the multiplication between two matrices, and its calculation formula is: Here, A and B are input matrices, and C is the output matrix. Input matrix A has dimensions M×K, input matrix B has dimensions K×N, and output matrix C has dimensions M×N. As one of the most common and fundamental operators in large model training and inference, the performance of the MMA operator has a significant impact on the overall model performance. Therefore, in GPU (Graphics Processing Unit) programming, it is often necessary to utilize high-speed caching resources such as L2 cache to improve the overall throughput of the operator and thus enhance MMA performance.

[0030] In various sizes of models, MMA operators need to handle various regular or irregular shapes. During chip design, for performance reasons, the execution granularity of MMA instructions is usually designed to be larger, so as to fully utilize the computing power of the Tensor Core, while the granularity of LDM instructions is designed to be relatively smaller. Figure 2 This is a schematic diagram illustrating the area of ​​the high-bandwidth memory's working space that is not covered by the memory loading instructions provided by this invention, as shown below. Figure 2 As shown, Figure 2 The green rectangle represents the area covered by an LDM instruction, while the length and width of the blue area correspond to the actual length and width of the output matrix, respectively. During the execution of the MMA operator described above, LDM typically only considers the actual output boundaries to avoid invalid computations. However, in irregularly shaped cases, some areas (red areas) within the allocated workspace (blue area) may not have their L2 cache priority correctly restored. This could lead to instruction blocking issues for subsequent operators when the L2 cache is consistently occupied by privileged access.

[0031] To address the above problems, this invention provides a computing resource management method. Figure 3 This is a flowchart illustrating the computing resource management method provided by the present invention, as shown below. Figure 3 As shown, the method includes steps 110 and 120.

[0032] Step 110: When performing matrix multiplication and accumulation operations, set the working state of the first memory to resource occupied state.

[0033] Specifically, this method is applied to computing devices. Figure 4 This is a schematic diagram of the structure of the computing device provided by the present invention, as shown below. Figure 4 As shown, the computing device includes a processing unit 41, a first memory 42, and a second memory 43. The computing device can be a general-purpose computing system or a dedicated computing system designed for specific computing tasks, such as artificial intelligence, machine learning, and scientific computing. For example, the computing device can be a server equipped with a graphics processor, a dedicated AI (Artificial Intelligence) accelerator chip, or a system-on-a-chip (SoC) including multiple computing cores, etc. This embodiment of the invention does not specifically limit the specific type of device.

[0034] Here, the processing unit 41 is the core component for performing computational tasks. The processing unit can be a GPU (Graphics Processing Unit), etc. In a preferred embodiment of the present invention, considering the characteristics of matrix multiplication and addition operations, the processing unit 41 can specifically be a streaming multiprocessor (SM) in a GPU or a hardware acceleration unit specifically designed for performing matrix multiplication and addition operations, such as a tensor core. The processing unit 41 is responsible for interpreting and executing various instructions, including matrix multiplication and addition operation instructions and data read / write instructions.

[0035] The first memory 42 is a type of memory that has a higher access speed but typically a smaller capacity compared to the second memory 43. The first memory 42 is typically used as a cache for the processing unit to reduce the latency caused by the processing unit accessing the second memory 43. The first memory 42 can specifically be a Level 1 cache, a Level 2 cache, or a Level 3 cache, etc., and this embodiment of the invention does not specifically limit this. In the application scenario of this embodiment, the first memory 42 is preferably a Level 2 cache.

[0036] Here, the second memory 43 is a memory with a larger capacity but a slower access speed compared to the first memory 42. The second memory 43 is typically used as the system's main memory to store raw data to be processed, intermediate results during calculations, and final calculation results. The second memory 43 can specifically be Dynamic Random-Access Memory (DRAM), Double Data Rate Synchronous Dynamic Random-Access Memory (DDR SDRAM), or high-bandwidth memory, etc., and this embodiment of the invention does not specifically limit its application. In practice, one or more preset workspaces can be allocated in the second memory 43 to temporarily store intermediate data or final results of matrix multiplication and addition operations.

[0037] Specifically, when performing matrix multiplication and accumulation operations, the operating state of the first memory 42 can be set to a resource-occupied state. Here, matrix multiplication and accumulation operations refer to mathematical operations performed on one or more matrices. For example, calculating... Here, A, B, and C are all matrices, representing one of the most common and fundamental operations in deep learning model training and inference scenarios. It should be noted that the matrix multiplication and addition operation in this embodiment is not limited to strict mathematical matrix multiplication; its core is an efficient multiplication-addition fusion tensor computation paradigm. It can be extended and mapped to a series of complex tensor operations that rely on dense multiplication and addition patterns, such as linear transformations incorporating activation functions, the core parts of convolution computation, etc. These operations can all benefit from the efficient reuse and low-latency access characteristics of MMA hardware for intermediate results.

[0038] It should be noted that when performing matrix multiplication and accumulation operations, the working state of the first memory 42 is set to resource occupied state. In other words, setting the working state of the first memory 42 to resource occupied state is directly configured on the write instruction of the matrix multiplication and accumulation operation. More specifically, the configuration and activation process is as follows: First, configuration is performed on the write instruction of the matrix multiplication and accumulation operation. After the processing unit 41 completes a block matrix multiplication operation, when the data stream carrying the operation result passes through the first memory 42 to be written out and accumulated into the preset working space in the second memory 43, the working state setting of the first memory 42 takes effect. Here, the working state refers to the caching strategy mode adopted by the first memory 42 for the memory working area associated with the matrix multiplication and accumulation operation. The working state can include resource occupied state and normal state.

[0039] The resource occupancy state is a special configuration state. In this state, the first memory prioritizes caching data related to matrix calculation tasks. Specifically, when processing unit 41 needs to access data in the second memory 43, if the first memory 42 is in a resource occupancy state and the data falls within the specified priority caching range, then after the data is loaded into the first memory 42, its cache line will have a lower priority for being replaced by new data, thus allowing it to be retained more persistently. This can significantly improve the speed of subsequent repeated accesses to the data.

[0040] Here, the normal state is the default operating state of the first memory 42. In the normal state, the first memory 42 caches all data according to a standard replacement algorithm, such as the LRU (Least Recently Used) algorithm, without any special priority distinction. In specific hardware implementations, the normal state can correspond to a no-priority state.

[0041] Therefore, when performing matrix multiplication and accumulation operations, the working state of the first memory 42 is set to the resource occupied state. The purpose is to provide high-speed cache support for intermediate results stored in the working space of the second memory 43 during the subsequent matrix calculation process, thereby accelerating the entire matrix multiplication and accumulation operation process.

[0042] Step 120: Execute the terminal write instruction for the matrix multiplication and accumulation operation; the terminal write instruction is used to write the final operation result of the matrix multiplication and accumulation operation into the second memory, and change the working state of the first memory from the resource occupied state to the normal state.

[0043] Specifically, a terminal write instruction is executed to perform matrix multiplication and accumulation operations. This instruction writes the final result of the matrix multiplication and accumulation operation to the second memory and changes the operating state of the first memory from a resource-occupied state to a normal state. This step, through a dual-function terminal write instruction, safely releases the occupied computing resources while completing the computation task.

[0044] The terminal write instruction for performing matrix multiplication and accumulation operations refers to the process by which processing unit 41, such as the tensor core, completes the operation specified by the instruction code. Specifically, in this context, the terminal write instruction refers specifically to the write operation instruction responsible for writing the final operation result to the second memory 43 at the final stage of the matrix multiplication and accumulation operation process. It should be noted that a complex matrix multiplication and accumulation operation may include multiple calculation stages and corresponding multiple write operations, such as writing intermediate results, and the terminal write instruction is the last one in this series of write operations.

[0045] Here, the final result refers to the final data obtained after the entire matrix multiplication and accumulation operation is completed. For example, in In the operation, the final result is the complete matrix C.

[0046] It should be understood that the specific functions of the terminal write instruction include data writing and state restoration. The core task of the data writing function is to write the calculated final result from the register or internal cache of the processing unit 41 into the address space of the designated second memory 43. Furthermore, the function of restoring the working state of the first memory 42 is coupled into this terminal write instruction. That is, when the processing unit 41 executes the terminal write instruction, in addition to completing the data writing operation, it also simultaneously sends an internal signal to switch the working state of the first memory 42 from the previously set resource-occupied state to the normal state.

[0047] In existing technical solutions, setting and restoring cache priorities are typically handled by different instructions. For example, a matrix multiplication-addition instruction sets a high-priority state, while a subsequent load instruction restores it to a normal state. However, due to the presence of components such as read caches in the hardware architecture, or uncertainties in data access paths when processing irregular matrix multiplication-addition operations, the load instruction responsible for state restoration may not be able to completely cover all storage areas that were previously set to a high-priority state. This incompleteness of the restoration mechanism results in some resources of the first memory 42 remaining in an irregular priority state and not being completely released, which may cause resource blocking and performance interference for other computing tasks that need to access the memory. This embodiment of the invention accurately identifies the critical node of terminal write, which inevitably occurs in the matrix multiplication-addition operation process, and binds the cache priority restoration operation to the terminal write instruction. This design ensures that when the matrix multiplication-addition operation is successfully completed and the final calculation result is output, the terminal write instruction, as a process termination signal, will synchronously trigger a reset operation on the state of the first memory 42, reliably restoring it from a resource-occupied state to a normal state. This mechanism fundamentally solves the problem of incomplete priority recovery at the logical level, ensuring that system resources are always fully released after the operation is completed. This significantly improves the overall stability of system resources under complex computing tasks, effectively avoids the risk of resource occupation caused by incomplete recovery, and improves the reliability and effectiveness of computing resource management.

[0048] The method provided in this invention, when performing matrix multiplication and accumulation operations, sets the working state of the first memory to a resource-occupied state; executes a terminal write instruction for the matrix multiplication and accumulation operation, which writes the final result of the operation to the second memory, and changes the working state of the first memory from the resource-occupied state to the normal state. This method couples the state transition operation with the write instruction for the final result through the terminal write instruction, ensuring that the occupied first memory resources are reliably released when the matrix multiplication and accumulation operation successfully outputs the final result. This effectively avoids the risk of resource occupation caused by incomplete recovery and improves the reliability and effectiveness of computing resource management.

[0049] Based on the above embodiments, the matrix multiplication accumulation operation includes performing multiple block matrix multiplication operations on the first matrix and the second matrix to obtain the final operation result; The terminal write instruction is the write instruction corresponding to the last block matrix multiplication operation in the multiple block matrix multiplication operations.

[0050] Specifically, in a particular application scenario, such as matrix multiplication commonly seen in large model training, the dimensions of matrices A and B to be computed are usually enormous. The processing unit 41 can only handle a limited number of matrix sizes in a single operation, making it impossible to complete the multiplication of the entire large matrix at once. Therefore, a common and efficient computational strategy is to employ the idea of ​​block-based computation.

[0051] Specifically, in this embodiment of the invention, the matrix multiplication accumulation operation includes performing multiple block matrix multiplication operations on the first matrix and the second matrix to obtain the final operation result. This means that the original large matrix, for example, the first matrix A and the second matrix B, is logically divided into several smaller sub-matrices (blocks), and then the processing unit performs multiple multiplication operations on these sub-matrices (blocks). Each block matrix multiplication operation produces an intermediate result.

[0052] Accordingly, the entire matrix multiplication and accumulation process includes multiple operations of writing intermediate results into memory. The terminal write instruction is the write instruction corresponding to the final block matrix multiplication operation in the multiple block matrix multiplication operations.

[0053] In one alternative embodiment, at the program logic level, the timing of the last block operation can be uniquely identified through a loop counter, an iteration termination flag, or a specific program flow control mechanism. For example, in a for loop structure that performs N block operations, the write instruction executed during the Nth iteration is the terminal write instruction.

[0054] The method provided in this invention concretizes the complex matrix multiplication and accumulation operation into performing multiple block matrix multiplication operations on a first matrix and a second matrix, and explicitly defines the terminal write instruction as the write operation corresponding to the last operation in the multiple block matrix multiplication operations. By decomposing the matrix multiplication and accumulation operation task into iteratively executable block operations, a reliable technical anchor is provided for accurately embedding resource recovery operations in complex cyclic calculation processes, thereby ensuring the reliability of computing resource management in complex calculation processes.

[0055] Based on the above embodiments, performing multiple block matrix multiplication operations on the first matrix and the second matrix to obtain the final operation result includes: Step 21: The results of each block matrix multiplication operation between the first matrix and the second matrix are accumulated with the results in the second memory to obtain the final operation result; Specifically, the result of the first block matrix multiplication operation between the first matrix and the second matrix is ​​written into the second memory.

[0056] Specifically, the final result of the matrix multiplication accumulation operation is a combination of the intermediate results generated by all the block matrix multiplication operations. Specifically, performing multiple block matrix multiplication operations on the first and second matrices to obtain the final result includes: The results of each block matrix multiplication operation of the first and second matrices are accumulated with the results in the second memory to obtain the final result.

[0057] Specifically, the first matrix block multiplication operation is performed, writing the block product result of the first and second matrices into the second memory 43. During this process, the processing unit 41 calculates the first partial result and directly stores it in the pre-allocated workspace of the second memory 43. At this time, the workspace stores the initial partial result. At the instruction level, this operation can be implemented by a matrix multiplication out (MMA out) type instruction.

[0058] Subsequently, in each subsequent block matrix multiplication operation, after the processing unit 41 calculates the new partial result, it does not directly overwrite the existing data in the workspace. Instead, it performs an accumulation operation: it sums the newly calculated partial result with the matrix multiplication accumulation result currently stored in the second memory workspace, and then rewrites the updated accumulation result back into the workspace. This process corresponds to a matrix multiplication accumulation operation type instruction in instruction implementation.

[0059] It should be understood that the above calculation-accumulation process will be executed cyclically until all matrix blocks have been operated on. After the last block of matrix multiplication and accumulation operation is completed, the data stored in the workspace of the second memory 43 is the final result of the entire matrix multiplication operation.

[0060] Based on the above embodiments, step 21 includes: Step 211: The result of the current block matrix multiplication operation is added to the result in the second memory to generate an updated result; Step 212: Write the update operation result back to the second memory; The final calculation result is the update calculation result corresponding to the last block matrix multiplication operation.

[0061] Specifically, the result of the current block matrix multiplication operation is accumulated with the result in the second memory to generate an updated result. The result of the current block matrix multiplication operation refers to the matrix operation result generated in real-time by the processing unit 41 in the current calculation iteration. The result in the second memory refers to the accumulated matrix multiplication result obtained from all previous accumulation operations, stored in the workspace of the second memory 43.

[0062] After obtaining the updated result, it can be written back to the second memory. By repeatedly executing steps 211 and 212, the value in the second memory accumulates until the last block operation is completed. Therefore, the final result is the updated result corresponding to the last block matrix multiplication operation.

[0063] It is understandable that the terminal write instruction is the write instruction that writes the final update result, i.e. the final result, back to the second memory during the final block matrix multiplication operation.

[0064] The method provided in this invention accumulates the result of the current block matrix multiplication operation with the result in the second memory to generate an updated result, and then writes the updated result back to the second memory; wherein, the final result is the updated result corresponding to the last block matrix multiplication operation. This method ensures that cache resources remain resident until the final result is generated, thereby guaranteeing high-speed access performance to all intermediate results and accurately releasing resources the moment the operation is completed, achieving the best balance between computing performance and resource utilization efficiency.

[0065] Based on the above embodiments, setting the working state of the first memory to a resource-occupied state in step 110 includes: Step 1101: Configure the first memory to a privileged occupancy state so that data associated with a preset workspace in the second memory is preferentially cached in the first memory.

[0066] Specifically, in this embodiment of the invention, the first memory 42 can be configured in a privileged state so that data associated with a preset workspace in the second memory 43 is preferentially cached in the first memory 42.

[0067] Privileged occupancy is a specific, software-configurable caching strategy. When a cache line in the first memory, such as L2Cache, is set to this state, it has the highest priority in cache replacement decisions, meaning it is least likely to be replaced by regular cache replacement algorithms.

[0068] The preset workspace is a memory area in the second memory 43 that is explicitly requested by the program, such as the MMA operator, and is specifically used to store the intermediate accumulation results and the final operation results of this matrix multiplication and accumulation operation.

[0069] In one specific embodiment, it should be noted that when performing matrix multiplication and accumulation operations, the operating state of the first memory 42 is set to a privileged occupancy state. That is, setting the operating state of the first memory 42 to privileged occupancy is configured directly on the write instruction of the matrix multiplication and accumulation operation. More specifically, the configuration and activation process is as follows: First, configuration is performed on the write instruction of the matrix multiplication and accumulation operation. After the processing unit 41 completes a block matrix multiplication operation, when the data stream carrying the operation result passes through the first memory 42 to be written out and accumulated into the preset workspace in the second memory 43, the operating state setting of the first memory 42 takes effect, so that data associated with the preset workspace in the second memory 43 is preferentially cached in the first memory 42.

[0070] The method provided in this invention configures a first memory in a privileged occupancy state, so that data associated with a preset workspace in a second memory is preferentially cached in the first memory. This method ensures that all intermediate result data can continuously reside in the cache throughout the entire matrix multiplication and accumulation operation, thereby guaranteeing high-speed, low-latency access to the intermediate result data by the processing unit. Simultaneously with the completion of the matrix multiplication and accumulation operation, the privileged occupancy state can be precisely released, and cache resources can be immediately released. This achieves efficient synergy between maximizing cache hit rate and dynamically reclaiming storage resources, thereby improving the computational efficiency of computing resource management.

[0071] Based on the above embodiments, after changing the working state of the first memory from the resource-occupied state to the normal state, the method further includes: The final calculation result is loaded from the second memory to the target register using a batch load instruction.

[0072] Specifically, after the working state of the first memory 42 is changed from resource occupied state to normal state, the final result of the matrix multiplication and accumulation operation has been completely and safely written into the working space of the second memory 43. At the same time, the cache access priority of the first memory 42 has also been restored to the normal setting.

[0073] Therefore, the final calculation result can be loaded from the second memory 43 to the target register using a batch load instruction. The batch load instruction is specifically designed for efficiently reading a data block (matrix) from memory into the target register. Here, the batch load instruction can be an LDM instruction.

[0074] Here, the target register refers to the high-speed storage unit inside the processing unit 41 used for temporary data storage. The target register can be a TLR (Tensor Load Register), etc., and this embodiment of the invention does not specifically limit it.

[0075] It should be understood that the execution timing of this operation step is crucial. The startup of this timing control logic must be strictly after the state of the first memory 42 has successfully recovered to its normal state. This design ensures that the execution process of subsequent LDM instructions is completely independent and will not be affected by any residual effects of the previous privileged state. This is the core innovation of this invention compared to the prior art. Existing solutions couple the cache state degradation task to the data loading process of LDM instructions, but this mechanism has inherent defects. When processing irregularly shaped matrices causes LDM instructions to load only part of the workspace data, or when the data requested by the instruction has already been hit in the cache, the batch load instruction may not access all memory areas that need to be degraded, resulting in incomplete state degradation and leaving the risk of resource locking.

[0076] This invention fundamentally solves the aforementioned problems by introducing a terminal write instruction with explicit timing constraints. The terminal write instruction is used to write the final result of matrix multiplication and accumulation operations into the second memory 43 and change the operating state of the first memory 42 from a resource-occupied state to a normal state. Thereafter, the LDM instruction only performs its primary function of data loading, thereby optimizing data path performance while ensuring system state consistency.

[0077] Based on the above embodiments, the first memory is a level 2 cache, and the second memory is a high-bandwidth memory.

[0078] Specifically, the first memory is the L2 cache. The L2 cache is located between the L1 cache and main memory in the storage hierarchy. It has a larger capacity than the L1 cache and a faster access speed than main memory. It is an ideal choice for balancing hit rate and access latency and is very suitable for use as an intermediate result cache for large-scale operations, such as matrix multiplication and accumulation operations.

[0079] The second memory is a high-bandwidth memory. HBM integrates multiple DRAM (Dynamic Random Access Memory) chips and the processor core onto the same package substrate using 3D stacking technology, providing extremely high memory bandwidth to meet the data throughput requirements of powerful processing units like the Tensor Core.

[0080] The method provided in this embodiment of the invention uses a first memory as a level 2 cache and a second memory as a high-bandwidth memory, which can maximize hardware performance.

[0081] Based on any of the above embodiments Figure 5 This is a schematic diagram illustrating the degradation of the high-bandwidth memory workspace during the final write of the matrix multiplication and accumulation operation instructions provided by this invention. Figure 5 As shown, by recognizing the completion of the last matrix multiplication and accumulation operation instruction, the L2cache privilege occupancy of the high-bandwidth memory workspace (HBM's workspace) is restored to a state of no privilege. This fundamentally avoids the problem of incomplete degradation of the HBM workspace by LDM instructions in the event of irregular shapes in the future.

[0082] First, during the initial matrix multiplication and accumulation operation, the processing unit calculates the first partial result and writes it to the workspace of the high-bandwidth memory. The working state of the first memory associated with this write operation is set to "none," i.e., the normal state. Next, during the second, third, and subsequent matrix multiplication and accumulation operations, the partial result from each operation is accumulated with the data already stored in the high-bandwidth memory's workspace, and the updated accumulated result is written back. During these intermediate steps, the write instruction sets the working state of the first memory to a privileged occupied state to ensure that frequent access to the workspace is cached and accelerated.

[0083] Then, during the final matrix multiplication and accumulation operation, the processing unit writes the calculated final result to the working space of the high-bandwidth memory. The terminal write instruction executing this write operation simultaneously restores the working state of the first memory from privileged occupation to non-privileged, i.e., normal state, thereby reliably releasing the occupied cache resources.

[0084] Finally, after the entire matrix multiplication and accumulation operation is completed and the cache state is restored, the final result stored in the high-bandwidth memory workspace can be loaded into the processing unit's target register, such as the TLR, via the Load Multiple Registers (LDM) instruction, and ultimately used for subsequent processing or directly as output.

[0085] The method provided in this invention can ensure, at the mechanism level, that MMA operators can safely and efficiently utilize L2 cache to optimize computation and memory access while avoiding potential performance risks to subsequent operators. This method has good code generality, requires no additional steps or overhead, and its impact on operator performance can be controlled to within 1%. It also helps reduce dependence on L2 cache hardware complexity, thereby reducing hardware power consumption and area.

[0086] Based on any of the above embodiments Figure 6 This is a schematic diagram illustrating the use of a second-level cache to accelerate matrix multiplication and accumulation operations provided by this invention, as shown below. Figure 6 As shown, matrices A and B are sent to the processing unit for tensor core computation, specifically performing matrix multiplication and accumulation operations. This operation involves multiple block calculations. Partial results are written to the storage system in two ways: the initial result is written directly using a matrix multiplication and accumulation write instruction; subsequent results are written back after being accumulated with existing intermediate results using a matrix multiplication and accumulation accumulation instruction. The target storage location for these write and accumulation operations is the workspace of a high-bandwidth memory. When accessing this workspace, data is accelerated through a secondary cache. In this embodiment, the secondary cache is set to a resource-occupied state to improve the access hit rate. Upon performing the final matrix multiplication and accumulation write operation, the secondary cache state is restored to its normal state.

[0087] After all matrix multiplication and summation operations are completed and the final result is stored in the high-bandwidth memory's workspace, the computing device executes a Load Multiple Registers (LDM) instruction. The LDM instruction loads the final operation result from the workspace into the target register. Finally, the data stored in the target register can be used as the final output for subsequent processing or storage.

[0088] The computing resource management device provided by the present invention is described below. The computing resource management device described below and the computing resource management method described above can be referred to in correspondence.

[0089] Based on any of the above embodiments, the present invention provides a computing resource management device. Figure 7 This is a schematic diagram of the computing resource management device provided by the present invention, as shown below. Figure 7As shown, the device is applied to a computing device, which includes a processing unit, a first memory, and a second memory. The device includes: Setting unit 710 is used to set the working state of the first memory to a resource occupied state when performing matrix multiplication and accumulation operations. The execution unit 720 is used to execute the terminal write instruction of the matrix multiplication and accumulation operation; the terminal write instruction is used to write the final operation result of the matrix multiplication and accumulation operation into the second memory, and change the working state of the first memory from the resource occupied state to the normal state.

[0090] The apparatus provided in this invention, when performing matrix multiplication and accumulation operations, sets the working state of the first memory to a resource-occupied state; executes a terminal write instruction for the matrix multiplication and accumulation operation, which writes the final result of the matrix multiplication and accumulation operation to the second memory, and changes the working state of the first memory from the resource-occupied state to the normal state. This method couples the state transition operation with the write instruction for the final operation result through the terminal write instruction, ensuring that the occupied first memory resources are reliably released when the matrix multiplication and accumulation operation successfully outputs the final operation result. Therefore, it effectively avoids the risk of resource occupation caused by incomplete recovery, improving the reliability and effectiveness of computing resource management.

[0091] Based on any of the above embodiments, the matrix multiplication accumulation operation includes performing multiple block matrix multiplication operations on the first matrix and the second matrix to obtain the final operation result; The terminal write instruction is the write instruction corresponding to the last block matrix multiplication operation in the multiple block matrix multiplication operations.

[0092] Based on any of the above embodiments, performing multiple block matrix multiplication operations on the first matrix and the second matrix to obtain the final operation result specifically includes: An accumulation unit is used to accumulate the results of each block matrix multiplication operation between the first matrix and the second matrix with the results in the second memory to obtain the final operation result; Specifically, the result of the first block matrix multiplication operation between the first matrix and the second matrix is ​​written into the second memory.

[0093] Based on any of the above embodiments, the accumulation unit is specifically used for: The result of the current block matrix multiplication operation is added to the result in the second memory to generate an updated result; Write the update operation result back to the second memory; The final calculation result is the update calculation result corresponding to the last block matrix multiplication operation.

[0094] Based on any of the above embodiments, the setting unit 710 is specifically used for: Configure the first memory to a privileged state so that data associated with a preset workspace in the second memory is preferentially cached in the first memory.

[0095] Based on any of the above embodiments, a loading unit is further included, wherein the loading unit is specifically used for: The final calculation result is loaded from the second memory to the target register using a batch load instruction.

[0096] Based on any of the above embodiments, the first memory is a level 2 cache, and the second memory is a high-bandwidth memory.

[0097] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 8 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a computing resource management method, which includes: setting the working state of a first memory to a resource-occupied state when performing a matrix multiplication and accumulation operation; executing a terminal write instruction for the matrix multiplication and accumulation operation; the terminal write instruction being used to write the final result of the matrix multiplication and accumulation operation into a second memory, and changing the working state of the first memory from the resource-occupied state to a normal state.

[0098] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0099] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the computing resource management method provided by the above methods. The method includes: setting the working state of a first memory to a resource-occupied state when performing a matrix multiplication and accumulation operation; executing a terminal write instruction for the matrix multiplication and accumulation operation; the terminal write instruction being used to write the final operation result of the matrix multiplication and accumulation operation into a second memory, and changing the working state of the first memory from the resource-occupied state to a normal state.

[0100] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the computing resource management method provided by the above methods. The method includes: setting the working state of a first memory to a resource-occupied state when performing a matrix multiplication and accumulation operation; executing a terminal write instruction for the matrix multiplication and accumulation operation; the terminal write instruction being used to write the final operation result of the matrix multiplication and accumulation operation into a second memory, and changing the working state of the first memory from the resource-occupied state to a normal state.

[0101] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0102] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A computing resource management method applied to a computing device, the computing device comprising a processing unit, a first memory, and a second memory, characterized in that, The method includes: When performing matrix multiplication and accumulation operations, the working state of the first memory is set to resource occupied state; A terminal write instruction is executed to perform the matrix multiplication and accumulation operation; the terminal write instruction is used to write the final result of the matrix multiplication and accumulation operation into the second memory, and to change the working state of the first memory from the resource occupied state to the normal state.

2. The computing resource management method according to claim 1, characterized in that, The matrix multiplication accumulation operation includes performing multiple block matrix multiplication operations on the first matrix and the second matrix to obtain the final operation result; The terminal write instruction is the write instruction corresponding to the last block matrix multiplication operation in the multiple block matrix multiplication operations.

3. The computing resource management method according to claim 2, characterized in that, The step of performing multiple block matrix multiplication operations on the first matrix and the second matrix to obtain the final operation result includes: The results of each block matrix multiplication operation between the first matrix and the second matrix are accumulated with the results in the second memory to obtain the final operation result; Specifically, the result of the first block matrix multiplication operation between the first matrix and the second matrix is ​​written into the second memory.

4. The computing resource management method according to claim 3, characterized in that, The step of accumulating the results of each block matrix multiplication operation between the first matrix and the second matrix with the results in the second memory to obtain the final result includes: The result of the current block matrix multiplication operation is added to the result in the second memory to generate an updated result; Write the update operation result back to the second memory; The final calculation result is the update calculation result corresponding to the last block matrix multiplication operation.

5. The computing resource management method according to any one of claims 1 to 4, characterized in that, Setting the working state of the first memory to a resource-occupied state includes: Configure the first memory to a privileged state so that data associated with a preset workspace in the second memory is preferentially cached in the first memory.

6. The computing resource management method according to any one of claims 1 to 4, characterized in that, After changing the operating state of the first memory from the resource-occupied state to the normal state, the method further includes: The final calculation result is loaded from the second memory to the target register using a batch load instruction.

7. The computing resource management method according to any one of claims 1 to 4, characterized in that, The first memory is a level 2 cache, and the second memory is a high-bandwidth memory.

8. A computing resource management device, applied to a computing device, the computing device comprising a processing unit, a first memory, and a second memory, characterized in that, The device includes: The setting unit is used to set the working state of the first memory to a resource occupied state when performing matrix multiplication and accumulation operations. An execution unit is used to execute a terminal write instruction for the matrix multiplication and accumulation operation; the terminal write instruction is used to write the final result of the matrix multiplication and accumulation operation into the second memory, and change the working state of the first memory from the resource occupied state to the normal state.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the computing resource management method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the computing resource management method as described in any one of claims 1 to 7.

Images