Operator computation method, electronic device, and storage medium

Abstract

The present application relates to artificial intelligence technology, provide a kind of operator computing method, electronic equipment and storage medium, wherein method includes: original data is divided into multiple segments;Intermediate result of one segment is obtained based on the operator calculation of multiple segments in one segment, read historical intermediate result from cache, fuse historical intermediate result and the intermediate result of one segment, obtain new historical intermediate result, new historical intermediate result is written into cache, and return to the next segment of one segment is carried out operator calculation, until all segments are completed operator calculation;Read historical intermediate result from cache, while reducing the priority of historical intermediate result in cache;Based on historical intermediate result, the calculation result of operator is output.The method, electronic equipment and storage medium provided by the present application realize the timely degradation processing of historical intermediate result in a software control manner, and avoid the blocking situation of subsequent operator memory request.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an operator computation method, electronic device, and storage medium. Background Technology

[0002] To improve the computational performance of operators, a common optimization method is to set the priority of intermediate results of operator computation in the cache, thereby increasing the cache hit rate and accelerating operator computation.

[0003] However, when other operators are still being calculated after one operator has finished calculating, the above optimization method will cause the cache to be occupied by the intermediate results of the already calculated operators. Most of the cache space is difficult to be released or replaced in time, which will block the memory access requests of subsequent operators. This may result in performance degradation or even cache deadlock, causing the system to freeze. Summary of the Invention

[0004] This invention provides an operator computation method, an electronic device, and a storage medium to address the shortcomings of related technologies where operator performance optimization methods lead to difficulties in subsequent operator memory access.

[0005] This invention provides an operator calculation method, comprising:

[0006] Obtain the raw data and divide the raw data into multiple segments;

[0007] Operator calculation is performed on one of the multiple segments to obtain an intermediate result of the segment. The historical intermediate result is read from the cache, and the historical intermediate result and the intermediate result of the segment are merged to obtain a new historical intermediate result. The new historical intermediate result is written to the cache, and operator calculation is performed on the next segment of the segment until all segments have completed operator calculation.

[0008] Read the historical intermediate results from the cache, and simultaneously reduce the priority of the historical intermediate results in the cache;

[0009] Based on the historical intermediate results, the calculation result of the operator is output.

[0010] According to an operator computation method provided by the present invention, the step of reading the historical intermediate result from the cache and simultaneously reducing the priority of the historical intermediate result in the cache includes:

[0011] Send a non-privileged access request to the cache so that the cache returns the historical intermediate result based on the non-privileged access request and lowers the priority of the historical intermediate result.

[0012] According to an operator calculation method provided by the present invention, the target address in the non-privileged access request is a register address.

[0013] According to an operator calculation method provided by the present invention, writing the new historical intermediate result into the cache includes:

[0014] A privileged access request is sent to the cache so that the cache stores the new historical intermediate result based on the privileged access request, and the priority of the new historical intermediate result is controlled to be the first priority.

[0015] The operator calculation method provided by the present invention further includes:

[0016] Operator calculations are performed on the first segment among the multiple segments to obtain an intermediate result of the first segment. The intermediate result is then written into the target memory block as a new historical intermediate result. Operator calculations are then performed on the next segment of the first segment.

[0017] According to an operator calculation method provided by the present invention, the size of the target memory block is consistent with the size of the cache.

[0018] According to the present invention, an operator calculation method is provided, wherein the operator is a matrix multiplication and accumulation operator.

[0019] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the operator calculation method as described above.

[0020] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the operator computation method as described above.

[0021] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the operator calculation method as described above.

[0022] The operator computation method, electronic device, and storage medium provided by this invention, after all segments have completed operator computation, simultaneously reduce the priority of historical intermediate results in the cache while reading them. This software-controlled approach achieves timely degradation of historical intermediate results, freeing up cache space for subsequent operator computations. It ensures operator computation performance while preventing memory access request blocking in subsequent operators, thus guaranteeing the performance optimization effect of multi-operator computation. The software-controlled priority degradation method is more versatile and less costly than hardware-based methods, avoiding increased cache hardware design complexity and reducing hardware power consumption and area. Furthermore, reducing the priority of historical intermediate results in the cache while reading them eliminates the need for additional steps, avoiding new cost overhead for operator computation and ensuring operator computation performance. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating the operator calculation method provided by the present invention.

[0025] Figure 2 This is a schematic diagram of the matrix multiplication and accumulation data structure provided by the present invention.

[0026] Figure 3 This is a flowchart illustrating the matrix multiplication and accumulation calculation method provided by the present invention.

[0027] Figure 4 This is a schematic diagram showing the usage of the secondary cache before and after priority downgrading provided by the present invention.

[0028] Figure 5 This is a schematic diagram of the operator computing device provided by the present invention.

[0029] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0031] In the field of artificial intelligence, operator optimization is a key step in improving the efficiency of model training and inference, especially in computing platforms centered around AI chips. The performance of operators directly affects the running speed and responsiveness of the entire AI model.

[0032] Taking the Matrix Multiply-Accumulate (MMA) operator as an example, the mainstream optimization strategy for MMA operators is to fully utilize on-chip cache resources, especially the L2 cache. Specifically, when executing the MMA operator, a dedicated temporary storage space, usually called the workspace, is allocated in memory to temporarily store intermediate results generated by the MMA computation for accumulation. To ensure the efficiency of reading and writing these intermediate results, developers use cache control strategies to set access to this workspace memory region as high cache priority, thereby increasing the hit rate of memory access operations in the L2 cache. This allows the Tensor Core (Tcore) to quickly complete the accumulation operation, significantly enhancing the overall performance of the MMA operator.

[0033] However, the optimization methods described above for the performance of a single MMA operator can lead to serious technical problems in complex artificial intelligence models.

[0034] A complete artificial intelligence model may consist of hundreds or thousands of different types of operators executed sequentially in a specific order. After the execution of a preceding MMA operator, its associated workspace is useless data for subsequent operators. However, because the corresponding cached data blocks in the secondary cache of this workspace are given high priority, these cached data blocks are not immediately downgraded or released after the execution of the MMA operator, but continue to occupy most of the secondary cache space.

[0035] When subsequent operators begin execution, they also require a L2 cache to accelerate their own instruction and data access. However, at this time, most of the L2 cache space is occupied by useless workspace data from the preceding MMA operators. The hardware's automatic replacement mechanism for the L2 cache is ineffective due to the high priority assigned to cached data blocks, causing multiple memory access requests from subsequent operators to be continuously blocked. In extreme cases, this resource contention can evolve into a L2 cache deadlock, where subsequent operators wait for L2 cache resources to be released, but high-priority data blocks in the L2 cache cannot be replaced due to the lack of an effective degradation mechanism, ultimately causing the AI ​​chip to freeze and the system to become unresponsive.

[0036] To address the aforementioned issues, since the secondary cache is invisible to developers and is mostly controlled by hardware, developers have attempted to use hardware design to automatically identify and downgrade these no longer used cached data blocks. However, this approach has become extremely difficult because the hardware cannot predict the subsequent operator types and operator execution durations. Furthermore, as artificial intelligence models become increasingly complex, the complexity of hardware design has also increased, making it difficult to achieve automatic downgrading through hardware design.

[0037] Based on this, the present invention provides an operator computation method. This method can be applied to artificial intelligence chips, which can be graphics processing units (GPUs), general-purpose computing on graphics processing units (GPGPUs), tensor processing units (TPUs), etc.

[0038] Figure 1 This is a flowchart illustrating the operator calculation method provided by the present invention, as shown below. Figure 1 As shown, the method includes:

[0039] Step 110: Obtain the raw data and divide the raw data into multiple segments.

[0040] Here, raw data refers to the input data used for operator computation. The operator can be an MMA operator, or other operators that require the application of intermediate data obtained from previous segment computations in segmented computations, such as the Softmax function.

[0041] The type and structure of the raw data can be determined according to different application scenarios. Taking the MMA operator as an example, MMA is often used in the computation of fully connected layers, convolutional layers, attention layers, etc. Specifically, in image processing scenarios, such as in image classification models, the raw data of the MMA operator can be image features obtained by feature extraction from image data; specifically in speech processing scenarios, such as in speech recognition models, the raw data of the MMA operator can be speech features obtained by feature extraction from speech data; specifically in text processing scenarios, such as in text generation models, the raw data of the MMA operator can be text features obtained by feature extraction from text data. This embodiment of the invention does not impose specific limitations on these aspects.

[0042] The original data is usually quite large. To facilitate operator calculations, the original data can be divided before the calculations. Thus, the original data can be divided into multiple smaller data blocks, which are referred to as fragments. It can be understood that multiple fragments can be combined to form the original data, and the size of a single fragment is much smaller than the size of the original data itself.

[0043] Step 120: Perform operator calculation based on one of the multiple segments to obtain an intermediate result of the segment. Read the historical intermediate result from the cache, merge the historical intermediate result and the intermediate result of the segment to obtain a new historical intermediate result, write the new historical intermediate result into the cache, and return to perform operator calculation on the next segment of the segment until all segments have completed operator calculation.

[0044] Specifically, in the process of operator calculation on the original data, operator calculation can be performed segment by segment at the granularity.

[0045] After performing operator calculations on one of the segments, the result of the operator calculation for that segment is obtained, which is referred to here as the intermediate result. Subsequently, in order to merge this intermediate result with the intermediate results obtained from the previous segment calculations, a memory access request can be sent to the cache to read the pre-stored intermediate result based on the previous segment calculations, which is referred to here as the historical intermediate result.

[0046] After retrieving the historical intermediate results from the cache, the retrieved historical intermediate results can be merged with the intermediate results obtained in the current calculation to obtain the merged result. Subsequently, the merged result can be written into the cache as a new historical intermediate result. At this point, the operator calculation process for one segment is complete, and the process can return to execute the operator calculation for the next segment, and so on, until all segments have completed the operator calculation.

[0047] Understandably, during the segment-by-segment operator computation process, each segment's operator computation process may involve reading and writing historical intermediate results, which may be frequently read and written during operator computation. Therefore, historical intermediate results typically have a high priority in the cache, ensuring that they can be retained in the cache for a longer period during operator computation. This improves the cache hit rate of historical intermediate results, reduces the number of accesses to main memory to read and write historical intermediate results, reduces the time latency caused by reading and writing historical intermediate results, and improves operator computation efficiency.

[0048] Step 130: Read the historical intermediate results from the cache, and at the same time reduce the priority of the historical intermediate results in the cache.

[0049] Specifically, after the operator calculations have been completed for all segments, historical intermediate results can be read from the cache, and based on the historical intermediate results read, the calculation results of the operators for the original data can be determined and output.

[0050] It's understandable that this involves reading historical intermediate results from the cache. This is the last time a historical intermediate result is read from the cache during the operator computation process. Afterward, the historical intermediate results cached in the cache are irrelevant to subsequent operator computations and will not be accessed again. Therefore, while reading historical intermediate results from the cache, it's possible to control the cache to lower the priority of historical intermediate results.

[0051] Here, reading historical intermediate results from the cache and controlling the cache to lower the priority of historical intermediate results are performed synchronously. For example, a single instruction can be used to read historical intermediate data while simultaneously performing priority downgrading processing on the historical intermediate data.

[0052] After prioritizing historical intermediate data, the historical intermediate data in the cache can participate in the automatic replacement mechanism in the cache, so that it can be replaced in time during the calculation of subsequent operators, thereby providing cache space for the intermediate results of subsequent operators.

[0053] Step 140: Based on the historical intermediate results, output the calculation result of the operator.

[0054] Specifically, after reading the historical intermediate results, the calculation result of the operator can be determined based on this and the calculation result of the operator can be output.

[0055] In the method provided in this embodiment of the invention, after all segments have completed operator calculations, the priority of historical intermediate results in the cache is reduced while reading them from the cache. This software-controlled approach achieves timely degradation of historical intermediate results, freeing up cache space for subsequent operator calculations. This ensures operator calculation performance while preventing memory access requests from subsequent operators from blocking, thus guaranteeing the performance optimization effect of multi-operator calculations. The software-controlled priority degradation method is more versatile and less costly than hardware-based methods, avoiding increased complexity in cache hardware design and reducing hardware power consumption and area. Furthermore, reducing the priority of historical intermediate results in the cache while reading them eliminates the need for additional steps, avoiding new cost overhead for operator calculations and ensuring operator calculation performance.

[0056] Based on the above embodiments, step 130, which involves reading the historical intermediate result from the cache and simultaneously lowering the priority of the historical intermediate result in the cache, includes:

[0057] Send a non-privileged access request to the cache so that the cache returns the historical intermediate result based on the non-privileged access request and lowers the priority of the historical intermediate result.

[0058] Specifically, in order to control the cache to lower the priority of historical intermediate results while reading them from the cache, a non-privileged access request can be sent to the cache.

[0059] Here, a non-privileged access request is equivalent to a "none privileged" access request. When requesting access to historical intermediate results from the cache, a non-privileged access request can indicate to the cache that the requested historical intermediate result is data for one-time or short-term use and will not be needed in the future. For example, a non-privileged access request can be a memory access request carrying the "none" flag.

[0060] Upon receiving a non-privileged access request, the cache can retrieve and return the requested historical intermediate result. Simultaneously, it can lower the priority of this historical intermediate result in the cache. Thus, the historical intermediate result changes from a high-priority to a low-priority cache, moving from being a fixed cache item to participating in the cache's automatic replacement mechanism, and is therefore promptly released or replaced, providing more cache space for subsequent operator computations.

[0061] In the method provided in this embodiment of the invention, by sending a non-privileged access request to the cache, the priority of the historical intermediate results can be downgraded while reading the historical intermediate results. This avoids introducing new steps and incurring new costs for operator computation, while also avoiding the impact of operator computation on subsequent operators.

[0062] Based on any of the above embodiments, the target address in the non-privileged access request is a register address.

[0063] Specifically, a target address can be included in a non-privileged access request. The target address refers to the address where the cache returns the data requested by the non-privileged access request; that is, the address where the cache outputs the historical intermediate results of the lookup hit.

[0064] For cases where the target address in a non-privileged access request is a register address, historical intermediate results can be read from the cache and then read into the register pointed to by the register address, such as the thread local register (TLR).

[0065] Therefore, after the operator calculation is completed for all segments, the historical intermediate results in the cache can be read into the register through a non-privileged access request, and the calculation result of the operator can be determined and output based on the historical intermediate results through the register.

[0066] Based on any of the above embodiments, step 120, writing the new historical intermediate result into the cache, includes:

[0067] A privileged access request is sent to the cache so that the cache stores the new historical intermediate result based on the privileged access request, and the priority of the new historical intermediate result is controlled to be the first priority.

[0068] Specifically, during the operator calculation process segment by segment, for segments that are not the first to be calculated, after the calculation of the segment is completed and the intermediate result of the segment is obtained, the historical intermediate results of the previous segments can be read from the cache.

[0069] After reading the historical intermediate results, the historical intermediate results can be merged with the intermediate results of the segment to obtain the historical intermediate results of the segment, that is, the new historical intermediate results.

[0070] Building upon this, new historical intermediate results can be written to the cache via privileged access requests. These privileged access requests, also known as privileged access requests, request access to historical intermediate results from the cache while simultaneously indicating to the cache that the requested results are long-term, frequently used data with a high probability of reuse. For example, a privileged access request can be a memory access request carrying the "privileged" flag.

[0071] Upon receiving a privileged access request, the cache can write the new historical intermediate result to the location specified in the privileged access request within the cache. Simultaneously, the priority of the new historical intermediate result requested by the privileged access request in the cache can be set, for example, to first priority. Here, first priority refers to a higher priority, specifically ensuring that the data is stably cached in the cache and is not easily replaced by other data loaded subsequently.

[0072] In this way, during the operator calculation process segment by segment, each new historical intermediate result is written to the cache. The historical intermediate results in the cache always maintain the first priority. Thus, during the operator calculation process, the historical intermediate results can be stably cached in the cache, thereby saving time for frequent reading and writing of historical intermediate results and improving the efficiency of operator calculation.

[0073] Based on any of the above embodiments, the method further includes:

[0074] Operator calculations are performed on the first of the multiple segments to obtain an intermediate result for the first segment. The intermediate result is then written into the target memory block, and operator calculations are performed on the next segment of the first segment.

[0075] Specifically, during the segment-by-segment operator computation, for the first computed segment (i.e., the first segment among multiple segments), since there are no segments preceding it, the cache does not contain historical intermediate results for segments preceding the first segment. Therefore, after performing operator computation based on the first segment and obtaining its intermediate result, there is no need to read historical intermediate results from the cache; instead, the intermediate result of the first segment is directly written into the target memory block as the new historical intermediate result. Here, the target memory block is the memory block allocated for storing intermediate results generated during the operator computation.

[0076] Then, the operator calculation for the second segment can be returned. After obtaining the intermediate result of the second segment, the historical intermediate result can be read from the cache. Here, since the historical intermediate result of the first segment was written to the target memory block, the cache cannot hit this historical intermediate result. Therefore, the cache can load the historical intermediate result from the target memory block and then return the historical intermediate result from the cache, thus realizing the reading of the historical intermediate result. Here, the historical intermediate result can be merged with the intermediate result of the second segment to obtain a new historical intermediate result, and then the new historical intermediate result is written to the cache, and so on, until all segments have completed the operator calculation.

[0077] Based on any of the above embodiments, the size of the target memory block is consistent with the size of the cache.

[0078] Specifically, when requesting a memory block to store intermediate results for an operator, a memory block of the same size as the cache can be requested as the target memory block. Requesting a target memory block of the same size as the cache allows for the maximum possible loading of historical intermediate results from the target memory block into the cache during operator computation, thereby maximizing cache utilization and ensuring operator computation performance.

[0079] Based on any of the above embodiments, the operator is a matrix multiplication and accumulation operator.

[0080] Specifically, the operator calculation methods in the above embodiments can be applied to the calculation of matrix multiplication and accumulation MMA operators.

[0081] For example, Figure 2 This is a schematic diagram of the matrix multiplication and accumulation data structure provided by the present invention. For example... Figure 2 As shown, for the MMA operator, the original data includes two matrices, matrix A and matrix B, where the size of matrix A is... The size of matrix B is Performing MMA calculations on matrices A and B yields matrix C, which has a size of [missing value]. To facilitate MMA calculations, matrices A and B can be partitioned separately, for example, in... Figure 2 In this context, the blocks in matrices A and B can be considered as partitioned segments. During MMA calculations, matrix multiplication can be performed on the segment labeled A11 in matrix A and the segment labeled B11 in matrix B, and the result of this matrix multiplication can be considered an intermediate result. After waiting for the segment labeled A12 in matrix A and the segment labeled B21 in matrix B to be multiplied, the previously calculated intermediate result can be added to the result of the current matrix multiplication, and this sum can be used as a data block in matrix C. This can be understood as... Figure 2In the matrix, squares filled with the same color in matrix A and matrix B represent intermediate results generated during the MMA calculation process that need to be accumulated.

[0082] In addition, a target memory block can be pre-allocated in memory to store intermediate results generated by MMA calculations, and the size of the target memory block is consistent with the size of the L2 cache.

[0083] Based on this, Figure 3 This is a flowchart illustrating the matrix multiplication and accumulation calculation method provided by the present invention, as shown below. Figure 3 As shown, the computation of the MMA operator may include the following steps:

[0084] First, input matrix A and matrix B.

[0085] The Tensor Core, used for MMA computation, can load data segment by segment. It first loads the first segment, which in this case comes from the first segment in matrix A and the first segment in matrix B. After performing MMA computation on the first segment, the Tensor Core obtains the intermediate result for that segment and stores it as a historical intermediate result in the target memory block.

[0086] Subsequently, the tensor core loads the second segment and performs MMA calculations on the second segment to obtain the intermediate result of the second segment. Then, it reads the historical intermediate result from the target memory block and accumulates the historical intermediate result with the intermediate result of the second segment. The accumulated result is then written into the target memory block as the new historical intermediate result. This process is repeated until all segments have completed the MMA calculation.

[0087] During fragment-by-fragment MMA computation, both the process of reading historical intermediate results from the target memory block and the process of writing new historical intermediate results to the target memory block involve the L2 cache. Therefore, during the read and write operations of historical intermediate results, their priority in the L2 cache can be set. By increasing the priority of historical intermediate results in the L2 cache, for example, by setting them to a higher priority (Privileged) level, the L2 cache hit rate can be achieved very high when accessing the target memory block. For example, when both the target memory block and the L2 cache block are 512B, the measured cache hit rate can reach over 90%.

[0088] After all segments have completed the MMA calculation, the historical intermediate results in the target memory block can be read into the register using the Load Matrix instruction. These historical intermediate results can be understood as the final MMA calculation result. During this process, the L2 cache still participates, and when reading the historical intermediate results into the register—that is, when executing the last read request operation—it can downgrade the high priority previously set for the historical intermediate results in the L2 cache.

[0089] After reading the historical intermediate results into the register, the historical intermediate results can be output as the calculation result through the Store Matrix instruction, thus obtaining matrix C. At this point, the entire MMA operator calculation is complete.

[0090] in, Figure 4 This diagram illustrates the occupancy of the secondary cache before and after priority downgrading, as provided by this invention. Each square in the diagram represents an entry. Squares filled with diagonal lines represent entries occupied by historical intermediate results of MMA calculations, while blank squares represent entries occupied by historical intermediate results not used for MMA calculations. The secondary cache arranged at the top represents the secondary cache during the segment-by-segment MMA calculation process, while the secondary cache arranged at the bottom represents the secondary cache after all segments have completed MMA calculations and priority downgrading has been executed.

[0091] like Figure 4 As shown, all entries in the target memory block of the MMA are occupied by the historical intermediate results of the MMA computation; during the segment-by-segment MMA computation, most entries in the L2 cache are occupied by the historical intermediate results of the MMA computation. However, after priority downgrading, most entries in the L2 cache are released and are no longer occupied by the historical intermediate results of the MMA computation, providing a large amount of cache space for subsequent operator computations.

[0092] The operator calculation device provided by the present invention is described below. The operator calculation device described below can be referred to in correspondence with the operator calculation method described above.

[0093] Figure 5 This is a schematic diagram of the operator computing device provided by the present invention, as shown below. Figure 5 As shown, the device includes:

[0094] The segmentation unit 510 is used to acquire raw data and divide the raw data into multiple segments;

[0095] The segment calculation unit 520 is used to perform operator calculation based on one of the multiple segments to obtain an intermediate result of the segment, read the historical intermediate result from the cache, merge the historical intermediate result and the intermediate result of the segment to obtain a new historical intermediate result, write the new historical intermediate result into the cache, and return to perform operator calculation on the next segment of the segment until all segments have completed operator calculation.

[0096] The read degradation unit 530 is used to read the historical intermediate result from the cache and simultaneously reduce the priority of the historical intermediate result in the cache.

[0097] The output unit 540 is used to output the calculation result of the operator based on the historical intermediate results.

[0098] In the apparatus provided in this embodiment of the invention, after all segments have completed operator calculations, the priority of historical intermediate results in the cache is reduced while reading them from the cache. This software-controlled approach achieves timely degradation of historical intermediate results, freeing up cache space for subsequent operator calculations. This ensures operator calculation performance while preventing memory access requests from subsequent operators from blocking, thus guaranteeing the performance optimization effect of multi-operator calculations. The software-controlled priority degradation method is more versatile and less costly than hardware-based methods, avoiding increased complexity in cache hardware design and reducing hardware power consumption and area. Furthermore, reducing the priority of historical intermediate results in the cache while reading them eliminates the need for additional steps, avoiding new cost overhead for operator calculations and ensuring operator calculation performance.

[0099] Based on any of the above embodiments, the read degradation unit is specifically used for:

[0100] Send a non-privileged access request to the cache so that the cache returns the historical intermediate result based on the non-privileged access request and lowers the priority of the historical intermediate result.

[0101] Based on any of the above embodiments, the target address in the non-privileged access request is a register address.

[0102] Based on any of the above embodiments, the fragment calculation unit is specifically used for:

[0103] A privileged access request is sent to the cache so that the cache stores the new historical intermediate result based on the privileged access request, and the priority of the new historical intermediate result is controlled to be the first priority.

[0104] Based on any of the above embodiments, the fragment calculation unit is further configured to:

[0105] Operator calculations are performed on the first segment among the multiple segments to obtain an intermediate result of the first segment. The intermediate result is then written into the target memory block as a new historical intermediate result. Operator calculations are then performed on the next segment of the first segment.

[0106] Based on any of the above embodiments, the size of the target memory block is consistent with the size of the cache.

[0107] Based on any of the above embodiments, the operator is a matrix multiplication and accumulation operator.

[0108] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute an operator calculation method, which includes:

[0109] Obtain the raw data and divide the raw data into multiple segments;

[0110] Operator calculation is performed on one of the multiple segments to obtain an intermediate result of the segment. The historical intermediate result is read from the cache, and the historical intermediate result and the intermediate result of the segment are merged to obtain a new historical intermediate result. The new historical intermediate result is written to the cache, and operator calculation is performed on the next segment of the segment until all segments have completed operator calculation.

[0111] Read the historical intermediate results from the cache, and simultaneously reduce the priority of the historical intermediate results in the cache;

[0112] Based on the historical intermediate results, the calculation result of the operator is output.

[0113] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0114] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer being able to execute the operator calculation method provided by the above methods, the method comprising:

[0115] Obtain the raw data and divide the raw data into multiple segments;

[0116] Operator calculation is performed on one of the multiple segments to obtain an intermediate result of the segment. The historical intermediate result is read from the cache, and the historical intermediate result and the intermediate result of the segment are merged to obtain a new historical intermediate result. The new historical intermediate result is written to the cache, and operator calculation is performed on the next segment of the segment until all segments have completed operator calculation.

[0117] Read the historical intermediate results from the cache, and simultaneously reduce the priority of the historical intermediate results in the cache;

[0118] Based on the historical intermediate results, the calculation result of the operator is output.

[0119] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the operator computation methods provided by the methods described above, the method comprising:

[0120] Obtain the raw data and divide the raw data into multiple segments;

[0121] Operator calculation is performed on one of the multiple segments to obtain an intermediate result of the segment. The historical intermediate result is read from the cache, and the historical intermediate result and the intermediate result of the segment are merged to obtain a new historical intermediate result. The new historical intermediate result is written to the cache, and operator calculation is performed on the next segment of the segment until all segments have completed operator calculation.

[0122] Read the historical intermediate results from the cache, and simultaneously reduce the priority of the historical intermediate results in the cache;

[0123] Based on the historical intermediate results, the calculation result of the operator is output.

[0124] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0125] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An operator calculation method, characterized in that, include: Obtain the raw data and divide the raw data into multiple segments; Operator calculation is performed on one of the multiple segments to obtain an intermediate result of the segment. The historical intermediate result is read from the cache, and the historical intermediate result and the intermediate result of the segment are merged to obtain a new historical intermediate result. The new historical intermediate result is written to the cache, and operator calculation is performed on the next segment of the segment until all segments have completed operator calculation. Read the historical intermediate results from the cache, and simultaneously reduce the priority of the historical intermediate results in the cache; Based on the historical intermediate results, the calculation result of the operator is output.

2. The operator calculation method according to claim 1, characterized in that, The step of reading the historical intermediate results from the cache and simultaneously lowering the priority of the historical intermediate results in the cache includes: Send a non-privileged access request to the cache so that the cache returns the historical intermediate result based on the non-privileged access request and lowers the priority of the historical intermediate result.

3. The operator calculation method according to claim 2, characterized in that, The target address in the non-privileged access request is a register address.

4. The operator calculation method according to claim 1, characterized in that, The step of writing the new historical intermediate results into the cache includes: A privileged access request is sent to the cache so that the cache stores the new historical intermediate result based on the privileged access request, and the priority of the new historical intermediate result is controlled to be the first priority.

5. The operator calculation method according to any one of claims 1 to 4, characterized in that, Also includes: Operator calculations are performed on the first segment among the multiple segments to obtain an intermediate result of the first segment. The intermediate result is then written into the target memory block as a new historical intermediate result. Operator calculations are then performed on the next segment of the first segment.

6. The operator calculation method according to claim 5, characterized in that, The size of the target memory block is the same as the size of the cache.

7. The operator calculation method according to any one of claims 1 to 4, characterized in that, The operator is a matrix multiplication and accumulation operator.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the operator computation method as described in any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the operator computation method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the operator computation method as described in any one of claims 1 to 7.

Images