Neural processing unit, computing task execution method, processor and computer equipment

By setting the first operation module in the neural processing unit to perform matrix vector multiplication operation on the data transmission path, the problem of degradation in the calculation efficiency caused by the key-value cache occupies RAM in the artificial intelligence model is solved, and efficient calculation and low power consumption are achieved. Solution.

CN119089953APending Publication Date: 2024-12-06SMARTER SILICON (SHANGHAI) TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411019603.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

During the decoder-side calculation process of large-scale artificial intelligence models, the size of RAM occupied by the key-value cache increases significantly with the increase in the number of tokens, resulting in a decrease in computing efficiency. Traditional solutions require distributed computing of softmax functions and require larger ones. Capacity RAM to cache KV data, increasing cost and power consumption.

Method used

By setting the first operation module in the neural processing unit to perform matrix vector multiplication operations on the data transmission path, the data transmission and cache pressure are reduced, and the calculation of the large model is split into the matrix processing unit and the vector processing unit using a distributed computing method. Perform and execute calculation tasks in parallel to reduce system bandwidth pressure.

Benefits of technology

It significantly saves RAM storage space, eliminates some overhead of data transmission, reduces power consumption, improves processing speed, and reduces implementation costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089953A_ABST
    Figure CN119089953A_ABST
Patent Text Reader

Abstract

The invention relates to the field of artificial intelligence, and discloses a neural processing unit, a computing task execution method, a processor and computer equipment. The neural processing unit comprises a random access memory and a first operation module. And the first operation module is configured to execute a first type of calculation task on the received target to-be-processed data to obtain a first operation result, and store the first operation result to the random access memory. Wherein the first operation module is arranged on a data transmission path between the random access memory and the data transmission interface of the neural processing unit.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and specifically to a neural processing unit, a computing task execution method, a processor, and a computer device. Background Art

[0002] In the reasoning process of large-scale artificial intelligence models, the calculation on the decoder side is usually performed through the key-value (KV) cache method. As the number of tokens processed by the model increases, the size of the RAM (random access memory) occupied by the required KV cache also increases accordingly. Due to the limited capacity of the RAM integrated on the chip, when the number of tokens exceeds the range that the KV cache of a single attention layer can accommodate, the computing efficiency will drop significantly.

[0003] To address this challenge, researchers have proposed a "flash attention" solution, which solves the problem of insufficient memory by performing tiling calculations on KV to adapt to the capacity limitations of the chip's internal RAM. However, this solution requires distributed calculations of the softmax function, and a large amount of RAM must still be maintained inside the chip to cache KV data. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a neural processing unit, a computing task execution method, a processor and a computer device, which can eliminate part of the overhead of the neural processing unit and reduce power consumption.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] An embodiment of the present application provides a neural processing unit for executing computing tasks of an artificial intelligence model, including: a random access memory; a first operation module, configured to execute a first type of computing task on the received target data to be processed, obtain a first operation result, and store the first operation result in the random access memory; wherein the first operation module is arranged on the data transmission path between the random access memory and the data transmission interface of the neural processing unit.

[0007] In some embodiments of the present application, the target data to be processed only participates in the first type of computing tasks; and / or the first type of computing tasks at least includes matrix-vector multiplication operations.

[0008] In some embodiments of the present application, the first operation module includes: a matrix processing unit and n vector processing units; n is greater than or equal to 1; each of the vector processing units is configured to perform the first type of computing task on at least part of the sub-data in the target data to be processed, obtain sub-operation results, and summarize the sub-operation results to the matrix processing unit; the matrix processing unit is configured to merge the sub-operation results of each of the vector processing units to obtain the first operation result.

[0009] In some embodiments of the present application, n transmission paths are arranged between the memory and the memory access module; and each of the vector processing units is arranged on a corresponding transmission path.

[0010] In some embodiments of the present application, the matrix processing unit is further configured to send vectors, configuration information and start commands to the n vector processing units; wherein the start command is used to control the vector processing unit to start multiplication and addition operations; the vectors and the configuration information are required for the vector processing unit to perform the first type of computing tasks; the configuration information includes: operation type and number of operations.

[0011] In some embodiments of the present application, each of the vector processing units is further configured to determine the position of each of the sub-data in the matrix of the target data to be processed according to the address of each of the sub-data in the memory; based on the position of each of the sub-data in the matrix, perform vector multiplication operation on each of the sub-data to obtain the sub-operation result; the matrix processing unit is further configured to merge the sub-operation results of each of the vector processing units based on the position of each of the sub-data in the matrix to obtain the first operation result.

[0012] In some embodiments of the present application, each of the vector processing units is further configured to count the number of the sub-operation results; when the number of the sub-operation results reaches a threshold, all of the sub-operation results are returned to the matrix processing unit.

[0013] In some embodiments of the present application, the neural processing unit also includes: a second computing unit, connected to the random access memory, and configured to perform a second type of computing task based on the first computing result in the random access memory.

[0014] An embodiment of the present application also provides a method for executing computing tasks of an artificial intelligence model, including: obtaining target data to be processed; executing a first type of computing task on the target data to be processed to obtain a first computing result; caching the first computing result to participate in other computing tasks; wherein the target data to be processed only participates in the first type of computing task.

[0015] In some embodiments of the present application, a first type of computing task is performed on the target data to be processed to obtain a first computing result, including: performing the first type of computing task on the target data to be processed input through n transmission paths of the artificial intelligence model respectively to obtain sub-computation results; wherein the target data to be processed input through each transmission path is divided into multiple sub-data, and the first type of computing task is performed multiple times; and the sub-computation results are merged to obtain the first computing result.

[0016] In some embodiments of the present application, after caching the first operation result, the method further includes: executing a second type of computing task according to the cached first operation result.

[0017] An embodiment of the present application also provides a processor, including: the neural processing unit described in the above scheme.

[0018] An embodiment of the present application also provides a computer device, including: a memory, and the processor described in the above solution.

[0019] It is understandable that by setting up the first operation module on the data transmission path, the target data to be processed can be operated conveniently without transmitting the target data to be processed to other operation modules, thereby eliminating part of the data transmission overhead and reducing power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Schematic diagram of the structure of the neural processing unit provided in the embodiment of the present application Figure 1 ;

[0021] Figure 2 Schematic diagram of the structure of the neural processing unit provided in the embodiment of the present application Figure 2 ;

[0022] Figure 3 A schematic diagram of the structure of a first computing module provided in an embodiment of the present application;

[0023] Figure 4 Schematic diagram of the structure of the neural processing unit provided in the embodiment of the present application Figure 3 ;

[0024] Figure 5 A schematic diagram of the operation flow of a neural processing unit provided in an embodiment of the present application;

[0025] Figure 6 Schematic diagram of the implementation process of the computing task execution method provided in the embodiment of the present application Figure 1 ;

[0026] Figure 7 Schematic diagram of the implementation process of the computing task execution method provided in the embodiment of the present application Figure 2 ;

[0027] Figure 8 Schematic diagram of the implementation process of the computing task execution method provided in the embodiment of the present application Figure 3 ;

[0028] Fig. 9 A schematic diagram of the structure of a processor provided in an embodiment of the present application;

[0029] Fig.10 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0031] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. The terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific order for the objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing this application and are not intended to limit this application.

[0033] It should be noted that the calculation of large models is usually limited by bandwidth, which means that the neural processing unit (NPU) system needs to be connected to the system bus (SOC BUS) through multiple bus interfaces. During the calculation process of the decoder, the traditional practice is to read the model parameters and key-value (KV) cache data into the RAM inside the chip for temporary storage, and then the internal controller sends this data to the centralized computing unit for processing. However, since the system bandwidth is usually much lower than the processing power of the internal computing unit, this will result in inefficient use of the computing unit during the calculation process of the decoder.

[0034] In addition, in order to ensure the continuity of data processing, the internal RAM usually needs to adopt a "ping-pong" operation mode, that is, while one memory area is processing data, another memory area is preparing the next batch of data. This operation requires the RAM capacity to be at least twice the required cache capacity to facilitate alternating use. Under this traditional scheme, the RAM capacity requirement is very high, especially considering that the token window length of current large models is usually very long. For example, for a model with a 16K token length, the internal RAM used to cache KV data needs to have a large capacity.

[0035] In addition, in response to the problem of internal RAM capacity limitation, the proposed "flash attention" solution can indeed alleviate the storage pressure caused by the increase in the number of tokens. However, this solution requires additional distributed softmax calculation units and is accompanied by more complex control logic, which will introduce certain costs in actual deployment.

[0036] Figure 1 This is an optional structural diagram of a neural processing unit provided in an embodiment of the present application. Figure 1 As shown, the neural processing unit 50 includes: a first operation module 10 and a random access memory 30.

[0037] In the embodiment of the present application, the first operation module 10 is configured to perform a first type of computing task on the received target data to be processed, obtain a first computing result, and store the first computing result in a random access memory 30. The random access memory 30 may be a RAM, configured to store data (including the first computing result) required for the calculation in the neural processing unit 50.

[0038] In the embodiment of the present application, the initial data of the first type of calculation input externally, as well as the intermediate data in the process of executing the first type of calculation task, will not be reused in other types of calculations, and therefore may not be cached in the random access memory 30. In this way, the storage space of the random access memory 30 can be significantly saved, and at the same time, part of the overhead of the data cache is eliminated, and the time and power consumption caused by accessing the random access memory 30 are reduced, that is, the processing speed is improved and the power consumption is reduced.

[0039] In some embodiments of the present application, the target data to be processed only participates in the first type of computing tasks; and / or the first type of computing tasks at least include matrix-vector multiplication (GEMV) operations. In other words, the first type of computing tasks include matrix-vector multiplication; or, the first type of computing tasks may also be other operations, and the data and intermediate data participating in the calculation as described above will not be reused in other types of calculations.

[0040] In the embodiment of the present application, the first operation module 10 is arranged on the data transmission path between the random access memory 30 and the data transmission interface of the neural processing unit 50. The target data to be processed includes a part of the KV data, that is, the target data to be processed is read by the neural processing unit 50 through the data transmission interface and transmitted via the data transmission path.

[0041] It is understandable that by setting the first operation module 10 on the data transmission path, the target data to be processed can be easily operated without transferring the target data to be processed to other operation modules, thereby reducing the cache pressure, eliminating part of the data transmission overhead, and reducing power consumption. At the same time, in the embodiment of the present application, no changes are made to other modules in the neural processing unit, and no complex control logic is added, so the implementation cost is lower and it is easier to implement.

[0042] Figure 2 A schematic diagram of another optional structure of the neural processing unit provided in the embodiment of the present application.

[0043] In some embodiments of the present application, Figure 2 As shown, the first operation module 10 includes: a matrix processing unit 101 and n vector processing units 102. Where n is greater than or equal to 1.

[0044] In the embodiment of the present application, each vector processing unit 102 is configured to perform a first type of computing task on at least part of the sub-data in the target data to be processed, obtain a sub-operation result, and summarize the sub-operation result to the matrix processing unit 101. The matrix processing unit 101 is configured to merge the sub-operation results of each vector processing unit 102 to obtain a first operation result.

[0045] In the present application examples, reference Figure 3 , each vector processing unit (VPE) 102 exchanges data with the matrix processing unit (MPE) 101 separately. After obtaining the sub-operation result of any vector processing unit 102, the matrix processing unit 101 will merge all the sub-operation results of the vector processing unit 102 to obtain the first operation result corresponding to the vector processing unit 102.

[0046] In the embodiment of the present application, the vector processing unit (VPE) 102 may include: a matrix multiplication and addition operation unit and a first transmission unit. The matrix multiplication and addition operation unit is configured to perform a matrix multiplication and addition operation on at least part of the sub-data in the target data to be processed to obtain a sub-operation result; the matrix multiplication and addition operation unit may include a multiplier and an adder. The first transmission unit is configured to perform data exchange with the matrix processing unit (MPE) 101.

[0047] In the embodiment of the present application, the matrix processing unit (MPE) 101 may include: a merging unit and a second transmission unit. The merging unit is configured to merge the sub-operation results calculated by the vector processing unit (VPE) 102, for example, adding or matrix concatenating. The first transmission unit is configured to exchange data with the vector processing unit (VPE) 102.

[0048] In the present application, continue to refer to Figure 3 , since matrix-vector multiplication involves multiplication and addition operations of vector data; at the same time, when the amount of matrix data is large, it is difficult to complete the operation of all data vectors in the matrix at one time. Therefore, in the vector processing unit 102, part of the data in the matrix can be operated first to obtain a sub-operation result; then, the sub-operation result is returned to the matrix processing unit 101, and the matrix processing unit 101 merges all the sub-operation results to obtain the final first operation result of the matrix-vector multiplication.

[0049] It should be noted that matrix decomposition uses the rules in the matrix to decompose the matrix into several smaller sub-matrices without the need to observe all the data globally, and then recombines and calculates them to achieve data processing, feature extraction, dimensionality reduction, etc. For example, matrix decomposition techniques can include: singular value decomposition (SVD), principal component analysis (PCA) and other methods.

[0050] It can be understood that the calculation of the entire large model is divided into the first type and the second type, and the calculation is performed by different computing units. At the same time, the calculation tasks of the first type are divided into the matrix processing unit 101 and the vector processing unit 102. In this way, a distributed on-the-fly calculation method is implemented, so that the calculation tasks can be executed in parallel in time sequence, ensuring that the calculation tasks are executed stably and efficiently. At the same time, using a distributed calculation method, data is sent to different computing units in a dispersed manner, thereby reducing the amount of data received by a single computing unit and reducing the bandwidth pressure of the system.

[0051] In some embodiments of the present application, reference Figure 2 Each vector processing unit 102 is further configured to determine the position of each sub-data in the matrix of the target data to be processed according to the address of each sub-data in the memory (including DDR0-DDR3); and, based on the position of each sub-data in the matrix, perform vector multiplication operation on each sub-data to obtain a sub-operation result. The matrix processing unit 101 is further configured to merge the sub-operation results of each vector processing unit based on the position of each sub-data in the matrix to obtain a first operation result.

[0052] It should be noted that the reference Figure 2 The memories DDR0 to DDR3 outside the neural processing unit 50 are internal memories, while the random access memory 30 in the neural processing unit 50 is a cache.

[0053] It can be understood that the position of the sub-data in the matrix is ​​determined based on the address of the sub-data in the memory, so that the matrix processing unit 101 can accurately merge the sub-operation results, thereby ensuring the accuracy of the first operation result.

[0054] In some embodiments of the present application, reference Figure 2 Each vector processing unit 102 is further configured to count the number of sub-operation results; and when the number of sub-operation results reaches a threshold, all sub-operation results are returned to the matrix processing unit 101.

[0055] It is understandable that when the number of sub-operation results reaches the threshold, the vector processing unit 102 promptly returns the sub-operation results to the matrix processing unit 101. In this way, on the one hand, it is ensured that the operation can be executed in time; on the other hand, the cache pressure of the vector processing unit 102 is reduced, and the sub-operation results are prevented from being lost.

[0056] In some embodiments of the present application, reference Figure 2The matrix processing unit 101 is further configured to send vectors, configuration information and start commands to the n vector processing units 102. The start command is used to control the vector processing unit to start multiplication and addition operations. The vectors and configuration information are required for the vector processing unit to perform the first type of computing tasks; wherein the configuration information includes: operation type and operation quantity.

[0057] In some embodiments of the present application, reference Figure 2 , the matrix processing unit 101 and the vector processing unit 102 can perform operations according to the following workflow:

[0058] 1. The matrix processing unit 101 sends the vector and configuration information of the broadcast operation and starts the operation;

[0059] 2. After receiving the relevant information, the vector processing unit 102 monitors the data returned by the address accessed by the system bus and performs relevant operations according to the address;

[0060] 3. The vector processing unit 102 counts the sub-operation results while performing the operation, and returns the sub-operation results to the matrix processing unit 101 when the number of sub-operation results reaches a threshold;

[0061] 4. The matrix processing unit 101 automatically monitors the return data of the vector processing unit 102. When any vector processing unit 102 completes the operation, the merging operation is started to merge all the sub-operation results of the vector processing unit 102. In this way, a complete matrix-vector multiplication operation is completed.

[0062] In the embodiment of the present application, operations need to be completed according to the configuration information inside the vector processing unit 102. The vector processing unit 102 will configure the number of related operations according to the width of the bus.

[0063] In some embodiments of the present application, each vector processing unit 102 includes: a plurality of multiply-add units (MACs), wherein the parameters of the multiply-add units are Int4 / int8 and FP16, and / or the parameters of the multiply-add units are FP16 and FP16.

[0064] In some embodiments of the present application, reference Figure 2 There are n transmission paths between the memory (including DDR0-DDR3) and the memory access module 40. Each vector processing unit 102 is arranged on a corresponding transmission path; that is, a total of n vector processing units 102 are arranged in the neural processing unit 50, and a vector processing unit 102 is arranged on each transmission path.

[0065] In some embodiments of the present application, n=4. Figure 2There are four transmission paths between the memory (including DDR0-DDR3) and the memory access module 40. Accordingly, a vector processing unit 102 is provided on each transmission path, and a total of four vector processing units 102 are provided. At the same time, the number of memories DDR0-DDR3 is also four, and the data in each memory is transmitted through a corresponding transmission path.

[0066] In the embodiment of the present application, each vector processing unit 102 will calculate the data on the transmission path where it is located, and at the same time, the matrix processing unit 101 will merge all sub-operation results of a vector processing unit 102. In other words, the first operation result output by the first operation module 10 is obtained by calculating the data on each transmission path separately.

[0067] It should be noted that the reference Figure 2 The memory access module 40 is configured to read / write data. That is, the memory access module 40 is configured to access data in the random access memory 30, and to obtain data in the external memory DDR0-DDR3 through the system bus. The memory access module 40 may include a direct memory access engine (DMA Engine), and the DDR can be directly accessed through the memory access engine.

[0068] Figure 4 A schematic diagram of another optional structure of the neural processing unit provided in the embodiment of the present application.

[0069] In some embodiments of the present application, Figure 4 As shown, the neural processing unit 50 further includes: a second operation unit 20. The second operation unit 20 is connected to the random access memory 30 and is configured to perform a second type of computing task according to the first operation result in the random access memory 30.

[0070] In the present application examples, reference Figure 4 The second computing unit 20 may include a computing unit set (computation PEs) 201, a loading unit (ie, a loading engine, Load Engine) 202 and an output processing unit (Output process) 203.

[0071] The loading unit 202 is configured to read data from the random access memory 30 and organize the data into a format suitable for operation. The operation unit set 201 is configured to perform a second type of computing task on the data. The output processing unit 203 is configured to organize the results obtained by the calculation and store them in the random access memory 30.

[0072] It should be noted that Figure 5This is a schematic diagram of the transform layer operation of the large model Llama2. Figure 5 The solution shown is for example only. The solution can be applied to the decoder side or encoder side of a large model based on the transformer architecture, or other similar AI (artificial intelligence) model operations where some data is used only once.

[0073] like Figure 5 As shown in the figure, input embedding converts the input data (such as language text) into a specific vector; then, root mean square normalization (RMSNorm) removes the mean term and normalizes it. The matrices Q (query) and K (key value) are subjected to rotation position encoding (RoPE) and causal mask, and matrix-vector multiplication is performed. The result is normalized by the exponential function (Softmax) and then matrix-vector multiplication is performed with the matrix V (value) The output result will also go through matrix-vector addition ⊕, root mean square error (RMSNorm), and the activation function of the FFN layer (such as SwiGLU) and finally be output.

[0074] It should be noted that Figure 5 The calculation process shown belongs to the well-known technology and does not involve any improvement to the algorithm. It is only briefly explained here.

[0075] In the present application examples, reference Figure 5 , the computation process of the decoder is recursive in nature, and each operation only uses the latest generated token to produce new Q, K, and V vectors, and performs subsequent attention calculations, which are essentially matrix-vector multiplications.

[0076] Combination Figure 4 The first operation module 10 arranged on the data transmission path can execute Figure 5 Matrix-vector multiplication shown That is, the first type of computing tasks includes: performing matrix-vector multiplication on matrices Q, K, and V

[0077] At the same time, the second operation module 20 can execute Figure 5 That is, the second type of computing tasks include: Figure 5 Other operations shown include, for example, normalized exponential function (Softmax), linear rectification function (RELU), Gaussian error linear unit function (GELU), layer normalization (Layernorm), rotation position encoding (RoPE) and other activation and operation functions.

[0078] It should be noted that the reference Figure 4 , the calculation result obtained by the second operation module 20 is cached in the random access memory 30 and can be used to participate in the subsequent operation process. If the calculation result obtained by the second operation module 20 is subsequently used in the first type of calculation, that is, it is only used once, then the memory access module 40 can also read the calculation result from the random access memory 30 and send it to the first operation module 10 for calculation, thereby reducing the storage space occupied by the random access memory 30. In other words, the data required for the first operation module 10 to perform calculations can be obtained from the external DDR through the bus, or from the internal random access memory 30, as long as (all / part of) the data involved in the calculation will not be used again later, and no limitation is made here.

[0079] Figure 6 A schematic diagram of the implementation flow of a computing task execution method of an artificial intelligence model provided in an embodiment of the present application, such as Figure 6 As shown, the method includes steps S101 to S103.

[0080] S101, obtaining target data to be processed.

[0081] S102: Execute a first type of computing task on the target data to be processed to obtain a first computing result.

[0082] S103: Cache the first operation result to participate in other computing tasks; wherein the target data to be processed only participates in the first type of computing tasks.

[0083] In the embodiment of the present application, the initial data of the first type of calculation input externally, as well as the intermediate data during the execution of the first type of calculation task, will not be reused in other types of calculations, and therefore may not be cached in the random access memory. In this way, the storage space of the random access memory can be significantly saved, and at the same time, part of the data cache overhead is eliminated, and the time and power consumption caused by accessing the random access memory are reduced, that is, the processing speed is increased and the power consumption is reduced.

[0084] In the present application examples, reference Figure 1 , steps S101 to S103 can be performed by the first operation module 10. The first operation module 10 is set on the data transmission path between the random access memory 30 and the data transmission interface of the neural processing unit 50. In this way, the target data to be processed can be easily operated without transmitting the target data to be processed to other operation modules, thereby eliminating part of the data transmission overhead and reducing power consumption.

[0085] In some embodiments of the present application, Figure 7Steps S201 to S202 are shown to achieve Figure 6 The step S102 is shown. The description will be made in combination with each step.

[0086] S201. Execute the first type of computing tasks on the target data to be processed inputted through the n transmission paths of the artificial intelligence model respectively to obtain sub-operation results; wherein the target data to be processed inputted through each transmission path is divided into a plurality of sub-data, and the first type of computing tasks are performed multiple times.

[0087] S202: Merge the sub-operation results to obtain a first operation result.

[0088] In the embodiment of the present application, the first type of computing task is performed multiple times for the target data to be processed input by each transmission path. That is, the first operation result obtained by the first type of computing task is obtained by respectively calculating the data on each transmission path.

[0089] In the embodiment of the present application, the first type of computing task can be decomposed to obtain sub-computation results first, and then the sub-computation results are combined to obtain the first computing result. In this way, a distributed and real-time computing method is implemented, which can ensure that the first type of computing task is executed stably and efficiently.

[0090] In some embodiments of the present application, Figure 6 After step S103 shown in the figure, the following steps are also included: Figure 8 The step S104 shown will be described in combination with each step.

[0091] S104: Execute a second type of computing task according to the cached first computing result.

[0092] In the present application examples, reference Figure 4 , step S104 may be performed by the second operation module 20. The second type of computing tasks may include: normalized exponential function (Softmax), linear rectification function (RELU), Gaussian error linear unit function (GELU), layer normalization (Layernorm), rotation position encoding (RoPE) and other activation and computing functions.

[0093] Fig. 9 A schematic diagram of the composition structure of a processor provided in an embodiment of the present application. Fig. 9 As shown, the processor 60 includes: a neural processing unit 50. The neural processing unit 50 includes the technical features of the above-mentioned embodiments.

[0094] Fig.10 The following is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.10 As shown, the computer device 80 includes: a memory 70, and Fig. 9 A processor 60 is shown.

[0095] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process mentioned above does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The serial numbers of the embodiments of the present application mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.

[0096] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0097] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0098] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0099] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units. A person of ordinary skill in the art can understand that all or part of the steps of the above method embodiments can be completed by hardware related to program instructions, and the above program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above method embodiments are executed; and the above storage medium includes: mobile storage devices, read-only memory (ROM), disks or optical disks, etc. Various media that can store program codes.

[0100] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0101] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A neural processing unit for performing computational tasks of an artificial intelligence model, comprising: Random Access Memory; The first operation module is configured to perform a first type of computing task on the received target data to be processed, obtain a first operation result, and store the first operation result in the random access memory; wherein the first operation module is arranged on the data transmission path between the random access memory and the data transmission interface of the neural processing unit.

2. The neural processing unit according to claim 1, The target data to be processed only participates in the first type of computing tasks; and / or the first type of computing tasks at least includes matrix-vector multiplication operations.

3. The neural processing unit according to claim 1 or 2, wherein the first computing module comprises: A matrix processing unit and n vector processing units; n is greater than or equal to 1; Each of the vector processing units is configured to perform the first type of computing task on at least part of the sub-data in the target data to be processed, obtain sub-operation results, and summarize the sub-operation results to the matrix processing unit; The matrix processing unit is configured to merge the sub-operation results of each of the vector processing units to obtain the first operation result.

4. The neural processing unit according to claim 3, n transmission paths are arranged between the memory and the memory access module; Each of the vector processing units is arranged on a corresponding transmission path.

5. The neural processing unit according to claim 3, The matrix processing unit is further configured to send vectors, configuration information and start-up commands to the n vector processing units; wherein, The start command is used to control the vector processing unit to start multiplication and addition operations; The vector and the configuration information are required for the vector processing unit to perform the first type of computing task; the configuration information includes: operation type and operation quantity.

6. The neural processing unit according to claim 3, Each of the vector processing units is further configured to determine the position of each of the sub-data in the matrix of the target data to be processed according to the address of each of the sub-data in the memory; based on the position of each of the sub-data in the matrix, perform a vector multiplication operation on each of the sub-data to obtain the sub-operation result; The matrix processing unit is further configured to merge the sub-operation results of each of the vector processing units based on the position of each of the sub-data in the matrix to obtain the first operation result.

7. The neural processing unit according to claim 3, Each of the vector processing units is further configured to count the number of the sub-operation results; when the number of the sub-operation results reaches a threshold, return all the sub-operation results to the matrix processing unit.

8. The neural processing unit according to claim 1 or 2, further comprising: The second operation unit is connected to the random access memory and is configured to perform a second type of computing task according to the first operation result in the random access memory.

9. A method for executing a computing task of an artificial intelligence model, comprising: Get the target data to be processed; Executing a first type of computing task on the target data to be processed to obtain a first computing result; The first operation result is cached to participate in other computing tasks; wherein the target data to be processed only participates in the first type of computing tasks.

10. The computing task execution method according to claim 9, executing a first type of computing task on the target data to be processed to obtain a first computing result, comprising: The first type of computing tasks are respectively performed on the target data to be processed inputted by the n transmission paths of the artificial intelligence model to obtain sub-operation results; wherein the target data to be processed inputted by each transmission path is divided into a plurality of sub-data, and the first type of computing tasks are performed a plurality of times; The sub-operation results are combined to obtain the first operation result.

11. The computing task execution method according to claim 9, after caching the first operation result, the method further comprises: A second type of computing task is executed according to the cached first computing result.

12. A processor, comprising: A neural processing unit as claimed in any one of claims 1 to 8.

13. A computer device comprising: A memory, and a processor as claimed in claim 12.