Chip-based fusion operator processing methods, devices, and related products

Abstract

This application provides a chip-based fusion operator processing method, apparatus, and related products, specifically applied in the field of data processing technology. The scheme includes: sequentially loading matrix units from the chip's storage device onto the chip's computing device, wherein the matrix units are obtained by splitting the target matrix according to a preset matrix unit dimension; iteratively calculating and summing the normalized denominators of each matrix unit using the computing device, determining the TopK value of each matrix unit, and iteratively processing until the sum of the normalized denominators of all matrix units in the target matrix and the corresponding TopK value of the target matrix are obtained; calculating the normalized numerator of each TopK value based on the corresponding TopK value of the target matrix using the computing device; and obtaining the processing result of the fusion operator for the target matrix based on the sum of the normalized denominators of the target matrix and the normalized numerator of each TopK value using the computing device.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data processing technology, and in particular to a chip-based fusion operator processing method, apparatus and related products. Background Art

[0002] With the continuous development of artificial intelligence and digital electronics technology, the rapid development of various artificial intelligence chips has led to increasingly higher requirements for data processing.

[0003] Currently, in the field of artificial intelligence, fast-transformer (FT) network models are widely used in natural language human-computer interaction technology due to their powerful semantic representation capabilities. When performing classification using a fast-transformer network model, a fully connected layer is typically used to map the feature representation matrix to the sample label space, followed by a softmax classifier for classification, ultimately selecting the top K optimal classification results.

[0004] Currently, when using chips to process classification operations of Fast-Transformer network models, the feature representation matrix has a large dimension, while the computing space on the chip is limited. Therefore, a large number of I / O operations need to be performed on the chip's storage device. Furthermore, the current processing of classification operations of Fast-Transformer network models using chips generates a lot of redundant calculations, resulting in low data processing efficiency. Summary of the Invention

[0005] This application provides a chip-based fusion operator processing method, apparatus, and related products to solve the technical problem that when using chips to process classification operations of fast-transformer network models, a large number of I / O operations need to be performed on the chip's storage device, and the current processing of fast-transformer network model classification operations using chips generates a large amount of redundant calculations, resulting in low data processing efficiency.

[0006] In a first aspect, this application provides a chip-based fusion operator processing method, comprising:

[0007] Matrix units are loaded sequentially from the chip's storage device onto the chip's computing device; wherein the matrix units are obtained by splitting the target matrix according to a preset matrix unit dimension; the order is obtained when splitting the target matrix into matrix units;

[0008] The computing device of the chip is used to iteratively calculate the normalized denominator of each matrix unit and sum them up, and to determine the TopK value of each matrix unit and process iteratively until the sum of the normalized denominators of all matrix units in the target matrix and the TopK value corresponding to the target matrix are obtained.

[0009] The computing device is used to calculate the normalized numerator of each TopK value based on the TopK value corresponding to the target matrix;

[0010] The computing device uses the sum of the normalized denominators corresponding to the target matrix and the normalized numerators of each TopK value to obtain the processing result of the fusion operator of the target matrix.

[0011] Secondly, this application provides a chip-based fusion operator processing device, comprising:

[0012] A loading unit is used to load matrix units sequentially from the chip's storage device onto the chip's computing device; wherein the matrix units are obtained by splitting the target matrix according to a preset matrix unit dimension; the order is obtained when splitting the target matrix into matrix units;

[0013] An iterative processing unit is used to iteratively calculate the normalized denominator of each matrix unit using the computing device of the chip and sum them up, and determine the TopK value of each matrix unit and iteratively process it until the sum of the normalized denominators of all matrix units in the target matrix and the TopK value corresponding to the target matrix are obtained.

[0014] The computing unit is used to calculate the normalized numerator of each TopK value based on the TopK value corresponding to the target matrix using the computing device;

[0015] The computing unit is further configured to use the computing device to obtain the processing result of the fusion operator of the target matrix based on the sum of the normalized denominators corresponding to the target matrix and the normalized numerator of each TopK value.

[0016] Thirdly, this application provides an artificial intelligence chip, including: a storage device, a computing device, and a chip-based data processing device as described in the second aspect.

[0017] Fourthly, this application provides an electronic device, the electronic device comprising: off-chip memory and an artificial intelligence chip as described in the third aspect.

[0018] Fifthly, this application provides a board, the board comprising: a storage device, an external interface device, a control device, and an artificial intelligence chip as described in the third aspect;

[0019] The artificial intelligence chip is connected to the storage device, the control device, and the external interface device, respectively.

[0020] The storage device is used to store the target data;

[0021] The external interface device is used to realize data transmission between the artificial intelligence chip and external devices;

[0022] The controller is used to monitor the state of the artificial intelligence chip.

[0023] The chip-based fusion operator processing method, apparatus, and related products provided in this application load matrix units sequentially from the chip's storage device onto the chip's computing device in the order of splitting the target matrix into matrix units. The matrix units are obtained by splitting the target matrix according to a preset matrix unit dimension. The computing device iteratively calculates and sums the normalized denominators of each matrix unit, determines the TopK value of each matrix unit, and iteratively processes this process until the sum of the normalized denominators of all matrix units in the target matrix and the corresponding TopK value of the target matrix are obtained. The computing device then calculates the normalized numerator of each TopK value based on the TopK value of the target matrix. Finally, the computing device obtains the processing result of the fusion operator for the target matrix based on the sum of the normalized denominators of the target matrix and the normalized numerator of each TopK value. Because the processes of reading matrix units, calculating and summing the normalized denominators of the matrix units, and determining the TopK value of the target matrix are integrated into a loop iterative calculation, redundant calculations are effectively reduced, and data processing efficiency is improved. Furthermore, it only requires reading the matrix unit from the storage device once, thus effectively reducing I / O operations. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0025] Figure 1 This diagram shows the structure of a board according to an embodiment of this application;

[0026] Figure 2 This diagram shows a structural diagram of a combined processing apparatus according to an embodiment of the present application;

[0027] Figure 3 This diagram illustrates the internal structure of a single-core computing device according to an embodiment of this application.

[0028] Figure 4 This diagram illustrates the internal structure of a multi-core computing device according to an embodiment of this application.

[0029] Figure 5 This diagram illustrates the internal structure of a processor core according to an embodiment of this application.

[0030] Figure 6 A flowchart illustrating a chip-based fusion operator processing method provided in one embodiment of this application;

[0031] Figure 7 A flowchart illustrating a chip-based fusion operator processing method provided in another embodiment of this application;

[0032] Figure 8 A flowchart of a chip-based fusion operator processing method provided in yet another embodiment of this application;

[0033] Figure 9 A flowchart of a chip-based fusion operator processing method provided in another embodiment of this application;

[0034] Figure 10 A flowchart of a chip-based fusion operator processing method is also provided in this application according to an embodiment;

[0035] Figure 11 This application also provides a schematic diagram of the structure of a chip-based fusion operator processing device according to an embodiment. Detailed Implementation

[0036] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0037] For ease of explanation, the computer hardware structure involved in this application will be described first:

[0038] Figure 1 A structural diagram of a board according to an embodiment of this application is shown. This board can serve as the aforementioned execution device. Figure 1As shown, board 10 includes chip 101, which is a system-on-chip (SoC) integrating one or more combined processing units. These combined processing units are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. Board 10 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and powerful computing capabilities.

[0039] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. The calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface forms, such as a PCIe interface.

[0040] The board 10 also includes a storage device 104 for storing data, which includes one or more memory cells 105. The storage device 104 is connected to and transmits data with the controller 106 and the chip 101 via a bus. The controller 106 in the board 10 is configured to regulate the state of the chip 101. Therefore, in one application scenario, the controller 106 may include a microcontroller (MCU).

[0041] It should be noted that, in this embodiment of the application, chip 101 is used to process classification operations of the fast-transformer network model. Specifically, Figure 2 This is a structural diagram illustrating a combined processing apparatus according to an embodiment of this application. Figure 2 As shown, the combined processing device 20 includes a computing device 201, an interface device 202, a processing device 203, and a storage device 204.

[0042] The computing device 201 is configured to perform user-specified operations. It is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0043] Interface device 202 is used to transmit data and control commands between computing device 201 and processing device 203. For example, computing device 201 can obtain input data from processing device 203 via interface device 202 and write it to on-chip storage device of computing device 201. Further, computing device 201 can obtain control commands from processing device 203 via interface device 202 and write them to on-chip control cache of computing device 201. Alternatively or optionally, interface device 202 can also read data from storage device of computing device 201 and transmit it to processing device 203.

[0044] Processing device 203, as a general-purpose processing device, performs basic controls including but not limited to data transfer, and starting and / or stopping computing device 201. Depending on the implementation, processing device 203 may be one or more types of processors, including but not limited to digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As previously stated, computing device 201 disclosed herein can be considered to have a single-core structure or a homogeneous multi-core structure. However, when computing device 201 and processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.

[0045] Storage device 204 is used to store data to be processed. It may be DRAM or DDR memory, typically 16G or larger in size, and is used to store data of computing device 201 and / or processing device 203.

[0046] It should be noted that the specific processing of classification operations for the fast-transformer network model in the embodiments of this application can be achieved in [specific ways]. Figure 2 In the computing device.

[0047] In one possible implementation, a single-core computing device is also provided. For example... Figure 3 As shown, the single-core computing device 301 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 301 includes three main modules: a control module 31, an arithmetic module 32, and a storage module 33.

[0048] The control module 31 coordinates and controls the operation of the computation module 32 and the storage module 33 to complete the deep learning task. It includes an instruction fetch unit (IFU) 311 and an instruction decode unit (IDU) 312. The instruction fetch unit 311 fetches instructions from the processing device 203, and the instruction decode unit 312 decodes the fetched instructions and sends the decoding result as control information to the computation module 32 and the storage module 33.

[0049] The computation module 32 includes a vector operation unit 321 and a matrix operation unit 322. The vector operation unit 321 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit 322 is responsible for the core computations of deep learning algorithms, namely matrix multiplication and convolution.

[0050] Storage module 33 is used to store or move relevant data, including neuron RAM (NRAM) 331, weight RAM (WRAM) 332, and direct memory access (DMA) module 333. NRAM 331 is used to store input neurons, output neurons, and intermediate results after computation; WRAM 332 is used to store the convolution kernels of the deep learning network, i.e., the weights; DMA 333 is connected to DRAM 204 through bus 34 and is responsible for data transfer between single-core computing device 301 and DRAM 204.

[0051] In another possible implementation, the computing device for processing the classification operation of the fast-transformer network model in this embodiment is a multi-core computing device.

[0052] Figure 4 A schematic diagram of the internal structure of a processor core according to an embodiment of this application is shown. The multi-core computing device 41 adopts a hierarchical structure design. As a system-on-a-chip, the multi-core computing device 41 includes at least one cluster, and each cluster includes multiple processor cores. In other words, the multi-core computing device 41 is constructed in a hierarchical structure of system-on-a-chip, cluster, and processor core.

[0053] From the perspective of system-on-a-chip hierarchy, such as Figure 4 As shown, the multi-core computing device 41 includes an external storage controller 401, a peripheral communication module 402, an on-chip interconnect module 403, a synchronization module 404, and multiple clusters 405.

[0054] There can be multiple external storage controllers 401; two are shown as an example in the figure. These controllers are used to access external storage devices, such as those issued by the processor core, in response to access requests from the processor core. Figure 2 The DRAM 204 in the chip allows data to be read from or written to external devices. The peripheral communication module 402 receives control signals from the processing device 203 via the interface device 202, initiating the computing device 201 to execute tasks. The on-chip interconnect module 403 connects the external memory controller 401, the peripheral communication module 402, and multiple clusters 405 to transmit data and control signals between modules. The synchronization module 404 is a global barrier controller (GBC) used to coordinate the working progress of each cluster and ensure information synchronization. The multiple clusters 405 are the computing cores of the multi-core computing device 41. Four are shown exemplary in the figure; however, with hardware development, the multi-core computing device 41 disclosed herein may also include 8, 16, 64, or even more clusters 405. The clusters 405 are used to efficiently execute deep learning algorithms.

[0055] From the perspective of cluster hierarchy, such as Figure 4 As shown, each cluster 405 includes multiple processor cores (IPU cores) 406 and one memory core (MEM core) 407.

[0056] Four processor cores 406 are shown in the figure as an example; this disclosure does not limit the number of processor cores 406. Its internal architecture is as follows: Figure 5 As shown. Each processor core 406 is similar to Figure 3 The single-core computing device 301 also includes three main modules: a control module 51, an arithmetic module 52, and a storage module 53. The functions and structures of the control module 51, arithmetic module 52, and storage module 53 are largely the same as those of the control module 31, arithmetic module 32, and storage module 33, and will not be described again. It should be noted that the storage module 53 includes an input / output direct memory access (IODMA) module 533 and a move direct memory access (MVDMA) module 534. The IODMA 533 controls the memory access of NRAM 531 / WRAM 532 and DRAM 204 via the broadcast bus 409; the MVDMA 534 controls the memory access of NRAM 531 / WRAM 532 and SRAM 408.

[0057] Back Figure 4The storage core 407 is primarily used for storage and communication, namely storing shared data or intermediate results among processor cores 406, and performing communication between cluster 405 and DRAM 204, communication between clusters 405, and communication between processor cores 406. In other embodiments, the storage core 407 has scalar operation capabilities and is used to perform scalar operations.

[0058] The storage core 407 includes an SRAM 408, a broadcast bus 409, a cluster direct memory access (CDMA) module 410, and a global direct memory access (GDMA) module 411. The SRAM 408 acts as a high-performance data relay station. Data multiplexed between different processor cores 406 within the same cluster 405 does not need to be obtained from the DRAM 204 by each processor core 406. Instead, it is relayed between processor cores 406 via the SRAM 408. The storage core 407 only needs to quickly distribute the multiplexed data from the SRAM 408 to multiple processor cores 406 to improve inter-core communication efficiency and greatly reduce on-chip and off-chip input / output access.

[0059] Broadcast bus 409, CDMA 410, and GDMA 411 are used to perform communication between processor cores 406, communication between clusters 405, and data transfer between cluster 405 and DRAM 204, respectively. These will be explained below.

[0060] The broadcast bus 409 is used to complete high-speed communication between the processor cores 406 within the cluster 405. In this embodiment, the broadcast bus 409 supports inter-core communication methods including unicast, multicast, and broadcast. Unicast refers to point-to-point (e.g., data transmission from one processor core to another) data transmission. Multicast is a communication method that transmits a piece of data from SRAM 408 to several specific processor cores 406. Broadcast is a communication method that transmits a piece of data from SRAM 408 to all processor cores 406, and is a special case of multicast.

[0061] CDMA 410 is used to control SRAM 408 access between different clusters 405 within the same computing device 201.

[0062] GDMA 411 works in conjunction with external memory controller 401 to control memory access from SRAM 408 to DRAM 204 in cluster 405, or to read data from DRAM 204 into SRAM 408. As previously described, communication between DRAM 204 and NRAM 431 or WRAM 432 can be achieved through two channels. The first channel is a direct connection between DRAM 204 and NRAM 431 or WRAM 432 via IODAM 433; the second channel involves first transmitting data between DRAM 204 and SRAM 408 via GDMA 411, and then transmitting data between SRAM 408 and NRAM 431 or WRAM 432 via MVDMA 534. Although the second channel appears to require more components and has a longer data flow, in some embodiments, the bandwidth of the second channel is significantly greater than that of the first channel. Therefore, communication between DRAM 204 and NRAM 431 or WRAM 432 may be more efficient via the second channel. The embodiments disclosed herein allow for the selection of data transmission channels based on the hardware capabilities available.

[0063] In other embodiments, the functions of GDMA 411 and IODMA 533 can be integrated into the same component. For ease of description, this disclosure treats GDMA 411 and IODMA 533 as different components. For those skilled in the art, any component that performs similar functions and achieves similar technical effects to this disclosure falls within its scope of protection. Furthermore, the functions of GDMA 411, IODMA 533, CDMA 410, and MVDMA 534 can also be implemented by the same component.

[0064] The following is based on Figure 2 The combined processing device shown Figure 4 Taking the multi-core computing device shown as an example, this paper will explain and analyze in detail the specific process of using chips to process classification operations of fast-transformer network models in the prior art.

[0065] The output matrix of a fully connected layer can be represented as FC, which is the result of matrix multiplication of the input matrix and the weight matrix.

[0066] If the input matrix has a dimension of N*Ci and the weight matrix has a dimension of Ci*Co, then the dimension of the fully connected (FC) matrix is ​​N*Co. However, in the Fast-Transformer network model, the dimension of the FC matrix is ​​generally very large. For example, if the input matrix has a dimension of 64*1024 and the weight matrix has a dimension of 1024*90000, then the dimension of the FC matrix is ​​64*90000. To enable the softmax classifier to operate smoothly on the chip and to determine the optimal TopK classification results, the FC is typically processed as row vectors, with each row vector having a dimension of 1*Co.

[0067] The formula for calculating the Softmax classifier is as follows: Where xi and xj are elements in the row vector of FC. m is the global maximum value in the row vector.

[0068] In existing technologies, because the dimensions of a fully connected (FC) vector are very large, it is impossible to store the FC at once in the on-chip space of a computing device and calculate the maximum value of the FC row vector. Therefore, the common approach is to split the row vector into multiple row vector units, each with a dimension of 1*Coi. For example, it can be 1*64.

[0069] exist Figure 2In the combined processing apparatus shown, computing unit 201 reads input and weight from storage device 204, calculates the full value (FC), and writes the FC into storage device 204. When computing unit 201 performs the softmax operation in the softmax classifier, to reduce the storage pressure on the on-chip space of computing unit 201, computing unit 201 reads the FC from storage device 204 in segments. If the row vectors read in segments are called matrix units, then each matrix unit can be represented as A0, B0, C0, D0, ... The computing unit 201 calculates the maximum value in each segment and writes the maximum value into storage device 204. After computing unit 201 has calculated the maximum values ​​of all matrix units, it reads the maximum values ​​of all matrix units from storage device 204, determines the global maximum value, and writes the global maximum value into storage device 204. Because the on-chip space of computing unit 201 is limited, after the first acquisition of a matrix unit, to save space, the matrix unit is discarded after calculating its maximum value. Therefore, when the computing device 201 calculates the softmax, it needs to reread the matrix units A0, B0, C0, D0, ... from the storage device 204. After the calculation is completed, the softmax calculation result of each element xi is written into the storage device 204. Finally, the computing device 201 retrieves the softmax calculation result of each element xi from the storage device 204 to determine and provide the TopK value.

[0070] Therefore, based on existing technologies, it is known that multiple readings of matrix units and intermediate result data from the storage device are required, and multiple writings of intermediate result data to the storage device are also necessary, resulting in a large number of I / O operations. Furthermore, when first using a fully connected layer to map the feature representation matrix to the sample label space, and then using a softmax classifier for classification, and finally extracting the optimal Top K classification results, there is a large amount of redundant computation, leading to low data processing efficiency.

[0071] In existing technologies, the following methods are adopted: Figure 4 When a multi-core computing device processes the classification operation of a fast-transformer network model, the input matrix and weight matrix are first stored in DRAM 204. The input and weight are loaded into SRAM 408 through GDMA 411. SRAM 408 synchronizes the input and weight to at least one processor core through a broadcast bus. The processor core calculates the FC and stores the FC in DRAM 204 through SRAM 408.

[0072] Then, SRAM408 reads matrix cells A0B0, C0, D0, ... from DRAM 204 in segments. SRAM distributes the matrix cells to at least one processor core, which calculates the maximum value in each matrix cell. SRAM408 writes the maximum value of the matrix cell to DRAM 204. DRAM 204 loads the maximum values ​​of all matrix cells into SRAM408, which then loads them into the processor core to calculate the global maximum value and writes it back into DRAM 204. Matrix cells A0, B0, C0, D0, ... are reloaded from DRAM 204 into SRAM408, which distributes them to the processor core for softmax calculation. The calculation result is written back to DRAM 204 via SRAM408. Finally, all softmax calculation results are distributed to the processor core via SRAM408 to calculate the TopK value.

[0073] Therefore from Figure 4 As can be seen from the existing technology, matrix cells and intermediate result data need to be read from DRAM 204 multiple times, and intermediate result data needs to be written back to DRAM 204 multiple times, resulting in a large number of I / O operations. Furthermore, when first using a fully connected layer to map the feature representation matrix to the sample label space, and then using a softmax classifier for classification, and finally extracting the optimal TopK classification results, there is a lot of redundant computation, which leads to low data processing efficiency.

[0074] Therefore, in order to solve the technical problems in the existing technology, the processes of FC calculation, softmax xi calculation and TopK value determination can be integrated to form the Fc-Logsoftmax-topk fusion operator, which can complete the calculation of TopK value without performing a lot of IO operations with the storage device.

[0075] Specifically, if Figure 2 As shown, matrix units can be loaded sequentially from the storage device into the computing device on the chip in the order of splitting the target matrix into matrix units. The computing device iteratively calculates the normalized denominator of each matrix unit and sums them, and determines the TopK value of each matrix unit and iteratively processes it until the sum of the normalized denominators of all matrix units in the target matrix and the corresponding TopK value of the target matrix are obtained. Then, the computing device calculates the normalized numerator of each TopK value based on the corresponding TopK value of the target matrix. Finally, based on the sum of the normalized denominators of the target matrix and the normalized numerator of each TopK value, the processing result of the fusion operator of the target matrix is ​​obtained.

[0076] Since the matrix elements are read from the storage device only once during the calculation of the Fc-Logsoftmax-topk fusion operator, I / O operations are effectively reduced. Furthermore, the calculation of the softmax value is merged with the calculation of the TopK value, thus reducing redundant computation and improving data processing efficiency.

[0077] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0078] Example 1

[0079] Figure 6 A flowchart of a chip-based fusion operator processing method provided in one embodiment of this application is shown below. Figure 6 As shown, the execution entity in this embodiment is a chip-based fusion operator processing device, which is located on a chip. Specifically, it can be... Figure 2 If the computing device in the embodiment is a chip-based fusion operator processing method, then the method provided in this embodiment includes the following steps:

[0080] Step 601: Load the matrix units from the chip's storage device onto the chip's computing device in the order in which the target matrix is ​​split into matrix units. The matrix units are obtained by splitting the target matrix according to a preset matrix unit dimension.

[0081] Here, the target matrix is ​​the matrix corresponding to the row vectors in the fully connected matrix FC. For example, if the dimension of FC is N*Co, then the dimension of the target matrix is ​​1*Co.

[0082] The storage device is used to store the data to be processed; it can be DRAM or DDR memory.

[0083] In this embodiment, when the target matrix is ​​split into matrix units, it can be split according to a preset matrix unit dimension, such as Coi, where Coi can be 64, 128, etc., and the dimension of the matrix unit is 1*Coi. The split matrix units can be represented as A0, B0, C0, D0, ...

[0084] Specifically, in this embodiment, the computing device reads the matrix units sequentially from the storage device according to the order of partitioning, such as... Figure 3 As shown, the read matrix units can be cached in DMA 333 of storage module 33 in the storage module. Or as... Figure 4 As shown, the read matrix cells are cached in SRAM 408.

[0085] It is understandable that after the computing device acquires a matrix unit, step 602 can be executed. That is, steps 601-602 is a cyclical iterative process of reading and processing data.

[0086] Step 602: The computing device iteratively calculates and sums the normalized denominators of each matrix element, determines the TopK value of each matrix element, and iteratively processes it until the sum of the normalized denominators of all matrix elements in the target matrix and the TopK value corresponding to the target matrix are obtained.

[0087] Specifically, in this embodiment, after each matrix unit is read, the computing device calculates the normalized denominator of the matrix unit and sums it with the normalized denominator of the matrix unit obtained in the previous calculation to obtain the sum of the normalized denominators of the current matrix unit.

[0088] In this embodiment, after each matrix unit is read, the computing device obtains the TopK value determined when the previous iteration calculation was completed, and filters the matrix unit read this time with the TopK value determined when the previous iteration calculation was completed, thereby determining the TopK value when the current calculation is completed.

[0089] Understandably, step 602 iteratively calculates the normalized denominator of each matrix element and sums them, and determines the TopK value of each matrix element and iteratively processes it. Figure 3 The computation module within it performs the execution; specifically, it can be performed by... Figure 3 The vector operation unit 321 in the computing device performs the operation. The normalized denominator of the matrix unit obtained in the previous calculation and the TopK value determined when the previous iteration calculation is completed, etc., can be stored in the DMA 333 of the computing device.

[0090] Alternatively, step 602 can iteratively calculate the normalized denominator of each matrix element and sum them, and determine the TopK value of each matrix element and iteratively process it. Figure 4 It is executed by at least one processor core.

[0091] Intermediate results such as the normalized denominator of the matrix unit obtained in the previous calculation and the TopK value determined when the previous iteration calculation was completed can be stored in SRAM 408.

[0092] It is understandable that whether DMA 333 or SRAM 408 stores intermediate results, it is necessary to store the intermediate result of the previous calculation at the current calculation time. After the current intermediate result is calculated, the intermediate result of the previous calculation is discarded, and the current intermediate result is used as the intermediate result for the next calculation. Therefore, it can effectively reduce the amount of data stored and will not cause insufficient storage space.

[0093] Therefore, after the iterative reading and processing of data in steps 601 and 602, the sum of the normalized denominators of all matrix elements in the target matrix and the TopK value corresponding to the target matrix can be obtained.

[0094] The sum of the normalized denominators of all matrix elements in the target matrix can be expressed as SumA0B0C0……Z0.

[0095] Step 603: Using a computing device, calculate the normalized numerator of each TopK value based on the TopK value corresponding to the target matrix.

[0096] In this embodiment, from Figure 3 DMA 333 or Figure 4 The TopK value obtained from the last calculation is retrieved from SRAM 408. This last calculated TopK value corresponds to the TopK value of the target matrix, and is then used... Figure 3 The vector operation module 32 in the middle determines the maximum value among the TopK values ​​corresponding to the target matrix or uses... Figure 4 At least one processor core in the process determines the maximum value among the TopK values ​​corresponding to the target matrix, and then calculates the normalized numerator of each TopK value based on the TopK value and the maximum value corresponding to the target matrix.

[0097] The maximum value among the TopK values ​​corresponding to the target matrix can be represented as tmax, and the normalized numerator of each TopK value can be represented as: e xi-tmax , where i is 1, 2, ..., K.

[0098] Understandably, the normalized numerator of each TopK value can be stored in... Figure 3 DMA 333 or Figure 4 In SRAM 408.

[0099] Step 604: Using a computing device, the processing result of the fusion operator of the target matrix is ​​obtained based on the sum of the normalized denominators corresponding to the target matrix and the normalized numerators of each TopK value.

[0100] Specifically, in this embodiment, Figure 3The computation module 32 obtains the sum of the normalized numerator of each TopK value from DMA 333 and the sum of the normalized denominators of the matrix elements obtained in the last calculation in step 602. The sum of the normalized denominators of the matrix elements obtained in the last calculation is then used as the sum of the normalized denominators of the target matrix. The computation module 32 then calculates the processing result of the fusion operator for the target matrix.

[0101] Or, in this embodiment, Figure 4 At least one processor core retrieves the sum of the normalized numerator and the normalized denominator of each TopK value from the SRAM408. Then, at least one processor core calculates the processing result of the fusion operator for the target matrix.

[0102] The result of the fusion operator on the target matrix can be expressed as: Where i is 1, 2, ..., K. topxi represents the i-th element in the TopK elements of the target matrix, and Z0 represents the last matrix element.

[0103] The chip-based fusion operator processing method provided in this embodiment loads matrix units sequentially from the chip's storage device onto the chip's computing device in the order of splitting the target matrix into matrix units. The matrix units are obtained by splitting the target matrix according to a preset matrix unit dimension. The computing device iteratively calculates and sums the normalized denominators of each matrix unit, determines the TopK value of each matrix unit, and iteratively processes this process until the sum of the normalized denominators of all matrix units in the target matrix and the corresponding TopK value of the target matrix are obtained. The computing device then calculates the normalized numerator of each TopK value based on the TopK value of the target matrix. Finally, the computing device obtains the processing result of the fusion operator for the target matrix based on the sum of the normalized denominators of the target matrix and the normalized numerator of each TopK value. Because the processes of reading matrix units, calculating and summing the normalized denominators of matrix units, and determining the TopK value of the target matrix are integrated into a loop iterative calculation, redundant calculations are effectively reduced, improving data processing efficiency. Furthermore, since only one matrix unit needs to be read from the storage device, I / O operations are also effectively reduced.

[0104] Example 2

[0105] Figure 7 A flowchart of a chip-based fusion operator processing method provided in another embodiment of this application is shown below. Figure 7 As shown, the chip-based fusion operator processing method provided in this embodiment further refines step 602 based on the chip-based fusion operator processing method provided in Embodiment 1, wherein, as... Figure 3As shown, the computing device includes a storage module and a processing module. In the chip-based fusion operator processing method provided in this embodiment, step 602 involves iteratively calculating and summing the normalized denominator of each matrix element using the computing device until a further refinement of the sum of the normalized denominators of all matrix elements in the target matrix is ​​obtained. This includes the following steps:

[0106] Step 6021: Move the matrix unit of the current iteration from the storage module to the operation module, and use the operation module to calculate the normalized denominator of the matrix unit of the current iteration.

[0107] As an optional implementation, in this embodiment, step 6021 includes the following steps:

[0108] Step 6021a: Move the matrix cell of the current iteration from the storage module to the operation module, and use the operation module to determine the initial maximum value element in the matrix cell of the current iteration.

[0109] For example, if the matrix element in the current iteration is B0, then after calculating the normalized denominator of A0 in the previous iteration, A0 can be expressed as SumA0=∑e xj-t0 , where xj is the j-th element in A0, and t0 is the maximum value element in A0, which is the initial maximum value element of A0.

[0110] When iterating and calculating the normalized denominator of the current matrix unit and summing it, B0 is moved from the storage module to the operation module. The operation module is used to determine the maximum value element of B0, which is the initial maximum value element.

[0111] Step 6021b: Move the target maximum value element of the matrix cell from the storage module to the operation module, and use the operation module to determine the maximum value between the target maximum value element and the initial maximum value element of the matrix cell from the previous iteration, and determine this maximum value as the target maximum value element corresponding to the matrix cell in the current iteration.

[0112] In this process, the target maximum element of the matrix unit in the previous iteration is the initial maximum element t0 in A0. t0 is moved from the storage module to the operation module, and the operation module determines the maximum value between t0 and t1, which can be represented as t'. If the maximum value t' = t0, then t0 is determined as the target maximum element corresponding to B0; if the maximum value is t' = t1, then t1 is determined as the target maximum element of B0.

[0113] Step 6021c: The calculation module calculates the normalized denominator of the matrix unit in the current iteration based on the target maximum value element.

[0114] Specifically, if t0 > t1, then the normalized denominator of B0 is expressed as: SumB0 = ∑exj-t0 If t0 < t1, the normalized denominator of B0 is expressed as: SumB0 = ∑e xj-t1 .

[0115] Exemplarily, taking the matrix cell of the current iteration as C0, steps 6021a - 6021c will be described.

[0116] Specifically, the matrix cell C0 of the current iteration is moved from the storage module to the operation module, and the operation module is used to determine the initial maximum value element in C0 as t2. The target maximum value element of the matrix cell in the previous iteration is t', t' is moved from the storage module to the operation module, and the operation module is used to determine the maximum value of t' and t2, and the maximum value is determined as the target maximum value element of C0. If t' > t2, then t' is the target maximum value element of C0, and the normalized denominator of C0 is expressed as: SumC0 = ∑e xj-t′ If t' < t2, then t2 is the target maximum value element of C0, and the normalized denominator of C0 is expressed as SumC0 = ∑e xj-t2 .

[0117] In this embodiment, when moving the matrix cell of the current iteration from the storage module to the operation module and using the operation module to calculate the normalized denominator of the matrix cell of the current iteration, the matrix cell of the current iteration is moved from the storage module to the operation module, and the operation module is used to determine the initial maximum value element in the matrix cell of the current iteration; the target maximum value element of the matrix cell in the previous iteration is moved from the storage module to the operation module, and the operation module is used to determine the maximum value between the target maximum value element and the initial maximum value element of the matrix cell in the previous iteration, and this maximum value is determined as the target maximum value element corresponding to the matrix cell of the current iteration; the operation module is used to calculate the normalized denominator of the matrix cell of the current iteration according to the target maximum value element, which can ensure that when calculating the normalized denominator of the matrix cell of the current iteration, it is calculated using the maximum value element in the target matrix after iterative calculation, thereby ensuring the accuracy of the calculation of the normalized denominator of the matrix cell of the current iteration.

[0118] Step 6022, use the operation module to sum the sum of the normalized denominators calculated in the previous iteration and the normalized denominator of the matrix cell of the current iteration to obtain the sum of the normalized denominators after the current iteration until the sum of the normalized denominators of all matrix cells in the target matrix is obtained.

[0119] For example, let's take the matrix element of the current iteration as C0. The sum of the normalized denominators calculated in the previous iteration can be represented as SumA0B0. The normalized denominator of the matrix element in the current iteration is SumC0. The operation module sums SumA0B0 and SumC0 to obtain the sum of the normalized denominators after the current iteration, which can be represented as SumA0B0C0 = SumA0B0 + SumC0.

[0120] Optionally, to ensure the accuracy of the sum of the normalized denominators after the current iteration, the following steps are included before step 6022:

[0121] Step 6022a: The operation module is used to determine whether the target maximum value element of the matrix cell in the previous iteration is less than the initial maximum value element of the matrix cell in the current iteration.

[0122] Step 6022b: If it is determined that the target maximum value element of the matrix cell in the previous iteration is less than the initial maximum value element of the matrix cell in the current iteration, then update the sum of the normalized denominators calculated in the previous iteration.

[0123] In this embodiment, the sum of the normalized denominators calculated in the previous iteration was based on the maximum value of the elements in the target matrix up to the end of the previous iteration. This maximum value is the target maximum value element of the matrix cell in the previous iteration. However, when processing the matrix cell in the current iteration, the initial maximum value element of the matrix cell is introduced. Therefore, to ensure that the sum of the normalized denominators after the current iteration is calculated using the maximum value of the elements in the target matrix up to the end of the current iteration, it is necessary to determine whether the sum of the normalized denominators calculated in the previous iteration needs to be updated.

[0124] Specifically, the calculation module determines whether the target maximum value element of the matrix cell in the previous iteration is less than the initial maximum value element of the matrix cell in the current iteration. If it is determined that the target maximum value element of the matrix cell in the previous iteration is less than the initial maximum value element of the matrix cell in the current iteration, it means that the maximum value of the elements in the target matrix when the previous iteration was completed is not the maximum value when the current iteration was calculated. Therefore, the sum of the normalized denominators calculated in the previous iteration is updated. When updating the sum of the normalized denominators calculated in the previous iteration, the specific update method is that SumY0 is updated to SumY0 / e. ty *e t′ .

[0125] Where ty represents the initial maximum value element in the matrix cell of the current iteration. t′ represents the target maximum value element in the matrix cell of the previous iteration.

[0126] It can be understood that if it is determined that the target maximum value element of the matrix unit in the previous iteration is greater than or equal to the initial maximum value element in the matrix unit of the current iteration, there is no need to update the sum of the normalized denominators calculated in the previous iteration.

[0127] Exemplarily, the matrix unit of the current iteration is used as C0 for illustration. The initial maximum value element of C0 is denoted as t2.

[0128] Specifically, the operation module obtains the target maximum value element of the matrix unit in the previous iteration from the storage module. For C0, the target maximum value element of the matrix unit in the previous iteration is t'. The magnitude relationship between t' and t2 is judged. If t' < t2, it indicates that the sum of the normalized denominators calculated in the previous iteration is not calculated using the maximum value of the elements in the target matrix up to the current iteration calculation. Then, the sum of the normalized denominators calculated in the previous iteration is updated. For C0, the sum of the normalized denominators SumA0B0 calculated in the previous iteration is updated to SumA0B0 / e t2 *e t′ .

[0129] It should be noted that if the sum of the normalized denominators calculated in the previous iteration is updated, step 6022 specifically includes:

[0130] The operation module is used to sum the updated sum of the normalized denominators calculated in the previous iteration and the normalized denominator of the matrix unit of the current iteration to obtain the sum of the normalized denominators after the current iteration.

[0131] Continuing with the above example, SumA0B0C0 = SumA0B0 / e t2 *e t′ +SumC0

[0132] In the chip-based fusion operator processing method provided in this embodiment, before using the operation module to sum the sum of the normalized denominators calculated in the previous iteration and the normalized denominator of the matrix unit of the current iteration to obtain the sum of the normalized denominators after the current iteration, the operation module judges whether the target maximum value element of the matrix unit in the previous iteration is less than the initial maximum value element in the matrix unit of the current iteration; if it is determined that the target maximum value element of the matrix unit in the previous iteration is less than the initial maximum value element in the matrix unit of the current iteration, the sum of the normalized denominators calculated in the previous iteration is updated. When it is determined that the target maximum value element of the matrix unit in the previous iteration is less than the initial maximum value element in the matrix unit of the current iteration, by updating the sum of the normalized denominators calculated in the previous iteration, the accuracy of the sum of the normalized denominators after the current iteration can be effectively guaranteed.

[0133] Example 3

[0134] Figure 8 A flowchart of a chip-based fusion operator processing method provided in another embodiment of this application is shown below. Figure 8 As shown, the chip-based fusion operator processing method provided in this embodiment, based on the chip-based fusion operator processing method provided in any of the above embodiments, further refines the TopK value of each matrix unit determined by the computing device in step 602 and iteratively processes it until the TopK value corresponding to the target matrix is ​​obtained. Therefore, in this embodiment, step 602 also includes the following steps:

[0135] Step 6023: Move the matrix unit of the current iteration from the storage module to the operation module, and use the operation module to determine the TopK value in the matrix unit of the current iteration.

[0136] In this embodiment, the matrix unit of the current iteration includes more than K matrix elements. After the matrix unit of the current iteration is moved from the storage module to the operation module, the operation module compares the more than K matrix elements in the matrix unit of the current iteration to determine the top K matrix elements. The top K matrix elements are the TopK values ​​in the matrix unit of the current iteration.

[0137] For example, the matrix unit of the current iteration is C0, which has 64 matrix elements. The operation unit compares the size of these 64 matrix elements and determines the top K matrix elements from largest to smallest as the TopK value of C0.

[0138] Step 6024: Move the TopK value determined when the previous iteration was completed from the storage module to the operation module, and use the operation module to filter the TopK value when the current iteration was completed from the TopK value of the matrix unit in the current iteration and the TopK value determined when the previous iteration was completed, until the TopK value corresponding to the target matrix is ​​obtained.

[0139] Specifically, the TopK value determined at the end of the previous iteration is stored in the storage module. The computation module reads the TopK value determined at the end of the previous iteration from the storage module. It then obtains the TopK value of the matrix unit in the current iteration, compares the two TopK values ​​again, sorts them from largest to smallest, and determines the TopK value at the end of the current iteration as the value in the top K values.

[0140] Similarly, in the next iteration, the TopK value determined when the current iteration is completed is obtained. The operation module then selects the TopK value for the next iteration from the TopK value of the matrix unit in the next iteration and the TopK value determined when the current iteration is completed. This process is repeated until the TopK value corresponding to the target matrix is ​​obtained.

[0141] For example, the matrix element in the current iteration is C0, and the TopK value of C0 is TopK_C0. The TopK value determined when the previous iteration was completed is TopK_A0B0. Then, TopK_C0 and TopK_A0B0 are compared again to determine the TopK value when the current iteration is completed as TopK_A0B0C0.

[0142] The matrix element for the next iteration is D0, and the TopK value of D0 is TopK_D0. The TopK value determined at the end of the previous iteration is TopK_A0B0C0. Therefore, TopK_D0 and TopK_A0B0C0 are compared again to determine the TopK value for the next iteration: TopK_A0B0C0D0. The TopK value corresponding to the target matrix is ​​then TopK_A0B0C0D0…Z0, where Z0 is the last matrix element in the target matrix.

[0143] The chip-based fusion operator processing method provided in this embodiment, after using a computing device to determine the TopK value of each matrix unit and iteratively processing it until the TopK value corresponding to the target matrix is ​​obtained, moves the matrix unit of the current iteration from the storage module to the operation module, and uses the operation module to determine the TopK value of the matrix unit in the current iteration; the TopK value determined at the end of the previous iteration is moved from the storage module to the operation module, and the operation module filters the TopK value at the end of the current iteration from the TopK value of the matrix unit in the current iteration and the TopK value determined at the end of the previous iteration, until the TopK value corresponding to the target matrix is ​​obtained. Since the iterative processing of the TopK value of the matrix unit is performed when calculating the TopK value of the target matrix, it is not necessary to read the matrix unit from the storage device again to determine the TopK value of the target matrix, thus effectively reducing I / O operations. Furthermore, it is not necessary to calculate the Softmax value of other matrix elements besides the TopK value, thus further reducing redundant calculations and improving data processing efficiency.

[0144] Example 4

[0145] Figure 9 A flowchart of a chip-based fusion operator processing method provided in another embodiment of this application is shown below. Figure 9As shown, the chip-based fusion operator processing method provided in this embodiment is a further refinement of step 603 based on the chip-based fusion operator processing method provided in any of the above embodiments. Therefore, step 603 of the chip-based fusion operator processing method provided in this embodiment includes the following steps:

[0146] Step 6031: The operation module obtains the TopK value corresponding to the target matrix from the storage module, and determines the maximum value among the TopK values ​​corresponding to the target matrix.

[0147] The storage module stores the TopK values ​​corresponding to the target matrix. The computation module reads the TopK values ​​corresponding to the target matrix from the storage module and determines the maximum value among the TopK values ​​corresponding to the target matrix as tmax.

[0148] Step 6032: The calculation module calculates the normalized numerator of each TopK value based on the maximum value among the TopK values ​​corresponding to each TopK value and the target matrix.

[0149] Wherein, the normalized numerator of each TopK value can be expressed as e topxi-tmax Where topxi represents the i-th element in the TopK values.

[0150] The chip-based fusion operator processing method provided in this embodiment, when using a computing device to calculate the normalized numerator of each TopK value based on the TopK value corresponding to the target matrix, uses a computation module to obtain the TopK value corresponding to the target matrix from the storage module and determine the maximum value among the TopK values ​​corresponding to the target matrix. The computation module calculates the normalized numerator of each TopK value based on each TopK value and the maximum value among the TopK values ​​corresponding to the target matrix. Since the TopK value corresponding to the target matrix is ​​calculated based on the TopK value of the target matrix, it is only necessary to read the TopK value corresponding to the target matrix from the storage module to calculate the maximum value among the TopK values ​​corresponding to the target matrix in the computation module, without having to read data from the storage device again, thus effectively reducing I / O operations.

[0151] Optionally, the chip-based fusion operator processing method provided in this embodiment, step 604, is based on the chip-based fusion operator processing method provided in any of the above embodiments, and includes the following steps:

[0152] Step 6041: The calculation module is used to calculate the quotient of the sum of the normalized numerator of each TopK value and the normalized denominator of the target matrix.

[0153] The quotient of the sum of the normalized numerator and the normalized denominator of the target matrix for each TopK value is expressed as:

[0154] Step 6042: Determine each quotient as the processing result of the fusion operator of the target matrix.

[0155] The result of the fusion operator for the target matrix can be represented as softmax Topxi, therefore

[0156] Example 5

[0157] Figure 10 This application also provides a flowchart of a chip-based fusion operator processing method according to an embodiment, such as... Figure 10 As shown, the chip-based fusion operator processing method provided in this embodiment is based on Figure 4 The multi-core computing device shown. Among them, such as... Figure 4 As shown, the storage device is DRAM, the storage module is shared random access memory (SRAM), and the processing module is at least one processor core. Therefore, the chip-based fusion operator processing method provided in this embodiment includes the following steps:

[0158] Step 701: Load the matrix units from the DRAM onto the SRAM on the chip in the order in which the target matrix is ​​split into matrix units.

[0159] The matrix unit is obtained by splitting the target matrix according to the preset matrix unit dimensions.

[0160] like Figure 4 As shown, the matrix units are loaded onto the SRAM via GDMA.

[0161] Step 702: Use at least one processor core to iteratively calculate and sum the normalized denominators of each matrix element, determine the TopK value of each matrix element and iteratively process it until the sum of the normalized denominators of all matrix elements in the target matrix and the TopK value corresponding to the target matrix are obtained.

[0162] Specifically, the matrix element of the current iteration is moved from SRAM to at least one processor core, and the normalized denominator of the matrix element of the current iteration is calculated using at least one processor core. The sum of the normalized denominators calculated in the previous iteration is then summed with the normalized denominator of the matrix element of the current iteration using at least one processor core to obtain the sum of the normalized denominators after the current iteration, until the sum of the normalized denominators of all matrix elements in the target matrix is ​​obtained.

[0163] For example, such as Figure 4As shown, four processor cores simultaneously read the matrix elements of the current iteration from SRAM, and each processor core calculates the normalized denominator of the matrix elements in the current iteration. The four processor cores simultaneously sum the sum of the normalized denominators calculated in the previous iteration with the normalized denominator of the matrix elements in the current iteration to obtain the sum of the normalized denominators after the current iteration. This process is repeated until the sum of the normalized denominators of all matrix elements in the target matrix is ​​obtained.

[0164] The method by which each processor core calculates the normalized denominator of the matrix element in the current iteration is similar to the method used by the arithmetic module in the above embodiment, and will not be described in detail here. Similarly, the method by which at least one processor core sums the sum of the normalized denominators calculated in the previous iteration with the normalized denominator of the matrix element in the current iteration to obtain the sum of the normalized denominators after the current iteration is similar to the method used by the arithmetic module in the above embodiment, and will not be described in detail here.

[0165] Step 703: Using at least one processor core, calculate the normalized numerator of each TopK value based on the TopK value corresponding to the target matrix.

[0166] For example, such as Figure 4 As shown, four processor cores are used simultaneously to calculate the normalized numerator of each TopK value based on the TopK value corresponding to the target matrix, or any one or more of the four processor cores are used to calculate the normalized numerator of each TopK value based on the TopK value corresponding to the target matrix.

[0167] The method of using the processor core to calculate the normalized numerator of each TopK value based on the TopK value corresponding to the target matrix is ​​similar to the method of using the operation module to calculate the normalized numerator of each TopK value based on the TopK value corresponding to the target matrix in the above embodiment, and will not be described in detail here.

[0168] Step 704: Using at least one processor core, the processing result of the fusion operator of the target matrix is ​​obtained based on the sum of the normalized denominators corresponding to the target matrix and the normalized numerators of each TopK value.

[0169] For example, such as Figure 4As shown, one or more of the four processor cores simultaneously calculate the result of the fusion operator for the target matrix based on the sum of the normalized denominators corresponding to the target matrix and the normalized numerators of each TopK value. The specific implementation is similar to that of the computation module, which calculates the result of the fusion operator for the target matrix based on the sum of the normalized denominators corresponding to the target matrix and the normalized numerators of each TopK value; therefore, it will not be elaborated upon here.

[0170] The chip-based fusion operator processing method provided in this embodiment is applicable to multiple computing devices. It integrates the processes of reading matrix units, calculating and summing the normalized denominators of the matrix units, and determining the TopK value of the target matrix into a single iterative computation, effectively reducing redundant calculations and improving data processing efficiency. Furthermore, it only requires reading the matrix units from the storage device once, thus effectively reducing I / O operations.

[0171] Example 6

[0172] Figure 11 This application also provides a schematic diagram of the structure of a chip-based fusion operator processing device according to an embodiment, as shown below. Figure 11 As shown, the chip-based fusion operator processing device provided in this application embodiment is used to execute... Figure 6 The chip-based fusion operator processing method shown in this embodiment includes a chip-based fusion operator processing device 80 comprising a loading unit 81, an iterative processing unit 82, and a calculation unit 83.

[0173] The loading unit 81 is used to load matrix units from the chip's storage device sequentially onto the computing device on the chip in the order in which the target matrix is ​​split into matrix units. The matrix units are obtained by splitting the target matrix according to a preset matrix unit dimension. The iterative processing unit 82 is used to iteratively calculate and sum the normalized denominators of each matrix unit using the computing device, and determine the TopK value of each matrix unit and iteratively process it until the sum of the normalized denominators of all matrix units in the target matrix and the corresponding TopK value of the target matrix are obtained. The computing unit 83 is used to calculate the normalized numerator of each TopK value based on the corresponding TopK value of the target matrix using the computing device. The computing unit 83 is also used to obtain the processing result of the fusion operator of the target matrix based on the sum of the normalized denominators of the target matrix and the normalized numerator of each TopK value using the computing device.

[0174] Optionally, the computing device includes a storage module and a processing module.

[0175] The iterative processing unit 82, when iteratively calculating and summing the normalized denominators of each matrix element using a computing device until the sum of the normalized denominators of all matrix elements in the target matrix is ​​obtained, is specifically used for:

[0176] The matrix element of the current iteration is moved from the storage module to the operation module, and the operation module is used to calculate the normalized denominator of the matrix element of the current iteration. The operation module is then used to sum the sum of the normalized denominators calculated in the previous iteration with the normalized denominator of the matrix element of the current iteration to obtain the sum of the normalized denominators after the current iteration, until the sum of the normalized denominators of all matrix elements in the target matrix is ​​obtained.

[0177] The iterative processing unit 82, when moving the matrix element of the current iteration from the storage module to the operation module and using the operation module to calculate the normalized denominator of the matrix element of the current iteration, is specifically used for:

[0178] The matrix element of the current iteration is moved from the storage module to the operation module, and the operation module is used to determine the initial maximum value element in the matrix element of the current iteration; the target maximum value element of the matrix element of the previous iteration is moved from the storage module to the operation module, and the operation module is used to determine the maximum value between the target maximum value element and the initial maximum value element of the matrix element of the previous iteration, and this maximum value is determined as the target maximum value element of the matrix element of the current iteration; the operation module is used to calculate the normalized denominator of the matrix element of the current iteration based on the target maximum value element.

[0179] Before the iterative processing unit 82 uses the arithmetic module to sum the sum of the normalized denominators calculated in the previous iteration with the normalized denominators of the matrix units in the current iteration to obtain the sum of the normalized denominators after the current iteration, it is also used for:

[0180] The calculation module determines whether the target maximum value element of the matrix cell in the previous iteration is less than the initial maximum value element of the matrix cell in the current iteration; if it is determined that the target maximum value element of the matrix cell in the previous iteration is less than the initial maximum value element of the matrix cell in the current iteration, then the sum of the normalized denominators calculated in the previous iteration is updated.

[0181] The iterative processing unit 82, when using the arithmetic module to sum the sum of the normalized denominators calculated in the previous iteration with the normalized denominators of the matrix units in the current iteration to obtain the sum of the normalized denominators after the current iteration, is specifically used for:

[0182] The calculation module sums the sum of the normalized denominators calculated in the previous iteration with the normalized denominators of the matrix units in the current iteration to obtain the sum of the normalized denominators after the current iteration.

[0183] Optionally, the iterative processing unit 82, when using a computing device to determine the TopK value of each matrix element and iteratively processing until the TopK value corresponding to the target matrix is ​​obtained, is specifically used for:

[0184] The matrix element of the current iteration is moved from the storage module to the operation module, and the operation module is used to determine the TopK value of the matrix element of the current iteration. The TopK value determined when the previous iteration was completed is moved from the storage module to the operation module, and the operation module is used to filter the TopK value of the current iteration from the TopK value of the matrix element of the current iteration and the TopK value determined when the previous iteration was completed, until the TopK value corresponding to the target matrix is ​​obtained.

[0185] Optionally, the computing unit 83, when using a computing device to calculate the normalized numerator of each TopK value based on the TopK value corresponding to the target matrix, is specifically used for:

[0186] The computation module retrieves the TopK values ​​corresponding to the target matrix from the storage module and determines the maximum value among the TopK values ​​corresponding to the target matrix. The computation module then calculates the normalized numerator of each TopK value based on each TopK value and the maximum value among the TopK values ​​corresponding to the target matrix.

[0187] Optionally, the computing unit, when using a computing device to obtain the processing result of the fusion operator of the target matrix based on the sum of the normalized denominators corresponding to the target matrix and the normalized numerators of each TopK value, is specifically used for:

[0188] The computation module calculates the quotient of the sum of the normalized numerator and the normalized denominator of the target matrix for each TopK value; each quotient is then used as the result of the fusion operator for the target matrix.

[0189] Optionally, the storage device is DRAM, the storage module is shared random access memory (SRAM), and the arithmetic module is at least one processor core.

[0190] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the chip-based fusion operator processing method as described in any of the embodiments.

[0191] Computer-readable storage media can be any available medium or data storage device that a computer can access, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MOs), etc.), optical storage (such as CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (such as ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs)).

[0192] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the chip-based fusion operator processing method of any one of the embodiments.

[0193] This application also provides an artificial intelligence chip, including a storage device, a computing device, and a chip-based data processing device as provided in the above embodiments.

[0194] This application also provides an electronic device, which includes: an off-chip memory and an artificial intelligence chip as provided in the above embodiments.

[0195] This application also provides a board, which includes: a storage device, an external interface device, a control device, and an artificial intelligence chip as provided in the above embodiments;

[0196] The AI ​​chip is connected to a storage device, a controller, and an external interface device, respectively. The storage device is used to store target data. The external interface device is used to realize data transmission between the AI ​​chip and external devices. The controller is used to monitor the status of the AI ​​chip.

[0197] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0198] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0199] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0200] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0201] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, an AI processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, storage units can be any suitable magnetic or magneto-optical storage medium, such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), etc.

[0202] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0203] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

Claims

1. A chip-based fusion operator processing method, characterized in that, include: Matrix units are loaded sequentially from the chip's storage device onto the chip's computing device; wherein the matrix units are obtained by splitting the target matrix according to a preset matrix unit dimension; the order is obtained when the target matrix is ​​split into matrix units, and the target matrix is ​​the matrix corresponding to the row vectors in the fully connected matrix; The computing device using the chip iteratively calculates and sums the normalized denominators of each matrix unit, determines the TopK value of each matrix unit, and iteratively processes it until the sum of the normalized denominators of all matrix units in the target matrix and the TopK value corresponding to the target matrix are obtained. The computing device includes a computation module. The computing device is used to calculate the normalized numerator of each TopK value based on the TopK value corresponding to the target matrix; The computing device uses the sum of the normalized denominators corresponding to the target matrix and the normalized numerators of each TopK value to obtain the processing result of the fusion operator of the target matrix.

2. The method according to claim 1, characterized in that, The computing device also includes a storage module; The normalized denominator of each matrix element is iteratively calculated using a computing device and then summed until the sum of the normalized denominators of all matrix elements in the target matrix is ​​obtained, including: Move the matrix element of the current iteration from the storage module to the operation module, and use the operation module to calculate the normalized denominator of the matrix element of the current iteration. The calculation module sums the sum of the normalized denominators calculated in the previous iteration with the normalized denominators of the matrix elements in the current iteration to obtain the sum of the normalized denominators after the current iteration, until the sum of the normalized denominators of all matrix elements in the target matrix is ​​obtained.

3. The method according to claim 2, characterized in that, The step of moving the matrix element of the current iteration from the storage module to the operation module, and using the operation module to calculate the normalized denominator of the matrix element of the current iteration, includes: Move the matrix element of the current iteration from the storage module to the operation module, and use the operation module to determine the initial maximum value element in the matrix element of the current iteration; The target maximum value element of the matrix unit in the previous iteration is moved from the storage module to the operation module, and the operation module is used to determine the maximum value between the target maximum value element of the matrix unit in the previous iteration and the initial maximum value element, and this maximum value is determined as the target maximum value element corresponding to the matrix unit in the current iteration. The computation module calculates the normalized denominator of the matrix unit in the current iteration based on the target maximum value element.

4. The method according to claim 3, characterized in that, Before the step of using the calculation module to sum the sum of the normalized denominators calculated in the previous iteration with the normalized denominators of the matrix units in the current iteration to obtain the sum of the normalized denominators after the current iteration, the method further includes: The calculation module is used to determine whether the target maximum value element of the matrix cell in the previous iteration is less than the initial maximum value element of the matrix cell in the current iteration; If it is determined that the target maximum value element of the matrix cell in the previous iteration is less than the initial maximum value element of the matrix cell in the current iteration, then update the sum of the normalized denominators calculated in the previous iteration. The step of using the computation module to sum the sum of the normalized denominators calculated in the previous iteration with the normalized denominators of the matrix units in the current iteration, to obtain the sum of the normalized denominators after the current iteration, includes: The calculation module sums the sum of the normalized denominators calculated in the previous iteration with the normalized denominators of the matrix units in the current iteration to obtain the sum of the normalized denominators after the current iteration.

5. The method according to any one of claims 1-4, characterized in that, The TopK value of each matrix element is determined using a computing device and iteratively processed until the TopK value corresponding to the target matrix is ​​obtained, including: Move the matrix unit of the current iteration from the storage module to the operation module, and use the operation module to determine the TopK value in the matrix unit of the current iteration; The TopK value determined at the end of the previous iteration is moved from the storage module to the operation module. The operation module then filters the TopK value at the end of the current iteration from the TopK value of the matrix unit in the current iteration and the TopK value determined at the end of the previous iteration, until the TopK value corresponding to the target matrix is ​​obtained.

6. The method according to any one of claims 1-4, characterized in that, The step of using the computing device to calculate the normalized numerator of each TopK value based on the TopK value corresponding to the target matrix includes: The computation module retrieves the TopK values ​​corresponding to the target matrix from the storage module, and determines the maximum value among the TopK values ​​corresponding to the target matrix. The computation module calculates the normalized numerator of each TopK value based on each TopK value and the maximum value among the TopK values ​​corresponding to the target matrix.

7. The method according to any one of claims 1-4, characterized in that, The process of obtaining the fusion operator result of the target matrix using the computing device based on the sum of the normalized denominators corresponding to the target matrix and the normalized numerators of each TopK value includes: The computation module is used to calculate the quotient of the sum of the normalized numerator of each TopK value and the normalized denominator corresponding to the target matrix; Each quotient is determined as the result of the fusion operator of the target matrix.

8. The method according to any one of claims 2-4, characterized in that, The storage device is DRAM, the storage module is shared random access memory (SRAM), and the arithmetic module is at least one processor core.

9. A chip-based fusion operator processing device, characterized in that, include: A loading unit is used to load matrix units sequentially from the chip's storage device onto the chip's computing device; wherein the matrix units are obtained by splitting the target matrix according to a preset matrix unit dimension; the order is obtained when the target matrix is ​​split into matrix units, and the target matrix is ​​the matrix corresponding to the row vectors in a fully connected matrix; An iterative processing unit is used to iteratively calculate the normalized denominator of each matrix unit using the computing device of the chip and sum them up, and determine the TopK value of each matrix unit and iteratively process it until the sum of the normalized denominators of all matrix units in the target matrix and the TopK value corresponding to the target matrix are obtained. The computing device includes a computation module. The computing unit is used to calculate the normalized numerator of each TopK value based on the TopK value corresponding to the target matrix using the computing device; The computing unit is further configured to use the computing device to obtain the processing result of the fusion operator of the target matrix based on the sum of the normalized denominators corresponding to the target matrix and the normalized numerator of each TopK value.

10. An artificial intelligence chip, characterized in that, include: Storage device, computing device, and chip-based data processing device as described in claim 9.

11. An electronic device, characterized in that, The electronic device includes: off-chip memory and the artificial intelligence chip as described in claim 10.

12. A circuit board, characterized in that, The board includes: a storage device, an external interface device, and a control device, as well as the artificial intelligence chip as described in claim 10; The artificial intelligence chip is connected to the storage device, the control device, and the external interface device, respectively. The storage device is used to store the target data; The external interface device is used to realize data transmission between the artificial intelligence chip and external devices; The controller is used to monitor the state of the artificial intelligence chip.

Images