Data processing method and apparatus, and accelerator and computing device

Abstract

The present application relates to the technical field of artificial intelligence (AI). Disclosed are a data processing method and apparatus, and an accelerator and a computing device. The method comprises: when an AI accelerator executes an attention operator, a memory access engine reading an input matrix block in a first memory into a second memory; a computing engine calling the attention operator to process the input matrix block, so as to obtain an output matrix block; and then the memory access engine writing the output matrix block into the first memory. During this process, the read-write speed of the second memory is higher than the read-write speed of the first memory, and a normalization factor of a normalization operation in the attention operator is updated in the second memory, that is, it is unnecessary to write, into the first memory having a relatively low read-write speed, intermediate data generated during the process of executing the attention operator. Therefore, the method reduces the frequency of accessing the first memory, and the operation performance of the AI accelerator is effectively improved.

Description

Data processing method, device, accelerator and computing equipment

[0001] This application claims priority to Chinese patent application number 202311148416.3, filed on September 6, 2023, entitled “Data processing method, device, accelerator and computing device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a data processing method, apparatus, accelerator, and computing device. Background Art

[0003] With the rapid development of AI technology, a series of AI accelerators have emerged, providing computing power for matrix and vector calculations to accelerate the computational process of AI models. Currently, AI accelerators are typically equipped with various types of memory, which vary in storage capacity and read / write speed. Generally, the smaller the storage capacity, the faster the read / write speed.

[0004] Taking the attention operator in the Transformer model as an example, when the AI ​​accelerator calls the attention operator to process its input matrix, it needs to continuously access the high-bandwidth memory (HBM) configured on the AI ​​accelerator. For example, it needs to read data from the HBM and write intermediate data generated during the calculation process to the HBM. The high memory access volume leads to poor operating performance of the AI ​​accelerator.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a data processing method, apparatus, accelerator, and computing device that can effectively improve the operating performance of an AI accelerator. The technical solution is as follows:

[0007] In a first aspect, a data processing method is provided, which is applied to an artificial intelligence (AI) accelerator. The AI ​​accelerator includes a memory access engine and a computing engine. The AI ​​accelerator is configured with a first memory and a second memory, where the read and write speed of the second memory is greater than the read and write speed of the first memory. The method includes:

[0008] The memory access engine reads an input matrix block of the attention operator in the first memory into the second memory, wherein the input matrix block is a block of the input matrix of the attention operator;

[0009] The computing engine calls the attention operator and processes the input matrix block to obtain an output matrix block of the attention operator, wherein, in the process of processing the input matrix block, a normalization factor of the normalization operation of the attention operator in the second memory is updated;

[0010] The memory access engine writes the output matrix block of the attention operator into the first memory.

[0011] During this process, since the read and write speed of the second memory is greater than the read and write speed of the first memory, and the normalization factor of the normalization operation in the attention operator is updated in the second memory, that is, the intermediate data generated during the execution of the attention operator does not need to be written to the first memory with a slower read and write speed. Therefore, the above method reduces the access to the first memory and effectively improves the operating performance of the AI ​​accelerator.

[0012] In some embodiments, the memory access engine is a direct memory access (DMA) engine, the first memory is a high-bandwidth memory (HBM) or a double-data rate (DDR), the second memory is a static random access memory (SRAM), and the storage capacity of the first memory is greater than the storage capacity of the second memory.

[0013] In some embodiments, the input matrix block includes: a query matrix block, a key matrix block, and a value matrix block, wherein the query matrix block is a block of the query matrix of the attention operator, the key matrix block is a block of the key matrix of the attention operator, and the value matrix block is a block of the value matrix of the attention operator;

[0014] The computing engine calls the attention operator and processes the input matrix block to obtain an output matrix block of the attention operator, including:

[0015] The computing engine updates a first normalization factor of the first normalization operation in the second memory based on the query matrix block and the key matrix block, and calculates an attention weight matrix corresponding to the query matrix block based on the updated first normalization factor and the first normalization operation;

[0016] The computing engine updates the second normalization factor of the second normalization operation in the second memory based on the attention weight matrix and the value matrix block, and calculates the output matrix block corresponding to the query matrix block based on the updated second normalization factor and the second normalization operation.

[0017] In some embodiments, the computing engine includes a matrix computing engine and a vector computing engine, wherein the matrix computing engine and the vector computing engine execute in parallel the process of calling the attention operator to process the input matrix block.

[0018] In this way, different types of operations involved in the attention operator are assigned to different computing engines for execution, allowing multiple computing engines to call the attention operator in parallel to execute corresponding steps, thereby saving the running time of the attention operator and improving the running performance of the AI ​​accelerator.

[0019] In some embodiments, the computing engine updates a first normalization factor of a first normalization operation in the second memory based on the query matrix block and the key matrix block, and calculates an attention weight matrix corresponding to the query matrix block based on the updated first normalization factor and the first normalization operation, including:

[0020] The matrix calculation engine performs a matrix multiplication operation on the query matrix block and the transposed matrix of the key matrix block to obtain an attention score matrix corresponding to the query matrix block;

[0021] The vector calculation engine updates the first normalization factor in the second memory based on the attention score matrix, and calculates the attention weight matrix corresponding to the query matrix block based on the updated first normalization factor and the first normalization operation, where the first normalization factor indicates the maximum value of each row in the global attention score matrix.

[0022] In some embodiments, the vector calculation engine updates the first normalization factor in the second memory based on the attention score matrix, and calculates the attention weight matrix corresponding to the query matrix block based on the updated first normalization factor and the first normalization operation, including:

[0023] The vector calculation engine updates the first normalization factor stored in the second memory based on the maximum value of each row in the attention score matrix and the first normalization factor stored in the second memory;

[0024] The vector calculation engine performs the first normalization operation on the attention score matrix based on the updated first normalization factor to obtain the attention weight matrix corresponding to the query matrix block.

[0025] In some embodiments, the computing engine updates a second normalization factor of a second normalization operation in the second memory based on the attention weight matrix and the value matrix block, and calculates an output matrix block corresponding to the query matrix block based on the updated second normalization factor and the second normalization operation, including:

[0026] The matrix calculation engine performs a matrix multiplication operation on the attention weight matrix and the value matrix block to obtain a sub-output matrix block corresponding to the query matrix block;

[0027] The vector calculation engine updates the second normalization factor in the second memory based on the attention weight matrix, and calculates the output matrix block corresponding to the query matrix block based on the updated second normalization factor, the sub-output matrix block and the second normalization operation, where the second normalization factor indicates the sum of each row in the global attention weight matrix.

[0028] In some embodiments, the vector calculation engine updates the second normalization factor stored in the second memory based on the attention weight matrix, and calculates the output matrix block corresponding to the query matrix block based on the updated second normalization factor, the sub-output matrix block, and the second normalization operation, including:

[0029] The vector calculation engine updates the second normalization factor in the second memory based on the sum of each row in the attention weight matrix, the updated first normalization factor, and the second normalization factor stored in the second memory;

[0030] The vector calculation engine updates the intermediate output matrix block corresponding to the query matrix block based on the sub-output matrix block;

[0031] If the query matrix block is the last block of the query matrix, the vector calculation engine performs the second normalization operation on the intermediate output matrix block based on the updated second normalization factor to obtain the output matrix block corresponding to the query matrix block.

[0032] In a second aspect, an embodiment of the present application provides a data processing device, which is applied to an AI accelerator, wherein the AI ​​accelerator includes a memory access engine and a computing engine. The AI ​​accelerator is configured with a first memory and a second memory, and the read and write speed of the second memory is greater than the read and write speed of the first memory. The device includes: a memory access module, configured in the memory access engine, for implementing the steps executed by the memory access engine as involved in the aforementioned first aspect or any possible implementation of the first aspect; and a computing module, configured in the computing engine, for implementing the steps executed by the computing engine as involved in the aforementioned first aspect or any possible implementation of the first aspect.

[0033] In a third aspect, an embodiment of the present application provides an accelerator, comprising a communication interface, a memory access engine, and a computing engine, wherein the communication interface is used to provide program instructions and / or data, and the accelerator is used to implement the data processing method provided in the first aspect or any possible implementation of the first aspect.

[0034] In a fourth aspect, an embodiment of the present application provides a computing device comprising a computing device host and an AI accelerator, wherein the host is used to send data to the AI ​​accelerator and receive data sent by the AI ​​accelerator, and the AI ​​accelerator is used to execute the data processing method provided in the first aspect or any possible implementation of the first aspect.

[0035] In a fifth aspect, an embodiment of the present application provides a computing device cluster comprising multiple computing devices, wherein the computing devices include a host and an AI accelerator, wherein the host is used to send data to the AI ​​accelerator and receive data sent by the AI ​​accelerator, and the AI ​​accelerator is used to execute the data processing method provided in the first aspect or any possible implementation of the first aspect.

[0036] In a sixth aspect, embodiments of the present application provide a computer-readable storage medium for storing at least one program code segment, wherein the at least one program code segment is used to execute the data processing method provided in the first aspect or any possible implementation of the first aspect. The storage medium includes, but is not limited to, volatile memory, such as random access memory, and non-volatile memory, such as flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0037] In a seventh aspect, embodiments of the present application provide a computer program product that, when executed on an AI accelerator, causes the AI ​​accelerator to perform the data processing method provided in the first aspect or any possible implementation of the first aspect. The computer program product may be a software installation package, and when the aforementioned data processing method is required, the computer program product may be downloaded and executed on the AI ​​accelerator. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a schematic diagram of the calculation process of an attention operator;

[0039] FIG2 is a schematic diagram of an implementation environment provided by the present application;

[0040] FIG3 is a schematic diagram of the hardware structure of an accelerator provided in an embodiment of the present application;

[0041] FIG4 is a flow chart of a data processing method provided in an embodiment of the present application;

[0042] FIG5 is a flowchart of an execution of an attention operator provided in an embodiment of the present application;

[0043] FIG6 is a flowchart of another execution of an attention operator provided in an embodiment of the present application;

[0044] FIG7 is a schematic diagram of a pipeline arrangement of a data processing method provided in an embodiment of the present application;

[0045] FIG8 is a schematic structural diagram of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings. It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the input matrix of the attention operator involved in this application is obtained with full authorization.

[0047] For ease of understanding, the key terms and key concepts involved in this application are explained below.

[0048] Artificial intelligence (AI) models are a type of mathematical algorithm model that uses machine learning ideas to solve practical problems. AI models include a large number of parameters and calculation formulas (or calculation rules).

[0049] AI accelerators are a type of specialized hardware accelerator or computer system designed to accelerate AI applications, especially neural networks, machine vision, and machine learning. For example, they are used to provide computing power for calculating matrices, vectors, etc. to accelerate the calculation of AI models. Schematically, AI accelerators are, for example, graphics processing units (GPUs), intelligent processing units (IPUs), tensor processing units (TPUs), domain specific architecture (DSA) chips, etc., but this application is not limited thereto.

[0050] Deep learning (DL) is a branch of machine learning (ML). Deep learning studies the inherent patterns and representational hierarchies of sample data. The information gained from this learning process is highly helpful in interpreting data such as text, images, and sound. Generally speaking, deep learning is a complex machine learning algorithm, and the typical AI model that utilizes deep learning is the neural network model.

[0051] An operator (OP) is a computational unit or function running on a computing device. In deep learning, neural network layers and even entire models are composed of operators, which correspond to the computational logic within the neural network layer. For example, a convolution layer is an operator; the weight summation process in a fully-connected layer (FC layer) is also an operator.

[0052] The attention operator is the core operator of the self-attention layer in the Transformer model. The core idea of ​​the self-attention layer is to make the machine notice the correlation between different parts of the entire input. Schematically, for any vector in the input sequence of the self-attention layer (such as words, letters, image features, etc.), a linear transformation is performed on the vector to obtain the query (query, Q) matrix, key (key, K) matrix, and value (value, V) matrix corresponding to the vector, that is, the input matrices Q, K, and V of the attention operator are obtained. By executing the attention operator, the input matrix is ​​processed to obtain the corresponding output (output, O) matrix, that is, the correlation between different parts of the input sequence is obtained. This process is shown in Figure 1, which is a schematic diagram of the calculation process of the attention operator. The calculation process is referred to the following formula (1):

[0053] In the above formula (1), the matrices Q, K, and V are the input matrices of the attention operator, with a size of N×d, the matrix S is the attention score matrix, the matrix P is the attention weight matrix, and the matrix O is the output matrix, d k is the dimension of the key vector k, and N and d are both positive integers. It should be understood that the Softmax normalization operation is performed row-wise, that is, a Softmax normalization operation is performed once for each row, for a total of N rows.

[0054] In order to avoid overflow and other problems in the calculation process of Softmax normalization operation, in actual computer calculation, both the numerator and the denominator are usually divided by e max(X), the actual calculation of the Softmax normalization operation formula is shown in the following formula (2):

[0055] It can be seen that the calculation of the attention operator requires additional storage space to save the intermediate matrix S and matrix P. Each step of the calculation requires reading and writing data, and for each row of matrix data, the calculation of the Softmax normalization operation needs to be traversed 3 times in total to get the final result (the first time to find the maximum value max(X) in the row, the second time to calculate The third traversal calculates the results in sequence. That is, during the calculation process of the attention operator, there are multiple read and write operations on intermediate variables, which reduces the overall calculation efficiency.

[0056] The following is an introduction to the application scenarios and implementation environment involved in this application.

[0057] The technical solution provided by the embodiment of the present application can be applied to the process of executing the attention operator based on the AI ​​accelerator. In combination with the attention operator calculation process shown in Figure 1 above, it can be seen that intermediate data will be generated during the execution of the attention operator: matrix S and matrix P, as well as the maximum value within the row, etc. Therefore, during the execution of the attention operator, the AI ​​accelerator needs to continuously access the memory of the AI ​​accelerator to read and write matrix S and matrix P, resulting in excessive memory access and poor performance of the AI ​​accelerator. In the related art, another implementation method of the attention operator is provided, the FlashAttention algorithm, which aims to transform the Softmax normalization operation that originally required global data into an implementation method through "local data calculation + rolling update". Schematically, the FlashAttention algorithm divides the input matrix of the attention operator into blocks to obtain input matrix blocks, and by processing the input matrix blocks, it reduces the access volume of the memory HBM and improves the performance of the AI ​​accelerator. However, this method still requires writing the intermediate data generated based on the input matrix blocks during the execution of the attention operator into the memory HBM, resulting in a high access volume of the memory HBM.

[0058] Based on this, this application provides a data processing method applied to the attention operator, which can reduce memory access and improve the operating performance of the AI ​​accelerator.

[0059] The implementation environment of this application is described below with reference to Figure 2. Figure 2 is a schematic diagram of an implementation environment provided by this application. As shown in Figure 2, the implementation environment includes a host 201 and an AI accelerator 202, and the host 201 and the AI ​​accelerator 202 are in communication with each other.

[0060] Host 201 is a device used to run AI models and can provide AI services to users. In the embodiment of this application, host 201 can control AI accelerator 202 to call the attention operator and implement the function of the attention operator. This process can also be understood as loading the AI ​​task into AI accelerator 202 for execution. In addition, the number of hosts 201 can be one or more, and this application does not limit this.

[0061] The AI ​​accelerator 202 is used to provide computing power for the running AI model to accelerate the calculation process of the AI ​​model, that is, to execute the data processing method provided in this application. For example, the AI ​​accelerator 202 is a DSA chip, a GPU, etc., but this application is not limited to this. Schematically, the AI ​​accelerator 202 calls the attention operator based on the input matrix block of the attention operator sent by the host 201, processes the input matrix block, obtains the output matrix block of the attention operator, and returns the output matrix block to the host 201 to implement the function of the attention operator. The architecture of the AI ​​accelerator 202 is introduced in detail through the content shown in Figure 3 below, and will not be repeated here. In addition, the number of AI accelerators 202 can be one or more, and this application is not limited to this.

[0062] Schematically, the host 201 and the AI ​​accelerator 202 are integrated into a computing device. The host 201 and the AI ​​accelerator 202 are communicatively connected via a peripheral component interconnect express (PCIe) link. The host 201 exchanges data with the AI ​​accelerator 202 via the PCIe link and controls the AI ​​accelerator 202 to call the attention operator. For example, the computing device can be an independent physical server, or a server cluster or distributed file system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0063] Taking a cloud server as an example, a computing device can also be called a cloud platform (short for cloud computing platform). This refers to services based on hardware and software resources, providing computing, networking, and storage capabilities. Through the network "cloud," massive amounts of data are processed and analyzed remotely before being returned to users. Cloud platforms offer advantages such as large-scale, distributed, virtualized, high availability, scalability, on-demand services, and security. Cloud platforms enable the rapid provisioning and release of configurable computing resources with minimal management overhead and interaction complexity between users and service providers.

[0064] It should be noted that in the implementation environment shown in Figure 2 above, the execution function of the attention operator is implemented through interaction between the host 201 and the AI ​​accelerator 202 as an example. In other embodiments, the AI ​​accelerator 202 has the function of running the AI ​​model, that is, the AI ​​accelerator 202 can directly run the user-specified AI model to implement the function of the attention operator. This application does not limit this.

[0065] In addition, the networks involved above include, but are not limited to, any combination of data center networks, storage area networks (SANs), local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), mobile, wired or wireless networks, dedicated networks or virtual private networks. In some implementations, technologies and / or formats including hypertext markup language (HTML) and extensible markup language (XML) are used to represent data exchanged over the network. In addition, conventional encryption technologies such as secure sockets layer (SSL), transport layer security (TLS), virtual private networks (VPNs), and internet protocol security (IPsec) can be used to encrypt all or part of the links. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the above data communication technologies.

[0066] The structure of the AI ​​accelerator 202 is introduced below.

[0067] An embodiment of the present application provides an accelerator, or an acceleration chip, which can be configured as the AI ​​accelerator 202 shown in the above implementation environment. Referring to Figure 3, Figure 3 is a schematic diagram of the hardware structure of an accelerator provided in an embodiment of the present application. As shown in Figure 3, the accelerator 300 includes a communication interface 301, a memory access engine 302, a computing engine 303, a memory 304, and a bus 305. Among them, the communication interface 301, the memory access engine 302, the computing engine 303, and the memory 304 are connected to each other through the bus 305.

[0068] The communication interface 301 is used to provide program instructions and / or data. The communication interface 301 may include a PCIe communication interface, other common peripheral interfaces, etc., which are not limited in this application. For example, when the accelerator 300 is used as an accelerator card for the host 201, data exchange is achieved with the host 201 via the PCIe communication interface. For another example, the accelerator 300 may communicate with other devices or communication networks via the peripheral interface.

[0069] The memory access engine 302 is used to read the input matrix block of the attention operator from the first memory of the AI ​​accelerator into the second memory of the AI ​​accelerator. The input matrix block is a block of the input matrix of the attention operator. The read and write speed of the second memory is greater than the read and write speed of the first memory. The memory access engine 302 is also used to write the output matrix block of the attention operator to the first memory. In some embodiments, the memory access engine 302 is a direct memory access engine (DMA engine), but this application is not limited to this.

[0070] The computing engine 303 is used to call the attention operator, process the input matrix block, and obtain the output matrix block of the attention operator, wherein, in the process of processing the input matrix block, the normalization factor of the normalization operation of the attention operator in the second memory is updated. In some embodiments, the computing engine 303 includes a matrix computing engine (cube engine), a vector computing engine (vector engine) and a scalar computing engine (scalar engine), wherein the matrix computing engine is used to perform matrix multiplication operations in the attention operator, the vector computing engine is used to perform vector operations in the attention operator, and the scalar computing engine is used to perform scalar operations in the attention operator. In some embodiments, the aforementioned computing engines can be executed in parallel to improve the accelerator operation performance.

[0071] The memory 304 includes high bandwidth memory (HBM), double data rate memory (DDR), static random-access memory (SRAM), or other types of dynamic storage devices that can store information and instructions, or also includes any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. Schematically, the first memory of the AI ​​accelerator is HBM and / or DDR, and the second memory of the AI ​​accelerator is SRAM, also referred to as shared memory in the embodiment of the present application. It should be understood that this is only an example, and the present application does not limit the implementation of the first memory and the second memory on the AI ​​accelerator. For example, taking the first memory as HBM as an example, the memory access engine 302 can read data stored in the HBM into the SRAM, and write data stored in the SRAM into the HBM. It should be understood that HBM has a large capacity but a slow read and write speed, while SRAM has a small capacity but a high read and write speed. Therefore, by reducing the amount of access to the HBM, the operating performance of the AI ​​accelerator can be greatly improved. For example, the A100 GPU has 40-80GB of HBM, with a bandwidth of 1.5-2.0TB / s; each of the 108 streaming multicore processors has 192KB of on-chip SRAM, with an estimated bandwidth of approximately 19TB / s. As can be seen, on-chip SRAM is an order of magnitude faster than HBM, but many orders of magnitude smaller.

[0072] The bus 305 may include a path for transmitting information between various components of the accelerator 300 (eg, the communication interface 301 , the memory access engine 302 , the computation engine 303 , and the memory 304 ).

[0073] It should be noted that the above Figure 3 is only a hardware structure diagram of an accelerator provided by this application that can be configured as the above-mentioned AI accelerator 202. In some embodiments, the accelerator 300 may also include other components to achieve more functions. For example, the accelerator 300 also includes a task scheduler (TS), a controller, etc., but this application is not limited to this.

[0074] The following is an introduction to the data processing method applied to the attention operator provided in this application.

[0075] Based on the above content, it can be seen that for an AI accelerator, in the process of executing the attention operator, the less access is made to the first memory with a slower read and write speed, the better the operating performance of the AI ​​accelerator. Based on this, the present application provides a data processing method, wherein a memory access engine reads the input matrix block in the first memory into a second memory, the computing engine calls the attention operator, processes the input matrix block, and after obtaining the output matrix block, the memory access engine writes the output matrix block into the first memory. In this process, since the read and write speed of the second memory is greater than the read and write speed of the first memory, and the normalization factor of the normalization operation in the attention operator is updated in the second memory, that is, the intermediate data generated during the execution of the attention operator does not need to be written to the first memory with a slower read and write speed. Therefore, the above method reduces the access to the first memory and effectively improves the operating performance of the AI ​​accelerator.

[0076] The data processing method provided by the present application is described below with reference to FIG4 . FIG4 is a flow chart of a data processing method provided by an embodiment of the present application. As shown in FIG4 , the method is applied to an AI accelerator, the AI ​​accelerator includes a memory access engine and a computing engine, the AI ​​accelerator is configured with a first memory and a second memory, the read and write speed of the second memory is greater than the read and write speed of the first memory, and the method includes the following steps 401 to 404.

[0077] 401. The memory access engine reads the input matrix block of the attention operator in the first memory into the second memory, where the input matrix block is a block of the input matrix of the attention operator.

[0078] In the embodiment of the present application, based on the above introduction to the attention operator, it can be seen that the input matrix of the attention operator is obtained by linearly transforming any vector in the input sequence, including the query matrix Q, the key matrix K, and the value matrix V. The present application does not limit the type of the input sequence. For example, the input sequence is a word, a letter, an image feature, etc. Correspondingly, the input matrix block of the attention operator includes: a query matrix block, a key matrix block, and a value matrix block, wherein the query matrix block is a block of the query matrix of the attention operator, the key matrix block is a block of the key matrix of the attention operator, and the value matrix block is a block of the value matrix of the attention operator.

[0079] It should be understood that since the present application divides the input matrix of the attention operator, that is, the attention operator is executed in blocks, the output matrix of the attention operator is also obtained in blocks. Therefore, in this step, the input matrix block of the attention operator is used to calculate the output matrix block. For example, the query matrix Q is divided into T rows. r query matrix block (T r is a positive integer, r is the abbreviation of row), the key matrix K and the value matrix V are divided into T by rowc Take a matrix block as an example (T c is a positive integer, c is the abbreviation of column column), in this step, the input matrix block includes the query matrix block Q i (i is a positive integer, 1≤i≤T r ), key matrix K j Sum matrix V j (j is a positive integer, 1≤j≤T c ), accordingly, the output matrix block is O i It should be noted that this application does not limit the timing of the memory access engine reading each input matrix block. For example, the memory access engine may read the output matrix block O in this step. i All the input matrix blocks involved are read into the second memory, or the query matrix block Q can be read first i Sum matrix block K j , in the subsequent calculation process, the value matrix V is involved j In the case of j Reading into the second memory can also dynamically read input matrix blocks into the second memory according to the size of the data stored in the second memory to improve the operating performance of the AI ​​accelerator, etc.

[0080] In addition, this application does not limit the size of the input matrix block, and it can be set according to needs. In some embodiments, the AI ​​accelerator divides the input matrix based on the storage capacity of the second memory and the size of the input matrix, obtains the input matrix block of the attention operator, and stores it in the first memory. This application does not limit the implementation method adopted for matrix block division. For example, taking the size of the input matrix as N×d and the storage capacity of the second memory as M as an example, the size of the input matrix block is set to B is the abbreviation of block. Then, the query matrix Q and the output matrix O are divided into T by row. r row blocks, each row block is of size [B r ,d], divide the key matrix K and value matrix V into T by row c row blocks, each row block is of size [B c , d]. In this way, the attention operator is executed in blocks, which provides technical support for the subsequent execution of the attention operator based on the second memory. It should be understood that although the read and write speed of the second memory is greater than the read and write speed of the first memory, its storage capacity is often smaller than the storage capacity of the first memory. Therefore, by dividing the input matrix of the attention operator, the storage capacity upper limit of the second memory can be met, thereby optimizing the calculation process of the attention operator and improving the operating performance of the AI ​​accelerator.

[0081] 402. The computing engine updates the first normalization factor of the first normalization operation in the second memory based on the query matrix block and the key matrix block in the input matrix block, and calculates the attention weight matrix corresponding to the query matrix block based on the updated first normalization factor and the first normalization operation.

[0082] In an embodiment of the present application, the normalization operation in the attention operator is split into a first normalization operation and a second normalization operation, corresponding to a first normalization factor and a second normalization factor, respectively, wherein the first normalization factor indicates the maximum value of each row in the global attention score matrix, and the second normalization factor indicates the sum of each row in the global attention weight matrix. It should be understood that these two normalization factors are global variables involved in the normalization operation in the attention operator, and involve continuous updates during the calculation process. Therefore, in the present application, the AI ​​accelerator initializes the normalization factor in the second memory and continuously updates the normalization factor in the second memory, thereby reducing the access to the first memory and improving the operating performance of the AI ​​accelerator.

[0083] The following describes the process in which the computing engine in this step performs the first normalization operation based on the query matrix block and the key matrix block to obtain the attention weight matrix corresponding to the query matrix block.

[0084] Schematically, the computing engine includes a matrix computing engine and a vector computing engine, wherein the matrix computing engine and the vector computing engine execute the process of calling the attention operator in parallel to process the input matrix block, including the following steps A1 and A2:

[0085] Step A1: The matrix calculation engine performs a matrix multiplication operation on the transposed matrices of the query matrix block and the key matrix block to obtain the attention score matrix corresponding to the query matrix block.

[0086] Among them, the query matrix block Q i and key matrix block K j For example, this step refers to the following formula (3):

[0087] In the above formula (3), S ij is the query matrix block Q i The corresponding attention score matrix, Denotes the key matrix block K j The transposed matrix, B r and B c are the row and column lengths of the matrix block, respectively.

[0088] Step A2: The vector calculation engine updates the first normalization factor in the second memory based on the attention score matrix, and calculates the attention weight matrix corresponding to the query matrix block based on the updated first normalization factor and the first normalization operation.

[0089] The first normalization factor indicates the maximum value of each row in the global attention score matrix, which is also a global variable stored in the second memory. It can speed up the reading and writing speed of the computing engine, thereby improving the operating performance of the AI ​​accelerator. It should be understood that this application executes the attention operator in blocks. Therefore, the AI ​​accelerator will continuously update the first normalization factor as the calculation progresses in each block of the attention operator to ensure the correctness of the calculation results.

[0090] Schematically, the vector calculation engine updates the first normalization factor stored in the second memory based on the maximum value of each row in the attention score matrix and the first normalization factor stored in the second memory; based on the updated first normalization factor, the first normalization operation is performed on the attention score matrix to obtain the attention weight matrix corresponding to the query matrix block. Based on the aforementioned formula (3), this process refers to the following formulas (4) to (8):

[0091] The above formulas (4) to (6) are also used to update the first normalization factor m i The process. Among them, formula (4) is used to calculate the query matrix block Q i The corresponding attention score matrix S ij The maximum value of each row in Formula (5) is used to determine the first normalization factor m′ stored in the second memory i and The maximum value between the two is used as the latest first normalization factor Then, based on the latest first normalization factor Update the first normalization factor m i .

[0092] The above formulas (7) and (8) are based on the updated first normalization factor m i , for the attention score matrix S ij The process of performing the first normalization operation. Wherein, formula (7) is used to calculate the attention score matrix S ij Each item in the first normalization factor m of the corresponding row i The difference between them, that is, point by point calculation, is used to obtain the updated attention score matrix Formula (8) is used to update the attention score matrix Perform exponential operation to obtain the query matrix block Q i The corresponding attention weight matrix

[0093] 403. The computing engine updates the second normalization factor of the second normalization operation in the second memory based on the attention weight matrix and the value matrix block, and calculates the output matrix block corresponding to the query matrix block based on the updated second normalization factor and the second normalization operation.

[0094] In the embodiment of the present application, taking the calculation engine including a matrix calculation engine and a vector calculation engine as an example, this step includes the following steps B1 and B2:

[0095] Step B1: The matrix calculation engine performs a matrix multiplication operation on the attention weight matrix and the value matrix block to obtain the sub-output matrix block corresponding to the query matrix block.

[0096] Wherein, based on the above formulas (3) to (8), this step refers to the following formula (9):

[0097] In the above formula (9), O ij is the query matrix block Q i The corresponding sub-output matrix block, V j is a block of value matrices.

[0098] Step B2: The vector calculation engine updates the second normalization factor in the second memory based on the attention weight matrix, and calculates the output matrix block corresponding to the query matrix block based on the updated second normalization factor, the sub-output matrix block and the second normalization operation.

[0099] The second normalization factor indicates the sum of each row in the global attention weight matrix, which is a global variable stored in the second memory. It can speed up the reading and writing speed of the computing engine, thereby improving the operating performance of the AI ​​accelerator. It should be understood that this application executes the attention operator in blocks. Therefore, the AI ​​accelerator will continuously update the second normalization factor as the calculation progresses in each block of the attention operator to ensure the correctness of the calculation results.

[0100] Schematically, the vector calculation engine updates the second normalization factor in the second memory based on the sum of each row in the attention weight matrix, the updated first normalization factor, and the second normalization factor stored in the second memory; the vector calculation engine updates the intermediate output matrix block corresponding to the query matrix block based on the sub-output matrix block; if the query matrix block is the last block of the query matrix, the vector calculation engine performs a second normalization operation on the intermediate output matrix block based on the updated second normalization factor to obtain the output matrix block corresponding to the query matrix block. Based on the above formulas (3) to (9), this process refers to the following formulas (10) to (15):

[0101] The above formulas (10) to (13) are also used to update the second normalization factor l i The process. Among them, formula (10) is used to calculate the query matrix block Q i The corresponding attention weight matrix The sum of each row in l ij It should be noted that this application does not limit the execution time of formula (10). For example, the vector calculation engine can also calculate the attention weight matrix in the aforementioned step 402. After that, execute formula (10). Formula (11) is used to calculate the first normalization factor m′ of the historical storage in the second memory. i With the latest first normalization factor The difference between It should be noted that this process can also be performed before executing formula (6) in the aforementioned step 402, and this application does not limit this. Formula (12) is used to Perform exponential operation. Formula (13) is used to calculate the second normalization factor l′ based on the second memory. i and the attention weight matrix The sum of each row in l ij , update the second normalization factor l i . O i =O i / l i (15)

[0102] The above formula (14) is based on the sub-output matrix block O ij Update the query matrix block Q i The corresponding intermediate output matrix block O i The above formula (15) is also the query matrix block Q i To query the last block of the matrix Q, based on the updated second normalization factor li, the intermediate output matrix block O i Perform the second normalization operation to obtain the query matrix block Q i The corresponding output matrix block O i process.

[0103] 404. The memory access engine writes the output matrix block of the attention operator into the first memory.

[0104] In an embodiment of the present application, after obtaining the output matrix block corresponding to any query matrix block based on the aforementioned steps 402 and 403, it is written into the first memory. After the output matrix blocks corresponding to all query matrix blocks of the query matrix are written into the first memory, the output matrix block of the attention operator is obtained. These output matrix blocks are combined to obtain the output matrix of the attention operator.

[0105] The data processing method shown in steps 401 to 404 is described below with reference to the following pseudo code, taking the first memory as HBM and the second memory as SRAM as an example.

[0106] {

[0107] Requirement: Matrix in HBM (i.e., query matrix Q, key matrix K, and value matrix V stored in the first memory HBM), SRAM of size M (i.e., storage capacity M of the second memory SRAM);

[0108] 1. Set block sizes Set the size of the output matrix block;

[0109] 2. Initialize in HBM, in SRAM, Initialize the output matrix O to 0 in the first memory HBM, which will serve as an accumulator to continuously update the output matrix. Initialize the first normalization factor m and the second normalization factor l in the second memory SRAM.

[0110] 3.Divide Q, O into blocks Q1,…,Q Tr and O1,…,O Tr of size B r ×d each, and divide K, V into blocks K1,…,K Tc and V1,…,V Tc of size B c ×d each; Divide the query matrix Q and output matrix O into T by row r row blocks, each row block is of size [B r ,d], divide the key matrix K and value matrix V into T by row c row blocks, each row block is of size [B c , d];

[0111] 4.for 1≤i≤T r do; outer loop, cross-row loop;

[0112] 5.Load Q i from HBM to SRAM; the memory access engine converts the query matrix block Q in the first memory HBM i Read into the second memory SRAM;

[0113] 6.for 1≤j≤T c do; inner loop, cross-column loop;

[0114] 7.Load K j From HBM to SRAM, compute The memory access engine writes the key matrix block K in the first memory HBM i Read into the second memory SRAM, the matrix calculation engine performs matrix multiplication operation to calculate the query matrix block Q i The corresponding attention weight matrix S ij ;

[0115] 8. Compute The vector calculation engine calculates the attention score matrix S ij The maximum value of each row in

[0116] 9. Compute The vector calculation engine calculates the stored first normalization factor m′ i and The maximum value between the two is used as the latest first normalization factor Calculate the stored first normalization factor m′ i With the latest first normalization factor The difference between Update the first normalization factor m i ;

[0117] 10. Compute The vector calculation engine calculates the attention score matrix S ij Each item in the first normalization factor m of the corresponding row i The difference between them, that is, point by point calculation;

[0118] 11.Compute The vector calculation engine updates the attention score matrix Perform exponential operations;

[0119] 12.Load Vj from HBM to SRAM; the memory access engine transfers the value matrix block V in the first memory HBM j Read into the second memory SRAM;

[0120] 13.Compute The matrix calculation engine performs matrix multiplication operations to calculate the query matrix block Q i The corresponding sub-output matrix block O ij ;

[0121] 14.Compute Vector calculation engine Perform exponential operations;

[0122] 15.Compute The vector calculation engine updates the second normalization factor l i ;

[0123] 16.Compute The vector calculation engine updates the query matrix block Q i The corresponding intermediate output matrix block O i ;

[0124] 17.end for; the inner loop ends;

[0125] 18.compute O i =O i / l i ; If the query matrix block Q i To query the last block of the matrix Q, the vector calculation engine performs the calculation on the intermediate output matrix block O i Perform the second normalization operation to obtain the query matrix block Q i The corresponding output matrix block O i ;

[0126] 19.write O i to HBM; the memory access engine will output the matrix block O i Write into the first memory HBM;

[0127] 20.end for; the outer loop ends;

[0128] 21.Return O; Return output matrix O;

[0129] }

[0130] Refer to Figure 5, which is an execution flow chart of an attention operator provided in an embodiment of the present application. As shown in Figure 5, combined with the above pseudo code, it can be seen that the data processing method of the attention operator provided in the present application includes 5 steps: matrix multiplication, maximum value update, subtraction and summation calculation, matrix multiplication, output and summation update. Moreover, all intermediate variables involved in the calculation process do not need to be written to HBM during the inner loop, which reduces the interaction with HBM. In addition, by first updating the maximum value, then calculating the attention weight matrix P and the second normalization factor l, and then calculating the division after the inner loop ends, the intermediate calculation steps can be reduced, the calculation efficiency can be improved, and the operating performance of the AI ​​accelerator can be improved.

[0131] Furthermore, since the data processing method provided by the present application assigns different types of operations involved in the attention operator to different computing engines for execution, multiple computing engines can call the attention operator in parallel to execute the corresponding steps. Referring to Figure 6, Figure 6 is an execution flow chart of another attention operator provided in an embodiment of the present application. As shown in Figure 6, the matrix calculation engine executes the matrix multiplication operation involved in the above data processing method, and the vector calculation engine executes the calculation of the maximum value, addition, subtraction, division and exponential operation, etc. involved in the above data processing method. Pipelining the matrix calculation engine and the vector calculation engine can achieve the effect of parallel computing. Referring to Figure 7, Figure 7 is a schematic diagram of the pipeline arrangement of a data processing method provided in an embodiment of the present application. As shown in Figure 7, combined with the above pseudo code, in the inner loop 0, after the matrix calculation engine executes step 7, the vector calculation engine executes steps 8 to 11. During the process of the vector calculation engine executing steps 8 to 11, if the vector calculation engine calculates the attention score matrix, the matrix calculation engine can execute step 13 in parallel to calculate the query matrix block Q i The corresponding sub-output matrix block O ij , and the matrix calculation engine calculates the sub-output matrix block O ij After that, the next inner loop 1 is entered to perform the matrix multiplication operation. During this process, the vector calculation engine executes steps 14 to 16 in the inner loop 0 in parallel. In this way, the data processing method provided by this application can save the running time of the attention operator and improve the running performance of the AI ​​accelerator.

[0132] Figure 8 is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. The data processing device can implement some or all of the functions of the aforementioned AI accelerator through software, hardware, or a combination of both. The AI ​​accelerator includes a memory access engine and a computing engine. The AI ​​accelerator is configured with a first memory and a second memory, and the read and write speed of the second memory is greater than the read and write speed of the first memory. As shown in Figure 8, the device includes: a memory access module 801 and a computing module 802.

[0133] A memory access module 801 is configured in the memory access engine and is used to read the input matrix block of the attention operator in the first memory into the second memory, where the input matrix block is a block of the input matrix of the attention operator;

[0134] The computing module 802 is configured in the computing engine and is used to call the attention operator, process the input matrix block, and obtain the output matrix block of the attention operator, wherein, in the process of processing the input matrix block, the normalization factor of the normalization operation of the attention operator in the second memory is updated.

[0135] In some embodiments, the memory access engine is a direct memory access (DMA) engine, the first memory is a high-bandwidth memory (HBM) or a double-data rate (DDR), the second memory is a static random access memory (SRAM), and the storage capacity of the first memory is greater than the storage capacity of the second memory.

[0136] In some embodiments, the input matrix block includes: a query matrix block, a key matrix block, and a value matrix block, the query matrix block is a block of the query matrix of the attention operator, the key matrix block is a block of the key matrix of the attention operator, and the value matrix block is a block of the value matrix of the attention operator;

[0137] The calculation module 802 includes:

[0138] a first updating unit, configured to update a first normalization factor of the first normalization operation in the second memory based on the query matrix block and the key matrix block, and calculate an attention weight matrix corresponding to the query matrix block based on the updated first normalization factor and the first normalization operation;

[0139] The second updating unit is used to update the second normalization factor of the second normalization operation in the second memory based on the attention weight matrix and the value matrix block, and calculate the output matrix block corresponding to the query matrix block based on the updated second normalization factor and the second normalization operation.

[0140] In some embodiments, the computing engine includes a matrix computing engine and a vector computing engine, wherein the matrix computing engine and the vector computing engine execute in parallel the process of calling the attention operator to process the input matrix block.

[0141] In some embodiments, the first updating unit is configured to:

[0142] Perform matrix multiplication on the transposed matrices of the query matrix block and the key matrix block to obtain the attention score matrix corresponding to the query matrix block;

[0143] Based on the attention score matrix, update the first normalization factor in the second memory, and based on the updated first normalization factor and the first normalization operation, calculate the attention weight matrix corresponding to the query matrix block, where the first normalization factor indicates the maximum value of each row in the global attention score matrix.

[0144] In some embodiments, the first updating unit is configured to:

[0145] Based on the maximum value of each row in the attention score matrix and the first normalization factor stored in the second memory, updating the first normalization factor stored in the second memory;

[0146] Based on the updated first normalization factor, the first normalization operation is performed on the attention score matrix to obtain the attention weight matrix corresponding to the query matrix block.

[0147] In some embodiments, the second updating unit is configured to:

[0148] Perform matrix multiplication on the attention weight matrix and the value matrix block to obtain the sub-output matrix block corresponding to the query matrix block;

[0149] Based on the attention weight matrix, update the second normalization factor in the second memory, and based on the updated second normalization factor, the sub-output matrix block and the second normalization operation, calculate the output matrix block corresponding to the query matrix block, where the second normalization factor indicates the sum of each row in the global attention weight matrix.

[0150] In some embodiments, the second updating unit is configured to:

[0151] updating the second normalization factor in the second memory based on the sum of each row in the attention weight matrix, the updated first normalization factor, and the second normalization factor stored in the second memory;

[0152] Based on the sub-output matrix block, update the intermediate output matrix block corresponding to the query matrix block;

[0153] If the query matrix block is the last block of the query matrix, the vector calculation engine performs a second normalization operation on the intermediate output matrix block based on the updated second normalization factor to obtain an output matrix block corresponding to the query matrix block.

[0154] Through the above device, since the read and write speed of the second memory is greater than the read and write speed of the first memory, and the normalization factor of the normalization operation in the attention operator is updated in the second memory, that is, the intermediate data generated during the execution of the attention operator does not need to be written into the first memory with a slower read and write speed. Therefore, the above method reduces the access to the first memory and effectively improves the operating performance of the AI ​​accelerator.

[0155] It should be noted that the data processing device provided in the above embodiment is merely illustrated by the division of the above functional modules when performing data processing. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the data processing device provided in the above embodiment and the data processing method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0156] In this application, the terms "first", "second", etc. are used to distinguish between identical or similar items having substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there a limitation on quantity and execution order. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various described examples, a first memory may be referred to as a second memory, and similarly, a second memory may be referred to as a first memory. The first memory and the second memory may both be memories, and in some cases, may be separate and different memories.

[0157] In this application, the term "at least one" means one or more, and the term "plurality" means two or more. For example, a plurality of memories means two or more memories.

[0158] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

[0159] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of program structure information. The program structure information includes one or more program instructions. When the program instructions are loaded and executed on a computing device, all or part of the processes or functions described in the embodiments of the present application are generated.

[0160] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0161] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: Applied to an artificial intelligence (AI) accelerator, the AI ​​accelerator includes a memory access engine and a computing engine, the AI ​​accelerator is configured with a first memory and a second memory, the read and write speed of the second memory is greater than the read and write speed of the first memory, the method includes: The memory access engine reads an input matrix block of the attention operator in the first memory into the second memory, wherein the input matrix block is a block of the input matrix of the attention operator; The computing engine calls the attention operator and processes the input matrix block to obtain an output matrix block of the attention operator, wherein in the process of processing the input matrix block, a normalization factor of a normalization operation of the attention operator in the second memory is updated; The memory access engine writes the output matrix block of the attention operator into the first memory.

2. The method according to claim 1, characterized in that The memory access engine is a direct memory access DMA engine, the first memory is a high bandwidth memory HBM or a double data rate memory DDR, the second memory is a static random access memory SRAM, and the storage capacity of the first memory is greater than the storage capacity of the second memory.

3. The method according to claim 1 or 2, characterized in that: The input matrix block includes: a query matrix block, a key matrix block and a value matrix block, wherein the query matrix block is a block of the query matrix of the attention operator, the key matrix block is a block of the key matrix of the attention operator, and the value matrix block is a block of the value matrix of the attention operator; The computing engine calls the attention operator, processes the input matrix block, and obtains the output matrix block of the attention operator, including: The computing engine updates a first normalization factor of a first normalization operation in the second memory based on the query matrix block and the key matrix block, and calculates an attention weight matrix corresponding to the query matrix block based on the updated first normalization factor and the first normalization operation; The computing engine updates the second normalization factor of the second normalization operation in the second memory based on the attention weight matrix and the value matrix block, and calculates the output matrix block corresponding to the query matrix block based on the updated second normalization factor and the second normalization operation.

4. The method according to claim 3, characterized in that The computing engine includes a matrix computing engine and a vector computing engine, wherein the matrix computing engine and the vector computing engine execute in parallel a process of calling the attention operator to process the input matrix block.

5. The method according to claim 4, characterized in that The computing engine updates a first normalization factor of a first normalization operation in the second memory based on the query matrix block and the key matrix block, and calculates an attention weight matrix corresponding to the query matrix block based on the updated first normalization factor and the first normalization operation, including: The matrix calculation engine performs a matrix multiplication operation on the query matrix block and the transposed matrix of the key matrix block to obtain an attention score matrix corresponding to the query matrix block; The vector calculation engine updates the first normalization factor in the second memory based on the attention score matrix, and calculates the attention weight matrix corresponding to the query matrix block based on the updated first normalization factor and the first normalization operation, wherein the first normalization factor indicates the maximum value of each row in the global attention score matrix.

6. The method according to claim 5, characterized in that The vector calculation engine updates the first normalization factor in the second memory based on the attention score matrix, and calculates the attention weight matrix corresponding to the query matrix block based on the updated first normalization factor and the first normalization operation, including: The vector calculation engine updates the first normalization factor stored in the second memory based on the maximum value of each row in the attention score matrix and the first normalization factor stored in the second memory; The vector calculation engine performs the first normalization operation on the attention score matrix based on the updated first normalization factor, Get the attention weight matrix corresponding to the query matrix block.

7. The method according to any one of claims 4 to 6, characterized in that The computing engine updates the second normalization factor of the second normalization operation in the second memory based on the attention weight matrix and the value matrix block, and calculates the output matrix block corresponding to the query matrix block based on the updated second normalization factor and the second normalization operation, including: The matrix calculation engine performs a matrix multiplication operation on the attention weight matrix and the value matrix block to obtain a sub-output matrix block corresponding to the query matrix block; The vector calculation engine updates the second normalization factor in the second memory based on the attention weight matrix, and calculates the output matrix block corresponding to the query matrix block based on the updated second normalization factor, the sub-output matrix block and the second normalization operation, wherein the second normalization factor indicates the sum of each row in the global attention weight matrix.

8. The method according to claim 7, characterized in that The vector calculation engine updates the second normalization factor stored in the second memory based on the attention weight matrix, and calculates the output matrix block corresponding to the query matrix block based on the updated second normalization factor, the sub-output matrix block and the second normalization operation, including: The vector calculation engine updates the second normalization factor in the second memory based on the sum of each row in the attention weight matrix, the updated first normalization factor, and the second normalization factor stored in the second memory; The vector calculation engine updates the intermediate output matrix block corresponding to the query matrix block based on the sub-output matrix block; If the query matrix block is the last block of the query matrix, the vector calculation engine performs the second normalization operation on the intermediate output matrix block based on the updated second normalization factor to obtain the output matrix block corresponding to the query matrix block.

9. A data processing device, characterized in that: The device is applied to an AI accelerator, the AI ​​accelerator includes a memory access engine and a computing engine, the AI ​​accelerator is configured with a first memory and a second memory, the read and write speed of the second memory is greater than the read and write speed of the first memory, and the device includes: A memory access module, configured in the memory access engine, for implementing the steps performed by the memory access engine as described in any one of claims 1 to 8; A computing module, configured in the computing engine, is used to implement the steps performed by the computing engine as described in any one of claims 1 to 8.

10. An accelerator, characterized in that: The accelerator comprises a communication interface, a memory access engine and a computing engine, wherein the communication interface is used to provide program instructions and / or data, and the accelerator is used to implement the data processing method as claimed in any one of claims 1 to 8.

11. A computing device, characterized in that: The computing device host and the AI ​​accelerator, the host is used to send data to the AI ​​accelerator and receive data sent by the AI ​​accelerator, and the AI ​​accelerator is used to execute the data processing method as described in any one of claims 1 to 8.

12. A computing device cluster, characterized in that: The method comprises a plurality of computing devices, wherein the computing devices include a host and an AI accelerator, wherein the host is used to send data to the AI ​​accelerator and receive data sent by the AI ​​accelerator, and wherein the AI ​​accelerator is used to execute the data processing method as claimed in any one of claims 1 to 8.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store at least one program code, and the at least one program code is used to execute the data processing method as claimed in any one of claims 1 to 8.

14. A computer program product, characterized in that When the computer program product runs on an AI accelerator, the AI ​​accelerator is enabled to perform the data processing method according to any one of claims 1 to 8.

Images