Data processing method and device, accelerator and computing equipment

Abstract

The invention discloses a data processing method and device, an accelerator and computing equipment, and relates to the technical field of AI (artificial intelligence). The method comprises the steps that when the AI accelerator executes an attention operator, a memory access engine reads an input matrix block in a first memory into a second memory, a computing engine calls the attention operator and processes the input matrix block, and the AI accelerator executes the attention operator; and after the output matrix block is obtained, the memory access engine writes the output matrix block into the first memory, and in the process, the read-write speed of the second memory is greater than the read-write speed of the first memory, and the normalization factor of the normalization operation in the attention operator is updated in the second memory, namely, the normalization factor of the normalization operation in the attention operator is updated in the second memory. According to the method, the intermediate data generated in the execution process of the attention operator does not need to be written into the first memory with relatively low read-write speed, so that the page view aiming at the first memory is reduced, and the operation performance of the AI accelerator is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a data processing method, device, accelerator and computing equipment. Background Art

[0002] With the rapid development of AI technology, a series of AI accelerators have emerged to provide computing power for matrices, vectors, etc., in order to accelerate the calculation process of AI models. At present, AI accelerators are usually configured with various forms of memory, which have different storage capacity and read and write speeds. Usually, the smaller the storage capacity of the memory, the faster the read and write speed.

[0003] Taking the attention operator in the Transformer model as an example, when the AI ​​accelerator calls the attention operator to process its input matrix, it needs to continuously access the high-bandwidth memory (HBM) configured on the AI ​​accelerator, for example, reading data from the HBM, writing intermediate data generated during the calculation process to the HBM, etc. The high memory access volume leads to poor operating performance of the AI ​​accelerator. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, device, accelerator and computing device, which can effectively improve the operating performance of the AI ​​accelerator. The technical solution is as follows:

[0005] In a first aspect, a data processing method is provided, which is applied to an artificial intelligence (AI) accelerator, wherein the AI ​​accelerator includes a memory access engine and a computing engine, and the AI ​​accelerator is configured with a first memory and a second memory, wherein the read and write speed of the second memory is greater than the read and write speed of the first memory, and the method includes:

[0006] The memory access engine reads an input matrix block of the attention operator in the first memory into the second memory, wherein the input matrix block is a block of the input matrix of the attention operator;

[0007] The computing engine calls the attention operator and processes the input matrix block to obtain an output matrix block of the attention operator, wherein in the process of processing the input matrix block, a normalization factor of a normalization operation of the attention operator in the second memory is updated;

[0008] The memory access engine writes the output matrix block of the attention operator into the first memory.

[0009] In this process, since the read and write speed of the second memory is greater than the read and write speed of the first memory, and the normalization factor of the normalization operation in the attention operator is updated in the second memory, that is, the intermediate data generated during the execution of the attention operator does not need to be written to the first memory with a slower read and write speed. Therefore, the above method reduces the access to the first memory and effectively improves the operating performance of the AI ​​accelerator.

[0010] In some embodiments, the memory access engine is a direct memory access DMA engine, the first memory is high bandwidth memory HBM or double data rate memory DDR, the second memory is static random access memory SRAM, and the storage capacity of the first memory is greater than the storage capacity of the second memory.

[0011] In some embodiments, the input matrix block includes: a query matrix block, a key matrix block and a value matrix block, the query matrix block is a block of the query matrix of the attention operator, the key matrix block is a block of the key matrix of the attention operator, and the value matrix block is a block of the value matrix of the attention operator;

[0012] The computing engine calls the attention operator, processes the input matrix block, and obtains the output matrix block of the attention operator, including:

[0013] The computing engine updates a first normalization factor of a first normalization operation in the second memory based on the query matrix block and the key matrix block, and calculates an attention weight matrix corresponding to the query matrix block based on the updated first normalization factor and the first normalization operation;

[0014] The computing engine updates the second normalization factor of the second normalization operation in the second memory based on the attention weight matrix and the value matrix block, and calculates the output matrix block corresponding to the query matrix block based on the updated second normalization factor and the second normalization operation.

[0015] In some embodiments, the computing engine includes a matrix computing engine and a vector computing engine, wherein the matrix computing engine and the vector computing engine execute in parallel a process of calling the attention operator to process the input matrix block.

[0016] In this way, different types of operations involved in the attention operator are assigned to different computing engines for execution, so that multiple computing engines can call the attention operator in parallel to execute the corresponding steps, thereby saving the running time of the attention operator and improving the running performance of the AI ​​accelerator.

[0017] In some embodiments, the computing engine updates a first normalization factor of a first normalization operation in the second memory based on the query matrix block and the key matrix block, and calculates an attention weight matrix corresponding to the query matrix block based on the updated first normalization factor and the first normalization operation, including:

[0018] The matrix calculation engine performs a matrix multiplication operation on the query matrix block and the transposed matrix of the key matrix block to obtain an attention score matrix corresponding to the query matrix block;

[0019] The vector calculation engine updates the first normalization factor in the second memory based on the attention score matrix, and calculates the attention weight matrix corresponding to the query matrix block based on the updated first normalization factor and the first normalization operation, wherein the first normalization factor indicates the maximum value of each row in the global attention score matrix.

[0020] In some embodiments, the vector calculation engine updates the first normalization factor in the second memory based on the attention score matrix, and calculates the attention weight matrix corresponding to the query matrix block based on the updated first normalization factor and the first normalization operation, including:

[0021] The vector calculation engine updates the first normalization factor stored in the second memory based on the maximum value of each row in the attention score matrix and the first normalization factor stored in the second memory;

[0022] The vector calculation engine performs the first normalization operation on the attention score matrix based on the updated first normalization factor to obtain the attention weight matrix corresponding to the query matrix block.

[0023] In some embodiments, the computing engine updates the second normalization factor of the second normalization operation in the second memory based on the attention weight matrix and the value matrix block, and calculates the output matrix block corresponding to the query matrix block based on the updated second normalization factor and the second normalization operation, including:

[0024] The matrix calculation engine performs a matrix multiplication operation on the attention weight matrix and the value matrix block to obtain a sub-output matrix block corresponding to the query matrix block;

[0025] The vector calculation engine updates the second normalization factor in the second memory based on the attention weight matrix, and calculates the output matrix block corresponding to the query matrix block based on the updated second normalization factor, the sub-output matrix block and the second normalization operation, wherein the second normalization factor indicates the sum of each row in the global attention weight matrix.

[0026] In some embodiments, the vector calculation engine updates the second normalization factor stored in the second memory based on the attention weight matrix, and calculates the output matrix block corresponding to the query matrix block based on the updated second normalization factor, the sub-output matrix block and the second normalization operation, including:

[0027] The vector calculation engine updates the second normalization factor in the second memory based on the sum of each row in the attention weight matrix, the updated first normalization factor, and the second normalization factor stored in the second memory;

[0028] The vector calculation engine updates the intermediate output matrix block corresponding to the query matrix block based on the sub-output matrix block;

[0029] If the query matrix block is the last block of the query matrix, the vector calculation engine performs the second normalization operation on the intermediate output matrix block based on the updated second normalization factor to obtain the output matrix block corresponding to the query matrix block.

[0030] In a second aspect, an embodiment of the present application provides a data processing device, which is applied to an AI accelerator, wherein the AI ​​accelerator includes a memory access engine and a computing engine, and the AI ​​accelerator is configured with a first memory and a second memory, and the read and write speed of the second memory is greater than the read and write speed of the first memory. The device includes: a memory access module, configured in the memory access engine, for implementing the steps executed by the memory access engine as described in the first aspect or any possible implementation of the first aspect; a computing module, configured in the computing engine, for implementing the steps executed by the computing engine as described in the first aspect or any possible implementation of the first aspect.

[0031] In a third aspect, an embodiment of the present application provides an accelerator, comprising a communication interface, a memory access engine, and a computing engine, wherein the communication interface is used to provide program instructions and / or data, and the accelerator is used to implement a data processing method as provided in the first aspect or any possible implementation of the first aspect.

[0032] In a fourth aspect, an embodiment of the present application provides a computing device, comprising a computing device host and an AI accelerator, wherein the host is used to send data to the AI ​​accelerator and receive data sent by the AI ​​accelerator, and the AI ​​accelerator is used to execute a data processing method provided in the first aspect or any possible implementation of the first aspect.

[0033] In a fifth aspect, an embodiment of the present application provides a computing device cluster, comprising multiple computing devices, wherein the computing devices include a host and an AI accelerator, the host is used to send data to the AI ​​accelerator, and receive data sent by the AI ​​accelerator, and the AI ​​accelerator is used to execute the data processing method provided in the first aspect or any possible implementation of the first aspect.

[0034] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store at least one program code, and the at least one program code is used to execute the data processing method provided in the first aspect or any possible implementation of the first aspect. The storage medium includes but is not limited to a volatile memory, such as a random access memory, a non-volatile memory, such as a flash memory, a hard disk drive (HDD), and a solid state drive (SSD).

[0035] In a seventh aspect, an embodiment of the present application provides a computer program product, which, when running on an AI accelerator, enables the AI ​​accelerator to perform a data processing method as provided in the first aspect or any possible implementation of the first aspect. The computer program product may be a software installation package, and when the data processing method needs to be implemented, the computer program product may be downloaded and executed on the AI ​​accelerator. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the calculation process of an attention operator;

[0037] Figure 2 It is a schematic diagram of an implementation environment provided by this application;

[0038] Figure 3 is a schematic diagram of the hardware structure of an accelerator provided in an embodiment of the present application;

[0039] Figure 4 is a flow chart of a data processing method provided in an embodiment of the present application;

[0040] Figure 5 is an execution flow chart of an attention operator provided in an embodiment of the present application;

[0041] Figure 6 is an execution flow chart of another attention operator provided in an embodiment of the present application;

[0042] Figure 7 It is a pipeline arrangement schematic diagram of a data processing method provided in an embodiment of the present application;

[0043] Figure 8 It is a structural diagram of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below in conjunction with the accompanying drawings. It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the input matrix of the attention operator involved in this application is obtained with full authorization.

[0045] For ease of understanding, the key terms and key concepts involved in this application are explained below.

[0046] Artificial intelligence (AI) model is a type of mathematical algorithm model that uses machine learning ideas to solve practical problems. The AI ​​model includes a large number of parameters and calculation formulas (or calculation rules).

[0047] AI accelerators are a type of specialized hardware accelerator or computer system designed to accelerate AI applications, especially neural networks, machine vision, and machine learning. For example, they are used to provide computing power for calculating matrices, vectors, etc. to accelerate the calculation of AI models. Schematically, AI accelerators are, for example, graphics processing units (GPUs), intelligent processing units (IPUs), tensor processing units (TPUs), domain specific architecture (DSA) chips, etc., but the present application is not limited thereto.

[0048] Deep learning (DL) is a branch of machine learning (ML). Deep learning is the study of the inherent laws and representation levels of sample data. The information obtained in the learning process is very helpful for interpreting data such as text, images, and sounds. Schematically, deep learning is a complex machine learning algorithm, and the typical AI model that uses deep learning ideas is the neural network model.

[0049] An operator (OP) refers to a computing unit or computing function running on a computing device. In the field of deep learning, the neural network layer and even the entire model are composed of operators, which correspond to the computing logic in the neural network layer. For example, the convolution layer is an operator; the weight summation process in the fully-connected layer (FClayer) is an operator.

[0050] The attention operator is the core operator of the self-attention layer in the Transformer model. The core idea of ​​the self-attention layer is to make the machine notice the correlation between different parts of the entire input. Schematically, for any vector in the input sequence of the self-attention layer (such as words, letters, image features, etc.), the vector is linearly transformed to obtain the query (query, Q) matrix, key (key, K) matrix and value (value, V) matrix corresponding to the vector, that is, the input matrices Q, K, and V of the attention operator are obtained. By executing the attention operator, the input matrix is ​​processed to obtain the corresponding output (output, O) matrix, that is, the correlation between different parts of the input sequence is obtained. This process refers to Figure 1 , Figure 1 It is a schematic diagram of the calculation process of the attention operator. The calculation process refers to the following formula (1):

[0051]

[0052] In the above formula (1), the matrices Q, K, and V are the input matrices of the attention operator, with a size of N×d, the matrix S is the attention score matrix, the matrix P is the attention weight matrix, and the matrix O is the output matrix, d k is the dimension of the key vector k, and N and d are both positive integers. It should be understood that the Softmax normalization operation is performed row-wise, that is, a Softmax normalization operation is performed once for each row, and a total of N rows are performed.

[0053] In order to avoid overflow and other problems in the calculation process of Softmax normalization operation, in actual computer calculation, both the numerator and the denominator are usually divided by e max(X) , the actual calculation of the Softmax normalization operation formula is shown in the following formula (2):

[0054]

[0055] It can be seen that the calculation of the attention operator requires additional storage space to save the intermediate matrices S and P. Each step of the calculation requires reading and writing data, and for each row of matrix data, the calculation of the Softmax normalization operation needs to be traversed three times in total to get the final result (the first time to find the maximum value max(X) in the row, the second time to calculate The third traversal calculates the results in sequence). That is, in the calculation process of the attention operator, there are multiple read and write operations on the intermediate variables, and the overall calculation efficiency is low.

[0056] The following is an introduction to the application scenarios and implementation environment involved in this application.

[0057] The technical solution provided in the embodiment of the present application can be applied to the process of executing the attention operator based on the AI ​​accelerator. Figure 1 As can be seen from the calculation process of the attention operator shown, intermediate data will be generated during the execution of the attention operator: matrix S and matrix P, as well as the maximum value within the row, etc. Therefore, in the process of executing the attention operator, the AI ​​accelerator needs to continuously access the memory of the AI ​​accelerator to read and write matrix S and matrix P, resulting in excessive memory access and poor performance of the AI ​​accelerator. In the related art, another implementation method of the attention operator is provided, the FlashAttention algorithm, which aims to transform the Softmax normalization operation that originally requires global data into an implementation method through "local data calculation + rolling update". Schematically, the FlashAttention algorithm divides the input matrix of the attention operator into blocks to obtain input matrix blocks, and processes the input matrix blocks to reduce the access to the memory HBM and improve the performance of the AI ​​accelerator. However, this method still requires the intermediate data generated based on the input matrix blocks during the execution of the attention operator to be written into the memory HBM, resulting in a high access to the memory HBM.

[0058] Based on this, the present application provides a data processing method applied to the attention operator, which can reduce memory access and improve the operating performance of the AI ​​accelerator.

[0059] Reference below Figure 2 , introduce the implementation environment of this application. Figure 2 is a schematic diagram of an implementation environment provided by this application. Figure 2 As shown, the implementation environment includes a host 201 and an AI accelerator 202, and the host 201 and the AI ​​accelerator 202 are communicatively connected.

[0060] The host 201 refers to a device for running an AI model, which can provide AI services to users. In the embodiment of the present application, the host 201 can control the AI ​​accelerator 202 to call the attention operator and implement the function of the attention operator. This process can also be understood as loading the AI ​​task into the AI ​​accelerator 202 for operation. In addition, the number of hosts 201 can be one or more, and this application does not limit this.

[0061] The AI ​​accelerator 202 is used to provide computing power for the running AI model to accelerate the computing process of the AI ​​model, that is, to execute the data processing method provided by this application. For example, the AI ​​accelerator 202 is a DSA chip, a GPU, etc., but this application is not limited thereto. Schematically, the AI ​​accelerator 202 calls the attention operator according to the input matrix block of the attention operator sent by the host 201, processes the input matrix block, obtains the output matrix block of the attention operator, and returns the output matrix block to the host 201 to implement the function of the attention operator. The architecture of the AI ​​accelerator 202 is as follows: Figure 3 The content shown is introduced in detail, and will not be repeated here. In addition, the number of AI accelerators 202 can be one or more, and this application does not limit this.

[0062] Schematically, the host 201 and the AI ​​accelerator 202 are integrated in a computing device, and the host 201 and the AI ​​accelerator 202 are connected to each other through a peripheral component interconnect express (PCIe) link. The host 201 exchanges data with the AI ​​accelerator 202 through the PCIe link and controls the AI ​​accelerator 202 to call the attention operator. For example, the computing device can be an independent physical server, or a server cluster or distributed file system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0063] Taking the computing device as a cloud server as an example, the computing device can also be called a cloud platform (short for cloud computing platform), which refers to services based on hardware resources and software resources, providing computing, network and storage capabilities. Through the network "cloud", huge data computing is processed and analyzed remotely and then returned to the user. It has the characteristics of large-scale, distributed, virtualized, high availability, scalability, on-demand service and security. The cloud platform can realize the rapid issuance and release of configurable computing resources with a small management cost or low interaction complexity between users and service providers.

[0064] It should be noted that in the above Figure 2 In the illustrated implementation environment, the execution function of the attention operator is implemented through interaction between the host 201 and the AI ​​accelerator 202 as an example. In other embodiments, the AI ​​accelerator 202 has the function of running the AI ​​model, that is, the AI ​​accelerator 202 can directly run the AI ​​model specified by the user to implement the function of the attention operator. This application does not limit this.

[0065] In addition, the above-mentioned networks include, but are not limited to, data center networks, storage area networks (SAN), local area networks (LAN), metropolitan area networks (MAN), wide area networks (WAN), mobile, wired or wireless networks, dedicated networks or any combination of virtual private networks. In some implementations, technologies and / or formats including hypertext markup language (HTML), extensible markup language (XML) are used to represent data exchanged through the network. In addition, conventional encryption technologies such as secure sockets layer (SSL), transport layer security (TLS), virtual private network (VPN), and Internet protocol security (IPsec) can also be used to encrypt all or part of the links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0066] The structure of the AI ​​accelerator 202 is introduced below.

[0067] The embodiment of the present application provides an accelerator, or an acceleration chip, which can be configured as the AI ​​accelerator 202 shown in the above implementation environment. Figure 3 , Figure 3 Schematic diagram of the hardware structure of an accelerator provided in an embodiment of the present application. Figure 3As shown, the accelerator 300 includes a communication interface 301, a memory access engine 302, a computing engine 303, a memory 304 and a bus 305. The communication interface 301, the memory access engine 302, the computing engine 303 and the memory 304 are connected to each other through the bus 305.

[0068] The communication interface 301 is used to provide program instructions and / or data, and the communication interface 301 includes a PCIe communication interface, other common peripheral interfaces, etc., which are not limited in this application. For example, when the accelerator 300 is used as an acceleration card of the host 201, data exchange is achieved through the PCIe communication interface and the host 201. For another example, the accelerator 300 realizes communication between the accelerator 300 and other devices or communication networks through the peripheral interface.

[0069] The memory access engine 302 is used to read the input matrix block of the attention operator in the first memory of the AI ​​accelerator into the second memory of the AI ​​accelerator, where the input matrix block is a block of the input matrix of the attention operator, and the read and write speed of the second memory is greater than the read and write speed of the first memory; the memory access engine 302 is also used to write the output matrix block of the attention operator into the first memory. In some embodiments, the memory access engine 302 is a direct memory access engine (DMA engine), but the present application is not limited thereto.

[0070] The computing engine 303 is used to call the attention operator, process the input matrix block, and obtain the output matrix block of the attention operator, wherein, in the process of processing the input matrix block, the normalization factor of the normalization operation of the attention operator in the second memory is updated. In some embodiments, the computing engine 303 includes a matrix computing engine (cube engine), a vector computing engine (vector engine) and a scalar computing engine (scalar engine), wherein the matrix computing engine is used to perform matrix multiplication operations in the attention operator, the vector computing engine is used to perform vector operations in the attention operator, and the scalar computing engine is used to perform scalar operation operations in the attention operator. In some embodiments, the aforementioned computing engines can be executed in parallel to improve the accelerator operation performance.

[0071] The memory 304 includes a high bandwidth memory (HBM), a double data rate memory (DDR), a static random access memory (SRAM), or other types of dynamic storage devices that can store information and instructions, or also includes any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. Schematically, the first memory of the AI ​​accelerator is HBM and / or DDR, and the second memory of the AI ​​accelerator is SRAM, which is also referred to as shared memory in the embodiment of the present application. It should be understood that this is only an example, and the present application does not limit the implementation of the first memory and the second memory on the AI ​​accelerator. For example, taking the first memory as HBM as an example, the memory access engine 302 can read the data stored in the HBM into the SRAM, and write the data stored in the SRAM into the HBM. It should be understood that the HBM has a large capacity but a slow read and write speed, and the SRAM has a small capacity but a high read and write speed. Therefore, by reducing the amount of access to the HBM, the operating performance of the AI ​​accelerator can be greatly improved. For example, the A100 GPU has 40-80GB of HBM with a bandwidth of 1.5-2.0TB / s; each of the 108 streaming multicore processors has 192KB of on-chip SRAM, with an estimated bandwidth of about 19TB / s. It can be seen that on-chip SRAM is an order of magnitude faster than HBM, but is many orders of magnitude smaller in size.

[0072] The bus 305 may include a path for transmitting information between various components of the accelerator 300 (eg, the communication interface 301 , the memory access engine 302 , the computing engine 303 , and the memory 304 ).

[0073] It should be noted that the above Figure 3 What is shown is only a hardware structure diagram of an accelerator provided by the present application that can be configured as the above-mentioned AI accelerator 202. In some embodiments, the accelerator 300 may also include other components to achieve more functions. For example, the accelerator 300 also includes a task scheduler (TS), a controller, etc., but the present application is not limited to this.

[0074] The following is an introduction to the data processing method applied to the attention operator provided in this application.

[0075] Based on the foregoing, it can be seen that for an AI accelerator, in the process of executing the attention operator, the less access is made to the first memory with a slower read and write speed, the better the operating performance of the AI ​​accelerator. Based on this, the present application provides a data processing method, in which a memory access engine reads an input matrix block in a first memory into a second memory, a computing engine calls an attention operator, processes the input matrix block, and after obtaining an output matrix block, the memory access engine writes the output matrix block into the first memory. In this process, since the read and write speed of the second memory is greater than the read and write speed of the first memory, and the normalization factor of the normalization operation in the attention operator is updated in the second memory, that is, the intermediate data generated during the execution of the attention operator does not need to be written to the first memory with a slower read and write speed. Therefore, the above method reduces the access to the first memory and effectively improves the operating performance of the AI ​​accelerator.

[0076] Reference below Figure 4 , the data processing method provided by this application is introduced. Figure 4 : is a flow chart of a data processing method provided in an embodiment of the present application. Figure 4 As shown, the method is applied to an AI accelerator, the AI ​​accelerator includes a memory access engine and a computing engine, the AI ​​accelerator is configured with a first memory and a second memory, the read and write speed of the second memory is greater than the read and write speed of the first memory, and the method includes the following steps 401 to 404.

[0077] 401. The memory access engine reads the input matrix block of the attention operator in the first memory into the second memory, where the input matrix block is a block of the input matrix of the attention operator.

[0078] In the embodiment of the present application, based on the above introduction to the attention operator, it can be known that the input matrix of the attention operator is obtained by linearly transforming any vector in the input sequence, including the query matrix Q, the key matrix K and the value matrix V. The present application does not limit the type of the input sequence, for example, the input sequence is a word, a letter, an image feature, etc. Correspondingly, the input matrix block of the attention operator includes: a query matrix block, a key matrix block and a value matrix block, wherein the query matrix block is a block of the query matrix of the attention operator, the key matrix block is a block of the key matrix of the attention operator, and the value matrix block is a block of the value matrix of the attention operator.

[0079] It should be understood that since the present application divides the input matrix of the attention operator, that is, the attention operator is executed in blocks, the output matrix of the attention operator is also obtained in blocks. Therefore, in this step, the input matrix block of the attention operator is used to calculate the output matrix block. For example, the query matrix Q is divided into T by row. r A query matrix block (T ris a positive integer, r is the abbreviation of row), the key matrix K and the value matrix V are divided into T by row respectively. c Take a matrix block as an example (T c is a positive integer, c is the abbreviation of column column), in this step, the input matrix block includes the query matrix block Q i (i is a positive integer, 1≤i≤T r ), key matrix K j Sum value matrix V j (j is a positive integer, 1≤j≤T c ), accordingly, the output matrix block is O i It should be noted that the present application does not limit the timing of the memory access engine reading each input matrix block. For example, the memory access engine may read the output matrix block O in this step. i All the input matrix blocks involved are read into the second memory, or the query matrix block Q can be read first i Sum matrix block K j , the value matrix V is involved in the subsequent calculation process j In the case of j The data can be read into the second memory, and the input matrix blocks can be dynamically read into the second memory according to the size of the data stored in the second memory to improve the operating performance of the AI ​​accelerator, etc.

[0080] In addition, the present application does not limit the size of the input matrix block, which can be set as needed. In some embodiments, the AI ​​accelerator divides the input matrix based on the storage capacity of the second memory and the size of the input matrix, obtains the input matrix block of the attention operator, and stores it in the first memory. The present application does not limit the implementation method adopted for matrix block division. For example, taking the size of the input matrix as N×d and the storage capacity of the second memory as M as an example, the size of the input matrix block is set to B is the abbreviation of block. Next, the query matrix Q and the output matrix O are divided into T by row. r row blocks, each row block is of size [B r , d], divide the key matrix K and value matrix V into T by row c row blocks, each row block is of size [B c , d]. In this way, the attention operator is executed in blocks, which provides technical support for the subsequent execution of the attention operator based on the second memory. It should be understood that although the read and write speed of the second memory is greater than the read and write speed of the first memory, its storage capacity is often smaller than the storage capacity of the first memory. Therefore, by dividing the input matrix of the attention operator, the storage capacity upper limit of the second memory can be met, thereby optimizing the calculation process of the attention operator and improving the operating performance of the AI ​​accelerator.

[0081] 402. The computing engine updates the first normalization factor of the first normalization operation in the second memory based on the query matrix block and the key matrix block in the input matrix block, and calculates the attention weight matrix corresponding to the query matrix block based on the updated first normalization factor and the first normalization operation.

[0082] In an embodiment of the present application, the normalization operation in the attention operator is split into a first normalization operation and a second normalization operation, corresponding to a first normalization factor and a second normalization factor, respectively, wherein the first normalization factor indicates the maximum value of each row in the global attention score matrix, and the second normalization factor indicates the sum of each row in the global attention weight matrix. It should be understood that these two normalization factors are global variables involved in the normalization operation in the attention operator, and involve continuous updates during the calculation process. Therefore, in the present application, the AI ​​accelerator initializes the normalization factor in the second memory and continuously updates the normalization factor in the second memory, thereby reducing the access to the first memory and improving the operating performance of the AI ​​accelerator.

[0083] The following introduces the process in which the computing engine in this step performs the first normalization operation based on the query matrix block and the key matrix block to obtain the attention weight matrix corresponding to the query matrix block.

[0084] Schematically, the computing engine includes a matrix computing engine and a vector computing engine, wherein the matrix computing engine and the vector computing engine execute in parallel a process of calling the attention operator to process the input matrix block, including the following steps A1 and A2:

[0085] Step A1: The matrix calculation engine performs a matrix multiplication operation on the transposed matrices of the query matrix block and the key matrix block to obtain an attention score matrix corresponding to the query matrix block.

[0086] Among them, the query matrix block Q i and key matrix block K j For example, this step refers to the following formula (3):

[0087]

[0088] In the above formula (3), S ij The query matrix block Q i The corresponding attention score matrix, Denotes the key matrix block K j The transposed matrix, B r and B c are the row and column lengths of the matrix block, respectively.

[0089] Step A2: The vector calculation engine updates the first normalization factor in the second memory based on the attention score matrix, and calculates the attention weight matrix corresponding to the query matrix block based on the updated first normalization factor and the first normalization operation.

[0090] Among them, the first normalization factor indicates the maximum value of each row in the global attention score matrix, that is, a global variable, which is stored in the second memory, and can speed up the reading and writing speed of the computing engine, thereby improving the operating performance of the AI ​​accelerator. It should be understood that this application executes the attention operator in blocks, so the AI ​​accelerator will continuously update the first normalization factor to ensure the correctness of the calculation results as the calculation progresses of each block in the attention operator.

[0091] Schematically, the vector calculation engine updates the first normalization factor stored in the second memory based on the maximum value of each row in the attention score matrix and the first normalization factor stored in the second memory; based on the updated first normalization factor, the attention score matrix is ​​subjected to the first normalization operation to obtain the attention weight matrix corresponding to the query matrix block. Based on the aforementioned formula (3), this process refers to the following formulas (4) to (8):

[0092]

[0093]

[0094]

[0095] The above formulas (4) to (6) are also used to update the first normalization factor m i The process. Formula (4) is used to calculate the query matrix block Q i The corresponding attention score matrix S ij The maximum value of each row in Formula (5) is used to determine the first normalization factor m′ stored in the second memory. i and The maximum value between Next, based on the latest first normalization factor, Update the first normalization factor m i .

[0096]

[0097]

[0098] The above formulas (7) and (8) are based on the updated first normalization factor m i , for the attention score matrix S ijThe process of performing the first normalization operation. Wherein, formula (7) is used to calculate the attention score matrix S ij Each item in the first normalization factor m of the corresponding row i The difference between them, that is, point by point calculation, obtains the updated attention score matrix Formula (8) is used to update the attention score matrix Perform exponential operation to obtain the query matrix block Q i The corresponding attention weight matrix

[0099] 403. The computing engine updates the second normalization factor of the second normalization operation in the second memory based on the attention weight matrix and the value matrix block, and calculates the output matrix block corresponding to the query matrix block based on the updated second normalization factor and the second normalization operation.

[0100] In the embodiment of the present application, taking the calculation engine including a matrix calculation engine and a vector calculation engine as an example, this step includes the following steps B1 and B2:

[0101] Step B1: The matrix calculation engine performs a matrix multiplication operation on the attention weight matrix and the value matrix block to obtain a sub-output matrix block corresponding to the query matrix block.

[0102] Wherein, based on the above formulas (3) to (8), this step refers to the following formula (9):

[0103]

[0104] In the above formula (9), O ij The query matrix block Q i The corresponding sub-output matrix block, V j is a block of value matrices.

[0105] Step B2: The vector calculation engine updates the second normalization factor in the second memory based on the attention weight matrix, and calculates the output matrix block corresponding to the query matrix block based on the updated second normalization factor, the sub-output matrix block and the second normalization operation.

[0106] Among them, the second normalization factor indicates the sum of each row in the global attention weight matrix, that is, a global variable, which is stored in the second memory, and can speed up the reading and writing speed of the computing engine, thereby improving the operating performance of the AI ​​accelerator. It should be understood that this application executes the attention operator in blocks, so the AI ​​accelerator will continuously update the second normalization factor to ensure the correctness of the calculation results as the calculation progresses of each block in the attention operator.

[0107] Schematically, the vector calculation engine updates the second normalization factor in the second memory based on the sum of each row in the attention weight matrix, the updated first normalization factor, and the second normalization factor stored in the second memory; the vector calculation engine updates the intermediate output matrix block corresponding to the query matrix block based on the sub-output matrix block; if the query matrix block is the last block of the query matrix, the vector calculation engine performs a second normalization operation on the intermediate output matrix block based on the updated second normalization factor to obtain the output matrix block corresponding to the query matrix block. Based on the above formulas (3) to (9), this process refers to the following formulas (10) to (15):

[0108]

[0109]

[0110]

[0111]

[0112] The above formulas (10) to (13) are to update the second normalization factor l i The process. Formula (10) is used to calculate the query matrix block Q i The corresponding attention weight matrix The sum of each row in ij It should be noted that the present application does not limit the execution time of formula (10). For example, the vector calculation engine can also calculate the attention weight matrix in the aforementioned step 402. After that, execute formula (10). Formula (11) is used to calculate the first normalization factor m′ of the historical storage in the second memory. i With the latest first normalization factor The difference between It should be noted that this process can also be performed before executing formula (6) in the aforementioned step 402, and this application does not limit this. Formula (12) is used to Perform exponential operation. Formula (13) is used to calculate the second normalization factor l′ based on the second memory. i and the attention weight matrix The sum of each row in ij , update the second normalization factor l i .

[0113]

[0114] O i =O i / l i (15)

[0115] The above formula (14) is based on the sub-output matrix block O ij Update the query matrix block Q i The corresponding intermediate output matrix block O i The above formula (15) means that if the query matrix block Q i The last block of the query matrix Q is based on the updated second normalization factor l i , for the intermediate output matrix block O i Perform the second normalization operation to obtain the query matrix block Q i The corresponding output matrix block O i process.

[0116] 404. The memory access engine writes the output matrix block of the attention operator into the first memory.

[0117] In an embodiment of the present application, after obtaining the output matrix block corresponding to any query matrix block based on the aforementioned steps 402 and 403, it is written into the first memory. After the output matrix blocks corresponding to all query matrix blocks of the query matrix are written into the first memory, the output matrix block of the attention operator is obtained. These output matrix blocks are combined to obtain the output matrix of the attention operator.

[0118] The following pseudo code is referred to, and the data processing method shown in the above steps 401 to 404 is described by taking the first memory as HBM and the second memory as SRAM as an example.

[0119] {

[0120] Requirement: Matrix in HBM (i.e. the query matrix Q, key matrix K and value matrix V stored in the first memory HBM), SRAM of size M (i.e. the storage capacity M of the second memory SRAM);

[0121] 1. Set block sizes Set the size of the output matrix block;

[0122] 2. Initialize inHBM, inSRAM, inSRAM; initialize the output matrix O to 0 in the first memory HBM, which will serve as an accumulator to continuously update the output matrix, and initialize the first normalization factor m and the second normalization factor l in the second memory SRAM;

[0123] 3.Divide Q, O into blocks Q1,...,QTr and O1,...,O Tr of size B r ×deach, and divide K, Vinto blocks K1,...,K Tc and V1,...,V Tc of size B c ×deach; Divide the query matrix Q and the output matrix O into T rows r row blocks, each row block is of size [B r , d], divide the key matrix K and value matrix V into T by row c row blocks, each row block is of size [B c , d];

[0124] 4.for 1≤i≤T r d o ; Outer loop, cross-row loop;

[0125] 5. Load Q i from HBM to SRAM; the memory access engine transfers the query matrix block Q in the first memory HBM i Read into the second memory SRAM;

[0126] 6.for 1≤j≤T c do; inner loop, cross-column loop;

[0127] 7.Load K j From HBM to SRAM, compute The memory access engine transfers the key matrix block K in the first memory HBM i Read into the second memory SRAM, the matrix calculation engine performs matrix multiplication operation, and calculates the query matrix block Q i The corresponding attention weight matrix S ij ;

[0128] 8. compute The vector calculation engine calculates the attention score matrix S ij The maximum value of each row in

[0129] 9. compute The vector calculation engine calculates the stored first normalization factor m′ i and The maximum value between Calculate the stored first normalization factor m′ iWith the latest first normalization factor The difference between Update the first normalization factor m i ;

[0130] 10. compute The vector calculation engine calculates the attention score matrix S ij Each item in the first normalization factor m of the corresponding row i The difference between them, that is, point by point calculation;

[0131] 11.Compute The vector calculation engine updates the attention score matrix Perform exponential operations;

[0132] 12.Load V j from HBM to SRAM; the memory access engine transfers the value matrix block V in the first memory HBM j Read into the second memory SRAM;

[0133] 13. compute The matrix calculation engine performs matrix multiplication operations to calculate the query matrix block Q i The corresponding sub-output matrix block O ij ;

[0134] 14. compute Vector calculation engine Perform exponential operations;

[0135] 15. compute The vector calculation engine updates the second normalization factor l i ;

[0136] 16.Compute The vector calculation engine updates the query matrix block Q i The corresponding intermediate output matrix block O i ;

[0137] 17.end for; the inner loop ends;

[0138] 18.compute O i =O i / l i ; If the query matrix block Qi is the last block of the query matrix Q, the vector calculation engine performs the calculation on the intermediate output matrix block O i Perform the second normalization operation to obtain the query matrix block Q i The corresponding output matrix block O i ;

[0139] 19.write O i to HBM; the memory access engine will output the matrix block O i Write into the first memory HBM;

[0140] 20.end for; the outer loop ends;

[0141] 21.Return O; Return output matrix O;

[0142] }

[0143] refer to Figure 5 , Figure 5 is an execution flow chart of an attention operator provided in an embodiment of the present application, such as Figure 5 As shown, combined with the above pseudo code, it can be seen that the data processing method of the attention operator provided in the present application includes five steps: matrix multiplication, maximum value update, subtraction and summation calculation, matrix multiplication, output and summation update. Moreover, all intermediate variables involved in the calculation process do not need to be written to HBM during the inner loop, which reduces the interaction with HBM. In addition, by first updating the maximum value, then calculating the attention weight matrix P and the second normalization factor l, and then calculating the division after the inner loop ends, the intermediate calculation steps can be reduced, the calculation efficiency can be improved, and the operating performance of the AI ​​accelerator can be improved.

[0144] Furthermore, since the data processing method provided by the present application assigns different types of operations involved in the attention operator to different computing engines for execution, multiple computing engines can call the attention operator in parallel to execute corresponding steps. Figure 6 , Figure 6 is an execution flow chart of another attention operator provided in an embodiment of the present application, such as Figure 6 As shown, the matrix calculation engine performs the matrix multiplication operation involved in the above data processing method, and the vector calculation engine performs the maximum value calculation, addition, subtraction, division and exponential operation involved in the above data processing method. Pipelining the matrix calculation engine and the vector calculation engine can achieve the effect of parallel calculation. Figure 7 , Figure 7 Schematic diagram of the pipeline arrangement of a data processing method provided in an embodiment of the present application. Figure 7 As shown, combined with the above pseudo code, in the inner loop 0, after the matrix calculation engine executes step 7, the vector calculation engine executes steps 8 to 11. During the process of the vector calculation engine executing steps 8 to 11, if the vector calculation engine calculates the attention score matrix, the matrix calculation engine can execute step 13 in parallel to calculate the query matrix block Q i The corresponding sub-output matrix block O ij , and the matrix calculation engine calculates the sub-output matrix block Oij After that, the next inner loop 1 is entered to perform the matrix multiplication operation. During this process, the vector computing engine executes steps 14 to 16 in the inner loop 0 in parallel. In this way, the data processing method provided by the present application can save the running time of the attention operator and improve the running performance of the AI ​​accelerator.

[0145] Figure 8 : is a structural diagram of a data processing device provided in an embodiment of the present application. The data processing device can realize part or all of the functions of the aforementioned AI accelerator through software, hardware, or a combination of both. The AI ​​accelerator includes a memory access engine and a computing engine. The AI ​​accelerator is configured with a first memory and a second memory, and the read and write speed of the second memory is greater than the read and write speed of the first memory. Figure 8 As shown, the device includes: a memory access module 801 and a calculation module 802.

[0146] A memory access module 801, configured in the memory access engine, is used to read the input matrix block of the attention operator in the first memory into the second memory, where the input matrix block is a block of the input matrix of the attention operator;

[0147] The computing module 802 is configured in the computing engine and is used to call the attention operator, process the input matrix block, and obtain the output matrix block of the attention operator, wherein, in the process of processing the input matrix block, the normalization factor of the normalization operation of the attention operator in the second memory is updated.

[0148] In some embodiments, the memory access engine is a direct memory access DMA engine, the first memory is high bandwidth memory HBM or double data rate memory DDR, the second memory is static random access memory SRAM, and the storage capacity of the first memory is greater than the storage capacity of the second memory.

[0149] In some embodiments, the input matrix block includes: a query matrix block, a key matrix block, and a value matrix block, the query matrix block is a block of the query matrix of the attention operator, the key matrix block is a block of the key matrix of the attention operator, and the value matrix block is a block of the value matrix of the attention operator;

[0150] The calculation module 802 includes:

[0151] A first updating unit, configured to update a first normalization factor of a first normalization operation in the second memory based on the query matrix block and the key matrix block, and calculate an attention weight matrix corresponding to the query matrix block based on the updated first normalization factor and the first normalization operation;

[0152] The second updating unit is used to update the second normalization factor of the second normalization operation in the second memory based on the attention weight matrix and the value matrix block, and calculate the output matrix block corresponding to the query matrix block based on the updated second normalization factor and the second normalization operation.

[0153] In some embodiments, the computing engine includes a matrix computing engine and a vector computing engine, wherein the matrix computing engine and the vector computing engine execute in parallel a process of calling an attention operator to process an input matrix block.

[0154] In some embodiments, the first updating unit is configured to:

[0155] Perform matrix multiplication on the transposed matrices of the query matrix block and the key matrix block to obtain the attention score matrix corresponding to the query matrix block;

[0156] Based on the attention score matrix, the first normalization factor in the second memory is updated, and based on the updated first normalization factor and the first normalization operation, the attention weight matrix corresponding to the query matrix block is calculated, and the first normalization factor indicates the maximum value of each row in the global attention score matrix.

[0157] In some embodiments, the first updating unit is configured to:

[0158] Based on the maximum value of each row in the attention score matrix and the first normalization factor stored in the second memory, update the first normalization factor stored in the second memory;

[0159] Based on the updated first normalization factor, the first normalization operation is performed on the attention score matrix to obtain the attention weight matrix corresponding to the query matrix block.

[0160] In some embodiments, the second updating unit is configured to:

[0161] Perform matrix multiplication on the attention weight matrix and the value matrix block to obtain the sub-output matrix block corresponding to the query matrix block;

[0162] Based on the attention weight matrix, the second normalization factor in the second memory is updated, and based on the updated second normalization factor, the sub-output matrix block and the second normalization operation, the output matrix block corresponding to the query matrix block is calculated, and the second normalization factor indicates the sum of each row in the global attention weight matrix.

[0163] In some embodiments, the second updating unit is configured to:

[0164] Based on the sum of each row in the attention weight matrix, the updated first normalization factor, and the second normalization factor stored in the second memory, updating the second normalization factor in the second memory;

[0165] Based on the sub-output matrix block, update the intermediate output matrix block corresponding to the query matrix block;

[0166] If the query matrix block is the last block of the query matrix, the vector calculation engine performs a second normalization operation on the intermediate output matrix block based on the updated second normalization factor to obtain an output matrix block corresponding to the query matrix block.

[0167] Through the above device, since the reading and writing speed of the second memory is greater than the reading and writing speed of the first memory, and the normalization factor of the normalization operation in the attention operator is updated in the second memory, that is, the intermediate data generated during the execution of the attention operator does not need to be written to the first memory with a slower reading and writing speed. Therefore, the above method reduces the access to the first memory and effectively improves the operating performance of the AI ​​accelerator.

[0168] It should be noted that: the data processing device provided in the above embodiment only uses the division of the above functional modules as an example when performing data processing. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the data processing device provided in the above embodiment and the data processing method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0169] In this application, the terms "first", "second", etc. are used to distinguish between identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is the quantity and execution order limited. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various described examples, the first memory may be referred to as the second memory, and similarly, the second memory may be referred to as the first memory. The first memory and the second memory may both be memories, and in some cases, may be separate and different memories.

[0170] In this application, the term "at least one" means one or more, and the term "plurality" means two or more. For example, a plurality of memories means two or more memories.

[0171] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0172] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of program structure information. The program structure information includes one or more program instructions. When the program instructions are loaded and executed on a computing device, all or part of the processes or functions in the embodiments of the present application are generated.

[0173] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0174] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: Applied to an artificial intelligence (AI) accelerator, the AI ​​accelerator includes a memory access engine and a computing engine, the AI ​​accelerator is configured with a first memory and a second memory, the read and write speed of the second memory is greater than the read and write speed of the first memory, the method includes: The memory access engine reads an input matrix block of the attention operator in the first memory into the second memory, wherein the input matrix block is a block of the input matrix of the attention operator; The computing engine calls the attention operator and processes the input matrix block to obtain an output matrix block of the attention operator, wherein in the process of processing the input matrix block, a normalization factor of a normalization operation of the attention operator in the second memory is updated; The memory access engine writes the output matrix block of the attention operator into the first memory.

2. The method according to claim 1, characterized in that The memory access engine is a direct memory access DMA engine, the first memory is a high bandwidth memory HBM or a double data rate memory DDR, the second memory is a static random access memory SRAM, and the storage capacity of the first memory is greater than the storage capacity of the second memory.

3. The method according to claim 1 or 2, characterized in that: The input matrix block includes: a query matrix block, a key matrix block and a value matrix block, wherein the query matrix block is a block of the query matrix of the attention operator, the key matrix block is a block of the key matrix of the attention operator, and the value matrix block is a block of the value matrix of the attention operator; The computing engine calls the attention operator, processes the input matrix block, and obtains the output matrix block of the attention operator, including: The computing engine updates a first normalization factor of a first normalization operation in the second memory based on the query matrix block and the key matrix block, and calculates an attention weight matrix corresponding to the query matrix block based on the updated first normalization factor and the first normalization operation; The computing engine updates the second normalization factor of the second normalization operation in the second memory based on the attention weight matrix and the value matrix block, and calculates the output matrix block corresponding to the query matrix block based on the updated second normalization factor and the second normalization operation.

4. The method according to claim 3, characterized in that The computing engine includes a matrix computing engine and a vector computing engine, wherein the matrix computing engine and the vector computing engine execute in parallel a process of calling the attention operator to process the input matrix block.

5. The method according to claim 4, characterized in that The computing engine updates a first normalization factor of a first normalization operation in the second memory based on the query matrix block and the key matrix block, and calculates an attention weight matrix corresponding to the query matrix block based on the updated first normalization factor and the first normalization operation, including: The matrix calculation engine performs a matrix multiplication operation on the query matrix block and the transposed matrix of the key matrix block to obtain an attention score matrix corresponding to the query matrix block; The vector calculation engine updates the first normalization factor in the second memory based on the attention score matrix, and calculates the attention weight matrix corresponding to the query matrix block based on the updated first normalization factor and the first normalization operation, wherein the first normalization factor indicates the maximum value of each row in the global attention score matrix.

6. The method according to claim 5, characterized in that The vector calculation engine updates the first normalization factor in the second memory based on the attention score matrix, and calculates the attention weight matrix corresponding to the query matrix block based on the updated first normalization factor and the first normalization operation, including: The vector calculation engine updates the first normalization factor stored in the second memory based on the maximum value of each row in the attention score matrix and the first normalization factor stored in the second memory; The vector calculation engine performs the first normalization operation on the attention score matrix based on the updated first normalization factor to obtain the attention weight matrix corresponding to the query matrix block.

7. The method according to any one of claims 4 to 6, characterized in that The computing engine updates the second normalization factor of the second normalization operation in the second memory based on the attention weight matrix and the value matrix block, and calculates the output matrix block corresponding to the query matrix block based on the updated second normalization factor and the second normalization operation, including: The matrix calculation engine performs a matrix multiplication operation on the attention weight matrix and the value matrix block to obtain a sub-output matrix block corresponding to the query matrix block; The vector calculation engine updates the second normalization factor in the second memory based on the attention weight matrix, and calculates the output matrix block corresponding to the query matrix block based on the updated second normalization factor, the sub-output matrix block and the second normalization operation, wherein the second normalization factor indicates the sum of each row in the global attention weight matrix.

8. The method according to claim 7, characterized in that The vector calculation engine updates the second normalization factor stored in the second memory based on the attention weight matrix, and calculates the output matrix block corresponding to the query matrix block based on the updated second normalization factor, the sub-output matrix block and the second normalization operation, including: The vector calculation engine updates the second normalization factor in the second memory based on the sum of each row in the attention weight matrix, the updated first normalization factor, and the second normalization factor stored in the second memory; The vector calculation engine updates the intermediate output matrix block corresponding to the query matrix block based on the sub-output matrix block; If the query matrix block is the last block of the query matrix, the vector calculation engine performs the second normalization operation on the intermediate output matrix block based on the updated second normalization factor to obtain the output matrix block corresponding to the query matrix block.

9. A data processing device, characterized in that: The device is applied to an AI accelerator, the AI ​​accelerator includes a memory access engine and a computing engine, the AI ​​accelerator is configured with a first memory and a second memory, the read and write speed of the second memory is greater than the read and write speed of the first memory, and the device includes: A memory access module, configured in the memory access engine, for implementing the steps performed by the memory access engine as described in any one of claims 1 to 8; A computing module, configured in the computing engine, is used to implement the steps performed by the computing engine as described in any one of claims 1 to 8.

10. An accelerator, characterized in that: The accelerator comprises a communication interface, a memory access engine and a computing engine, wherein the communication interface is used to provide program instructions and / or data, and the accelerator is used to implement the data processing method as claimed in any one of claims 1 to 8.

11. A computing device, characterized in that: The computing device host and the AI ​​accelerator, the host is used to send data to the AI ​​accelerator and receive data sent by the AI ​​accelerator, and the AI ​​accelerator is used to execute the data processing method as described in any one of claims 1 to 8.

12. A computing device cluster, characterized in that: The method comprises a plurality of computing devices, wherein the computing devices include a host and an AI accelerator, wherein the host is used to send data to the AI ​​accelerator and receive data sent by the AI ​​accelerator, and wherein the AI ​​accelerator is used to execute the data processing method as claimed in any one of claims 1 to 8.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store at least one program code, and the at least one program code is used to execute the data processing method as claimed in any one of claims 1 to 8.

14. A computer program product, characterized in that When the computer program product runs on an AI accelerator, the AI ​​accelerator is enabled to perform the data processing method according to any one of claims 1 to 8.

Images