Data processing method and related equipment

Abstract

The invention discloses a data processing method and related equipment, and aims to improve the utilization rate of a matrix calculation unit in an intelligent chip, improve the vector class operation efficiency and further improve the throughput of the intelligent chip. The method comprises the steps that after a vector operator is obtained, whether the vector operator can be converted into an equivalent matrix multiplication operator or not is judged, and if yes, the vector operator is converted into a corresponding target matrix multiplication operator, so that a matrix calculation unit of an intelligent chip can be used for executing the target matrix multiplication operator, and the vector operator is converted into the equivalent matrix multiplication operator. And the overall operation efficiency of the intelligent chip is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data processing method and related equipment. Background Art

[0002] In the context of big data and big computing, artificial intelligence technology represented by machine learning has developed rapidly, becoming the core foundation of key technologies such as computer vision, intelligent speech, natural language processing, biometric recognition, and recommendation systems. It has been widely used in financial risk control, medical diagnosis, smart cities, and other fields, and has gradually become one of the main forces driving the information revolution and social development. The rapid development of artificial intelligence is due to two important factors, namely the innovation of algorithm models and the continuous improvement of the computing power of smart chips.

[0003] In the post-Moore era, although the density of chip transistors continues to increase, it is difficult to further improve the power density and performance density, which means that it is impossible to improve computing power through process improvements. Therefore, an important branch of chip development is the domain specific architecture (DSA), also known as smart chips. This type of chip is highly specialized and simple in design. According to the specific characteristics of the application, the computing unit is customized, the control logic is simplified, and the storage structure and data path that are adapted to the domain computing characteristics are designed. Although it sacrifices versatility and flexibility, it has achieved higher performance and energy efficiency. It has been widely used in high-performance computing, artificial intelligence, cryptography and other fields. The core computing unit of the smart chip consists of two parts, one is the matrix computing unit, which is used to perform matrix multiplication calculations, and the other is the vector computing unit, which mainly completes the acceleration of vector operations.

[0004] The core of the design of smart chips is to accelerate matrix multiplication operations, so the area of ​​matrix computing units on smart chips accounts for a high proportion. Compared with the computing power of vector computing units, matrix computing units have an order of magnitude improvement. For example, in some smart chips, the computing power ratio of matrix computing units to vector computing units reaches 100:1, and the calculation of neural network models is often performed alternately on these two independent computing units. This results in vector computing with low computational complexity and low computing power requirements becoming a bottleneck that restricts the computing power of smart chips. Summary of the invention

[0005] The present application provides a data processing method and related equipment to solve the problem that the computing power of the vector computing unit of the smart chip is weak and restricts the computing power of the smart chip.

[0006] The first aspect provides a data processing method. The method includes: determining whether a vector operator meets the conditions, and when the vector operator meets the conditions, converting the vector operator into a target matrix multiplication operator that can be executed by the matrix calculation unit of the chip. Wherein, the vector operator is an operator executed by the vector calculation unit of the chip. The first calculation result of the target matrix multiplication operator is the same as the second calculation result of the vector operator, that is, the target matrix multiplication operator is an equivalent operator of the vector operator, and the calculation results before and after the vector operator conversion are consistent. When the vector operator meets the conditions, the vector operator is converted into a target matrix multiplication operator, that is, the vector operator is converted into a data format that can be processed by the matrix calculation unit, so that the converted target matrix multiplication operator can be operated on the matrix calculation unit, thereby accelerating vector operations and improving the reasoning efficiency of the neural network model.

[0007] In a possible implementation, determining whether a vector operator meets the conditions includes: determining whether the vector operator is a target vector operator, and if the vector operator is a target vector operator, determining that the vector operator meets the conditions. If the vector operator is not a target vector operator, determining that the vector operator does not meet the conditions. The target vector operator is an operator that can be converted into a matrix multiplication operator, and the calculation result of the target vector operator is the same as the calculation result of the converted matrix multiplication operator. This ensures that the vector operator can be converted into a matrix multiplication operator, and ensures that the calculation result of the converted target matrix multiplication operator is consistent with the calculation result of the vector operator.

[0008] In a possible implementation, determining whether the vector operator meets the conditions also includes: determining whether the first cost is less than the second cost. The first cost is the predicted time required for the matrix calculation unit to execute the target matrix multiplication operator to complete the calculation, and the second cost is the predicted time required for the vector calculation unit to execute the vector operator to complete the calculation. If the vector operator is the target vector operator, determining that the vector operator meets the conditions includes: if the vector operator is the target vector operator and the first cost is less than the second cost, determining that the vector operator meets the conditions. Thus, by predicting and comparing the time overhead required by the matrix calculation unit and the vector calculation unit, it can be ensured that the calculation is ultimately performed in a manner with the least time overhead, thereby optimizing the processing efficiency of the smart chip.

[0009] In one possible implementation, the target vector operator includes at least one of matrix row summation, matrix column summation, matrix multiplication, vector outer product, and Hadamard product operation between a matrix and a vector.

[0010] In a possible implementation, the target matrix multiplication operator includes a construction matrix, where the construction matrix is ​​a matrix constructed according to the vector operator so that the first calculation result is the same as the second calculation result.

[0011] In one possible implementation, the chip includes a buffer connected to the matrix calculation unit, and the buffer is used to store a block matrix of the construction matrix. Converting the vector operator to the target matrix multiplication operator includes: constructing a target block matrix according to the size of the buffer and the vector operator, the size of the target block matrix is ​​less than or equal to the size of the buffer, the target block matrix is ​​a part of the construction matrix, and the values ​​in the target block matrix are not all 0. Therefore, it is not necessary to construct a complete construction matrix, reduce the occupation of storage space, reduce invalid operations and reduce data handling overhead, thereby improving the processing efficiency of the matrix calculation unit.

[0012] In a possible implementation, the matrix calculation unit includes at least one multiplication accumulator, and the multiplication accumulator is used to calculate matrix multiplication between matrices of a preset size. The size of the target block matrix is ​​an integer multiple of the preset size (greater than or equal to 1).

[0013] The second aspect provides a data processing method. The method is applied to a chip, and the chip includes a vector calculation unit and a matrix calculation unit. The method includes: obtaining a target vector operator, converting the target vector operator into a target matrix multiplication operator, and then executing the target matrix multiplication operator using the matrix calculation unit. Among them, the first calculation result of the target matrix multiplication operator is the same as the second calculation result of the target vector operator. The target vector operator is an operator that can be converted into a matrix multiplication operator, and the calculation result of the converted matrix multiplication operator is the same as the calculation result of the vector operator.

[0014] In a possible implementation, the target matrix multiplication operator includes a construction matrix, where the construction matrix is ​​a matrix constructed according to the target vector operator so that the first calculation result is the same as the second calculation result.

[0015] In one possible implementation, the chip includes a buffer connected to the matrix computing unit, the buffer is used to store a block matrix of a construction matrix, and converting a target vector operator into a target matrix multiplication operator includes: constructing a target block matrix according to the size of the buffer and the target vector operator, the size of the target block matrix is ​​less than or equal to the size of the buffer, the target block matrix is ​​part of the construction matrix, and the values ​​in the target block matrix are not all zero.

[0016] The third aspect provides a data processing method. The method is applied to a host central processing unit. The method includes: determining whether a vector operator meets a condition, and if the vector operator meets the condition, issuing an instruction. The instruction instructs the chip to convert the vector operator into a matrix multiplication operator. The vector operator is an operator executed by a vector calculation unit of the chip.

[0017] In a possible implementation, determining whether the vector operator meets the condition includes: determining whether the vector operator is a target vector operator, and if the vector operator is a target vector operator, determining that the vector operator meets the condition. The target vector operator is an operator that can be converted into a matrix multiplication operator, and the calculation result of the target vector operator is the same as the calculation result of the converted matrix multiplication operator.

[0018] In a possible implementation, determining whether the vector operator meets the condition further includes: determining whether the first cost is less than the second cost, the first cost being the time required for the predicted matrix calculation unit to execute the target matrix multiplication operator to complete the calculation, and the second cost being the time required for the predicted vector calculation unit to execute the vector operator to complete the calculation. If the vector operator is the target vector operator, determining that the vector operator meets the condition includes: if the vector operator is the target vector operator and the first cost is less than the second cost, determining that the vector operator meets the condition.

[0019] In one possible implementation, the target vector operator includes at least one of matrix row summation, matrix column summation, matrix multiplication, vector outer product, and Hadamard product operation between a matrix and a vector.

[0020] A fourth aspect provides a device. The device includes a first processing module and a second processing module. The first processing module is used to determine whether a vector operator meets a condition, and the vector operator is an operator executed by a vector calculation unit of a chip. The second processing module is used to convert the vector operator into a target matrix multiplication operator when the vector operator meets the condition, so that the matrix calculation unit of the chip can execute the target matrix multiplication operator, and the first calculation result of the target matrix multiplication operator is the same as the second calculation result of the vector operator.

[0021] In a possible implementation, the first processing module is used to determine whether the vector operator is a target vector operator, where the target vector operator is an operator that can be converted into a matrix multiplication operator, and the calculation result of the converted matrix multiplication operator is the same as the calculation result of the vector operator. The first processing module is used to determine whether the vector operator meets the condition when the vector operator is the target vector operator.

[0022] In one possible implementation, the target vector operator includes at least one of matrix row summation, matrix column summation, matrix multiplication, vector outer product, and Hadamard product operation between a matrix and a vector.

[0023] In a possible implementation, the first processing module is used to determine whether the first cost is less than the second cost, the first cost is the predicted time required for the matrix calculation unit to execute the target matrix multiplication operator to complete the calculation, and the second cost is the predicted time required for the vector calculation unit to execute the vector operator to complete the calculation. The first processing module is used to determine that the vector operator meets the condition when the vector operator is the target vector operator and the first cost is less than the second cost.

[0024] In a possible implementation, the target matrix multiplication operator includes a construction matrix, which is a matrix constructed according to a vector operator so that the first calculation result is the same as the second calculation result. The chip includes a buffer connected to the matrix calculation unit, and the buffer is used to store a block matrix of the construction matrix. The second processing module is used to construct a target block matrix according to the size of the buffer and the vector operator, the size of the target block matrix is ​​less than or equal to the size of the buffer, the target block matrix is ​​a part of the construction matrix, and the values ​​in the target block matrix are not all 0.

[0025] A fifth aspect provides a device. The device is applied to a chip, the chip includes a vector calculation unit and a matrix calculation unit, and the device includes a processing module. The processing module is used to obtain a target vector operator. The processing module is used to convert the target vector operator into a target matrix multiplication operator so that the matrix calculation unit can execute the target matrix multiplication operator, and the first calculation result of the target matrix multiplication operator is the same as the second calculation result of the target vector operator.

[0026] In a possible implementation, the target matrix multiplication operator includes a construction matrix, which is a matrix constructed according to a vector operator so that the first calculation result is the same as the second calculation result. The chip includes a buffer connected to the matrix calculation unit, and the buffer is used to store a block matrix of the construction matrix. A processing module is used to construct a target block matrix according to the size of the buffer and the vector operator, the size of the target block matrix is ​​less than or equal to the size of the buffer, the target block matrix is ​​a part of the construction matrix, and the values ​​in the target block matrix are not all 0.

[0027] A sixth aspect provides a device. The device includes a processing module. The processing module is used to determine whether a vector operator meets a condition, where the vector operator is an operator executed by a vector computing unit of a chip. The processing module is used to issue an instruction when the vector operator meets the condition, the instruction instructing the chip to convert the vector operator into a matrix multiplication operator.

[0028] In a possible implementation, the processing module is used to determine whether the vector operator is a target vector operator, the target vector operator is an operator that can be converted into a matrix multiplication operator, and the calculation result of the target vector operator is the same as the calculation result of the converted matrix multiplication operator. The processing module is used to determine that the vector operator meets the condition when the vector operator is the target vector operator.

[0029] In one possible implementation, the target vector operator includes at least one of matrix row summation, matrix column summation, matrix multiplication, vector outer product, and Hadamard product operation between a matrix and a vector.

[0030] In a possible implementation, the processing module is used to determine whether the first cost is less than the second cost, the first cost is the predicted time required for the matrix calculation unit to execute the target matrix multiplication operator to complete the calculation, and the second cost is the predicted time required for the vector calculation unit to execute the vector operator to complete the calculation. The processing module is used to determine that the vector operator meets the condition when the vector operator is the target vector operator and the first cost is less than the second cost.

[0031] A seventh aspect provides an electronic device. The electronic device includes a processor, the processor is coupled to a memory, and the processor is configured to execute the data processing method in the first aspect or any possible implementation of the first aspect, or the second aspect or any possible implementation of the second aspect, or the third aspect or any possible implementation of the third aspect based on instructions stored in the memory.

[0032] In an eighth aspect, a computer-readable storage medium is provided. The computer-readable storage medium includes instructions, and when the computer-readable storage medium is run on a computer, the computer executes the steps in the data processing method in the first aspect or any possible implementation of the first aspect, or the second aspect or any possible implementation of the second aspect, or the third aspect or any possible implementation of the third aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a schematic diagram of the architecture of an artificial intelligence chip;

[0034] Figure 2 A hardware structure of a chip provided for this application;

[0035] Figure 3 A schematic diagram of the structure of a data processing system provided for this application;

[0036] Figure 4 A schematic diagram of the system architecture used in this application;

[0037] Figure 5 A flowchart of a data processing method provided in this application;

[0038] Figure 6a A schematic diagram of converting a vector operator provided in this application into a corresponding target matrix multiplication operator;

[0039] Figure 6bA schematic diagram of converting another vector operator provided in this application into a corresponding target matrix multiplication operator;

[0040] Figure 6c A schematic diagram of converting another vector operator provided in this application into a corresponding target matrix multiplication operator;

[0041] Figure 6d A schematic diagram of converting another vector operator provided in this application into a corresponding target matrix multiplication operator;

[0042] Figure 6e A schematic diagram of converting another vector operator provided in this application into a corresponding target matrix multiplication operator;

[0043] Figure 6f A schematic diagram of converting another vector operator provided in this application into a corresponding target matrix multiplication operator;

[0044] Figure 7 A flowchart of another data processing method provided for this application;

[0045] Figure 8 A flowchart of another data processing method provided by the present application;

[0046] Figure 9a It is a comparison of the time taken for a vector operator to complete the operation in the vector calculation unit and the time taken for the operator to complete the operation in the matrix calculation unit after the operation is completed in the vector calculation unit;

[0047] Figure 9b A comparison of the time taken for another vector operator to complete the operation in the vector computing unit and its conversion thereafter in the matrix computing unit;

[0048] Fig.9c A comparison of the time taken for another vector operator to complete an operation in a vector computing unit and the time taken for the operator to complete an operation in a matrix computing unit after conversion;

[0049] Figure 9d The benefits of completing end-to-end reasoning of a neural network model based on the solution provided by this application;

[0050] Fig.10 A schematic diagram of the structure of a device provided in this application;

[0051] Fig.11 A schematic diagram of the structure of another device provided in this application;

[0052] Fig.12 A schematic diagram of the structure of another device provided in this application;

[0053] Fig.13 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0054] The present application provides a processing method and related equipment to improve the utilization rate of matrix computing units in smart chips, improve the throughput of smart chips, and thus improve the reasoning or training efficiency of neural network models.

[0055] The following describes the embodiments of the present application in conjunction with the accompanying drawings. Obviously, the described embodiments are only embodiments of a part of the present application, rather than all embodiments. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0056] The terms "first", "second", etc. in the specification and claims of this application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. "Multiple" means greater than or equal to 2.

[0057] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0058] In order to better explain the technical solution of the present application, the relevant terms involved in the present application are explained below.

[0059] 1. Operator (OP for short)

[0060] In this application, the operator is an operation performed by a matrix multiplication calculation unit or a vector calculation unit in an AI chip, that is, the operator in this embodiment is a basic operation performed on the calculation circuit of the AI ​​chip. The operation performed by the matrix calculation unit is called a matrix multiplication operator, such as matrix multiplication. The operations performed by the vector calculation unit are collectively referred to as vector operators. Vector operators include, for example, operations between vectors, operations between vectors and matrices, operations between vectors and scalars, operations between matrices and scalars, and operations such as vector / matrix reduction and summation. Vector operators are not matrix multiplication operators.

[0061] In this embodiment, a layer in a neural network is called a neural network operator, and a neural network operator can be composed of one or more basic calculation operations.

[0062] 2. Matrix

[0063] A matrix of dimension m×n is a rectangular array formed by arranging elements of m rows and n columns. For example, matrix A is shown in equation (1) below, and matrix B is shown in equation (2) below.

[0064]

[0065] 3. Matrix multiplication

[0066] Two matrices can be multiplied only if the number of columns of the first matrix (the matrix on the left, hereinafter referred to as the left matrix) is the same as the number of rows of the second matrix (the matrix on the right, hereinafter referred to as the right matrix). ij ) is a matrix with dimension m×p, matrix B=(b ij ) is a matrix of dimension p×n. The product of matrix A and matrix B is defined as an m×n matrix C=(c ij ). Among them, the matrix C=(c ij )'s i-th row and j-th column element c ij is the i-th row element a of matrix A i1 ,a i2 ,…,a ip The element b corresponding to the jth column of matrix B 1j ,b 2j ,…,b pj The sum of the products of

[0067]

[0068] Among them, 1≤i≤m, 1≤j≤n, 1≤k≤p.

[0069] 4. Row vector

[0070] A row vector is a matrix with a dimension of 1×m, where m is a positive integer. For example, a row vector is shown in formula (3).

[0071] X=[x1 x2 ... x m ], formula (3).

[0072] 5. Column vector

[0073] A column vector is a matrix with a dimension of m×1, where m is a positive integer. For example, a column vector is shown in formula (4).

[0074]

[0075] 6. Vector outer product

[0076] The tensor product of two vectors is a matrix. For example, given a column vector U of dimension m×1 and a row vector V of dimension 1×n, the outer product of vector U and vector V, U×V, is defined as a matrix D of dimension m×n, as shown in equation (5).

[0077]

[0078] 7. Matrix row / column summation

[0079] Summing a matrix by row / column is a dimensionality reduction operation. Summing a matrix by row means summing the elements of each row of the matrix to obtain a column vector. Summing a matrix by column means summing the elements of each column of the matrix to obtain a row vector. For example, summing the rows of matrix A gives column vector C, as shown in equation (6). Summing the columns of matrix B gives row vector D, as shown in equation (7).

[0080]

[0081]

[0082] D=[b 11 +b 21 +...+b m1 b 12 +b 22 ...+b m2 ... b 1n +b 2n +...+b mn ]=[d1 d2 ... d n ]

[0083] 8. Hadamard product

[0084] The Hadamard product is an operation that multiplies the elements of the same position of two matrices. The Hadamard product operation can be performed between two two-dimensional matrices, or between a two-dimensional matrix and a row vector, or between a two-dimensional matrix and a column vector. The Hadamard product operation between two two-dimensional matrices requires that the dimensions of the two two-dimensional matrices are the same. The Hadamard product operation between a two-dimensional matrix and a row vector requires that the number of columns of the two-dimensional matrix is ​​the same as the number of columns of the row vector. The Hadamard product operation between a two-dimensional matrix and a column vector requires that the number of rows of the two-dimensional matrix is ​​the same as the number of rows of the column vector. The Hadamard product operation between two two-dimensional matrices is shown in equation (8). The Hadamard product operation between a two-dimensional matrix and a column vector is shown in equation (9). The Hadamard product operation between a two-dimensional matrix and a row vector is shown in equation (10).

[0085]

[0086]

[0087]

[0088] 9. Matrix multiplication

[0089] Matrix multiplication is the multiplication of a matrix and a scalar (constant). The matrix obtained by multiplying all elements of matrix A by a constant s is called matrix multiplication, denoted as sA or As, as shown in formula (11).

[0090]

[0091] 10. Block Matrix

[0092] For a matrix A with a high number of rows and columns, the matrix A is divided into several small matrices during operation. Each small matrix is ​​called a sub-block of A, and a matrix with sub-blocks as elements is called a block matrix. These small matrices are treated as elements during operation to simplify matrix multiplication operations. The sizes of the block matrices can be the same or different. For example, the matrix is ​​divided into block matrices as shown in formula (12). A 4*4 matrix A can be divided into 4 2*2 block matrices. Of course, there can be other division methods, and the matrix A can also be of other dimensions, which is only used as an example here.

[0093]

[0094] Machine learning is a method to achieve artificial intelligence. The goal of this method is to design and analyze some algorithms (also known as models) that allow computers to "learn" automatically. The designed algorithms are called machine learning models. Machine learning models are a type of algorithm that automatically analyzes data to obtain patterns and uses the patterns to predict unknown data.

[0095] Deep learning is a new technical field that emerged in the process of machine learning research. Specifically, deep learning is a method in machine learning based on deep representation learning of data. Deep learning interprets data by establishing a neural network that simulates the human brain for analysis and learning. Since in machine learning methods, almost all features need to be determined by industry experts and then encoded. However, deep learning algorithms try to learn features from data by themselves. Algorithms designed based on deep learning ideas are called deep learning models.

[0096] The typical structure of the current deep learning model is a deep neural network. A neural network model is a mathematical model or computational model that imitates the structure and function of a biological neural network (the central nervous system of an animal, especially the brain). A neural network model is calculated by connecting a large number of neurons. A neural network model can include a variety of neural network layers with different functions, each layer including parameters and calculation rules. Different layers in a neural network have different names according to different calculation formulas or functions. For example, the layer that performs convolution calculations is called a convolution layer, which is often used to extract features from input signals (e.g., images). A neural network can also be composed of a combination of multiple sub-neural networks. Neural networks with different structures can be applied to different scenarios (e.g., classification, recognition) or provide different effects when used in the same scenario. The different structures of neural networks specifically include one or more of the following: different number of layers in the neural network, different order of each network layer, and different weights, parameters or calculation formulas in each network layer. There are many different neural networks with high accuracy for application scenarios such as recognition or classification in the industry. Some neural networks can be trained with specific data sets and used alone to complete a task or combined with other neural networks (or other functional modules) to complete a task. In other words, the deep learning model is actually a machine learning model with a complex structure of a neural network.

[0097] In the context of big data and big computing, artificial intelligence technology represented by machine learning has developed rapidly, becoming the core foundation of key technologies such as computer vision, intelligent speech, natural language processing, biometric recognition, and recommendation systems. It has been widely used in transportation, education, finance, medical care, smart cities, and other fields, and has gradually become one of the main forces driving the information revolution and social development. The rapid development of artificial intelligence is due to two important factors, namely the innovation of algorithm models and the continuous improvement of the computing power of smart chips.

[0098] Neural networks include convolution neural network (CNN), recurrent neural network (RNN), feedforward neural network (FNN) and self-attention mechanism. The transformer neural network model based on the self-attention mechanism was originally used for natural language processing tasks, and later expanded to computer vision, speech recognition, recommendation systems and other fields. Taking the transformer architecture as an example, this type of neural network consists of several transformer encoders / decoders. Each encoder or decoder contains a self-attention mechanism structure and a feedforward network structure. In this network model, most of the computational complexity comes from matrix multiplication, such as convolution or full connection, and the remaining part of the computational overhead comes from vector operators such as normalized exponential function (softmax) and layer normalization (layernorm). In addition, there are also a small number of scalar operations, such as conditional judgment.

[0099] In the post-Moore era, although the density of chip transistors continues to increase, it is difficult to further improve the power density and performance density, which means that it is impossible to increase computing power through process improvements. Therefore, an important branch of chip development is domain specific architecture (DSA), also known as smart chips, artificial intelligence (AI) chips or AI processors. This type of chip is highly specialized and simple in design. According to the specific characteristics of the application, the computing unit is customized, the control logic is simplified, and the storage structure and data path that are adapted to the computing characteristics of the domain are designed. Although it sacrifices versatility and flexibility, it has achieved higher performance and energy efficiency. It has been widely used in high-performance computing, artificial intelligence, cryptography and other fields. For example, AI chips designed specifically for artificial intelligence scenarios include Nvidia's graphics processing unit (GPU), Google's tensor processing unit (TPU), and Huawei's Ascend AI chip, etc. Take the DaVinci chip architecture designed specifically for artificial intelligence scenarios as an example. Figure 1 As shown, its core computing unit includes a matrix computing unit and a vector computing unit. The matrix computing unit is used to perform matrix multiplication operations, and the vector computing unit is used to perform vector operations.

[0100] The core of the design of smart chips is to accelerate matrix multiplication operations. Therefore, the area of ​​the matrix computing unit on the smart chip accounts for a relatively high proportion, which is an order of magnitude higher than the computing power of the vector computing unit. For example, in some smart chips, the computing power ratio of the matrix computing unit to the vector computing unit reaches 100:1, and the calculation of neural networks is often performed alternately on the matrix computing unit and the vector computing unit. This results in vector operations with low computational complexity and low computing power requirements becoming a bottleneck that restricts the computing power of smart chips, affecting the throughput of smart chips and reducing the training and reasoning efficiency of smart chips.

[0101] Figure 2 A hardware structure of a chip provided for the present application. The chip can be set in an electronic device for training a neural network model, or can be set in an electronic device for reasoning using a neural network model. The chip includes a matrix calculation unit, which is used to execute a matrix multiplication operator to perform a matrix multiplication operation. The matrix calculation unit can also be called a cube unit or a tensor core, etc. The chip also includes a vector calculation unit, which is used to execute a vector operator to perform a vector operation. The vector calculation unit can also be called a vector unit, a cudacore, etc. Optionally, the chip also includes a buffer (such as a buffer, a cache, or a register, etc.), which is used to store data input to the matrix calculation unit, and to supply input data to the matrix calculation unit in a timely and accurate manner. Optionally, the chip can also include a scalar calculation unit (not shown), which is mainly responsible for various types of scalar data operations and program flow control. The algorithms of each layer in the neural network can be implemented in the following manner: Figure 2 is implemented in the chip shown.

[0102] The AI ​​chip is mounted on the main central processing unit (CPU) (host CPU) as a coprocessor, and the main CPU assigns tasks. The AI ​​chip and the main CPU can be two independent devices with a connection relationship. The AI ​​chip can be connected to the main CPU in a pluggable manner, and the AI ​​chip and the main CPU can also be integrated into the same system on a chip (system on a chip, SoC). In another possible implementation, the main CPU can also be embedded in the AI ​​chip as part of the AI ​​chip, which is not limited here.

[0103] In this embodiment, when the main CPU obtains a certain operator in the neural network, if the operator belongs to a vector operator, it determines whether the vector operator meets the conditions. If the vector operator meets the conditions, the main CPU issues an instruction, which instructs the AI ​​chip to convert the vector operator into the corresponding target matrix multiplication operator. Thus, the AI ​​chip can use the matrix calculation unit to execute the target matrix multiplication operator without using the vector calculation unit to execute the vector operator, and use the powerful computing power of the matrix calculation unit to complete the operation that should be completed by the vector calculation unit, which can reduce the computing pressure of the vector calculation unit, accelerate the vector operator operation, improve the computing bottleneck caused by insufficient computing power of the vector calculation unit, and improve the reasoning or training efficiency of the neural network model.

[0104] Of course, in another implementation, when the main CPU determines that a certain vector operator meets the conditions, the main CPU can convert it into a target matrix multiplication operator, and the matrix calculation unit of the AI ​​chip executes the target matrix multiplication operator.

[0105] The main CPU determines whether the vector operator meets the conditions, and the detailed process of converting the vector operator into the matrix multiplication operator will be described below.

[0106] For example, Figure 3 As shown, Figure 3 A schematic diagram of the structure of a data processing system provided in the present application. The data processing system includes an AI chip 50, a main CPU and an external memory connected to each other. The core part of the AI ​​chip 50 is a matrix calculation unit 503, and the controller 504 controls the matrix calculation unit 503 to extract data from the memory (weight memory 502 or input memory 501) and perform matrix multiplication operations.

[0107] In some implementations, the matrix calculation unit 503 includes multiple processing units (process engines, PEs) inside. In some implementations, the matrix calculation unit 503 is a two-dimensional systolic array. The matrix calculation unit 503 can also be a one-dimensional systolic array or other electronic circuits that can perform mathematical operations such as multiplication and addition. In some implementations, the matrix calculation unit 503 is a general matrix multiplication calculation unit. Each PE includes a multiplication accumulator (MAC). MAC is an operation circuit for implementing matrix multiplication. One MAC can perform matrix multiplication between two matrices of preset size at a time. The MAC operator implements parallel computing through a specific circuit design to accelerate multiplication and accumulation.

[0108] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The matrix calculation unit takes the corresponding data of matrix B from the weight memory 502 and caches it on each PE in the matrix calculation unit 503. The matrix calculation unit takes the matrix A data from the input memory 501 and performs matrix operations with the matrix B, and the partial results or the final results of the matrix are stored in the accumulator 508.

[0109] The vector calculation unit 507 can further process the output of the matrix calculation unit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 507 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0110] In some implementations, the vector calculation unit can 507 store the processed output vector to the unified memory 506. For example, the vector calculation unit 507 can apply a nonlinear function to the output of the matrix calculation unit 503, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 507 generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the matrix calculation unit 503, such as for use in a subsequent layer in a neural network.

[0111] The unified memory 506 is used to store input data and output data.

[0112] The weight data is directly transferred from the external memory to the input memory 501 and / or the unified memory 506 through the direct memory access controller 505 (DMAC), the weight data in the external memory is stored in the weight memory 502, and the data in the unified memory 506 is stored in the external memory.

[0113] The bus interface unit (BIU) 510 is used to implement the interaction between the main CPU, DMAC and instruction fetch memory 509 through the bus.

[0114] An instruction fetch buffer 509 connected to the controller 504 is used to store instructions used by the controller 504 .

[0115] The controller 504 is used to call the instructions cached in the memory 509 to control the working process of the computing accelerator.

[0116] Generally, the unified memory 506, the input memory 501, the weight memory 502 and the instruction fetch memory 509 are all on-chip memories, and the external memory is a memory outside the AI ​​chip 50, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.

[0117] The operations of operators at each layer in the neural network may be performed by the matrix calculation unit 503 or the vector calculation unit 507 .

[0118] The instructions issued by the main CPU in response to the vector operator meeting the conditions can be stored in the external memory, and then transmitted to the instruction fetch memory 509 through the BIU510, and the controller 504 calls the instruction cached in the instruction memory 509 to realize the conversion of the control vector operator to the matrix multiplication operator. Of course, when the main CPU is embedded in the AI ​​chip 50, the instructions issued by the main CPU in response to the vector operator meeting the conditions can also be not stored in the external memory, but directly stored in the on-chip memory for the controller 504 to call, and there is no restriction here.

[0119] The application scenario of this application is to use smart chips to accelerate tasks such as artificial intelligence and high-performance computing, especially the training and reasoning of large-scale neural networks. Figure 4 As shown, the solution provided by this application is based on the operator library level of the system software stack and can be applied to different neural network operators as long as some operations in the neural network operator meet the conditions, such as softmax, layernorm and other neural network operators. It supports a variety of hardware downward, such as Huawei Ascend, NVIDIA GPU and Google TPU. If there are matrix computing units and vector computing units on smart chips from different manufacturers, the vector calculations originally performed on the vector computing units can be accelerated based on the solution provided by this application. It supports a variety of scenarios upward, such as artificial intelligence, high-performance computing and database retrieval. The algorithms in such scenarios often contain both matrix multiplication operations and vector operations.

[0120] The data processing method of the present application is described in detail below with reference to the accompanying drawings.

[0121] like Figure 5 As shown, Figure 5A flowchart of a data processing method provided in the present application. This embodiment is performed by an electronic device. The electronic device includes an AI chip, and the AI ​​chip includes a matrix calculation unit and a vector calculation unit. The electronic device can be an inference device or a training device. The electronic device is, for example, a mobile phone, a tablet computer, a laptop computer, a computer, a server, a workstation or a computer cluster. This embodiment includes the following steps:

[0122] S501: Determine whether the vector operator meets the conditions.

[0123] It is determined whether the vector operator meets the conditions to determine whether the vector operator can or is suitable for conversion into a matrix multiplication operator.

[0124] In this embodiment, determining whether a vector operator meets the conditions includes determining whether the vector operator is a target vector operator. The target vector operator is a vector operator that can be converted into a matrix multiplication operator, and the calculation result of the matrix multiplication operator after the conversion is the same as the calculation result of the vector operator before the conversion. Not all vector operators can be smoothly converted into matrix multiplication operators. To be able to be converted into a matrix multiplication operator, a linear relationship needs to be satisfied mathematically to ensure that the calculation results of the two operators before and after the conversion are consistent. Specifically, after obtaining a vector operator, it can be queried whether the vector operator exists in the target operator set. If the vector operator exists in the target operator set, it can be determined that the target vector operator is the target vector operator, that is, the vector operator meets the conditions. The target operator set includes vector operators in the operator library that can be converted into matrix multiplication operators, and the calculation result of the matrix multiplication operator after the conversion is the same as the calculation result of the vector operator before the conversion.

[0125] Exemplarily, the target vector operator includes, but is not limited to, at least one of matrix row summation (such as formula (6)), matrix column summation (such as formula (7)), matrix multiplication (such as formula (11)), vector outer product (such as formula (5)), and Hadamard product operation between matrix and vector (such as formula (8)-(10)). The operations corresponding to formula (5)-formula (11) are implemented on the vector computing unit. The vector computing unit may be called the bottleneck that restricts the computing power of the AI ​​chip. On the one hand, the vector computing unit has weak computing power and low throughput compared to the matrix computing unit. On the other hand, when the vector operator includes a matrix, it is necessary to split the matrix into several vector operations, introduce multiple loops, and take a long time to calculate. This causes delays in the calculation of the subsequent matrix computing unit, which restricts the computing power of the matrix computing unit. For example, the vector computing unit performs the matrix row / column reduction summation. It is necessary to split the matrix into vectors by row / column, and each vector is further split into sub-vectors that can be operated by the vector computing unit and solved with one instruction. The results of several sub-vectors are then reduced and summed to obtain the final result. For example, the vector computing unit performs the Hadamard product operation between the matrix and the column vector. It is necessary to split the matrix into multiple first row vectors according to the row, broadcast the value of each row in the column vector to obtain multiple second row vectors, and then multiply the first row vector and the second row vector of the same row element by element. Matrix multiplication and the Hadamard product operation between the matrix and the row vector are similar. Both require splitting the matrix into multiple vectors, introducing multiple loops, and affecting the computing efficiency of the AI ​​chip.

[0126] Therefore, this embodiment determines whether the vector operator is the target vector operator. If it is the target vector operator, the vector operator can be subsequently converted into a corresponding matrix multiplication operator, so that the target matrix multiplication operator can be executed using a matrix calculation unit with greater computing power, thereby improving the overall computing efficiency.

[0127] If the vector operator is the target vector operator, it is determined that the vector operator meets the condition. If the vector operator is not the target vector operator, it is determined that the vector operator does not meet the condition.

[0128] Optionally, since the vector operator originally processed by the vector calculation unit is converted into a target matrix multiplication operator and transported to the matrix calculation unit for processing, additional transportation overhead will be generated. Therefore, in order to ensure that the benefits of accelerating vector operations using the matrix calculation unit are greater than the transportation overhead, in addition to determining whether the vector operator can be converted into a matrix multiplication operator, it is also possible to predict the time overhead corresponding to the conversion of the vector operator into the target matrix multiplication operator and the processing by the matrix calculation unit, and the time overhead corresponding to the processing by the vector calculation unit without converting the vector operator, and compare the two, and determine whether the vector operator meets the conditions based on the cost comparison result.

[0129] Specifically, judging whether the vector operator meets the conditions also includes judging whether the first cost is less than the second cost. The first cost is the time required for the predicted matrix calculation unit to execute the target matrix multiplication operator to complete the calculation. The first cost includes the time overhead caused by moving the target matrix multiplication operator. The second cost is the time required for the predicted vector calculation unit to execute the vector operator to complete the calculation. The first cost and the second cost can be calculated according to the cost model. The calculation method of the first cost is shown in formula (13), and the calculation method of the second cost is shown in formula (14).

[0130] T 矩阵 =α1×M+β1, formula (13).

[0131] T 向量 =α2×M+β2, formula (14).

[0132] Among them, T 矩阵 is the first cost, T 向量 is the second cost. α1, β1, α2 and β2 are coefficients. M is the matrix size in the vector operator. M can be obtained according to the vector operator. For example, if the vector operator is a matrix multiplication as shown in formula (11), or a Hadamard product operation as shown in formula (9) (10), or a matrix row / column summation operation as shown in formula (6), then M is the size of the matrix A therein. If the vector operator is a vector outer product operation as shown in formula (5), then the matrix size M is the product between the row vector and the column vector.

[0133] α1, β1, α2 and β2 can be fitted according to the test results. Specifically, the time T' required for the vector operator to be processed on the vector computing unit can be tested respectively. 向量 , and the time required for the vector operator to be converted into the corresponding matrix multiplication operator and processed on the matrix calculation unit T' 矩阵 , change the matrix size M in the vector operator to obtain multiple sets of test data, substitute them into equations (13) and (14) for linear fitting, and obtain α1, β1, α2 and β2.

[0134] If the first cost is less than the second cost, it means that the benefit of converting the vector operator to the corresponding target matrix multiplication operator and executing the target matrix multiplication operator by the matrix calculation unit is greater than the overhead introduced by converting the vector operator to the target matrix multiplication operator. If the first cost is greater than the second cost, it means that the benefit of converting the vector operator to the corresponding target matrix multiplication operator and executing the target matrix multiplication operator by the matrix calculation unit is less than the overhead introduced by converting the vector operator to the target matrix multiplication operator. If the first cost is equal to the second cost, the benefit and overhead brought by the conversion are equal, and no acceleration effect is achieved.

[0135] In one implementation, the step of determining whether the vector operator is the target vector operator may be performed first, and if the vector operator is determined to be the target operator, the step of determining whether the first cost is less than the second cost may be performed. If the vector operator is not the target operator, the step of determining whether the first cost is less than the second cost does not need to be performed. In another implementation, the step of determining whether the vector operator is the target vector operator and the step of determining whether the first cost is less than the second cost may be performed in parallel.

[0136] When the vector operator is the target vector operator and the first cost is less than the second cost, it is determined that the vector operator meets the condition. When the vector operator is not the target vector operator, or the vector operator is the target vector operator but the first cost is greater than or equal to the second cost, it is determined that the vector operator does not meet the condition.

[0137] If whether the vector operator meets the condition is determined only by judging whether the vector operator is the target vector operator, then when the vector operator is the target vector operator, it is determined that the vector operator meets the condition; when the vector operator is not the target vector operator, it is determined that the vector operator does not meet the condition.

[0138] If the vector operator meets the condition, S502 is executed. If the vector operator does not meet the condition, S504 is executed.

[0139] S502: Convert the vector operator to a target matrix multiplication operator, wherein a first calculation result of the target matrix multiplication operator is the same as a second calculation result of the vector operator.

[0140] After determining that the vector operator meets the conditions, the vector operator is converted to the target matrix multiplication operator corresponding to the vector operator, so that the matrix calculation unit can execute the target matrix multiplication operator, thereby migrating the vector operation originally performed by the vector calculation unit to the matrix calculation unit for performing an equivalent matrix multiplication operation, thereby improving the operation efficiency. In this embodiment, it is required that the first calculation result of the target matrix multiplication operator is the same as the second calculation result of the vector operator to ensure the accuracy of the converted calculation result. That is, for the same input data, the first calculation result obtained by using the matrix calculation unit to perform the operation corresponding to the target matrix multiplication operator is the same as the second calculation result obtained by using the vector calculation unit to perform the operation corresponding to the vector operator.

[0141] Different vector operators correspond to different target matrix multiplication operators. The following lists four types of conversions from vector operators to matrix multiplication operators. It is understood that the present application is not limited to the conversions of these four types of vector operators. Figure 6a-6f In the figure, the dimensions of the matrix and vector are only used as examples and should not be understood as limitations on the present application. In actual applications, the dimensions of the matrix and vector of the operator depend on the actual situation.

[0142] 1. Sum the matrix by rows or columns

[0143] like Figure 6a As shown in the figure, the vector operator on the left is the matrix row summation, and the right is the target matrix multiplication operator converted from the vector operator. The left matrix of the target matrix multiplication operator comes from the vector operator and does not need to be changed. The right matrix is ​​the constructed construction matrix. Among them, the first column element value of the construction matrix is ​​all 1, and the remaining elements are all 0, so that the calculation results of the vector operator and the target matrix multiplication operator are the same. For example, b1=a 11 +a 12 +a 13 +a 14 , b2=a 21 +a 22 +a 23 +a 24 , b3 and b4 are the same, the calculation results of the vector operator and the target matrix operator are consistent. The matrix calculation unit performs Figure 6a The target matrix multiplication operator in outputs a matrix, and the first column of the output matrix is ​​taken as the calculation result.

[0144] like Figure 6b As shown in the figure, the vector operator above is the matrix column summation, and the target matrix multiplication operator below is obtained by converting the vector operator. The left matrix of the target matrix multiplication operator is the constructed construction matrix, and the right matrix comes from the vector operator and does not need to be changed. Among them, the first row of the construction matrix has all 1 elements and the rest are all 0, so that the calculation results of the vector operator and the target matrix multiplication operator are the same. For example, b1=a 11 +a 21 +a 31 +a 41 , b2=a 12 +a 22 +a 32 +a 42 , b3 and b4 are the same, the calculation results of the vector operator and the target matrix operator are consistent. The matrix calculation unit performs Figure 6b If the target matrix multiplication operator in outputs a matrix, the first row of data in the output matrix is ​​taken as the calculation result.

[0145] 2. Hadamard product of matrix and vector

[0146] like Figure 6cAs shown, the vector operator above is a Hadamard operation between a matrix and a column vector, and the number of column vector elements is the same as the number of matrix rows. The target matrix multiplication operator obtained by the transformation of the vector operator is shown below. The left matrix of the target matrix multiplication operator is a construction matrix constructed according to the vector in the vector operator. The construction matrix is ​​a diagonal matrix, and the right matrix does not need to be changed from the matrix in the vector operator. The values ​​of the elements on the main diagonal of the construction matrix (from the upper left corner to the lower right corner of the matrix) are the values ​​of the same row elements of the column vector in the vector operator, and the rest of the positions are all 0, so that the calculation results of the vector operator and the target matrix multiplication operator are the same.

[0147] like Figure 6d As shown in the figure, the vector operator on the left is a Hadamard operation between a matrix and a row vector, and the number of row vector elements is the same as the number of matrix columns. The right side is the target matrix multiplication operator converted from the vector operator. The left matrix of the target matrix multiplication operator does not need to be changed from the matrix in the vector operator, and the right matrix is ​​a construction matrix constructed from the row vector in the vector operator. The construction matrix is ​​a diagonal matrix, and the values ​​of the elements on the main diagonal of the construction matrix are the values ​​of the same column elements of the row vector in the vector operator, and the rest of the positions are all 0, so that the calculation results of the vector operator and the target matrix multiplication operator are the same.

[0148] 3. Matrix multiplication

[0149] like Figure 6e As shown in the figure, the vector operator above is a matrix multiplication, and the target matrix multiplication operator is obtained by converting the vector operator. The left matrix of the target matrix multiplication operator comes from the matrix in the vector operator and does not need to be changed. The right matrix is ​​a construction matrix constructed based on the scalar in the vector operator. The construction matrix is ​​a diagonal matrix, and the values ​​of the elements on the main diagonal of the construction matrix are all the values ​​of the scalar in the vector operator, and the rest of the positions are all 0.

[0150] 4. Vector outer product

[0151] like Figure 6f As shown, the vector operator above is the vector outer product, and the target matrix multiplication operator obtained by transformation according to the vector operator is shown below. The left matrix of the target matrix multiplication operator is the first construction matrix constructed according to the column vector in the vector operator, and the right matrix is ​​the second construction matrix constructed according to the row vector in the vector operator. The first construction matrix is ​​a two-dimensional matrix obtained by broadcasting along the rows according to the column vector in the vector operator, and the number of columns of the first construction matrix is ​​equal to the number of columns of the row vector in the vector operator. The second construction matrix is ​​a diagonal matrix, and the number of rows and columns of the second construction matrix are the same as the number of columns of the row vector in the vector operator. The values ​​of the elements on its main diagonal are the values ​​of the same column elements of the row vector in the vector operator, and the rest of the positions are all 0, so that the calculation results of the vector operator and the target matrix multiplication operator are the same.

[0152] In the above four conversion scenarios, the size of the construction matrix in the target matrix multiplication operator depends on the size of the matrix / vector in the vector operator. For example, Figure 6a-6e In the target matrix multiplication operator, the matrix size from the vector operator is (M,N). If the construction matrix is ​​the left matrix, the size of the construction matrix is ​​(M,M); if the construction matrix is ​​the right matrix, the size of the construction matrix is ​​(N,N).

[0153] When the dimension of the constructed matrix is ​​large, the constructed matrix will take up more storage space and increase the transportation overhead. If the on-chip storage space of the AI ​​chip is insufficient, it needs to be stored in an external memory, which will further increase the transportation overhead. For matrix multiplication operations of matrices with larger dimensions, the entire matrix multiplication operation cannot be completed at one time due to the hardware limitations of the matrix calculation unit. Generally, the matrix is ​​divided into multiple block matrices. Each time the AI ​​chip loads a block matrix into the on-chip buffer for calculation, it performs a block matrix multiplication operation.

[0154] Optionally, in order to reduce the storage space and handling overhead of the construction matrix, combined with the block matrix multiplication operation characteristics of the AI ​​chip and the characteristics of the construction matrix, in this example, when constructing the construction matrix of the target matrix multiplication operator, only the block matrix can be constructed without constructing a complete matrix, so as to reduce the storage space and handling overhead of the construction matrix. Specifically, from Figure 6a-6f It can be seen from the construction matrix shown that there are a large number of elements in the construction matrix whose values ​​are 0. Therefore, among the multiple block matrices corresponding to the construction matrix, there will be a situation where all the elements of a part of the block matrices have values ​​of 0. The values ​​of all elements in the matrix obtained by multiplying the block matrix with all elements of 0 with any matrix are still all 0, which will not affect the final calculation result. Such calculation can be regarded as invalid calculation, which not only wastes storage space, but also increases the handling cost and wastes computing resources.

[0155] Therefore, when constructing the construction matrix of the target matrix multiplication operator, only the target block matrix that can perform effective calculations can be constructed without constructing a complete matrix. The target block matrix is ​​part of the construction matrix, and the values ​​in the target block matrix are not all zero. That is, there are valid non-zero data in the target block matrix. If the construction matrix in the target matrix multiplication operator is a diagonal matrix, then the target block matrix can also be a diagonal matrix. Block matrices in which the values ​​of all elements are all 0 do not need to be constructed. And other block matrices that are the same as the target block matrix do not need to be constructed, and the target block matrix can be reused, thereby reducing the storage space occupied by the construction matrix, reducing the data that needs to be transported, and reducing invalid calculations, thereby improving the efficiency of constructing the target matrix multiplication operator and the efficiency of the matrix calculation unit in processing the target matrix multiplication operator.

[0156] To Figure 6aFor example, the construction matrix in is divided into 4 2*2 block matrices. The 4 block matrices include 2 and 2 Block Matrix All elements in are 0, which can be regarded as an invalid block matrix and does not need to be constructed. If there are elements that are not 0, they need to be constructed. Since the constructed matrix has two identical Can be reused, so you can construct only one As the target block matrix, without constructing a complete matrix, the amount of data in the constructed matrix can be greatly reduced. It can be understood that the construction matrix and the block matrix here are only used as examples, and the dimensions of the construction matrix and the block matrix in actual applications are often larger.

[0157] If the construction matrix in the target matrix multiplication operator is a diagonal matrix, and the values ​​of the elements on the diagonal of the construction matrix are different from the values ​​of other elements on the diagonal, such as Figure 6c , 6d and the construction matrix shown in 6f, then multiple target block matrices can be constructed, all of which are diagonal matrices, and the elements on the diagonal of the multiple target block matrices contain all the elements of the corresponding vectors in the vector operator. In this case, multiple target block matrices can be constructed at the same time. Alternatively, a target block matrix can be constructed first for the matrix calculation unit to perform calculations, and then the values ​​on the diagonal of the target block matrix are replaced to obtain a new target block matrix and then used for the matrix calculation unit to perform calculations. This process can be completed on the AI ​​chip to improve processing efficiency and reduce data handling overhead.

[0158] Optionally, when constructing the construction matrix of the target matrix multiplication operator, the target block matrix can be constructed according to the size of the buffer of the AI ​​chip. The buffer can be the input buffer of the matrix calculation unit, which can be a buffer, cache or register. Taking the Da Vinci architecture as an example, the buffer can be an L1 buffer, or a buffer L0A / B. Taking the GPU as an example, the buffer can be an L0 / L1 cache. The size of the target block matrix is ​​less than or equal to the size of the buffer, so that the target block matrix can be directly stored in the on-chip buffer of the AI ​​chip, thereby reducing the data required to be transported, improving the efficiency of constructing the target matrix multiplication operator and the efficiency of the matrix calculation unit processing the target matrix multiplication operator.

[0159] Generally, the MAC in the matrix calculation unit is configured to calculate matrix multiplication between matrices of a preset size. The preset size is, for example, 4*4, 8*8, 16*16 or other sizes. Optionally, in order to maximize the performance of the matrix calculation unit, the size of the target block matrix can be an integer (greater than or equal to 1) times the preset size.

[0160] Exemplarily, taking the buffer size as 64 kilobytes (KB), the matrix calculation unit using half-precision floating-point (FP16) precision (one FP16 value occupies two bytes), and the preset size of the matrix processed by MAC as 16*16 as an example, the buffer can store a maximum of 16384 FP16 values, then the target block matrix size can be any size of 16*16, 32*16, 48*16, 32*32, ..., 128*128, etc.

[0161] S503: The matrix calculation unit executes the target matrix multiplication operator.

[0162] After completing the construction of the target matrix multiplication operator, the matrix calculation unit can execute the target matrix multiplication operator. Compared with the vector calculation unit, the matrix calculation unit can complete more calculations with one instruction, thereby improving the calculation efficiency.

[0163] S504: The vector calculation unit executes the vector operator.

[0164] If the vector operator does not meet the conditions, it means that the vector operator cannot be converted into an equivalent matrix multiplication operator, or the vector operator is more / equally efficient to be executed by the vector computing unit. The vector operator is executed by the vector computing unit to ensure maximum efficiency.

[0165] In this embodiment, by converting the vector operator that meets the conditions into the target matrix multiplication operator, the operation originally processed by the vector computing unit can be migrated to the matrix computing unit for calculation, and the more powerful computing power of matrix multiplication can be used to perform calculations more efficiently, which can improve the reasoning or training efficiency of the neural network model.

[0166] Figure 5 S501 in the example can be executed by the main CPU of the electronic device, and S502-S504 can be executed by the AI ​​chip of the electronic device. Of course, S501-S502 can also be executed by the main CPU of the electronic device, and S503-S504 can be executed by the AI ​​chip of the electronic device. This is not limited here.

[0167] like Figure 7 As shown, Figure 7 This is a flow chart of another data processing method provided by the present application. This embodiment is executed by the main CPU.

[0168] The main CPU is connected to the AI ​​chip. This embodiment includes the following steps:

[0169] S701: Determine whether the vector operator meets the conditions.

[0170] This step is similar to S501, and the relevant description can be found above, which will not be repeated here.

[0171] If the vector operator meets the condition, S702 is executed. If the vector operator does not meet the condition, S703 is executed.

[0172] S702: Issue a first instruction, which instructs the AI ​​chip to convert the vector operator into a target matrix multiplication operator.

[0173] The vector operator meets the conditions, indicating that the vector operator can be converted into a matrix multiplication operator. Therefore, the main CPU issues a first instruction instructing the AI ​​chip to convert the vector operator into a target matrix multiplication operator, so that the matrix calculation unit of the AI ​​chip can execute the target matrix multiplication operator.

[0174] S703: Issue a second instruction, where the second instruction instructs the vector computing unit of the AI ​​chip to execute the vector operator.

[0175] The vector operator does not meet the conditions, which means that the vector operator cannot be converted into a matrix multiplication operator, or it is more advantageous to process the vector operator by the vector computing unit. Therefore, the main CPU issues a second instruction instructing the vector computing unit of the AI ​​chip to execute the vector operator.

[0176] In this embodiment, after obtaining the vector operator, the main CPU first determines whether the vector operator can be converted into a matrix multiplication operator that can be executed by the matrix calculation unit. After determining that the vector operator meets the conditions, it instructs the AI ​​chip to convert the vector operator into the corresponding target matrix multiplication operator, thereby ensuring the accuracy of the calculation results and improving the computing efficiency.

[0177] like Figure 8 As shown, Figure 8 A flowchart of another data processing method provided by the present application. This embodiment is performed by an AI chip. The AI ​​chip includes a matrix calculation unit and a vector calculation unit. This embodiment includes the following steps:

[0178] S801: Obtain a vector operator.

[0179] The AI ​​chip moves vector operators from off-chip memory to the AI ​​chip.

[0180] S802: Obtain a first instruction, which instructs the AI ​​chip to convert the vector operator into a target matrix multiplication operator.

[0181] The first instruction is issued by the main CPU after determining that the vector operator meets the conditions.

[0182] S803: Obtain a target matrix multiplication operator according to the vector operator.

[0183] This step is similar to S502, so it will not be repeated here.

[0184] S803: Utilize the matrix calculation unit to execute the target matrix multiplication operator.

[0185] After obtaining the target matrix multiplication operator according to the vector operator, the matrix calculation unit can execute the target matrix multiplication operator without the need for the vector calculation unit to execute the vector operator. The matrix multiplication operator can process multiple element multiplications in parallel, has higher computing power than the vector calculation unit, and can improve data computing efficiency. In the training and / or reasoning scenarios of the neural network model, it can improve the training or reasoning efficiency.

[0186] In order to verify the effect of the technical solution provided by this application, tests were conducted on different AI chips. The results show that the technical solution provided by this application can greatly improve the throughput of AI chips and significantly reduce the end-to-end reasoning efficiency.

[0187] 1. Implementation on NVIDIA V100 GPU

[0188] We will provide the technical solution provided by this application ( Figure 5 , Figure 8 ), implemented on NVIDIA V100 GPU, and compared with the implementation of open source operator library on the same hardware platform. The GPU CUDA version used was 10.2, the GCC compiler version was 7.5, the matrix size in the vector operator was (256,4194304), and the data type of matrix operation was half precision. Each test was run 10 times, and the average value after removing the maximum and minimum values ​​was finally calculated as a reference.

[0189] In the operation of summing the rows of a matrix, the comparison objects are: the reduction interface provided by NVIDIA, the reduction interface in the computing library Thrust provided by NVIDIA, the vector sum interface cudaSasum in the operator library Cublas provided by NVIDIA, and the matrix-vector multiplication gemv interface in the operator library Cublas provided by NVIDIA. The results are shown in Table 1, where the throughput is obtained by dividing the amount of calculated data by the running time.

[0190] Table 1. Matrix row summation test results

[0191] method Throughput Run time Nvidia official reduction example 20.047GB / s 0.217s Thrust optimization library reduction 308.981GB / s 0.0139s Cublas Vector Sum (cudaSasum) 404.909GB / s 0.0106s Cublas matrix-vector multiplication (gemv) 423.163GB / s 0.0101s The technical solution provided by this application 765.156GB / s 0.00561s

[0192] In the operations of matrix-vector Hadamard product and matrix multiplication, the objects of comparison include kernels customized through NVIDIA CUDA interface and transform interface in Thrust computing library provided by NVIDIA. The results are shown in the following table, where throughput is obtained by dividing the amount of data calculated by the running time.

[0193] Table 2. Test results of matrix-vector Hadamard product and matrix multiplication

[0194]

[0195] For the operation of summing the rows of a matrix, the technical solution provided by this application has an 80.8% improvement in throughput compared to the open source matrix-vector multiplication gemv interface, which clearly shows the performance benefits and can improve the computing power utilization of the matrix multiplication operation unit in practical applications. For the Hadamard product of matrices and vectors and matrix multiplication, in terms of throughput indicators, the technical solution provided by this application is 2.76 times and 3.65 times that of other solutions, respectively, which can also greatly improve the computing power utilization of the matrix multiplication operation unit.

[0196] 2. Implemented on Huawei Ascend 910 chip

[0197] On the Huawei Ascend 910 chip, the matrix row summation is implemented based on the technical solution provided by this application and the existing solution implemented on the vector computing unit. By changing the size of the matrix, the benefits of the conversion operation on different matrix sizes are determined. In addition, the optimized implementation of the matrix row summation is integrated into the reasoning of the transformer model such as Wukong Huahua to verify the performance improvement brought by end-to-end reasoning. Figures 9a to 9d In the above diagram, vector means that the vector operator is calculated directly on the vector computing unit without conversion, and cube means that the vector operator is converted into the corresponding matrix multiplication operator and calculated on the matrix computing unit.

[0198] First, the columns of the original matrix are fixed, and the benefit of converting the row summation into the corresponding matrix multiplication operation is determined by increasing the number of rows, such as Figure 9a shown. Figure 9a The horizontal axis is the matrix size, and the vertical axis is the time required to complete the calculation.

[0199] Secondly, the rows of the original matrix are fixed and the benefit of converting the matrix row summation into matrix multiplication can be determined by increasing the number of columns, such as Figure 9b shown. Figure 9b The horizontal axis is the matrix size, and the vertical axis is the time required to complete the calculation.

[0200] Furthermore, we increase the rows and columns of the original matrix in the same proportion to determine the benefit of converting the row summation into a matrix multiplication operation, such as Fig.9c shown. Fig.9c The horizontal axis is the matrix size, and the vertical axis is the time required to complete the calculation.

[0201] Finally, the optimization of matrix row summation is integrated into the end-to-end reasoning of the Wukong painting model to observe the end-to-end reasoning performance benefits brought by the optimization. The size of the accelerated matrix is ​​(1024, 64), as shown in Figure 9d shown. Figure 9d The vertical axis is the time required for end-to-end inference.

[0202] Compared with executing vector operators on the vector computing unit, in the single operator evaluation, as the size of the original matrix (the matrix in the vector operator) increases, the benefits of converting the matrix row-by-row sum into the corresponding matrix multiplication operator on the matrix computing unit are more obvious. For example, when the matrix size is (1024, 1024), the time taken to do the calculation on the vector computing unit is 3.54 times that of the time taken to do the calculation on the matrix computing unit. Applying the method provided in this application to the end-to-end reasoning of the Wukong painting model can reduce the reasoning delay by 46.8%, bringing significant benefits to actual business.

[0203] Combination of the above Figure 2 9 to 9 describe the data processing system and data processing method provided by the present application. Fig.11 , describes the device embodiment of the present application. It should be understood that the description of the data processing system and the data processing method corresponds to the description of the device embodiment, so the parts not described in detail can refer to the above description.

[0204] like Fig.10 As shown, Fig.10 A schematic diagram of the structure of a device provided in the present application. The device 1000 includes a first processing module 1001 and a second processing module 1002. The device 1000 can be a chip (such as an AI chip or a system on a chip), and can also be a terminal device, a module, a board, an intelligent station, a server or a server cluster. The first processing module 1001 and the second processing module 1002 can be the same module, for example, a CPU. The first processing module 1001 and the second processing module 1002 can also be different modules. For example, the first processing module is a CPU, and the second processing module is an AI chip.

[0205] The first processing module 1001 is used to determine whether the vector operator meets the conditions, and the vector operator is an operator executed by the vector calculation unit of the chip. The first processing module 1001 is used to convert the vector operator into a target matrix multiplication operator when the vector operator meets the conditions, so that the matrix calculation unit of the chip can execute the target matrix multiplication operator, and the first calculation result of the target matrix multiplication operator is the same as the second calculation result of the vector operator.

[0206] In a possible implementation, the first processing module 1001 is used to determine whether the vector operator is a target vector operator, the target vector operator is an operator that can be converted to a matrix multiplication operator, and the calculation result of the target vector operator is the same as the calculation result of the converted matrix multiplication operator. The first processing module 1001 is used to determine whether the vector operator meets the condition when the vector operator is the target vector operator.

[0207] In one possible implementation, the target vector operator includes at least one of matrix row summation, matrix column summation, matrix multiplication, vector outer product, and Hadamard product operation between a matrix and a vector.

[0208] In a possible implementation, the first processing module 1001 is used to determine whether the first cost is less than the second cost, the first cost is the time required for the predicted matrix calculation unit to execute the target matrix multiplication operator to complete the calculation, and the second cost is the time required for the predicted vector calculation unit to execute the vector operator to complete the calculation. The first processing module 1001 is used to determine that the vector operator meets the condition when the vector operator is the target vector operator and the first cost is less than the second cost.

[0209] In a possible implementation, the target matrix multiplication operator includes a construction matrix, which is a matrix constructed according to a vector operator so that the first calculation result is the same as the second calculation result. The chip includes a buffer connected to the matrix calculation unit, and the buffer is used to store a block matrix of the construction matrix. The second processing module 1002 is used to construct a target block matrix according to the size of the buffer and the vector operator, the size of the target block matrix is ​​less than or equal to the size of the buffer, the target block matrix is ​​a part of the construction matrix, and the values ​​in the target block matrix are not all 0.

[0210] like Fig.11 As shown, Fig.11A schematic diagram of the structure of another device provided by the present application. The device 1100 is applied to a chip, and the chip includes a vector calculation unit and a matrix calculation unit. The device 1100 includes a processing module 1101. The processing module 1101 is used to obtain a target vector operator. The processing module 1101 is used to convert the target vector operator into a target matrix multiplication operator so that the matrix calculation unit can execute the target matrix multiplication operator, and the first calculation result of the target matrix multiplication operator is the same as the second calculation result of the target vector operator.

[0211] In a possible implementation, the target matrix multiplication operator includes a construction matrix, which is a matrix constructed according to a vector operator so that the first calculation result is the same as the second calculation result. The chip includes a buffer connected to the matrix calculation unit, and the buffer is used to store a block matrix of the construction matrix. The processing module 1101 is used to construct a target block matrix according to the size of the buffer and the vector operator, the size of the target block matrix is ​​less than or equal to the size of the buffer, the target block matrix is ​​a part of the construction matrix, and the values ​​in the target block matrix are not all 0.

[0212] like Fig.12 As shown, Fig.12 A schematic diagram of the structure of another device provided in the present application. The device 1200 includes a processing module 1201. The device 1200 is, for example, a CPU. The processing module 1201 is used to determine whether a vector operator meets a condition, where the vector operator is an operator executed by a vector computing unit of a chip. The processing module 1201 is used to issue an instruction when the vector operator meets the condition, the instruction instructing the chip to convert the vector operator into a matrix multiplication operator.

[0213] In a possible implementation, processing module 1201 is used to determine whether the vector operator is a target vector operator, the target vector operator is an operator that can be converted into a matrix multiplication operator, and the calculation result of the target vector operator is the same as the calculation result of the converted matrix multiplication operator. Processing module 1201 is used to determine whether the vector operator meets the condition when the vector operator is the target vector operator.

[0214] In one possible implementation, the target vector operator includes at least one of matrix row summation, matrix column summation, matrix multiplication, vector outer product, and Hadamard product operation between a matrix and a vector.

[0215] In a possible implementation, processing module 1201 is used to determine whether a first cost is less than a second cost, the first cost being the time required for a predicted matrix calculation unit to execute a target matrix multiplication operator to complete a calculation, and the second cost being the time required for a predicted vector calculation unit to execute a vector operator to complete a calculation. Processing module 1201 is used to determine that the vector operator meets the condition when the vector operator is a target vector operator and the first cost is less than the second cost.

[0216] like Fig.13 As shown, Fig.13 A schematic diagram of the structure of an electronic device provided in the present application. The electronic device 1300 includes a processor 1301, the processor 1301 is coupled to a memory, and the processor 1301 is configured to execute the data processing method in any of the above embodiments based on instructions stored in the memory. The electronic device 1300 can be a chip (such as an AI chip or a system on a chip), and can also be a terminal device, a module, a board, an intelligent station, a server or a server cluster.

[0217] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a computer, the steps of the data processing method of any of the above method embodiments are implemented.

[0218] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0219] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical or other forms.

[0220] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0221] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0222] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the technical solution of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A data processing method, characterized in that: The method comprises: Determine whether a vector operator meets a condition, wherein the vector operator is an operator executed by a vector computing unit of the chip; If the vector operator meets the conditions, the vector operator is converted into a target matrix multiplication operator so that the matrix calculation unit of the chip can execute the target matrix multiplication operator, and the first calculation result of the target matrix multiplication operator is the same as the second calculation result of the vector operator.

2. The method according to claim 1, characterized in that The determining whether the vector operator meets the conditions includes: Determine whether the vector operator is a target vector operator, the target vector operator is an operator that can be converted into a matrix multiplication operator, and the calculation result of the target vector operator is the same as the calculation result of the converted matrix multiplication operator; If the vector operator is the target vector operator, it is determined that the vector operator meets the condition.

3. The method according to claim 2, characterized in that The determining whether the vector operator meets the conditions further includes: Determine whether the first cost is less than the second cost, the first cost being the predicted time required for the matrix calculation unit to execute the target matrix multiplication operator to complete the calculation, and the second cost being the predicted time required for the vector calculation unit to execute the vector operator to complete the calculation; If the vector operator is the target vector operator, determining that the vector operator meets the condition includes: If the vector operator is the target vector operator and the first cost is less than the second cost, it is determined that the vector operator meets the condition.

4. The method according to claim 2 or 3, characterized in that: The target vector operator includes at least one of matrix row summation, matrix column summation, matrix multiplication, vector outer product and Hadamard product operation between a matrix and a vector.

5. The method according to any one of claims 1 to 4, characterized in that The target matrix multiplication operator includes a construction matrix, which is a matrix constructed according to the vector operator so that the first calculation result is the same as the second calculation result.

6. The method according to claim 5, characterized in that The chip includes a buffer connected to the matrix calculation unit, the buffer is used to store the block matrix of the construction matrix, and the converting the vector operator into a target matrix multiplication operator includes: A target block matrix is ​​constructed according to the size of the buffer and the vector operator, the size of the target block matrix is ​​less than or equal to the size of the buffer, the target block matrix is ​​a part of the construction matrix, and the values ​​in the target block matrix are not all zero.

7. A data processing method, characterized in that: The method is applied to a chip, the chip includes a vector calculation unit and a matrix calculation unit, and the method includes: Get the target vector operator; Converting the target vector operator to a target matrix multiplication operator, wherein a first calculation result of the target matrix multiplication operator is the same as a second calculation result of the target vector operator; The target matrix multiplication operator is executed using the matrix computation unit.

8. The method according to claim 7, characterized in that The target matrix multiplication operator includes a construction matrix, which is a matrix constructed according to the target vector operator to make the first calculation result the same as the second calculation result.

9. The method according to claim 8, characterized in that The chip includes a buffer connected to the matrix calculation unit, the buffer is used to store the block matrix of the construction matrix, and the conversion of the target vector operator into a target matrix multiplication operator includes: A target block matrix is ​​constructed according to the size of the buffer and the target vector operator, the size of the target block matrix is ​​less than or equal to the size of the buffer, the target block matrix is ​​a part of the construction matrix, and the values ​​in the target block matrix are not all zero.

10. A data processing method, characterized in that: The method comprises: Determine whether a vector operator meets a condition, wherein the vector operator is an operator executed by a vector computing unit of the chip; If the vector operator meets the condition, an instruction is issued, wherein the instruction instructs the chip to convert the vector operator into a matrix multiplication operator.

11. The method according to claim 10, characterized in that The determining whether the vector operator meets the conditions includes: Determine whether the vector operator is a target vector operator, the target vector operator is an operator that can be converted into a matrix multiplication operator, and the calculation result of the target vector operator is the same as the calculation result of the converted matrix multiplication operator; If the vector operator is the target vector operator, it is determined that the vector operator meets the condition.

12. The method according to claim 11, characterized in that The determining whether the vector operator meets the conditions further includes: Determine whether the first cost is less than the second cost, the first cost being the predicted time required for the matrix calculation unit to execute the target matrix multiplication operator to complete the calculation, and the second cost being the predicted time required for the vector calculation unit to execute the vector operator to complete the calculation; If the vector operator is the target vector operator, determining that the vector operator meets the condition includes: If the vector operator is the target vector operator and the first cost is less than the second cost, it is determined that the vector operator meets the condition.

13. A device, characterized in that: The device comprises: A first processing module, used to determine whether a vector operator meets a condition, wherein the vector operator is an operator executed by a vector computing unit of the chip; The second processing module is used to convert the vector operator into a target matrix multiplication operator when the vector operator meets the conditions, so that the matrix calculation unit of the chip can execute the target matrix multiplication operator, and the first calculation result of the target matrix multiplication operator is the same as the second calculation result of the vector operator.

14. The device according to claim 13, characterized in that The first processing module is used to determine whether the vector operator is a target vector operator, the target vector operator is an operator that can be converted into a matrix multiplication operator, and the calculation result of the target vector operator is the same as the calculation result of the converted matrix multiplication operator; The first processing module is used to determine whether the vector operator meets the condition when the vector operator is the target vector operator.

15. The device according to claim 14, characterized in that The first processing module is used to determine whether a first cost is less than a second cost, the first cost being the predicted time required for the matrix calculation unit to execute the target matrix multiplication operator to complete the calculation, and the second cost being the predicted time required for the vector calculation unit to execute the vector operator to complete the calculation; The first processing module is used to determine that the vector operator meets the condition when the vector operator is the target vector operator and the first cost is less than the second cost.

16. The device according to any one of claims 13 to 15, characterized in that The target matrix multiplication operator includes a construction matrix, which is a matrix constructed according to the vector operator so that the first calculation result is the same as the second calculation result; the chip includes a buffer connected to the matrix calculation unit, and the buffer is used to store a block matrix of the construction matrix; The second processing module is used to construct a target block matrix according to the size of the buffer and the vector operator, the size of the target block matrix is ​​less than or equal to the size of the buffer, the target block matrix is ​​a part of the construction matrix, and the values ​​in the target block matrix are not all 0.

17. A device, characterized in that: The device is applied to a chip, the chip includes a vector calculation unit and a matrix calculation unit, and the device includes: A processing module, used for obtaining a target vector operator; The processing module is used to convert the target vector operator into a target matrix multiplication operator so that the matrix calculation unit can execute the target matrix multiplication operator, and a first calculation result of the target matrix multiplication operator is the same as a second calculation result of the target vector operator.

18. A device, characterized in that: The device comprises: A processing module, used to determine whether a vector operator meets the conditions, wherein the vector operator is an operator executed by a vector computing unit of the chip; The processing module is used to issue an instruction when the vector operator meets the condition, and the instruction instructs the chip to convert the vector operator into a matrix multiplication operator.

19. An electronic device, characterized in that: The electronic device comprises a processor, the processor is coupled to a memory, and the processor is configured to execute the data processing method according to any one of claims 1 to 12 based on instructions stored in the memory.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises instructions, which, when the computer-readable storage medium is executed on a computer, causes the computer to execute the data processing method according to any one of claims 1 to 12.

Images