Vector matrix calculation component, vector matrix calculation method and chip

Abstract

The invention discloses a vector matrix calculation component, a vector matrix calculation method and a chip, and belongs to the technical field of chips. The vector matrix calculation component comprises an input cache unit, i calculation processing units, i groups of first beat registers and an output cache unit, the input cache unit is used for storing one-dimensional vector data and two-dimensional matrix data; the i calculation processing units are connected with the corresponding group of first beat registers; the i groups of first beat registers are used for temporarily storing two-dimensional matrix data read according to a beat sequence; the i calculation processing units are used for sequentially transmitting one-dimensional vector data according to a beating sequence; and the i calculation processing units are also used for obtaining corresponding matrix data parts from a group of connected first beat registers, carrying out multiply-accumulate processing on the matrix data parts and the one-dimensional vector data, and inputting an obtained multiply-accumulate result into the output cache unit.

Description

Technical Field

[0001] The present application relates to the field of chip technology, and in particular to a vector matrix calculation component, a vector matrix calculation method and a chip. Background Art

[0002] Artificial Intelligence (AI) chips, also known as AI accelerators or computing cards, are modules specifically designed to handle large amounts of computing tasks in AI applications.

[0003] In the related technology, the AI ​​chip includes storage units corresponding to one-dimensional vectors and storage units corresponding to two-dimensional matrices, and the storage units corresponding to the one-dimensional vectors and the storage units corresponding to the two-dimensional matrices are respectively connected to several computing processing units. During the calculation process, the computing processing unit reads the one-dimensional vector and the corresponding data in the two-dimensional matrix from the two storage units respectively, multiplies and accumulates the two, and outputs them.

[0004] However, the AI ​​chip of the above solution will be limited by the timing bottleneck problem, which has a great impact on the main frequency design and has poor scalability. Summary of the invention

[0005] Each embodiment of the present application provides a vector matrix calculation component, a vector matrix calculation method and a chip, and the technical solution is as follows:

[0006] According to one aspect of the present application, a vector matrix calculation component is provided, the vector matrix calculation component comprising: an input cache unit, i calculation processing units, i groups of first beat registers and an output cache unit; the input cache unit is used to store one-dimensional vector data and two-dimensional matrix data; i is an integer greater than or equal to 2;

[0007] The i computing processing units correspond one-to-one to the i groups of first beat registers, and the computing processing units are connected to the corresponding group of first beat registers;

[0008] The first computing processing unit among the i computing processing units and the input buffer unit, and every two adjacent computing processing units among the i computing processing units are connected in sequence;

[0009] The i groups of first beat registers are respectively connected to the input buffer units;

[0010] The i computing processing units are respectively connected to the output buffer units;

[0011] The i groups of first beat registers are used to temporarily store the two-dimensional matrix data read in a beat order; each group of the i groups of first beat registers is used to temporarily store the matrix data portion of the two-dimensional matrix data corresponding to the connected calculation processing unit;

[0012] The i computing processing units are used to sequentially transmit the one-dimensional vector data according to the beat order;

[0013] The i computing processing units are also used to obtain corresponding matrix data parts from a connected group of first beat registers, perform multiplication and accumulation processing on the obtained matrix data parts and the one-dimensional vector data, and input the obtained multiplication and accumulation results into the output cache unit.

[0014] According to one aspect of the present application, a vector matrix calculation method is provided, the method being applied to the vector matrix calculation component as described above, the method comprising:

[0015] The i groups of first beat registers temporarily store the two-dimensional matrix data read in a beat order; each group of the i groups of first beat registers is used to temporarily store the matrix data portion of the two-dimensional matrix data corresponding to the connected calculation processing unit;

[0016] The i computing processing units obtain corresponding matrix data parts from a connected group of first beat registers, perform multiplication and accumulation processing on the obtained matrix data parts and the one-dimensional vector data, and input the obtained multiplication and accumulation results into the output cache unit.

[0017] In a possible implementation, each of the i computing processing units includes a cache register, j multiplication and accumulation unit groups, and a second beat register; j is an integer greater than or equal to 1;

[0018] The i computing processing units obtain corresponding matrix data parts from a connected group of first beat registers, perform multiplication and accumulation processing on the obtained matrix data parts and the one-dimensional vector data, and input the obtained multiplication and accumulation results into the output buffer unit, including:

[0019] The cache register obtains the corresponding matrix data portion from a connected group of first beat registers and caches it;

[0020] The multiplication-accumulation unit group performs multiplication-accumulation processing on the matrix data part stored in the cache register and the one-dimensional vector data, and inputs the obtained multiplication-accumulation result into the output cache unit;

[0021] The second beat register transfers the one-dimensional vector data in a beat order.

[0022] In a possible implementation, a group of first beat registers corresponding to the s-th computing processing unit among the i computing processing units includes s first beat registers, 1≤s≤i, and s is an integer;

[0023] When the value of s is greater than 1, the first one of the s first beat registers is connected to the input buffer unit, and each two adjacent first beat registers of the s first beat registers are connected in sequence, and the last one of the s first beat registers is connected to the sth calculation processing unit;

[0024] The cache register obtains and caches the corresponding matrix data portion from a connected group of first beat registers, including:

[0025] The cache register in the sth computing processing unit is used to obtain and cache the corresponding matrix data portion from the sth beat register among the s first beat registers.

[0026] In a possible implementation, the i groups of first beat registers temporarily storing the two-dimensional matrix data read in a beat order include:

[0027] The first of the s first beat registers, in one beat, when a matrix data portion of the two-dimensional matrix data is cached, transfers the cached matrix data portion to the second of the s first beat registers; the first of the s first beat registers, in one beat, when there is a matrix data portion corresponding to the s-th computing processing unit in the two-dimensional matrix data and has not been read by the first of the s first beat registers, reads a matrix data portion corresponding to the s-th computing processing unit from the two-dimensional matrix data and caches it;

[0028] The t-th first beat register among the s first beat registers, in a beat, when a matrix data portion of the two-dimensional matrix data is cached, transfers the cached matrix data portion to the t+1-th first beat register among the s first beat registers; the t-th first beat register among the s first beat registers, in a beat, when a matrix data portion exists in the t-1-th first beat register among the s first beat registers, obtains the matrix data portion transferred from the t-1-th first beat register among the s first beat registers and caches it; 2≤t≤s-1, and t is an integer;

[0029] In one beat, when the s-th first beat register among the s first beat registers has a matrix data portion in the s-1-th first beat register among the s first beat registers, the matrix data portion transferred from the s-1-th first beat register among the s first beat registers is obtained and cached.

[0030] In a possible implementation, the second beat registers in the i computing processing units constitute i second beat registers connected in sequence, and the first second beat register among the i second beat registers is connected to the input buffer unit;

[0031] The second beat register transmits the one-dimensional vector data according to the beat order, including:

[0032] The first second beat register of the i second beat registers, in one beat, when there is one-dimensional vector data cached, transfers the cached one-dimensional vector data to the second second beat register of the i second beat registers; the first first beat register, in one beat, when there is one-dimensional matrix data corresponding to the two-dimensional data matrix in the input buffer unit and not read by the first second beat register, reads the one-dimensional matrix data from the input buffer unit and caches it;

[0033] The qth second beat register among the i second beat registers, in one beat, when there is one-dimensional vector data cached, passes the cached one-dimensional vector data to the q+1th second beat register among the i first beat registers for caching; the qth beat register is also used to obtain and cache the one-dimensional vector data passed by the q-1th second beat register; 2≤q≤i-1, and q is an integer;

[0034] The i-th second beat register among the i second beat registers obtains and caches the one-dimensional vector data transmitted from the i-1-th second beat register in one beat.

[0035] In a possible implementation, the one-dimensional vector is a vector with a dimension of 1*k, and the two-dimensional matrix is ​​a matrix with a dimension of k*n; k and n are integers greater than or equal to 2;

[0036] Each of the matrix data parts in the two-dimensional matrix data includes a data part corresponding to one computing processing unit in r rows of data in the two-dimensional matrix data; 1≤r<n, and r is an integer;

[0037] The second beat register transmits the one-dimensional vector data according to the beat order, including:

[0038] After the corresponding cache register of the first second beat register completes caching of the entire matrix data portion of the two-dimensional matrix data corresponding to the first computing processing unit, in one beat, if there is one-dimensional matrix data corresponding to the two-dimensional data matrix in the input cache unit and has not been read by the first second beat register, the one-dimensional matrix data is read from the input cache unit and cached.

[0039] In a possible implementation manner, the cache register includes a first register and a second register; and the method further includes:

[0040] After the first register completes caching of all the matrix data parts corresponding to the calculation processing unit in the two-dimensional matrix data, the first register writes all the matrix data parts of the calculation processing unit into the second register, and obtains and caches the corresponding new matrix data parts from a group of connected first beat registers;

[0041] The multiplication-accumulation unit group performs multiplication-accumulation processing on the matrix data portion stored in the second register and the one-dimensional vector data, and inputs the obtained multiplication-accumulation result into the output buffer unit.

[0042] In one possible implementation, the one-dimensional vector data and the two-dimensional matrix data are data for performing vector-matrix multiplication operations during the operation of the AI ​​model.

[0043] On the other hand, a chip is provided, comprising at least one vector matrix calculation component as described above.

[0044] The beneficial effects of the technical solution provided by this application include at least:

[0045] The vector matrix calculation component includes an input cache unit, i calculation processing units, i groups of first beat registers and an output cache unit; the i calculation processing units are connected to the i groups of first beat registers in a one-to-one correspondence; the i calculation processing units are connected to the input cache unit in sequence; the i groups of first beat registers are connected to the input cache units respectively; the i groups of first beat registers temporarily store the matrix data parts corresponding to the connected calculation processing units in the two-dimensional matrix data read in the beat order; the i calculation processing units are used to transmit one-dimensional vector data in sequence in the beat order; the calculation processing unit obtains the corresponding matrix data part from the connected first beat register, performs multiplication and accumulation processing on the obtained matrix data part and the one-dimensional vector data, and inputs the obtained multiplication and accumulation result into the output cache unit; in the above-mentioned vector matrix calculation component, the matrix data part is transmitted to multiple calculation processing units in the beat order. One-dimensional vector data and two-dimensional matrix data are delivered. On the one hand, only one-dimensional vector data needs to be transferred to the adjacent computing processing unit in each beat. The main frequency design only needs to consider the time required for transferring one-dimensional vector data between two computing processing units. The fan-out in the data transmission process is small, the routing is close, and the impact on the chip main frequency is small. It can support high main frequency chip design, thereby expanding the adaptive main frequency range. On the other hand, through i groups of first beat registers, the two-dimensional matrix data is read from the unified output cache unit according to the beat order and loaded into the corresponding computing processing unit. There is no need to set an output cache unit for storing two-dimensional matrix data for each computing processing unit, which improves the flexibility of the circuit design between the computing processing unit and the output cache unit. In other words, the above-mentioned vector matrix calculation component improves the scalability from both the main frequency design and the circuit design. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a structural diagram of a vector-matrix multiplication operation component involved in this application;

[0047] Figure 2 is a schematic diagram of the structure of a vector matrix calculation component provided by an exemplary embodiment of the present application;

[0048] Figure 3 is a schematic diagram of the structure of a vector matrix calculation component provided by an exemplary embodiment of the present application;

[0049] Figure 4 It is a schematic diagram of the vector-matrix multiplication operation involved in this application;

[0050] Figure 5 It is the main structure diagram of the vector-matrix multiplication operation involved in this application;

[0051] Figure 6 yes Figure 3 A schematic diagram of the structure of another vector matrix calculation component involved in the illustrated embodiment;

[0052] Figure 7 is a structural diagram of each PE involved in this application;

[0053] Figure 8 It is a circuit structure diagram of a one-dimensional vector multiplication and accumulation calculation unit involved in the present application;

[0054] Fig. 9 is a schematic diagram of cache switching involved in this application;

[0055] Fig.10 It is a timing diagram of cache switching involved in this application;

[0056] Fig.11 It is a method flow chart of a vector matrix calculation method provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0058] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0059] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0060] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the object behaviors such as attack operations involved in this application are all obtained with full authorization.

[0061] It should be understood that although the terms first, second, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first parameter may also be referred to as the second parameter, and similarly, the second parameter may also be referred to as the first parameter. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0062] For ease of understanding, several terms involved in this application are explained below.

[0063] 1) Artificial Intelligence AI is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0064] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0065] 2) Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0066] In the current mainstream computing models in the field of artificial intelligence, most of the computing can be achieved through vector-matrix multiplication operations. Therefore, in the design of artificial intelligence cloud inference / training processor chips, the vector-matrix multiplication unit plays a very important role in the entire AI processor. In a sense, the design area, power consumption and performance of the vector-matrix multiplication unit determine the success or failure of the AI ​​processor. Therefore, how to design an efficient and feasible vector-matrix operation solution is significantly more important.

[0067] Please refer to Figure 1 , which shows a structural diagram of a vector-matrix multiplication operation component involved in the present application, such as Figure 1 As shown, to complete the multiplication operation of one-dimensional vector and two-dimensional matrix, there are usually two independent input cache units, of which DIN_MEM_A is used to cache the input one-dimensional vector, and DIN_MEM_B is used to cache the two-dimensional input matrix data. DOUT_MEM is the calculation result output cache unit. PE is a one-dimensional vector multiplication and accumulation calculation unit. When calculating the vector matrix, PE0~PE15 needs to read 1*k one-dimensional vector data from DIM_MEM_A and k*n two-dimensional matrix data from DIN_MEM_B at the same time in each beat. After all PEs read the corresponding data, they start calculating at the same time. After n beats of calculation, the 1*n one-dimensional vector calculation result is written into DOUT_MEM for cache processing. Among them, DIN_MEM_B needs to be divided into 16 groups of independent small cache units, each independent small cache unit corresponds to a PE calculation unit, so as to ensure that k*n two-dimensional matrix data is read from DIN_MEM_B at the same time every time.

[0068] However, the above Figure 1 In the scheme shown, when the computational scale of the multiplication operation of a one-dimensional vector and a two-dimensional matrix is ​​further increased, especially when the number of columns of the input two-dimensional matrix is ​​increased, there are some problems in scalability and feasibility, which are mainly reflected in the following aspects:

[0069] 1) DIN_MEM_A and DIM_MEM_B cannot share a unified cache, and the cache space must be strictly divided in advance. When the number of PEs is adjusted, the division units of DIM_MEM_B must also be reallocated and adjusted, so the scalability of the solution is poor.

[0070] 2) The 1*k vector data read from DIN_MEM_A in each beat needs to be transmitted to all PE computing units at the same time. When the number of PE computing units is large, there are problems of excessive fan-out of 1*k vector matrix data and long routing distance, which leads to timing bottleneck problems and affects the design main frequency.

[0071] 3) When the multiplication of a one-dimensional vector and a two-dimensional matrix is ​​calculated continuously for multiple times and the two-dimensional matrix data remains unchanged and only the one-dimensional vector is updated, the same two-dimensional matrix data still needs to be read from DIN_MEM_B and transmitted to the PE computing unit in each time, resulting in a problem of high dynamic power consumption.

[0072] The solution shown in the subsequent embodiments of this application provides a highly scalable vector-matrix multiplication operation solution, which can provide scalability, effectively reduce the power consumption and area of ​​the AI ​​chip, and effectively improve the design main frequency of the AI ​​processor.

[0073] Figure 2 FIG. 4 shows a schematic diagram of the structure of a vector matrix calculation component provided by an exemplary embodiment of the present application. Figure 2 As shown, the vector matrix calculation component includes:

[0074] Input buffer unit 210 ( Figure 2 DIN_MEM), i computing processing units 220 ( Figure 2 PE in the figure), i group first beat register 230 ( Figure 2 denoted by DFF in FIG) and the output buffer unit 240 ( Figure 2 MEM in the figure).

[0075] The input buffer unit 210 is used to store one-dimensional vector data and two-dimensional matrix data; i is an integer greater than or equal to 2.

[0076] That is to say, in the scheme shown in the embodiment of the present application, one-dimensional vector data and two-dimensional matrix data share storage space, that is, one-dimensional vector data and two-dimensional matrix data are stored in the same cache unit (ie, the above-mentioned input cache unit 210).

[0077] There is a one-to-one correspondence between i computing units 220 and i groups of first beat registers 230 , and the computing unit 220 and the corresponding group of first beat registers 230 are connected.

[0078] That is to say, in the above-mentioned vector matrix calculation component, each calculation processing unit 220 is connected to a group of first beat registers 230 , and different calculation processing units 220 are connected to different groups of first beat registers 230 .

[0079] The first computing unit 220 among the i computing units 220 and the input buffer unit 210 , and every two adjacent computing units 220 among the i computing units 220 are connected in sequence.

[0080] In the embodiment of the present application, the input cache unit 210 and i computing processing units 220 are connected in sequence; specifically, starting from the input cache unit 210, the input cache unit 210 is connected to the first computing processing unit 220, the first computing processing unit 220 is connected to the second computing processing unit 220, and so on, until the i-1th computing processing unit 220 is connected to the i-th computing processing unit 220; that is, for the input cache unit 210 and i computing processing units, starting from the input cache unit 210, two adjacent units are connected in sequence.

[0081] The i groups of first beat registers 230 are respectively connected to the input buffer unit 210 .

[0082] In an embodiment of the present application, each group of first beat registers 230 in the above-mentioned i groups of first beat registers 230 is respectively connected to the input cache unit 210, that is, each group of first beat registers 230 has an independent channel between the input cache unit 210, so that each group of first beat registers 230 can read data directly from the input cache unit 210.

[0083] Among the above i groups of first beat registers 230 , each group of first beat registers 230 includes one or more first beat registers, and different groups of first beat registers 230 may include different numbers of first beat registers.

[0084] The first beat register can perform a data temporary storage operation once in each beat, and perform an operation of transferring data to the next register or computing processing unit once.

[0085] The i computing processing units 220 are respectively connected to the output buffer units 240 .

[0086] The i computing processing units 220 share one output buffer. That is, each computing processing unit 220 can independently write data to the output buffer unit 240, and the i computing processing units 220 can write data to the output buffer unit 240 at different times.

[0087] Among them, the above-mentioned i groups of first beat registers 230 are used to temporarily store the two-dimensional matrix data read according to the beat order; and each group of first beat registers in the i groups of first beat registers is used to temporarily store the matrix data part corresponding to the connected computing processing unit in the two-dimensional matrix data.

[0088] In the embodiment of the present application, for i computing processing units 220, the i computing processing units 220 are respectively used to multiply and accumulate a part of the data in the two-dimensional matrix data with the one-dimensional vector data. Accordingly, the i computing processing units 220 respectively correspond to different data parts in the two-dimensional matrix data.

[0089] For example, taking a two-dimensional matrix data of k*n as an example, the two-dimensional matrix data contains n columns of data, each column of data contains data of k dimensions, and the n columns of data are divided into i portions of data, each portion of data corresponds to one or more columns of the n columns of data. For each computing processing unit, the computing processing unit corresponds to one of the i portions of data, and during the calculation process, the one portion of data corresponding to the computing processing unit will be read in multiple beats according to the beat order and temporarily stored in a group of first beat registers connected to the computing processing unit. That is to say, the one portion of data corresponding to the computing processing unit can be divided into multiple matrix data parts, and the multiple matrix data parts are respectively read in multiple beats and temporarily stored in a group of first beat registers corresponding to the computing processing unit.

[0090] i computing processing units 220 are used to sequentially transmit one-dimensional vector data according to the beat order.

[0091] The i computing processing units 220 are also used to obtain the corresponding matrix data part from the connected first beat register 230, perform multiplication and accumulation processing on the obtained matrix data part and the one-dimensional vector data, and input the obtained multiplication and accumulation result into the output cache unit.

[0092] In an embodiment of the present application, each matrix data part in the two-dimensional matrix data is read in sequence according to the beat order and temporarily stored in each group of first beat registers 230. Correspondingly, the computing processing unit 220 can read the temporarily stored matrix data part from a correspondingly connected group of beat registers 230, and one-dimensional vector data is also transmitted between the computing processing units 220, so that each computing processing unit 220 can perform multiplication and accumulation processing on the corresponding matrix data part and the one-dimensional vector data, thereby realizing vector-matrix multiplication calculation.

[0093] To summarize, the vector matrix calculation component involved in the embodiment of the present application includes an input cache unit, i calculation processing units, i groups of first beat registers and an output cache unit; the i calculation processing units are connected to the i groups of first beat registers in a one-to-one correspondence; the i calculation processing units are connected to the input cache units in sequence; the i groups of first beat registers are connected to the input cache units respectively; the i groups of first beat registers temporarily store the matrix data parts corresponding to the connected calculation processing units in the two-dimensional matrix data read in the beat order; the i calculation processing units are used to transmit one-dimensional vector data in sequence in the beat order; the calculation processing unit obtains the corresponding matrix data part from the connected first beat register, performs multiplication and accumulation processing on the obtained matrix data part and the one-dimensional vector data, and inputs the obtained multiplication and accumulation result into the output cache unit; in the above-mentioned vector matrix calculation component, multiple The computing processing unit transmits one-dimensional vector data and two-dimensional matrix data. On the one hand, only one-dimensional vector data needs to be transmitted to the adjacent computing processing unit in each beat. The main frequency design only needs to consider the time required to transmit one-dimensional vector data between two computing processing units. The fan-out during data transmission is small and the routing is close, which has little impact on the chip main frequency and can support high main frequency chip design, thereby expanding the adaptive main frequency range. On the other hand, through the i groups of first beat registers, the two-dimensional matrix data is read from the unified output cache unit according to the beat order and loaded into the corresponding computing processing unit. There is no need to set an output cache unit for storing two-dimensional matrix data for each computing processing unit, which improves the flexibility of the circuit design between the computing processing unit and the output cache unit. In other words, the above-mentioned vector matrix calculation component improves the scalability from both the main frequency design and the circuit design.

[0094] based on Figure 2 For the vector matrix calculation components shown, please refer to Figure 3 , which shows a schematic diagram of the structure of a vector matrix calculation component provided by an exemplary embodiment of the present application, such as Figure 3 As shown, the vector matrix calculation component includes:

[0095] An input buffer unit 310 , i computing processing units 320 , i groups of first beat registers 330 and an output buffer unit 340 .

[0096] The input buffer unit 310 is used to store one-dimensional vector data and two-dimensional matrix data; i is an integer greater than or equal to 2.

[0097] In some embodiments, one-dimensional vector data and two-dimensional matrix data are data for performing vector-matrix multiplication operations during the operation of the AI ​​model.

[0098] There is a one-to-one correspondence between i computing units 320 and i groups of first beat registers 330 , and the computing unit 320 and the corresponding group of first beat registers 330 are connected.

[0099] The first computing processing unit 320 among the i computing processing units 320 and the input buffer unit 310 , and every two adjacent computing processing units 320 among the i computing processing units 320 are connected in sequence.

[0100] The i groups of first beat registers 330 are respectively connected to the input buffer units 310 .

[0101] The i computing processing units 320 are connected to the output buffer units 340 respectively.

[0102] i computing processing units 320 are used to sequentially transmit one-dimensional vector data according to the beat order.

[0103] The I group of first beat registers 330 is used to temporarily store the two-dimensional matrix data read in the beat order; and each group of first beat registers in the i groups of first beat registers is used to temporarily store the matrix data part corresponding to the connected computing processing unit in the two-dimensional matrix data.

[0104] The i computing processing units 320 are also used to obtain the corresponding matrix data part from the connected first beat register 330, perform multiplication and accumulation processing on the obtained matrix data part and the one-dimensional vector data, and input the obtained multiplication and accumulation result into the output cache unit.

[0105] The vector-matrix multiplication operation refers to the operation of multiplying a one-dimensional vector and a two-dimensional matrix to obtain a one-dimensional vector. Figure 4 , which shows a schematic diagram of the vector-matrix multiplication operation involved in this application, such as Figure 4 As shown, the input is usually a 1*k dimensional one-dimensional vector and a k*n dimensional two-dimensional matrix. The 1*k dimensional vector is multiplied and added with the data of each column vector in the k*n two-dimensional matrix to obtain a 1*n dimensional one-dimensional vector result. In the above operation process, the 1*k dimensional vector is reused n times. Figure 4 In this example, k=64 and n=64.

[0106] Please refer to Figure 5 , which shows the main architecture diagram of the vector-matrix multiplication operation involved in this application. Figure 5As shown, DIN_MEM is an input unified cache unit (i.e., the above-mentioned input cache unit), which is used to cache input data of one-dimensional vectors of 1*k dimension and two-dimensional matrices of k*n dimension. DOUT_MEM is an output cache unit of one-dimensional vectors of 1*n dimension. PE0~PE15 are vector multiplication and accumulation calculation units (i.e., the above-mentioned calculation processing units). WPR is an internal cache unit of a two-dimensional matrix of k*n dimension. DFF is a beat register. Among them, each PE corresponds to a set of DFFs (i.e., the above-mentioned first beat register), and the number of DFFs in a set of DFFs corresponding to each PE may be different. Taking the two-dimensional matrix data of k*n dimension, which contains 64 columns (i.e., n=64) of matrix data (represented by B0~B63 respectively), as an example, there are 16 PEs, and each PE corresponds to 4 columns of matrix data, as shown in the following figure. Figure 5 As shown, a group of DFFs of PE0 is used to read and temporarily store B0-B3, a group of DFFs of PE1 is used to read and temporarily store B3-B7, a group of DFFs of PE2 is used to read and temporarily store B8-B11, and so on, a group of DFFs of PE15 is used to read and temporarily store B60-B63. At the same time, the one-dimensional vector data contains 64 vector elements (i.e., k=64), which are represented by A0-A63 respectively. The above one-dimensional vector data A0-A63 is first read by PE0, and then transferred among PE1, PE2, PE3, etc. in the beat order until it is transferred to PE15. In addition, each PE in the above PE0-PE5 can perform multiplication and accumulation calculations on the 4-column matrix data obtained by the corresponding group of DFFs and the one-dimensional vector data transferred from DIN_MEM or the previous PE, and output the calculation results to DOUT_MEM.

[0107] like Figure 3 As shown, in some embodiments, each of the i computing processing units 320 includes a cache register 321 ( Figure 3 WPR), j multiplication and accumulation unit groups 322 and a second beat register 323 ( Figure 3 DFF in the table); j is an integer greater than or equal to 1;

[0108] A cache register 321, used for obtaining and caching a corresponding matrix data portion from a connected set of first beat registers;

[0109] The multiplication-accumulation unit group 322 is used to perform multiplication-accumulation processing on the matrix data part stored in the cache register and the one-dimensional vector data, and input the obtained multiplication-accumulation result into the output cache unit;

[0110] The second beat register 323 is used to transfer one-dimensional vector data according to the beat order.

[0111] like Figure 3As shown, the matrix data portion temporarily stored in a group of first beat registers corresponding to each PE can be loaded into the cache register 321 in the corresponding PE, and the one-dimensional vector data transferred from DIN_MEM or the previous PE is loaded into the second beat register 323 in the PE. Afterwards, in one beat, the multiplication and accumulation unit group 322 can perform multiplication and accumulation calculations on the matrix data portion cached in the cache register 321 and the one-dimensional vector data cached in the second beat register 323, and the calculation results can be written into the output cache unit.

[0112] Through the above Figure 3 In the scheme shown, each computing processing unit is provided with a second beat register, through which one-dimensional vector data can be temporarily stored and transferred backwards, thereby supporting the transfer of one-dimensional vector data between each computing processing unit at the hardware level. The main frequency design only needs to consider the time required for transferring one-dimensional vector data between two computing processing units. The fan-out during data transfer is small and the routing is closer, which has little impact on the chip main frequency and can support high main frequency chip design, thereby expanding the adaptive main frequency range.

[0113] In addition, in the scheme shown in the embodiment of the present application, a cache register can be set in the computing processing unit. In the scenario where the two-dimensional matrix data remains unchanged and the one-dimensional vector data changes, there is no need to re-read and load the matrix data part of the two-dimensional matrix data through the first beat register, which can reduce the dynamic power consumption of the entire vector matrix computing component and improve computing efficiency.

[0114] At the same time, since the i groups of first beat registers read and temporarily store the matrix data part of the two-dimensional matrix data in the beat order, that is to say, for a group of first beat registers corresponding to a computing processing unit, it can obtain and calculate the matrix data required by the processing unit through multiple beats, and the scheme of setting a cache register in the computing processing unit can reduce the capacity requirement of the first beat register.

[0115] In another possible implementation, the above-mentioned second beat register 323 can also be set outside the computing processing unit 320, that is, i second beat registers 323 are connected in sequence, and the first second beat register 323 is connected to the input cache unit, and each second beat register 323 among the i second beat registers 323 is connected to a computing processing unit 320.

[0116] based on Figure 3 In the scheme shown, in some embodiments, a set of first beat registers 330 corresponding to the s-th computing processing unit among the i computing processing units 320 includes s first beat registers, 1≤s≤i, and s is an integer;

[0117] When the value of s is greater than 1, the first first beat register among the s first beat registers is connected to the input buffer unit, and each two adjacent first beat registers among the s first beat registers are connected in sequence, and the last first beat register among the s first beat registers is connected to the sth calculation processing unit;

[0118] The cache register in the sth computing processing unit is used to obtain and cache the corresponding matrix data portion from the sth beat register among the s first beat registers.

[0119] based on Figure 3 For the scheme shown, please refer to Figure 6 , which shows a schematic diagram of the structure of another vector matrix calculation component involved in an embodiment of the present application. Figure 6 As shown, the first computing unit 320 corresponds to 1 first beat register, the second computing unit 320 corresponds to 2 first beat registers, the third computing unit 320 corresponds to 3 beat registers, and so on, the i-th computing unit 320 corresponds to i first beat registers. Among them, the cache register in each computing unit 320 is connected to the last first beat register in the corresponding group of first beat registers, and the first first beat register in each group of first beat registers is connected to DIN_MEM. In the process of loading the matrix data part, when the last first beat register in the group of first beat registers corresponding to the computing unit 320 temporarily stores the matrix data part in one beat, the matrix data part temporarily stored in the last first beat register can be loaded into the cache register in the computing unit 320 in the current beat or the next beat.

[0120] pass Figure 6 The vector matrix calculation component shown, for each calculation processing unit corresponding to the multiple first beat registers, the multiple first beat registers are connected in series and connected to DIN_MEM and the cache register in the calculation processing unit. There is no need to connect each first beat register to DIN_MEM and the cache register in the calculation processing unit respectively. In other words, each first beat register only needs to be set with one entrance and one exit, and the cache register also only needs to be set with one entrance. While meeting the requirement of temporarily storing the matrix data part according to the beat, it can reduce the wiring difficulty and the interface complexity of the components and reduce the area occupied by the vector matrix calculation component.

[0121] In some embodiments, the first first beat register among the s first beat registers is used to, in one beat, transfer the cached matrix data portion to the second first beat register among the s first beat registers when a matrix data portion of the two-dimensional matrix data is cached; the first first beat register among the s first beat registers is also used to, in one beat, read a matrix data portion corresponding to the s-th computing processing unit from the two-dimensional matrix data and cache it when there is a matrix data portion corresponding to the s-th computing processing unit in the two-dimensional matrix data and has not been read by the first first beat register among the s first beat registers;

[0122] The t-th first beat register among the s first beat registers is used to, in one beat, transfer the cached matrix data portion to the t+1-th first beat register among the s first beat registers when a matrix data portion of the two-dimensional matrix data is cached; the t-th first beat register among the s first beat registers is also used to, in one beat, obtain and cache the matrix data portion transferred from the t-1-th first beat register among the s first beat registers when a matrix data portion exists in the t-1-th first beat register among the s first beat registers; 2≤t≤s-1, and t is an integer;

[0123] The sth first beat register among the s first beat registers is used to obtain and cache the matrix data part transferred from the s-1th first beat register among the s first beat registers in one beat when there is a matrix data part in the s-1th first beat register among the s first beat registers.

[0124] In the above embodiment, during the process of reading the matrix data portion from DIN_MEM and temporarily storing it in the first beat register, in the first beat, a matrix data portion corresponding to each computing processing unit can be read from DIN_MEM and temporarily stored in the first beat register in a group of first beat registers corresponding to each computing processing unit. For the first beat register of the first computing processing unit, the temporarily stored matrix data portion can be loaded into the cache register of the first computing processing unit in the current beat; in the second beat, another matrix data portion corresponding to each computing processing unit can be read from DIN_MEM and temporarily stored in the first beat register in a group of first beat registers corresponding to each computing processing unit. A beat register, for the first beat register of the first computing processing unit, it can load the temporarily stored matrix data part into the cache register of the first computing processing unit in the current beat, for other computing processing units other than the first computing processing unit, the first first beat register of the computing processing unit transfers the temporarily stored matrix data part in the previous beat to the second beat register for temporary storage, and so on. For each computing processing unit, in a certain beat, if the last first beat register of the computing processing unit temporarily stores a new matrix data part, then in the current beat, the matrix data part in the last first beat register can be loaded into the cache register of the computing processing unit.

[0125] It can be seen from the above-mentioned beat control logic that, among i computing processing units, the progress of loading the matrix data part of the latter computing unit is always one beat slower than the progress of loading the matrix data part of the former computing processing unit. The above-mentioned process of loading data in beat order can improve the efficiency of data loading; specifically, the chip usually contains one or more vector matrix computing components, each of which contains multiple computing processing units. The computing processing unit needs to be started before working, and the startup process of the computing processing unit has certain requirements on voltage or power consumption, which makes it impossible for the vector matrix computing component to start multiple or all computing processing units at the same time. In addition, due to the difference in routing distance between multiple computing processing units and the control unit in the chip, multiple computing processing units cannot be achieved. Therefore, in the vector matrix computing component, each computing processing unit will be started one by one according to the distance between it and DIN_MEM, and one computing processing unit will be started for each beat. That is to say, when the first computing processing unit is started, the second computing processing unit has not yet been started. At this time, if data cannot be loaded into the second computing processing unit, if the two-dimensional matrix data is read and recorded after waiting for all the computing processing units to be successfully started, it is necessary to wait for i beats first, that is, it can only passively wait in these i beats; and through the scheme shown in the embodiment of the present application, in the process of each computing processing unit being started one by one according to the beat, the matrix data part can be read from DIN_MEM and temporarily stored in a corresponding set of first beat registers. At the same time, since the number of each set of first beat registers matches the startup order of the computing processing unit, therefore, when the corresponding computing processing unit is successfully started, the corresponding first beat register can immediately load the matrix data part to the computing processing unit, so that the reading and loading of the two-dimensional matrix data can be started in the beat order during the process of starting the computing processing unit, thereby improving the efficiency of data loading.

[0126] Specifically, in the first beat, the first computing processing unit is started up. At this time, a matrix data part is temporarily stored in the first beat register corresponding to the first computing processing unit. Therefore, in the first beat, the first computing processing unit can immediately load the matrix data part, and for other computing processing units, their respective first first beat registers temporarily store the matrix data part; in the second beat, the matrix data part is updated in the first beat register corresponding to the first computing processing unit. In the second beat, the first computing processing unit can immediately load the matrix data part, and the first first beat registers corresponding to the other computing processing units update the matrix data temporarily stored in the previous beat. The data part is transferred to the second first beat register, and the new matrix data part is read and temporarily stored at the same time. At this time, the second calculation processing unit is also started. Therefore, the second calculation processing unit can load the corresponding matrix data part in the second first beat register in the second beat. Similarly, in the ith beat, the ith calculation processing unit is also started and completed, and the last first beat register of the ith calculation processing unit just temporarily stores the matrix data part. The ith calculation processing unit can directly load the matrix data part; all matrix data parts in the two-dimensional matrix data are read in sequence according to the above-mentioned beat order and temporarily stored in the first beat register, and then loaded into the corresponding calculation processing unit.

[0127] In some embodiments, the second beat registers in the i computing processing units constitute i second beat registers connected in sequence, and the first second beat register among the i second beat registers is connected to the input buffer unit;

[0128] The first second beat register of the i second beat registers is used to transfer the cached one-dimensional vector data to the second second beat register of the i second beat registers in one beat when there is one-dimensional vector data cached; the first first beat register is also used to read the one-dimensional matrix data from the input buffer unit and cache it when there is one-dimensional matrix data corresponding to the two-dimensional data matrix in the input buffer unit and not read by the first second beat register in one beat;

[0129] The qth second beat register among the i second beat registers is used to transfer the cached one-dimensional vector data to the q+1th second beat register among the i first beat registers for caching in one beat when there is one-dimensional vector data cached; the qth beat register is also used to obtain and cache the one-dimensional vector data transferred from the q-1th second beat register; 2≤q≤i-1, and q is an integer;

[0130] The i-th second beat register among the i-th second beat registers is used to obtain and cache the one-dimensional vector data transmitted from the i-1-th second beat register in one beat.

[0131] In the above embodiment, one-dimensional vector data is read from DIN_MEM and temporarily stored in the first computing processing unit, and is transmitted to subsequent computing processing units in sequence according to the beat order. In each beat, the one-dimensional vector data can be transmitted backward once. For example, in the first beat, the one-dimensional vector data can be read from DIN_MEM and temporarily stored in the second beat register corresponding to the first computing processing unit. In the second beat, the second beat register in the first computing processing unit transmits the one-dimensional vector data temporarily stored in the first beat to the second beat register in the second computing processing unit for temporary storage. In three beats, the second beat register in the second computing processing unit passes the one-dimensional vector data temporarily stored in the second beat to the second beat register in the third computing processing unit for temporary storage, and so on, until the i-th beat, the second beat register in the i-1-th computing processing unit passes the one-dimensional vector data temporarily stored in the i-1-th beat to the second beat register in the i-th computing processing unit for temporary storage; in the above process, each time the one-dimensional vector data is transmitted, it is transmitted between two adjacent computing processing units, the transmission distance is shorter, and the required time is also shorter, which can support the design of a higher chip main frequency.

[0132] Among them, for the first computing processing unit, it may take multiple beats for the computing processing unit to load all the required matrix data parts, which depends on the data amount of the two-dimensional matrix data, while the first computing processing unit only needs one beat to obtain the one-dimensional vector data. Therefore, the time for each computing processing unit to start transmitting one-dimensional vector data only needs to be no later than the time when the first computing processing unit completely loads all the matrix data parts it needs.

[0133] For example, assuming that a computing processing unit requires a matrix data portion that is divided into 10 matrix data parts, in each beat, a set of first beat registers of each computing processing unit can read a matrix data portion from DIN_MEM and temporarily store it, which means that, starting from the start of reading the two-dimensional matrix data, the first computing processing unit needs 10 beats to load all the required matrix data parts, the second computing processing unit needs 11 beats to load all the required matrix data parts, and so on. Therefore, starting from any beat of the 10 beats after the start of reading the two-dimensional matrix data and temporarily storing it in the first computing processing unit will not affect the computing efficiency.

[0134] In some embodiments, the one-dimensional vector data is a vector of dimension 1*k, and the two-dimensional matrix data is a matrix of dimension k*n; k and n are integers greater than or equal to 2;

[0135] Each matrix data portion in the two-dimensional matrix data includes a data portion corresponding to one computing processing unit in r rows of data in the two-dimensional matrix data; 1≤r<n, and r is an integer;

[0136] The first second beat register is used to read and cache one-dimensional vector data from the input cache unit in a beat after the corresponding cache register completes caching of the entire matrix data portion corresponding to the first computing processing unit in the two-dimensional matrix data, if there is one-dimensional vector data corresponding to the two-dimensional data matrix in the input cache unit and has not been read by the first second beat register.

[0137] In the embodiment of the present application, the first second beat register can start reading the one-dimensional vector data and pass it backward when the cache register of the first computing processing unit completes the cache of all the matrix data parts of the first computing processing unit. Of course, the first second beat register can also read the one-dimensional vector data in advance and pass it backward.

[0138] In some embodiments, the cache register includes a first register and a second register;

[0139] The first register is used to write all the matrix data parts of the computing processing unit into the second register after completing the caching of all the matrix data parts of the computing processing unit in the two-dimensional matrix data, and obtain the corresponding new matrix data parts from a connected group of first beat registers and cache them;

[0140] The multiplication-accumulation unit group is used to perform multiplication-accumulation processing on the matrix data part stored in the second register and the one-dimensional vector data, and input the obtained multiplication-accumulation result into the output buffer unit.

[0141] In an embodiment of the present application, since among i computing processing units, the progress of loading the matrix data part by the latter computing unit is always one beat slower than the progress of loading the matrix data part by the former computing processing unit, accordingly, when the former computing processing unit has loaded all the matrix data parts required in the two-dimensional matrix data to be processed this time and completed the calculation, the latter computing processing unit has not yet loaded all the matrix data parts required in the two-dimensional matrix data to be processed this time. If the former computing processing unit has been waiting for the latter computing processing unit to load and calculate data, it will lead to a waste of processing time and processing resources. In this regard, in an embodiment of the present application, two registers can be set in the cache register. During the calculation process after one of the registers is loaded, if it is necessary to switch to calculate other two-dimensional matrix data later, the other two-dimensional matrix data can be loaded in a beating manner through another register. When the computing processing unit completes the current calculation, it can quickly start the calculation of the next two-dimensional matrix data by switching registers. This scheme can improve the computing performance when the two-dimensional matrix data changes but the one-dimensional vector data remains unchanged.

[0142] Please refer to Figure 7 , which shows the structural diagram of each PE involved in this application; Figure 7 As shown, each PE contains four independent one-dimensional vector multiplication and accumulation calculation units. Each one-dimensional vector multiplication and accumulation calculation unit completes the multiplication and accumulation operation of a one-dimensional vector of 1*k dimension and one column vector in a two-dimensional matrix of k*n dimension, that is, each PE can complete the multiplication and accumulation calculation of a one-dimensional vector of 1*k dimension and a two-dimensional matrix of k*4 dimension. Each one-dimensional vector multiplication and accumulation calculation unit contains 64 independent MAC calculation units. Each MAC calculation unit completes the multiplication operation of a pair of input data, where one input of the multiplication operation comes from the one-dimensional vector of 1*k dimension read by DIN_MEM, and the other input comes from the WPR cache module. Each one-dimensional vector multiplication and accumulation calculation unit has a corresponding set of WPR two-dimensional matrix cache units. The WPR two-dimensional matrix cache unit has two sets of cache units with the same capacity, WPR1 and WPR0, and the total cache capacity of WPR1 and WPR0 is equal to the size of the k*n two-dimensional matrix. The DFF module in the PE unit is used to beat the input one-dimensional vector of 1*k dimension and transmit it to the next PE calculation unit after pulsation.

[0143] Please refer to Figure 8 , which shows a circuit structure diagram of a one-dimensional vector multiplication and accumulation calculation unit involved in the present application. Based on timing considerations, it is difficult to achieve the accumulation of 64 multiplication results in one beat. For this, Figure 8As shown, the first stage completes the parallel multiplication calculation of 64 MACs to obtain 64 multiplication result values; the second stage accumulates the calculation results of 64 groups of MACs according to the MAC numbers from low to high, with each group of 8 MACs to obtain 8 accumulated result values; the third stage further accumulates the 8 accumulated result values ​​obtained in the second stage to obtain the final calculation result value.

[0144] Take the multiplication process of a 1*64 dimensional one-dimensional vector and a 64*64 dimensional two-dimensional matrix as an example. Figure 5 The structure shown in the figure, the calculation process can be described as follows:

[0145] Step S1: Load a 64*64 dimensional two-dimensional matrix data from DIN_MEM to the WPR unit inside the PE. Since the data reading bandwidth of DIN_MEM is limited at one time, it is impossible to read the 64*64 dimensional two-dimensional matrix data from DIN_MEM at one time, so it needs to be loaded in multiple beats. Taking reading 2*64 dimensional data from DIN_MEM as an example, a total of 32 beats are required to complete the reading operation.

[0146] The 2*64 dimensional data read in the first shot are {B00, B10, B20, B30, ..., B620, B630} and {B01, B11, B21, B31, ..., B621, B631}.

[0147] The read {B00,B01},{B10,B11},{B20,B21},{B30,B31} are reorganized in this format and loaded into the WPR unit inside PE0 for cache processing after being beaten by the 1st level DFF. The read {B40,B41},{B50,B51},{B60,B61},{B70,B71} are reorganized in this format and loaded into the WPR unit inside PE1 for cache processing after being beaten by the 2nd level DFF. Similarly, the read {B600,B601},{B610,B611},{B620,B621},{B630,B631} are reorganized in this format and loaded into the WPR unit inside PE15 for cache processing after being beaten by the 16th level DFF. It can be seen that for a load of 2*64 dimensional data, the 2*4 dimensional data loaded into the WPR unit inside PEn is one beat later than the 2*4 dimensional data loaded into the WPR unit inside PEn-1.

[0148] The 2*64 dimensional data read in the second shot are {B02, B12, B22, B32, ..., B622, B632} and {B03, B13, B23, B33, ..., B623, B633}.

[0149] Similarly, the 2*64 dimensional data read in the 32nd beat are {B062, B162, B262, B362, ..., B6262, B6362} and {B063, B163, B263, B363, ..., B6263, B6363}. The process of loading the read data pulse beat into the WPR unit is the same as the loading process of the 1st beat.

[0150] It can be seen that after 32 load operations, the WPR unit data inside PE0 has completed the data cache of 64*4 dimensions. After 33 load operations, the WPR unit data inside PE1 has completed the data cache of 64*4 dimensions. Similarly, after 47 load operations, the WPR unit data inside PE15 has completed the data cache of 64*4 dimensions.

[0151] Step S2: Read 1*64 one-dimensional vector from DIN_MEM and start pulsation calculation. After waiting for the WPR unit inside PE0 to complete the 64*4 dimensional two-dimensional matrix data cache, you can start reading 1*64 one-dimensional vector data from DIN_MEM and send it to PE0. After PE0 receives the 1*64 one-dimensional vector data, it starts the multiplication and accumulation operation of the 1*64 one-dimensional vector and the 64*4 two-dimensional matrix. At the same time, PE0 processes the 1*64 one-dimensional vector and sends it to PE1 for vector matrix multiplication and accumulation operation. It can be seen that the 1*64 one-dimensional vector received by PEn is one beat later than the 1*64 dimensional vector received by PEn-1. Because the loading of the 2*4 dimensional data of the WPR unit inside PEn is also one beat later than the 2*4 dimensional data of the WPR unit inside PEn-1. Therefore, the timing can be matched. But PEn outputs the calculation result one beat later than PEn-1.

[0152] Because PE needs to wait for the WPR inside PE to complete the loading of the two-dimensional input matrix before starting calculation, there is a certain waiting time in the loading process. Therefore, there will be a loss of computing performance when the two-dimensional input matrix is ​​switched. In order to avoid the problem of computing performance loss caused by switching the two-dimensional input matrix, two groups of WPR units are designed in this application, as shown in the figure below. The two-dimensional matrix data read from DIN_MEN is first loaded into the WPR1 unit for cache processing. After waiting for all the two-dimensional matrix data that WPR1 needs to cache to be loaded, the two-dimensional matrix data is loaded into WPR0 for cache processing at one time. At this time, WPR1 can start loading a new set of two-dimensional matrix data, and can use the two-dimensional matrix data of WPR0 for vector-matrix multiplication operations. Please refer to Fig. 9 and Fig.10 ,in, Fig. 9 shows a schematic diagram of cache switching involved in this application, Fig.10 FIG. 1 shows a schematic diagram of the timing of cache switching involved in the present application. Fig. 9 and Fig.10 As shown, after using this solution, seamless calculation operations can be achieved when the two-dimensional input matrix is ​​switched, solving the problem of calculation performance loss.

[0153] When there are multiple different 1*k one-dimensional vectors multiplied with the same k*n two-dimensional matrix, after waiting for the WPR unit inside the PE to complete the k*n dimension two-dimensional matrix data cache, it is only necessary to read different 1*k one-dimensional vector data from DIN_MEM and send it to PE for calculation every beat until all 1*k one-dimensional vectors and k*n two-dimensional matrices are calculated. In this process, the k*n two-dimensional matrix does not need to be read and updated repeatedly.

[0154] based on Figure 2 or Figure 3 For the vector matrix calculation components shown, please refer to Fig.11 , which shows a method flow chart of a vector matrix calculation method provided by an exemplary embodiment of the present application. The vector matrix calculation method can be Figure 2 or Figure 3 The vector matrix calculation component shown is executed as Fig.11 As shown, the method may include:

[0155] Step 1101: i groups of first beat registers temporarily store the two-dimensional matrix data read in the beat order; each group of the i groups of first beat registers is used to temporarily store the matrix data part of the two-dimensional matrix data corresponding to the connected computing processing unit.

[0156] Step 1102: i computing processing units obtain corresponding matrix data parts from a connected group of first beat registers, perform multiplication and accumulation processing on the obtained matrix data parts and one-dimensional vector data, and input the obtained multiplication and accumulation results into the output cache unit.

[0157] In a possible implementation, each of the i computing processing units includes a cache register, j multiplication and accumulation unit groups, and a second beat register; j is an integer greater than or equal to 1;

[0158] i computing processing units obtain corresponding matrix data parts from a connected group of first beat registers, perform multiplication and accumulation processing on the obtained matrix data parts and one-dimensional vector data, and input the obtained multiplication and accumulation results into the output buffer unit, including:

[0159] The cache register obtains the corresponding matrix data portion from a connected group of first beat registers and caches it;

[0160] The multiplication-accumulation unit group performs multiplication-accumulation processing on the matrix data part stored in the cache register and the one-dimensional vector data, and inputs the obtained multiplication-accumulation result into the output cache unit;

[0161] The second beat register is used to transfer one-dimensional vector data in the beat order.

[0162] In a possible implementation, a group of first beat registers corresponding to the s-th computing processing unit among the i computing processing units includes s first beat registers, where 1≤s≤i, and s is an integer;

[0163] When the value of s is greater than 1, the first first beat register among the s first beat registers is connected to the input buffer unit, and each two adjacent first beat registers among the s first beat registers are connected in sequence, and the last first beat register among the s first beat registers is connected to the sth calculation processing unit;

[0164] The cache register obtains and caches the corresponding matrix data portion from a connected set of first beat registers, including:

[0165] The cache register in the sth computing processing unit is used to obtain and cache the corresponding matrix data portion from the sth beat register among the s first beat registers.

[0166] In a possible implementation, the i groups of first beat registers temporarily store the two-dimensional matrix data read in the beat sequence, including:

[0167] The first first beat register among the s first beat registers, in a beat, when a matrix data portion of the two-dimensional matrix data is cached, transfers the cached matrix data portion to the second first beat register among the s first beat registers; the first first beat register among the s first beat registers, in a beat, when there is a matrix data portion corresponding to the s-th calculation processing unit in the two-dimensional matrix data and has not been read by the first first beat register among the s first beat registers, reads a matrix data portion corresponding to the s-th calculation processing unit from the two-dimensional matrix data and caches it;

[0168] In one beat, if the t-th first beat register among the s first beat registers has a matrix data portion of two-dimensional matrix data cached, the cached matrix data portion is transferred to the t+1-th first beat register among the s first beat registers; in one beat, if the t-th first beat register among the s first beat registers has a matrix data portion in the t-1-th first beat register among the s first beat registers, the matrix data portion transferred from the t-1-th first beat register among the s first beat registers is obtained and cached; 2≤t≤s-1, and t is an integer;

[0169] In one beat, when the s-th first beat register among the s first beat registers has a matrix data portion in the s-1-th first beat register among the s first beat registers, the matrix data portion transferred from the s-1-th first beat register among the s first beat registers is obtained and cached.

[0170] In a possible implementation, the second beat registers in the i computing processing units constitute i second beat registers connected in sequence, and the first second beat register among the i second beat registers is connected to the input buffer unit;

[0171] The second beat register is used to transfer one-dimensional vector data in the beat order, including:

[0172] The first second beat register of the i second beat registers, in one beat, when there is one-dimensional vector data cached, transfers the cached one-dimensional vector data to the second second beat register of the i second beat registers; the first first beat register, in one beat, when there is one-dimensional matrix data corresponding to the two-dimensional data matrix in the input buffer unit and not read by the first second beat register, reads the one-dimensional matrix data from the input buffer unit and caches it;

[0173] The qth second beat register among the i second beat registers, in a beat, when there is one-dimensional vector data cached, passes the cached one-dimensional vector data to the q+1th second beat register among the i first beat registers for caching; the qth beat register is also used to obtain and cache the one-dimensional vector data passed by the q-1th second beat register; 2≤q≤i-1, and q is an integer;

[0174] The i-th second beat register among the i second beat registers obtains and caches the one-dimensional vector data transmitted from the i-1-th second beat register in one beat.

[0175] In a possible implementation, a one-dimensional vector is a vector with a dimension of 1*k, and a two-dimensional matrix is ​​a matrix with a dimension of k*n; k and n are integers greater than or equal to 2;

[0176] Each matrix data portion in the two-dimensional matrix data includes a data portion corresponding to one computing processing unit in r rows of data in the two-dimensional matrix data; 1≤r<n, and r is an integer;

[0177] The second beat register is used to transfer one-dimensional vector data in the beat order, including:

[0178] After the first second beat register completes caching of the entire matrix data portion corresponding to the first computing processing unit in the two-dimensional matrix data in the corresponding cache register, in one beat, if there is one-dimensional matrix data corresponding to the two-dimensional data matrix in the input cache unit and has not been read by the first second beat register, the one-dimensional matrix data is read from the input cache unit and cached.

[0179] In a possible implementation manner, the cache register includes a first register and a second register; and the method further includes:

[0180] After the first register completes caching of all matrix data parts corresponding to the calculation processing unit in the two-dimensional matrix data, the first register writes all matrix data parts of the calculation processing unit into the second register, and obtains the corresponding new matrix data parts from a connected group of first beat registers and caches them;

[0181] The multiplication-accumulation unit group performs multiplication-accumulation processing on the matrix data portion stored in the second register and the one-dimensional vector data, and inputs the obtained multiplication-accumulation result into the output buffer unit.

[0182] In one possible implementation, one-dimensional vector data and two-dimensional matrix data are the data for vector-matrix multiplication operations during the operation of the AI ​​model.

[0183] In other words, this application proposes a new acceleration scheme for one-dimensional vector and two-dimensional matrix multiplication operations, which mainly includes the following features:

[0184] 1) The DIN_MEM unified shared cache solution is used to store the input one-dimensional vector and two-dimensional matrix data. The two-dimensional matrix data is loaded from DIN_MEM to the WPR unit inside the PE for cache processing through multi-beat pulse beating. The solution has strong scalability. The more beats the data is read out from DIN_MEM and sent to the PE unit at a farther distance, the more beats there are, which can effectively solve the timing bottleneck problem caused by the long routing distance, thereby effectively improving the design main frequency.

[0185] 2) The one-dimensional vector data is transmitted from DIN_MEM to all PE units in a pulsating manner, which can effectively solve the timing bottleneck problem caused by excessive data fan-out when the one-dimensional vector data is transmitted to all PE units at the same time, thereby effectively improving the design main frequency.

[0186] 3) The PE computing unit is designed with a WPR unit for caching two-dimensional matrix data. By reducing repeated read operations on DIM_MEM, the dynamic power consumption of the chip can be effectively reduced. At the same time, two sets of two-dimensional matrix data cache units, WPR1 and WPR0, are designed to improve computing performance through preloading.

[0187] In an exemplary embodiment, a chip is also provided, the chip comprising at least one Figure 2 , Figure 3 or Figure 5 The vector matrix calculation component shown in the figure is used to implement the above-mentioned Fig.11 The method shown.

[0188] In some exemplary embodiments, a computer program product is further provided, which is executed by a control unit in a chip to implement the above Fig.11 All or part of the steps in the method shown.

[0189] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0190] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A vector matrix calculation component, characterized in that: The vector matrix calculation component includes: an input buffer unit, i calculation processing units, i groups of first beat registers and an output buffer unit; the input buffer unit is used to store one-dimensional vector data and two-dimensional matrix data; i is an integer greater than or equal to 2; The i computing processing units correspond one-to-one to the i groups of first beat registers, and the computing processing units are connected to the corresponding group of first beat registers; The first computing processing unit among the i computing processing units and the input buffer unit, and every two adjacent computing processing units among the i computing processing units are connected in sequence; The i groups of first beat registers are respectively connected to the input buffer units; The i computing processing units are respectively connected to the output buffer units; The i groups of first beat registers are used to temporarily store the two-dimensional matrix data read in a beat order; each group of the i groups of first beat registers is used to temporarily store the matrix data portion of the two-dimensional matrix data corresponding to the connected calculation processing unit; The i computing processing units are used to sequentially transmit the one-dimensional vector data according to the beat order; The i computing processing units are also used to obtain corresponding matrix data parts from a connected group of first beat registers, perform multiplication and accumulation processing on the obtained matrix data parts and the one-dimensional vector data, and input the obtained multiplication and accumulation results into the output cache unit.

2. The vector matrix calculation component according to claim 1, characterized in that: Each of the i computing processing units comprises a cache register, j multiplication and accumulation unit groups and a second beat register; j is an integer greater than or equal to 1; The cache register is used to obtain and cache the corresponding matrix data portion from a group of connected first beat registers; The multiplication-accumulation unit group is used to perform multiplication-accumulation processing on the matrix data part stored in the cache register and the one-dimensional vector data, and input the obtained multiplication-accumulation result into the output cache unit; The second beat register is used to transfer the one-dimensional vector data according to the beat order.

3. The vector matrix calculation component according to claim 2, characterized in that: A group of first beat registers corresponding to the s-th computing processing unit among the i computing processing units includes s first beat registers, 1≤s≤i, and s is an integer; When the value of s is greater than 1, the first one of the s first beat registers is connected to the input buffer unit, and each two adjacent first beat registers of the s first beat registers are connected in sequence, and the last one of the s first beat registers is connected to the sth calculation processing unit; The cache register in the sth computing processing unit is used to obtain and cache the corresponding matrix data portion from the sth beat register among the s first beat registers.

4. The method according to claim 3, characterized in that The first one of the s first beat registers is used to, in one beat, transfer the cached matrix data part to the second one of the s first beat registers when a matrix data part of the two-dimensional matrix data is cached; the first one of the s first beat registers is also used to, in one beat, read a matrix data part corresponding to the s-th computing processing unit from the two-dimensional matrix data and cache it when there is a matrix data part corresponding to the s-th computing processing unit in the two-dimensional matrix data and has not been read by the first one of the s first beat registers; The t-th first beat register among the s first beat registers is used to, in one beat, transfer the cached matrix data part to the t+1-th first beat register among the s first beat registers when a matrix data part of the two-dimensional matrix data is cached; the t-th first beat register among the s first beat registers is also used to, in one beat, obtain and cache the matrix data part transferred from the t-1-th first beat register among the s first beat registers when a matrix data part exists in the t-1-th first beat register among the s first beat registers; 2≤t≤s-1, and t is an integer; The sth first beat register among the s first beat registers is used to obtain and cache the matrix data part transferred from the s-1th first beat register among the s first beat registers in one beat when there is a matrix data part in the s-1th first beat register among the s first beat registers.

5. The vector matrix calculation component according to claim 2, characterized in that: The second beat registers in the i computing processing units constitute i second beat registers connected in sequence, and the first second beat register among the i second beat registers is connected to the input buffer unit; The first second beat register of the i second beat registers is used to, in one beat, transfer the cached one-dimensional vector data to the second second beat register of the i second beat registers when there is one-dimensional vector data cached; the first first beat register is also used to, in one beat, read the one-dimensional matrix data from the input buffer unit and cache it when there is one-dimensional matrix data corresponding to the two-dimensional data matrix in the input buffer unit and not read by the first second beat register; The qth second beat register among the i second beat registers is used to transfer the cached one-dimensional vector data to the q+1th second beat register among the i first beat registers for caching in one beat; the qth beat register is also used to obtain and cache the one-dimensional vector data transferred from the q-1th second beat register; 2≤q≤i-1, and q is an integer; The i-th second beat register among the i second beat registers is used to obtain and cache the one-dimensional vector data transmitted from the i-1-th second beat register in one beat.

6. The vector matrix calculation component according to claim 5, characterized in that: The one-dimensional vector data is a vector with a dimension of 1*k, and the two-dimensional matrix data is a matrix with a dimension of k*n; k and n are integers greater than or equal to 2; Each of the matrix data parts in the two-dimensional matrix data includes a data part corresponding to one computing processing unit in r rows of data in the two-dimensional matrix data; 1≤r<n, and r is an integer; The first second beat register is used to read and cache one-dimensional vector data from the input cache unit in one beat after the corresponding cache register completes caching of the entire matrix data portion of the two-dimensional matrix data corresponding to the first computing processing unit, if there is one-dimensional vector data corresponding to the two-dimensional data matrix in the input cache unit and has not been read by the first second beat register.

7. The vector matrix calculation component according to claim 6, characterized in that: The cache register includes a first register and a second register; The first register is used to write all the matrix data parts of the computing processing unit into the second register after completing the caching of all the matrix data parts of the computing processing unit in the two-dimensional matrix data, and obtain the corresponding new matrix data parts from a group of connected first beat registers and cache them; The multiplication-accumulation unit group is used to perform multiplication-accumulation processing on the matrix data part stored in the second register and the one-dimensional vector data, and input the obtained multiplication-accumulation result into the output buffer unit.

8. The vector matrix calculation component according to any one of claims 1 to 7, characterized in that: The one-dimensional vector data and the two-dimensional matrix data are data for performing vector-matrix multiplication operations during the operation of the AI ​​model.

9. A vector matrix calculation method, characterized in that: The method is applied to the vector matrix calculation component according to any one of claims 1 to 8, and the method comprises: The i groups of first beat registers temporarily store the two-dimensional matrix data read in a beat order; each group of the i groups of first beat registers is used to temporarily store the matrix data portion of the two-dimensional matrix data corresponding to the connected calculation processing unit; The i computing processing units obtain corresponding matrix data parts from a connected group of first beat registers, perform multiplication and accumulation processing on the obtained matrix data parts and the one-dimensional vector data, and input the obtained multiplication and accumulation results into the output cache unit.

10. The method according to claim 9, characterized in that Each of the i computing processing units comprises a cache register, j multiplication and accumulation unit groups and a second beat register; j is an integer greater than or equal to 1; The i computing processing units obtain corresponding matrix data parts from a connected group of first beat registers, perform multiplication and accumulation processing on the obtained matrix data parts and the one-dimensional vector data, and input the obtained multiplication and accumulation results into the output buffer unit, including: The cache register obtains the corresponding matrix data portion from a connected group of first beat registers and caches it; The multiplication-accumulation unit group performs multiplication-accumulation processing on the matrix data part stored in the cache register and the one-dimensional vector data, and inputs the obtained multiplication-accumulation result into the output cache unit; The second beat register transfers the one-dimensional vector data in a beat order.

11. The method according to claim 10, characterized in that A group of first beat registers corresponding to the s-th computing processing unit among the i computing processing units includes s first beat registers, 1≤s≤i, and s is an integer; When the value of s is greater than 1, the first one of the s first beat registers is connected to the input buffer unit, and each two adjacent first beat registers of the s first beat registers are connected in sequence, and the last one of the s first beat registers is connected to the sth calculation processing unit; The cache register obtains and caches the corresponding matrix data portion from a connected group of first beat registers, including: The cache register in the sth computing processing unit is used to obtain and cache the corresponding matrix data portion from the sth beat register among the s first beat registers.

12. The method according to claim 11, characterized in that The i groups of first beat registers temporarily store the two-dimensional matrix data read in a beat order, including: The first one of the s first beat registers, in a beat, when a matrix data part of the two-dimensional matrix data is cached, transfers the cached matrix data part to the second one of the s first beat registers; In a beat, when a matrix data portion corresponding to the s-th computing processing unit exists in the two-dimensional matrix data and has not been read by the first first beat register among the s first beat registers, a matrix data portion corresponding to the s-th computing processing unit is read from the two-dimensional matrix data and cached; The t-th first beat register among the s first beat registers, in a beat, when a matrix data portion of the two-dimensional matrix data is cached, transfers the cached matrix data portion to the t+1-th first beat register among the s first beat registers; In a beat, when the t-th first beat register among the s first beat registers has a matrix data part in the t-1-th first beat register among the s first beat registers, the matrix data part transferred from the t-1-th first beat register among the s first beat registers is obtained and cached; 2≤t≤s-1, and t is an integer; In one beat, when the s-th first beat register among the s first beat registers has a matrix data portion in the s-1-th first beat register among the s first beat registers, the matrix data portion transferred from the s-1-th first beat register among the s first beat registers is obtained and cached.

13. The method according to claim 10, characterized in that The second beat registers in the i computing processing units constitute i second beat registers connected in sequence, and the first second beat register among the i second beat registers is connected to the input buffer unit; The second beat registers in the i computing processing units transmit the one-dimensional vector data according to the beat order, including: The first second beat register of the i second beat registers, in one beat, when there is one-dimensional vector data cached, transfers the cached one-dimensional vector data to the second second beat register of the i second beat registers; the first first beat register, in one beat, when there is one-dimensional matrix data corresponding to the two-dimensional data matrix in the input buffer unit and not read by the first second beat register, reads the one-dimensional matrix data from the input buffer unit and caches it; The qth second beat register among the i second beat registers is used to transfer the cached one-dimensional vector data to the q+1th second beat register among the i first beat registers for caching in one beat; the qth beat register is also used to obtain and cache the one-dimensional vector data transferred from the q-1th second beat register; 2≤q≤i-1, and q is an integer; The i-th second beat register among the i second beat registers is used to obtain and cache the one-dimensional vector data transmitted from the i-1-th second beat register in one beat.

14. The method according to claim 13, characterized in that The one-dimensional vector is a vector with a dimension of 1*k, and the two-dimensional matrix is ​​a matrix with a dimension of k*n; k and n are integers greater than or equal to 2; Each of the matrix data parts in the two-dimensional matrix data includes a data part corresponding to one computing processing unit in r rows of data in the two-dimensional matrix data; 1≤r≤n, and r is an integer; The first first beat register, in a beat, when there is one-dimensional matrix data corresponding to the two-dimensional data matrix in the input buffer unit and not read by the first second beat register, reads the one-dimensional matrix data from the input buffer unit and caches it, comprising: The first second beat register, after the corresponding cache register completes caching of the entire matrix data portion of the two-dimensional matrix data corresponding to the first computing processing unit, reads and caches the one-dimensional matrix data from the input cache unit in one beat if there is one-dimensional matrix data corresponding to the two-dimensional data matrix in the input cache unit and has not been read by the first second beat register.

15. A chip, characterized in that: The chip comprises at least one vector matrix calculation component as described in any one of claims 1 to 8.

Images