Matrix multiplication operation circuit and method

Abstract

The present application provides a matrix multiplication operation circuit and method, the circuit comprising: an IO core, a memory connected to the IO core and storing target matrix data to be calculated, and multiple AI computing cores connected to the IO core and deployed together with the IO core on a target chip; the IO core comprises a matrix multiplication real-time block module, the matrix multiplication real-time block module determines the optimal block strategy based on the calculation task information of the target matrix data and the current chip parameter information when the target chip is running, and determines the block task information of each AI computing core according to the optimal block strategy and the weights corresponding to the multiple AI computing cores; the AI ​​computing core obtains the matrix block data based on the block task information and outputs the local calculation results through calculation; the local calculation results output by the multiple AI computing cores are combined into a matrix multiplication result. The present application can effectively improve the operation efficiency of the matrix multiplication operation and solve the limitations of the offline static matrix block strategy.

Description

Technical Field

[0001] The present application relates to the technical field of circuit design, and in particular to a matrix multiplication operation circuit and method. Background Art

[0002] Matrix multiplication is the basis of artificial intelligence (AI) algorithms and is widely used in image recognition, natural language processing, speech recognition, recommendation systems, and other fields. As a major component of AI algorithm computing time, matrix multiplication is also the focus of performance optimization of dedicated AI accelerators.

[0003] A matrix is ​​a two-dimensional data structure consisting of rows and columns. Figure 1 A typical matrix multiplication operation is shown, where a matrix A of size (M, K) is multiplied by a matrix B of size (K, N) to obtain a matrix C of size (M, N).

[0004] With the increase in data scale and computing requirements, matrix blocking technology has begun to be widely used to improve computing efficiency. That is, the matrix multiplication task is broken down into blocks, distributed to different computing resources, and then the block calculation results are integrated to obtain the final result.

[0005] Figure 2A An example of a slicing method is shown, which divides the matrix into several blocks in the M dimension and multiplies them with the B matrix respectively to obtain the corresponding part of the C matrix, and then splices them to obtain the complete C matrix. Similarly, Figure 2B , Figure 2C The methods of block calculation in N dimension and K dimension are shown respectively. The block calculation in K dimension will produce multiple intermediate results with the same size as the C matrix, and the final result can be obtained by accumulation. In practical applications, the calculation strategy can be a combination of the above basic strategies, or no block calculation (when the task size is small).

[0006] The current matrix partitioning strategies for graphics processors (GPU), central processing units (CPU), AI acceleration chips, etc. mainly rely on static offline judgment by the compiler. Figure 3A , 3B The typical structures of a typical GPU and a multi-chip-based AI accelerator are shown respectively. The advantage of the matrix partitioning strategy relying on the static offline judgment of the compiler is that it is simple to implement, but in complex on-chip systems such as multi-chip accelerators, it is difficult to adjust the strategy in real time according to the status of different computing resources and communication resources at runtime, which easily leads to resource waste and reduced efficiency. Summary of the invention

[0007] In view of the above problems, the embodiments of the present application provide a matrix multiplication circuit and method that overcomes the above problems or at least partially solves the above problems.

[0008] In a first aspect, an embodiment of the present application provides a matrix multiplication operation circuit, comprising: an input / output IO core particle, a memory, and a plurality of artificial intelligence AI computing core particles, wherein the IO core particle and the plurality of AI computing core particles are deployed on a target chip, the IO core particle is connected to the memory and the plurality of AI computing core particles, and the memory stores target matrix data to be calculated;

[0009] The IO core particle includes a matrix multiplication real-time block module, which determines an optimal block strategy based on the calculation task information of the target matrix data and the current chip parameter information when the target chip is running, and determines the block task information matched by the multiple AI calculation core particles according to the optimal block strategy and the weights corresponding to the multiple AI calculation core particles respectively;

[0010] The AI ​​computing core obtains matrix block data from the memory through the IO core based on the matching block task information, calculates the matrix block data and outputs local calculation results; wherein the local calculation results respectively output by the multiple AI computing cores are combined into a matrix multiplication result corresponding to the target matrix data.

[0011] In a second aspect, an embodiment of the present application provides a matrix multiplication operation method, which is applied to an input and output IO core particle, including:

[0012] Determine an optimal block partitioning strategy based on computing task information of target matrix data to be calculated and current chip parameter information when the target chip is running, and deploy the IO core particle and multiple AI computing core particles connected to the IO core particle on the target chip;

[0013] Determine the block task information that the multiple AI computing core particles match respectively according to the optimal block strategy and the weights corresponding to the multiple AI computing core particles respectively;

[0014] Sending the block task information respectively matched by the multiple AI computing core particles to the multiple AI computing core particles;

[0015] Receiving local calculation results respectively fed back by the multiple AI calculation core particles and calculated based on the matched matrix block data, and combining the multiple local calculation results into a matrix multiplication result corresponding to the target matrix data;

[0016] The matrix block data matched by the AI ​​computing core particle is determined by the AI ​​computing core particle from the target matrix data based on the matched block task information.

[0017] The technical solution of the embodiment of the present application determines the optimal block strategy based on the computing task information of the target matrix data and the current chip parameter information of the target chip when it is running, determines the block task information of multiple AI computing core particles respectively matched according to the optimal block strategy and the weights of multiple AI computing core particles, and the AI ​​computing core particles perform block operations based on the block task information, and after obtaining the local calculation results corresponding to the multiple AI computing core particles, they are combined to obtain the matrix multiplication result. It can be realized that the optimal block method is calculated according to the real-time state of the chip at runtime, and the computational efficiency of the matrix multiplication operation is effectively improved, the limitations of the offline static matrix block cutting strategy are solved, and the compiler overhead is reduced, so that the compiler does not need to perform the underlying task cutting. Moreover, by considering the weight of the AI ​​computing core particles, the matrix is ​​segmented with the help of the optimal block strategy and the weight of the AI ​​computing core particles, which can effectively avoid the waste of resources caused by the need for global waiting due to load imbalance such as some AI computing core particles have heavy computing tasks and some AI computing core particles have light computing tasks. Furthermore, the matrix multiplication circuit provided by the present application has small area and power consumption overhead, and is easy to integrate and implement. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram showing matrix multiplication calculation;

[0019] Figure 2A A schematic diagram showing the matrix being divided into blocks and operated in M ​​dimensions;

[0020] Figure 2B A schematic diagram showing the matrix being divided into blocks and operated in N dimensions;

[0021] Figure 2C A schematic diagram showing the matrix being divided into blocks and operated in K dimensions;

[0022] Figure 3A A schematic diagram showing the connection between the graphics processor and the host and the memory;

[0023] Figure 3B A schematic diagram showing a multi-chip-based AI accelerator;

[0024] Figure 4 A schematic diagram of a matrix multiplication circuit provided in an embodiment of the present application is shown in FIG. Figure 1 ;

[0025] Figure 5 A schematic diagram showing a target chip provided in an embodiment of the present application;

[0026] Figure 6 A schematic diagram showing a cost calculation module provided in an embodiment of the present application;

[0027] Figure 7 A schematic diagram showing a matrix block module provided in an embodiment of the present application;

[0028] Figure 8 A schematic diagram showing a cascade of block size calculation units of the same dimension in a core task size submodule provided in an embodiment of the present application;

[0029] Fig. 9A Schematic diagram of a block size calculation unit provided in an embodiment of the present application Figure 1 ;

[0030] Fig. 9B Schematic diagram 2 showing a block size calculation unit provided in an embodiment of the present application;

[0031] Fig.10 Schematic diagram 2 of a matrix multiplication operation circuit provided in an embodiment of the present application;

[0032] Fig.11 Schematic diagram 3 of a matrix multiplication operation circuit provided in an embodiment of the present application;

[0033] Fig.12 A schematic diagram showing a matrix multiplication method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0035] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. The multiple in the embodiments of the present application can include two and more than two.

[0036] The matrix multiplication operation circuit provided in the embodiment of the present application is introduced below. Figure 4 and Figure 5 As shown, the matrix multiplication operation circuit includes: an input and output IO core particle 1, a memory 2, and multiple artificial intelligence AI computing core particles 3, the IO core particle 1 and the multiple AI computing core particles 3 are deployed on a target chip 4, the IO core particle 1 is connected to the memory 2 and the multiple AI computing core particles 3, and the memory 2 stores the target matrix data to be calculated;

[0037] The IO core particle 1 includes a matrix multiplication real-time block module 10, which determines the optimal block strategy based on the calculation task information of the target matrix data and the current chip parameter information of the target chip 4 when it is running, and determines the block task information that matches the multiple AI calculation core particles 3 respectively according to the optimal block strategy and the weights corresponding to the multiple AI calculation core particles 3 respectively;

[0038] Based on the matching block task information, the AI ​​computing core 3 obtains the matrix block data from the memory 2 through the IO core 1, calculates the matrix block data and outputs the local calculation results; wherein the local calculation results output by multiple AI computing cores 3 are combined into the matrix multiplication result corresponding to the target matrix data.

[0039] The matrix multiplication circuit provided in the embodiment of the present application includes an IO core particle 1, a memory 2 and a plurality of AI computing core particles 3. The memory 2 is used to store target matrix data to be subjected to matrix multiplication operation, for example, the memory 2 is a graphics double data rate (Graphics Double Data Rate, GDDR) dynamic random access memory (Dynamic Random Access Memory, DRAM) or a double data rate (Double Data Rate, DDR) synchronous dynamic random access memory (Synchronous Dynamic Random Access Memory, SDRAM). The memory 2 is connected to the IO core particle 1, and the AI ​​computing core particle 3 is connected to the IO core particle 1, and the AI ​​computing core particle 3 can be connected to the memory 2 through the IO core particle 1.

[0040] The IO core 1 and multiple AI computing cores 3 are deployed on the target chip 4. As the core processing component of the matrix multiplication operation circuit, the IO core 1 includes a matrix multiplication real-time block module 10. The matrix multiplication real-time block module 10 obtains the calculation task information of the target matrix data. Since the IO core 1 is deployed on the target chip 4, the matrix multiplication real-time block module 10 included in the IO core 1 can obtain the current chip parameter information when the target chip 4 is running, and realize the real-time acquisition of the relevant parameters when the target chip 4 is running. The matrix multiplication real-time block module 10 determines the optimal block strategy based on the calculation task information of the target matrix data and the current chip parameter information when the target chip 4 is running. The optimal block strategy is the matrix block strategy that is most suitable for the target matrix data selected from multiple matrix block strategies; for any matrix block strategy, it indicates segmentation in at least one dimension (segmentation in the row dimension and / or column dimension of the matrix), and it can also indicate no segmentation.

[0041] The real-time matrix multiplication block module 10 determines the block task information that matches the multiple AI computing core particles 3 respectively based on the optimal block strategy and the weights corresponding to the multiple AI computing core particles 3 when determining the optimal block strategy. The optimal block strategy indicates in which dimension the matrix is ​​segmented. In actual segmentation, in addition to the optimal block strategy, it is also necessary to determine the weights corresponding to each AI computing core particle 3. Based on the optimal block strategy and the weights corresponding to the multiple AI computing core particles 3, the computing tasks adapted to each AI computing core particle 3 are determined, and then the block task information of the AI ​​computing core particle 3 is determined. The weight corresponding to the AI ​​computing core particle 3 can be determined according to the load, computing power and bandwidth of the AI ​​computing core particle 3, and can be adjusted dynamically. By considering the weight of the AI ​​computing core particle 3 and performing matrix segmentation with the help of the optimal block strategy and the weight of the AI ​​computing core particle 3, it can effectively avoid the waste of resources caused by the need for global waiting due to load imbalance such as some AI computing core particles have heavy computing tasks and some AI computing core particles have light computing tasks.

[0042] For the AI ​​computing core 3, it can obtain the matrix block data corresponding to the block task information from the memory 2 through the IO core 1 based on the matching block task information. After obtaining the matrix block data, it performs matrix multiplication on the matrix block data to output the local calculation result. The local calculation result output by the AI ​​computing core 3 is stored in the specified location. The IO core 1 obtains the local calculation results output by multiple AI computing cores 3 respectively, combines the multiple local calculation results, and generates the matrix multiplication result corresponding to the target matrix data, so as to determine the matrix multiplication result based on the block operation and the combination of the operation results, and effectively improve the calculation efficiency of the matrix multiplication task. It should be noted that after the IO core 1 obtains multiple local calculation results, many factors need to be considered in the actual combination, such as the combination position. The technicians in this field make the combination based on the actual situation, which will not be described in detail here.

[0043] The matrix multiplication operation circuit provided in the above-mentioned implementation scheme of the present application is suitable for an accelerator with a multi-core architecture. The optimal block strategy is determined based on the computing task information of the target matrix data and the current chip parameter information when the target chip is running. The block task information of multiple AI computing core particles respectively matched is determined according to the optimal block strategy and the weights of multiple AI computing core particles. The AI ​​computing core particles perform block operations based on the block task information. After obtaining the local calculation results corresponding to the multiple AI computing core particles, they are combined to obtain the matrix multiplication operation result. It can be realized that the optimal block method is calculated according to the real-time state of the chip at runtime, and the computing efficiency of the accelerator based on the multi-core architecture is effectively improved. The limitations of the offline static matrix block cutting strategy are solved, and the compiler overhead is reduced, so that the compiler does not need to perform the underlying task cutting. Moreover, by considering the weight of the AI ​​computing core particles, the matrix is ​​segmented with the help of the optimal block strategy and the weight of the AI ​​computing core particles, which can effectively avoid the waste of resources caused by the need for global waiting due to load imbalance such as some AI computing core particles have heavy computing tasks and some AI computing core particles have light computing tasks. Furthermore, the matrix multiplication operation circuit provided by the present application has small area and power consumption overhead, and is easy to integrate and implement.

[0044] The following is an introduction to the real-time block module for matrix multiplication. Figure 4 As shown, the matrix multiplication real-time block module 10 includes a cost calculation module 11, a matrix block module 12 and an AI configuration module 13;

[0045] The cost calculation module 11 determines the strategy costs corresponding to P matrix partition strategies based on the computing task information and the current chip parameter information, determines the optimal partition strategy based on the P strategy costs, and determines the weights corresponding to multiple AI computing core particles 3 based on the current chip parameter information, where P is the total number of matrix partition strategies corresponding to the target matrix data;

[0046] The matrix partitioning module 12 is connected to the cost calculation module 11, and divides the subtask size information corresponding to the multiple AI computing core particles 3 based on the optimal partitioning strategy, the weights corresponding to the multiple AI computing core particles 3, and the computing task information;

[0047] The AI ​​configuration module 13 is connected to the matrix block module 12, and configures the block task information respectively matched by the multiple AI computing core particles 3 based on the subtask size information respectively corresponding to the multiple AI computing core particles 3.

[0048] The matrix multiplication real-time block module 10 includes a cost calculation module 11, a matrix block module 12 connected to the cost calculation module 11, and an AI configuration module 13 connected to the matrix block module 12. The cost calculation module 11 obtains the computing task information and the current chip parameter information, and determines the strategy costs corresponding to the P matrix block strategies based on the acquired computing task information and the current chip parameter information. The P matrix block strategies are all the matrix block strategies corresponding to the target matrix data. After determining the P strategy costs, the cost calculation module 11 determines the final strategy cost based on the P strategy costs, and determines the matrix block strategy corresponding to the final strategy cost as the optimal block strategy, so as to select the optimal block strategy after determining the P strategy costs.

[0049] After obtaining the current chip parameter information, the cost calculation module 11 also needs to determine the weights corresponding to the multiple AI computing core particles 3 based on the current chip parameter information, so as to provide the matrix blocking module 12 with the optimal blocking strategy and the weights corresponding to the multiple AI computing core particles 3 .

[0050] For the matrix block module 12, it obtains the optimal block strategy, the weights corresponding to the multiple AI computing core particles 3, and also needs to obtain the computing task information corresponding to the target matrix data. Based on the optimal block strategy, the weights corresponding to the multiple AI computing core particles 3, and the computing task information, the matrix block module 12 divides the subtask size information corresponding to the multiple AI computing core particles 3, and realizes the calculation of the subtask size information adapted to each AI computing core particle 3 based on the optimal block strategy and the weight of the AI ​​computing core particle 3.

[0051] The AI ​​configuration module 13 is connected to the matrix block module 12, and obtains the subtask size information corresponding to each AI computing core particle 3 provided by the matrix block module 12. For each AI computing core particle 3, based on its corresponding subtask size information, the block task information of the AI ​​computing core particle 3 is configured. For the AI ​​configuration module 13, the AI ​​configuration module 13 parses the subtask size information corresponding to the AI ​​computing core particle 3 into the block task information acceptable to the AI ​​computing core particle 3. The block task information includes the block storage address corresponding to the matrix block data to be calculated, the block size information, the cutting dimension (including the dimension to be cut, and the original size of the dimension) and the first storage address for storing the local calculation results. Based on the block task information, the AI ​​computing core particle 3 can obtain the corresponding matrix block data to perform matrix operations to provide local calculation results.

[0052] As an example, the block task information of the AI ​​calculation core 3 configured by the AI ​​configuration module 13 is introduced. For example, the data volume of the original A matrix (M, K) corresponding to M rows and K columns is z bytes (z = M*K*number of bytes of data type), the storage address is a_i, the output result is to be placed at address a_o, and the output result matrix size is y bytes. Now we need to cut the A matrix into two halves to obtain two matrices: (M / 2, K) and (M / 2, K), which are divided into two AI calculation cores 3. The subtask size information obtained by the first AI calculation core is (M / 2, K), the starting address is a_i, "cut in M ​​dimension, the original dimension size is M", and the calculation result is placed in a_o; the subtask size information obtained by the second AI calculation core is (M / 2, K), the starting address is (a_i+z / 2), "cut in M ​​dimension, the original dimension size is M", and the calculation result is placed in a_o+y / 2.

[0053] In the above implementation process, the cost calculation module determines the optimal blocking strategy and the weights corresponding to multiple AI computing cores and provides them to the matrix blocking module. The matrix blocking module calculates the subtask size information adapted to each AI computing core based on the optimal blocking strategy and the weights of the AI ​​computing cores. The AI ​​configuration module configures the blocking task information of the AI ​​computing core based on the subtask size information, so as to allocate adapted computing tasks to each AI computing core.

[0054] The computing task information corresponding to the target matrix data provided in the embodiment of the present application includes matrix size information corresponding to the first matrix and the second matrix, and the first matrix and the second matrix are matrices to be multiplied corresponding to the target matrix data; the current chip parameter information provided in the embodiment of the present application includes first chip parameter information related to communication resources and second chip parameter information related to computing power.

[0055] The following is an introduction to the process of determining the optimal block strategy and the weights corresponding to multiple AI computing cores in the cost calculation module. Figure 6 As shown, the cost calculation module 11 includes a decision maker 111 and P strategy cost submodules 112, each strategy cost submodule 112 corresponds to a matrix block strategy, and the strategy cost submodule 112 includes a communication cost unit 1121, a calculation cost unit 1122 and a cost merging unit 1123;

[0056] The communication cost unit 1121 obtains the communication time cost based on the first chip parameter information and the matrix size information, and the calculation cost unit 1122 obtains the calculation time cost based on the second chip parameter information and the matrix size information; the cost merging unit 1123 connects the communication cost unit 1121 and the calculation cost unit 1122, and determines the strategy cost adapted to the current strategy cost submodule based on the communication time cost and the calculation time cost;

[0057] The decision maker 111 is connected to P strategy cost submodules 112 , and determines a final strategy cost according to the strategy costs respectively corresponding to the P strategy cost submodules 112 . The matrix partitioning strategy corresponding to the final strategy cost is the optimal partitioning strategy.

[0058] The cost calculation module 11 includes a decision maker 111 and P strategy cost submodules 112, each strategy cost submodule 112 is adapted to a matrix block strategy, and the P strategy cost submodules 112 are connected to the decision maker 111. It can be understood that: the number of strategy cost submodules 112 is the same as the number of matrix block strategies corresponding to the target matrix data. If two matrix block strategies use the same calculation logic, such as a certain addition, the adapted addition circuit can be shared / reused by the two matrix block strategies, that is, in hardware implementation, some common modules can be reused; but logically speaking, there still needs to be two strategy cost submodules 112, each strategy cost submodule 112 corresponds to a matrix block strategy, and it is not that one strategy cost submodule 112 is responsible for two matrix block strategies.

[0059] For each strategy cost submodule 112, which corresponds to a matrix block strategy, the strategy cost submodule 112 includes a communication cost unit 1121, a calculation cost unit 1122, and a cost merging unit 1123. The input end of the communication cost unit 1121 inputs the first chip parameter information related to the communication resource and the matrix size information corresponding to the first matrix and the second matrix, and the communication cost unit 1121 determines the communication time cost based on the acquired first chip parameter information and matrix size information; the input end of the calculation cost unit 1122 inputs the second chip parameter information related to the computing power and the matrix size information corresponding to the first matrix and the second matrix, and the calculation cost unit 1122 determines the calculation time cost based on the acquired second chip parameter information and matrix size information. The input end of the cost merging unit 1123 is connected to the output ends of the communication cost unit 1121 and the calculation cost unit 1122, receives the communication time cost provided by the communication cost unit 1121 and the calculation time cost provided by the calculation cost unit 1122, and determines the strategy cost adapted to the current strategy cost submodule according to the communication time cost and the calculation time cost. Among them, the cost merging unit 1123 adopts a preset strategy to determine the strategy cost based on the communication time cost and the computing time cost. For example, the larger time cost can be selected as the strategy cost from the communication time cost and the computing time cost. Since it is necessary to ensure that the strategy cost covers the communication time cost and the computing time cost, the larger time cost of the two is selected as the strategy cost. Alternatively, a weighted summation strategy is adopted to determine the strategy cost based on the communication time cost and the computing time cost. At this time, it is necessary to determine the cost weights corresponding to the communication time cost and the computing time cost respectively, and then determine the strategy cost based on the cost weights, the communication time cost, and the computing time cost. A direct summation strategy can also be adopted to directly determine the strategy cost based on the communication time cost and the computing time cost. Of course, other methods can also be used to determine the strategy cost, which will not be listed here one by one.

[0060] Since the cost calculation module 11 includes P policy cost submodules 112, each policy cost submodule 112 can provide a policy cost, and the decision maker 111 is connected to the P policy cost submodules 112, specifically connected to the output end of the cost merging unit 1123. The decision maker 111 obtains the policy costs respectively provided by the P policy cost submodules 112, and determines the final policy cost according to the policy costs respectively corresponding to the P policy cost submodules 112. Since the policy cost is determined based on the communication time cost and the computing time cost, it represents the cost of the time dimension. When determining the final policy cost among the P policy costs, the decision maker 111 selects the smallest policy cost among the P policy costs as the final policy cost, which can select the smallest time cost, and then determine the optimal block strategy based on the smallest time cost.

[0061] In the above implementation process, the strategy cost submodule determines the communication time cost based on the first chip parameter information and the matrix size information, determines the calculation time cost based on the second chip parameter information and the matrix size information, and determines the strategy cost based on the communication time cost and the calculation time cost. The decision maker selects the smaller strategy cost among P strategy costs as the final strategy cost, and then determines the optimal blocking strategy based on the minimum time cost, thereby selecting a matrix blocking strategy that takes less time.

[0062] As an optional embodiment, the first chip parameter information includes first chip specification information associated with bandwidth resources and the number of AI computing chiplets, and first chip status information associated with communication load; the second chip parameter information includes second chip specification information associated with the peak computing power of the AI ​​computing chiplet and the number of AI computing chiplets, and second chip status information associated with the AI ​​computing chiplet load; and the first chip specification information includes chiplet bandwidth information characterizing the connection bandwidth between the AI ​​computing chiplet and the IO chiplet.

[0063] Since the first chip specification information is associated with bandwidth resources and the number of AI computing cores, and the first chip status information is associated with communication load, the communication cost unit 1121 can obtain the communication time cost based on the first chip specification information, the first chip status information and the matrix size information; since the second chip specification information is associated with the peak computing power of the AI ​​computing cores and the number of AI computing cores, and the second chip status information is associated with the AI ​​computing core load, the computing cost unit 1122 can obtain the computing time cost based on the second chip specification information, the second chip status information and the matrix size information.

[0064] The weight corresponding to the AI ​​computing core can be determined according to the load, computing power and bandwidth of the AI ​​computing core. For the decision maker 111, it can obtain the current chip parameter information, which includes the first chip parameter information related to the communication resources and the second chip parameter information related to the computing power. Since the first chip parameter information includes the first chip specification information associated with the bandwidth resources and the number of AI computing cores and the first chip status information associated with the communication load, the second chip parameter information includes the second chip specification information associated with the peak computing power of the AI ​​computing core and the number of AI computing cores, and the second chip status information associated with the load of the AI ​​computing core, the decision maker 111 can obtain the core bandwidth information, AI computing core load and AI computing core peak computing power corresponding to multiple AI computing cores in the current chip parameter information, and determine the weights corresponding to multiple AI computing cores based on the information obtained; wherein the weight corresponding to the AI ​​computing core is negatively correlated with the AI ​​computing core load and positively correlated with the core bandwidth information and the peak computing power of the AI ​​computing core.

[0065] By extracting relevant information from the current chip parameter information and determining the weights corresponding to multiple AI computing core particles, the weights of the AI ​​computing core particles can be considered, and matrix segmentation can be performed with the help of the optimal blocking strategy and the weights of the AI ​​computing core particles to achieve the allocation of adaptive computing tasks to each AI computing core particle. This can effectively avoid the waste of resources caused by the need for global waiting due to load imbalance such as some AI computing core particles have heavy computing tasks and some AI computing core particles have light computing tasks.

[0066] As an optional embodiment, the computing task information corresponding to the target matrix data also includes: the matrix storage address corresponding to the first matrix and the second matrix and the result storage address; the block task information corresponding to the AI ​​computing core particle includes the block storage address corresponding to the matrix block data to be calculated, the block size information, the cutting dimension (including the dimension to be cut, and the original size of the dimension) and the first storage address; the block storage address is determined based on the matrix storage address, and the first storage address is determined based on the result storage address.

[0067] like Figure 4 As shown, after obtaining the subtask size information corresponding to each AI computing core particle 3 provided by the matrix blocking module 12, the AI ​​configuration module 13 configures the block task information of the AI ​​computing core particle 3 for each AI computing core particle 3 based on its corresponding subtask size information. Since the subtask size information corresponding to the AI ​​computing core particle 3 is known, the block size information and the cutting dimension (including the dimension to be cut and the original size of the dimension) can be determined, and the AI ​​configuration module 13 can also configure the block storage address and the first storage address for the AI ​​computing core particle 3.

[0068] Among them, the AI ​​computing core 3 obtains the matrix block data from the memory 2 based on the block storage address; after determining the local calculation result corresponding to the matrix block data through matrix multiplication operation, the local calculation result is stored in the specified position of the memory 2 based on the first storage address.

[0069] The block task information corresponding to the AI ​​computing core 3 includes a block storage address, which is determined based on the matrix storage address. Since the memory 2 stores the target matrix data, the target matrix data corresponds to the matrix storage address, and the AI ​​computing core 3 can extract the matrix data from the target matrix data based on the block storage address to obtain the matrix block data. After obtaining the matrix block data, a matrix multiplication operation is performed to obtain the local calculation result corresponding to the matrix block data, and then the local calculation result is stored in the specified position of the memory 2 based on the first storage address, so as to obtain the matrix block data from the memory 2 and return the local calculation result corresponding to the matrix block data to the memory 2. The IO core 1 obtains the local calculation results respectively provided by multiple AI computing cores 3 from the memory 2, and obtains the matrix multiplication result corresponding to the target matrix data based on the multiple local calculation results.

[0070] In the above implementation process, the AI ​​computing core obtains the adapted matrix block data from the memory based on the block storage address, determines the local calculation result corresponding to the matrix block data through matrix multiplication operation, and stores the local calculation result to the specified position of the memory based on the first storage address. The IO core obtains multiple local calculation results from the memory, and obtains the matrix multiplication result corresponding to the target matrix data by combining the multiple local calculation results, thereby improving the computing efficiency based on block calculation and result combination.

[0071] The following is an introduction to the matrix block module. Figure 7 As shown, the matrix block module 12 includes a plurality of cascaded core-grain task size submodules 121, the plurality of core-grain task size submodules 121 correspond one-to-one to a plurality of AI computing core-grains, and the core-grain task size submodule 121 includes a block size calculation unit 1211 corresponding to the M dimension, the K dimension, and the N dimension respectively;

[0072] Among them, M is the number of rows corresponding to the first matrix, K is the number of columns corresponding to the first matrix and the number of rows corresponding to the second matrix, and N is the number of columns corresponding to the second matrix.

[0073] like Figure 4 and Figure 7 As shown, the matrix block module 12 includes a plurality of core task size submodules 121 corresponding to a plurality of AI computing cores 3, and the plurality of core task size submodules 121 form a cascade relationship. For any core task size submodule 121, it is adapted to an AI computing core 3, and is used to determine the subtask size information corresponding to the adapted AI computing core 3 based on the optimal block strategy, the weight corresponding to the adapted AI computing core 3, and the number of rows and columns of the M dimension, N dimension, and K dimension.

[0074] The target matrix data of the embodiment of the present application corresponds to the first matrix and the second matrix, and the matrix multiplication circuit is used to calculate the multiplication of the first matrix by the second matrix, the number of rows corresponding to the first matrix is ​​M and the number of columns is K, and the number of rows corresponding to the second matrix is ​​K and the number of columns is N. The core task size submodule 121 includes a block size calculation unit 1211 corresponding to the M dimension, the K dimension, and the N dimension respectively, and the block size calculation unit 1211 is used to determine the size to be obtained by segmentation in the matching dimension based on the input number of rows or columns.

[0075] It should be noted that the cascade between the core task size submodules 121 in the embodiment of the present application is actually the cascade between the block size calculation units 1211 of the same dimension, that is, Figure 7 and Figure 8As shown, for multiple core-grain task size submodules 121, multiple block size calculation units 1211 in M ​​dimension form a cascade, multiple block size calculation units 1211 in K dimension form a cascade, and multiple block size calculation units 1211 in N dimension form a cascade.

[0076] Since the cascade between the core grain task size submodules 121 is the cascade between the block size calculation units 1211 of the same dimension, for the cascaded multiple core grain task size submodules 121, the block size calculation unit 1211 corresponding to the X dimension of the previous core grain task size submodule 121 is connected to the block size calculation unit 1211 corresponding to the X dimension of the subsequent core grain task size submodule 121.

[0077] The block size calculation unit 1211 includes an input port and an output port, and also includes a selector. The input port is used to input the number of rows or columns corresponding to the X dimension, and the output port is used to output the block size corresponding to the X dimension, and the output port of the block size calculation unit 1211 is the selected output port of the selector. The block size calculation unit 1211 also includes a cascade port related to cascading, and multiple block size calculation units 1211 of the same dimension form a cascade relationship based on the cascade port.

[0078] like Fig. 9A and 9B As shown, the block size calculation unit 1211 includes a selector 12111, and the selector 12111 includes a first selection input port, a second selection input port, a third selection input port and a selection output port. The first selection input port is used to input a first parameter associated with the X dimension, the second selection input port is used to input a second parameter associated with the X dimension, and the third selection input port is used to input 0; the selection output port outputs the first parameter, the second parameter or 0 based on the optimal block strategy, and the output information of the selection output port is the block size corresponding to the X dimension; the X dimension is the M dimension, the K dimension or the N dimension, and the output information of the selection output port of the block size calculation unit 1211 corresponding to the M dimension, the K dimension and the N dimension respectively constitutes the subtask size information of the AI ​​computing core particle.

[0079] Among them, the second parameter is the number of rows or columns corresponding to the X dimension, the second parameter is input by the input port of the block size calculation unit 1211, the first parameter is a first integer determined based on the second parameter and the weight corresponding to the current AI calculation core particle, or the first parameter is a second integer determined based on the second parameter and the first integers corresponding to the first Q block size calculation units 1211, and Q is the number corresponding to multiple AI calculation core particles minus 1.

[0080] Since each core task size submodule 121 matches an AI computing core, each core task size submodule 121 includes block size calculation units 1211 corresponding to M dimension, K dimension, and N dimension, respectively. Therefore, one AI computing core is adapted to block size calculation units 1211 corresponding to M dimension, K dimension, and N dimension, respectively. For any block size calculation unit 1211, it includes a selector 12111, and the block size calculation unit 1211 adapted to the last AI computing core and the block size calculation unit 1211 adapted to other AI computing cores are different in structure. That is, multiple AI computing cores can correspond to a number respectively, and the order of an AI computing core among multiple AI computing cores is determined based on the number to determine whether the AI ​​computing core is the first AI computing core or the last AI computing core.

[0081] like Fig. 9A As shown, the block size calculation unit 1211 adapted for the non-last AI computing core particle includes a selector 12111, a multiplier 12112, a rounding module 12113 and an adder 12114, wherein the input end of the multiplier 12112 is used to input the weight corresponding to the AI ​​computing core particle and the number of rows or columns corresponding to the X dimension (the second parameter associated with the X dimension), the output end of the multiplier 12112 is connected to the input end of the rounding module 12113, the rounding module 12113 is used to round down the input data, and the output end of the rounding module 12113 is connected to the first selection input port of the selector 12111 and an input port (such as input port 1) of the adder 12114.

[0082] The adder 12114 includes two input ports (such as input port 1 and input port 2) and an output port. The output port of the adder 12114 of the previous stage block size calculation unit 1211 (the cascade port of the block size calculation unit 1211, specifically the cascade port used as output) is connected to another input port (such as input port 2, which is the cascade port of the block size calculation unit 1211, specifically the cascade port used as input) of the adder 12114 of the next stage block size calculation unit 1211. For the block size calculation unit 1211 adapted to the first AI computing core particle, another input port (such as input port 2) of the corresponding adder 12114 inputs 0; for the block size calculation units 1211 adapted to other AI computing core particles different from the first AI computing core particle and the last AI computing core particle, one input port (such as input port 1) of the corresponding adder 12114 is used to input the output result of the rounding module 12113, and the other input port (such as input port 2) is used to input the output result of the adder 12114 of the previous block size calculation unit 1211.

[0083] like Fig. 9B As shown, the block size calculation unit 1211 adapted for the last AI calculation core particle includes a selector 12111 and a subtractor 12115. An input port of the subtractor 12115 inputs the number of rows or columns corresponding to the X dimension (i.e., the second parameter), and the value input to the input port is used as the minuend. Another input port of the subtractor 12115 inputs the cumulative sum of the first integers corresponding to the first Q block size calculation units 1211, and the value input to the input port is used as the subtrahend. The difference between the number of rows or columns corresponding to the X dimension and the cumulative sum of the first integers corresponding to the first Q block size calculation units 1211 is subtracted, and the difference is the output value of the output port of the subtractor 12115. The output port of the subtractor 12115 is connected to the first selection input port of the selector 12111, that is, the first selection input port of the selector 12111 is used to input the output result of the subtractor 12115.

[0084] Among them, Fig. 9A and 9B As shown, for the block size calculation unit 1211 that does not match the last AI computing core particle, the first parameter is a first integer, which is determined by rounding down based on the product of the second parameter and the weight corresponding to the current AI computing core particle; for the block size calculation unit 1211 that matches the last AI computing core particle, the first parameter is a second integer, which is determined based on the difference between the second parameter and the target parameter, and the target parameter is the cumulative sum of the first integers corresponding to the first Q block size calculation units 1211.

[0085] For the first AI computing core particle, the input port of the adder 12114 of the adapted block size calculation unit 1211 inputs 0, and the output port outputs the first integer; and the output port of the adder 12114 is the cascade port of the block size calculation unit 1211, which is used to output the first integer.

[0086] For the last AI computing core particle, one input port (the input port for providing the minuend) of the subtractor 12115 of the adapted block size calculation unit 1211 is used to input the number of rows or columns corresponding to the X dimension, and the other input port (the input port for providing the subtrahend) is used to input the cumulative sum of the first integers corresponding to the first Q block size calculation units 1211 respectively; and the input port for providing the subtrahend is the cascade port of the block size calculation unit 1211, which is used to connect to the previous block size calculation unit 1211 and to input data.

[0087] For other AI computing core particles that are different from the first AI computing core particle and the last AI computing core particle among the multiple AI computing core particles, one input port of the adder 12114 of the block size calculation unit 1211 in the X dimension is used to input the first integer output by the rounding module 12113, and the other input port is used to input the cumulative sum of the first integers corresponding to the AI ​​computing core particles located before the current AI computing core particle, and the output port of the adder 12114 of the block size calculation unit 1211 is used to output the cumulative sum of the first integers corresponding to the current AI computing core particle and the AI ​​computing core particle located before the current AI computing core particle. The input port corresponding to the adder 12114 for inputting the accumulated sum is the cascade port of the block size calculation unit 1211, which is used to connect with the previous block size calculation unit 1211 and to input data; the output port corresponding to the adder 12114 is the cascade port of the block size calculation unit 1211, which is used to connect with the next block size calculation unit 1211 and to output data.

[0088] Optionally, output information of the selection output port of the selector 12111 included in the block size calculation unit 1211 is determined based on the optimal block strategy. If the optimal block strategy indicates segmentation in the X dimension, the selection output port of the selector 12111 outputs a first parameter. At this time, the block size calculation unit 1211 determines to perform matrix segmentation in the X dimension based on the first parameter; if the optimal block strategy indicates no segmentation in the X dimension, the selection output port of the selector 12111 outputs a second parameter. At this time, the block size calculation unit 1211 determines not to segment in the X dimension; if the optimal block strategy indicates no segmentation in any of the three dimensions, the selection output port of the selector 12111 corresponding to the first AI computing core particle outputs the second parameter, and the selection output ports of the selectors 12111 corresponding to the remaining AI computing core particles output 0 (indicating that the AI ​​computing core particles are not assigned to computing tasks). At this time, the target matrix data is not segmented, and the first AI computing core particle performs matrix operations on the first matrix and the second matrix, and the other AI computing core particles do not obtain computing tasks.

[0089] In the above implementation process of the present application, the block size calculation unit of each dimension included in the core particle task size submodule inputs the row number or column number information and weight information of the adaptation dimension, performs matrix segmentation in the adaptation dimension based on the optimal block strategy, and outputs the segmentation situation adapted to the dimension, so as to determine the subtask size information corresponding to the AI ​​calculation core particle through the output results of the block size calculation units corresponding to the M, N, and K dimensions respectively.

[0090] As an optional embodiment, Fig.10As shown, the IO chip 1 also includes: a communication bus 20 connected to the matrix multiplication real-time block module 10 and a plurality of chip interconnection interfaces 30 connected to the communication bus 20; a memory 2 connected to the communication bus 20; a plurality of chip interconnection interfaces 30 are matched one-to-one with a plurality of AI computing chiplets 3, and the AI ​​computing chiplets 3 are connected to the communication bus 20 through the matching chip interconnection interfaces 30 and connected to the memory 2 through the communication bus 20.

[0091] The IO chip 1 also includes a communication bus 20 and a plurality of chip interconnection interfaces 30 ( Fig.10 4 are used as examples for illustration), the matrix multiplication real-time block module 10 is connected to the communication bus 20, specifically, the output end of the AI ​​configuration module 13 is connected to the communication bus 20, the communication bus 20 is connected to multiple core interconnection interfaces 30, and multiple core interconnection interfaces 30 are connected to multiple AI computing cores 3 in a one-to-one correspondence, and the memory 2 is connected to the communication bus 20. The AI ​​computing core 3 is connected to the matching core interconnection interface 30, and is connected to the communication bus 20 through the core interconnection interface 30. Since the communication bus 20 is connected to the memory 2, the AI ​​computing core 3 can be sequentially based on the connection with the matching core interconnection interface 30, the connection between the core interconnection interface 30 and the communication bus 20, and the connection between the communication bus 20 and the memory 2 to realize the connection between the AI ​​computing core 3 and the memory 2, and then the matrix block data can be obtained from the memory 2, and the local calculation results determined by the matrix multiplication operation based on the matrix block data are stored in the memory 2.

[0092] like Fig.11 As shown, the matrix multiplication operation circuit also includes: a target device 5 connected to the IO chip 1, the target device 5 transmits the target matrix data to the memory 2 via the IO chip 1, and sends the calculation task information of the target matrix data to the matrix multiplication real-time block module 10 of the IO chip 1.

[0093] The matrix multiplication operation circuit provided in the embodiment of the present application also includes a target device 5, such as a host. The target device 5 is connected to the IO chip 1. Since the IO chip 1 is connected to the memory 2, the target device 5 can be connected to the memory 2 based on the IO chip 1 to transmit the target matrix data corresponding to the first matrix and the second matrix to the memory 2, so as to store the target matrix data in the memory 2. While providing the target matrix data, the target device 5 also provides the calculation task information corresponding to the target matrix data, and transmits the calculation task information to the matrix multiplication real-time blocking module 10 of the IO chip 1. The matrix multiplication real-time blocking module 10 determines the optimal blocking strategy, performs matrix blocking based on the optimal blocking strategy, and determines the blocking task information corresponding to each AI calculation chip.

[0094] The above is the matrix multiplication operation circuit provided in the embodiment of the present application. The optimal block strategy is determined based on the computing task information of the target matrix data and the current chip parameter information when the target chip is running. The block task information of multiple AI computing core particles respectively matched is determined according to the optimal block strategy and the weights of multiple AI computing core particles. The AI ​​computing core particles perform block operations based on the block task information. After obtaining the local calculation results corresponding to the multiple AI computing core particles, they are combined to obtain the matrix multiplication operation result. It can be realized that the optimal block method is calculated according to the real-time state of the chip at runtime, and the computing efficiency of the accelerator based on the multi-core architecture is effectively improved. The limitations of the offline static matrix slicing strategy are solved, and the compiler overhead is reduced, so that the compiler no longer needs to perform underlying task slicing; and by considering the weight of the AI ​​computing core particles, the matrix is ​​segmented with the help of the optimal block strategy and the weight of the AI ​​computing core particles, which can effectively avoid the waste of resources caused by the need for global waiting due to load imbalance such as some AI computing core particles have heavy computing tasks and some AI computing core particles have light computing tasks; further, the matrix multiplication operation circuit provided by the present application has small area and power consumption overhead, and is easy to integrate and implement.

[0095] The present application also provides a matrix multiplication operation method, which is applied to IO core particles, such as Fig.12 As shown, the method includes:

[0096] Step 1201: determine the optimal block partitioning strategy based on the computing task information of the target matrix data to be calculated and the current chip parameter information when the target chip is running, and deploy the IO core particle and multiple AI computing core particles connected to the IO core particle on the target chip.

[0097] In an embodiment of the present application, the IO core particles and multiple AI computing core particles are deployed on the target chip, and the IO core particles are connected to the multiple AI computing core particles respectively. The IO core particles are connected to the target device (such as a host), obtain the computing task information of the target matrix data provided by the target device, and the target device provides the target matrix data to the memory. Among them, the memory is connected to the IO core particles on the target chip, the target device is connected to the IO core particles, the target device provides the target matrix data to the memory through the IO core particles, and the target matrix data to be performed on the matrix multiplication operation is stored in the memory. Since the IO core particles are deployed on the target chip, the IO core particles can obtain the current chip parameter information of the target chip when it is running in real time.

[0098] The IO core particle determines the optimal block partitioning strategy based on the computing task information provided by the target device and the current chip parameter information obtained when the target chip is running. The optimal block partitioning strategy mentioned here is the optimal strategy for performing matrix partitioning on the first matrix and the second matrix corresponding to the target matrix data determined in multiple matrix partitioning strategies. The first matrix is ​​the matrix to be multiplied by the second matrix.

[0099] Step 1202: Determine the block task information that matches the multiple AI computing core particles respectively according to the optimal block strategy and the weights corresponding to the multiple AI computing core particles respectively.

[0100] The optimal blocking strategy indicates the dimension in which the matrix is ​​to be segmented. During the actual segmentation, in addition to the optimal blocking strategy, it is also necessary to determine the weights corresponding to each AI computing core particle. Based on the optimal blocking strategy and the weights corresponding to multiple AI computing core particles, the computing tasks adapted to each AI computing core particle are determined, and then the block task information of each AI computing core particle is determined.

[0101] The weight corresponding to the AI ​​computing core can be determined according to the load, computing power and bandwidth of the AI ​​computing core, and can be adjusted dynamically. By considering the weight of the AI ​​computing core and using the optimal block strategy and the weight of the AI ​​computing core to perform matrix segmentation, the AI ​​computing core can be assigned adaptive computing tasks, effectively avoiding the waste of resources caused by global waiting due to load imbalance such as some AI computing cores have heavy computing tasks and some AI computing cores have light computing tasks.

[0102] Step 1203: Send the block task information matched by the multiple AI computing core particles to the multiple AI computing core particles, and the AI ​​computing core particles determine the matrix block data from the target matrix data based on the matched block task information.

[0103] After determining the block task information corresponding to the multiple AI computing core particles, the IO core particle sends the block task information matched by the multiple AI computing core particles to the multiple AI computing core particles. Since the storage device stores the target matrix data, the AI ​​computing core particle is connected to the memory through the IO core particle, and then the AI ​​computing core particle can obtain the matching matrix block data from the memory.

[0104] For the AI ​​computing core, based on the matching block task information and after obtaining the matrix block data from the memory through the IO core, it performs matrix multiplication on the matrix block data and outputs the local calculation results. The local calculation results output by the AI ​​computing core can be stored in the specified location of the memory through the IO core.

[0105] Step 1204: Receive local calculation results respectively fed back by multiple AI calculation core particles, which are calculated based on the matching matrix block data, and combine the multiple local calculation results into a matrix multiplication result corresponding to the target matrix data.

[0106] Based on the connection with the memory, the IO core obtains the local calculation results output by multiple AI computing cores and stored in the memory, combines the multiple local calculation results, generates the matrix multiplication result corresponding to the target matrix data, and determines the matrix multiplication result based on block operation and combination of operation results, effectively improving the calculation efficiency of matrix multiplication tasks.

[0107] The above implementation scheme determines the optimal block partitioning strategy based on the computing task information of the target matrix data and the current chip parameter information of the target chip when it is running, determines the block task information that matches multiple AI computing core particles respectively according to the optimal block partitioning strategy and the weights of multiple AI computing core particles, and the AI ​​computing core particles perform block operations based on the block task information, and after obtaining the local calculation results corresponding to the multiple AI computing core particles, they are combined to obtain the matrix multiplication result. This can realize the calculation of the optimal block partitioning method according to the real-time status of the chip at runtime, effectively improve the computing efficiency, solve the limitations of the offline static matrix block partitioning strategy, reduce the compiler overhead, and make the compiler no longer need to perform the underlying task partitioning; and by considering the weight of the AI ​​computing core particles, using the optimal block partitioning strategy and the weight of the AI ​​computing core particles to perform matrix partitioning, it can effectively avoid the waste of resources caused by the need for global waiting due to load imbalance such as some AI computing core particles have heavy computing tasks and some AI computing core particles have light computing tasks.

[0108] Optionally, determining the optimal block strategy based on the calculation task information of the target matrix data to be calculated and the current chip parameter information when the target chip is running includes:

[0109] Based on the computing task information and the current chip parameter information, determining the strategy costs corresponding to P matrix partition strategies respectively, where P is the total number of matrix partition strategies corresponding to the target matrix data;

[0110] The smallest strategy cost is selected from the P strategy costs as the final strategy cost, and the matrix partitioning strategy corresponding to the final strategy cost is the optimal partitioning strategy.

[0111] After obtaining the computing task information and the current chip parameter information, the IO core particle determines the strategy costs corresponding to the P matrix partition strategies based on the computing task information and the current chip parameter information. The P matrix partition strategies are all the matrix partition strategies corresponding to the target matrix data. After determining the P strategy costs, the final strategy costs are determined based on the P strategy costs, and the matrix partition strategy corresponding to the final strategy cost is determined as the optimal partition strategy, so as to achieve the optimal partition strategy after determining the P strategy costs.

[0112] The computing task information includes matrix size information corresponding to the first matrix and the second matrix, and the first matrix and the second matrix are matrices to be multiplied corresponding to the target matrix data; the current chip parameter information includes first chip parameter information related to communication resources and second chip parameter information related to computing power; when determining the strategy costs corresponding to the P matrix partitioning strategies respectively based on the computing task information and the current chip parameter information, it includes:

[0113] For each matrix blocking strategy, the communication time cost is obtained based on the matrix blocking strategy, the first chip parameter information and the matrix size information; the computing time cost is obtained based on the matrix blocking strategy, the second chip parameter information and the matrix size information; the strategy cost corresponding to the matrix blocking strategy is determined according to the communication time cost and the computing time cost.

[0114] For any matrix partitioning strategy among the P matrix partitioning strategies, when determining the corresponding strategy cost, the communication time cost is determined based on the first chip parameter information and the matrix size information, the calculation time cost is determined based on the second chip parameter information and the matrix size information, and the strategy cost adapted to the current strategy cost submodule is determined according to the communication time cost and the calculation time cost. When determining the strategy cost based on the communication time cost and the calculation time cost, a preset strategy can be adopted, such as selecting a larger time cost as the strategy cost in the communication time cost and the calculation time cost. Since it is necessary to ensure that the strategy cost covers the communication time cost and the calculation time cost, the larger time cost of the two is selected as the strategy cost. Alternatively, a weighted summation strategy is adopted to determine the strategy cost based on the communication time cost and the calculation time cost. At this time, it is necessary to determine the cost weights corresponding to the communication time cost and the calculation time cost, and then determine the strategy cost based on the cost weight, the communication time cost, and the calculation time cost. A direct summation strategy can also be adopted to directly determine the strategy cost based on the communication time cost and the calculation time cost. Of course, other methods can also be used to determine the strategy cost, which will not be listed here one by one.

[0115] Since the strategy cost is determined based on the communication time cost and the computing time cost, it represents the cost of the time dimension. When determining the final strategy cost among P strategy costs, selecting the smallest strategy cost among the P strategy costs as the final strategy cost can achieve the selection of the smallest time cost, and then determine the optimal blocking strategy based on the smallest time cost.

[0116] Optionally, since it is necessary to determine the block task information that matches multiple AI computing core particles respectively based on the optimal block strategy and the weights corresponding to the multiple AI computing core particles respectively, it is necessary to determine the weights corresponding to the multiple AI computing core particles respectively while determining the optimal block strategy. The first chip parameter information includes first chip specification information associated with bandwidth resources and the number of AI computing core particles, and first chip status information associated with communication load, the first chip specification information includes core particle bandwidth information characterizing the connection bandwidth between the AI ​​computing core particle and the IO core particle; the second chip parameter information includes second chip specification information associated with the peak computing power of the AI ​​computing core particle and the number of AI computing core particles, and second chip status information associated with the AI ​​computing core particle load; the method also includes:

[0117] The chip bandwidth information, AI computing chip load and AI computing chip peak computing power corresponding to the multiple AI computing chiplets are obtained, and the weights corresponding to the multiple AI computing chiplets are determined based on the obtained information; wherein the weight corresponding to the AI ​​computing chiplet is negatively correlated with the AI ​​computing chiplet load, and positively correlated with the chip bandwidth information and the AI ​​computing chip peak computing power.

[0118] In this embodiment, the current chip parameter information includes first chip parameter information related to communication resources and second chip parameter information related to computing power. The first chip parameter information includes first chip specification information associated with bandwidth resources and the number of AI computing chiplets, and first chip status information associated with communication load. The first chip specification information includes chip bandwidth information characterizing the connection bandwidth between the AI ​​computing chiplet and the IO chiplet; the second chip parameter information includes second chip specification information associated with the peak computing power of the AI ​​computing chiplet and the number of AI computing chiplets, and second chip status information associated with the AI ​​computing chiplet load.

[0119] When calculating the weights corresponding to multiple AI computing cores, the IO core obtains the core bandwidth information, AI computing core load and AI computing core peak computing power corresponding to the multiple AI computing cores in the current chip parameter information, and determines the weights corresponding to the multiple AI computing cores based on the obtained information; the weight corresponding to the AI ​​computing core is negatively correlated with the AI ​​computing core load, and positively correlated with the core bandwidth information and the AI ​​computing core peak computing power.

[0120] By extracting relevant information from the current chip parameter information and determining the weights corresponding to multiple AI computing core particles, the weights of the AI ​​computing core particles can be considered, and matrix segmentation can be performed with the help of the optimal blocking strategy and the weights of the AI ​​computing core particles to achieve the allocation of adaptive computing tasks to each AI computing core particle. This can effectively avoid the waste of resources caused by the need for global waiting due to load imbalance such as some AI computing core particles have heavy computing tasks and some AI computing core particles have light computing tasks.

[0121] Optionally, when determining the block task information that matches the multiple AI computing core particles respectively according to the optimal block strategy and the weights corresponding to the multiple AI computing core particles respectively, it includes:

[0122] Based on the optimal block division strategy, the weights corresponding to the multiple AI computing core particles, and the computing task information, subtask size information corresponding to the multiple AI computing core particles is divided;

[0123] The block task information respectively matched by the multiple AI computing core particles is configured based on the subtask size information respectively corresponding to the multiple AI computing core particles.

[0124] When calculating the block task information matching the AI ​​computing core particles, based on the optimal block strategy, the weights corresponding to the multiple AI computing core particles and the computing task information, the sub-task size information corresponding to the multiple AI computing core particles is divided out, and the sub-task size information adapted to each AI computing core particle is calculated based on the optimal block strategy and the weights of the AI ​​computing core particles.

[0125] The subtask size information is then parsed into block task information acceptable to the AI ​​computing core, which includes the block storage address corresponding to the matrix block data to be calculated, the block size information, the cutting dimension (including the dimension to be cut and the original size of the dimension), and the first storage address for storing the calculation result. The AI ​​computing core can obtain the corresponding matrix block data based on the block task information to perform matrix multiplication operations to provide local calculation results.

[0126] Among them, when calculating the subtask size information adapted to each AI computing core particle based on the optimal block strategy and the weight of the AI ​​computing core particle, it is necessary to determine the size that needs to be divided in the dimension based on the optimal block strategy for each AI computing core particle in the matching dimension, based on the number of rows or columns input and the weight of the AI ​​computing core particle, and determine the subtask size information according to the segmentation corresponding to the M dimension, K dimension, and N dimension respectively. The detailed process can be found in the relevant introduction on the matrix multiplication operation circuit side, which will not be further elaborated here.

[0127] By performing matrix multiplication operations using the above method, the optimal blocking method can be calculated according to the real-time status of the chip at runtime, effectively improving the computing efficiency, solving the limitations of the offline static matrix slicing strategy, reducing the compiler overhead, and making it unnecessary for the compiler to perform underlying task slicing; and by considering the weights of the AI ​​computing core particles, using the optimal blocking strategy and the weights of the AI ​​computing core particles to perform matrix segmentation, it is possible to effectively avoid the waste of resources caused by global waiting due to load imbalance such as some AI computing core particles having heavy computing tasks and some AI computing core particles having light computing tasks.

[0128] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A matrix multiplication circuit, characterized in that: include: An input / output IO core particle, a memory, and a plurality of artificial intelligence AI computing core particles, wherein the IO core particle and the plurality of AI computing core particles are deployed on a target chip, the IO core particle is connected to the memory and the plurality of AI computing core particles, and the memory stores target matrix data to be calculated; The IO core particle includes a matrix multiplication real-time block module, and the matrix multiplication real-time block module includes: a cost calculation module, a matrix block module and an AI configuration module; The cost calculation module determines the strategy costs corresponding to P matrix partition strategies based on the calculation task information of the target matrix data and the current chip parameter information when the target chip is running, determines the optimal partition strategy based on the P strategy costs, and determines the weights corresponding to the multiple AI calculation core particles based on the current chip parameter information, where P is the total number of matrix partition strategies corresponding to the target matrix data; The matrix partitioning module is connected to the cost calculation module, and based on the optimal partitioning strategy, the weights corresponding to the multiple AI computing core particles, and the computing task information, divides the subtask size information corresponding to the multiple AI computing core particles respectively; The AI ​​configuration module is connected to the matrix block module, and configures the block task information respectively matched by the multiple AI computing core particles based on the subtask size information respectively corresponding to the multiple AI computing core particles; The AI ​​computing core obtains matrix block data from the memory through the IO core based on the matching block task information, calculates the matrix block data and outputs local calculation results; wherein the local calculation results respectively output by the multiple AI computing cores are combined into a matrix multiplication result corresponding to the target matrix data.

2. The matrix multiplication circuit according to claim 1, characterized in that: The computing task information includes matrix size information corresponding to a first matrix and a second matrix, wherein the first matrix and the second matrix are matrices to be multiplied corresponding to the target matrix data; the current chip parameter information includes first chip parameter information related to communication resources and second chip parameter information related to computing power.

3. The matrix multiplication circuit according to claim 2, characterized in that: The cost calculation module includes a decision maker and P strategy cost submodules, each strategy cost submodule corresponds to a matrix block strategy, and the strategy cost submodule includes a communication cost unit, a calculation cost unit and a cost merging unit; The communication cost unit acquires the communication time cost based on the first chip parameter information and the matrix size information, and the calculation cost unit acquires the calculation time cost based on the second chip parameter information and the matrix size information; The cost merging unit is connected to the communication cost unit and the calculation cost unit, and determines the strategy cost adapted to the current strategy cost submodule based on the communication time cost and the calculation time cost; The decision maker is connected to the P strategy cost submodules, and determines a final strategy cost according to the strategy costs respectively corresponding to the P strategy cost submodules. The matrix partitioning strategy corresponding to the final strategy cost is the optimal partitioning strategy.

4. The matrix multiplication circuit according to claim 3, characterized in that: The first chip parameter information includes first chip specification information associated with bandwidth resources and the number of AI computing cores, and first chip status information associated with communication load; The second chip parameter information includes second chip specification information associated with the peak computing power of the AI ​​computing core particles, the number of AI computing core particles, and second chip status information associated with the AI ​​computing core particle load.

5. The matrix multiplication circuit according to claim 4, characterized in that: The first chip specification information includes chip bandwidth information representing a connection bandwidth between the AI ​​computing chip and the IO chip; The decision maker obtains chip bandwidth information, AI computing chip load, and AI computing chip peak computing power respectively corresponding to the multiple AI computing chiplets, and determines weights respectively corresponding to the multiple AI computing chiplets based on the obtained information; Among them, the weight corresponding to the AI ​​computing core is negatively correlated with the load of the AI ​​computing core, and positively correlated with the core bandwidth information and the peak computing power of the AI ​​computing core.

6. The matrix multiplication circuit according to claim 3, characterized in that: The cost merging unit adopts a preset strategy to determine the strategy cost based on the communication time cost and the computing time cost; The decision maker selects the smallest strategy cost among P strategy costs as the final strategy cost.

7. The matrix multiplication circuit according to claim 2, characterized in that: The computing task information also includes: matrix storage addresses corresponding to the first matrix and the second matrix and result storage addresses; The AI ​​configuration module parses the subtask size information into block task information acceptable to the AI ​​computing core particle, wherein the block task information includes a block storage address corresponding to the matrix block data to be calculated, block size information, cutting dimension, and a first storage address for storing local calculation results; the block storage address is determined based on the matrix storage address, and the first storage address is determined based on the result storage address.

8. The matrix multiplication circuit according to claim 7, characterized in that: The AI ​​computing core obtains matrix block data from the memory based on the block storage address; After determining the local calculation result corresponding to the matrix block data, the local calculation result is stored in a designated location of the memory based on the first storage address.

9. The matrix multiplication circuit according to claim 2, characterized in that: The matrix block module includes a plurality of cascaded core-grain task size sub-modules, the plurality of core-grain task size sub-modules correspond one-to-one to the plurality of AI computing core-grains, and the core-grain task size sub-module includes block size calculation units corresponding to M, K, and N dimensions respectively; Among them, M is the number of rows corresponding to the first matrix, K is the number of columns corresponding to the first matrix and the number of rows corresponding to the second matrix, and N is the number of columns corresponding to the second matrix.

10. The matrix multiplication circuit according to claim 9, characterized in that: The block size calculation unit includes: a selector; The selector comprises a first selection input port, a second selection input port, a third selection input port and a selection output port, wherein the first selection input port is used to input a first parameter associated with the X dimension, the second selection input port is used to input a second parameter associated with the X dimension, and the third selection input port is used to input 0; the selection output port outputs the first parameter, the second parameter or 0 based on the optimal block strategy, and the output information of the selection output port is the block size corresponding to the X dimension; Wherein, the X dimension is M dimension, K dimension or N dimension, and the output information of the selected output port of the block size calculation unit corresponding to the M dimension, K dimension and N dimension respectively constitutes the subtask size information of the AI ​​calculation core particle; The second parameter is the number of rows or columns corresponding to the X dimension, the first parameter is a first integer determined based on the second parameter and the weight corresponding to the current AI computing core particle, or the first parameter is a second integer determined based on the second parameter and the first integers corresponding to the first Q block size calculation units, and Q is the number corresponding to the multiple AI computing core particles minus 1.

11. The matrix multiplication circuit according to claim 10, characterized in that: For the cascaded multiple core-grain task size submodules, the block size calculation unit corresponding to the X dimension of the previous core-grain task size submodule is connected to the block size calculation unit corresponding to the X dimension of the subsequent core-grain task size submodule; Among them, for the first AI computing core particle, the block size calculation unit of the X dimension outputs the first integer, and for the last AI computing core particle, the block size calculation unit of the X dimension inputs the cumulative sum of the first integers corresponding to the first Q block size calculation units of the X dimension.

12. The matrix multiplication circuit according to claim 11, characterized in that: If the block size calculation unit does not match the last AI computing core particle, the first parameter is the first integer, which is determined by rounding down the product of the second parameter and the weight corresponding to the current AI computing core particle; If the block size calculation unit matches the last AI calculation core particle, the first parameter is the second integer, the second integer is determined based on the difference between the second parameter and the target parameter, and the target parameter is the cumulative sum of the first integers corresponding to the first Q block size calculation units respectively.

13. The matrix multiplication circuit according to claim 10, characterized in that: If the optimal block partitioning strategy indicates partitioning in the X dimension, the selection output port of the selector outputs the first parameter; If the optimal block partitioning strategy indicates no partitioning in the X dimension, the selection output port of the selector outputs the second parameter; If the optimal block partitioning strategy indicates no partitioning in any of the three dimensions, the selection output port of the selector corresponding to the first AI computing core particle outputs the second parameter, and the selection output ports of the selectors corresponding to the remaining AI computing core particles output 0.

14. The matrix multiplication circuit according to any one of claims 1 to 13, characterized in that: The IO core also includes: a communication bus connected to the matrix multiplication real-time block module and a plurality of core interconnection interfaces connected to the communication bus; The memory is connected to the communication bus; The multiple core particle interconnection interfaces are matched one by one with the multiple AI computing core particles, and the AI ​​computing core particles are connected to the communication bus through the matched core particle interconnection interfaces and connected to the memory through the communication bus.

15. The matrix multiplication circuit according to any one of claims 1 to 13, characterized in that: The matrix multiplication operation circuit also includes: a target device connected to the IO core particle, the target device transmits the target matrix data to the memory via the IO core particle, and sends the calculation task information of the target matrix data to the matrix multiplication real-time blocking module of the IO core particle.

16. A matrix multiplication method, applied to an input / output IO chip, characterized in that: include: The optimal block strategy is determined based on the calculation task information of the target matrix data to be calculated and the current chip parameter information when the target chip is running, including: based on the calculation task information and the current chip parameter information, determining the strategy costs corresponding to P matrix block strategies respectively, P is the total number of matrix block strategies corresponding to the target matrix data; selecting the smallest strategy cost among the P strategy costs as the final strategy cost, and the matrix block strategy corresponding to the final strategy cost is the optimal block strategy; The IO core particle and a plurality of AI computing core particles connected to the IO core particle are deployed on the target chip; Determining the block task information that the multiple AI computing core particles match respectively according to the optimal block strategy and the weights that the multiple AI computing core particles respectively correspond to, including: dividing the subtask size information that the multiple AI computing core particles respectively correspond to based on the optimal block strategy, the weights that the multiple AI computing core particles respectively correspond to, and the computing task information; configuring the block task information that the multiple AI computing core particles respectively match based on the subtask size information that the multiple AI computing core particles respectively correspond to; Sending the block task information respectively matched by the multiple AI computing core particles to the multiple AI computing core particles; Receiving local calculation results respectively fed back by the multiple AI calculation core particles and calculated based on the matched matrix block data, and combining the multiple local calculation results into a matrix multiplication result corresponding to the target matrix data; The matrix block data matched by the AI ​​computing core particle is determined by the AI ​​computing core particle from the target matrix data based on the matched block task information.

17. The method according to claim 16, characterized in that The computing task information includes matrix size information corresponding to a first matrix and a second matrix, wherein the first matrix and the second matrix are matrices to be multiplied corresponding to the target matrix data; the current chip parameter information includes first chip parameter information related to communication resources and second chip parameter information related to computing power; The determining, based on the computing task information and the current chip parameter information, the strategy costs corresponding to the P matrix partition strategies respectively includes: For each matrix block strategy, a communication time cost is obtained based on the matrix block strategy, the first chip parameter information and the matrix size information, and a calculation time cost is obtained based on the matrix block strategy, the second chip parameter information and the matrix size information; The strategy cost corresponding to the matrix partitioning strategy is determined according to the communication time cost and the calculation time cost.

18. The method according to claim 17, characterized in that The first chip parameter information includes first chip specification information associated with bandwidth resources and the number of AI computing core particles, and first chip status information associated with communication load, wherein the first chip specification information includes core particle bandwidth information characterizing the connection bandwidth between the AI ​​computing core particle and the IO core particle; the second chip parameter information includes second chip specification information associated with the peak computing power of the AI ​​computing core particle and the number of AI computing core particles, and second chip status information associated with the AI ​​computing core particle load; The method further comprises: Obtain chip bandwidth information, AI computing chip load, and AI computing chip peak computing power respectively corresponding to the multiple AI computing chiplets, and determine weights respectively corresponding to the multiple AI computing chiplets based on the obtained information; Among them, the weight corresponding to the AI ​​computing core is negatively correlated with the load of the AI ​​computing core, and positively correlated with the core bandwidth information and the peak computing power of the AI ​​computing core.

Images