A conversion method and circuit for implementing convolution-to-matrix multiplication online

Abstract

A conversion method and circuit for implementing convolution-to-matrix multiplication online, the circuit comprising: a storage unit for caching feature map data and convolution kernel parameters, and reading convolution data according to a memory access instruction generated by a control unit, and outputting a one-dimensional vector unfolded in a preset order; the control unit is used for generating the memory access instruction according to the convolution kernel parameters, the size parameters of the feature map data and the size parameters of the non-blocking full permutation network; and generating the exchange instruction according to the convolution kernel parameters, the one-dimensional vector and the size parameters of the non-blocking full permutation network; the non-blocking full permutation network is used for rearranging and converting the one-dimensional vector received from the storage unit according to the exchange instruction, and generating matrix multiplication data; wherein the exchange instruction contains the network structure transformation information corresponding to the non-blocking full permutation network in each conversion. Thus, the artificial intelligence processor calculation efficiency and performance can be significantly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep learning, and in particular to a conversion method and circuit for online implementation of convolution-to-matrix multiplication. BACKGROUND

[0002] Convolution operation conversion to matrix multiplication (im2col) is a core technology for implementing efficient convolution calculation in deep learning. By expanding the input convolution data and convolution kernel in local regions into matrix form, matrix operations can utilize dedicated matrix acceleration hardware to achieve fast calculation, enabling efficient convolution calculation. The core idea of convolution-to-matrix multiplication is to map nonlinear operations to a linear algebra framework, facilitating the use of efficient numerical calculation methods to accelerate deep learning model training and inference, thereby significantly improving processor calculation speed and hardware utilization. In summary, convolution-to-matrix multiplication has been widely used in deep learning frameworks such as TensorFlow and PyTorch for bottom numerical calculation.

[0003] In convolutional neural networks, padding operations are often required on input data (such as images), which involves adding an extra ring of padding data to the boundaries of input data. The main purpose of this operation is to control the output size of the convolution layer, preserve edge information and features, adapt to specific network structures, and maintain model symmetry, so a reasonable and efficient padding strategy is crucial for model performance.

[0004] Currently, the offline method of pre-converting convolution to matrix and caching it in memory is commonly used in processors, which often consumes a large amount of memory resources. In particular, when the input convolution data needs to be padded, a large amount of memory space is consumed to store the padding data. In addition, existing convolution-to-matrix multiplication implementation schemes require multiple repeated access to storage space, occupying memory bandwidth and affecting memory efficiency. SUMMARY

[0005] The present application aims to provide a conversion method and circuit for online implementation of convolution-to-matrix multiplication. Based on a non-blocking full permutation network (such as Benes network), the input convolution data is rearranged to realize the mapping process from convolution data to matrix data. This conversion circuit can complete data conversion online during data transmission, without the need for repeated memory access operations on the original convolution data, and without the consumption of additional memory resources to cache the converted matrix data. This can significantly improve the calculation efficiency and performance of artificial intelligence processors.

[0006] To achieve the above object, the conversion circuit for implementing convolution-to-matrix multiplication online according to an embodiment of the present application can include: a storage unit, a control unit and a non-blocking full permutation network; the storage unit is configured to cache feature map data and convolution kernel parameters, read the convolution data according to a memory access instruction generated by the control unit, and output a one-dimensional vector unfolded in a preset order; the control unit is configured to generate the memory access instruction according to the convolution kernel parameters, the size parameters of the feature map data and the size parameters of the non-blocking full permutation network, and generate a swap instruction according to the convolution kernel parameters, the one-dimensional vector and the size parameters of the non-blocking full permutation network; the non-blocking full permutation network is configured to perform online rearrangement conversion on the one-dimensional vector received from the storage unit according to the swap instruction, and generate matrix multiplication data; wherein the swap instruction includes network structure transformation information corresponding to the non-blocking full permutation network in each conversion.

[0007] In an embodiment of the present application, the control unit is specifically configured to calculate the size of the matrix multiplication data according to the size and step length in the one-dimensional vector and the convolution kernel parameters, determine the network structure transformation information according to the size of the matrix multiplication data and the calculation parallelism of the non-blocking full permutation network, and generate the swap instruction according to the network structure transformation information.

[0008] In an embodiment of the present application, the non-blocking full permutation network is further configured to perform value compensation on the output port of the non-blocking full permutation network that lacks valid data when the calculation parallelism of the non-blocking full permutation network is not fully utilized.

[0009] In an embodiment of the present application, the conversion circuit further includes at least one first multiplexing module arranged at an input port of the non-blocking full permutation network; wherein the control unit is further configured to control at least one first multiplexing module to select a padding value or the convolution data at the corresponding input port of the non-blocking full permutation network when the convolution data needs to be padded.

[0010] In an embodiment of the present application, the conversion circuit further includes a plurality of second multiplexing modules arranged at respective output ports of the non-blocking full permutation network; wherein the control unit is further configured to control the second multiplexing modules to select a padding value or the matrix multiplication data at the corresponding output ports of the non-blocking full permutation network when the convolution data needs to be padded.

[0011] In an embodiment of the present application, the control unit is further configured to control the storage unit to read the feature map data in blocks to obtain the convolution data and unfold the convolution data into a one-dimensional vector in a preset order through the memory access instruction when the size of the feature map data exceeds the input capacity of the non-blocking full permutation network.

[0012] In one embodiment of this application, the control unit is further configured to: control the storage unit to read the non-overlapping part of the feature map data with the previous convolutional data according to the overlap relationship of adjacent convolutional data, and combine it with the overlapping data temporarily stored in the input port of the non-blocking full permutation network to generate the current convolutional data.

[0013] In one embodiment of this application, the non-blocking permutation network is a Benes network.

[0014] In one embodiment of this application, a method for online convolution to matrix multiplication is also provided. The method includes: generating a memory access instruction based on the size parameters of a non-blocking fully permuted network and the size parameters of the convolution kernel parameters and feature map data cached in a storage unit; controlling the storage unit to read convolution data and output a one-dimensional vector unfolded in a preset order through the memory access instruction; generating a swap instruction based on the convolution kernel parameters, the one-dimensional vector, and the size parameters of the non-blocking fully permuted network; controlling the non-blocking fully permuted network to perform online rearrangement transformation on the input one-dimensional vector to generate a matrix data stream through the swap instruction; wherein the swap instruction includes network structure transformation information of the non-blocking fully permuted network in each transformation.

[0015] In one embodiment of this application, the method further includes: when the feature map data needs edge padding, controlling at least one first selection module disposed at the input port of the non-blocking full permutation network to select a padding value or the convolutional data at the input port corresponding to the non-blocking full permutation network.

[0016] In one embodiment of this application, the method further includes: when the feature map data needs edge padding, controlling multiple second selection modules disposed at each output port of the non-blocking permutation network to select padding values ​​or matrix multiplication data at the corresponding input port of the non-blocking permutation network.

[0017] In one embodiment of this application, controlling the storage unit to read convolutional data through the memory access instruction and unfolding the convolutional data into a one-dimensional vector in a preset order includes: when the feature map data size exceeds the input capacity of the non-blocking full permutation network, controlling the storage unit to read the feature map data in blocks through the memory access instruction to obtain convolutional data, and unfolding the convolutional data into a one-dimensional vector in a preset order; wherein, the size of the convolutional data is less than or equal to the input capacity of the non-blocking full permutation network.

[0018] In one embodiment of this application, controlling the storage unit to read feature map data in blocks to obtain convolutional data through the memory access instruction includes: controlling the storage unit to read the non-overlapping part of the feature map data with the previous convolutional data according to the overlap relationship of adjacent convolutional data, and combining it with the overlapping data temporarily stored in the input port of the non-blocking full permutation network to generate the current convolutional data.

[0019] In one embodiment of this application, a chip is also provided, comprising a matrix operation unit and the above-described conversion circuit; the matrix operation unit is used to perform multiplication and accumulation operations on the matrix multiplication data output in real time by the non-blocking permutation network.

[0020] In one embodiment of this application, a board card is also provided, including the chip described above.

[0021] In one embodiment of this application, an electronic device is also provided, which is equipped with the aforementioned board or the aforementioned chip.

[0022] The beneficial technical effects of this application are: greatly reducing processor memory access overhead and storage resource overhead, and effectively improving processor computing efficiency and performance.

[0023] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the present application. Other features and aspects of the present application will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, do not constitute a limitation thereof. In the drawings:

[0025] Figure 1A This is a schematic diagram of the conversion circuit provided in an embodiment of this application;

[0026] Figure 1B This is a schematic diagram of intermediate-level cross-connection in a non-blocking full-permutation network provided in an embodiment of this application;

[0027] Figure 2 This is a schematic diagram of the convolutional data unrolling logic provided in an embodiment of this application;

[0028] Figure 3A This is a schematic diagram illustrating the input principle of convolutional data unrolling provided in an embodiment of this application;

[0029] Figure 3B This is a schematic diagram illustrating the principle of converting convolutional data to matrix multiplication data according to an embodiment of this application;

[0030] Figure 4This is a schematic diagram of the complement logic provided in an embodiment of this application;

[0031] Figure 5A This is a schematic diagram of edge-filled convolutional data block segmentation logic provided in an embodiment of this application;

[0032] Figure 5B A schematic diagram illustrating the principle of adding selection logic to the input port provided in an embodiment of this application;

[0033] Figure 5C A schematic diagram illustrating the principle of adding selection logic to the output port provided in an embodiment of this application;

[0034] Figure 6A This is a schematic diagram illustrating the principle of large-size convolutional data segmentation in one embodiment of this application;

[0035] Figure 6B This is a schematic diagram illustrating the principle of reading segmented convolutional data according to an embodiment of this application;

[0036] Figure 6C This is a schematic diagram illustrating the principle of feature map data reading provided in an embodiment of this application;

[0037] Figure 7A This is a schematic diagram of a conversion method provided in an embodiment of this application;

[0038] Figure 7B This is a schematic diagram of the processing chip provided in one embodiment of this application;

[0039] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0040] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0041] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0042] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0043] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed description. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0044] Please refer to Figure 1A As shown, an embodiment of this application provides a conversion circuit for online implementation of convolution to matrix multiplication, comprising: a storage unit, a control unit, and a non-blocking permutation network;

[0045] The storage unit is used to cache feature map data and convolution kernel parameters, and read convolution data according to the memory access instructions generated by the control unit, and output a one-dimensional vector unfolded in a preset order;

[0046] The control unit is configured to generate the memory access instruction based on the convolution kernel parameters, the size parameters of the feature map data, and the size parameters of the non-blocking permutation network; and to generate a swap instruction based on the convolution kernel parameters, the one-dimensional vector, and the size parameters of the non-blocking permutation network.

[0047] The non-blocking permutation network is used to perform online rearrangement transformation on the one-dimensional vector received from the storage unit according to the exchange instruction, and generate matrix multiplication data;

[0048] The exchange instruction contains network structure transformation information corresponding to each transition of the non-blocking full permutation network.

[0049] In the above embodiments, the non-blocking fully permuted network can be a strictly non-blocking network or a generalized non-blocking network such as the Benes network or the Clos network; the conversion circuit rearranges the input convolutional data based on the non-blocking fully permuted network, thereby realizing the mapping process from convolutional data to matrix data. This conversion circuit can complete the data conversion online during data transmission, without requiring repeated memory access operations on the feature map data, and without consuming additional memory resources to cache the converted matrix data. It can significantly improve the computational efficiency and performance of artificial intelligence processors and has great practical value in the field of artificial intelligence processors.

[0050] The non-blocking permutation network can adopt a three-level topology, including input ports, intermediate-level cross-connections, and output ports. The number of input ports and output ports is equal to the length N of the one-dimensional vector, and the intermediate level is configured as a cross-connection. During the data unpacking process, the input convolutional data is unpacked into a one-dimensional vector in row-major or column-major order. Then, based on the convolution kernel size, the convolutional data is mapped in real-time to matrix multiplication data of a matrix structure, which is directly fed into the matrix operation unit for matrix operations. The size of the non-blocking fully permutation network can be set based on the maximum supported convolutional kernel size and computational parallelism. For example, if the maximum supported convolutional kernel element size is 4, then the maximum supported computational parallelism during the conversion of convolutional data to matrix multiplication data is also 4. The size of the non-blocking fully permutation network would then be 16 input ports and 16 output ports. Based on this 16×16 size, all convolutional data with kernel elements less than or equal to 4 (sizes such as 2×2, 1×2, 2×1, 1×3, 3×1, 1×4, 4×1, etc.) can be converted from convolutional data to matrix multiplication data using the non-blocking fully permutation network. Please refer to [reference needed]. Figure 1B As shown, since the connection relationship between the input and output ports of a non-blocking permutation network is not unique, intermediate-level crossovers can be adjusted accordingly based on network structure transformation information, thereby changing the input and output transmission paths to achieve the desired result. Figure 1B The sequential input A shown 0，0 To A 3，3 The output order is different; the network structure transformation information refers to the connection relationship of the multi-level interconnection network in the non-blocking full permutation network. In this application, the transmission path from input to output can be dynamically reconstructed by changing the connection relationship through the network structure transformation information. The specific application of this control logic will be described in detail in subsequent embodiments, and will not be described in detail here; the above-mentioned conversion method provided by this application can realize zero-buffered data flow control, which improves computational efficiency and reduces memory bandwidth usage compared with traditional offline data reconstruction methods.

[0051] In one embodiment, the control unit is specifically configured to: calculate the size of the matrix multiplication data based on the size and stride in the one-dimensional vector and the convolution kernel parameters, determine the network structure transformation information based on the size of the matrix multiplication data and the computational parallelism of the non-blocking full permutation network, and generate the exchange instruction based on the network structure transformation information.

[0052] Specifically, in some implementations of this embodiment, the unfolding logic for expanding the read convolutional data into a one-dimensional vector is as follows: The m×n convolutional data can be unfolded one-dimensionally in a certain order; this order can be column-first then row-first or row-first then column-first, etc. Figure 2As shown, an m×n convolutional data is unfolded into a one-dimensional vector of length m×n. This process A i，j In this context, i and j represent the row and column coordinates, respectively. Figure 2 The coordinates of the origin point are (0, 0), representing the first row and first column. Subsequently, the coordinates of the first row and second column are (0, 1), where 0 represents the first row and 1 represents the second column, and so on. A i，j When i and j are m and n respectively, the coordinates of the m-th row and n-th column are (m-1, n-1). This can be obtained by unfolding the m×n convolutional data into a one-dimensional vector using a row-first, column-second arrangement. Figure 2 The right side shows a one-dimensional vector of length m×n, ranging from (0,0) to (m-1,n-1). It is worth noting that there can be many possible arrangements of the one-dimensional unfolded convolutional data; any definite arrangement can be used. The example only shows one such arrangement.

[0053] Taking a 16×16 non-blocking permutation network as an example, if the kernel size of the input feature map data is 2×2 with a stride of 1, and the feature map data size is 4×4, then the feature map data size (4×4) does not exceed the input scale (16) of the non-blocking permutation network. Therefore, feature map data does not need to be segmented; it can be expanded as the read convolutional data to obtain a one-dimensional vector. This one-dimensional vector is then used as the input data for the non-blocking permutation network. Since the computational parallelism of the non-blocking permutation network is 4, nine input data groups are generated with four input data groups as a set, forming a 9×4 data matrix. Given that the non-blocking permutation network has 16 input ports, the 9×4 data matrix can be segmented into three inputs using four input data groups as the actual input data for each non-blocking permutation network operation. Thus, the number of transformations (3) and the input data for each transformation can be determined. For details, please refer to... Figure 3A As shown, in the process of online convolution to matrix multiplication, assuming the size of the input feature map data is 4×4, since 4×4 does not exceed 16, there is no need to divide the feature map data into blocks. The feature map data is directly used as the read convolution data to perform one-dimensional vector expansion, resulting in a one-dimensional vector of length 16, which is used as the input of the full permutation network.

[0054] The convolution kernel size is 2×2, and the stride is 1. The principle of converting convolutional data into matrix multiplication data is as follows: Figure 3B As shown, this means that the 4×4 feature map data needs to be converted into a 9×4 matrix to participate in matrix multiplication. The parallelism is 4, therefore the matrix operation needs to be completed in 3 steps. Figure 3B As shown, after the matrix is ​​expanded into a one-dimensional vector and input into the permutation network, the network needs to perform three transformations (i.e., three internal network reconnections) to output the complete mapped matrix multiplication data in three steps (three clock cycles). Figure 3BAs shown, step1 represents the output result of the first beat, step2 represents the output result of the second beat, and step3 represents the output result of the third beat. During this process, A... 0，0 To A 3，3 This refers to reading the input data obtained from the one-dimensional vector. In steps 1, 2, and 3, each row of data is a constructed input data group. Step 1, 2, and 3 represent the number of transformations. The input data for each transformation is shown in steps 1, 2, and 3.

[0055] In some embodiments of this application, the number of transformations required for a single input data and the network structure for each transformation can be indicated by providing a switching instruction to the non-blocking permutation network. A single switching instruction can indicate multiple transformations, or a single switching instruction can indicate a single transformation. In the aforementioned embodiments, one or three switching instructions can be used to control the permutation network to complete three transformations and output.

[0056] In one embodiment, the non-blocking permutation network is further used to: fill in the missing valid data output ports of the non-blocking permutation network when the computational parallelism of the non-blocking permutation network is not fully utilized.

[0057] In some embodiments of this example, the non-blocking fully permutation network is further used to perform padding when extracting input data when the computational parallelism of the convolution kernel is less than that of the non-blocking fully permutation network; see details for further information. Figure 4 As shown, when the convolution kernel size is 2×1 and the stride is 1, the parallelism of the non-blocking fully permuted network is 4. Therefore, the last two bits need to be padded. The padded data can be zero or other constant values ​​that will not cause computational confusion, such as... Figure 4 As shown, when the feature map data is read, features such as (0,0), (0,1), (0,1), (0,2), (0,2), and (0,3) are read sequentially with a computational parallelism of 2 and a length of 1. Since the computational parallelism of the convolutional kernel is less than the computational parallelism of the non-blocking full permutation network is 4, the difference of two bits needs to be padded, which is the structure of Step 1. At this time, the structure is a combination of reading features and zero values.

[0058] In one embodiment of this application, the conversion circuit further includes at least one first selection module disposed at the input port of the non-blocking full permutation network; wherein, the control unit is further configured to control at least one first selection module to select a padding value or the convolutional data at the input port corresponding to the non-blocking full permutation network when the convolutional data needs to be edge-padded.

[0059] Furthermore, the conversion circuit also includes multiple second selection modules, which are respectively disposed at each output port of the non-blocking full permutation network; wherein, the control unit is also used to control the second selection modules to select the padding value or the matrix multiplication data at the output port corresponding to the non-blocking full permutation network when the convolution data needs to be edge-padded.

[0060] In some implementations of this embodiment, the first selection module and the second selection module may be selection modules with the same structure or similar logic; due to the difference in their setting positions, they are distinguished here by "first" and "second," and "first" and "second" do not impose any structural or functional limitations on them. In actual work, the application of convolutional data varies, and in some scenarios, edge padding of the convolutional data is required. The logic for converting convolutional data to matrix multiplication data is as follows: Taking a convolution kernel size of 2×2, a stride of 1, and edge padding as an example, 4×4 convolutional data, after edge padding, can be converted into a 25×4 matrix for matrix multiplication using the aforementioned logic. The parallelism is 4, therefore, 7 transformations are required to complete the matrix operation. Please refer to... Figure 5A As shown, after the edge-padded convolutional data is expanded into a one-dimensional vector and input into the full permutation network, the network needs to perform seven transformations (i.e., seven internal network switching and reconnection operations) to output the complete mapped matrix multiplication data in seven steps (seven steps). Step 1 represents the output result of the first step, Step 2 represents the output result of the second step, Step 3 represents the output result of the third step, and so on.

[0061] In this scenario, to improve computational efficiency, a selection logic can be added before the input port of the non-blocking full permutation network, or after the output port of the non-blocking full permutation network. The function of this selection logic is to choose whether the input is padded or convolutional data, or to choose whether the output is padded or matrix multiplication data. Figure 5B As shown, by adding selection logic at the input end, the padding value 0 and the convolutional data A0 are selected and provided to the first input port, and so on. Other input ports can also use this selection logic for selection. Please refer to [further details omitted]. Figure 5C As shown, by adding selection logic to the output, the output result (either the fill value or the matrix multiplication data) is selected and output. Similarly, other output ports can also use this selection logic for filtering. Please refer to [further details]. Figure 5A As shown, in the first cycle, the multiplexing logic corresponding to output ports 1-3, 5-6, 9-10, and 13-14 selects constant output, while the remaining output ports select data output for network transmission.

[0062] It should be noted that, due to the characteristics of the permutation network, multiplexing logic can be added to only one input port. After a constant value is input through the multiplexing logic, it is copied by the permutation network and output as a constant value at the appropriate output port. Similarly, in this embodiment, the constant value also includes zero or other constant values ​​that will not cause computational confusion. Another implementation method is to add multiplexing logic to each input port, such as... Figure 5A and Figure 5B As shown, in the first cycle, the selection logic of the input ports corresponding to elements A0, A1, A2, and A3 in the convolutional data selects elements from the convolutional data as input, while the selection logic of other input ports selects constant values ​​as input (or, some selection logic of other input ports selects constant values ​​as input, and after being copied by the full permutation network, the constant values ​​are output at appropriate output ports; the remaining selection logic of other input ports selects elements from the convolutional data as input, but the full permutation network does not output these elements). Figure 5B and Figure 5C In the diagram, the dashed arrow perpendicular to the checkmarks refers to the input logic of the control command, which can be controlled by a control unit in actual operation; A0, A1, A... m*n-1 B0, B1, B m-1 "etc." refers to the relevant data in the convolutional data, namely A in the aforementioned attached figure. 0，0 A 0，1 The above is only for the purpose of understanding how the selection logic is used, and does not involve any restrictions on the input or output data.

[0063] In one embodiment of this application, the control unit is further configured to: when the feature map data size exceeds the input capacity of the non-blocking full permutation network, control the storage unit to read the feature map data in blocks to obtain convolutional data through the memory access instruction, and unfold the convolutional data into a one-dimensional vector in a preset order.

[0064] In some implementations of this embodiment, the size of the feature map data may be larger than the number of input ports of the non-blocking fully permutation network. In this case, the feature map data needs to be segmented to obtain convolutional data with a computational parallelism less than or equal to that of the non-blocking fully permutation network, thereby batch-converting the convolutional data to obtain the matrix multiplication data corresponding to the feature map data; please refer to Figure 6AAs shown, taking a feature map data of size 8×10 as an example; the non-blocking full permutation network is 16×16, and its computational parallelism is 4. The size of the feature map data is larger than the 16 input ports of the non-blocking full permutation network. Therefore, the 8×10 feature map data needs to be divided into blocks to obtain multiple 4×4 convolution data to correspond to the 16 input ports of the non-blocking full permutation network. For each convolution data, it can be converted into a 9×4 matrix to participate in matrix multiplication operations in the manner described in the previous embodiment. The conversion calculation process has been described in the previous embodiment and will not be described in detail here.

[0065] In one embodiment of this application, the control unit is further configured to: control the storage unit to read the non-overlapping part of the feature map data with the previous convolutional data according to the overlap relationship of adjacent convolutional data, and combine it with the overlapping data temporarily stored in the input port of the non-blocking full permutation network to generate the current convolutional data.

[0066] In the above embodiment, since the convolution kernel size of the convolutional data is 2×2 and the stride is 1, the data read in the first and second reads partially overlap in a preset direction during block processing. Therefore, for efficiency considerations, only the non-overlapping data can be read in the second read, while the overlapping data reuses the temporary data at the input of the non-blocking full permutation network, and the non-overlapping data is provided to other input ports of the non-blocking full permutation network. Please refer to [further details omitted]. Figure 6A and Figure 6B As shown, its block processing flow can be as follows:

[0067] 1. Since the input to the non-blocking full permutation network is 16, it can only process 4×4 feature map data at a time. Therefore, the convolutional data corresponding to the feature map data needs to be read in blocks. Figure 6A In this process, convolutional data is read in column-major order.

[0068] 2. First, read A from the feature map data. 0，0 Start by reading a 4×4 convolutional data and passing it into the input port of the non-blocking permutation network; after the 4×4 convolutional data completes the convolution-to-matrix multiplication operation, read the second 4×4 convolutional data in the column direction.

[0069] 3. Since the convolution kernel size is 2×2, the convolutional data read in the first reading will partially overlap with the convolutional data read in the second reading in the column direction. Therefore, in the second reading, only the non-overlapping data needs to be read, and the overlapping data is temporarily stored at the input port of the non-blocking full permutation network. The read non-overlapping data is passed to the other input ports of the non-blocking full permutation network. In the second reading, A... 3，0 A 3，1 A3，2 and A 3，3 It refers to the reused overlapping data.

[0070] 4. The third convolutional data read only contains a portion of valid data. The invalid portion is padded with 0s at the input port of the non-blocking full permutation network. Therefore, after removing the multiplexed data, only A needs to be loaded. 7，0 A 7，1 A 7，2 and A 7，3 These four sets of data;

[0071] 5. Repeat steps 2-4 to begin loading the convolutional data for the second column, such as... Figure 6B The fourth read is shown.

[0072] Please refer to the overall information. Figure 6C As shown, in the above process, for feature map data of size x×y, a non-blocking full permutation network with (m×n) input and output ports is used, where x×y>m×n; the convolution kernel size is h×w, and the convolution stride is 1. During the convolution-to-matrix multiplication process, the feature map data is divided into m×n convolution data and fed into the non-blocking full permutation network. After one convolution data completes the convolution-to-matrix multiplication in the non-blocking full permutation network, the next convolution data is selected according to the column direction, that is, the convolution data shifted down by h-1, and transported to the input port of the non-blocking full permutation network. Considering data reuse, except for the first convolution data in each column, subsequent convolution data only need to load the non-overlapping data. The overlapping data directly reuses the temporary data stored in the input port of the non-blocking full permutation network.

[0073] Therefore, the efficient online conversion logic for convolution to matrix multiplication provided in this application is based on a non-blocking permutation network to rearrange the input convolutional data, thereby achieving efficient online conversion of convolutional data to matrix data, greatly reducing processor memory access overhead and storage resource overhead, and effectively improving processor computing efficiency and performance.

[0074] Please refer to Figure 7A As shown, in one embodiment of this application, a method for online implementation of convolution to matrix multiplication is also provided, the method comprising:

[0075] S701: Generate a memory access instruction based on the size parameters of the non-blocking fully permuted network and the size parameters of the convolution kernels and feature map data cached in the storage unit. Control the storage unit to read the convolution data through the memory access instruction and output a one-dimensional vector unfolded in a preset order.

[0076] S702: Generate a swap instruction based on the convolution kernel parameters, the one-dimensional vector, and the size parameters of the non-blocking permutation network; control the non-blocking permutation network to perform online rearrangement transformation on the input one-dimensional vector to generate a matrix data stream through the swap instruction;

[0077] The exchange instruction contains network structure transformation information corresponding to each transition of the non-blocking full permutation network.

[0078] Since the principle behind this method is similar to that of the conversion circuit, the implementation of this circuit can be found in the implementation of the circuit method, and the repetitive parts will not be repeated.

[0079] In the above embodiments, controlling the non-blocking permutation network to perform online rearrangement transformation on the input one-dimensional vector through the exchange instruction to generate matrix multiplication data includes: when the computational parallelism of the non-blocking permutation network is not fully utilized, the non-blocking permutation network fills in the missing valid data output ports, and the matrix multiplication data is obtained based on the filled-in output data of the non-blocking permutation network. For specific implementation details, please refer to the foregoing... Figure 4 The corresponding circuit structures are shown in the embodiments, which will not be described in detail here.

[0080] Furthermore, the method further includes, when the convolutional data requires edge padding, controlling at least one first selection module disposed at an input port of the non-blocking full permutation network to select a padding value or the convolutional data at the corresponding input port of the non-blocking full permutation network. Also, when the convolutional data requires edge padding, controlling multiple second selection modules disposed at each output port of the non-blocking full permutation network to select a padding value or the matrix multiplication data at the corresponding input port of the non-blocking full permutation network. See the foregoing for details. Figure 5A to Figure 5B The corresponding circuit structures are shown in the embodiments, which will not be described in detail here.

[0081] In one embodiment of this application, controlling the storage unit to read convolutional data through the memory access instruction and unfolding the convolutional data into a one-dimensional vector in a preset order includes: when the feature map data size exceeds the input capacity of the non-blocking fully permutation network, controlling the storage unit to read the feature map data in blocks through the memory access instruction to obtain convolutional data, and unfolding the convolutional data into a one-dimensional vector in a preset order; wherein, the size of the convolutional data is less than or equal to the input capacity of the non-blocking fully permutation network. For details, please refer to the foregoing. Figure 6A to Figure 6C The corresponding circuit structures are shown in the embodiments, which will not be described in detail here.

[0082] In one embodiment of this application, controlling the storage unit to read feature map data in blocks to obtain convolutional data via the memory access instruction includes: controlling the storage unit to read the non-overlapping portion of the feature map data with the previous convolutional data based on the overlap relationship of adjacent convolutional data, and combining it with the overlapping data temporarily stored at the input port of the non-blocking full permutation network to generate the current convolutional data. For details, please refer to the foregoing. Figure 6A to Figure 6C The corresponding circuit structures are shown in the embodiments, which will not be described in detail here.

[0083] Since the principle behind this method is similar to that of the conversion circuit, the implementation of this circuit can be found in the implementation of the circuit method, and the repetitive parts will not be repeated.

[0084] In one embodiment of this application, a chip is also provided, comprising a matrix operation unit and the above-described conversion circuit;

[0085] The matrix operation unit is used to perform multiplication and accumulation operations on the matrix multiplication data output in real time by the non-blocking permutation network.

[0086] For further details, please refer to... Figure 7B As shown, in the above embodiment, the chip may include a storage unit, a permutation network, a matrix operation unit, and a control unit. The storage unit is used to cache feature map data and convolution kernel data that need to be convolved, and the data reading and writing of the storage unit is controlled by the memory access instructions issued by the control unit. The permutation network is used to realize the conversion of feature map data from convolution to matrix, and is controlled by the data exchange instructions issued by the control unit. The matrix operation unit is used to receive matrix data streams and perform matrix multiplication and accumulation operations, and is also controlled by the matrix operation instructions issued by the control unit.

[0087] In one embodiment of this application, a board is also provided, including the aforementioned artificial intelligence processor chip.

[0088] In one embodiment of this application, an electronic device is also provided, which is equipped with the aforementioned board or the aforementioned chip.

[0089] Therefore, the efficient online conversion logic for convolution to matrix multiplication provided in this application is based on a non-blocking permutation network to rearrange the input convolutional data, thereby achieving efficient online conversion of convolutional data to matrix data, greatly reducing processor memory access overhead and storage resource overhead, and effectively improving processor computing efficiency and performance.

[0090] One embodiment of this application also proposes an electronic device, which includes the chip or board described above. The electronic device may include user equipment (UE), mobile device, user terminal, terminal, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, and other devices with AI application requirements.

[0091] Figure 8 A block diagram of an electronic device 1900 according to an embodiment of this application is shown. For example, the electronic device 1900 may be provided as a server or a terminal device. (Refer to...) Figure 8 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0092] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output (I / O) interface 1958. Electronic device 1900 can operate on an operating system stored in memory 1932, such as Microsoft Windows Server™, Apple's graphical user interface-based operating system (Mac OSX™), a multi-user, multi-process computer operating system (Unix™), a free and open-source Unix-like operating system (Linux™), an open-source Unix-like operating system (FreeBSD™), or similar.

[0093] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0094] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0095] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A conversion circuit for online implementation of convolution to matrix multiplication, characterized in that, The conversion circuit includes: a storage unit, a control unit, and a non-blocking permutation network; The storage unit is used to cache feature map data and convolution kernel parameters, and read convolution data according to the memory access instructions generated by the control unit, and output a one-dimensional vector unfolded in a preset order; The control unit is configured to generate the memory access instruction based on the convolution kernel parameters, the size parameters of the feature map data, and the size parameters of the non-blocking permutation network; and to generate a swap instruction based on the convolution kernel parameters, the one-dimensional vector, and the size parameters of the non-blocking permutation network. The non-blocking permutation network is used to perform online rearrangement transformation on the one-dimensional vector output by the storage unit according to the exchange instruction, and generate matrix multiplication data; The switching instruction includes network structure transformation information corresponding to each transition of the non-blocking full permutation network; the network structure transformation information is the connection relationship of multi-level interconnected networks in the non-blocking full permutation network; the control unit configures the connection relationship through the network structure transformation information to dynamically reconstruct the input-to-output transmission path of the non-blocking full permutation network.

2. The conversion circuit according to claim 1, characterized in that, The control unit is specifically used for: The size of the matrix multiplication data is calculated based on the size and stride in the one-dimensional vector and the convolution kernel parameters. The network structure transformation information is determined based on the size of the matrix multiplication data and the computational parallelism of the non-blocking full permutation network. The exchange instruction is generated based on the network structure transformation information.

3. The conversion circuit according to claim 2, characterized in that, The non-blocking full permutation network is also used for: When the computational parallelism of the non-blocking full permutation network is not fully utilized, the output ports of the non-blocking full permutation network that lack valid data are supplemented.

4. The conversion circuit according to claim 1, characterized in that, Also includes: At least one first selection module is disposed at the input port of the non-blocking full permutation network; The control unit is further configured to, when the convolutional data requires edge padding, control at least one of the first selection modules to select a padding value or the convolutional data at the input port corresponding to the non-blocking full permutation network.

5. The conversion circuit according to claim 1, characterized in that, Also includes: Multiple second selection modules are respectively set at each output port of the non-blocking full permutation network; The control unit is further configured to control the second selection module to select either the padding value or the matrix multiplication data at the output port corresponding to the non-blocking full permutation network when the convolutional data requires edge padding.

6. The conversion circuit according to claim 1, characterized in that, The control unit is also used for: When the feature map data size exceeds the input capacity of the non-blocking full permutation network, the memory access instruction controls the storage unit to read the feature map data in blocks to obtain convolutional data, and the convolutional data is unfolded into a one-dimensional vector in a preset order.

7. The conversion circuit according to claim 6, characterized in that, The control unit is also used for: The memory access instruction controls the storage unit to read the non-overlapping part of the feature map data with the previous convolutional data according to the overlap relationship of adjacent convolutional data, and combines it with the overlapping data temporarily stored in the input port of the non-blocking full permutation network to generate the current convolutional data.

8. The conversion circuit according to any one of claims 1 to 7, characterized in that, The non-blocking full permutation network is a Benes network.

9. A method for online implementation of convolution to matrix multiplication, characterized in that, The method includes: Based on the size parameters of the non-blocking full permutation network and the size parameters of the convolution kernels and feature map data cached in the storage unit, a memory access instruction is generated. The memory access instruction controls the storage unit to read the convolution data and outputs a one-dimensional vector unfolded in a preset order. Based on the convolution kernel parameters, the one-dimensional vector, and the size parameters of the non-blocking permutation network, a swap instruction is generated. The swap instruction controls the non-blocking permutation network to perform online rearrangement transformation on the input one-dimensional vector to generate a matrix data stream. The switching instruction includes network structure transformation information corresponding to each transition of the non-blocking full permutation network; the network structure transformation information is the connection relationship of the multi-level interconnection network in the non-blocking full permutation network; the connection relationship is configured through the network structure transformation information to dynamically reconstruct the input-to-output transmission path of the non-blocking full permutation network.

10. The conversion method according to claim 9, characterized in that, The method further includes: When the feature map data needs edge padding, at least one first selection module located at the input port of the non-blocking full permutation network selects either the padding value or the convolutional data at the corresponding input port of the non-blocking full permutation network.

11. The conversion method according to claim 9, characterized in that, The method further includes: When the feature map data needs edge padding, multiple second selection modules located at the output ports of the non-blocking permutation network select padding values ​​or matrix multiplication data at the corresponding input ports of the non-blocking permutation network.

12. The conversion method according to claim 9, characterized in that, The memory access instruction controls the storage unit to read convolutional data, and the convolutional data is unfolded into a one-dimensional vector in a preset order, including: When the feature map data size exceeds the input capacity of the non-blocking full permutation network, the memory access instruction controls the storage unit to read the feature map data in blocks to obtain convolutional data, and expands the convolutional data into a one-dimensional vector in a preset order; The size of the convolutional data is less than or equal to the input capacity of the non-blocking full permutation network.

13. The conversion method according to claim 12, characterized in that, The convolutional data is obtained by controlling the memory access instructions to read feature map data in blocks from the memory unit, including: The memory access instruction controls the storage unit to read the non-overlapping part of the feature map data with the previous convolutional data according to the overlap relationship of adjacent convolutional data, and combines it with the overlapping data temporarily stored in the input port of the non-blocking full permutation network to generate the current convolutional data.

14. A chip, characterized in that, It includes a matrix operation unit and the conversion circuit according to any one of claims 1 to 8; The matrix operation unit is used to perform multiplication and accumulation operations on the matrix multiplication data output in real time by the non-blocking permutation network.

15. A circuit board, characterized in that, Includes the chip as described in claim 14.

16. An electronic device, characterized in that, The board as described in claim 15 or the chip as described in claim 14 is deployed.

Images