Matrix transformation method, device and convolutional neural network accelerator
By buffering and transforming the sub-eigrams of the input feature maps in a predetermined order in the CNN accelerator of FPGA, the problem of low matrix computing efficiency is solved, and higher computing speed and lower storage access redundancy is achieved.
Patent Information
- Application Number
- CN202111277687.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-29
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-10-29
AI Technical Summary
The input feature map matrix transformation efficiency of the CNN accelerator based on FPGA in the prior art in matrix operations is low, resulting in slow computing speed and access memory and bandwidth redundancy.
By buffering the sub-feature maps in the input feature map in a predetermined order, and performing matrix transformation on the sub-feature maps cached each time, the target sub-feature matrix is obtained. The output feature map is determined based on the weight parameters of these matrices and neural network models, and the output feature map is reduced. Number of data accesses to avoid redundant data access.
It improves the computing speed, reduces memory access and bandwidth redundancy, and improves the efficiency of matrix computing.
Smart Images

Figure CN114003201B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the field of communications, and more specifically, to a matrix transformation method, device, convolutional neural network accelerator, storage medium, and electronic device. Background Art
[0002] At present, the field of deep learning has received extensive attention in the industry, among which convolutional neural networks (CNN) have become a research hotspot in many fields such as image classification, target detection and semantic segmentation, and can achieve good results. The most computationally intensive parts of CNN are convolutional layers and fully connected layers, and their underlying layers can be implemented based on matrix multiplication. As the computational scale and complexity of CNN models continue to increase, the use of traditional CPU platforms can no longer meet the requirements of practicality. Therefore, the implementation of accelerators using computing platforms such as GPUs and FPGAs has received widespread attention. However, compared with GPUs, FPGAs are more energy efficient, easy to reconfigure, fast iterative updates, and easy to deploy on mobile edge terminals, which can better meet the needs of the rapid development of deep learning algorithms.
[0003] In the related art, the implementation methods of CNN accelerators based on FPGA mainly include two aspects: loop unrolling parallel computing and systolic array computing. However, the first method achieves computing acceleration by increasing the degree of parallelism, but faces the problem of high fan-in / fan-out, resulting in a low final speed and poor universality of the computing mode; the second method converts the convolution operation and full connection operation in CNN into matrix multiplication, but the implementation of the matrix transformation img2col method has a greater impact on the overall performance.
[0004] It can be seen from this that the related art has the problem of low efficiency of input feature map matrix transformation in the matrix operation involved in the accelerator operation process.
[0005] Currently, no effective solution has been proposed for the above-mentioned problems existing in the related technologies. Summary of the invention
[0006] Embodiments of the present invention provide a matrix transformation method, device, convolutional neural network accelerator, storage medium and electronic device to at least solve the problem of low efficiency of input feature map matrix transformation in matrix operations involved in accelerator operation processes existing in related technologies.
[0007] According to one embodiment of the present invention, a matrix transformation method is provided, comprising: caching a plurality of sub-feature maps included in an input feature map in sequence according to a predetermined order; performing a matrix transformation on the cached sub-feature map each time to obtain a plurality of target sub-feature matrices; and determining a target output feature map corresponding to the input feature map based on the plurality of target sub-feature matrices and weight parameters of a neural network model.
[0008] According to another embodiment of the present invention, a matrix transformation device is provided, comprising: a cache module for caching a plurality of sub-feature maps included in an input feature map in sequence according to a predetermined order; a transformation module for performing a matrix transformation on the sub-feature maps cached each time to obtain a plurality of target sub-feature matrices; and a determination module for determining a target output feature map corresponding to the input feature map based on the plurality of target sub-feature matrices and weight parameters of a neural network model.
[0009] According to another embodiment of the present invention, a convolutional neural network accelerator is provided, comprising the device in the above embodiment.
[0010] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, wherein the computer program implements the steps of any of the above methods when executed by a processor.
[0011] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0012] Through the present invention, multiple sub-feature maps included in the input feature map are cached in sequence according to a predetermined order, and matrix transformation is performed on each cached sub-feature map to obtain multiple target sub-feature matrices, and the target output feature map corresponding to the input feature map is determined according to the multiple sub-feature matrices and the weight parameters of the neural network model. Since only one sub-feature map included in the output feature map is cached each time, the sub-feature map is matrix transformed, which reduces the number of times the data is accessed and read during calculation, and only one sub-feature map is cached each time, avoiding the problem of redundant data access. Therefore, the problem of low efficiency of matrix transformation of the input feature map in the matrix operation involved in the accelerator operation process in the related art can be solved, and the memory access and bandwidth redundancy are reduced, thereby improving the operation speed. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 It is a hardware structure block diagram of a mobile terminal of a matrix transformation method according to an embodiment of the present invention;
[0014] Figure 2 is a flowchart of a matrix transformation method according to an embodiment of the present invention;
[0015] Figure 3 is a schematic diagram of the structure of a convolutional neural network accelerator according to an exemplary embodiment of the present invention;
[0016] Figure 4 It is a schematic diagram of the calculation process of convolution in related technologies;
[0017] Figure 5 is a schematic diagram of dividing an input feature map according to input channel parallelism, row width, and number of rows according to an exemplary embodiment of the present invention;
[0018] Figure 6 is a schematic diagram of overlapping blocks after segmenting an input feature map according to an exemplary embodiment of the present invention;
[0019] Figure 7 This is a schematic diagram of converting convolution calculation into matrix operation in related technologies Figure 1 ;
[0020] Figure 8 This is a schematic diagram of converting convolution calculation into matrix operation in related technologies Figure 2 ;
[0021] Fig. 9 is a schematic diagram of a sub-feature graph of a cache according to an exemplary embodiment of the present invention;
[0022] Fig.10 is a schematic diagram of a row cache sliding window process according to an exemplary embodiment of the present invention;
[0023] Fig.11 is a structural diagram of a pulse array unit according to an exemplary embodiment of the present invention;
[0024] Fig.12 is a structural block diagram of a matrix conversion device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0025] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.
[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0027] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of a matrix transformation method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.
[0028] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the matrix transformation method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.
[0029] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0030] In this embodiment, a matrix transformation method is provided. Figure 2 is a flowchart of matrix transformation according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:
[0031] Step S202, caching multiple sub-feature maps included in the input feature map in sequence according to a predetermined order;
[0032] Step S204, performing matrix transformation on the sub-feature graph cached each time to obtain a plurality of target sub-feature matrices;
[0033] Step S206, determining a target output feature map corresponding to the input feature map based on the plurality of sub-feature matrices and weight parameters of the neural network model.
[0034] In the above embodiment, when the convolutional neural network performs a convolution operation, the input feature map of the original input image can be first determined, the sub-feature maps in the input feature map can be cached, and the sub-feature maps cached each time can be matrix transformed to obtain multiple target sub-feature matrices, and then the target output feature map corresponding to the input feature map can be determined according to the multiple target sub-feature matrices and the weight parameters of the neural network model. Among them, the input feature map can be a feature map for systolic array matrix calculation.
[0035] Optionally, the executor of the above steps may be an FPGA, a convolutional neural network accelerator, a background processor, or other devices with similar processing capabilities, or a machine that at least integrates a data processing device, wherein the data processing device may include terminals such as computers and mobile phones, but is not limited thereto.
[0036] In the above embodiment, when the execution subject of the above steps is FPGA or convolutional neural network accelerator, the structural diagram of convolutional neural network accelerator can be found in the attached Figure 3 ,like Figure 3 As shown, the original input image can be used as the input feature map of the first convolution layer and the corresponding weight parameters for convolution operation. Among them, the original input feature map and the weight parameters of each convolution layer or fully connected layer in the network are stored in an external memory (generally DDR). Before performing the calculation, the hardware module of the programmable logic end can be configured by the software end (i.e., the processing system). For example, during the convolution calculation, the processing system configures the working register of the direct memory access controller DMA through the bus, so that the DMA moves data (input feature map and weight parameters) from the external DDR to the FPGA on-chip logic. Since the on-chip resources of the FPGA are limited, an input feature map cache unit and a weight cache unit can be added to cache part of the input feature map (i.e., part of the feature map) and the storage of parameter data, respectively. After the two parts of the data perform several operations, the direct memory access controller reads and caches the next batch of data. The input feature map cache unit can use a channel priority strategy to implement storage, and the matrix transformation unit can use a row cache window sliding method to implement the convolution-to-matrix operation, i.e., the hardware implementation of the img2col operation. The systolic array unit is the main computing unit, which implements the matrix operation of the input feature map and weight parameters, and its result is the partial sum of the convolution operation. Therefore, the accumulator can be used to cache the next batch of results until the input feature map corresponding to the current weight parameter is calculated and the final result is output. The bias module is responsible for adding the bias parameter to the output channel direction of the input feature map. The result is processed nonlinearly by the activation function (such as ReLU) module and output to the next level module. At this point, the calculation of one convolution layer is completed.
[0037] In the above embodiment, the next layer of the convolution layer can be a pooling layer or an element-level operation layer (such as a concatenation Concat layer). Therefore, the pooling / element-level operation processing unit is responsible for the pooling operation and the element-level operation, and its output result is written back to the external memory through DMA as the input feature map of the next convolution layer, and a new round of convolution layer-bias-activation function-pooling / element-level operation processing process is executed until all layers of the network are fully calculated.
[0038] In the above embodiment, the input of the convolution calculation includes the weight parameter weight(Co, Ci, Ky, Kx) and the input feature map ifmp(N, Ci, Hi, Wi), and the final calculation result is the output feature map ofmp(N, Co, Ho, Wo). Wherein, N represents the batch size, which can be set to 1 (this value is only an exemplary description and the present invention is not limited to this). Co and Ci represent the output and input channel data, Kx and Ky represent the length and width of the convolution kernel respectively, Hi, Wi and Ho, Wo represent the height and width of the input and output feature maps respectively, and the final result is added with bias bias(Co, 1), that is, each output channel corresponds to a bias. The calculation of the convolution layer is shown in formula (1), where S represents the step size of the sliding window.
[0039]
[0040]
[0041] Among them, Ho=((Hi-Ky+2*P) / S)+1, Wo=((Wi-Kx+2*P) / S)+1, P represents the number of zero-padding rows. Taking the first convolutional layer as an example, the input feature map ifmp is (1, 3, 416, 416), weight is (16, 3, 3, 3), S=1, P=1, then the output feature map ofmp of the first layer is (1, 16, 416, 416).
[0042] At the same time, the fully connected layer can be regarded as a vector-matrix multiplication, that is, ifmp is (Ci, 1), the weight weight is (Co, Ci), and the bias bias is (Co, 1), then ofmp(Co, 1) = weight × ifmp + bias.
[0043] Through the present invention, multiple sub-feature maps included in the input feature map are cached in sequence according to a predetermined order, and matrix transformation is performed on each cached sub-feature map to obtain multiple target sub-feature matrices. The multiple sub-feature matrices and the weight parameters of the neural network model determine the target output feature map corresponding to the input feature map. Since only one sub-feature map included in the output feature map is cached each time, the sub-feature map is matrix-transformed, which reduces the number of times the data is accessed and read during calculation, and only one sub-feature map is cached each time, which avoids the problem of redundant data access. Therefore, the problem of low efficiency of matrix transformation of the input feature map in the matrix operation involved in the accelerator operation process in the related art can be solved, and the memory access and bandwidth redundancy are reduced, thereby improving the speed of operation.
[0044] In an exemplary embodiment, caching multiple sub-feature maps included in an input feature map in a predetermined order includes: determining the input channel parallelism and row width of the input feature map; determining the number of rows to be cached each time; dividing the input feature map according to the input channel parallelism, the row width, and the number of rows to obtain multiple sub-feature maps; caching the multiple sub-feature maps in sequence in the predetermined order. In this embodiment, when caching the sub-feature maps included in the input feature map in a predetermined order, the input channel parallelism and the row width of the input feature map can be determined first, and then the number of rows to be cached each time can be determined, and the input feature map can be divided according to the input channel parallelism, the row width, and the number of rows to obtain multiple sub-feature maps. After obtaining the multiple sub-feature maps, the multiple sub-feature maps are cached in sequence in a predetermined order. Among them, the predetermined order can be an order from top to bottom.
[0045] In the related art, the schematic diagram of the convolution calculation process can be found in the attached Figure 4 ,like Figure 4 As shown, each filter weight slides on the input feature map ifmp from left to right and from top to bottom, and the corresponding positions of the two are multiplied and accumulated to generate a feature point of the output feature map ofmp. When the sliding window traverses the entire ifmp, an output feature map of one channel is generated, and Co weights generate the final Co channels ofmp. Since the convolution process involves the storage and reuse of a large number of input feature maps and parameter data, the on-chip resources of the FPGA are limited. Therefore, only part of the data can be cached on the chip for calculation each time, as shown in the gray part in the figure. The number of gray cubes in the Ci direction is PC, which is called the input channel parallelism, and the number of gray cubes in the Co direction is PF, which is called the output channel parallelism. It can be seen that the accelerator only calculates a subset of the entire input feature map and weight parameters each time ( Figure 4In order to effectively read the corresponding calculation data in the external memory, in the related art, the input feature map is divided into several blocks horizontally and vertically, and only one block of the input feature map is read and calculated each time. However, due to the existence of the sliding window, it is obvious that there is data overlap between two adjacent blocks up and down / left and right, so the data of the overlapping part needs to be considered. In addition, after calculating the convolution result of each block, additional control and operation are required to restore the physical form of the original output feature map, which introduces additional delay.
[0046] In the above embodiment, the schematic diagram of dividing the input feature graph according to the input channel parallelism, row width and number of rows can be found in the attached Figure 5 ,like Figure 5 As shown, the row W direction and the input channel Ci direction of the input feature map are all stored, and only part of the rows, such as 32 rows (this value is configurable) are stored in the column H direction. A total of W×Ci×32 data need to be stored, and the resource addresses stored on the chip are continuous, and the row size of each layer is upwardly compatible to the power of 2, such as 416→512, 208→256, etc. In addition, it should be noted that the input channel parallelism PC can be 8 (this value is only an exemplary description, and the present invention is not limited to this, for example, it can also be set to 16, 32, etc.). PC indicates that each input data is stored externally, stored on the chip, and calculated on the chip in groups of 8 in the input channel direction. In the figure, step 1 indicates that the first group of 8 parallelisms are calculated in the row→column direction during calculation, and step 2 indicates that the next group of 8 parallelisms are calculated in the input channel direction until the input channel direction is calculated. That is, the storage strategy with input channel direction priority can effectively alleviate the problem of limited on-chip storage resources. At the same time, after the output feature map is calculated, it is not necessary to splice and restore the output feature maps corresponding to each block. In addition, the overlapping parts introduced by regular blocking can reduce memory access bandwidth through multiplexing.
[0047] In an exemplary embodiment, determining the number of rows cached each time includes: determining a sliding step of a sliding window; determining the number of overlapping rows overlapped in two adjacent slides based on the sliding step; obtaining a predetermined pre-cached number of pre-cached rows; and determining the difference between the pre-cached number and the overlapping number as the number of rows. In this embodiment, when determining the number of rows cached each time, the sliding step of the sliding window may be determined first, the number of overlapping rows overlapped in two adjacent slides may be determined based on the sliding step, the number of pre-cached rows overlapped in two adjacent slides may be obtained, and the difference between the pre-cached number and the overlapping number may be determined as the number of rows.
[0048] In the above embodiment, after the input feature map is segmented, n feature maps are obtained, and only one block Blockn is stored on each slice, where n=1, 2, 3, 4... After the current block is calculated, the next block is cached. Taking the first channel as an example, when Ky=Kx=3, S=1, and P=1, two upper and lower adjacent blocks overlap. When the number of pre-caches is 32, the schematic diagram of the overlapping of the blocks after the input feature map is segmented can be seen in the attached figure. Figure 6 ,like Figure 6 As shown in the figure, Block1 and Block2 overlap by two lines. Therefore, the on-chip cache can be reduced by 2 lines each time; the final results of blocks do not require additional recovery operations, and can be directly written back to the external memory at consecutive addresses; each block contains all input feature maps of the input channel dimension, so when all filters traverse and calculate the current block, the final output feature map result can be directly generated, and the on-chip cache of the intermediate convolution calculation part and results is no longer needed. This reduces the number of frequent accesses to the external memory and reduces bandwidth requirements.
[0049] In an exemplary embodiment, matrix transformation is performed on the sub-feature graph cached each time to obtain multiple target sub-feature matrices, including: performing the following operations for each time period: determining the row data included in the sub-feature graph acquired in the time period; matrix transformation is performed on the row data to obtain the target sub-feature matrix. In this embodiment, when determining the target feature matrix, the row data acquired in each time period can be matrix transformed to obtain the target sub-feature matrix, and the output feature graph is obtained by convolution calculation through the target sub-feature matrix. After the calculation is completed, the row data in the next time period is acquired again for matrix transformation to obtain the target sub-feature matrix, and the output feature matrix is calculated in sequence. Therefore, redundant data access can be effectively avoided, and the hardware design of the pipeline can be realized, which is of great significance to the storage space, memory access bandwidth and computing resources of the FPGA. Among them, the time period can be N clock beats, such as 3 clock beats. The time period can be determined according to the sliding window and the number of rows in each cache.
[0050] In the related art, the convolution layer and the fully connected layer in the CNN calculation can be converted into matrix multiplication, that is, they can be implemented using a systolic array. The fully connected layer is a matrix-vector multiplication, and its calculation process is not repeated here. The matrix conversion of the convolution calculation is the hardware implementation of img2col. Figure 7 As shown in the figure, the process of img2col is to convert the input feature map into the matrix format corresponding to the filter. Since there will be data overlap in the sliding window, there will be a large amount of redundant data inside.
[0051] Through img2col Figure 4 The convolution operation is converted into the matrix multiplication operation process, such as Figure 8 As shown, the systolic array is used to perform the corresponding calculations. A more intuitive method is to use a high-level language to implement img2col on the software side, and then transfer the transformed data to the FPGA for calculation. However, a large amount of redundant data and each layer of calculation requires matrix conversion operations, and the time overhead of this processing method is huge. Figure 7 and Figure 8 It can be seen that img2col converts the original convolution input feature map into a matrix form. It uses the data of the corresponding position of ifmp covered by each convolution kernel sliding on the input feature map ifmp, and expands the data covered by the convolution kernel into a one-dimensional vector, which is the corresponding result of img2col.
[0052] In the above embodiment, it is assumed that the data currently stored on the chip is as follows: Fig. 9 The cube shown in the figure has PC=8, with a total of Ci×W×32 data. Assume that the convolution kernel of the current layer is 3×3×Ci, and its parallelism PC corresponds to ifmp. Then the position of the first window is window 1, the second is window 2, and then window 3, window 4, and window 5, until a row is slid. Each window covers PKKC=3×3×PC=72 data, corresponding to the 1-dimensional vector of img2col. Fig.10 As shown, assuming that the input feature map data bit width is DW, then each beat of data is PC×DW. For a 3×3 convolution kernel, a 2-line cache can be used, where the cache can be implemented based on FIFO, and the depth of the FIFO depends on the maximum value of the row of the input feature map, such as 512. It is easy to know that after 2 lines of data are cached, when the third line of data flows in, the data of the first 2 lines flow into the register group in sequence and the data of the third line at the same time, and 3 groups of PC×DW flow out in each clock beat. After waiting for 3 clock beats, the 9 groups of data corresponding to the 3×3 window in the register group are all valid, and then they are output to the systolic array for vertical input once, and the next beat corresponds to the next batch of 9 groups of data corresponding to the sliding window once, and so on, then the corresponding input feature map after img2col matrix conversion can be obtained.
[0053] In the above embodiment, the matrix transformation operation of img2col is implemented by adopting a row cache sliding window, which effectively avoids the redundant data memory access problem faced by the img2col operation. At the same time, this method is based on hardware implementation, which effectively reduces the requirements for on-chip storage space and external memory access bandwidth.
[0054] In an exemplary embodiment, determining the target output feature map corresponding to the input feature map based on multiple target sub-feature matrices and weight parameters of the neural network model includes: determining the output feature map corresponding to each sub-feature matrix based on each sub-feature matrix and the weight parameters; caching the output feature map corresponding to each sub-feature matrix to obtain multiple output feature maps; and determining the target output feature map based on multiple output feature maps. In this embodiment, when determining the target output feature map, the output feature map corresponding to each sub-feature matrix is determined according to each sub-feature matrix and the weight parameters, and the output feature map corresponding to each sub-feature matrix is cached, that is, each output feature map obtained from the sub-feature map is cached, and the target output feature map is determined according to the multiple output feature maps.
[0055] In an exemplary embodiment, determining the output feature map corresponding to each sub-feature matrix based on each sub-feature matrix and the weight parameter includes: determining the sub-weight parameter of the target input channel corresponding to the sub-feature matrix based on the weight parameter; determining the current sub-feature matrix obtained in the current time period; determining the first product of the current sub-feature matrix and the sub-weight parameter; determining the second product of the previous feature matrix in the time period before the current time period and the sub-weight parameter; and determining the sum of the first product and the second product as the output feature map. In this embodiment, when determining the output feature map, for each target input channel, the sub-weight parameter of the target input channel can be determined, the current sub-feature matrix obtained in the current time period is determined, the first product of the current sub-feature matrix and the sub-weight parameter is determined, and then the second product of the previous feature matrix in the time period before the current time period and the weight parameter is determined, and the sum of the first product and the second product is determined as the output feature map.
[0056] In the above embodiment, the pulse array unit can be used to determine the output characteristic diagram. The structure diagram of the pulse array unit can be found in the attached Fig.11 ,like Fig.11As shown, the pulse array unit may include a two-dimensional PE matrix. The weight parameters of the target input channel may be determined first, wherein the weight parameters used for each input channel may be fixed. The input feature map data and the output feature map data may be calculated in a directed propagation manner, wherein the weight parameters may be pre-stored in the longitudinal direction, after which the input feature map data enters the pulsating array unit in a horizontal manner for calculation, and flows to the adjacent right PE, and the calculation result of each PE is directed propagated in the longitudinal direction. All flows are once for each clock beat. Each PE is a multiplication and accumulation unit, that is, when Wi and Xi meet in each PE, multiplication is performed, and the product result and the result flowing from the adjacent upper part are accumulated, and then flow to the adjacent lower PE. After calculating part of the input feature map of the current batch, a partial sum result is obtained. Since the result of the final output feature map is the sum of the products of all data in the input channel direction, therefore. It is necessary to temporarily store the partial sum corresponding to the next batch and accumulate it until the final calculation result is output, which is the output feature map of the current convolution layer.
[0057] In an exemplary embodiment, determining the current sub-feature matrix obtained in the current time period includes: in the case where there is no input channel before the target input channel, determining the target sub-feature matrix as the current sub-feature matrix; in the case where there is an input channel before the target input channel, determining the first product of adjacent input channels included in the input channel and adjacent to the target input channel as the current sub-feature matrix. In this embodiment, in the case where there is no input channel before the target input channel, the target sub-feature matrix is determined as the current sub-feature matrix; in the case where there is an input channel before the target input channel, the first product of adjacent input channels included in the input channel and adjacent to the target input channel is determined as the current sub-feature matrix. Fig.11 As shown, when the target input channel is the first column channel, the target sub-feature matrix is determined as the current sub-feature matrix. When the target input channel is the channel before the first column channel, such as the second column channel, the first product of the first column channel is determined as the current sub-feature matrix.
[0058] In the above embodiment, based on the systolic array method, each PE is connected only to adjacent PEs, and the fan-in and fan-out are 1. At the same time, the regular rectangular physical structure of the systolic array makes it more efficient when mapped to the FPGA, because the physical space where the DSP resources on the corresponding FPGA are distributed is a rectangular area, thus enabling the accelerator to achieve a higher clock frequency.
[0059] In the aforementioned embodiment, the problem of limited on-chip storage resources is effectively alleviated by the storage strategy that prioritizes the input channel direction. At the same time, compared with other block strategies, after the output feature map is calculated, it is not necessary to splice and restore the output feature maps corresponding to each block. In addition, the overlapping parts introduced by the regular block can reduce the memory access bandwidth through multiplexing. The fan-in / fan-out is 1 through the systolic array solution, and its regular matrix physical structure is easy to layout and route the FPGA, so a higher clock frequency can be achieved, improving the overall performance of the accelerator. In addition, the FPGA-based accelerator has the characteristics of flexible design, reconfigurability and low power consumption.
[0060] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0061] In the present embodiment, a device for determining an output feature map is also provided, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.
[0062] Fig.12 is a structural block diagram of a matrix conversion device according to an embodiment of the present invention. Fig.12 As shown, the device comprises:
[0063] A cache module 1202 is used to cache multiple sub-feature maps included in the input feature map in sequence according to a predetermined order;
[0064] A transformation module 1204 is used to perform matrix transformation on the sub-feature graph cached each time to obtain multiple target sub-feature matrices;
[0065] The determination module 1206 is used to determine the target output feature map corresponding to the input feature map based on the multiple target sub-feature matrices and the weight parameters of the neural network model.
[0066] The cache module 1202 corresponds to the attached Figure 3The input feature map cache unit in the transform module 1204 corresponds to the attached Figure 3 The matrix transformation (row buffer window sliding) unit in the determination module 1206 corresponds to the attached Figure 3 The systolic array and accumulator cache in .
[0067] In an exemplary embodiment, the cache module 1202 can cache multiple sub-feature maps included in the input feature map in a predetermined order in the following manner: determine the input channel parallelism and the row width of the input feature map; determine the number of rows to be cached each time; divide the input feature map according to the input channel parallelism, the row width and the number of rows to obtain multiple sub-feature maps; and cache the multiple sub-feature maps in the predetermined order.
[0068] In an exemplary embodiment, the cache module 1202 can determine the number of rows cached each time in the following manner: determine the sliding step size of the sliding window; determine the number of overlapping rows overlapping in two adjacent slides based on the sliding step size; obtain a pre-cache number of pre-cache rows determined in advance; and determine the difference between the pre-cache number and the overlapping number as the number of rows.
[0069] In an exemplary embodiment, the transformation module 1204 can implement matrix transformation of the sub-feature graph cached each time to obtain multiple target sub-feature matrices in the following manner: perform the following operations for each time period: determine the row data included in the sub-feature graph acquired within the time period; and perform matrix transformation on the row data to obtain the target sub-feature matrix.
[0070] In an exemplary embodiment, the determination module 1206 can determine the target output feature map corresponding to the input feature map based on multiple target sub-feature matrices and weight parameters of the neural network model in the following manner: determine the output feature map corresponding to each sub-feature matrix based on each sub-feature matrix and the weight parameters; cache the output feature map corresponding to each sub-feature matrix to obtain multiple output feature maps; and determine the target output feature map based on multiple output feature maps.
[0071] In an exemplary embodiment, the determination module 1206 can determine the output feature map corresponding to each sub-feature matrix based on each sub-feature matrix and the weight parameter in the following manner: determine the sub-weight parameter of the target input channel corresponding to the sub-feature matrix based on the weight parameter; determine the current sub-feature matrix obtained in the current time period; determine the first product of the current feature matrix and the sub-weight parameter; determine the second product of the previous feature matrix in the time period before the current time period and the weight parameter; and determine the sum of the first product and the second product as the output feature map.
[0072] In an exemplary embodiment, the determination module 1206 can determine the current sub-feature matrix obtained in the current time period in the following manner: when there is no input channel before the target input channel, the target sub-feature matrix is determined as the current sub-feature matrix; when there is an input channel before the target input channel, the first product of adjacent input channels included in the input channel and adjacent to the target input channel is determined as the current sub-feature matrix.
[0073] In this embodiment, a convolutional neural network accelerator is also provided, which includes the device in the above embodiment and can run the method in the above method embodiment. The structure diagram can be found in the attached Figure 3 .
[0074] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.
[0075] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above methods are implemented.
[0076] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.
[0077] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0078] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.
[0079] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.
[0080] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.
[0081] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A matrix transformation method, It is characterized in that include: Cache the multiple sub-feature maps included in the input feature map in sequence according to a predetermined order; Performing matrix transformation on the sub-feature graph cached each time to obtain multiple target sub-feature matrices; Determine a target output feature map corresponding to the input feature map based on the plurality of target sub-feature matrices and weight parameters of a neural network model; The method of sequentially caching the plurality of sub-feature maps included in the input feature map in a predetermined order comprises: determining the input channel parallelism and the row width of the input feature map; determining the number of rows to be cached each time; dividing the input feature map according to the input channel parallelism, the row width and the number of rows to obtain the plurality of sub-feature maps; sequentially caching the plurality of sub-feature maps in the predetermined order; Determining the number of rows cached each time includes: determining a sliding step of a sliding window; determining the number of overlapping rows in two adjacent slides based on the sliding step; obtaining a pre-cached number of pre-cached rows determined in advance; and determining the difference between the pre-cached number and the overlapping number as the number of rows.
2. The method according to claim 1, It is characterized in that The sub-feature graph cached each time is subjected to matrix transformation to obtain a plurality of target sub-feature matrices including: For each time period the following operations are performed: Determining row data included in the sub-feature map acquired within the time period; Perform matrix transformation on the row data to obtain the target sub-feature matrix.
3. The method according to claim 1, It is characterized in that Determining a target output feature map corresponding to the input feature map based on the plurality of target sub-feature matrices and weight parameters of a neural network model includes: Determine an output feature map corresponding to each of the sub-feature matrices based on each of the sub-feature matrices and the weight parameter; Cache the output feature map corresponding to each of the sub-feature matrices to obtain a plurality of the output feature maps; The target output feature map is determined based on the multiple output feature maps.
4. The method according to claim 3, It is characterized in that Determining the output feature map corresponding to each of the sub-feature matrices based on each of the sub-feature matrices and the weight parameter includes: Determine a sub-weight parameter of a target input channel corresponding to the sub-feature matrix based on the weight parameter; Determine the current sub-feature matrix obtained in the current time period; Determine a first product of the current sub-feature matrix and the sub-weight parameter; Determine a second product of a previous feature matrix in a time period before the current time period and the sub-weight parameter; A sum of the first product and the second product is determined as the output feature map.
5. The method according to claim 4, It is characterized in that Determine the current sub-feature matrix obtained in the current time period includes: In a case where there is no input channel before the target input channel, determining the target sub-feature matrix as the current sub-feature matrix; In the case that an input channel exists before the target input channel, a first product of adjacent input channels included in the input channels and adjacent to the target input channel is determined as the current sub-feature matrix.
6. A device for determining an output feature map, It is characterized in that include: A cache module, used to cache the sub-feature maps included in the input feature map in sequence according to a predetermined order; A transformation module, used for performing matrix transformation on the sub-feature graph cached each time to obtain multiple target sub-feature matrices; A determination module, used to determine a target output feature map corresponding to the input feature map based on a plurality of the target sub-feature matrices and weight parameters of a neural network model; The cache module caches the multiple sub-feature maps included in the input feature map in a predetermined order in sequence by: determining the input channel parallelism and row width of the input feature map; Determine the number of rows to be cached each time; divide the input feature map according to the input channel parallelism, the row width and the number of rows to obtain a plurality of sub-feature maps; Cache the plurality of sub-feature graphs in sequence according to the predetermined order; The device determines the number of rows cached each time in the following manner: determining the sliding step of the sliding window; determining the number of overlapping rows overlapped in two adjacent slides based on the sliding step; obtaining a pre-cached number of pre-cached rows determined in advance; and determining the difference between the pre-cached number and the overlapping number as the number of rows.
7. A convolutional neural network accelerator, It is characterized in that Comprising the device as claimed in claim 6.
8. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 5 when executed by a processor.
9. An electronic device comprising a memory and a processor, It is characterized in that A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 5.
