A deep convolutional neural network acceleration method, module, system and storage medium
Through the second-level cascade convolution method, high-bit features are used to perform approximate convolution and maximum pooling area are determined, and the pooling results of low-bit features are directly calculated, which solves the problems of low-bit computing speed and high power consumption in deep convolution neural networks, and realizes the Improved computing speed and reduced energy consumption.
Patent Information
- Application Number
- CN202110789039.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-13
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-07-13
AI Technical Summary
Existing deep convolutional neural networks have low computing speed and high computing power consumption, especially in convolution and maximum pooling operations, which lead to inefficiency.
The second-level cascade convolution method is adopted to obtain the high and low-bit features of the input features, and perform convolution and maximum pooling respectively. The maximum pooling area is determined using the high-bit pooling results to reduce the redundant convolution multiplication of the low-bit features. Add operations to achieve increased computing speed and reduced energy consumption.
By reducing the redundant convolution multiplication and addition operation, the speed of convolution pooling calculation is significantly improved and the calculation energy consumption is reduced, which solves the problems of low computing speed and high power consumption in deep convolution neural networks.
Smart Images

Figure CN113408710B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of deep learning technology, and in particular to a deep convolutional neural network acceleration method, module, system and storage medium. Background Art
[0002] Deep convolutional neural networks (DCNNs) have been widely used in computer vision, speech recognition, etc. At the same time, with the popularity of deep convolutional neural network algorithms, more and more people are paying attention to the use of hardware to implement neural networks.
[0003] In conventional convolution and pooling calculations, 75% of the data that has passed through convolution will be discarded after passing through maximum pooling (taking 2×2 pooling size as an example). Obviously, the processing of this part of the data takes up most of the computing time and consumes a lot of power.
[0004] Therefore, how to speed up the calculation speed of deep convolutional neural networks and reduce computing power consumption has become a difficulty in deep convolutional neural network technology. Summary of the invention
[0005] The present invention provides a deep convolutional neural network acceleration method, module, system and storage medium, which are intended to solve the problems of low computing speed and high computing power consumption of existing deep convolutional neural networks.
[0006] According to the first aspect, an embodiment provides a deep convolutional neural network acceleration method, including:
[0007] Get input features;
[0008] Get the high-order features and low-order features of the input features;
[0009] Convolution and maximum pooling are performed on high-order features to obtain high-order pooling results;
[0010] According to the high-order pooling results, the corresponding maximum pooling area is obtained;
[0011] Convolve the low-order corresponding features to obtain the low-order convolution result, where the low-order corresponding features are the data corresponding to the maximum pooling area in the low-order features;
[0012] Based on the high-order pooling results and the low-order convolution results, the maximum pooling result of the input features is obtained.
[0013] According to the second aspect, an embodiment provides an acceleration module for a deep convolutional neural network, including a control module and a computing module;
[0014] The control module is used to obtain input features;
[0015] The calculation module is used to obtain the high-order features and low-order features of the input features; convolution and maximum pooling are performed on the high-order features to obtain the high-order pooling results;
[0016] The control module is also used to obtain the corresponding maximum pooling area according to the high-order pooling result;
[0017] The calculation module is also used to convolve the low-order corresponding features to obtain low-order convolution results, wherein the low-order corresponding features are data corresponding to the maximum pooling area in the low-order features; and the maximum pooling result of the input features is obtained based on the high-order pooling results and the low-order convolution results.
[0018] According to the third aspect, an embodiment provides an acceleration system for a deep convolutional neural network, comprising a processor, an off-chip storage module, and the acceleration module described in the above technical solution; the acceleration module also includes an on-chip storage module;
[0019] The processor is used to control data exchange between an off-chip storage module and an on-chip storage module, and / or the processor is a Rocket general-purpose RISC-V processor.
[0020] According to the fourth aspect, an embodiment provides a computer-readable storage medium, on which a program is stored, and the program can be executed by a processor to implement the deep convolutional neural network acceleration method as described in the above technical solution.
[0021] According to a deep convolutional neural network acceleration method, module, system and storage medium provided by the above embodiment, by obtaining input features; obtaining high-order features and low-order features of the input features; convolving and maximum pooling the high-order features to obtain high-order pooling results; obtaining the corresponding maximum pooling area according to the high-order pooling results; convolving the low-order corresponding features to obtain low-order convolution results, wherein the low-order corresponding features are the data corresponding to the maximum pooling area in the low-order features; obtaining the maximum pooling result of the input features according to the high-order pooling results and the low-order convolution results. It can be seen that by using the high-order features of the input features for approximate convolution, finding the maximum pooling area, and then convolving the corresponding part of the low-order features, the low-order pooling results of the low-order features can be directly obtained, and finally the maximum pooling results of the input features can be obtained. Therefore, when the input features are subjected to maximum pooling, the redundant convolution multiplication and addition operations of the low-order features can be reduced by 75%, and finally the calculation speed of the convolution pooling calculation can be improved and the calculation energy consumption can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 A schematic diagram of a convolution-maximum pooling calculation process in the prior art;
[0023] Figure 2A schematic diagram of the structure of an acceleration module and system for a deep convolutional neural network provided by an embodiment;
[0024] Figure 3 A schematic flow chart of a deep convolutional neural network acceleration method provided by an embodiment;
[0025] Figures 4 to 6 , Figure 8 A schematic diagram of a convolution-maximum pooling calculation process in a deep convolutional neural network acceleration method provided by an embodiment;
[0026] Figure 7 A schematic diagram of a convolution-maximum pooling calculation process in the prior art;
[0027] Fig. 9 A schematic diagram of a process for generating non-zero values and sparse mapping matrices in a deep convolutional neural network acceleration method provided by an embodiment;
[0028] Fig.10 A schematic diagram of a full-jump-to-zero calculation process in a deep convolutional neural network acceleration method provided by an embodiment;
[0029] Fig.11 A schematic diagram of establishing a logical mapping of a PE array in a deep convolutional neural network acceleration method provided by an embodiment;
[0030] Fig.12 A schematic diagram of the structure of a computing unit in an acceleration module for a deep convolutional neural network provided by an embodiment;
[0031] Fig.13 A schematic diagram of the structure of a sub-computing unit in an acceleration module for a deep convolutional neural network provided by an embodiment;
[0032] Fig.14 A schematic diagram of the structure of an acceleration module and system for a deep convolutional neural network provided in another embodiment;
[0033] Fig.15 A schematic diagram of the structure of a sparse mapping module in an acceleration module for a deep convolutional neural network provided by an embodiment.
[0034] Figure markings: 10-processor; 20-off-chip storage module; 30-acceleration module; 31-control module; 32-on-chip storage module; 33-computing module; 331-PE array; 3301-controller; 3302-multiplier; 3303-register; 3304-multiplexer; 332-activation function module; 34-sparse mapping module; 341-write address generation module; 342-write cache module; 343-asynchronous FIFO data module; 344-read address generation module. DETAILED DESCRIPTION
[0035] The present invention is further described in detail below by specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments adopt associated similar element numbers. In the following embodiments, many detailed descriptions are for making the present application better understood. However, those skilled in the art can easily recognize that some features can be omitted in different situations, or can be replaced by other elements, materials, methods. In some cases, some operations related to the present application are not shown or described in the specification, this is to avoid the core part of the present application being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations according to the description in the specification and the general technical knowledge in the art.
[0036] In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various implementations. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a required sequence, unless otherwise specified that a certain sequence must be followed.
[0037] The serial numbers of the components in this document, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "coupling" mentioned in this application, unless otherwise specified, include direct and indirect connections (couplings).
[0038] Embodiment 1
[0039] like Figure 1As shown, existing convolutional neural networks (CNN) or deep convolutional neural networks (DCNN) involve convolution and pooling calculations. After the input feature (generally a pixel matrix) is convolved, the corresponding convolution result is generated. Generally, each pixel point (or data) of the input feature is used as the convolution center for convolution, and padding is generally involved when convolving edge data. The convolution result is then subjected to maximum pooling to produce a maximum pooling result. It can be seen that the input feature needs to be convolved 16 times to obtain the convolution result, and the maximum pooling only extracts 4 data from the convolution result. In other words, when using the conventional convolution-maximum pooling calculation method, 12 data in the convolution result are redundant calculation results and do not participate in the maximum pooling calculation, that is, the redundant calculation reaches 12 / 16=75%. At the same time, the calculation of each convolution result requires the convolution kernel to multiply and add the convolution area of the input feature. When the size of the convolution kernel is 3×3, the 9 weights of the convolution kernel are multiplied by the 9 data in the convolution area of the input feature and then added. It can be seen that the existing convolution-maximum pooling calculation has many redundant calculation processes. Especially when the existing input feature size is getting larger and larger, the existing deep convolutional neural network has the problems of low calculation speed and high calculation power consumption. Further explanation, the pooling mentioned in this embodiment is maximum pooling unless otherwise specified.
[0040] In view of this, the present invention provides a deep convolutional neural network acceleration method, module, system and storage medium, aiming to solve the problems of low computing speed and high computing power consumption of existing deep convolutional neural networks.
[0041] like Figure 2 As shown, an acceleration module 30 for a deep convolutional neural network provided by the present invention includes a control module 31, an on-chip storage module 32 and a computing module 33. An acceleration system for a deep convolutional neural network provided by the present invention includes the above-mentioned acceleration module 30, a processor 10 and an off-chip storage module 20.
[0042] like Figure 3 As shown, the present invention provides a deep convolutional neural network acceleration method, which adopts a two-level cascade convolution method and includes the following steps:
[0043] Step 1: The control module 31 obtains input features, such as a pixel matrix of an image, where the input features come from the off-chip storage module 20. The control module 31 sends a read instruction to the off-chip storage module 20 through the processor 10, and the input features in the off-chip storage module 20 can be transmitted to the on-chip storage module 32.
[0044] Step 2: The calculation module 33 obtains the high-order features and low-order features of the input features. This may be done by processing the input features through the activation function module 332, such as the activation function module 332 using the ReLU function, so as to obtain the high-order features and low-order features of the input features.
[0045] It should be noted that the above-mentioned high-bit features and low-bit features can use the first half of the input features as high-bit features and the second half as low-bit features. For example, when the pixel value (such as grayscale value) of each pixel of the input feature is eight-bit data, the data corresponding to the high-bit feature is the first four bits of data (defined as high-bit data), and the data corresponding to the low-bit feature is the last four bits of data (defined as low-bit data). Specifically, taking binary as an example, when the pixel value of a pixel is 200, the number of bits is expressed as 200D=11001000B in binary, then the high-bit data of this pixel is 1100B=12D, and the second data is 1000B=8D. The high-bit data of each pixel in the input feature forms the above-mentioned high-bit features, and the low-bit data of each pixel forms the above-mentioned low-bit features. It should be understood that the above is only an example and does not limit the number of bits of the input feature. Whether it is 8-bit, 16-bit or 32-bit data or even-bit data, the acceleration method provided by the present invention can be applied.
[0046] Further explanation, such as Figure 4 As shown, the input feature is a pixel matrix of an image, and the pixel value of each pixel can be a non-zero value (such as a, b, c...s shown in the figure) or a zero value (such as 0 shown in the figure). The pixel point a in the input feature corresponds to the high-order data a1 in the high-order feature and the low-order data a2 in the low-order feature. It can be simply understood that a=a1+a2. It can be seen that the matrix positions of the data in the high-order feature and the low-order feature are one-to-one corresponding. In other words, each data or pixel point (zero value or non-zero value) of the input feature has its own position coordinates.
[0047] Step 3: The calculation module 33 performs convolution and maximum pooling on the high-order features to obtain a high-order pooling result.
[0048] For example, Figure 5 As shown, each high-order data in the high-order feature is used as the convolution center, and the high-order data is convolved to obtain the high-order convolution result. Before the data at the edge of the convolution, the high-order feature can be padded to expand the high-order feature; the high-order convolution result is subjected to maximum pooling calculation to obtain the high-order pooling result. The high-order pooling result can be a register stored in the computing module 33, or an on-chip storage module on the storage acceleration module 30, or an off-chip storage module of the acceleration system.
[0049] Step 4: The control module 31 obtains the corresponding maximum pooling area according to the high-order pooling result.
[0050] For example, Figure 5 As shown, the data in the high-order pooling result can be used to infer the high-order data corresponding to the convolution center in the high-order feature. As shown in the figure, the data A1 in the high-order pooling result corresponds to the data A1 of the high-order convolution result, and the data A1 of the high-order convolution result is the convolution calculation result with the data a1 of the high-order feature as the convolution center. Therefore, according to the high-order pooling result, the area corresponding to the maximum pooling result in the high-order convolution result and the high-order feature (that is, the data in the shaded part of the figure) can be obtained (for the convenience of expression, this area is defined as the maximum pooling area). In other words, the position of the high-order data (data when used as the convolution center) that produces the maximum pooling result in the high-order feature can be obtained. From the above analysis, it can be seen that the high-order data of the high-order feature corresponds one-to-one to the input feature and the low-order data of the low-order feature. Therefore, the maximum pooling area can be applied to the convolution calculation of the next low-order feature.
[0051] Step 5: The calculation module 33 performs convolution on the low-order corresponding features to obtain low-order convolution results, wherein the low-order corresponding features are data corresponding to the maximum pooling area in the low-order features.
[0052] For example, Figure 6 As shown, the control module 31 obtains the low-bit data corresponding to the maximum pooling area in the low-bit feature (such as the data in the shaded part of the figure) according to the above-mentioned maximum pooling area, and sends these low-bit data to the calculation module 33. For the convenience of description, the set of low-bit data corresponding to the maximum pooling area in the low-bit feature is defined as the low-bit corresponding feature. The calculation module 33 only convolves the convolution area corresponding to the low-bit data of the low-bit corresponding feature as the convolution center to obtain the low-bit convolution result, which is equivalent to obtaining the low-bit pooling result corresponding to the low-bit feature.
[0053] Further explanation, such as Figure 7 As shown in the figure, in the prior art convolution-maximum pooling calculation method, each data in the low-order feature is used as the convolution center for a convolution, and finally the low-order pooling result shown in the figure is obtained. It can be seen that the data in the low-order convolution result except the shaded part are redundant convolution results, accounting for 75% (corresponding to the pooling window size of 2×2). Obviously, this part of data is not required to participate in the maximum pooling of the input feature.
[0054] Therefore, through this step 3, we can directly obtain Figure 6 The low-order convolution result shown is equivalent to Figure 7 The low-level pooling result is shown.
[0055] Step 6: The calculation module 33 obtains the maximum pooling result of the input feature according to the high-order pooling result and the low-order convolution result.
[0056] like Figure 8As shown, this acceleration method performs approximate convolution on the high-order features of the input features, finds the maximum pooling area based on the high-order pooling results, and then performs convolution on the corresponding parts of the low-order features based on the correspondence between the high-order features and the low-order features, so as to directly obtain the low-order convolution results of the low-order features (equivalent to the result of convolution-maximum pooling calculation on the low-order features), and finally obtains the maximum pooling results of the input features based on the high-order pooling results and the low-order convolution results. The above correspondence between the high-order features and the low-order features and the convolution correspondence are the above-mentioned two-level cascade convolution method. Therefore, when the input features are subjected to maximum pooling, 75% of the redundant convolution multiplication and addition calculations of the low-order features can be reduced, and finally the calculation speed of the convolution pooling calculation can be improved and the calculation energy consumption can be reduced.
[0057] The acceleration method provided in this embodiment can be used for edge computing, for example, it can be applied to cloud offloading, video analysis, smart home, and can also be applied to face recognition of security cameras, face recognition of access control, or accelerated computing of field detection.
[0058] For example, in face recognition applications, when the acceleration method is implemented by means of a computer program, the program can be stored in the storage medium of the face recognition terminal, and the acceleration method is executed by the processor of the face recognition terminal. The camera of the face recognition terminal obtains the face image, and performs recognition calculations through the acceleration method, generates face recognition results in the face recognition terminal, and uploads them to the server and other systems, and performs operations such as opening the door and playing voice through the face recognition results.
[0059] Embodiment 2
[0060] In the application of deep convolutional neural networks, the input features have more pixel data that are zero, and there are fewer non-zero values. In the prior art, the conventional acceleration method for convolution calculation is: for zero values, generally the method of skipping calculation after judging them as zero is adopted to reduce redundant calculations. At this time, a clock cycle is required to make a zero value judgment, and then skip the calculation.
[0061] In order to further improve the speed of deep convolutional neural network calculation, the deep convolutional neural network acceleration method provided by the present invention can also make the following further improvements: adopting a full-skip-zero calculation method to achieve convolution acceleration.
[0062] The above step 2: the calculation module 33 obtains the high-order features and low-order features of the input features, which may include:
[0063] Step 201: The calculation module 33 obtains the non-zero value of the input feature and the sparse mapping matrix of the input feature according to the input feature.
[0064] Step 202: The calculation module 33 obtains non-zero high-order features and low-order features.
[0065] like Fig. 9 As shown, the "1" data bit in the sparse mapping matrix represents a non-zero value, and the "0" data bit is a zero value, that is, "1" is a sparse mapping flag for a non-zero value, and "0" is a sparse mapping flag for a zero value. "1" and "0" form a corresponding matrix according to the positional relationship of the pixel points, and the matrix defines the sparse mapping matrix. The pixel matrix of the input feature can be restored through the non-zero value and the sparse mapping matrix. For example, the non-zero values are arranged from left to right and from top to bottom and filled into the "1" data bit of the sparse mapping matrix to obtain the input feature.
[0066] By processing the data corresponding to the non-zero value, the high-order features and low-order features of the non-zero data set can be obtained. In the subsequent convolution results, the data involved in the convolution are all non-zero values, and there is no need to judge the zero value, thereby accelerating the convolution calculation. In actual data storage, only non-zero values are stored in the on-chip storage module 32 or the off-chip storage module 20, which reduces the time in both calculation and data reading.
[0067] Similarly, the above sparse mapping matrix is applied to the corresponding feature images of the high-order convolution results and the low-order convolution results. That is, when the high-order convolution results and the low-order convolution results are used as a new input feature and a secondary convolution is performed, the above acceleration method can be used to accelerate the secondary convolution of the high-order convolution results and the low-order convolution results. Or it can be applied to other calculations, such as pooling, that is, only the non-zero values in the convolution results participate in the pooling.
[0068] In the above step 3: before the calculation module 33 performs convolution and maximum pooling on the high-order features, the acceleration method may further include:
[0069] Step 203: The control module 31 determines the weights of the positions corresponding to the non-zero values in the convolution kernel according to the sparse mapping matrix.
[0070] When convolving high-order data, it is still necessary to perform a convolution on all pixel points in the high-order feature as the convolution center. At this time, the convolution kernel moves point by point, and the convolution center of the convolution area corresponding to the high-order feature can be zero or non-zero, but the data in the convolution area must include at least one of non-zero and zero values. In order to increase the calculation speed, this embodiment proposes an acceleration method for all-skip zero calculation.
[0071] like Fig.10As shown, the position of non-zero values and zero values in the next convolution area can be obtained through the sparse mapping matrix. For example, when the "0" in the high-order feature (the shaded data in the figure, the second row and the third column) is used as the convolution center, the convolution area and its data are determined according to the size of the convolution kernel (3×3 in the figure). According to the sparse mapping matrix, the non-zero values (b1, e1 and h1 in the figure) and their positions of the convolution area corresponding to the "0" data as the convolution center can be determined. At the same time, according to the sparse mapping matrix, the weights (such as W, A and X in the figure) of the convolution kernel corresponding to the non-zero values (b1, e1 and h1 in the figure) when convolving this convolution area can also be determined. Then the convolution result P(0) when the "0" data is used as the convolution center is P(0) = W×b1+A×e1+X×h1. The calculation of P(0) only requires three clock cycles to complete.
[0072] In the existing convolution process, the convolution result when the "0" data is calculated as the convolution center is:
[0073] P(0)=Q×0+W×b1+E×0+A×e1+S×0+D×0+Z×0+X×h1+C×0, which requires 9 clock cycles. The above method can reduce the redundant calculation of zero values and the clock cycles occupied during the convolution process. That is to say, the calculation of zero values in high-order features (corresponding to input features) is skipped, which is referred to as zero skipping.
[0074] Therefore, by adopting the above technical solution, the weights of the positions corresponding to non-zero values in the convolution kernel can be determined when each convolution region is convolved, and all-skip-zero calculations can be achieved for high-order features, greatly improving the calculation speed. Based on the correspondence between high-order features and low-order features, all-skip-zero calculations can also be achieved when convolution calculations are performed on low-order features, and finally all-skip-zero calculations can be achieved for input features.
[0075] Based on this, in step 2 of the above technical solution, convolving the high-order features may include: convolving the high-order features of non-zero values through the weights of the positions corresponding to the non-zero values in the convolution kernel;
[0076] In step 5 of the above technical solution, convolving the low-order corresponding features may include: convolving the low-order corresponding features through the weights of the positions corresponding to the non-zero values in the convolution kernel.
[0077] Embodiment 3
[0078] The existing convolution kernels are generally 3×3 or 5×5 in size, but larger sizes are also used. The computing module 33 often uses a computing unit array (PE array) to perform convolution operations. When the matrix size of the PE array is smaller than the size of the convolution kernel, the convolution area needs to be divided and operated multiple times, which has complex logic and slow calculation speed. For example, when a 3×3 PE array calculates the convolution of a 5×5 convolution kernel, since the 3×3 PE array can calculate up to 9 data in sequence, it needs to be calculated at least three times.
[0079] In order to further improve the speed of deep convolutional neural network calculation, the deep convolutional neural network acceleration method provided by the present invention can also make the following further improvements: establish a logical mapping between convolution data (weights and non-zero values of the convolution kernel) and the PE array to simplify the calculation logic.
[0080] In the above step 3: convolution and maximum pooling are performed on the high-order features to obtain the high-order pooling results, which may include:
[0081] Step 301: The control module 31 matches the computing units of the same matrix according to the size of the convolution kernel. The computing units that match the same matrix are computing units that match the same number of weights as the convolution kernel, and each computing unit calculates the weight of a position. For example, for a 3×3 convolution kernel, 9 computing units are matched to form a PE array for convolution operations. The convolution kernel corresponds to 1 to 9 weights, which are numbered from left to right and from top to bottom, and the computing units in the PE array are numbered accordingly. For example, Fig.11 The "Q" weight in the convolution kernel in can be numbered as 1, that is, the number in the upper left corner of the PE array is 1. In other words, the number of computing units cannot be less than the number of weights of the convolution kernel.
[0082] Step 302: The control module 31 sends the data of each convolution area in the high-order feature to the corresponding computing unit according to the position relationship based on the sparse mapping matrix; and sends the weight of the corresponding non-zero value position in the convolution kernel to the corresponding computing unit.
[0083] For example, Fig.10 and Fig.11 As shown, when convolution is performed on the "0" data as the convolution center, the weights of the corresponding position relationships of the convolution kernels are sent to the corresponding computing units through the sparse mapping matrix according to the matrix position relationship, that is, the weight "W" is sent to the computing unit numbered 2 (as shown in the figure PE-2), the weight "A" is sent to the computing unit numbered 4 (as shown in the figure PE-4), and the weight "X" is sent to the computing unit numbered 8 (as shown in the figure PE-8).
[0084] Step 303: The computing module 33 performs convolution and maximum pooling on the high-order features to obtain a high-order pooling result; the computing unit is used to perform convolution on the high-order features.
[0085] like Figure 5 , Fig.10 as well as Fig.11 As shown, by adopting the above logical mapping of sending and calculating data according to the corresponding position relationship, the PE array can quickly complete the convolution calculation and ensure that the size of the PE array always meets the size of the convolution kernel.
[0086] Similarly, the above step 5: convolve the low-order corresponding features to obtain the low-order convolution results, which may include:
[0087] Step 501: Match the computing units of the same matrix according to the size of the convolution kernel.
[0088] Step 502: According to the sparse mapping matrix, the data of each convolution area in the low-order corresponding feature is sent to the corresponding computing unit according to the position relationship; and the weight of the corresponding non-zero value position in the convolution kernel is sent to the corresponding computing unit.
[0089] Step 503: Convolution is performed on the low-order corresponding features. The computing unit is used to perform convolution on the low-order features.
[0090] It can be seen that the above convolution process of high-order features and low-order corresponding features is illustrated by the process diagram of all zero skipping, but in fact it is also applicable to the conventional convolution calculation without zero skipping. The logical mapping of the convolution kernel or convolution area to the PE array, the simple mapping logic saves the time wasted on data flow and convolution kernel interception, and improves the calculation speed.
[0091] In combination with the above three embodiments, the deep convolutional neural network acceleration method provided by the present invention respectively describes the technical solutions from three aspects: two-level cascade (step-by-step convolution of high-order features and low-order features), full-skip zero calculation, and logical mapping relationship of calculation units. The technical solutions provided by the above three embodiments can improve the speed of convolution calculation and reduce the energy consumption of calculation. The various embodiments can be freely combined and can be implemented independently or in combination with each other.
[0092] Embodiment 4
[0093] The acceleration method provided in the above embodiment can be implemented in a variety of hardware structures. This embodiment provides an acceleration module 30 for a deep convolutional neural network, which may include a control module 31, a computing module 33, and an on-chip storage module 32. The acceleration module 30 is used to complete the convolution calculation using the above acceleration method to improve the speed of the convolution calculation.
[0094] The control module 31 is used to obtain input features and send the input features to the calculation module 33 .
[0095] The calculation module 33 is used to obtain high-order features and low-order features of the input features; perform convolution and maximum pooling on the high-order features to obtain high-order pooling results.
[0096] The control module 31 is also used to obtain the corresponding maximum pooling area according to the high-order pooling result, and send the low-order corresponding features to the calculation module 33 according to the maximum pooling area, wherein the low-order corresponding features are the data corresponding to the maximum pooling area in the low-order features.
[0097] The above-mentioned calculation module 33 is also used to convolve the low-order corresponding features to obtain a low-order convolution result; and obtain the maximum pooling result of the input feature according to the high-order pooling result and the low-order convolution result.
[0098] The on-chip storage module 32 is used to store at least one of the input features, high-order features, low-order features, high-order pooling results, maximum pooling areas, low-order convolution results, and maximum pooling results.
[0099] like Figure 1 and Fig.11 As shown, the computing module 33 may include a computing unit array (PE array) composed of M×M computing units (PE), and the computing unit array is used to perform convolution on high-order features and low-order corresponding features, wherein the size of the convolution kernel used by the computing module 33 is N×N, M≥N, and M and N are positive integers. By ensuring a sufficient number of computing units, a simpler mapping logic can be implemented, that is, the weights at different positions of the convolution kernel are directly convolved in the PE with the same number. For example, using a 5×5 PE array 331, when using a 3×3 convolution kernel for convolution calculation, the control module 31 controls 9 computing units to participate in the convolution calculation, and the remaining 16 computing units are dormant.
[0100] like Fig.12 As shown, each computing unit is composed of multiple sub-computing units, for example, each computing unit includes 36 sub-computing units (also called Sub-Bundles). Fig.13As shown, each sub-computing unit may include a controller 3301, a multiplier 3302, a register 3303 and two multiplexers 3304 to complete the convolution operation of the input data. The controller 3301 needs to output SEL and Out_SEL signals to select the source of the input signal and the destination of the output signal. The input source includes data from the storage part and the memory, data from the PE register and data from the external register. At the same time, the output destination includes entering the activation function module 332, entering the part responsible for storing the sum and the memory, the PE register and the external register. Of course, the sub-computing unit can also adopt any structure of the existing minimum multiplication and addition operation unit that can achieve the above effect.
[0101] Among them, several sub-computing units can form a group of sub-computing units, and one sub-computing unit or a group of sub-computing units is used to calculate the weight of a corresponding position of the convolution kernel of a channel. Because the convolution kernel has a depth parameter in addition to size, one depth corresponds to one channel. When the convolution kernel has multiple channels, each sub-computing unit or each group of sub-computing units calculates for one channel. According to the actual size of the convolution kernel and the number of channels, the corresponding computing unit and sub-computing unit are matched.
[0102] Before performing convolution, the control module 31 can also enable the corresponding number of sub-computation units according to the number of channels (depth) of the convolution kernel. Therefore, when performing convolution on a convolution region in a high-order feature or a low-order feature, a PE array 331 composed of a computing unit having multiple sub-computation units is used to complete the convolution calculation in the shortest time with the convolution kernels of multiple channels.
[0103] The calculation module 33 is used to obtain the non-zero value of the input feature and the sparse mapping matrix of the input feature, where the input feature is a pixel matrix; and obtain the high-order feature and the low-order feature of the non-zero value.
[0104] In a practical application, Figure 1 As shown, the calculation module 33 may also include an activation function module 332. The activation function module 332 is used to obtain the non-zero value of the input feature, the sparse mapping flag of the non-zero value, and the sparse mapping matrix of the input feature, where the input feature is a pixel matrix; and obtain the high-order feature and low-order feature of the non-zero value. The activation function module 332 may use a ReLU function.
[0105] like Fig.13 As shown, the acceleration module 30 provided by the present invention also includes a sparse mapping flag and data storage module (referred to as a sparse mapping module 34), which is used to generate addresses according to the sparse mapping matrix, and write non-zero values and non-zero sparse mapping flags to the off-chip storage module 20. The sparse mapping module 34 is also used to read non-zero values and non-zero sparse mapping flags from the off-chip storage module 20.
[0106] For example, Fig.14 As shown, the sparse mapping module 34 may include a write address generation module 341, a write cache module 342, an asynchronous FIFO data module 343, and a read address generation module 344. The non-zero value of the input feature generates an output feature (i.e., a high-order feature and a low-order feature), a sparse mapping flag, and coordinate information data associated with the output feature (high-order feature and low-order feature) after passing through the activation function module 332. The output feature and the sparse mapping flag data are sent to the write cache module 342 and arithmetically expanded into corresponding bit data, and the coordinate information data associated with the data is sent to the write address generation module 341 and converted into a write address according to a specific formula. Since the clock frequency of the on-chip storage module 32 is different from the clock frequency of the off-chip storage module 20, the asynchronous FIFO data module 343 is used for data interaction. The read address generation module 344 obtains the read address according to the coordinate information sent by the control module 31 and reads the non-zero value and sparse mapping mark data from the off-chip storage module 20 to the on-chip storage module 32, and sends a data loading success signal to the control module 31.
[0107] More specifically, the coordinates of a pixel in the input feature (pixel matrix) can be obtained through the sparse mapping matrix and can be expressed as (x, y, ch), where x represents the horizontal coordinate, y represents the vertical coordinate, and ch represents the channel. The write address ad1 of the sparse mapping flag is ad1 = x×y×ch / 16+y×ch / 16+ch / 16; the write address ad2 of the output feature (high-order feature and low-order feature) is ad2 = x×y×ch / 2+y×ch / 2+ch / 2.
[0108] The above-mentioned on-chip storage module 32 can be divided according to different types of data, one part of which stores input features, high-order features and low-order features, another part stores convolution kernel data, another part stores sparse mapping matrix data, and the last part stores the partial sum (Partial sum) generated by the convolution of the PE array 331.
[0109] Embodiment 5
[0110] like Figure 1 As shown, an acceleration system for a deep convolutional neural network provided by the present invention includes a processor 10, an off-chip storage module 20, and an acceleration module 30 described in the above technical solution;
[0111] The processor 10 is used to control data exchange between the off-chip storage module 20 and the on-chip storage module 32, and / or the processor 10 is a Rocket general-purpose RISC-V processor.
[0112] The Rocket processor includes an open source processor, a data path, a data cache (DCache) and an instruction cache (ICache), and its main function is to control the working state of the acceleration module 30. Compared with other neural network acceleration systems that use dedicated compilers, the Rocket processor is a RISC-V-based processor and is an open source project. It is very convenient for development and application, reducing the cost of additional learning. The Rocket processor has better versatility and portability. Different types of processors are generally developed for corresponding computer languages. Some processors are strictly bound to computer languages and are not open source, so their portability is low.
[0113] Embodiment 6
[0114] In this embodiment, the acceleration system for deep convolutional neural network provided by the present invention to implement the acceleration method will be further illustrated to realize a two-level cascade, full-jump-to-zero acceleration method.
[0115] Step 10: The off-chip storage module 20 loads the input features, and the processor 10 preprocesses the input features so that the number of bits of the pixel points in the input features matches the number of bits calculated by the acceleration module 30, such as converting into 16-bit data.
[0116] Step 20 : The processor 10 sends the input features to the on-chip storage module 20 , and the control module 31 sends the input features to the calculation module 33 .
[0117] Step 30: The calculation module 33 uses the activation function module to process the input features to generate non-zero high-order features, low-order features, sparse mapping matrix and coordinate information. The control module 31 sends the high-order features, low-order features, sparse mapping matrix and coordinate information to the sparse mapping module 34.
[0118] Step 40 : The sparse mapping module 34 generates corresponding write addresses and read addresses of high-order data and low-order data according to the coordinate information of the high-order features and low-order features, and stores the high-order features and low-order features in the off-chip storage module 20 or the on-chip storage module 32 .
[0119] Step 50: The control module 31 sends the high-order features to the calculation module 33, and the calculation module 33 performs convolution and maximum pooling on the high-order features to obtain the high-order pooling results. When performing convolution, the participating data in the high-order features and the weights corresponding to the convolution kernels are sent to the corresponding calculation units for calculation according to the position relationship.
[0120] Step 60: the control module 31 obtains the maximum pooling area according to the high-order pooling result, and determines the data corresponding to the maximum pooling area in the low-order features according to the maximum pooling area, that is, determines the low-order corresponding features.
[0121] Step 70: The control module 31 sends the low-order corresponding features to the calculation module 33, and the calculation module 33 performs convolution on the low-order corresponding features to obtain the low-order convolution result. When performing convolution, the participating data in the low-order corresponding features and the weights corresponding to the convolution kernel are sent to the corresponding calculation unit for calculation according to the position relationship.
[0122] Step 80: The calculation module 33 superimposes the high-order pooling result and the low-order convolution result to obtain the maximum pooling result of the input feature.
[0123] Step 90: The processor generates a recognition result based on the maximum pooling result, and sends the result to the display for display, reflecting the recognition result of the input feature.
[0124] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above-mentioned embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above-mentioned functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above-mentioned functions can be implemented. In addition, when all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and can be downloaded or copied and saved in the memory of the local device, or the system of the local device is updated, and when the program in the memory is executed by the processor, all or part of the functions in the above-mentioned embodiments can be implemented.
[0125] The above specific examples are used to illustrate the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. For those skilled in the art, according to the concept of the present invention, some simple deductions, modifications or substitutions can be made.
Claims
1. A deep convolutional neural network acceleration method, characterized in that: include: Obtaining input features, where the input features are a pixel matrix of the image; Obtaining high-order features and low-order features of the input features; Performing convolution and maximum pooling on the high-order features to obtain a high-order pooling result; According to the high-order pooling result, a corresponding maximum pooling area is obtained; Convolving the low-order corresponding features to obtain a low-order convolution result, wherein the low-order corresponding features are data corresponding to the maximum pooling area in the low-order features; Obtaining a maximum pooling result of the input feature according to the high-order pooling result and the low-order convolution result; The step of obtaining the high-order features and the low-order features of the input features includes: According to the input feature, obtain a non-zero value of the input feature and a sparse mapping matrix of the input feature; Obtain the high-order feature and the low-order feature of the non-zero value; The high-bit data of each pixel in the input feature forms the high-bit feature, and the low-bit data of each pixel forms the low-bit feature.
2. The deep convolutional neural network acceleration method according to claim 1, characterized in that: Before performing convolution and maximum pooling on the high-order features, the method further includes: determining the weight of the position corresponding to the non-zero value in the convolution kernel according to the sparse mapping matrix; The convolving the high-order features comprises: convolving the high-order features of the non-zero values by weights of the positions corresponding to the non-zero values in the convolution kernel; The convolving the low-order corresponding features includes: convolving the low-order corresponding features through the weights of the positions corresponding to the non-zero values in the convolution kernel.
3. The deep convolutional neural network acceleration method according to claim 1 or 2, characterized in that: The convolution and maximum value pooling are performed on the high-order features to obtain a high-order pooling result, including: According to the size of the convolution kernel, match the computing units of the same matrix; According to the sparse mapping matrix, the data of each convolution area in the high-order feature is sent to the corresponding computing unit according to the position relationship; and the weight corresponding to the non-zero value position in the convolution kernel is sent to the corresponding computing unit; Convolution and maximum pooling are performed on the high-order features to obtain a high-order pooling result; the computing unit is used to convolve the high-order features.
4. The deep convolutional neural network acceleration method according to claim 1 or 2, characterized in that: The convolution of the low-order corresponding features to obtain the low-order convolution result includes: According to the size of the convolution kernel, match the computing units of the same matrix; According to the sparse mapping matrix, the data of each convolution area in the low-order corresponding feature is sent to the corresponding computing unit according to the position relationship; and the weight corresponding to the non-zero value position in the convolution kernel is sent to the corresponding computing unit; Convolution is performed on the low-order corresponding features, and the computing unit is used to perform convolution on the low-order features.
5. An acceleration module for a deep convolutional neural network, characterized in that: It includes a control module and a calculation module; The control module is used to obtain input features, where the input features are pixel matrices of images; The calculation module is used to obtain the high-order features and low-order features of the input features; Performing convolution and maximum pooling on the high-order features to obtain a high-order pooling result; The control module is also used to obtain a corresponding maximum pooling area according to the high-order pooling result; The calculation module is also used to perform convolution on the low-order corresponding features to obtain a low-order convolution result, wherein the low-order corresponding features are data corresponding to the maximum pooling area in the low-order features; according to the high-order pooling result and the low-order convolution result, the maximum pooling result of the input feature is obtained; The step of obtaining the high-order features and low-order features of the input features by the calculation module includes: The calculation module obtains the non-zero value of the input feature and the sparse mapping matrix of the input feature; obtains the high-bit feature and the low-bit feature of the non-zero value; wherein the high-bit data of each pixel point in the input feature forms the high-bit feature, and the low-bit data of each pixel point forms the low-bit feature.
6. The acceleration module according to claim 5, characterized in that: The computing module includes a computing unit array consisting of M×M computing units, and the computing unit array is used to convolve the high-order features and the low-order corresponding features, wherein the size of the convolution kernel used by the computing module is N×N, M≥N, and M and N are both positive integers.
7. An acceleration system for deep convolutional neural networks, characterized in that: It comprises a processor, an off-chip storage module and the acceleration module according to claim 5 or 6; the acceleration module also comprises an on-chip storage module; The processor is used to control data exchange between the off-chip storage module and the on-chip storage module, and / or the processor is a Rocket general-purpose RISC-V processor.
8. A computer-readable storage medium, characterized in that: The medium stores a program, which can be executed by a processor to implement the method according to any one of claims 1 to 4.
