Pooler and pooling acceleration circuit for the maximum pooling layer of convolutional neural network
By designing an acceleration circuit for the maximum pooling layer of convolutional neural network in the spacecraft, using components such as selectors, comparators and control units to achieve efficient maximum pooling layer calculation, solving the calculation of CNN algorithm in the spacecraft Force, power consumption and real-time requirements reduce circuit complexity and resource usage.
Patent Information
- Application Number
- CN202111632969.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-28
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2041-12-28
AI Technical Summary
Existing deep learning technologies based on general-purpose processors are difficult to meet the high computing power, low power consumption and real-time requirements of CNN algorithms in spacecraft, especially in the calculation of the maximum pooling layer of convolutional neural networks.
A pooler including selector, comparator, constant register and pooling register is designed. Combined with an acceleration circuit of control unit and pooling cache, it realizes efficient calculation of the maximum pooling layer of the convolutional neural network, and supports arbitrary Pooling operations of window shapes and sizes, and optimize data loading through the data sorting adjuster.
Efficient maximum pooling layer calculations are achieved with the smallest possible FPGA resource consumption, solving real-time and power consumption problems, reducing circuit logic complexity and resource usage, and improving the matching degree of data throughput and bandwidth.
Smart Images
Figure CN114265696B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of deep learning acceleration circuits, and specifically relates to a pooler and a pooling acceleration circuit for a maximum pooling layer of a convolutional neural network. Background Art
[0002] As the performance of deep learning technology represented by Convolutional Neural Network (CNN) continues to improve in the fields of image classification and target detection. In the process of the development of the new generation of aerospace models towards intelligence, a large number of application requirements have emerged for using CNN for target detection, fault identification and other aspects. However, due to the following reasons, the previous information processing methods based on general-purpose processors are increasingly difficult to meet the application requirements of models: first, the calculation amount and parameter amount of deep learning algorithms based on CNN are huge, and high-computing power devices are required to perform real-time reasoning on them; second, the internal and external heat dissipation conditions of spacecraft are harsh, and high-power computing devices are not allowed to work for a long time; finally, most applications in the aerospace field are strong real-time application scenarios, and low-computing power devices are difficult to meet real-time requirements. The general solution to the above problems is to design a hardware acceleration unit for CNN based on FPGA, and a complete and available acceleration unit design usually needs to include multiple functions such as convolution, pooling, nonlinear activation, preprocessing, post-processing, ShortCut, etc. In order to realize the most general and efficient CNN acceleration unit design within the limited FPGA resource consumption, it is necessary to realize the most commonly used computing functions at the lowest possible resource cost. The present invention discloses a simplified maximum pooling layer hardware acceleration circuit design method with high data throughput that covers the computing requirements of commonly used CNN algorithms. Summary of the invention
[0003] In view of the above analysis, the present invention aims to disclose a pooler and a pooling acceleration circuit for the maximum pooling layer of a convolutional neural network, to achieve a streamlined hardware acceleration circuit for the maximum pooling layer with high data throughput that covers the computational requirements of commonly used CNN algorithms.
[0004] The present invention discloses a pooler for a maximum pooling layer of a convolutional neural network, comprising a first selector S1, a second selector S2, a comparator, a constant register and a pooling register;
[0005] The first input terminal of the comparator inputs the feature data in the pooling window, and the second input terminal is connected to the output data of the first selector S1, and the maximum value obtained by comparison is output to the second selector S2;
[0006] The first input terminal of the first selector S1 is connected to the output terminal of the constant register, the second input terminal is connected to the external pooling cache to read data therefrom, and the third input terminal is connected to the output terminal of the pooling register; under the control of the external instruction, the input data of one of the input terminals is selected to be output to the second input terminal of the comparator;
[0007] The first output end of the second selector S2 serves as the output end of the final pooling result, the second output end is connected to the external pooling cache to write data to it, and the third output end is connected to the input end of the pooling register; under the control of the external instruction, an output end is selected to output the comparison result of the comparator.
[0008] Furthermore, the external control unit determines the position of the input feature data in the pooling window according to the configuration information of the current pooling layer and the position of the current input feature data in the feature map; and outputs a control instruction to the first selector for input gating control and to the second selector for output gating control according to the position, so as to realize the pooling calculation of the pooling window.
[0009] Further, when the input feature data is the first data in the pooling window, the control unit controls the first selector S1 to select the first input terminal, and the second selector S2 to select the third output terminal;
[0010] The data in the constant register is selected and output to the comparator for comparison with the data, and the maximum value is obtained and output to the pooling register through the second selector.
[0011] Further, when the input feature is the last data of each row in the pooling window and not the last data of the entire pooling window, the control unit controls the first selector S1 to select the third input terminal and the second selector S2 to select the second output terminal;
[0012] The data in the pooling register is selected and output to the comparator for comparison with the data, and the maximum value is obtained as the pooling result of the row and output to the external pooling cache through the second selector S2 for storage.
[0013] Further, when the input feature is the first data of each row of the pooling window and not the first data of the entire pooling window, the control unit controls the first selector S1 to select the second input terminal and the second selector S2 to select the third output terminal;
[0014] By selecting the data of the corresponding address in the pooling cache and comparing it with the data, the maximum value is output to the pooling register through the second selector S2; the data of the corresponding address in the pooling cache is the pooling result of the previous row stored in the pooling cache address in advance.
[0015] Further, when the input feature is the last number of the pooling window, the control unit controls the first selector S1 to select the third input terminal, and the second selector S2 to select the first output terminal;
[0016] The data in the pooling register is selected and output to the comparator for comparison with the data, and the maximum value is obtained as the pooling result of the pooling window and output to the external memory for storage through the second selector S2.
[0017] Further, when the input feature is data other than the first column and the last column in the pooling window, the control unit controls the first selector S1 to select the third input terminal, and the second selector S2 to select the third output terminal;
[0018] The data in the pooling register is selected and output to the comparator for comparison with the data, and the maximum value is obtained and stored in the pooling register through the second selector S2.
[0019] The present invention also discloses a pooling acceleration circuit for a maximum pooling layer of a convolutional neural network, comprising a control unit, a pooler array and a pooling cache;
[0020] The pooler array comprises n poolers according to any one of claims 1 to 7;
[0021] The control unit is connected to n poolers, and controls the calculation of each pooler according to the position of the feature data input into each pooler in the pooling window;
[0022] The pooling cache is connected to n poolers and is used to support parallel data reading and writing of the n poolers.
[0023] Furthermore, the pooling cache caches the pooling intermediate results of one row in each pooler; the depth of the pooling cache is greater than or equal to W max / S',W max is the maximum feature map width supported, and S' is the corresponding pooling step size.
[0024] Furthermore, it also includes a data sorting adjuster;
[0025] The data sorting adjuster re-sorts the feature data and loads them sequentially during calculation;
[0026] The default sorting of feature data is expressed as Fin[N][Hin][Win];
[0027] The feature data after the data sorting adjuster adjusts the sorting mode is a 4-dimensional tensor, Fin[N / n][Hin][Win][n]; where H in is the height of the feature map, W inis the width of the feature map, N is the number of channels of the feature map, and n is the number of poolers;
[0028] And the feature data of the 4-dimensional tensor is continuously stored in the external DDR in the order from low dimension to high dimension, and is called when the pooling acceleration circuit is calculated.
[0029] The present invention can achieve at least one of the following beneficial effects:
[0030] The present invention realizes efficient calculation of the maximum pooling layer in common CNN with the smallest possible FPGA resource consumption, thereby solving the real-time and power consumption problems encountered when deploying CNN in embedded devices.
[0031] And it realizes efficient calculation of the maximum pooling layer of arbitrary window shapes and sizes in CNN;
[0032] Reduce the logic structure complexity of this part of the circuit and reduce the usage of FPGA logic resources, so that the data throughput and data bandwidth are highly matched. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.
[0034] Figure 1 It is a schematic diagram of the connection of the pooling device in the embodiment of the present invention;
[0035] Figure 2 A schematic diagram of the connection components of a pooling acceleration circuit in an embodiment of the present invention;
[0036] Figure 3 4 is a schematic diagram of a maximum pooling process with a pooling window size of 2×2 in an embodiment of the present invention. DETAILED DESCRIPTION
[0037] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used to illustrate the principles of the present invention together with the embodiments of the present invention.
[0038] One embodiment of the present invention discloses a pooler for a maximum pooling layer of a convolutional neural network, such as Figure 1 As shown, it includes: a first selector S1, a second selector S2, a comparator, a constant register and a pooling register;
[0039] The first input terminal of the comparator inputs the feature data in the pooling window, and the second input terminal is connected to the output data of the first selector S1, and the maximum value obtained by comparison is output to the second selector S2;
[0040] The first input terminal of the first selector S1 is connected to the output terminal of the constant register, the second input terminal is connected to the external pooling cache to read data therefrom, and the third input terminal is connected to the output terminal of the pooling register; under the control of the external instruction, the input data of one of the input terminals is selected to be output to the second input terminal of the comparator;
[0041] The first output end of the second selector S2 serves as the output end of the final pooling result, the second output end is connected to the external pooling cache to write data to it, and the third output end is connected to the input end of the pooling register; under the control of the external instruction, an output end is selected to output the comparison result of the comparator.
[0042] Specifically, the constant register stores the minimum value that can be represented by the current feature map data bit width. If the feature map is a signed integer represented by 8 bits, the minimum value is 0xFF. The pooling register is used to store the intermediate results of the pooling calculation.
[0043] Specifically, the external control unit determines the position of the input feature data in the input pooling window for pooling calculation according to the configuration information of the current pooling layer and the position (x, y) of the current input feature data in the feature map; outputs a control instruction to the first selector for input gating control according to the position, and to the second selector S2 for output gating control to realize the pooling calculation of the pooling window; the configuration information includes the height H of the feature map f , the width of the feature map W f , the number of channels of the feature map N, the height of the pooling window H w , the width of the pooling window W w , pooling step size S.
[0044] The external control unit performs gating control according to the position of the determined input feature data in the input pooling window for performing pooling calculation, including:
[0045] 1) When the input feature data is the first data in the pooling window, the control unit controls the first selector S1 to select the first input terminal and the second selector S2 to select the third output terminal;
[0046] The data in the constant register is selected and output to the comparator for comparison with the data, and the maximum value is obtained and output to the pooling register through the second selector S2.
[0047] 2) When the input feature is the last data of each row in the pooling window and not the last data of the entire pooling window, the control unit controls the first selector S1 to select the third input terminal and the second selector S2 to select the second output terminal;
[0048] The data in the pooling register is selected and output to the comparator for comparison with the data, and the maximum value is obtained as the pooling result of the row and output to the external pooling cache through the second selector S2 for storage.
[0049] 3) When the input feature is the first data of each row of the pooling window and not the first data of the entire pooling window, the control unit controls the first selector S1 to select the second input terminal and the second selector S2 to select the third output terminal;
[0050] The maximum value is obtained by selecting the data of the corresponding address in the pooling cache and comparing it with the data, and is output to the pooling register through the second selector S2.
[0051] The data at the corresponding address in the pooling cache is the pooling result of the previous row previously stored in the pooling cache address.
[0052] 4) When the input feature is the last number of the pooling window, the control unit controls the first selector S1 to select the third input terminal and the second selector S2 to select the first output terminal;
[0053] The data in the pooling register is selected and output to the comparator for comparison with the data, and the maximum value is obtained as the pooling result of the pooling window and output to the external memory for storage through the second selector S2; the external memory is such as DDR.
[0054] 5) When the input feature is data other than the first column and the last column in the pooling window, the control unit controls the first selector S1 to select the third input terminal, and the second selector S2 to select the third output terminal;
[0055] The data in the pooling register is selected and output to the comparator for comparison with the data, and the maximum value is obtained and stored in the pooling register through the second selector S2.
[0056] According to the structure and workflow of the pooler, the pooling window input into the pooler for pooling calculation can be a pooling window of any shape and size; and it supports edge expansion of any size of the feature map, but the pooling step size must be consistent with the pooling window size.
[0057] For the case where the pooling window is incomplete at the edge of the feature map, special processing can be performed according to the above design ideas.
[0058] The present invention also discloses a pooling acceleration circuit for the maximum pooling layer of a convolutional neural network, such as Figure 2 As shown, it includes a control unit, a pooler array and a pooling cache;
[0059] The pooler array includes n poolers as described above;
[0060] The control unit is connected to n poolers, and controls the calculation of each pooler according to the position of the feature data input into each pooler in the pooling window;
[0061] The pooling cache is connected to n poolers to support parallel data reading and writing of the n poolers. If the bit width of the feature data is 8 bits, the width of the pooling cache is n×8 bits.
[0062] In this embodiment, the pooling cache only needs to cache one row of pooling intermediate results. For example, the maximum feature map width supported by the current design is W. max , and the corresponding pooling step is S', then the cache depth must be greater than or equal to W max / S'.
[0063] Specifically, in order to improve the continuity of loading data from the external storage DDR, and thus improve the actual bandwidth of the system during operation; the pooling circuit of this embodiment also includes a data sorting adjuster;
[0064] The data sorting adjuster reorders the feature data and loads them sequentially during calculation;
[0065] The default sorting of feature data is expressed as: Fin[N][Hin][Win];
[0066] The feature data after the data sorting adjuster adjusts the sorting method can be expressed as a 4-dimensional tensor as shown below, Fin[N / n][Hin][Win][n]; where H in is the height of the feature map, W in is the width of the feature map, N is the number of channels of the feature map, and n is the number of poolers.
[0067] The feature data is stored continuously in the external DDR in order from low dimension to high dimension, and is called when the pooling acceleration circuit is calculated.
[0068] A specific solution in this embodiment provides a maximum pooling process with a pooling window size of 2×2, such as Figure 3 The above is a diagram illustrating the principle of the 2×2 maximum pooling process. Figure 3 In the example, the parameters of the maximum pooling layer are set as follows: the height of the feature map is H in , the width of the feature map is W in , the number of channels of the feature map is N, the pooling window size is 2×2, and the pooling step size is 2.
[0069] In this embodiment, the number of poolers is n, which means that feature maps of n channels can be processed simultaneously.
[0070] In the maximum pooling process with a pooling window size of 2×2,
[0071] The data sorter adjusts the feature data represented by the default sorting method Fin[N][Hin][Win] to a 4-dimensional tensor of Fin[N / n][Hin][Win][n]; and stores it continuously in the external DDR in order from low dimension to high dimension for calling during calculation by the pooling acceleration circuit.
[0072] The implementation process of the present invention is described by taking the first channel pooler as an example. The data of the first channel is as follows: Figure 3 As shown in the figure. When the first number 7 enters the pooler, it is compared with the minimum value 0xFF and stored in the pooling register as the maximum value; when 8 enters the pooler, it is compared with 7 in the register and stored in the pooling cache as the maximum value. This process is repeated until the first row of feature maps is processed.
[0073] When the second line of feature data 0 enters the pooler, it is compared with 8 in the pooling cache, and 8 is still stored in the pooling register as the maximum value; when the last number 5 in the pooling window enters the pooler, it is compared with 8 in the register, and 8 is output to the outside of the chip as the pooling result of this pooling window.
[0074] And so on, the whole feature map is processed. The working principle of other channels is exactly the same. Each pooler works independently and performs pooling operations on the feature data of each channel.
[0075] In summary, this embodiment achieves efficient calculation of the maximum pooling layer in common CNN with the smallest possible FPGA resource consumption, thereby solving the real-time and power consumption problems encountered when deploying CNN in embedded devices; and achieves efficient calculation of the maximum pooling layer of any window shape and size in CNN; reduces the logic structure complexity of this part of the circuit, reduces the use of FPGA logic resources, and makes the data throughput highly matched with the data bandwidth.
[0076] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.
Claims
1. A pooler for a maximum pooling layer of a convolutional neural network, characterized in that: It includes a first selector S1, a second selector S2, a comparator, a constant register and a pooling register; The first input terminal of the comparator inputs the feature data in the pooling window, and the second input terminal is connected to the output data of the first selector S1, and the maximum value obtained by comparison is output to the second selector S2; The first input terminal of the first selector S1 is connected to the output terminal of the constant register, the second input terminal is connected to the external pooling cache to read data therefrom, and the third input terminal is connected to the output terminal of the pooling register; under the control of the external instruction, the input data of one of the input terminals is selected to be output to the second input terminal of the comparator; The first output end of the second selector S2 serves as the output end of the final pooling result, the second output end is connected to the external pooling cache to write data to it, and the third output end is connected to the input end of the pooling register; under the control of the external instruction, an output end is selected to output the comparison result of the comparator.
2. The pooling device according to claim 1, characterized in that: The external control unit determines the position of the input feature data in the pooling window according to the configuration information of the current pooling layer and the position of the current input feature data in the feature map; outputs a control instruction to the first selector for input gating control and to the second selector for output gating control according to the position, so as to realize the pooling calculation of the pooling window.
3. The pooling device according to claim 2, characterized in that: When the input feature data is the first data in the pooling window, the control unit controls the first selector S1 to select the first input terminal, and the second selector S2 to select the third output terminal; The data in the constant register is selected and output to the comparator for comparison with the data, and the maximum value is obtained and output to the pooling register through the second selector.
4. The pooling device according to claim 2, characterized in that: When the input feature is the last data of each row in the pooling window and not the last data of the entire pooling window, the control unit controls the first selector S1 to select the third input terminal and the second selector S2 to select the second output terminal; The data in the pooling register is selected and output to the comparator for comparison with the data, and the maximum value is obtained as the pooling result of the row and output to the external pooling cache through the second selector S2 for storage.
5. The pooling device according to claim 2, characterized in that: When the input feature is the first data of each row of the pooling window and not the first data of the entire pooling window, the control unit controls the first selector S1 to select the second input terminal and the second selector S2 to select the third output terminal; By selecting the data of the corresponding address in the pooling cache and comparing it with the data, the maximum value is output to the pooling register through the second selector S2; the data of the corresponding address in the pooling cache is the pooling result of the previous row stored in the pooling cache address in advance.
6. The pooling device according to claim 2, characterized in that: When the input feature is the last number of the pooling window, the control unit controls the first selector S1 to select the third input terminal, and the second selector S2 to select the first output terminal; The data in the pooling register is selected and output to the comparator for comparison with the data, and the maximum value is obtained as the pooling result of the pooling window and output to the external memory for storage through the second selector S2.
7. The pooling device according to claim 2, characterized in that: When the input feature is data other than the first column and the last column in the pooling window, the control unit controls the first selector S1 to select the third input terminal, and the second selector S2 to select the third output terminal; The data in the pooling register is selected and output to the comparator for comparison with the data, and the maximum value is obtained and stored in the pooling register through the second selector S2.
8. A pooling acceleration circuit for the maximum pooling layer of a convolutional neural network, characterized in that: Includes a control unit, a pooler array and a pooling cache; The pooler array comprises n poolers according to any one of claims 1 to 7; The control unit is connected to n poolers, and controls the calculation of each pooler according to the position of the feature data input into each pooler in the pooling window; The pooling cache is connected to n poolers and is used to support parallel data reading and writing of the n poolers.
9. The pooling acceleration circuit according to claim 8, characterized in that: The pooling cache caches the pooling intermediate results of one row in each pooler; the depth of the pooling cache is greater than or equal to W max / S',W max is the maximum feature map width supported, and S' is the corresponding pooling step size.
10. The pooling acceleration circuit according to claim 8, characterized in that: Also includes data sort adjuster; The data sorting adjuster reorders the feature data and loads them sequentially during calculation; The default sorting of feature data is expressed as Fin[N][Hin][Win]; The feature data after the data sorting adjuster adjusts the sorting mode is a 4-dimensional tensor, Fin[N / n][Hin][Win][n]; where H in is the height of the feature map, W in is the width of the feature map, N is the number of channels of the feature map, and n is the number of poolers; And the feature data of the 4-dimensional tensor is continuously stored in the external DDR in the order from low dimension to high dimension, and is called when the pooling acceleration circuit is calculated.