An Optimization and Speed-Up Method for a Lightweight CNN Classifier Based on FPGA
Through quantitative perception training and collaborative parallel pipeline methods, the CNN network deployment problem caused by FPGA resource constraints is solved, and an efficient, real-time and low-power CNN classifier is realized.
Patent Information
- Application Number
- CN202210679183.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-16
- Publication Date
- 2025-03-11
- Estimated Expiration
- 2042-06-16
AI Technical Summary
FPGA resources are limited, resulting in large amounts of calculation and parameters when deploying CNN networks on FPGAs. The compression model will reduce the recognition accuracy, and storing off-chip parameters will affect real-time.
Quantitative awareness training (QAT) is used to compress the 32-bit floating-point model parameters into 8-bit fixed-point integers, and implement fixed-point number operation on FPGA hardware. At the same time, a collaborative parallel pipeline method is used within and between adjacent layers of lightweight CNN networks.
It improves the real-time and accuracy of CNN classifiers on FPGAs, reduces power consumption and resource usage, and is suitable for embedded devices with limited resources.
Smart Images
Figure CN114925780B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of target classification, and in particular relates to an optimization and speed-up method for a lightweight CNN classifier based on FPGA. Background Art
[0002] Convolutional Neural Network (CNN) is good at extracting target features from massive data. Due to its excellent classification and recognition capabilities, it has been widely used in image classification, speech recognition, medical diagnosis, national defense and security. In order to improve the recognition accuracy of convolutional networks, the complexity and computational complexity of convolutional neural networks are also increasing. The huge number of parameters makes it difficult to deploy convolutional neural networks on embedded devices with limited resources. The birth of lightweight networks has greatly reduced the number of network parameters. The current mainstream lightweight networks, such as MobilenetV2, use a depth-separable convolution structure, which consists of spatial convolution (Depthwise Convolution, DW) and pointwise convolution (Pointwise Convolution, PW) operations. While reducing the number of network parameters and ensuring recognition accuracy, it is very suitable for deployment on small-capacity embedded devices and applied to target classification scenarios.
[0003] Deploying CNN on a traditional processor CPU will result in high latency. If parallel computing is used, the GPU will consume a lot of power. Field Programmable Gate Array (FPGA) has the advantages of real-time, low power consumption, and reconfigurability, and is particularly suitable as a hardware accelerator for CNN. At present, the main problem faced by deploying the mainstream CNN network in the AI field on FPGA is that FPGA resources are limited, and the amount of calculation and parameters of the CNN network is relatively large. If the parameters are reduced by compressing the model, the final recognition accuracy of the network will be reduced. If the network parameters are stored off-chip, there will be bandwidth limitations, affecting real-time performance. Summary of the invention
[0004] The purpose of the present invention is to provide an optimization speed-up method for a lightweight CNN classifier based on FPGA, which is beneficial to improving the real-time performance of the classifier.
[0005] To achieve the above purpose, the technical solution adopted by the present invention is: an optimization and speed-up method of a lightweight CNN classifier based on FPGA, comprising the following steps:
[0006] 1) On the PC side, use the built lightweight CNN network to train with the data set for image classification to obtain the network model with the best classification effect after multiple trainings; at the same time, design the model architecture exported from the PC side on the FPGA hardware side;
[0007] 2) Perform quantization-aware training on the trained network model, save the 32-bit floating-point model parameters as 8-bit fixed-point integers, and implement the complete architecture of the lightweight CNN network on the FPGA hardware side;
[0008] 3) The weights after quantization perception training are written into the coe file format according to the designed parallelism and the read and write order required by the hardware, and imported into the on-chip storage of the FPGA. The lightweight CNN classifier can then read the input image data to realize classification reasoning.
[0009] Furthermore, the overall architecture of the lightweight CNN classifier system is mainly composed of a host computer, a hardware acceleration module, an on-chip memory and a display module. The network weights are quantized and arranged in order according to the parallelism of the DW and PW layers and the data reading rules, and written into the on-chip memory in advance; the host computer is only responsible for transmitting the input image data to the input buffer through the PCIE bus; after the hardware acceleration module detects that the image data has been loaded, it reads the weights from the on-chip memory and starts forward reasoning acceleration.
[0010] Furthermore, the lightweight CNN network adopts the lightweight network MobilenetV2, and the hardware acceleration module is deployed corresponding to the MobileNetV2 network structure, mainly consisting of a Conv2d layer, a DW layer, a PW layer, and a pooling layer; the final classification result is displayed on the display module.
[0011] Furthermore, after the parameters are quantized, a new fixed-point calculation formula is needed to replace the original floating-point convolution operation formula to be implemented on the hardware; assuming that the convolution weight is w, the bias is b, the input is x, and the output activation value is a, since convolution is essentially a matrix operation, it is expressed as shown in the following formula:
[0012]
[0013] Let r represent a floating-point real number, q represent a quantized fixed-point integer, S represent the quantized scale factor, and Z represent the integer value corresponding to 0 after quantization. The conversion formula between floating-point numbers and fixed-point numbers is as follows:
[0014] r=S(qZ) (2)
[0015]
[0016] Represent w, b, and x in formula (1) with quantized fixed-point numbers, and we get formula (4):
[0017]
[0018] The entire formula (4) only has is a floating point number, where S w is the quantization scale of the weight, S x is the quantization scale of the input, S a is the quantization scale of the output; assuming As long as we find the right n and M0 so that M≈2 -n M0, formula (4) can be completely converted into fixed-point operations.
[0019] Furthermore, the hardware implementation steps of the quantization calculation are as follows:
[0020] A) For each convolution operation, the result of the multiplication and accumulation of the image and weight is stored in a 32-bit register, the multiplier is also stored in a 32-bit register, and the multiplication result of the two is stored in a 64-bit register;
[0021] B) Perform a round operation on the hardware side for the multiplication and accumulation result of the image and the weight, followed by a shift operation, with the round and shift operations working together as one beat;
[0022] C) The shifted result is truncated into 8-bit output after relu, and the relu and truncation are combined in one shot.
[0023] Furthermore, a collaborative parallel pipeline method is adopted within the layers of the lightweight CNN network and between adjacent PW layers to accelerate the forward propagation process and improve the real-time performance of the classifier; the lightweight CNN network is mainly composed of PW layers and DW layers. In order to improve the computational parallelism of the DW layer, a strategy combining single convolution operation expansion with input channel expansion is adopted, specifically:
[0024] The convolution window N of the DW layer is 3, that is, a 3×3 convolution window. The 3×3 convolution calculation is flattened, and the 3×3 data at the input image position is multiplied by the corresponding weight, that is, 9 multiplication calculations are completed in one clock cycle; the k-channel parallel mode is adopted on the input channel, and the entire DW layer completes 9×k multiplication calculations in one FPGA clock cycle, and outputs the data at the same position of k channels after subsequent operations;
[0025] The PW layer adopts a parallel mode combining k input channels and k output channels; it reads the data at the same position of the k input channels and the weights of the k groups of channels at the same time in one CLK to calculate, reads the input Map data in the direction of the input channels in turn, and outputs the data at the same position of the k output channels after n CLKs; it runs the calculation in a pipeline manner between two adjacent PW layers of the lightweight CNN classifier to further speed up the reasoning speed of the network. The inter-layer pipeline structure is as follows:
[0026] The weight of PW1 is in the form of a two-dimensional array of (m,n), where m is the number of output channels and n is the number of input channels. The parallelism of the PW layer is set to k. Then, after the PW1 layer reads the k data of the k pictures output by the previous layer in one cycle, it also reads the k numbers on the k groups of n input channels, that is, k×k weight data, and enters the PW1 layer for calculation. In the next cycle, after the PW1 layer reads the k data of the next group of k pictures, the weight reads the next group of k×k weight data in the direction of the output channel, and so on. The output picture size of the PW1 layer is (y,y,m). When the first digit of the m pictures is When all the data are written into the cache, the PW1 layer sends a start signal to PW2, and PW2 starts to read in data for calculation. Since the number of output channels of PW2 is greater than that of PW1, PW2 waits for the first data of the m pictures written into the cache by the PW1 layer before starting, so that there is no conflict between the writing of PW1 and the reading of PW2, and the two layers run in parallel; the next layer of PW2 is the DW layer, and the convolution kernel is 3×3. The cache data must be read in a sliding window in each cycle, so the weight reading of PW2 reads each group of k×k weight data in the direction of the input channel; thus, parallel pipelines within and between layers are realized.
[0027] Compared with the prior art, the present invention has the following beneficial effects: providing an optimization and speed-up method for a lightweight CNN classifier based on FPGA, which aims at the problem of FPGA resource shortage, compresses the parameters of the lightweight CNN network into INT8 type, and optimizes the use of on-chip resources; and arranges the weight parameters in the on-chip storage, which increases the bandwidth of the network and reduces the additional power consumption caused by accessing the off-chip memory; at the same time, adopts quantization-aware training (QAT) to conduct secondary training on the network, overcoming the accuracy loss of the classifier caused by quantization; in addition, a collaborative pipeline method is adopted within the layer of the lightweight CNN network and between adjacent PW layers, which improves the real-time performance of the classifier, and is applicable to all lightweight CNN networks with a deep separable structure. The classifier optimized and accelerated by this method has the advantages of real-time performance, low power consumption, and compact size, and is applicable to scenes such as industrial assembly line defective product classification, automatic driving, and national defense security. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 It is an implementation flow chart of an embodiment of the present invention.
[0029] Figure 2 4 is a system framework diagram of a lightweight CNN classifier in an embodiment of the present invention.
[0030] Figure 3 It is a schematic diagram of the hardware implementation principle of quantitative calculation in an embodiment of the present invention.
[0031] Figure 4 4 is a diagram of the parallel structure of the DW layer in an embodiment of the present invention.
[0032] Figure 5 4 is a diagram of a parallel structure of a PW layer in an embodiment of the present invention.
[0033] Figure 6 2 is a diagram of the convolutional layer pipeline structure in an embodiment of the present invention. DETAILED DESCRIPTION
[0034] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0035] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.
[0036] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0037] like Figure 1 As shown, this embodiment provides an optimization and speed-up method for a lightweight CNN classifier based on FPGA, which is characterized by comprising the following steps:
[0038] 1) Use Python to build a lightweight CNN network on the PC, train it with a dataset for image classification, and obtain the network model with the best classification effect after multiple trainings; at the same time, design the model architecture exported from the PC on the FPGA hardware side.
[0039] 2) Perform quantization-aware training on the trained network model, save the 32-bit floating-point model parameters as 8-bit fixed-point integers (INT8), and implement the complete architecture of the lightweight CNN network on the FPGA hardware side.
[0040] 3) The weights after quantization perception training are written into the coe file format according to the designed parallelism and the read and write order required by the hardware, and imported into the on-chip storage of the FPGA. The lightweight CNN classifier can then read the input image data to realize classification reasoning.
[0041] In this embodiment, the lightweight CNN network adopts the current mainstream lightweight network MobilenetV2. The overall architecture of the lightweight CNN classifier system is mainly composed of a host computer, a hardware acceleration module, an on-chip memory and a display module. Among them, the host computer corresponds to the above-mentioned PC end, the hardware acceleration module and the on-chip memory constitute the FPGA hardware end, and the display module is a display screen mounted on a separate board. After quantization, the weights of the network are arranged in order according to the parallelism of the DW and PW layers and the data reading rules, and are written into the on-chip memory (On-chip Memory) in advance. The host computer is only responsible for transmitting the input image data to the input buffer (Input-buffer) through the PCIE bus. After monitoring that the image data has been loaded, the hardware acceleration module reads the weights from the on-chip memory, and starts forward reasoning acceleration. The hardware acceleration module is deployed corresponding to the MobileNetV2 network structure, and is mainly composed of Conv2d layer, DW layer, PW layer, and pooling layer (Pooling Layer). The final classification result is displayed on the display module. The lightweight CNN classifier system framework is as follows Figure 2 shown.
[0042] The network model that has been trained and saved on the PC side has a weight of 32 floating-point data. For lightweight CNN, such as the MobileNetV2 network, the parameter volume is close to 3.2M. If the storage space required for caching inter-layer images is considered, it will be even larger, and the on-chip BRAM of the FPGA is very limited, which requires the parameters of the lightweight CNN network to be compressed. Although the static quantization method can effectively compress the parameter model, it will cause serious accuracy loss for the lightweight CNN network. The present invention conducts experimental tests on the hot-rolled strip surface defect data set released by Northeastern University under the PyTorch framework. The MobileNetV2 network is used. After the model parameters are compressed to INT8 type by static quantization, the defect classification accuracy drops to 70.1%. The quantization-aware training method is used to compress the model parameters to INT8 type and then retrain. After 8 rounds of EPOCH, the highest accuracy can reach 93.89%, as shown in Table 1.
[0043] Table 1. Changes in the accuracy of quantization-aware training
[0044] EPOCH First round Round 2 Round 3 Round 4 Fifth round Round 6 Round 7 Round 8 Accuracy 72.78％ 85.00％ 88.33％ 90.56％ 91.11％ 92.22％ 93.89％ 92.78％
[0045] Compared with floating-point operations, FPGA is better at processing fixed-point operations. After the parameters are quantized, a new fixed-point calculation formula is needed to replace the original floating-point convolution calculation formula to be implemented in hardware. Assuming that the convolution weight is w, the bias is b, the input is x, and the output activation value is a, since convolution is essentially a matrix operation, it can be expressed as shown below:
[0046]
[0047] Let r represent a floating point real number, q represent a quantized fixed point integer, S represent the quantized scale factor, and Z represent the integer value corresponding to 0 after quantization. The conversion formula between floating point numbers and fixed point numbers is as follows:
[0048] r=S(qZ) (2)
[0049]
[0050] Represent w, b, and x in formula (1) with quantized fixed-point numbers, and we get formula (4):
[0051]
[0052] The entire formula (4) only has is a floating point number, where S w is the quantization scale of the weight, S x is the quantization scale of the input, S a is the quantization scale of the output. Assume As long as we find the right n and M0 so that M≈2 -n M0, formula (4) can be completely converted into fixed-point operations.
[0053] The hardware implementation of quantitative calculation is as follows Figure 3 As shown, the implementation steps are as follows:
[0054] A) For each convolution operation, the result of the multiplication and accumulation of the image and weight is stored in a 32-bit register, the multiplier is also stored in a 32-bit register, and the multiplication result of the two is stored in a 64-bit register.
[0055] B) Since the PC side has the final rounding, the hardware side also performs a Round operation on the multiplication and accumulation result of the image and the weight, and the Round operation is followed by a shift operation. The Round and shift operations are combined into one beat.
[0056] C) The shifted result is truncated into 8-bit output after relu, and the relu and truncation are combined in one shot.
[0057] In this embodiment, a collaborative parallel pipeline method is adopted within the layer of the lightweight CNN network and between adjacent PW layers to accelerate the forward propagation process and improve the real-time performance of the classifier. The lightweight CNN network is mainly composed of a PW layer and a DW layer. In order to improve the computational parallelism of the DW layer, a strategy combining single convolution operation expansion and input channel expansion is adopted. The specific expansion method is as follows: Figure 4 As shown:
[0058] The convolution window N of the DW layer is 3, that is, a 3×3 convolution window. The 3×3 convolution calculation is flattened, and the 3×3 data at the input image position is multiplied by the corresponding weight, that is, 9 multiplication calculations are completed in one clock cycle; if the k-channel parallel method is adopted on the input channel, the entire DW layer completes 9×k multiplication calculations in one FPGA clock cycle, and outputs the data at the same position of k channels after subsequent operations.
[0059] Since one address of the memory stores the data of k channels in one cycle, the parallelism of the DW input channel needs to be consistent with the parallelism of the input channel of the next PW layer. Therefore, the PW layer adopts a parallel mode combining k input channels with k output channels; the data at the same position of the k input channels is read at the same time in one CLK and the weights on the k groups of channels are calculated, the input Map data is read in the direction of the input channel in turn, and the data at the same position of the k output channels is output after n CLKs. The parallel design of the PW layer is as follows: Figure 5 shown.
[0060] In this embodiment, the calculation is run in a pipeline manner between two adjacent PW layers of the lightweight CNN classifier to further speed up the network reasoning speed. Figure 6 shown.
[0061] The weight of PW1 is in the form of a two-dimensional array of (m,n), where m is the number of output channels and n is the number of input channels. If the parallelism of the PW layer is set to k, after the PW1 layer reads the k data of the k pictures output by the previous layer in one cycle, it also reads the k numbers on the k groups of n input channels, that is, k×k weight data, and enters the PW1 layer for calculation. In the next cycle, after the PW1 layer reads the k data of the next group of k pictures, the weight reads the next group of k×k weight data in the direction of the output channel, and so on. The output image size of the PW1 layer is (y,y,m). When the first data of the m pictures are all written into the cache, the PW1 layer sends a start signal to PW2, and PW2 starts to read in data for calculation. Since the number of output channels of PW2 is greater than that of PW1, PW2 only needs to wait for the first data of the m pictures written into the cache by the PW1 layer to start. There will be no conflict between the writing of PW1 and the reading of PW2, and the two layers can run in parallel. Since the next layer of PW2 is the DW layer, the convolution kernel is 3×3, and the cache data needs to be read in a sliding window in each cycle, the weight reading of PW2 reads each group of k×k weight data in the direction of the input channel. So far, parallel pipelines within each layer and between layers are realized.
[0062] The present invention provides an optimization and speed-up method for a lightweight CNN classifier based on FPGA. The method adopts the QAT quantization method to quantize 32-bit floating-point numbers into 8-bit fixed-point integers, thereby reducing the number of network parameters. The weight parameters and inter-layer cache data are stored in the on-chip resources of the FPGA. Parallel pipelining is adopted within the layers of the lightweight CNN network and between PW layers to accelerate the forward propagation process, and the final classification result is displayed on a display screen.
[0063] The present invention uses the QAT quantization method on the PC side for the model trained for the specified data set, and performs secondary training on the network during the quantization process, so that the network parameters can adapt to the errors caused by quantization, thereby reducing the final accuracy loss caused by quantization. The quantized weights and the image data to be detected are written into the on-chip BRAM. The weight data is merged according to the parallelism, and each address in the BRAM can store the weight of the corresponding number of parallelism. While improving the data reading speed, it can also save on-chip storage resources.
[0064] A lightweight CNN hardware network structure is built on FPGA. The network structure is mainly composed of Conv2d, PW, DW, quantization calculation and other modules. Each module occupies hardware resources. The input image on FPGA and the PC side must be consistent in inference mode and output data. After layer-by-layer inference of the network, the final classification result is output.
[0065] In order to speed up the reasoning speed of the classifier, the present invention proposes a pipeline method for intra-layer and inter-layer coordination. In the inter-layer cache area, once the data enters the layer, the convolution operation begins. Taking resources and speed into consideration, the convolution cycle is performed in the input channel, output channel, and convolution kernel, and the convolution is reasonably unfolded. The output result is directly output to the quantization module after the next beat. Between adjacent PW layers, the next layer does not need to wait for the previous layer to be completed. As long as the computing unit of the next layer obtains enough data, the calculation can be carried out. The pipelines within and between layers are arranged in sequence and cooperate with each other.
[0066] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.
Claims
1. An optimization and speed-up method for a lightweight CNN classifier based on FPGA, characterized in that: The following steps are involved: 1) On the PC side, use the built lightweight CNN network to train with the data set for image classification to obtain the network model with the best classification effect after multiple trainings; at the same time, design the model architecture exported from the PC side on the FPGA hardware side; 2) Perform quantization-aware training on the trained network model, save the 32-bit floating-point model parameters as 8-bit fixed-point integers, and implement the complete architecture of the lightweight CNN network on the FPGA hardware side; 3) The weights after quantization perception training are written into the coe file format according to the designed parallelism and the read and write order required by the hardware, and imported into the on-chip storage of the FPGA. The lightweight CNN classifier can then read the input image data to realize classification reasoning; The overall architecture of the lightweight CNN classifier system is mainly composed of a host computer, a hardware acceleration module, an on-chip memory, and a display module. The network weights are quantized and arranged in order according to the parallelism of the DW and PW layers and the data reading rules, and written into the on-chip memory in advance; The host computer is only responsible for transmitting the input image data to the input buffer through the PCIE bus; After the hardware acceleration module detects that the image data has been loaded, it reads the weights from the on-chip memory and starts forward reasoning acceleration; The lightweight CNN network adopts the lightweight network MobilenetV2, and the hardware acceleration module is deployed corresponding to the MobileNetV2 network structure, mainly consisting of a Conv2d layer, a DW layer, a PW layer, and a pooling layer; the final classification result is displayed on the display module; After parameter quantization, a new fixed-point calculation formula is needed to replace the original floating-point convolution operation formula to be implemented on hardware; assuming that the convolution weight is w, the bias is b, the input is x, and the output activation value is a, since convolution is essentially a matrix operation, it is expressed as shown in the following formula: Let r represent a floating-point real number, q represent a quantized fixed-point integer, S represent the quantized scale factor, and Z represent the integer value corresponding to 0 after quantization. The conversion formula between floating-point numbers and fixed-point numbers is as follows: r=S(qZ) (2) Represent w, b, and x in formula (1) with quantized fixed-point numbers, and we get formula (4): The entire formula (4) only has is a floating point number, where S w is the quantization scale of the weight, S x is the quantization scale of the input, S a is the quantization scale of the output; assuming As long as we find the right n and M0 so that M≈2 -n M0, formula (4) can be completely converted into fixed-point operations; The hardware implementation steps of quantitative calculation are as follows: A) For each convolution operation, the result of the multiplication and accumulation of the image and weight is stored in a 32-bit register, the multiplier is also stored in a 32-bit register, and the multiplication result of the two is stored in a 64-bit register; B) Perform a round operation on the hardware side for the multiplication and accumulation result of the image and the weight, followed by a shift operation, with the round and shift operations working together as one beat; C) After the shifted result passes through relu, it is truncated into 8-bit output, and relu and truncation are combined in one beat; A collaborative parallel pipeline method is adopted within the layers of the lightweight CNN network and between adjacent PW layers to accelerate the forward propagation process and improve the real-time performance of the classifier; the lightweight CNN network is mainly composed of PW layers and DW layers. In order to improve the computational parallelism of the DW layer, a strategy combining single convolution operation expansion with input channel expansion is adopted, specifically: The convolution window N of the DW layer is 3, that is, a 3×3 convolution window. The 3×3 convolution calculation is flattened, and the 3×3 data at the input image position is multiplied by the corresponding weight, that is, 9 multiplication calculations are completed in one clock cycle; the k-channel parallel mode is adopted on the input channel, and the entire DW layer completes 9×k multiplication calculations in one FPGA clock cycle, and outputs the data at the same position of k channels after subsequent operations; The PW layer adopts a parallel mode combining k input channels and k output channels; it reads the data at the same position of the k input channels and the weights of the k groups of channels at the same time in one CLK to calculate, reads the input Map data in the direction of the input channels in turn, and outputs the data at the same position of the k output channels after n CLKs; it runs the calculation in a pipeline manner between two adjacent PW layers of the lightweight CNN classifier to further speed up the reasoning speed of the network. The inter-layer pipeline structure is as follows: The weight of PW1 is in the form of a two-dimensional array of (m,n), where m is the number of output channels and n is the number of input channels. The parallelism of the PW layer is set to k. Then, after the PW1 layer reads the k data of the k pictures output by the previous layer in one cycle, it also reads the k numbers on the k groups of n input channels, that is, k×k weight data, and enters the PW1 layer for calculation. In the next cycle, after the PW1 layer reads the k data of the next group of k pictures, the weight reads the next group of k×k weight data in the direction of the output channel, and so on. The output picture size of the PW1 layer is (y,y,m). When the first digit of the m pictures is When all the data are written into the cache, the PW1 layer sends a start signal to PW2, and PW2 starts to read in data for calculation. Since the number of output channels of PW2 is greater than that of PW1, PW2 waits for the first data of the m pictures written into the cache by the PW1 layer before starting, so that there is no conflict between the writing of PW1 and the reading of PW2, and the two layers run in parallel; the next layer of PW2 is the DW layer, and the convolution kernel is 3×3. The cache data must be read in a sliding window in each cycle, so the weight reading of PW2 reads each group of k×k weight data in the direction of the input channel; thus, parallel pipelines within and between layers are realized.