Heterogeneous hardware acceleration method for plant disease and insect pest image recognition based on FPGA and ARM

Abstract

The invention discloses a heterogeneous hardware acceleration method for plant disease and insect pest image recognition based on an FPGA and an ARM, and the method comprises the steps: employing a CSPdarknet53tiny as a trunk feature extraction network of Yolov4tiny, employing a layer fusion principle during the export of operator parameters, carrying out the optimization of a data flow and an interface of a designed hardware accelerator, employing a dual-buffer structure, and enabling the hardware acceleration to be more stable and reliable. An AXI bus Central DMA is designed at an interface, and finally, prediction frame decoding after network post-processing and a non-maximum suppression algorithm are implemented at an embedded PS end, so that an identification result can be obtained in real time when an accelerator network obtains a result. According to the method, an efficient and accurate hardware accelerator is designed in an embedded application scene, particularly in the aspect of plant disease and insect pest image recognition, the network recognition accuracy is improved, meanwhile, the parameter quantity of a network and the transmission efficiency of hardware are considered, greater possibility is provided for deployment of a recognition system at a mobile terminal, and the application prospect is wide. Meanwhile, the labor cost is reduced, and a powerful technical basis is provided for subsequent plant disease and pest control.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a heterogeneous hardware acceleration method for plant pest and disease image recognition based on FPGA and ARM. Background Art

[0002] At present, the continuous optimization and maturity of computer hardware (such as GPUs) and software technologies (programming software and various integrated libraries) provide a solid platform foundation for the implementation of deep learning technologies. However, many models used for leaf recognition consume a large amount of computing resources and can only be deployed on the server side, resulting in poor popularity and real-time performance. Therefore, there are certain limitations for leaf recognition. Compared with classical machine learning algorithms, neural network models with asynchronous architectures have more advantages in real-time acquisition, recognition, and analysis of images, and are better in quickly recognizing plant leaves and mobile deployment with better performance and application prospects.

[0003] In the actual hardware application field, adopting the heterogeneous method of FPGA and ARM to implement the deployment of neural networks can effectively improve the efficiency of FPGA accelerators. The ARM side is responsible for controlling the function implementation of the FPGA and data transfer. Usually, ARM, FPGA, and external memories are connected through a high-speed Advanced Extensible Interface (AXI), which can significantly improve the memory access efficiency. Thus, the heterogeneous computing method has gradually become a highly sought-after hardware acceleration architecture for deploying neural network algorithms on FPGAs. Although a large number of dedicated heterogeneous FPGA accelerators have emerged and achieved good results. Some scholars use Zynq FPGAs to accelerate neural networks, and the published acceleration scheme can achieve a convolutional layer operation effect of 187.8 Gop / s in the Zynq XC7Z045 development board with a power consumption of only 9.63 W.

[0004] However, the application in the field of plant leaf image recognition is more complex than other image recognition fields. The similarity between different plant leaves is high, and it is difficult to distinguish their shape and other features. In addition, the lesion locations of plant leaf diseases are random, and the shooting of plant leaf images is easily affected by objective factors such as light, background, and angle. Even the manifestation forms of the same disease are inconsistent and vary greatly with external factors such as region and climate. Therefore, it is particularly important to study a more accurate, fast, and intelligent plant leaf image recognition system, which has certain theoretical significance and practical engineering application value. Summary of the Invention

[0005] To solve the above technical problems, the present invention proposes a heterogeneous hardware acceleration method for plant pest and disease image recognition based on FPGA and ARM.

[0006] The technical solution adopted by the present invention is as follows: A heterogeneous hardware acceleration method for plant pest and disease image recognition based on FPGA and ARM, and the specific steps are as follows:

[0007] S1. Use the lightweight Yolov4_tiny convolutional neural network as the hardware implementation network of the accelerator, and obtain the Conv2D and BN operator parameters required by the accelerator through weight export and layer fusion principle;

[0008] The Yolov4_tiny convolutional neural network selects CSPdarknet53_tiny as the backbone feature extraction network, and then performs lightweight processing. First, make a dataset of the plant leaf pest and disease image recognition network and input it into the model for training. Export the weights of the trained model results to obtain the trained parameters of the Conv2D and BN operators, and then fuse the Conv2D and BN operator parameters through the layer fusion principle.

[0009] Among them, a total of 7 types of operators are obtained through lightweight processing, namely Conv2D, LeakyReLU, Contact, Maxpool2D, and Upsample with 3 configurations.

[0010] S2. Build the architecture of the accelerator through HLS, and adopt the overlap architecture;

[0011] The overlap architecture adopted by the accelerator is the overlay-based co-designs, which has one or more accelerator cores. The tensor core in the architecture of the accelerator divides the Yolov4_tiny convolutional neural network in step S1 into a series of accelerator call commands, determines the acceleration layer sorting, and executes the network through the generated control program running on the CPU.

[0012] Among them, the architecture of the accelerator executes the calculation part of the convolution process through a reusable processing engine PE, and the hardware control and operation scheduling are executed on the upper-layer software. Then the architecture can be extended according to the input model and available FPGA resources, that is, an independent PE is set for each operator of the network, and the operators are reused across layers.

[0013] S3. Based on the accelerator built in step S2, design and implement the IP of each operator of the hardware accelerator and the AXI bus interface IP of the plant pest and disease leaf;

[0014] S31. Design and implement the IP of each operator of the hardware accelerator;

[0015] Perform operator IP design on the 7 types of operators obtained after lightweight processing in step S1. First, implement the fixed size of the architecture in step S2 on the PL side, that is, fix the size of the output. First, determine the output block ofm, and then reverse the size and coordinates of the corresponding block ifm based on this block ofm for padding calculation.

[0016] S32. Based on step S31, design and implement the AXI bus interface IP for plant disease and pest leaves;

[0017] Based on the IP of each operator designed in step S31, design an AXI-based Central DMA IP suitable for data transmission of plant disease and pest leaves. The module uses an AXI interface to complete the transfer of data in different address spaces, and can copy up to 255 2D rectangular data frames at most. And match the transmission of plant leaf disease and pest image data through the characteristics of non-aligned data transmission on the AXI bus, that is, by supporting 32-bit non-aligned data transmission, and discard non-critical data in the plant leaf disease image by shifting to align the data within the DMA. Combine with the STRB signal and then read and write the data.

[0018] Among them, the width, depth, and row spacing of the source and destination rectangular frames of the Central DMA IP can be configured according to bytes. The configuration register uses an APB interface, and the data transmission uses an AXI interface.

[0019] S4. Implement the double-buffer mechanism for the 3 configurations of Conv2D obtained in step S1, and implement the prediction box decoding and non-maximum suppression operations on the PS side;

[0020] The double-buffer mechanism is as follows: Design two buffers to perform the data loading load and calculation compute operations simultaneously, complete the coverage of the calculation time by the loading time to improve the transmission efficiency; at the same time, implement the prediction box decoding and non-maximum suppression NMS operations for post-processing after the network output on the PS side.

[0021] Among them, the double-buffer mechanism is implemented by the BRAM on the PL side.

[0022] S5. Implement the top-level interface of the IP optimized in step S4 on the PL side, and schedule and control it by the PS-side ARM CPU. At the same time, synthesize and implement the Block design;

[0023] Integrate each IP designed in steps S3 and S4 into a Block design in Vivado, and connect each IP interface. The control is performed by the PS side, and the data operation is performed by the PL side. After completing the Block design, synthesize and implement the design, check the resource usage and the power consumption of the model, and obtain the acceleration effect of the accelerator.

[0024] S6. Load the Block design obtained through steps S3, S4, and S5 and the plant pest and disease leaves to be identified into the PYNQ-Z2 board, start the board to obtain the identification result, and complete the acceleration.

[0025] Further, in step S1, the weight export and layer fusion principles are specifically as follows:

[0026] The weights are obtained by reading the values of the given keys in the dictionary. For the layer fusion principle, through logical operations, the networks of the Conv2D layer and the BN layer are fused. Before fusion, the output expressions of the two layers are as follows:

[0027] y conv = wx + b

[0028]

[0029] where x represents the input of the current layer, y conv and y BN represent the outputs of the Conv2D layer and the BN layer respectively, b represents the bias term, w represents the weights, γ and β represent the learnable parameters of the network model, represents the batch mean, σ represents the batch variance, and ε represents a small constant used to prevent division by zero.

[0030] After fusion, the total output y expression of the current layer is as follows:

[0031]

[0032] And the output expression of the fused convolution Y is as follows:

[0033] Y = w'x + b'

[0034] where w' and b' represent the fused weights and bias, and the expressions are as follows:

[0035]

[0036] Finally, use the fused weights and bias to update the original weights w and bias b in the network to obtain a new Conv2D layer, which can replace the operations of the original Conv2D layer and the BN layer.

[0037] Further, in step S3, the specific implementation of the fixed size of the architecture in step S2 on the PL side is as follows:

[0038]

[0039] Among them, H out and W out represent the size of the output feature map, and H in and W in represent the size of the input feature map. When Padding is set to p, dilation is set to 1, kernel_size is set to k, and the stride is s, the size expression of the output feature map can be obtained as follows:

[0040]

[0041] That is:

[0042]

[0043] According to the output feature map size formula, the following expression can be deduced:

[0044] fm_size = H out × s - (2 × p + s - k)

[0045] Set the size of the input feature block block ifm to T rin , and set the size of the output feature block block ofm to T rout , then there is the following expression:

[0046] T rin = (T rout - 1) × s - 2 × p + k

[0047] Finally, the convolution with a fixed output size on the PL side is obtained.

[0048] Advantages of the present invention: By using CSPdarknet53_tiny as the backbone feature extraction network of Yolov4_tiny in the method of the present invention, and applying the principle of layer fusion when exporting operator parameters, the number of operator parameters is reduced, making the entire model have a more lightweight structure. Then, the data stream and interface of the designed hardware accelerator are optimized. The dual-buffer structure adopted greatly reduces the on-chip storage pressure of the FPGA chip for this pest and disease identification application scenario with an embedded nature. At the same time, the AXI bus Central DMA design at the interface can complete the rapid matching and transfer of plant pest and disease leaf image data blocks, improving the data reading efficiency and recognition accuracy. Finally, the prediction box decoding and non-maximum suppression algorithms for network post-processing are implemented on the embedded PS side, enabling the accelerator network to obtain the recognition result in real time when getting the result, which is more in line with the scenario of plant leaf pest and disease recognition on the mobile side. The method of the present invention solves the problem that the plant pest and disease recognition model is too large and not easy to be deployed on the mobile side, and designs an efficient and accurate hardware accelerator in the embedded application scenario. Especially in the aspect of plant pest and disease image recognition, while improving the network recognition accuracy, it takes into account the number of network parameters and the hardware transmission efficiency, providing greater possibilities for the plant leaf pest and disease recognition system to be deployed on the mobile side, and at the same time reducing labor costs, providing a strong technical foundation for the subsequent prevention and control of plant pests and diseases. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flowchart of a heterogeneous hardware acceleration method for plant pest and disease image recognition based on FPGA and ARM according to the present invention.

[0050] Figure 2 It is a schematic diagram of the architecture of the plant pest and disease leaf recognition accelerator in an embodiment of the present invention.

[0051] Figure 3 It is a schematic diagram of the module division of the AXI-based Central DMA design IP in an embodiment of the present invention.

[0052] Figure 4 It is a schematic diagram of the principle of the dual-buffer mechanism in an embodiment of the present invention.

[0053] Figure 5 It is a Block design diagram in FPGA in an embodiment of the present invention.

[0054] Figure 6 It is a schematic diagram of the resource consumption after the synthesis and implementation of the Block design in an embodiment of the present invention.

[0055] Figure 7 It is a schematic diagram of the power consumption analysis after the synthesis and implementation of the Block design in an embodiment of the present invention.

[0056] Figure 8 This is a schematic diagram of the output result of the hardware accelerator with grape disease leaves as the input in the embodiment of the present invention.

[0057] Figure 9 This is a schematic diagram of the output result of the hardware accelerator with tomato disease leaves as the input in the embodiment of the present invention.

[0058] Figure 10 This is a schematic diagram for comparing the output results of plant pest and disease leaves on the FPGA hardware accelerator and the server network under complex conditions in the embodiment of the present invention.

[0059] Figure 11 This is a schematic diagram of the time consumption of each layer of the hardware acceleration network and the result of post-processing after hardware implementation displayed in real time through the serial port in the embodiment of the present invention. Detailed implementation manners

[0060] The method of the present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0061] As Figure 1 shown, the flowchart of a heterogeneous hardware acceleration method for plant pest and disease image recognition based on FPGA and ARM of the present invention is as follows:

[0062] S1. Use the lightweight Yolov4_tiny convolutional neural network as the hardware implementation network of the accelerator, and obtain the Conv2D and BN operator parameters required by the accelerator through weight export and layer fusion principles;

[0063] In this embodiment, the Yolov4_tiny convolutional neural network selects CSPdarknet53_tiny as the backbone feature extraction network (to obtain a lightweight convolutional network neural model), and then performs lightweight processing. First, a dataset for the plant leaf pest and disease image recognition network is made and input into the model for training. The trained results of the model are exported to obtain the trained parameters of the Conv2D and BN operators. Then, through the layer fusion principle, the Conv2D and BN operator parameters are fused to reduce the number of parameters.

[0064] Among them, a total of 7 types of operators are obtained through lightweight processing, namely Conv2D, LeakyReLU, Contact, Maxpool2D, and Upsample with 3 configurations.

[0065] S2. Build the architecture of the accelerator through HLS, and adopt the overlap architecture (when the input model changes but the calculation operators do not change, there is no need to redesign the hardware);

[0066] The accelerator adopts an overlap architecture, namely an overlay-based co-designs, which has one or more accelerator cores. The advantage of adopting this architecture is that it is very flexible. When the input model changes but the computational operators remain unchanged, there is no need to redesign the hardware. The greatest benefit of this is that it has a certain degree of compatibility. As long as the operators in the network are supported by this accelerator, then this accelerator can be used for hardware acceleration. The accelerator architecture of this embodiment is as Figure 2 shown.

[0067] The tensor core in the accelerator architecture divides the Yolov4_tiny convolutional neural network in step S1 into a series of accelerator call commands, determines the acceleration layer sorting (that is, which layer to accelerate first and which layer to accelerate later), and executes the network through the generated control program running on the CPU.

[0068] Among them, the accelerator architecture executes the computational part of the convolution process through a reusable processing engine PE, and the hardware control and operation scheduling are executed by the upper-layer software (running on the CPU). Then the architecture can be extended according to the input model and available FPGA resources, that is, an independent PE is set for each operator of the network, and the operators are reused across layers.

[0069] S3. Based on the accelerator built in step S2, design and implement the IP of each operator of the hardware accelerator and the AXI bus interface IP of the plant pest and disease leaves;

[0070] S31. Design and implement the IP of each operator of the hardware accelerator;

[0071] When designing the operator IP, if following the design idea of a general-purpose accelerator and adopting an operator-based acceleration scheme, the PL needs to implement a fixed-size PE array to implement a specific operator, that is, the operator implemented by the PL is to perform a convolution of a fixed-size ifm and kernel, that is, read the ifm for each convolution window, perform convolutions with 32 kernels respectively, and cache the calculation results.

[0072] In this embodiment, for the 7 types of operators obtained after lightweight processing in step S1, operator IP design is carried out. First, the PL side implements the fixed size in the architecture in step S2, that is, fixes the size of the output. First, determine the output block ofm, and then reverse-deduce the size and coordinates of the corresponding block ifm based on this block ofm for padding calculation.

[0073] Among them, it is more convenient to implement Concat on the PS side through pointer operations. In fact, only four operators, namely Conv2D, LeakyReLU, Maxpool2D, and Upsample, need to be implemented on the PL side. According to different convolution kernel sizes and strides, Conv2D needs to implement three configurations: K3 P1 S2||K3 P1 S1||K1 P0 S1. The first two are both K3 P1, only the strides are different. Therefore, in this embodiment, an operator that supports strides S = 1 or 2 is used to implement them. K1 P0 S1 is Pointwise Convolution, which is a special convolution. In this embodiment, it is implemented according to the implementation method of the convolution operator and combined with the characteristics of PWConv. The implementation of the Maxpool2D and Upsample operators is relatively simple. The configuration of Maxpool2D is K2 S2 P0, and Upsample is implemented using the Nearest neighbor interpolation algorithm with a scaling factor of 2.0.

[0074] S32. Based on step S31, design and implement the AXI bus interface IP for plant pest and disease leaves;

[0075] Based on the IP of each operator designed in step S31, design an AXI-based Central DMA IP suitable for the data transmission of plant pest and disease leaves. For the design of Central DMA, the main function to be realized is the fast matching and transfer of plant pest and disease leaf image data, which is used to replace the memcpy function on the PS side to speed up the transmission speed of the entire module. The module uses an AXI interface to complete the transfer of data in different address spaces, and can implement the copying of up to 255 2D rectangular data frames at most. It matches the transmission of plant leaf pest and disease image data through the characteristic of non-aligned AXI bus data transmission, that is, configures the first rectangular data frame through apb_inf, stores the configuration information of subsequent rectangular frames in the Command Linked List, reads and writes the rectangular frame data and transmits the subsequent data rectangular frame information through AXI_inf. At the same time, it supports 32-bit non-aligned data transmission. The non-critical data in the plant leaf disease image is discarded by shifting to align the data within the DMA. Combined with the STRB signal, the data is then read (split) and written (stitched) to achieve the purpose that the transmitted data has obvious characteristics, that is, obvious disease characteristics. According to the characteristics of the AXI bus, this embodiment supports the functions of read_cmd outstanding and UserARID to improve the bus read data efficiency. The structure diagram of the Central DMA module is as Figure 3 shown.

[0076] Among them, the widths, depths, and row spacings of the source and destination rectangular frames of the IP of the Central DMA can be configured according to bytes. The configuration register uses the APB interface, and data transmission uses the AXI interface.

[0077] S4. Implement the double-buffering mechanism for the three configured Conv2Ds obtained in step S3, and implement the prediction box decoding and non-maximum suppression (NMS) operations on the PS side;

[0078] The described double-buffering mechanism is as follows: Design two buffers to enable the simultaneous execution of data loading and computing operations, covering the computing time with the loading time to improve the transmission efficiency (design throughput); at the same time, implement the prediction box decoding and non-maximum suppression NMS operations for post-processing of the network output on the PS side.

[0079] Among them, the double-buffering mechanism is implemented by the BRAM on the PL side.

[0080] In the case of only one buffer, loading and computing cannot be performed simultaneously. It is necessary to wait until the input feature map (ifm) is loaded into the buffer on the PL before subsequent calculations can be performed on the ifm stored in the buffer. However, if there are two input buffers, starting from Time2, ifm0 can be loaded into buffer0 while computing the ifm1 that has been loaded into the other buffer1. Usually, the time required for computing is longer than the loading time, so the loading time is covered. If there are more slices (blocks), the more loading time saved in this way, the greater the improvement in throughput. At the same time, the double-loop buffering mechanism in this embodiment is nested inside and outside. The outer double buffering is from off-chip to on-chip buffer, and the inner double buffering is from on-chip buffer to internal register. The schematic diagram of the ping-pong buffer is as Figure 4 shown.

[0081] In this embodiment, fm_out1 and fm_out2 form an output double buffer. When actually implementing the double buffer strategy, a flag signal of the double buffer strategy will be used for buffer switching. The data is transferred to the buffer in the on-chip buffer through the AXI interface, and then the convolution calculation function is called to start the calculation. The calculation result is saved in fm_out1. Then, the block ofm coordinates are updated, and the calculation is performed again. This time, the result is saved in fm_out2, and at the same time, the function for storing the execution result is executed to transfer the result in fm_out1 of the PL back to the DDR on the PS side. Then, similarly, the result in fm_out2 is stored in the DDR of the PS, and the calculation result is saved in fm_out1. Finally, the block ofm coordinates are updated.

[0082] The non-maximum suppression (NMS) operation in this embodiment is the process of selecting the best result among the 3 prediction boxes of each grid at two scales (13*13 and 26*26). First, the coordinates of the upper left and lower right points of the bounding box are calculated through the coordinates of the center point. The x coordinate and y coordinate of the upper left point of each bounding box are calculated respectively. The x (y) coordinate of the center point minus half of the width w (height h) of the prediction box is used to obtain the x coordinate of the upper left point. The coordinates of the lower right point are obtained by adding half of the width w (height h) of the prediction box to the x (y) coordinate of the center point. First, all the prediction results are traversed. For each prediction result, first, the maximum value is taken from the predicted class confidence and the corresponding class is recorded. Secondly, the first-round screening of the prediction results is performed using the confidence threshold to remove the results with lower confidence. Then, the screened results are connected with the class confidence and class index (label) to form new detection results. And all the classes included in the prediction results are obtained and processed. Finally, non-maximum suppression (NMS) processing is performed on each class to remove the boxes with high overlap and retain the boxes with higher confidence as the final output. The intersection over union is the basis for object detection NMS and is used to calculate the overlap degree between the true bounding box Bgt (the annotation of the dataset) and the predicted bounding box Bp (the model prediction result). The expression is as follows:

[0083]

[0084] S5. Implement the top-level interface of the IP optimized in step S4 on the PL side, and it is scheduled and controlled by the PS-side ARM CPU. At the same time, perform synthesis and implementation on the Block design;

[0085] Integrate each IP designed in steps S3 and S4 into a Block design in Vivado, and connect each IP interface. The PS side is used for operation control, and the PL side is used for data calculation. After completing the Block design, perform synthesis and implementation on the design, view the resource usage and the power consumption of the model, and obtain the acceleration effect of the accelerator.

[0086] In this embodiment, the design of the Block Design is implemented in Vivado 2019.2. Import the IP designed in HLS into Vivado to create a Block Design. After completing the configuration with the ZYNQ7 on the PS side, the design as shown in Figure 5 is obtained. After synthesis and implementation, when the timing requirements are met, view the resource utilization rate and the power consumption of the model, as shown in Figure 6 Figure 7 shown.

[0087] S6. Load the Block design obtained through steps S3, S4, and S5 and the plant disease and pest leaves to be identified into the PYNQ-Z2 board, start the board to obtain the identification result (the identification result is stored in the SD card or directly connected to the host computer to view the result), and complete the acceleration.

[0088] In this embodiment, in step S6, the trained model weights and biases are loaded into the board together with the input plant disease and pest images. The input images are grape disease leaf images with occlusion and tomato leaf disease images under complex backgrounds collected, and the corresponding output images are obtained. As shown in Figure 8 、 Figure 9 shown. At the same time, test with the network model on the server side under the same parameters and weights, and compare the two results as shown in Figure 10 shown.

[0089] Among them, Figure 10 (a) is the output result on the FPGA hardware accelerator, Figure 10 (b) is the output result on the server network.

[0090] At the same time, connect to the host computer and view the time spent on hardware inference for each layer of the network in real time through the serial port. The experimental results obtained are as shown in Figure 11 shown.

[0091] In this embodiment, in step S1, the weight export and layer fusion principle are specifically as follows:

[0092] The weights are obtained by reading the values of the given keys in the dictionary. After finding the prefix rule, it is convenient to export the weights of all Conv2D and BN layers. For the layer fusion principle, the networks of the Conv2D layer and the BN layer are fused through logical operations. Before fusion, the output expressions of the two layers are as follows:

[0093] y conv = wx + b

[0094]

[0095] Among them, x represents the input of the current layer, y conv and y BN represent the outputs of the Conv2D layer and the BN layer respectively, b represents the bias term, w represents the weight, γ and β represent the parameters that can be learned by the network model, and are used to scale and translate the normalized values. represents the mean of the batch, σ represents the variance of the batch, ε represents a small constant, which is 10 -5 in this embodiment, and is used to prevent division by zero.

[0096] After fusion, the total output y expression of the current layer is as follows:

[0097]

[0098] And the output expression of the fused convolution Y is as follows:

[0099] Y = w'x + b'

[0100] Among them, w' and b' represent the fused weights and biases, and the expressions are as follows:

[0101]

[0102] Finally, the weights w and biases b in the original network are updated with the fused weights and biases to obtain a new Conv2D layer, which can replace the operations of the original Conv2D layer and the BN layer. In the subsequent forward inference operation of the hardware, through such layer fusion operation, the effect of the two-layer network is achieved, and the output of the network is reduced by half.

[0103] In this embodiment, in step S3, the specific implementation of the fixed size of the architecture in step S2 on the PL side is as follows:

[0104]

[0105] Among them, H out and W out represent the output feature map size, H in and W inDenote the size of the input feature map. Set Padding to p, dilation to the default value of 1, kernel_size to k, and stride to s, then the expression for the size (shape) of the output feature map can be obtained as follows:

[0106]

[0107] That is:

[0108]

[0109] According to the formula for the output feature map size, the following expression can be derived by reverse deduction:

[0110] fm_size = H out × s - (2 × p + s - k)

[0111] Set the size of the input feature block block ifm to T rin , and set the size of the output feature block block ofm to T rout , then there is the following expression:

[0112] T rin =(T rout - 1)× s - 2 × p + k

[0113] Finally, the convolution with a fixed output size on the PL side can be obtained.

[0114] In summary, the method of the present invention adopts the idea of software and hardware co - design, and proposes a method for accelerating a small - scale neural network based on FPGA + ARM heterogeneous architecture. By performing layer fusion and fixing the operator size on the original lightweight convolutional neural network of yolov4_tiny, while ensuring the original accuracy, the number of parameters of the original network model is reduced, and the on - chip storage problem of the chip in the image recognition of plant diseases and pests, which is a kind of embedded application scenario, is alleviated. At the same time, in terms of data transmission, the method of the present invention uses an internally and externally embedded double - buffer mechanism to overlap the time of image data loading and calculation, greatly improving the data transmission efficiency. In terms of interface design, based on the characteristics of the AXI bus and combined with the characteristics that the shapes of leaf diseases are difficult to distinguish, the present invention designs a Central DMA interface IP for transporting 2D rectangular frames matching plant diseases and pests leaf images to replace the memcpy function on the PS side, improving the data reading efficiency of the bus and the recognition accuracy.

[0115] Those of ordinary skill in the art will realize that the embodiments described herein are to assist the reader in understanding the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. Various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the scope of the claims of the present invention.

Claims

1. A heterogeneous hardware acceleration method for plant pest and disease image recognition based on FPGA and ARM, the specific steps are as follows: S1. Use the lightweight Yolov4_tiny convolutional neural network as the hardware implementation network of the accelerator, and obtain the Conv2D and BN operator parameters required by the accelerator through weight export and layer fusion principle; The Yolov4_tiny convolutional neural network uses CSPdarknet53_tiny as the backbone feature extraction network, and then performs lightweight processing. First, a data set of a plant leaf pest image recognition network is prepared and input into the model for training. The results of the model training are weighted and derived to obtain the trained parameters of the Conv2D and BN operators. Then, the Conv2D and BN operator parameters are fused through the layer fusion principle. in, A total of 7 types of operators are obtained through lightweight processing, namely Conv2D, LeakyReLU, Contact, Maxpool2D and Upsample with 3 configurations; S2, build the accelerator architecture through HLS, using the overlap architecture; The accelerator adopts an overlap architecture, i.e., an overlapping data flow architecture, overlay-based co-designs, and has one or more accelerator cores; the tensor core in the accelerator architecture divides the Yolov4_tiny convolutional neural network of step S1 into a series of accelerator call commands, determines the order of the acceleration layers, and executes the network through a generated control program running on the CPU; The accelerator architecture uses a reusable processing engine PE to perform the computational part of the convolution process, and the hardware control and operation scheduling are performed by the upper-layer software. The architecture can be expanded according to the input model and available FPGA resources, that is, an independent PE is set for each operator of the network, and the operators are reused across layers. S3. Based on the accelerator built in step S2, design and implement the operator IPs of the hardware accelerator and the AXI bus interface IP of the plant pest and disease leaves; S31. Design and implement each operator IP of the hardware accelerator; Operator IP design is performed for the seven types of operators obtained after lightweight processing in step S1. First, the PL side implements the fixed size of the architecture in step S2, that is, the output size is fixed. The output block ofm is first determined, and then the size and coordinates of the corresponding block ifm are reversed based on this block ofm to perform filling calculations; S32, based on step S31, designing and implementing the AXI bus interface IP of plant pests and diseases leaves; Based on the operator IPs designed in step S31, an AXI-based Central DMA IP suitable for plant pest and disease leaf data transmission is designed. The module adopts the AXI interface to complete the data transfer in different address spaces, and can copy up to 255 2D rectangular data frames. The transmission of plant leaf pest and disease image data is matched through the characteristics of AXI bus data non-aligned transmission, that is, by supporting 32-bit non-aligned data transmission, non-critical data in the plant leaf disease image is discarded by shift, so that the data is aligned in the DMA, and the data is read and written in combination with the STRB signal; The width, depth and row spacing of the source and destination rectangular frames of the Central DMA IP can be configured in bytes; the configuration register uses the APB interface, and the data transmission uses the AXI interface; S4, implementing a double buffering mechanism for the three configurations of Conv2D obtained in step S1, and implementing prediction frame decoding and non-maximum suppression operations on the PS side; The double buffer mechanism is as follows: design two buffers to allow data loading and computing operations to be performed simultaneously, so that computing time covers loading time to improve transmission efficiency; at the same time, prediction box decoding and non-maximum suppression (NMS) operations for post-processing of network output are implemented on the PS side; Among them, the double buffer mechanism is implemented by the BRAM on the PL side; S5. Implement the top-level interface of the IP optimized in step S4 on the PL side, and schedule and control it by the ARMCPU on the PS side, and synthesize and implement the Block design at the same time; In Vivado, integrate the IPs designed in steps S3 and S4 into a Block design and connect the interfaces of the IPs. The PS side controls the operation and the PL side performs data calculations. After the Block design is completed, synthesize and implement the design, check the resource usage and power consumption of the model, and obtain the acceleration effect of the accelerator. S6. Load the Block design obtained through steps S3, S4, and S5 and the plant pest leaves to be identified into the PYNQ-Z2 board, start the board to obtain the identification result, and complete the acceleration.

2. According to claim 1, a heterogeneous hardware acceleration method for plant disease and insect pest image recognition based on FPGA and ARM is characterized in that: In step S1, the weight derivation and layer fusion principle are as follows: The weight is obtained by reading the value of a given key in the dictionary. For the layer fusion principle, the networks of the Conv2D layer and the BN layer are fused through logical operations. Before fusion, the output expressions of the two layers are as follows: y conv =wx+b Among them, x represents the input of the current layer, y conv ,y BN They represent the output of the Conv2D layer and the BN layer respectively, b represents the bias term, w represents the weight, γ and β represent the learnable parameters of the network model, represents the mean of the batch, σ represents the variance of the batch, and ε represents a small constant used to prevent division by zero; After fusion, the total output y of the current layer is expressed as follows: The output expression of the fused convolution Y is as follows: Y=w′x+b′ Among them, w′ and b′ represent the fused weight and bias, and the expressions are as follows: Finally, the fused weights and biases are used to update the weights w and biases b in the original network, and the new Conv2D layer can replace the operations of the original Conv2D layer and BN layer.

3. The heterogeneous hardware acceleration method for plant pest and disease image recognition based on FPGA and ARM according to claim 1, characterized in that: In step S3, the PL side implements the fixed size of the architecture in step S2 as follows: Among them, H out With W out represents the output feature map size, H in and W in Represents the input feature map size; Padding is set to p, dilation is set to 1, kernel_size is set to k, and stride is s, then the size expression of the output feature map is as follows: Right now: According to the output feature map size formula, the expression can be reversed as follows: fm_size=H out ×s-(2×p+s-k) Set the size of the input feature block size block ifm to T rin , the size of the output feature block size block ofm is set to T rout , then the expression is as follows: T rin =(T rout -1)×s-2×p+k Finally, we get the convolution with fixed output size performed by the PL side.

Images