FPGA-based convolutional neural network image recognition system, method, device and medium

Abstract

The application discloses a kind of FPGA-based convolutional neural network image recognition system, method, equipment and medium, it is related to image recognition technical field, including: ARM processor, AHB bus, FPGA-based lightweight convolutional neural network acceleration module, including depth convolution and target point convolution, target point convolution realizes point convolution and standard convolution;ARM processor pre-processes image data, and the data after processing is transmitted to lightweight convolutional neural network acceleration module by AHB bus, then receives the operation result of the module, and carries out image recognition based on received operation result;Lightweight convolutional neural network acceleration module carries out depth convolution to the data after pre-processing by depth convolution module, and carries out point convolution and standard convolution to the convolution result by target point convolution module, obtains operation result;The application can improve image recognition processing speed using less resources under the condition of lower operating power consumption.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to an FPGA-based convolutional neural network image recognition system, method, device, and medium. Background Technology

[0002] In recent years, with the vigorous promotion of information technology construction by countries around the world, artificial intelligence technology has also made rapid progress. As an important component of artificial intelligence networks, convolutional neural networks (CNNs) fully combine strategies such as local connectivity and weight sharing, along with excellent fully connected feedforward networks, giving them superb image processing capabilities. CNNs have achieved great success in fields such as facial recognition, object detection, action prediction, and image classification.

[0003] As the complexity of artificial intelligence applications increases, the unique structure of convolutional neural network models necessitates that their hardware platforms possess extremely high performance. Traditional hardware platforms are increasingly unable to meet the core requirements of contemporary convolutional neural networks in terms of power consumption and real-time performance.

[0004] Existing technologies lack design capabilities for lightweight convolutional neural networks, and often rely on increasing hardware resources to improve image recognition and classification speeds. This results in problems such as long convolution computation time, high resource consumption, and low acceleration efficiency.

[0005] Therefore, how to improve image recognition processing speed with low power consumption and limited resources is an urgent problem to be solved in this field. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide an FPGA-based convolutional neural network image recognition system, method, device, and medium that can improve image recognition processing speed with low power consumption and limited resources. The specific solution is as follows:

[0007] In a first aspect, this application discloses an FPGA-based convolutional neural network image recognition system, comprising: an ARM processor, an advanced high-performance bus, and an FPGA-based lightweight convolutional neural network acceleration module; the lightweight convolutional neural network acceleration module includes a depth convolution module and a target point convolution module, wherein the target point convolution module is a convolution module capable of performing point convolution operations and standard convolution operations;

[0008] The ARM processor is used to preprocess the acquired image data and transmit the preprocessed image data to the lightweight convolutional neural network acceleration module through the advanced high-performance bus; and is also used to receive the calculation results of the lightweight convolutional neural network acceleration module through the advanced high-performance bus, and then perform image recognition based on the received calculation results.

[0009] The lightweight convolutional neural network acceleration module is used to perform depth convolution operations on the preprocessed image data through the depth convolution module to obtain a first operation result, and to perform point convolution and standard convolution operations on the first operation result through the target point convolution module to obtain a second operation result; and to transmit the second operation result to the ARM processor through the advanced high-performance bus so that the ARM processor can perform image recognition based on the second operation result.

[0010] Optionally, the lightweight convolutional neural network acceleration module is a convolutional neural network acceleration module constructed using an in-memory computing approach that integrates convolution operations and read / write caching.

[0011] Optionally, the depthwise convolution module includes:

[0012] A distributed parallel multiplication structure is used to perform multiplication operations on the preprocessed image data to obtain the results of the multiplication operations;

[0013] An addition tree is used to accumulate the results of the multiplication operation based on a number of adders to obtain the first operation result.

[0014] A streamlined register is used to temporarily store the accumulation result of each adder.

[0015] Optionally, the target point convolution module is a parallel convolution module that reorganizes the point convolution structure to achieve point convolution operations and standard convolution operations.

[0016] Optionally, the FPGA-based convolutional neural network image recognition system further includes:

[0017] The instruction register is used to receive instructions sent by the user terminal and control the lightweight convolutional neural network acceleration module to perform operations on the preprocessed image data according to the instructions.

[0018] A data register is used to temporarily store the results of various operations, including the first operation result and the second operation result.

[0019] Optionally, the FPGA-based convolutional neural network image recognition system further includes:

[0020] An external storage device is used to store the result of the second operation;

[0021] A display is used to show the result of the second calculation.

[0022] Optionally, the FPGA-based convolutional neural network image recognition system further includes:

[0023] The camera is used to acquire image data and send the acquired image data to the ARM processor.

[0024] Secondly, this application discloses an FPGA-based convolutional neural network image recognition method, applied to an FPGA-based convolutional neural network image recognition system. The FPGA-based convolutional neural network image recognition system includes: an ARM processor, an advanced high-performance bus, and an FPGA-based lightweight convolutional neural network acceleration module. The lightweight convolutional neural network acceleration module includes a depthwise convolution module and a target point convolution module, wherein the target point convolution module is a convolution module capable of performing point convolution operations and standard convolution operations. The method includes:

[0025] The ARM processor preprocesses the acquired image data and transmits the preprocessed image data to the lightweight convolutional neural network acceleration module via the advanced high-performance bus. The processor also receives the computation results from the lightweight convolutional neural network acceleration module via the advanced high-performance bus and performs image recognition based on the received computation results.

[0026] The preprocessed image data is subjected to depth convolution operation by the depth convolution module in the lightweight convolutional neural network acceleration module to obtain a first operation result. The first operation result is then subjected to point convolution operation and standard convolution operation by the target point convolution module in the lightweight convolutional neural network acceleration module to obtain a second operation result. The second operation result is then transmitted to the ARM processor through the advanced high-performance bus so that the ARM processor can perform image recognition based on the second operation result.

[0027] Thirdly, this application discloses an electronic device, including:

[0028] Memory, used to store computer programs;

[0029] A processor is used to execute the computer program to implement the aforementioned FPGA-based convolutional neural network image recognition method.

[0030] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned FPGA-based convolutional neural network image recognition method.

[0031] As can be seen, this application proposes an FPGA-based convolutional neural network image recognition system, comprising: an ARM processor, an advanced high-performance bus, and an FPGA-based lightweight convolutional neural network acceleration module; the lightweight convolutional neural network acceleration module includes a depthwise convolution module and a target point convolution module, wherein the target point convolution module is a convolution module capable of performing point convolution operations and standard convolution operations; the ARM processor is used to preprocess the acquired image data and transmit the preprocessed image data to the lightweight convolutional neural network acceleration module through the advanced high-performance bus; and is used to receive the operation results of the lightweight convolutional neural network acceleration module through the advanced high-performance bus, and then perform image recognition based on the received operation results; the lightweight convolutional neural network acceleration module is used to perform depthwise convolution operations on the preprocessed image data through the depthwise convolution module to obtain a first operation result, and perform point convolution operations and standard convolution operations on the first operation result through the target point convolution module to obtain a second operation result; and is used to transmit the second operation result to the ARM processor through the advanced high-performance bus, so that the ARM processor can perform image recognition based on the second operation result. In summary, on the one hand, the lightweight convolutional neural network acceleration module based on FPGA reduces the number of parameters in the convolutional neural network model due to its lightweight nature, thereby improving the convolution operation speed. On the other hand, this application separates data transmission and convolution operation; that is, it receives image data first, then performs convolution operation, and sends the result only after the convolution operation is completed. This reduces transmission latency and improves computational efficiency. Finally, the target point convolution module of this application can simultaneously meet the neural network computation requirements of both standard convolution and point convolution calculations, further reducing computational latency. In conclusion, this application can improve image recognition processing speed with low power consumption and limited resources. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0033] Figure 1This is a schematic diagram of an FPGA-based convolutional neural network image recognition system disclosed in this application;

[0034] Figure 2 This is a schematic diagram of a depthwise separable convolutional structure for a MobileNet network disclosed in this application.

[0035] Figure 3 This is a schematic diagram of the structure of a lightweight convolutional neural network acceleration module disclosed in this application;

[0036] Figure 4 This is a schematic diagram of the structure of a depthwise convolutional module disclosed in this application;

[0037] Figure 5 This is a schematic diagram of the structure of a target point convolution module disclosed in this application;

[0038] Figure 6 This is a schematic diagram of a specific FPGA-based convolutional neural network image recognition system disclosed in this application;

[0039] Figure 7 This is a flowchart of an FPGA-based convolutional neural network image recognition method disclosed in this application;

[0040] Figure 8 This is a schematic diagram illustrating one of the verification effects disclosed in this application;

[0041] Figure 9 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] Existing technologies lack design capabilities for lightweight convolutional neural networks, and often rely on increasing hardware resources to improve image recognition and classification speeds. This results in problems such as long convolution computation time, high resource consumption, and low acceleration efficiency.

[0044] Therefore, this application proposes an FPGA-based convolutional neural network image recognition scheme that can improve image recognition processing speed with less power consumption and fewer resources.

[0045] This application discloses an FPGA-based convolutional neural network image recognition system. See also... Figure 1As shown, the system includes: an ARM processor 11, an advanced high-performance bus 12, and a lightweight convolutional neural network acceleration module 13 based on FPGA; the lightweight convolutional neural network acceleration module includes a depth convolution module 131 and a target point convolution module 132, wherein the target point convolution module is a convolution module capable of performing point convolution operations and standard convolution operations.

[0046] The ARM processor 11 is used to preprocess the acquired image data and transmit the preprocessed image data to the lightweight convolutional neural network acceleration module 13 through the advanced high-performance bus 12; and is used to receive the calculation results of the lightweight convolutional neural network acceleration module 13 through the advanced high-performance bus 12, and then perform image recognition based on the received calculation results.

[0047] In this embodiment, the ARM processor 11 is connected to the FPGA (Field-Programmable Gate Array) via the Advanced High-Performance Bus (AHB) 12. The ARM processor 11 is used to preprocess and postprocess the image data. Postprocessing is to perform image recognition based on the received calculation results, including but not limited to: performing pupil recognition, confirming the identity information of the image object, etc. After confirmation, it can be further determined whether the image object has the corresponding access rights, etc.

[0048] The lightweight convolutional neural network acceleration module 13 is used to perform depth convolution operations on the preprocessed image data through the depth convolution module 131 to obtain a first operation result, and to perform point convolution operations and standard convolution operations on the first operation result through the target point convolution module 132 to obtain a second operation result; and is used to transmit the second operation result to the ARM processor 11 through the advanced high-performance bus 12 so that the ARM processor 11 can perform image recognition based on the second operation result.

[0049] In this embodiment, the lightweight convolutional neural network uses the MobileNet network as an example. The core of the MobileNet network design lies in replacing the traditional standard convolutional structure with depthwise separable convolution, which greatly reduces the computational cost and number of parameters of the network. Furthermore, for ease of computation, this model is usually decomposed into two smaller convolutional operations: depthwise convolution and pointwise convolution. The depthwise separable convolution structure of the MobileNet network is as follows: Figure 2 As shown, D represents the resolution of the input feature map, M represents the number of channels, Dk represents the kernel size, Dg represents the resolution of the output feature map, and N represents the number of output channels.

[0050] In this embodiment, the lightweight convolutional neural network acceleration module 13 based on FPGA, also known as the convolution acceleration unit, is the core of the entire hardware architecture design. The lightweight convolutional neural network acceleration module 13 accelerates the computation process of depthwise separable convolutions, including depthwise convolutions and target point convolutions, both of which are accelerated using FPGA. Furthermore, the lightweight convolutional neural network acceleration module 13 is a convolutional neural network acceleration module built using an in-memory computing approach that integrates convolution operations with read / write caching. Its maximum designed input feature map is a 128×128 grayscale image. The depthwise convolution kernel is designed as a fixed 3×3 kernel, and the target point convolution kernel is designed as a fixed 1×1 kernel. The lightweight convolutional neural network acceleration module 13 adopts a 16-channel convolution acceleration design. When the single-layer convolution computation exceeds 16 channels, the convolution channels are reused, using 16 channels as a complete convolution, and the computation is rolled over until the convolution computation requirements of all channels are met. The specific design structure of the lightweight convolutional neural network acceleration module 13 is as follows: Figure 3 As shown.

[0051] In this embodiment, the workflow of the lightweight convolutional neural network acceleration module 13 is as follows:

[0052] (1) After the system is powered on, it receives the model instructions, weight parameters, quantization parameters, quantization offset, and input feature map input by the user and stores them in the cache;

[0053] (2) The instruction control unit decodes the instruction, determines the type of convolution that needs to be accelerated, and passes the parameters to the corresponding module. In this embodiment, determining the type of convolution that needs to be accelerated specifically refers to determining whether it is a convolutional neural network that meets the acceleration conditions, such as a depth-separable convolutional neural network or other types of convolutional neural networks. The corresponding module is the lightweight convolutional neural network acceleration module 13.

[0054] (3) The convolution acceleration module reads the cached data, that is, it reads the weight parameters, quantization parameters, quantization offset, and input feature map, and then implements convolution through hardware.

[0055] (4) The convolution process is complete, and the convolution result is output.

[0056] In this embodiment, the depthwise convolutional module 131 and the target point convolutional module 132 in the lightweight convolutional neural network acceleration module 13 specifically include the following:

[0057] The deep convolution module 131 is the core of the MobileNet network structure. The deep convolution module of this application contains several deep convolution computation units, each of which includes a convolution readout module, a deep convolution operation module, and a ReLU (Rectified Linear Unit, activation function) processing module. The specific structure of the deep convolution module is as follows: Figure 4 As shown, in the deep convolutional array, a distributed parallel multiplication structure is used to perform multiplication operations on the preprocessed image data to obtain the multiplication results. The multiplication results are then transmitted to the addition tree for accumulation. Specifically, the addition tree accumulates the multiplication results based on a number of adders to obtain the first operation result. Furthermore, a pipelined register (Reg) is added between each layer of adders in the addition tree to temporarily store the accumulation results of each adder, thus forming a pipelined addition tree structure. Interconnection logic switches are added between the input and the buffer FIFO to realize the correspondence between the jump input neurons and the weight parameters and to satisfy the movement of the convolution window. In the figure, the input image pixels are cut by row, and pixels in every two rows are grouped into three groups, such as neurons in rows 1+3i, 2+3i, and 3+3i. Neurons in the same group are stored in the same storage unit (i is a non-negative integer). If the number of rows in the input feature map is not divisible by 3, it is filled with 0. Similarly, the weight matrix is ​​grouped into 3 groups by row. The depthwise convolutional computation unit uses a 3×3 convolutional kernel to extract the input feature map. Each pixel in the image remains independent during the cumulative multiplication, and pixels are processed in parallel. After the multiplication operation, the results are connected through an addition tree, and the computation of one pixel can be completed in each clock cycle. The calculated image pixels are then processed through a BN layer and a ReLU activation function before being temporarily stored in a register.

[0058] For the target point convolution module 132, the MobileNet network structure only has one standard convolutional layer. If an independent standard convolutional layer is designed, the FPGA resources used for standard convolutional layer computation will become idle after the first layer computation is completed. Since the functional structures of the standard convolutional layer and the point convolutional layer are quite similar, differing only in the size of the convolutional kernel, this application reorganizes the traditional point convolution computation unit structure and designs a parallel point convolution unit capable of performing both standard and point convolution computations, namely the target point convolution module 132. The parallel point convolution unit does not consider parallel processing within the convolutional kernel and includes a 1×1 multiplier, a ReLU module, a BN processing module, etc. The specific structure of the parallel point convolution unit is as follows: Figure 5As shown, a 1×1 multiplier is invoked to perform convolution. The convolution result is then subjected to BN and ReLU operations before being fed into an addition tree. The final result is temporarily stored in a register. Due to the data dependency between depthwise convolutional layers and pointwise convolutional layers, the number of computational units in depthwise convolution and pointwise convolution must be consistent.

[0059] In this embodiment, the preprocessed image data is subjected to depth convolution operation by the depth convolution module 131 constructed based on the above design method to obtain a first operation result. The first operation result is then subjected to point convolution operation and standard convolution operation by the target point convolution module 132 constructed based on the above design method to obtain a second operation result. The second operation result is then transmitted to the ARM processor 11 through the advanced high-performance bus 12 so that the ARM processor 11 can perform image recognition based on the second operation result.

[0060] This application discloses a specific FPGA-based convolutional neural network image recognition system. Compared to the previous embodiment, this embodiment further explains and optimizes the technical solution. See also... Figure 6 As shown, the FPGA-based convolutional neural network image recognition system further includes:

[0061] Camera 14 is used to acquire image data and send the acquired image data to the ARM processor 11;

[0062] Instruction register 15 is used to receive instructions sent by the user terminal and control the lightweight convolutional neural network acceleration module 13 to perform operations on the preprocessed image data according to the instructions.

[0063] Data register 16 is used to temporarily store the results of each operation, including the first operation result and the second operation result;

[0064] Peripheral storage device 17 is used to store the second calculation result;

[0065] Display 18 is used to display the result of the second calculation.

[0066] For details, see Figure 6As shown, image data is acquired by the camera 14 peripheral and then transmitted to the ARM processor 11 for preprocessing. The preprocessed image information is then transmitted via the AHB bus 12 to the programmable logic section, namely the lightweight convolutional neural network acceleration module 13, for further processing. The processed data is then transmitted back to the ARM processor 11 via the AHB bus 12 for subsequent processing. The image information after convolutional processing can be transmitted to the display via the peripheral storage device 17, such as DDR3 peripheral storage, or via the HDMI interface. To facilitate the analysis and execution of system instructions and data, the AHB bus also includes a data register 16 (DSRAM) and an instruction register 15 (ISRAM). In addition, the APB bus is also mounted on the AHB bus 12 of the system, and peripheral debugging interfaces such as USI (UART, SPI, IIC) and JTAG are added. On the embedded processor ARM side of this system architecture, apart from initial configuration and data preprocessing and post-processing, it does not participate in other intermediate system operations or data transmission processes, thus significantly improving computational efficiency.

[0067] This application discloses an FPGA-based convolutional neural network image recognition method, applied to an FPGA-based convolutional neural network image recognition system. The FPGA-based convolutional neural network image recognition system includes: an ARM processor, an advanced high-performance bus, and an FPGA-based lightweight convolutional neural network acceleration module. The lightweight convolutional neural network acceleration module includes a depthwise convolution module and a target point convolution module. The target point convolution module is a convolution module capable of performing point convolution operations and standard convolution operations. See also... Figure 7 As shown, the method includes:

[0068] Step S11: The ARM processor preprocesses the acquired image data and transmits the preprocessed image data to the lightweight convolutional neural network acceleration module via the advanced high-performance bus; the advanced high-performance bus receives the calculation results from the lightweight convolutional neural network acceleration module, and then performs image recognition based on the received calculation results.

[0069] Step S12: The preprocessed image data is subjected to depth convolution operation by the depth convolution module in the lightweight convolutional neural network acceleration module to obtain a first operation result. The first operation result is then subjected to point convolution operation and standard convolution operation by the target point convolution module in the lightweight convolutional neural network acceleration module to obtain a second operation result. The second operation result is then transmitted to the ARM processor through the advanced high-performance bus so that the ARM processor can perform image recognition based on the second operation result.

[0070] Therefore, this application proposes an FPGA-based convolutional neural network image recognition method, applied to an FPGA-based convolutional neural network image recognition system. The FPGA-based convolutional neural network image recognition system includes: an ARM processor, an advanced high-performance bus, and an FPGA-based lightweight convolutional neural network acceleration module. The lightweight convolutional neural network acceleration module includes a depthwise convolution module and a target point convolution module, wherein the target point convolution module is a convolution module capable of performing point convolution operations and standard convolution operations. The ARM processor preprocesses the acquired image data, and the preprocessed image data is transmitted to the lightweight convolutional neural network acceleration module via the advanced high-performance bus. The image data is processed via a network acceleration module and the advanced high-performance bus. The system receives the computation results from the lightweight convolutional neural network acceleration module and performs image recognition based on these results. The system then performs depth convolution on the preprocessed image data using the depth convolution module within the lightweight convolutional neural network acceleration module to obtain a first computation result. The system further performs point convolution and standard convolution on the first computation result using the target point convolution module within the lightweight convolutional neural network acceleration module to obtain a second computation result. Finally, the second computation result is transmitted to the ARM processor via the advanced high-performance bus, enabling the ARM processor to perform image recognition based on the second computation result. In summary, on the one hand, the lightweight convolutional neural network acceleration module based on FPGA reduces the number of parameters in the convolutional neural network model due to its lightweight nature, thereby improving the convolution operation speed. On the other hand, this application separates data transmission and convolution operation; that is, it receives image data first, then performs convolution operation, and sends the result only after the convolution operation is completed. This reduces transmission latency and improves computational efficiency. Finally, the target point convolution module of this application can simultaneously meet the neural network computation requirements of both standard convolution and point convolution calculations, further reducing computational latency. In conclusion, this application can improve image recognition processing speed with low power consumption and limited resources.

[0071] This application conducts an in-depth analysis of existing lightweight convolutional neural network models and optimizes the design from the hardware structure level, taking into account the special structure of the MobileNet network. Under the premise of maximizing resource conservation, the data transmission and operation of convolution are separated by a multi-data-stream driven approach. A high-speed image recognition system based on FPGA convolutional neural network is developed using the "tree" design concept. The control strategy of this system can simultaneously meet the neural network computation requirements of standard convolution and point convolution calculations, reducing computation latency. The example results show that at a system clock frequency of 100MHz, the designed accelerator achieves a unit frame rate of 203.7fps with a total power consumption of only 2.36W.

[0072] To verify the reliability of the FPGA-based high-speed convolutional neural network image recognition system of this application, image recognition experiments were conducted at a room temperature of 26.5℃ using a self-developed system test platform. The experiments tested the convolutional acceleration implementation and obtained the resource usage and efficiency of the entire architecture. The FPGA image recognition acceleration experiment results were output to a 23.8-inch LCD screen via HDMI interface. The image recognition output results are shown below. Figure 8 As shown.

[0073] Table 1 shows the FPGA resource usage of the overall architecture of the convolution accelerator designed in this application at a system operating frequency of 100MHz.

[0074] Table 1

[0075]

[0076]

[0077] Subsequently, the operating power consumption of the system designed in this application was evaluated. At an operating temperature of 27.6℃, the total operating power consumption of the entire acceleration architecture was found to be 2.36W, of which 83% was dynamic power consumption and 17% was static power consumption.

[0078] In summary, the FPGA-based high-speed image recognition system for convolutional neural networks achieves high image recognition processing speed with low power consumption and minimal resources.

[0079] Furthermore, embodiments of this application also provide an electronic device. Figure 9 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0080] Figure 9 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a display screen 23, an input / output interface 24, a communication interface 25, a power supply 26, and a communication bus 27. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the FPGA-based convolutional neural network image recognition method disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0081] In this embodiment, the power supply 26 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 25 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 24 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0082] Furthermore, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored thereon may include computer programs 221, and the storage method may be temporary storage or permanent storage. The computer programs 221 may include, in addition to computer programs capable of performing the FPGA-based convolutional neural network image recognition method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, computer programs capable of performing other specific tasks.

[0083] Furthermore, embodiments of this application also disclose a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned FPGA-based convolutional neural network image recognition method.

[0084] For the specific steps of this method, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0085] The various embodiments in this application are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. For the same or similar parts between the various embodiments, refer to each other. As for the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and relevant parts can be referred to in the method section.

[0086] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0087] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0088] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0089] The foregoing has provided a detailed description of the FPGA-based convolutional neural network image recognition system, method, device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An FPGA-based convolutional neural network image recognition system, characterized in that, include: The system includes an ARM processor, an advanced high-performance bus, and a lightweight convolutional neural network acceleration module based on an FPGA. The lightweight convolutional neural network acceleration module includes a depthwise convolution module and a target point convolution module, wherein the target point convolution module is a convolution module capable of performing point convolution operations and standard convolution operations. The ARM processor is used to preprocess the acquired image data and transmit the preprocessed image data to the lightweight convolutional neural network acceleration module through the advanced high-performance bus. It is used to receive the computation results of the lightweight convolutional neural network acceleration module through the advanced high-performance bus, and then perform image recognition based on the received computation results; The lightweight convolutional neural network acceleration module is used to perform depth convolution operations on the preprocessed image data through the depth convolution module to obtain a first operation result, and to perform point convolution and standard convolution operations on the first operation result through the target point convolution module to obtain a second operation result; and is used to transmit the second operation result to the ARM processor through the advanced high-performance bus so that the ARM processor can perform image recognition based on the second operation result; The lightweight convolutional neural network acceleration module is a convolutional neural network acceleration module constructed using an in-memory computing approach that integrates convolution operations and read / write caching. The depthwise convolution module includes: A distributed parallel multiplication structure is used to perform multiplication operations on the preprocessed image data to obtain the results of the multiplication operations; An addition tree is used to accumulate the results of the multiplication operation based on a number of adders to obtain the first operation result. A streamlined register is used to temporarily store the accumulation result of each adder; The target point convolution module is a parallel convolution module that reorganizes the point convolution structure to achieve point convolution operations and standard convolution operations. In the deep convolutional computation array, a distributed parallel multiplication structure is used to perform multiplication operations on the preprocessed image data to obtain the multiplication results. These results are then transmitted to an addition tree for accumulation. The addition tree accumulates the multiplication results based on a number of adders to obtain the first computation result. Pipeline registers are added between each adder layer of the addition tree to temporarily store the accumulation results of each adder, thus forming a pipelined addition tree structure. Interconnection logic switches are added between the input and the buffer FIFO to realize the correspondence between the jump input neurons and the weight parameters, and to satisfy the movement of the convolution window. Image pixels are cut into rows, and pixels in every two rows are grouped into three groups. Neurons in the same group are stored in the same storage unit. If the number of rows in the input feature map is not divisible by 3, it is filled with zeros. The weight matrix is ​​grouped into three groups by rows. The depthwise convolutional computation unit uses a 3×3 convolution kernel to extract the input feature map. Each pixel in the image remains independent during the cumulative multiplication, and the pixels are processed in parallel. After the multiplication operation, the results are connected through an addition tree. The calculation of one pixel can be completed in each clock cycle. After the calculated image pixels are processed by the BN layer and the ReLU activation function, they are sent to a register for temporary storage.

2. The FPGA-based convolutional neural network image recognition system according to claim 1, characterized in that, The FPGA-based convolutional neural network image recognition system further includes: The instruction register is used to receive instructions sent by the user terminal and control the lightweight convolutional neural network acceleration module to perform operations on the preprocessed image data according to the instructions. A data register is used to temporarily store the results of various operations, including the first operation result and the second operation result.

3. The FPGA-based convolutional neural network image recognition system according to claim 1, characterized in that, The FPGA-based convolutional neural network image recognition system further includes: An external storage device is used to store the result of the second operation; A display is used to show the result of the second calculation.

4. The FPGA-based convolutional neural network image recognition system according to any one of claims 1 to 3, characterized in that, The FPGA-based convolutional neural network image recognition system further includes: The camera is used to acquire image data and send the acquired image data to the ARM processor.

5. An FPGA-based convolutional neural network image recognition method, characterized in that, An application is made to an FPGA-based convolutional neural network image recognition system. The FPGA-based convolutional neural network image recognition system includes: an ARM processor, an advanced high-performance bus, and an FPGA-based lightweight convolutional neural network acceleration module. The lightweight convolutional neural network acceleration module includes a depthwise convolution module and a target point convolution module, wherein the target point convolution module is a convolution module capable of performing point convolution operations and standard convolution operations. The method includes: The ARM processor preprocesses the acquired image data and transmits the preprocessed image data to the lightweight convolutional neural network acceleration module via the advanced high-performance bus. The processor also receives the computation results from the lightweight convolutional neural network acceleration module via the advanced high-performance bus and performs image recognition based on the received computation results. The preprocessed image data is subjected to depthwise convolution operation by the depthwise convolution module in the lightweight convolutional neural network acceleration module to obtain a first operation result. The first operation result is then subjected to point convolution operation and standard convolution operation by the target point convolution module in the lightweight convolutional neural network acceleration module to obtain a second operation result. The second operation result is then transmitted to the ARM processor through the advanced high-performance bus so that the ARM processor can perform image recognition based on the second operation result. The lightweight convolutional neural network acceleration module is a convolutional neural network acceleration module constructed using an in-memory computing approach that integrates convolution operations and read / write caching. The depthwise convolution module includes: A distributed parallel multiplication structure is used to perform multiplication operations on the preprocessed image data to obtain the results of the multiplication operations; An addition tree is used to accumulate the results of the multiplication operation based on a number of adders to obtain the first operation result. A streamlined register is used to temporarily store the accumulation result of each adder; The target point convolution module is a parallel convolution module that reorganizes the point convolution structure to achieve point convolution operations and standard convolution operations. In the deep convolutional computation array, a distributed parallel multiplication structure is used to perform multiplication operations on the preprocessed image data to obtain the multiplication results. These results are then transmitted to an addition tree for accumulation. The addition tree accumulates the multiplication results based on a number of adders to obtain the first computation result. Pipeline registers are added between each adder layer of the addition tree to temporarily store the accumulation results of each adder, thus forming a pipelined addition tree structure. Interconnection logic switches are added between the input and the buffer FIFO to realize the correspondence between the jump input neurons and the weight parameters, and to satisfy the movement of the convolution window. Image pixels are cut into rows, and pixels in every two rows are grouped into three groups. Neurons in the same group are stored in the same storage unit. If the number of rows in the input feature map is not divisible by 3, it is filled with zeros. The weight matrix is ​​grouped into three groups by rows. The depthwise convolutional computation unit uses a 3×3 convolution kernel to extract the input feature map. Each pixel in the image remains independent during the cumulative multiplication, and the pixels are processed in parallel. After the multiplication operation, the results are connected through an addition tree. The calculation of one pixel can be completed in each clock cycle. After the calculated image pixels are processed by the BN layer and the ReLU activation function, they are sent to a register for temporary storage.

6. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor is provided for executing the computer program to implement the FPGA-based convolutional neural network image recognition method as described in claim 5.

7. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the FPGA-based convolutional neural network image recognition method as described in claim 5.

Images