Automatic deployment method of convolutional neural network based on FPGA

Abstract

The present invention discloses a method and device for automatic deployment of convolutional neural networks based on FPGA. By constructing an automated neural network compilation device, universal, hardware-oriented optimization processing is performed on various input convolutional neural network models to reduce network complexity. The custom hardware instructions obtained after compilation are used to configure the FPGA-based neural network hardware acceleration device that includes a series of universal acceleration modules to control its operations, thereby realizing the automatic and efficient deployment of different convolutional neural network models on the universal hardware acceleration device.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an automatic deployment method of a convolutional neural network based on FPGA. Background Art

[0002] In recent years, artificial intelligence algorithms based on convolutional neural networks have achieved widespread application in image processing. Convolutional neural networks are characterized by large number of network parameters and are computationally intensive. Therefore, most current CNN algorithms utilize high-performance devices such as CPUs and GPUs for training and inference. Although CPUs and GPUs can achieve high performance when deploying CNNs, their significant power consumption limits their application in power-constrained scenarios. Embedded hardware devices, with their low power consumption and high energy efficiency, are increasingly being used to deploy network algorithms in power-sensitive scenarios. FPGA-based network deployment is the most widely used solution. However, in applications such as onboard remote sensing, hardware computing resources are strictly limited, and the content of applications is complex and diverse, requiring hardware accelerators that can flexibly handle diverse applications. However, existing FPGA neural network hardware accelerators often only accelerate specific models, resulting in limited flexibility and inability to handle diverse networks. Furthermore, the current mainstream network deployment method still relies on manually writing deployment code, which is not only cumbersome, but also time-consuming and labor-intensive. These issues make the flexible deployment of various deep neural networks on FPGAs challenging. Summary of the Invention

[0003] In view of this, the present invention provides an automatic deployment method of convolutional neural networks based on FPGA, which can realize real-time and automatic deployment of different convolutional neural networks on FPGA.

[0004] The present disclosure provides an automatic deployment method for a convolutional neural network based on FPGA, comprising the following steps:

[0005] Build a universal convolutional neural network optimization method for hardware;

[0006] Based on the optimization method, a configurable and universal hardware acceleration device based on FPGA is constructed;

[0007] Building a custom hardware instruction set for configuring the hardware acceleration device;

[0008] Constructing a neural network compilation device, optimizing various convolutional neural network models according to the optimization method, and converting the optimized data structure into the custom hardware instructions;

[0009] The generalized network hardware acceleration device is configured according to the hardware instructions to achieve automatic deployment of a specific convolutional neural network on the generalized hardware acceleration device.

[0010] Furthermore, the convolutional neural network optimization method includes one or more of a network quantization method, an operation unification method, and a dynamic segmentation method, wherein:

[0011] In the network quantization method, the number of quantization bits of the convolution weight and the number of quantization bits of the convolution feature map are both set to 8;

[0012] The operation unification method unifies the full connection operation and convolution operation of the network into a convolution operation, and unifies the LeakyReLU operation and ReLU operation into a LeakyReLU operation;

[0013] The dynamic segmentation method is based on the dimensional information of each layer of the neural network and determines whether the feature map of each layer needs to be segmented through analysis.

[0014] Furthermore, the dynamic segmentation method includes the following steps:

[0015] Analyze the feature map storage requirements of each layer in the neural network and derive the storage threshold at which most layers can fully store feature maps;

[0016] Based on the storage threshold, analyzing whether the storage requirement of each layer exceeds the storage threshold;

[0017] For layers that do not exceed the threshold, their feature maps do not need to be split;

[0018] For layers that exceed the threshold, the blocks into which the feature map is split are determined according to the ratio of the storage threshold and the storage requirement.

[0019] Furthermore, in the operation unification method, the unification of the full connection operation and the convolution operation is achieved by replacing the full connection operation with the convolution operation, which specifically includes the following steps:

[0020] Determine custom weight size;

[0021] Reconstruct the one-dimensional input vector of the fully connected operation into a three-dimensional input tensor, ensuring that the size of the input tensor is equal to the custom weight size;

[0022] Reconstruct the two-dimensional weight matrix of the fully connected operation into a four-dimensional weight tensor, ensuring that the size of the weight tensor is equal to the customized weight size;

[0023] The input tensor and weight tensor are convolved to obtain a convolution result that is exactly the same as the original fully connected result.

[0024] Furthermore, the hardware acceleration device includes acceleration modules for various common computing layers in neural networks, and corresponding configuration systems.

[0025] Furthermore, the instructions in the custom hardware instruction set include:

[0026] Configuration instructions, used to transmit configuration information to the hardware acceleration device;

[0027] Data movement instructions, used to control the interaction between the processing engine of the hardware acceleration device and the off-chip storage;

[0028] Handshake instructions are used to mark the beginning and end of the configuration phase, data transmission phase, and calculation phase;

[0029] The instruction includes a 32-bit binary code, with the first 8 bits serving as an identification header of the instruction, wherein the data movement instruction is a multi-level instruction.

[0030] Furthermore, the neural network compilation device includes a front-end parsing module, a function channel, a memory allocation module and an instruction generation module, wherein:

[0031] The front-end parsing module converts the input neural network model into a fixed-format data structure;

[0032] The memory allocation module allocates memory addresses for data in the network;

[0033] A functional channel, configured to optimize the data structure according to the optimization method and make the optimized data structure correspond to the architecture of the hardware acceleration device;

[0034] The instruction generation module realizes the mapping of the compilation result to the hardware instruction set.

[0035] The present disclosure provides a method for dynamic segmentation of a neural network, comprising the following steps:

[0036] Analyze the feature map storage requirements of each layer in the neural network and derive the storage threshold at which most layers can fully store feature maps;

[0037] Based on the storage threshold, analyzing whether the storage requirement of each layer exceeds the storage threshold;

[0038] For layers that do not exceed the threshold, their feature maps do not need to be split;

[0039] For layers that exceed the threshold, the blocks into which the feature map is split are determined according to the ratio of the storage threshold and the storage requirement.

[0040] The present disclosure also provides an FPGA-based convolutional neural network automatic deployment device, comprising:

[0041] FPGA-based, configurable, general-purpose hardware acceleration devices, including acceleration modules for various common computing layers in neural networks, and corresponding configuration systems;

[0042] A neural network compilation device is used to perform hardware-oriented optimization processing on various specific convolutional neural network models and convert the optimized data structure into custom hardware instructions. The instructions are used to configure the hardware acceleration device to achieve automatic deployment of different network models on the hardware acceleration device.

[0043] Furthermore, the custom hardware instructions include:

[0044] Configuration instructions, used to transmit configuration information to the hardware acceleration device;

[0045] Data movement instructions, used to control the interaction between the processing engine of the hardware acceleration device and the off-chip storage;

[0046] Handshake instructions are used to mark the beginning and end of the configuration phase, data transmission phase, and calculation phase;

[0047] The instruction includes a 32-bit binary code, with the first 8 bits serving as an identification header of the instruction, wherein the data movement instruction is a multi-level instruction.

[0048] Furthermore, the neural network compilation device includes a front-end parsing module, a function channel, a memory allocation module and an instruction generation module, wherein:

[0049] The front-end parsing module converts the input neural network model into a fixed-format data structure;

[0050] The memory allocation module allocates memory addresses for data in the network;

[0051] The function channel performs the hardware-oriented optimization processing on the data structure, and makes the optimized data structure correspond to the architecture of the hardware acceleration device;

[0052] The instruction generation module realizes the mapping of the compilation result to the hardware instruction set.

[0053] The present invention provides an FPGA-based convolutional neural network automatic deployment method. By constructing an automated neural network compilation device, the method performs hardware-oriented, universal optimization processing on various input convolutional neural network models to reduce network complexity. The method also uses the custom hardware instructions obtained after compilation to configure the FPGA-based neural network hardware acceleration device that includes a series of universal acceleration modules to control its operations, thereby realizing the automatic and real-time deployment of different convolutional neural network models on the universal hardware acceleration device.

[0054] Beneficial effects: ① Different from the traditional method of hardware implementation specifically for a single network, this method provides an integrated network deployment solution including network optimization methods, intermediate compilers, and FPGA hardware accelerators, realizing the automated deployment of various specific network models; ② By designing network optimization methods, the network complexity is reduced, providing the prerequisites for network deployment; ③ By designing configurable hardware accelerators, a highly flexible and efficient deployment platform is provided for network deployment; ④ A neural network compiler with a complete compilation process from model to instruction is provided, automating the network deployment process, effectively avoiding the problems of complex manual deployment process, time-consuming and labor-intensive coding process, and difficulty in deploying different networks in traditional FPGA implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Flowchart of the automatic deployment method of convolutional neural network based on FPGA according to the present disclosure;

[0056] Figure 2 Schematic diagram of a process of replacing a fully connected operation with a convolution operation in an exemplary embodiment;

[0057] Figure 3 FIG1 is a schematic diagram showing the structure of a convolutional neural network hardware accelerator in an exemplary embodiment;

[0058] Figure 4 An example diagram of a custom hardware instruction set in an exemplary embodiment;

[0059] Figure 5 FIG1 is an overall framework diagram of a neural network compiler in an exemplary embodiment;

[0060] Figure 6 This is an example diagram of a “computational graph” data structure used within a neural network compiler in an exemplary embodiment;

[0061] Figure 7 is a flowchart of a compiler internal function channel process in an exemplary embodiment;

[0062] Figure 8 This is a schematic diagram of the instruction generator interface function processing process in an exemplary embodiment. DETAILED DESCRIPTION

[0063] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0064] This invention provides an FPGA-based automatic deployment method for convolutional neural networks, applicable to deep convolutional neural networks. With the development of deep learning, a large number of different types of deep convolutional neural networks have emerged. The automatic deployment method provided by the present invention can deploy various neural networks that include convolution, batch normalization (BN), leaky ReLU activation, ReLU activation, max pooling, global max pooling, global average pooling, and fully connected networks, including but not limited to the VGG series networks, YOLOv2, and the like.

[0065] The flowchart of the exemplary FPGA-based convolutional neural network automatic deployment method is as follows: Figure 1 As shown, the following steps are included:

[0066] Step S1: Design a hardware-oriented, universal optimization method for convolutional neural networks with large parameter counts and intensive computations to reduce network complexity. The optimization method includes one or more of a network quantization method, an operation unification method, and a dynamic segmentation method.

[0067] The network quantization method uses a symmetric interval hybrid quantization method to quantize the parameters and feature maps of the convolution operation in the network, quantizing the 32-bit floating-point data into fixed-point data, reducing the amount of network data and facilitating network deployment; at the same time, inverse quantization is performed before the BN operation and activation operation to maintain the floating-point nature of these two operations and reduce the accuracy loss of the network. Considering the quantization of any N bits, the symmetric interval quantization algorithm is shown as follows:

[0068]

[0069] Among them, q represents the fixed-point matrix after quantization, r represents the floating-point matrix before quantization, and the clamp function is used to limit the range of the quantization result to [(-2 N-1 +1),(2 N-1 -1)] to prevent data overflow. The quantization scaling factor S is used to determine the mapping relationship between floating-point numbers and fixed-point numbers, which is defined by the following formula:

[0070]

[0071] In contrast, the inverse quantization algorithm is used to restore the fixed-point matrix back to a floating-point matrix. The floating-point matrix r can be obtained as follows:

[0072] r=S f S w ×q (1)

[0073] Among them, S f represents the scaling factor of the feature map in the previous quantized convolution, S wIndicates the scaling factor of the weight in the previous quantized convolution. In this embodiment, it is preferred that the number of quantization bits for the convolution weight and the number of quantization bits for the convolution feature map are both set to 8. After the 32-bit floating-point data is quantized, 8-bit fixed-point data is obtained.

[0074] The operation unification method unifies the full connection operation and the convolution operation into the convolution operation, and unifies the LeakyReLU operation and the ReLU operation into the LeakyReLU operation, thereby reducing the resource consumption of hardware implementation. The unification of the full connection and convolution operations is achieved in this embodiment by replacing the full connection operation with the convolution operation. The process diagram of replacing the full connection operation with the convolution operation is shown in the figure. Figure 2 As shown: Full connection is essentially the inner product process of the input vector and the weight matrix, while convolution is the multiplication and addition process of the three-dimensional feature map and the four-dimensional weight. Therefore, by reconstructing the low-dimensional data into a high-dimensional tensor, the full connection can be replaced by convolution. If the custom convolution kernel size is N kh ×N kw , then the length of the full connection is N if The input vector is reconstructed into N new_if ×N kh ×N kw A three-dimensional tensor, and N if ×N of The weight matrix of size N is reshaped new_if ×N kh ×N kw ×N of In this way, the result obtained after the convolution operation is exactly the same as the result of the original full connection.

[0075] The unification of the LeakyReLU and ReLU operations, analyzing the properties of the two operations, reveals that compared to the ReLU operation, the LeakyReLU operation simply adds a slope coefficient α to the negative half-axis. Therefore, as long as α is set to 0, the LeakyReLU calculation is transformed into a ReLU calculation. Therefore, in the embodiments of the present invention, the LeakyReLU operation is used to unify ReLU and LeakyReLU.

[0076] The dynamic segmentation method is based on the dimensional information of each layer of the neural network, and determines whether the feature map of each layer needs to be segmented through analysis. In a neural network, the size of the feature maps of each layer is different, and the required storage space is not consistent. However, the on-chip cache of the hardware accelerator needs to be set according to the size of the largest feature map. On the one hand, it leads to excessive on-chip storage overhead, and on the other hand, it leads to idle on-chip cache for most of the computing time, which is inefficient. The preferred dynamic segmentation method in the exemplary embodiment includes the following steps:

[0077] Analyze the feature map storage requirements of each layer in the neural network and derive the storage threshold at which most layers can fully store feature maps;

[0078] Based on the storage threshold, analyze whether the storage requirements of each layer exceed the storage threshold;

[0079] For layers that do not exceed the threshold, their feature maps do not need to be split; for layers that exceed the threshold, the blocks into which the feature maps are split are determined according to the ratio of the storage threshold to the storage requirement.

[0080] After applying the dynamic splitting method, the storage overhead of the on-chip cache is greatly reduced and its idle state is also improved.

[0081] Step S2: Based on the optimization method in Step 1, a configurable, FPGA-based, general-purpose network hardware accelerator is constructed to address the computational characteristics and acceleration requirements of convolutional neural networks. The hardware accelerator includes acceleration modules for commonly used computational layers in various neural networks and a complete configuration system that supports the acceleration of various neural networks.

[0082] As a preferred embodiment, the structural diagram of the hardware accelerator provided in this embodiment is as follows: Figure 3 As shown, a series of acceleration modules are provided for general operation layers in various neural networks and meet the requirements of the optimization method in step one, including convolution module, BN module, LeakyReLU module, maximum pooling module, and global pooling module. In addition, the hardware accelerator also includes a complete configuration system, namely a data control module, a finite state machine module, and a routing module, which correspond to convolution type configuration, convolution size configuration, and module scheduling configuration. Under the control of the configuration system, the hardware accelerator provided by the present disclosure has a high degree of versatility and flexibility, and can accelerate neural networks of different structures.

[0083] Step S3: design a custom hardware instruction set for configuring the hardware accelerator obtained in step 2.

[0084] Based on the configuration system of the hardware accelerator designed in step 2, this embodiment designs a custom hardware instruction set. Examples of preferred hardware instruction sets are as follows: Figure 4As shown, the instructions in the instruction set are a series of 32-bit binary codes, with the first 8 bits of the instructions serving as an identifier. Instructions can be divided into three categories based on their function: configuration instructions, data movement instructions, and handshake instructions. Configuration instructions convey the information required by the hardware accelerator to configure the system, including the type of operation, convolution size, and convolution type. Different pieces of information carry varying amounts of data, so a single 32-bit instruction can consist of multiple pieces of information. For example, bits 0 to 9 of configuration instruction 3 correspond to the number of output channels, bits 10 to 19 correspond to the number of input channels, and bits 20 to 23 correspond to the convolution kernel size. Data movement instructions control the interaction between the hardware accelerator's processing engine and off-chip memory, including reading and writing data from and to specific addresses. It's important to note that memory addresses and the amount of data read or written are often large numbers, requiring a longer bit length. Therefore, data movement instructions are designed as multi-stage instructions, requiring multiple 32-bit instructions to complete a single read or write operation. The handshake instruction does not contain specific information. It is mainly responsible for marking the start and end of the configuration phase, data transmission phase and calculation phase.

[0085] Step S4: Construct a neural network compiler to implement the optimization method proposed in Step 1 and automatically convert the network model into custom hardware instructions and network parameters. The instructions and parameters obtained from the model conversion can be directly transmitted to the hardware accelerator. The configuration system on the accelerator decodes the instructions and controls the hardware accelerator's operations based on the instruction information, thereby achieving automatic deployment of the convolutional neural network on the hardware accelerator.

[0086] Specifically, based on the optimization method proposed in step 1, facing the hardware accelerator structure designed in step 2, in order to output the hardware instructions designed in step 3, the overall block diagram of the automated neural network compiler provided by the exemplary embodiment is as follows: Figure 5 As shown, the compiler is a full-stack software written in C++, primarily consisting of a front-end parser, a function pipeline, a memory allocator, and an instruction generator. The compiler inputs various neural network models defined in the PyTorch framework. The front-end parser converts the input model into a fixed-format data structure. The function pipeline performs specific compilation functions, and the memory allocator allocates memory addresses for data in the network. The instruction generator maps the compilation results to the hardware instruction set.

[0087] The front-end parser, with the help of the operator definition library, converts the input model into a data structure called a "computational graph". An example computational graph is as follows Figure 6As shown in Figure 1, the computation graph consists of operation nodes and data tensors. Operation nodes represent various operations in a convolutional neural network and store operation properties. Data tensors are divided into runtime tensors and parameter tensors. Runtime tensors represent the forward data flow during network inference, while parameter tensors store the network's static parameter data.

[0088] The function channel is the core part of the compiler, which is used to complete the compilation of the computation graph. The function channel consists of a series of sub-modules, each of which implements a specific compilation function. It should be noted that in order to achieve automatic deployment of the network, these compilation functions not only need to implement the optimization method proposed in step 1, but also make the compiled computation graph correspond to the architecture of the hardware accelerator in step 2. The processing flow of the function channel is as follows Figure 7 As shown, the operation unification submodule, node fusion submodule, data quantization submodule, and dynamic segmentation submodule are implemented. The operation unification submodule is used to convert the fully connected nodes into convolution nodes, convert the ReLU nodes into LeakyReLU nodes, and reconstruct the corresponding operation properties. The node fusion submodule is used to fuse adjacent nodes and eliminate the adjacent tensors between nodes. This can eliminate the access to off-chip storage during the calculation process, so that the calculation graph conforms to the pipeline calculation structure of the hardware accelerator. The data quantization module adjusts the amount of data recorded in the runtime tensor according to the proposed symmetric interval quantization algorithm, and completes the quantization operation of the floating-point parameters stored in the parameter tensor. The dynamic segmentation submodule implements the dynamic segmentation algorithm to obtain the segmentation results of the feature map corresponding to each layer of the network, copies the corresponding operation nodes according to the segmentation results, and adjusts the operation properties in the nodes.

[0089] The memory allocator allocates memory for each tensor in the computation graph using a linear scan algorithm. First, it traverses all tensors in the computation graph and calculates the required memory space based on their dimensionality. Next, it calculates the active intervals of each tensor based on the order in which they are executed in the computation graph. The active interval represents the time period during which a tensor needs to be stored and accessed. After the active interval table for each tensor is created, the overlap between active intervals is analyzed. If the active intervals of two tensors overlap, they cannot be allocated to the same memory block. Conversely, if the active intervals of two tensors do not overlap, their memory spaces can overlap, reducing off-chip storage overhead. It is important to note that, given that a deployed network algorithm often needs to be run multiple times in real-world applications, parameter tensors are stored in fixed off-chip storage until the deployed network is replaced. In contrast, the memory space of a runtime tensor is released immediately after the operations involving the tensor are completed. After memory allocation, each tensor stores its allocated address and space size.

[0090] The instruction generator converts the compiled computation graph into parameter files and hardware instructions. Parameters can be directly extracted from the parameter tensor and organized into parameter files that can be directly stored. The generation of hardware instructions requires the creation of an instruction generation interface function based on the hardware instruction set designed in step 3. The workflow of the interface function is as follows: Figure 8 As shown, each type of compute node has a corresponding interface function. This interface function contains a pre-programmed instruction block based on the instruction set. This instruction block performs the configuration and computation of the corresponding compute node. After receiving the compute node input, the interface function passes parameters to the pre-programmed instruction block based on the compute properties stored in the node. Within the instruction block, configuration instructions are defined first, followed by data movement instructions. A handshake instruction separates the configuration and data movement instructions, separating the different processing steps. In hardware terms, this first involves configuring the hardware accelerator. Then, the data movement instructions control the entry of feature map data and parameter data into the hardware accelerator, driving the accelerator to begin computation. After the computation is complete, the data movement instructions control the transfer of the results back to off-chip storage. Multiple instruction blocks are combined to form the hardware deployment instruction file for a convolutional neural network.

[0091] The hardware parsing module on the hardware accelerator can directly decode the instruction file and control the hardware accelerator's operations based on the instruction information, thereby automatically deploying different convolutional neural networks on the hardware accelerator. In this way, an automated, integrated deployment method for various convolutional neural networks has been established.

[0092] The embedded image processing system constructed using the FPGA-based convolutional neural network automatic deployment method proposed in this invention can flexibly handle a variety of different applications. A single processing platform can quickly respond to different application scenarios such as scene classification, target detection, and semantic segmentation, which can greatly improve the flexibility of neural network algorithm applications and reduce development difficulty and system costs.

[0093] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for automatic deployment of convolutional neural networks based on FPGA, characterized in that: The following steps are involved: Build a universal convolutional neural network optimization method for hardware; Based on the optimization method, a configurable and universal hardware acceleration device based on FPGA is constructed; Building a custom hardware instruction set for configuring the hardware acceleration device; Constructing a neural network compilation device, optimizing various convolutional neural network models according to the optimization method, and converting the optimized data structure into the custom hardware instructions; Configuring the generalized network hardware acceleration device according to the hardware instructions to achieve automatic deployment of a specific convolutional neural network on the generalized hardware acceleration device; The convolutional neural network optimization method includes: a network quantization method, an operation unification method, and a dynamic segmentation method, wherein: In the network quantization method, the number of quantization bits of the convolution weight and the number of quantization bits of the convolution feature map are both set to 8; The operation unification method unifies the full connection operation and convolution operation of the network into a convolution operation, and unifies the LeakyReLU operation and ReLU operation into a LeakyReLU operation; The dynamic segmentation method determines whether the feature map of each layer needs to be segmented by analyzing the dimensional information of each layer of the neural network; The dynamic segmentation method comprises the following steps: Analyze the feature map storage requirements of each layer in the neural network and derive the storage threshold at which most layers can fully store feature maps; Based on the storage threshold, analyzing whether the storage requirement of each layer exceeds the storage threshold; For layers that do not exceed the threshold, their feature maps do not need to be split; For layers that exceed the threshold, the blocks into which the feature map is split are determined according to the ratio of the storage threshold to the storage requirement; In the operation unification method, the unification of the full connection operation and the convolution operation is achieved by replacing the full connection operation with the convolution operation, which specifically includes the following steps: Determine custom weight size; Reconstruct the one-dimensional input vector of the fully connected operation into a three-dimensional input tensor, ensuring that the size of the input tensor is equal to the custom weight size; Reconstruct the two-dimensional weight matrix of the fully connected operation into a four-dimensional weight tensor, ensuring that the size of the weight tensor is equal to the customized weight size; The input tensor and weight tensor are convolved to obtain a convolution result that is exactly the same as the original fully connected result.

2. The method according to claim 1, wherein The hardware acceleration device includes acceleration modules for various common computing layers in neural networks, as well as corresponding configuration systems.

3. The method according to claim 2, wherein The instructions in the custom hardware instruction set include: Configuration instructions, used to transmit configuration information to the hardware acceleration device; Data movement instructions, used to control the interaction between the processing engine of the hardware acceleration device and the off-chip storage; Handshake instructions are used to mark the beginning and end of the configuration phase, data transmission phase, and calculation phase; The instruction includes a 32-bit binary code, with the first 8 bits serving as an identification header of the instruction, wherein the data movement instruction is a multi-level instruction.

4. The method according to claim 1, wherein The neural network compilation device includes a front-end parsing module, a function channel, a memory allocation module and an instruction generation module, wherein: The front-end parsing module converts the input neural network model into a fixed-format data structure; The memory allocation module allocates memory addresses for data in the network; A functional channel, configured to optimize the data structure according to the optimization method and make the optimized data structure correspond to the architecture of the hardware acceleration device; The instruction generation module realizes the mapping of the compilation result to the hardware instruction set.

5. An FPGA-based convolutional neural network automatic deployment device, characterized in that: include: FPGA-based, configurable, general-purpose hardware acceleration devices, including acceleration modules for various common computing layers in neural networks, and corresponding configuration systems; Neural network compilation equipment, which is used to perform hardware-oriented optimization processing on various specific convolutional neural network models and convert the optimized data structure into custom hardware instructions. The instructions are used to configure the hardware acceleration equipment and realize the automatic deployment of different network models on the hardware acceleration equipment; The FPGA-based, configurable, universal hardware acceleration device is obtained based on a universal hardware-oriented convolutional neural network optimization method; The convolutional neural network optimization method includes: a network quantization method, an operation unification method, and a dynamic segmentation method, wherein: In the network quantization method, the number of quantization bits of the convolution weight and the number of quantization bits of the convolution feature map are both set to 8; The operation unification method unifies the full connection operation and convolution operation of the network into a convolution operation, and unifies the LeakyReLU operation and ReLU operation into a LeakyReLU operation; The dynamic segmentation method determines whether the feature map of each layer needs to be segmented by analyzing the dimensional information of each layer of the neural network; The dynamic segmentation method comprises the following steps: Analyze the feature map storage requirements of each layer in the neural network and derive the storage threshold at which most layers can fully store feature maps; Based on the storage threshold, analyzing whether the storage requirement of each layer exceeds the storage threshold; For layers that do not exceed the threshold, their feature maps do not need to be split; For layers that exceed the threshold, the blocks into which the feature map is split are determined according to the ratio of the storage threshold to the storage requirement; In the operation unification method, the unification of the full connection operation and the convolution operation is achieved by replacing the full connection operation with the convolution operation, which specifically includes the following steps: Determine custom weight size; Reconstruct the one-dimensional input vector of the fully connected operation into a three-dimensional input tensor, ensuring that the size of the input tensor is equal to the custom weight size; Reconstruct the two-dimensional weight matrix of the fully connected operation into a four-dimensional weight tensor, ensuring that the size of the weight tensor is equal to the customized weight size; The input tensor and weight tensor are convolved to obtain a convolution result that is exactly the same as the original fully connected result.

6. The deployment device according to claim 5, wherein: The custom hardware instructions include: Configuration instructions, used to transmit configuration information to the hardware acceleration device; Data movement instructions, used to control the interaction between the processing engine of the hardware acceleration device and the off-chip storage; Handshake instructions are used to mark the beginning and end of the configuration phase, data transmission phase, and calculation phase; The instruction includes a 32-bit binary code, with the first 8 bits serving as an identification header of the instruction, wherein the data movement instruction is a multi-level instruction.

7. The deployment device according to claim 5 or 6, characterized in that The neural network compilation device includes a front-end parsing module, a function channel, a memory allocation module and an instruction generation module, wherein: The front-end parsing module converts the input neural network model into a fixed-format data structure; The memory allocation module allocates memory addresses for data in the network; The function channel performs the hardware-oriented optimization processing on the data structure, and makes the optimized data structure correspond to the architecture of the hardware acceleration device; The instruction generation module realizes the mapping of the compilation result to the hardware instruction set.

Images