Convolution kernel and CNN separated FPGA hardware acceleration platform design method
By separating the convolution kernel from the CNN and adopting the dynamic partial reconstruction (DPR) method, the problem that the FPGA hardware acceleration platform cannot realize online reconstruction in intelligent manufacturing is solved, and the rapid online adjustment and reconstruction of convolutional neural networks on FPGA is realized, which improves production efficiency and reduces cost and power consumption.
Patent Information
- Application Number
- CN202411991739.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2025-05-02
AI Technical Summary
In intelligent manufacturing, the FPGA hardware acceleration platform cannot realize online reconstruction when inference scenarios are complex and resources are limited, resulting in frequent system downtime and affecting production efficiency.
By separating the convolution kernel from the CNN and using the dynamic partial reconstruction (DPR) method, the convolution kernel is partially configured on the reconfigurable partition of the FPGA, and the rapid online adjustment and reconstruction of the convolutional neural network on the FPGA is realized.
The rapid online adjustment and reconstruction of convolutional neural networks on FPGAs is realized, which avoids system downtime, improves production efficiency, and reduces hardware costs and power consumption.
Smart Images
Figure CN119918599A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the application of deep learning technology in the field of intelligent manufacturing, and specifically refers to a design method for an FPGA hardware acceleration platform with a convolution kernel separated from a CNN. Background Art
[0002] At present, driven by intelligent manufacturing, my country's manufacturing industry is in a critical stage of rapid development and upgrading and transformation. In this process, artificial intelligence, especially deep learning technology, has become the core force leading the transformation of China's manufacturing industry. Deep learning covers multiple branch technologies and application fields such as vision and voice. Its integration with the manufacturing industry aims to improve production quality and efficiency and reduce production costs.
[0003] Among many deep learning applications, using convolutional neural network (CNN) algorithms to achieve image classification and target detection is one of the most common AI applications. Due to the programmable and reconfigurable features of FPGA (field programmable gate array), it has become a common choice for CNN end-side reasoning hardware acceleration. FPGA hardware acceleration uses a data-driven customized CNN accelerator to design a special data flow strategy based on the data characteristics of different convolutional networks, thereby achieving better acceleration effects under specific convolutional neural networks, effectively solving problems such as insufficient computing power of the end-side reasoning platform and slow target recognition speed.
[0004] However, the most commonly used reconfiguration method for FPGAs is static reconfiguration. This method requires reloading the configuration information stored in the external memory to the FPGA under the control of the main processor, thereby changing the circuit structure and function composed of the FPGA reconfigurable logic devices. Figure 1 As shown in the figure, before the FPGA is powered on, the configuration file must be loaded into the FPGA. After powering on, the FPGA will configure all resources to make it a circuit with set functions. In this configuration method, the logic circuit function implemented internally remains unchanged during the operation of the FPGA until the operation ends. If the logic function of the circuit is to be changed, the FPGA must be powered off and then a new configuration file must be downloaded again. It can be seen that the FPGA static reconstruction method does not support real-time changes in circuit functions according to actual needs during operation.
[0005] In the application scenario of intelligent manufacturing, CNN hardware acceleration faces many special challenges. For example, the production of various parts on the production line usually presents the characteristics of small batches, and there are many types of materials and a large range of material sizes. These characteristics often lead to the following difficult-to-solve technical problems when using FPGA for CNN hardware acceleration:
[0006] Technical problem ①: Due to the complexity and variety of inference scenarios, different modules need to be deployed on the resource-limited FPGA end-side platform to adapt to different scenarios and requirements. This requires static reconstruction (also known as reconfiguration) of the FPGA, which must enter the configuration mode to download the switched modules into the FPGA. However, this operation will cause frequent system downtime, seriously affecting the production cycle.
[0007] Technical issue ②: Given that the AI reasoning system needs to continuously optimize the algorithm, perform iterations and upgrades. However, to update the algorithm model in the entire FPGA, the FPGA must be powered off, and then the configuration file of the corresponding function must be downloaded to the FPGA again to realize the new circuit function. In this way, each algorithm upgrade will inevitably lead to downtime, which will also affect production line production.
[0008] In summary, the core of these two technical problems is how to achieve online reconstruction without affecting the normal operation of CNN when using FPGA as a CNN hardware acceleration platform. In response to these problems, the present invention conducts in-depth analysis from two aspects: CNN working mechanism and FPGA reconstruction, and proposes targeted strategies and measures. Summary of the invention
[0009] The present invention aims to solve the above technical problems and provide a design method for an FPGA hardware acceleration platform with a convolution kernel separated from a CNN. Through a design method for dynamic partial reconstruction of the convolution kernel, rapid online adjustment and reconstruction of the convolution neural network on the FPGA is achieved, thereby adapting to the needs of identifying different types of targets and improving the flexibility and adaptability of the convolution neural network FPGA hardware acceleration system.
[0010] In order to solve the above technical problems, the technical solution provided by the present invention is: a design method of an FPGA hardware acceleration platform with a convolution kernel and a CNN separated, comprising the following steps:
[0011] S1: Analysis of the convolutional neural network operation process:
[0012] Analyze the operation process of the convolutional neural network and find out the parts that are constantly iterating and changing;
[0013] In the process of target recognition, taking two-dimensional convolution as an example, the convolution input is clearly defined as a feature map matrix representing an image, and the convolution kernel is presented in the form of a numerical matrix to represent target features, such as target edges, corners, and textures;
[0014] It is determined that each CNN training is a process of continuously optimizing and adjusting the convolution kernel until the loss function meets the predetermined requirements. In the above process, except for the constantly changing convolution kernel, the other parts of CNN remain unchanged. Therefore, it is necessary to separate the constantly changing convolution kernel from the overall CNN, divide it into the convolution kernel part and other parts, and implement it in two parts in FPGA.
[0015] Furthermore, in the step of separating the convolution kernel from the overall CNN, for different input feature maps and convolution kernel dimensions, if the input channels are 3 and the size of each input feature map is 6×6, then the input dimension is [3,6,6], there are N convolution kernels, and the size of each convolution kernel is 3×3, then the convolution kernel dimension is [N,3,3,3]. It is clear that the convolution operation is to perform a convolution operation on each convolution kernel on the input feature map to obtain the output feature map. Figure 1 ~N, each output feature map is 4×4 in size, and then the output feature maps of these N channels are superimposed as the output feature map of this convolution, with a dimension of [3,4,4].
[0016] Also includes:
[0017] S2: Using the Dynamic Partial Reconstruction (DPR) method:
[0018] The convolution kernel part is configured in the FPGA reconfigurable partition, and the rest of the CNN is configured in the FPGA static configuration area;
[0019] Based on the dynamic partial reconfiguration (DPR) method, one or more areas of the specified PL (programmable logic) can be reconfigured at any time during runtime. These areas are designated as reconfigurable partitions (RP), and a set of reconfigurable modules RM (a set of bitstream fragments) is designed for each RP. DPR is implemented under the control of software applications running on the PS (processing system). That is, during the normal operation of other parts of the PL, the functions of some RP areas are changed and switched to different RMs without affecting any other normal working areas, thereby realizing time division multiplexing of different functional modules RM in specific RP areas and system online reconfiguration.
[0020] Furthermore, when the dynamic partial reconstruction (DPR)-based method is used to realize the reconfiguration of the specified PL area, the specific content and structure of the bitstream fragment of the reconfigurable module RM designed for each reconfigurable partition (RP) are clarified to ensure the accuracy and stability of the RP area function switching under the control of the PS.
[0021] Furthermore, in the process of implementing DPR under the control of the software application running on the PS, the trigger conditions and monitoring mechanism of the software application for switching the RP area function are set to ensure that when other parts of the PL are working normally, the change of the RP area function does not affect other areas and can accurately realize time division multiplexing and system online reconstruction.
[0022] Also includes:
[0023] S3: FPGA hardware acceleration system running convolution kernel and CNN separately:
[0024] In the configuration stage, the convolution kernel is configured separately from the rest of the CNN. The convolution kernel is configured in multiple reconfigurable partitions to facilitate later iteration and optimization, and the rest of the CNN is configured in the static configuration area.
[0025] During runtime, the convolution kernel is connected to the CNN part for convolution operation. When the convolution kernel of a reconfigurable area is iteratively reconstructed, other reconfigurable areas and static configuration areas are not affected, and normal operation can still be maintained, ensuring that the system does not shut down during the iteration process.
[0026] Furthermore, during the configuration phase, a specific allocation strategy of the convolution kernels in the multiple reconfigurable partitions and a layout of the remaining parts of the CNN in the static configuration area are determined to improve the overall operating efficiency of the system.
[0027] Furthermore, during the operation, a communication mechanism is established between the convolution kernel and the CNN part to ensure the accuracy and efficiency of the convolution operation data transmission, and at the same time, specific isolation and synchronization measures are clearly defined to ensure that the normal operation of other areas is not affected when the convolution kernel of a certain reconfigurable area is iteratively reconstructed.
[0028] This method is not limited to the separation of convolution kernels and CNNs, but is also used in scenarios where different modules need to be deployed on resource-limited FPGA-side platforms to achieve time-division multiplexing. It provides general rules and methods for module division, configuration, and operation for different module deployment scenarios.
[0029] An FPGA hardware acceleration platform constructed by a design method of an FPGA hardware acceleration platform with a convolution kernel separated from a CNN comprises an FPGA chip, wherein the FPGA chip is provided with a reconfigurable partition for configuring the convolution kernel and a static configuration area for configuring the rest of the CNN, and is provided with a control module for realizing the switching of the reconfigurable partition function based on a dynamic partial reconstruction (DPR) method, and a communication module for realizing partial connection and data interaction between the convolution kernel and the CNN.
[0030] The advantages of the present invention compared with the prior art are:
[0031] The FPGA hardware acceleration platform design method for separating the convolution kernel and CNN based on dynamic partial reconstruction technology and the platform constructed by it show significant advantages in many aspects, making it more adaptable in application scenarios such as industrial production lines.
[0032] 1. Smaller chips: Through innovative algorithm separation design and DPR technology, time-division multiplexing of different functional modules in the same reconfigurable area has been successfully achieved. This multiplexing method allows the selection of smaller chips while meeting the functional requirements of the system. Compared with traditional designs, there is no need to use large chips to accommodate all functional modules, thereby effectively reducing the physical size of the chip. This not only has advantages in hardware costs, but also has higher practicality in space-constrained application scenarios, such as miniaturized equipment in industrial production lines.
[0033] 2. Richer functions: Thanks to the algorithm separation design and DPR technology, the number of dynamically configured functional modules is no longer strictly limited when PL resource conditions permit. This means that the system can achieve richer and more diverse functions. For example, in the production line inspection scenario of intelligent manufacturing, a variety of modules with different functions can be dynamically loaded according to different inspection tasks, such as defect detection modules and dimension measurement modules for different parts, which greatly expands the application scope and functional flexibility of the system.
[0034] 3. Lower power consumption: On the one hand, algorithm separation design and DPR technology make time-division multiplexing of modules and functions possible, so that smaller chips can be selected. Smaller chips usually have lower system static power consumption because their internal circuit scale is relatively small and the static current consumption is lower. On the other hand, since there is no need to implement a large number of functional modules at the same time, the high power consumption state when multiple modules are running at the same time is avoided, thereby reducing the dynamic power consumption of the system. This low power consumption feature is of great significance in industrial production lines. It can not only reduce energy costs, but also reduce heat dissipation requirements and improve system stability and reliability.
[0035] 4. Lower cost: The use of algorithm separation design and DPR technology, and the use of smaller chips directly reduce the hardware procurement cost. At the same time, the reduction in power consumption also means the reduction of operating costs, which saves a lot of operating funds for enterprises in the long run. In addition, due to the enhanced flexibility of system functions, the cost of purchasing multiple devices to meet different needs is reduced, further improving cost-effectiveness.
[0036] 5. It is more convenient to add new functions: Through algorithm separation design and DPR technology, when new functions need to be added, they can be directly imported in the form of bitstream fragment files. This method does not require re-design of complex PL hardware, which greatly shortens the development cycle and cost of new functions. For example, when upgrading the production line or introducing new detection tasks, engineers can quickly develop new functional modules and easily load them into the FPGA in the form of bitstream files to achieve rapid expansion and upgrade of system functions. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 It is the flow chart of FPGA static reconstruction and operation function circuit.
[0038] Figure 2 It is a schematic diagram of the convolution operation process in a convolutional neural network.
[0039] Figure 3 This is a schematic diagram of the reconstruction of part of the FPGA area where the convolution kernel and CNN are designed separately.
[0040] Figure 4 It is a schematic diagram of the FPGA dynamic partial area reconstruction process with the convolution kernel and CNN designed separately. DETAILED DESCRIPTION
[0041] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0042] 1. Working principle of the present invention:
[0043] 1.1 Design method of FPGA hardware acceleration platform with separation of convolution kernel and CNN
[0044] S1: Analysis of the convolutional neural network operation process
[0045] Find out the parts that are constantly iterating and changing: Deeply analyze the computational process of the convolutional neural network and accurately identify the parts that are constantly iterating and changing.
[0046] Clarify the convolution input and convolution kernel: In the target recognition process, taking two-dimensional convolution as a specific example, it is clearly defined that the convolution input is the feature map matrix representing the image, while the convolution kernel is presented in the form of a numerical matrix. Its main function is to represent various features of the target, such as the edge, corners, and texture of the target.
[0047] Determine the optimization process and separation of the convolution kernel: It is clear that each CNN training is essentially a process of continuous optimization and adjustment of the convolution kernel until the loss function meets the pre-set requirements. Based on this, the convolution kernel that is always in a changing state is separated from the overall CNN, divided into the convolution kernel part and other parts, and the two parts are implemented separately in the FPGA.
[0048] Clarify the convolution operation for different dimensions: In the step of separating the convolution kernel from the overall CNN, for different input feature maps and convolution kernel dimensions, for example, if the input channels are 3 and the size of each input feature map is 6×6, then the input dimension is [3,6,6], there are N convolution kernels, and the size of each convolution kernel is 3×3, then the convolution kernel dimension is [N,3,3,3]. Clarify the convolution operation that each convolution kernel performs a convolution operation on the input feature map to obtain the output feature map. Figure 1 ~N, each output feature map has a size of 4×4, and then the output feature maps of these N channels are superimposed as the output feature map of this convolution, with a dimension of [3,4,4]. This process is as follows Figure 2 As shown in the figure, the detailed analysis of the convolution operation provides a solid theoretical foundation for the subsequent separation design.
[0049] S2: Using Dynamic Partial Reconstruction (DPR) method
[0050] Regional configuration: The convolution kernel part is carefully configured in the reconfigurable partition of the FPGA, while the rest of the CNN is configured in the static configuration area of the FPGA. This configuration method lays the foundation for dynamic partial reconstruction.
[0051] Achieve reconfigurable and related designs: Based on the dynamic partial reconfiguration (DPR) method, one or more areas of the specified PL (programmable logic) are reconfigurable at runtime. These areas are explicitly designated as reconfigurable partitions (RP), and a set of reconfigurable modules RM (i.e., a set of bitstream fragments) is tailored for each RP. DPR is implemented by running software applications on the PS (processing system) to implement control. Specifically, while other parts of the PL are working normally, the function of the RP area can be completely changed, that is, the switching of different RMs is achieved, and this process will not affect any other normal working areas. In this way, time division multiplexing of different functional modules RM in a specific RP area and online reconstruction of the system are achieved.
[0052] Ensure the accuracy and stability of RP area function switching: When implementing the reconfiguration of a specified PL area based on the dynamic partial reconstruction (DPR) method, it is important to clarify the specific content and structure of the bitstream fragment of the reconfigurable module RM designed for each reconfigurable partition (RP). By accurately designing the bitstream fragment, it is ensured that under the precise control of the PS, the RP area function switching can be carried out accurately, stably and reliably. This design point ensures that during the operation of the system, the reconfigurable partition can realize flexible switching of functions as expected, avoiding functional disorder or instability caused by unreasonable bitstream fragment design.
[0053] Setting trigger conditions and monitoring mechanisms: When running software applications on PS to control DPR, carefully set the trigger conditions and monitoring mechanisms for software applications to switch RP regional functions. By setting trigger conditions reasonably, it is ensured that the RP regional functions are changed only when specific requirements are met, avoiding unnecessary function switching from interfering with the system. At the same time, by establishing an effective monitoring mechanism, the impact of RP regional function changes on other regions is monitored in real time, ensuring that, under the premise that other parts of the PL are working normally, RP regional function changes will not only not affect other regions, but also accurately realize time division multiplexing and online reconstruction of the system. The setting of this mechanism further improves the stability and reliability of system operation, and ensures that the entire system can continue to operate stably in a complex and changing working environment.
[0054] S3: FPGA hardware acceleration system running convolution kernels and CNN separately
[0055] Configuration phase: In the configuration phase, the convolution kernel is separated from the rest of the CNN. Specifically, the convolution kernel is configured in multiple reconfigurable partitions. This configuration method facilitates subsequent iteration and optimization of the convolution kernel. At the same time, the rest of the CNN is configured in the static configuration area. Through reasonable separation configuration, the flexibility and efficiency of the system during operation are effectively guaranteed.
[0056] Determine the configuration strategy and layout method: In the configuration stage, further determine the specific allocation strategy of the convolution kernel in multiple reconfigurable partitions, as well as the layout of the rest of the CNN in the static configuration area. By scientifically and rationally planning the allocation strategy of the convolution kernel and the layout of the rest of the CNN, the overall operating efficiency of the system can be effectively improved. For example, according to the different characteristics and usage frequency of the convolution kernel, it is allocated to the most appropriate reconfigurable partition to reduce data transmission delays and resource competition; at the same time, the rest of the CNN is optimized in the static configuration area to ensure smooth communication between modules, thereby improving the operating performance of the entire system.
[0057] Runtime connection and operation: At runtime, the convolution kernel is connected to the CNN part to perform the convolution operation. It is worth emphasizing that when the convolution kernel of a reconfigurable area is iteratively reconstructed, the operation will not affect other reconfigurable areas and static configuration areas, which can still maintain normal working state, which ensures that the system will not stop during the iteration process.
[0058] Establish communication mechanisms and isolation and synchronization measures: During operation, establish a communication mechanism between the convolution kernel and the CNN part. By designing efficient communication protocols and data transmission channels, ensure the accuracy and efficiency of data transmission during the convolution operation. At the same time, clarify the specific isolation and synchronization measures that do not affect the normal operation of other areas when iteratively reconstructing the convolution kernel of a reconfigurable area. For example, use data buffering and synchronization signal mechanisms to ensure that the data processing in other areas is not disturbed while ensuring the smooth iterative reconstruction of the convolution kernel in the reconfigurable area, so as to maintain the overall stable operation of the system.
[0059] General rules and methods: This method is not only limited to the separation of convolution kernels and CNNs, but is also widely applicable to scenarios where different modules are deployed on resource-limited FPGA-side platforms, in order to achieve time-division multiplexing. For different module deployment scenarios, the present invention provides a set of general rules and methods for module division, configuration, and operation. These general rules and methods can help users flexibly divide, accurately configure, and efficiently operate different modules according to specific application requirements and FPGA resource conditions, thereby giving full play to the advantages of FPGA and meeting the needs of diverse application scenarios.
[0060] 1.2 FPGA hardware acceleration platform built based on the above method
[0061] An FPGA hardware acceleration platform constructed based on the above-mentioned FPGA hardware acceleration platform design method for separating convolution kernels from CNNs, the core component of which is an FPGA chip. Inside the FPGA chip, a reconfigurable partition specifically used to configure the convolution kernel and a static configuration area for configuring the rest of the CNN are provided. At the same time, a control module is provided for implementing the function switching of the reconfigurable partition based on the dynamic partial reconstruction (DPR) method. The control module can accurately control the function switching of the reconfigurable partition according to the preset trigger conditions and monitoring mechanism to ensure the stability and flexibility of the system. In addition, a communication module for realizing the connection and data interaction between the convolution kernel and the CNN part is also provided. Through the communication module, the efficiency and accuracy of the data transmission between the convolution kernel and the CNN part during operation can be guaranteed, thereby realizing the stable operation and efficient processing capability of the entire FPGA hardware acceleration platform. 2. Specific implementation methods:
[0063] 2.1 Analysis of the convolutional neural network operation process
[0064] 2.11 Analysis of the operation process: Taking product appearance inspection in intelligent manufacturing as an example, when performing image recognition tasks, the input product image is first preprocessed and converted into a feature map matrix suitable for convolutional neural network processing. Using the visualization tools of deep learning frameworks (such as TensorFlow or PyTorch), the operation process of each layer of the convolutional neural network is deeply analyzed. It is observed that in the convolution layer, the input feature map is convolved with the convolution kernel to extract features in the image. For example, when detecting scratches on the surface of a product, the convolution kernel will be sensitive to features such as edges and lines in the image. Through multiple experiments and analysis of the parameter changes of different convolution kernels during the training process, it is determined that the convolution kernel is the part that is constantly iterating and changing in the entire operation process.
[0065] 2.12 Detailed explanation of convolution operation: Assume that the dimension of the input feature map is [3,224,224], which means that the image has 3 color channels (such as RGB) and the size is 224×224 pixels. The number of convolution kernels is set to 64, and the dimension is [64,3,3,3], that is, each convolution kernel has 3 channels and the size is 3×3. When performing the convolution operation, each convolution kernel slides on the input feature map, and the elements at the corresponding positions are multiplied and summed to obtain a value on the output feature map. Each convolution kernel operates independently to generate 64 output feature maps. Then these 64 output feature maps are superimposed to obtain the final convolution output feature map, whose dimension may be [64,222,222] (assuming the step size is 1 and no padding). Through detailed simulation and calculation of this process, the specific steps and data flow of the convolution operation are clarified.
[0066] 2.13 Convolution kernel separation: Based on the above analysis, the convolution kernel module and other parts of CNN are designed separately using hardware description language (such as Verilog). Independent input and output interfaces are designed for the convolution kernel module so that it can easily interact with other modules. For example, a parameterized convolution kernel module is designed that can adapt to convolution kernels of different sizes and numbers of channels by changing parameters. At the same time, other parts of CNN, such as pooling layers and fully connected layers, are modularly designed to ensure that the interfaces between modules are clear and easy to layout and connect on FPGA.
[0067] 2.2 Dynamic Partial Reconstruction (DPR) method
[0068] 2.21FPGA resource planning: Choose a suitable FPGA chip, such as Xilinx's Zynq series. According to the chip's data sheet and the resource analysis function provided by the development tool, make a detailed plan for the programmable logic part (PL) of the FPGA. Divide the PL area into multiple reconfigurable partitions (RP) and a static configuration area. For example, about 30% of the PL resources are allocated to the reconfigurable partition to place the convolution kernel module; the remaining 70% of the resources are used as the static configuration area to implement other fixed parts of CNN. This resource allocation method can not only meet the needs of dynamic reconstruction of the convolution kernel, but also ensure the stable operation of other parts of CNN.
[0069] 2.22 Reconfigurable module design and implementation: For each reconfigurable partition, design a reconfigurable module RM for the convolution kernel. Take a reconfigurable module of a 3×3 convolution kernel as an example, and use Verilog language to describe its logical structure. The module includes components such as multipliers, adders, and registers to implement convolution operations. These components are connected according to the requirements of the convolution algorithm, and their working modes are controlled by configuration signals. For example, different convolution kernel parameters (such as weight values) can be selected through a set of configuration signals. Compile these logical descriptions to generate corresponding bitstream fragments, which constitute the reconfigurable module RM. Design a unique identifier for each reconfigurable module RM so that it can be accurately selected and loaded at runtime.
[0070] 2.23DPR control software writing: On the processing system (PS), use C language or Python language to write software applications to control DPR. First, establish a connection through the communication interface (such as AXI bus) between PS and PL to ensure that configuration data and control instructions can be sent to FPGA. In the software, set the trigger conditions for function switching. For example, when a change in product type is detected and a different convolution kernel is required for feature extraction, the function switching of the reconfigurable partition is triggered. At the same time, write a monitoring program to monitor the working status and data transmission of each module inside the FPGA in real time. For example, by reading the status register inside the FPGA, it is determined whether the reconfigurable partition is undergoing a reconstruction operation and whether the reconstruction process has been successfully completed. If an abnormal situation is found, take appropriate measures in time, such as resending the configuration data or suspending the system operation.
[0071] 2.3 FPGA hardware acceleration system with separate convolution kernel and CNN
[0072] 2.31 Configuration phase operations: In the configuration phase, use FPGA development tools (such as Xilinx's Vivado) to configure the convolution kernel module into multiple reconfigurable partitions. According to the function and frequency of use of the convolution kernel, the reconfigurable partitions are reasonably allocated. For example, the convolution kernel used to detect the edge features of the product is allocated to a reconfigurable partition, and the convolution kernel used to detect the texture features of the product is allocated to another reconfigurable partition. At the same time, other parts of CNN, such as the pooling layer and the fully connected layer, are configured to the static configuration area. During the configuration process, carefully set the parameters and connection relationships of each module to ensure that the system can start normally. For example, set the data transmission channel between the convolution kernel module and the pooling layer to ensure that the convolution output feature map can be smoothly passed to the pooling layer for the next round of processing.
[0073] 2.32 Runtime operation and management: When the system is running, a communication mechanism is established between the convolution kernel and other parts of the CNN. By designing a dedicated bus interface and data buffer, data can be efficiently and accurately transmitted between different modules. For example, a FIFO (first-in-first-out) buffer is set between the convolution kernel module and the pooling layer to temporarily store the convolution output feature map to avoid data loss or conflict. When it is necessary to iteratively reconstruct the convolution kernel of a certain reconfigurable area, the reconstruction instruction and the new convolution kernel configuration data are first sent through the PS. During the reconstruction process, the isolation mechanism inside the FPGA is used to ensure that the normal operation of other reconfigurable areas and static configuration areas is not affected. For example, the data connection between the reconfigurable partition being reconstructed and other modules is temporarily cut off through the control signal, while the clock signal of other modules is kept running normally. After the reconstruction is completed, the data connection is reestablished to restore the normal operation of the system. During the entire operation process, the system performance indicators such as processing speed and accuracy are continuously monitored, and the system is optimized and adjusted according to the actual situation. For example, if it is found that the convolution kernel of a certain reconfigurable partition is less efficient in processing a specific type of product image, the performance can be improved by reconfiguring the convolution kernel parameters or adjusting its position in the reconfigurable partition.
[0074] 2.4 General Implementation of Multi-Module Deployment
[0075] 2.41 Module division: When faced with a scenario where different modules need to be deployed on a resource-limited FPGA-side platform, first analyze the functional requirements of the system in detail. For example, in a multifunctional industrial detection system, it may be necessary to deploy image recognition modules, data processing modules, and communication modules. According to the characteristics and interrelationships of each functional module, it is divided into different functional units. For the image recognition module, it can be further subdivided into convolutional neural network modules, feature extraction modules, etc.; the data processing module can include data filtering modules, data analysis modules, etc. Ensure that each module has clear functional boundaries to facilitate subsequent configuration and management.
[0076] 2.42 Configuration strategy: Develop a reasonable configuration strategy based on the FPGA's resource situation and the resource requirements of each module. For modules with large and relatively stable resource requirements, such as communication modules, configure them to the static configuration area to ensure their stable operation. For modules that need to be dynamically adjusted according to different tasks, such as convolution kernel modules in convolutional neural networks, configure them to reconfigurable partitions. In the reconfigurable partitions, further optimize the configuration based on the frequency of use and relevance of the modules. For example, if there are multiple convolution kernel modules with different functions, modules with higher frequency of use and related functions will be assigned to adjacent reconfigurable partitions to reduce data transmission delays.
[0077] 2.43 Operation management: When the system is running, a unified scheduling mechanism is established to coordinate and manage the operation of different modules. For example, when a new detection task is received, the scheduling mechanism determines the module that needs to be started according to the task type, and reasonably arranges the operation order and data interaction process of each module. At the same time, the operation status and resource usage of each module are monitored in real time, and when a module is found to have resource bottlenecks or performance problems, timely adjustments are made. For example, if the convolution kernel module has a slow processing speed when processing a large amount of data, the scheduling mechanism can temporarily reduce the resource usage of other modules and give priority to ensuring the operation efficiency of the convolution kernel module. In this way, efficient operation and time-division multiplexing of different modules on the resource-limited FPGA end-side platform can be achieved.
[0078] The present invention and its embodiments are described above, and such description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it, without departing from the purpose of the invention, they can design a structure and embodiment similar to the technical solution without creativity, which should belong to the protection scope of the present invention.
Claims
1. A design method for an FPGA hardware acceleration platform with separated convolution kernel and CNN, characterized by: The following steps are involved: S1: Analysis of the convolutional neural network operation process: Analyze the operation process of the convolutional neural network and find out the parts that are constantly iterating and changing; In the process of target recognition, taking two-dimensional convolution as an example, the convolution input is clearly defined as a feature map matrix representing an image, and the convolution kernel is presented in the form of a numerical matrix to represent target features, such as target edges, corners, and textures; It is determined that each CNN training is a process of continuously optimizing and adjusting the convolution kernel until the loss function meets the predetermined requirements; in the above process, except for the constantly changing convolution kernel, the other parts of CNN remain unchanged. Therefore, it is necessary to separate the constantly changing convolution kernel from the overall CNN, divide it into the convolution kernel part and other parts, and implement it in two parts in FPGA.
2. The method for designing an FPGA hardware acceleration platform for separating convolution kernels from CNN according to claim 1, characterized in that: In the step of separating the convolution kernel from the overall CNN, for different input feature maps and convolution kernel dimensions, if the input channels are 3 and the size of each input feature map is 6×6, then the input dimension is [3,6,6], there are N convolution kernels, and the size of each convolution kernel is 3×3, then the convolution kernel dimension is [N,3,3,3]. It is clear that the convolution operation is that each convolution kernel performs a convolution operation on the input feature map to obtain output feature maps 1 to N, each of which is 4×4 in size, and then the output feature maps of these N channels are superimposed as the output feature map of this convolution, with a dimension of [3,4,4].
3. The method for designing an FPGA hardware acceleration platform for separating convolution kernel and CNN according to claim 1, characterized in that: Also includes: S2: Using the Dynamic Partial Reconstruction (DPR) method: The convolution kernel part is configured in the FPGA reconfigurable partition, and the rest of the CNN is configured in the FPGA static configuration area; Based on the dynamic partial reconfiguration (DPR) method, one or more areas of the specified PL (programmable logic) can be reconfigured at any time during runtime. These areas are designated as reconfigurable partitions (RP), and a set of reconfigurable modules RM (a set of bitstream fragments) is designed for each RP. DPR is implemented under the control of software applications running on the PS (processing system). That is, during the normal operation of other parts of the PL, the functions of some RP areas are changed and switched to different RMs without affecting other normal working areas, thereby realizing time division multiplexing of different functional modules RM in specific RP areas and system online reconfiguration.
4. The method for designing an FPGA hardware acceleration platform for separating convolution kernels from CNN according to claim 3, characterized in that: When the dynamic partial reconstruction (DPR)-based method is used to realize the reconfiguration of the specified PL area, the specific content and structure of the bitstream fragment of the reconfigurable module RM designed for each reconfigurable partition (RP) are clarified to ensure the accuracy and stability of the RP area function switching under the control of the PS.
5. The method for designing an FPGA hardware acceleration platform for separating convolution kernels from CNN according to claim 3, characterized in that: In the process of implementing DPR under the control of the software application running on the PS, the trigger conditions and monitoring mechanism of the software application for switching the RP area function are set to ensure that when other parts of the PL are working normally, the change of the RP area function does not affect other areas and can accurately realize time division multiplexing and system online reconstruction.
6. The method for designing an FPGA hardware acceleration platform for separating convolution kernels from CNN according to claim 1, characterized in that: Also includes: S3: FPGA hardware acceleration system running convolution kernel and CNN separately: In the configuration stage, the convolution kernel is configured separately from the rest of the CNN. The convolution kernel is configured in multiple reconfigurable partitions to facilitate later iteration and optimization, and the rest of the CNN is configured in the static configuration area. During runtime, the convolution kernel is connected to the CNN part for convolution operation. When the convolution kernel of a reconfigurable area is iteratively reconstructed, other reconfigurable areas and static configuration areas are not affected, and normal operation can still be maintained, ensuring that the system does not shut down during the iteration process.
7. The method for designing an FPGA hardware acceleration platform for separating convolution kernels from CNN according to claim 6, characterized in that: In the configuration phase, the specific allocation strategy of the convolution kernels in the multiple reconfigurable partitions and the layout of the rest of the CNN in the static configuration area are determined to improve the overall operating efficiency of the system.
8. The method for designing an FPGA hardware acceleration platform for separating convolution kernels from CNN according to claim 6, characterized in that: During the operation, a communication mechanism is established between the convolution kernel and the CNN part to ensure the accuracy and efficiency of the convolution operation data transmission, and at the same time, specific isolation and synchronization measures are clearly defined to ensure that the normal operation of other areas is not affected when iteratively reconstructing the convolution kernel of a certain reconfigurable area.
9. The method for designing an FPGA hardware acceleration platform for separating convolution kernels from CNN according to any one of claims 1 to 8, characterized in that: This method is not limited to the separation of convolution kernels and CNNs, but is also used in scenarios where different modules need to be deployed on resource-limited FPGA-side platforms to achieve time-division multiplexing. It provides general rules and methods for module division, configuration, and operation for different module deployment scenarios.
10. An FPGA hardware acceleration platform constructed based on the FPGA hardware acceleration platform design method for separating convolution kernel and CNN according to any one of claims 1 to 9, characterized in that: The invention comprises an FPGA chip, wherein the FPGA chip is provided with a reconfigurable partition for configuring a convolution kernel and a static configuration area for configuring the rest of a CNN, and is provided with a control module for realizing function switching of the reconfigurable partition based on a dynamic partial reconstruction (DPR) method, and a communication module for realizing partial connection and data interaction between the convolution kernel and the CNN.