A real-time image defogging method and device based on FPGA convolutional neural network

By using the quantized convolutional neural network model on the FPGA platform for image processing, the problem of high power consumption of GPU in edge-side image processing is solved, real-time fog removal and efficient processing of images are realized, and drone and ground are improved Image processing accuracy of unmanned platforms.

CN118761938BActive Publication Date: 2024-12-03CHINA ORDNANCE EQUIP GRP AUTOMATION RES INST CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411254637.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-09
Publication Date
2024-12-03
Estimated Expiration
2044-09-09

AI Technical Summary

Technical Problem

In the edge-end image processing, due to the high power consumption and high cost of the GPU computing power platform, it is difficult to provide a highly reliable and stable image real-time defog solution, which affects the image processing accuracy of drones and ground unmanned platforms.

Method used

Using a convolutional neural network based on FPGA, the floating-point data training and integer quantization are used on the GPU platform, the floating-point number weight is mapped to the integer weight and stored in the FPGA RAM resource to realize the multi-scale features of the image. Extraction, feature fusion and feature learning, output defogging images.

Benefits of technology

Real-time defog processing of images is realized on drones and ground unmanned platforms, making full use of the advantages of low power consumption, miniaturization and parallel processing of FPGAs, reducing computing costs and power consumption, and improving the reliability of image processing and stability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118761938B_ABST
    Figure CN118761938B_ABST
Patent Text Reader

Abstract

The present invention discloses a real-time image defogging method and device based on FPGA convolutional neural network. Through this method, foggy images are input into the convolutional neural network model loaded on the FPGA. The model has completed network pre-training on the server-side GPU platform through a large number of pre-collected labeled data sets. The model parameters are integer quantized and tested and stored in the on-chip resources of the FPGA. When the FPGA performs convolutional neural network processing, multi-scale feature extraction, feature fusion and feature learning can be performed on the foggy image, and the output result is the defogging image. This method can give full play to the advantages of FPGA low power consumption, miniaturization, pipeline design and parallel processing, and realize real-time image defogging based on convolutional neural network at the edge end such as drones and ground unmanned platforms.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of embedded image processing technology, and in particular to a real-time image defogging method and device based on FPGA convolutional neural network. Background Art

[0002] With the continuous development of science and technology, drones, ground unmanned platforms, etc. have been widely used in various fields of industry and life, and edge visual image processing technology has developed rapidly. However, in outdoor scenes, unmanned platforms are seriously affected by low light illumination, fog, haze, etc., and the acquired images will degrade to varying degrees, manifested as reduced image contrast, color distortion, increased noise, and loss of detail information, which reduces the quality and usability of the image and greatly affects the accuracy of high-level visual processing tasks. In order to solve these problems, the image can be dehazed to obtain a clearer image.

[0003] Image dehazing technology is an image processing technology that aims to extract clear images from fogged images. In many applications, such as unmanned driving, aerial photography, and surveillance, it is very important to obtain clear images.

[0004] There are currently three main research directions for the dehazing problem:

[0005] The first is to use image enhancement algorithms to improve the overall contrast of the image, obtain richer detail information, simulate the human eye visual system to correct the scene color, and complete image dehazing.

[0006] The second is to restore the image by studying atmospheric physics models and using prior knowledge.

[0007] The third is based on deep learning. Through a large number of pre-collected labeled data sets, the mapping relationship between foggy degraded images and clear images is directly or indirectly fitted, and the image defogging is completed using a neural network.

[0008] Compared with the other two methods, the image dehazing effect based on neural network is better. However, due to the large number of parameters of convolutional neural network, most of the processing methods and means are implemented on the GPU platform. GPU is a processor specially used for graphics rendering. Compared with the traditional central processing unit (CPU), it has obvious advantages in parallel computing capabilities. The GPU computing power platform uses the parallel computing characteristics of GPU to provide efficient computing power and processing speed for various applications. Whether it is scientific computing, data analysis or game development, the GPU computing power platform can greatly improve computing efficiency and processing capabilities.

[0009] Although GPU has performance advantages in image processing and faster processing speed in more complex and large-scale parallel computing tasks, its power consumption is relatively high and it is expensive. The reliability, stability and environmental adaptability of edge GPU computing platforms in engineering integration applications still have many problems.

[0010] Therefore, how to provide a highly reliable and stable real-time image defogging device to reduce power consumption and computing costs is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the invention

[0011] In view of the above problems, the present invention provides a real-time image defogging method and device based on FPGA convolutional neural network for overcoming the above problems or at least partially solving the above problems.

[0012] The present invention provides the following scheme:

[0013] A real-time image defogging method based on FPGA convolutional neural network, comprising:

[0014] Obtain the target input feature map and target convolution kernel network parameters required for each convolution layer in the trained target convolution neural network; the target input feature map is stored in the display memory module of the FPGA peripheral support device, and the target convolution kernel network parameters are stored in the RAM resources of the FPGA device, and the target convolution kernel network parameters are integer network parameters obtained after integer quantization;

[0015] Inputting the target input feature map and the target convolution kernel network parameters into the corresponding convolution layer in the target convolution neural network for convolution operation processing to obtain a defogging image;

[0016] Among them, the target convolutional neural network only contains several convolutional layers; the integer quantization includes performing floating-point data training on a GPU platform to obtain the optimal network model weights, mapping the floating-point weights to integer weights through a quantized network, loading the mapped integer weights as pre-trained weights into the network model of integer parameters, testing and evaluating the integer quantization results, and obtaining the integer network parameters.

[0017] Preferably: in one convolution operation, all parameters are multiplied in parallel with corresponding data, and then all multiplication results are summed.

[0018] Preferably: multiple layers of feature maps in the current convolutional layer are read to perform parallel calculations on the same convolution kernel of the current layer.

[0019] Preferably: the target convolutional neural network includes a pipeline structure in which row delay is aligned with the computing cycle.

[0020] Preferably: the target convolutional neural network combines low-dimensional information with high-dimensional abstract information by using feature concatenation and feature addition to perform feature fusion of different layers; several layers of the convolutional layers are connected in series; the output feature map size of each convolutional layer is the same as the input feature map size.

[0021] Preferably: the 7th convolution layer and the 8th convolution layer use two groups of 3×3 convolution kernels to achieve feature learning.

[0022] Preferably: the target input feature map required by the first convolution layer is the original image, and the first convolution layer uses multiple groups of convolution kernels of different sizes to perform multi-scale feature extraction on the original image.

[0023] Preferably: the target input feature map required by the convolutional layer other than the first convolutional layer is the output feature map of the convolutional layer in the previous layer.

[0024] A real-time image defogging device based on FPGA convolutional neural network, comprising an FPGA peripheral support device and an FPGA device;

[0025] The FPGA peripheral support device includes an FPGA peripheral support module and a display memory module; the FPGA device includes a convolutional neural network module, a model parameter control module, a data transmission and priority arbitration module, and a memory data read and write control module;

[0026] The FPGA peripheral support module is used to complete the communication interaction with the outside through the interface and complete the data input of the camera video source; the display memory module is used to store the input feature map;

[0027] The model parameter control module is used to use RAM resources to complete the storage of all network parameters that have been quantized into integers; the data transmission and priority arbitration module is used to complete the management of writing and reading feature map data streams, and control the timing and data alignment of the convolution process; the memory data read and write control module is used to generate the display memory module read and write control commands and calculate the corresponding addresses of the display memory module read and write operations according to the read and write operation commands and the read and write operation space set by the user, so that each feature map is ID-encoded and the address space is allocated;

[0028] The convolutional neural network module is used to store the trained target convolutional neural network and execute the above-mentioned real-time image defogging method based on FPGA convolutional neural network.

[0029] Preferably: the model parameter control module is also used to group the model parameters trained in different scenarios, communicate with the outside through the FPGA peripheral support module, and complete the switching of convolution kernel network parameters according to external commands.

[0030] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0031] The embodiment of the present application provides a method and device for real-time image defogging based on FPGA convolutional neural network. Through this method, foggy image input is loaded into the convolutional neural network model on the FPGA. The model has completed network pre-training on the server-side GPU platform through a large number of pre-collected labeled data sets. The model parameters are integer quantized and tested and stored in the on-chip resources of the FPGA. When the FPGA performs convolutional neural network processing, multi-scale feature extraction, feature fusion and feature learning can be performed on the foggy image, and the output result is the defogging image. This method can give full play to the advantages of FPGA low power consumption, miniaturization, pipeline design and parallel processing, and realize real-time image defogging based on convolutional neural network at the edge end such as drones and ground unmanned platforms.

[0032] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 It is a flowchart of a real-time image defogging method based on FPGA convolutional neural network provided by an embodiment of the present invention;

[0035] Figure 2 Schematic diagram of a real-time image defogging device based on FPGA convolutional neural network provided by an embodiment of the present invention;

[0036] Figure 3 is a convolutional neural network calculation schematic diagram provided by an embodiment of the present invention;

[0037] Figure 4 2. It is a schematic diagram of a 2-dimensional convolution multiplication-accumulation array design provided by an embodiment of the present invention;

[0038] Figure 5 It is a schematic diagram of a convolutional neural network design architecture provided by an embodiment of the present invention;

[0039] Figure 6 It is a schematic diagram of the convolutional neural network pipeline design provided by an embodiment of the present invention.

[0040] In the figure: FPGA peripheral support module 1, display memory module 2, convolutional neural network module 3, model parameter control module 4, data transmission and priority arbitration module 5, memory data read and write control module 6. DETAILED DESCRIPTION

[0041] The technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0042] See also Figure 1 , a real-time image defogging method based on FPGA convolutional neural network is provided in an embodiment of the present invention, such as Figure 1 As shown, the method may include:

[0043] S101: Obtain the target input feature map and target convolution kernel network parameters required for each convolution layer in the trained target convolution neural network; the target input feature map is stored in the display memory module of the FPGA peripheral support device, and the target convolution kernel network parameters are stored in the RAM resources of the FPGA device, and the target convolution kernel network parameters are integer network parameters obtained after integer quantization;

[0044] S102: Inputting the target input feature map and the target convolution kernel network parameters into the corresponding convolution layer in the target convolution neural network for convolution operation processing to obtain a defogging image;

[0045] Among them, the target convolutional neural network only contains several convolutional layers; the integer quantization includes performing floating-point data training on a GPU platform to obtain the optimal network model weights, mapping the floating-point weights to integer weights through a quantized network, loading the mapped integer weights as pre-trained weights into the network model of integer parameters, testing and evaluating the integer quantization results, testing and evaluating the integer quantization results, and obtaining the integer network parameters.

[0046] The real-time image defogging method based on FPGA convolutional neural network provided in the embodiment of the present application, the target convolutional neural network provided only includes several convolutional layers, and the network model is designed to be lightweight, thereby ensuring that the target convolutional neural network can run smoothly on the FPGA device. At the same time, since the lightweight target convolutional neural network may still have a huge amount of calculation, the embodiment of the present application provides a means of integer quantization of the target convolution kernel network parameters to reduce the amount of calculation involved in the calculation. In order to maintain the accuracy of the network while also performing integer quantization, floating-point data training is first performed on the GPU platform, and the best network model weights are obtained. The floating-point weights are mapped to integer weights through the quantization network, and then the mapped weights are loaded into the network model of the integer parameters as pre-trained weights. After the training of the integer network, the integer weights with almost the same accuracy are obtained as much as possible.

[0047] The target convolution kernel network parameters are stored in the RAM resources of the FPGA device. The target convolution kernel network parameters of each layer can be stored separately. When each layer participates in the calculation, it can be temporarily called from the RAM resources of the FPGA device, giving full play to the advantages of FPGA's low power consumption, miniaturization, pipeline design and parallel processing, and realizing real-time image defogging based on convolutional neural networks at the edge of drones, ground unmanned platforms, etc.

[0048] In order to further give full play to the parallel computing advantages of FPGA, the embodiment of the present application can provide that in one convolution operation, all parameters are multiplied with the corresponding data in parallel, and then all multiplication results are summed.

[0049] Read multiple layers of feature maps in the current convolution layer and perform parallel calculations on the same convolution kernel in this layer. The advantage is that one feature map read can complete all convolution operations of one convolution kernel on multiple layers of feature maps.

[0050] The target convolutional neural network includes a pipeline structure in which row delay and calculation cycle are aligned. The pipeline structure of the entire convolutional neural network operation is redesigned, and the two methods of row delay and calculation cycle alignment are used to further improve the calculation efficiency of the convolutional neural network.

[0051] In order to retain more original image details and enhance the generalization ability of the network, the embodiment of the present application can provide that the target convolutional neural network combines low-dimensional information with high-dimensional abstract information by using two methods, feature concatenation and feature addition, so as to perform feature fusion of different layers.

[0052] Furthermore, the embodiment of the present application can provide a plurality of convolutional layers connected in series; the output feature map size of each convolutional layer is the same as the input feature map size. The convolutional neural network is only designed with convolutional layers, and the output feature map size of each convolutional layer is the same as the input image.

[0053] Furthermore, the embodiment of the present application can provide that the convolution layer of the 7th layer and the convolution layer of the 8th layer use two groups of 3×3 convolution kernels to realize feature learning. Convolution layer 7 and convolution layer 8 use two groups of 3×3 convolution kernels to realize feature learning, improve the nonlinear mapping ability of the network, and estimate the haze-free image.

[0054] Furthermore, the embodiment of the present application can provide that the target input feature map required by the first convolution layer is the original image, and the first convolution layer uses multiple groups of convolution kernels of different sizes to perform multi-scale feature extraction on the original image. In particular, the first layer uses four groups of convolution kernels of different sizes (1×1, 3×3, 5×5, 7×7) to perform multi-scale feature extraction on the original image, which can not only avoid the loss of important information of the original image but also effectively achieve scale invariance, but it also requires the adjustment of the number of row caches of each convolution kernel convolution process through the data transmission and priority arbitration module.

[0055] It can be understood that when each convolution layer is operating, the target input feature map of each convolution layer except the first layer can be the output feature map of the previous layer. Specifically, the target input feature map required by the convolution layer other than the first layer is the output feature map of the convolution layer of the previous layer. Since the layers of the convolutional neural network are connected in series, the calculation result depends on the output of the upper layer, that is, the output feature map of the upper layer will become the input feature map of the next layer. Then the network parameters of the next layer are read out and the convolution operation is performed again, and the operation results are stored in DDR3 SDRAM, and so on, until the entire network is completed.

[0056] The real-time image defogging method based on FPGA convolutional neural network provided in the embodiment of the present application is that the foggy image input is loaded into the convolutional neural network model on the FPGA. The model has completed network pre-training on the server-side GPU platform through a large number of pre-collected labeled data sets. The model parameters are integer-quantized and stored in the on-chip resources of the FPGA. When the FPGA performs convolutional neural network processing, multi-scale feature extraction, feature fusion and feature learning can be performed on the foggy image, and the output result is the defogging image. This method can give full play to the advantages of FPGA's low power consumption, miniaturization, pipeline design and parallel processing, and realize real-time image defogging based on convolutional neural networks at the edge of drones, ground unmanned platforms, etc.

[0057] like Figure 2As shown, the embodiment of the present application can also provide a real-time image defogging device based on FPGA convolutional neural network, see Figure 2 , the device may include an FPGA peripheral support device and an FPGA device;

[0058] The FPGA peripheral support device includes an FPGA peripheral support module 1 and a display memory module 2; the FPGA device includes a convolutional neural network module 3, a model parameter control module 4, a data transmission and priority arbitration module 5 and a memory data read and write control module 6;

[0059] The FPGA peripheral support module 1 is used to complete the communication interaction with the outside through the interface and complete the data input of the camera video source; the display memory module 2 is used to store the input feature map;

[0060] The model parameter control module 4 is used to use RAM resources to complete the storage of all network parameters that have been quantized into integers; the data transmission and priority arbitration module 5 is used to complete the management of writing and reading feature map data streams, and control the timing and data alignment of the convolution process; the memory data read and write control module 6 is used to generate the display memory module read and write control commands and calculate the corresponding addresses of the display memory module read and write operations according to the read and write operation commands and the read and write operation space set by the user, so that each feature map is ID-encoded and the address space is allocated;

[0061] The convolutional neural network module 3 is used to store the trained target convolutional neural network and execute the above-mentioned real-time image defogging method based on FPGA convolutional neural network.

[0062] Furthermore, the embodiment of the present application can also provide that the model parameter control module 4 is also used to group the model parameters trained in different scenarios, communicate with the outside through the FPGA peripheral support module 1, and complete the switching of the convolution kernel network parameters according to external commands.

[0063] The solution provided in the embodiment of the present application is introduced in detail below in conjunction with a real-time image defogging device based on an FPGA convolutional neural network.

[0064] In the present invention, the real-time image defogging device based on FPGA includes two parts: FPGA and important peripheral supporting modules of FPGA.

[0065] FPGA assumes the function of the core computing processing unit, and the software functional modules are implemented by its coding.

[0066] FPGA's important peripheral support modules mainly include:

[0067] The front-end video input hardware unit (supports multiple video interfaces such as SDI, CSI MIPI, etc.) transmits the video source signal to the FPGA through the video input hardware unit. The FPGA completes video analysis, pixel extraction, image control signal extraction, and size-crops the image to be processed (divided into 640×480, 640×360, 320×240) to complete data caching.

[0068] The back-end video output hardware unit (supports multiple video display interfaces such as SDI, HDMI, VGA, etc.) realizes the output display of the defogged image.

[0069] Serial port, CAN, USB Type-C and other communication interfaces.

[0070] DDR3 SDRAM memory module (display memory module).

[0071] The real-time image dehazing method based on FPGA convolutional neural network is mainly divided into five parts according to the functional modules:

[0072] 1. FPGA peripheral support module, video analysis and acquisition module, serial port, CAN, USB and other communication modules, video display module.

[0073] 2. DDR3 SDRAM data read and write control module (memory data read and write control module) completes data storage interaction with DDR3 SDRAM.

[0074] 3. Model parameter control module, which controls the reading and writing of pre-trained model network parameters.

[0075] 4. Data transmission and priority arbitration module (completes the interaction with the DDR3 SDRAM data read and write control module).

[0076] 5. Convolutional neural network module, CNN dehazing algorithm module.

[0077] The functional modules are described in detail.

[0078] FPGA peripheral support module:

[0079] First, it completes external communication interaction through serial ports, CAN, USB Type-C and other interfaces. The functions of this module follow the corresponding interface standard specifications, and the software functions are encoded and processed by FPGA.

[0080] The second is to complete the data input of the camera video source. FPGA parses the video stream format of the unmanned platform camera and extracts pixels and video control signals (data decoding follows the video format standard specification, and the function is implemented by FPGA coding), and completes the first-level cache of the extracted pixel data in the corresponding asynchronous FIFO, and then reads data from the corresponding asynchronous FIFO under the control of the data transmission and priority arbitration module, and transmits it to the DDR3 SDRAM data read and write control module cache.

[0081] The third is to output and display the processed dehazed image according to the required video format. The module function complies with the corresponding video format standard specifications, and the software function is implemented by FPGA encoding processing.

[0082] DDR3 SDRAM data read and write control module: Generates DDR3 read and write control commands and calculates the corresponding addresses of DDR3 read and write operations according to the read and write operation commands and the read and write operation space set by the user. If it is a write command, the feature map data will be written into the DDR3 SDRAM memory. If it is a read command, the required feature map data will be read from the corresponding address space and output to the asynchronous FIFO of the data cache module. Each feature map data is ID-encoded and the address space is allocated.

[0083] Model parameter control module: Use the FPGA's on-chip RAM resources to complete the storage of all network parameters that have been quantized into integers. Integer data is numerical data that does not contain a decimal part and is represented by the letter I. Integer data is only used to represent integers and is stored in binary form.

[0084] Each time a calculation is performed, data is extracted from RAM for the current convolution layer calculation. At the same time, the model parameters trained in different scenarios can be grouped, communicated with the outside through the FPGA peripheral support module, and the network model parameters can be switched according to external commands.

[0085] Convolutional neural network module: Figure 3 As shown in the figure, first, the feature map in DDR3 SDRAM and the convolution kernel network parameters in the model parameter control module are read to the convolution neural network module according to the address assigned when writing data. The goal of each layer of convolution is to convert N input feature maps into M output feature maps. The conversion relationship is determined by N×M convolution kernels, that is, the connection weights between the two layers of neurons. The data results calculated by each layer of convolution network are then written into DDR3 SDRAM.

[0086] The layers of the convolutional neural network are connected in series, and the calculation results depend on the output of the upper layer, that is, the output feature map of the upper layer will become the input feature map of the next layer. Then the parameters of the next layer are read out and the convolution operation is performed again, and the operation results are stored in DDR3 SDRAM, and so on, until the entire network is completed.

[0087] Data transmission and priority arbitration module:

[0088] The data transmission and priority arbitration module mainly completes the overall management of feature map data flow writing and reading, and controls the timing and data alignment of the convolution process. Figure 4 As shown in the figure, during the convolution operation, a shift register (RAM-based shift register) is required for row caching. The shift register completes the feature map data cache of (N+1) / 2 rows and performs the convolution operation after the data is aligned. The uncached row data is zeroed, and the input and output sizes of the feature map are the same by padding the edges with zeros. That is, a 3×3 convolution operation requires 2 rows of data caching and data alignment, and a 5×5 convolution operation requires 3 rows of data caching and data alignment.

[0089] Target convolutional neural network design (completed network pre-training model). Figure 5 As shown in Table 1.

[0090] Table 1

[0091]

[0092] Although the target convolutional neural network model is lightweight because it only contains convolutional layers, the amount of computation is still very large. The hardware platform resources cannot deploy 32-bit floating-point operations, so it is necessary to use integer quantization to reduce the amount of computation involved.

[0093] In order to maintain network accuracy while performing integer quantization, floating-point data training is first performed on the GPU platform to obtain the best network model weights. The floating-point weights are mapped to integer weights through the quantized network, and then the mapped weights are loaded into the network model of integer parameters as pre-trained weights. After training the integer network, integer weights with almost the same accuracy are obtained as much as possible.

[0094] Feature extraction:

[0095] This convolutional neural network is designed with only convolutional layers, and the output feature map size of each convolutional layer is the same as the input image. The original image can be processed in standard sizes (640×480, 640×360, 320×240) as needed.

[0096] In particular, the first layer uses four groups of convolution kernels of different sizes (1×1, 3×3, 5×5, 7×7) to perform multi-scale feature extraction on the original image, which not only avoids the loss of important information of the original image but also effectively realizes scale invariance. However, it is also necessary to adjust the number of row caches of each convolution kernel convolution process through the data transmission and priority arbitration module.

[0097] Feature Fusion:

[0098] The target convolutional neural network provided in the embodiment of the present application combines low-dimensional information with high-dimensional abstract information by using feature concatenation and feature addition. By fusing features at different layers, more original image details are retained and the generalization ability of the network is enhanced.

[0099] Feature Learning:

[0100] Convolutional layers 7 and 8 of the target convolutional neural network use two sets of 3×3 convolutional kernels to achieve feature learning, improve the network's nonlinear mapping capability, and estimate the haze-free image.

[0101] In particular, the method provided in the embodiment of the present application performs parallel computing design in three dimensions, giving full play to the parallel computing advantages of FPGA, which is mainly manifested in:

[0102] 1. For a convolution operation, all parameters are multiplied with the corresponding data in parallel, and then all multiplication results are summed, such as Figure 4 shown.

[0103] Second, read the multi-layer feature maps in the current convolution layer and perform parallel calculations on the same convolution kernel in this layer. Figure 3 As shown in the figure. The advantage is that one feature map read can complete all convolution operations of a convolution kernel on multiple layers of feature maps. At the same time, the result of the convolution operation completed in each cycle needs to be summed again and added to the offset value, and then the output feature map result is obtained. The output feature map data will be cached in DDR3 SDRAM again.

[0104] Third, the pipeline structure of the entire convolutional neural network operation was redesigned, and the computational efficiency of the convolutional neural network was further improved by aligning row delays and computation cycles. Figure 6 As shown in the figure. Convolution layer 1 includes 4 feature layer outputs of 1X1, 3X3, 5X5, and 7X7 convolution kernels; the output feature map is cached in shift register rows and input into convolution layer 2. At the same time, the 4 feature layers output by convolution layer 1 are aligned with the row delay and calculation cycle by using feature splicing to ensure that the output feature layer data of convolution layer 2 and the output feature layer data of convolution layer 1 are synchronously input into convolution layer 3 to realize pipeline design.

[0105] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0106] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0107] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0108] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A real-time image defogging device based on FPGA convolutional neural network, characterized in that: Including FPGA peripheral support devices and FPGA devices; The FPGA peripheral support device includes an FPGA peripheral support module and a display memory module; the FPGA device includes a convolutional neural network module, a model parameter control module, a data transmission and priority arbitration module, and a memory data read and write control module; The FPGA peripheral support module is used to complete the communication interaction with the outside through the interface and complete the data input of the camera video source; the display memory module is used to store the input feature map; The model parameter control module is used to utilize RAM resources to complete the storage of all network parameters that have been quantized into integers; The data transmission and priority arbitration module is used to complete the management of feature map data stream writing and reading, and control the timing and data alignment of the convolution process; during the convolution operation, a shift register is used for line caching; the shift register completes the (N+1) / 2 lines of feature map data caching and performs the convolution operation after the data is aligned, and the uncached line data is zeroed to ensure that the input and output sizes of the feature map are the same by zero padding the edges; That is, a 3×3 convolution operation requires 2 rows of data cache and data alignment, and a 5×5 convolution operation requires 3 rows of data cache and data alignment; The memory data read / write control module is used to generate the display memory module read / write control command and calculate the corresponding address of the display memory module read / write operation according to the read / write operation command and the read / write operation space set by the user, so that each feature map is ID-coded and the address space is allocated; The model parameter control module is also used to group the model parameters trained in different scenarios, communicate with the outside through the FPGA peripheral support module, and complete the switching of the convolution kernel network parameters according to the external command; The convolutional neural network module is used to store the trained target convolutional neural network and execute the real-time image defogging method based on FPGA convolutional neural network; The real-time image defogging method based on FPGA convolutional neural network includes: Obtain the target input feature map and target convolution kernel network parameters required for each convolution layer in the trained target convolution neural network; the target input feature map is stored in the display memory module of the FPGA peripheral support device, and the target convolution kernel network parameters are stored in the RAM resources of the FPGA device, and the target convolution kernel network parameters are integer network parameters obtained after integer quantization; Inputting the target input feature map and the target convolution kernel network parameters into the corresponding convolution layer in the target convolution neural network for convolution operation processing to obtain a defogging image; The target convolutional neural network only includes several convolutional layers; the integer quantization includes performing floating-point data training on a GPU platform to obtain the optimal network model weights, mapping the floating-point weights to integer weights through a quantized network, loading the mapped integer weights as pre-trained weights into a network model of integer parameters, testing and evaluating the integer quantization results, and obtaining the integer network parameters; In one convolution operation, all parameters are multiplied with the corresponding data in parallel, and then all multiplication results are summed; multiple layers of feature maps in the current convolution layer are read to perform parallel calculations on the same convolution kernel in this layer; The target convolutional neural network includes a pipeline structure in which row delays are aligned with calculation cycles; convolution layer 1 includes 4 feature layer outputs of 1X1, 3X3, 5X5, and 7X7 convolution kernels; the output feature map is cached in shift register rows and input into convolution layer 2; a feature splicing method is adopted to align row delays and calculation cycles of the 4 feature layers output by convolution layer 1, so that the output feature layer data of convolution layer 2 and the output feature layer data of convolution layer 1 are synchronously input into convolution layer 3 to realize the pipeline structure.

2. The real-time image defogging device based on FPGA convolutional neural network according to claim 1 is characterized in that: The target convolutional neural network combines low-dimensional information with high-dimensional abstract information by using feature concatenation and feature addition, so as to perform feature fusion of different layers; several layers of the convolutional layers are serially connected; The output feature map size of each convolutional layer is the same as the input feature map size.

3. The real-time image defogging device based on FPGA convolutional neural network according to claim 2 is characterized in that: The 7th convolution layer and the 8th convolution layer use two groups of 3×3 convolution kernels to achieve feature learning.

4. The real-time image defogging device based on FPGA convolutional neural network according to claim 2 is characterized in that: The target input feature map required by the first convolution layer is the original image, and the first convolution layer uses multiple groups of convolution kernels of different sizes to perform multi-scale feature extraction on the original image.

5. The real-time image defogging device based on FPGA convolutional neural network according to claim 4 is characterized in that: The target input feature map required by the convolutional layers other than the first convolutional layer is the output feature map of the convolutional layer in the previous layer.