A real-time image defogging method based on FPGA+ARM

Abstract

The present invention discloses a real-time image defogging method based on FPGA+ARM, comprising the following steps: step 1, collecting foggy images; step 2, converting the image color space in FPGA, converting the image from RGB to HSV space; using an improved Retinex defogging algorithm, defogging the foggy images collected by the system in real time, and introducing a Sigmod function to correct the color of the defogged image; step 3, outputting the defogged image and displaying it on a display screen. The present invention improves the Retinex algorithm, and the brightness of the image obtained by the improved algorithm processing is improved, the linear mapping effect of the image data is improved, the original color of the image is effectively restored, the image contrast is increased, and more details are displayed.

Description

Technical Field

[0001] The invention relates to the fields of image processing and computer vision, and in particular to a real-time image defogging method based on FPGA+ARM. Background Art

[0002] In recent years, with the rapid development of smart cities and smart transportation, the demand for intelligent outdoor vision systems has become more urgent. In addition to adversely affecting people's lives and health, haze weather also hinders the effective application of intelligent vision systems. Light propagation in foggy days is affected by the scattering of suspended matter in the atmosphere, resulting in the loss of main information and significant color distortion of images obtained by imaging equipment such as urban monitoring systems, aerial photography systems, and assisted driving systems, which limits the widespread use of computer vision.

[0003] At present, most intelligent solutions for visual systems use a dark channel prior defogging algorithm, which is more complex to implement and less costly than the Retinex algorithm. The Retinex-based defogging optimization solution, including the solution using the high-level synthesis (HLS) tool to complete the single-scale (SSR) algorithm design, has relatively large resource consumption, high cost, and insufficient defogging effect. Another solution based on software and hardware collaboration implements the Retinex algorithm based on guided filtering. This solution still has a lot of room for optimization in terms of defogging effect and resource utilization. In summary, there is still a lot of room for improvement in resource utilization and cost control in current related research, which makes the Retinex-based image defogging algorithm encounter bottlenecks in the efficient application of embedded devices. Therefore, it is very necessary to propose a new type of fast image defogging system based on ARM+FPGA. Summary of the invention

[0004] In order to solve the problem that the main information of images acquired by imaging devices such as urban monitoring systems, aerial photography systems, and assisted driving systems are lost and they cannot work effectively under the influence of severe weather environments such as haze and dust, the present invention uses machine vision technology to perform real-time defogging of images based on FPGA+ARM to ensure that subsequent image work is not affected.

[0005] In order to achieve the above object, the present invention is implemented by the following technical solutions:

[0006] The present invention is a real-time image defogging method based on FPGA+ARM, comprising the following steps:

[0007] Step 1, the system collects foggy images through a 5-megapixel complementary metal oxide semiconductor camera on the FPGA side;

[0008] Step 2, convert the image color space in FPGA, converting the image from RGB to HSV space;

[0009] The improved Retinex defogging algorithm is used to defog the foggy images collected by the system in real time, and the Sigmod function is introduced to correct the color of the defogged images.

[0010] Step 3: The system outputs the dehazed image and displays it on a liquid crystal display device.

[0011] A further improvement of the present invention is that the specific steps of step 2 are as follows:

[0012] Step 2.1, convert the image color space in FPGA, converting the image from RGB to HSV space;

[0013] Step 2.2, the first channel performs illumination component estimation optimization: the V component is enhanced by using the Retinex dehazing algorithm with a bilateral filter function, where the bilateral filter function is as follows:

[0014]

[0015] Where BF[I] p is the pixel after filtering, s is the filtering window domain, and They are Gaussian filters in spatial domain and pixel range domain respectively;

[0016] Step 2.2: The second channel is saturated and normalized; specifically:

[0017] The S component of the image is linearly stretched using an adaptive linear stretching algorithm; the adaptive linear stretching algorithm is as follows:

[0018]

[0019] In the formula, S is the saturation before treatment; is the saturation after processing; M v is the mean value of the image before processing; max(R,G,B), min(R,G,B) are the maximum and minimum values ​​of the R, G, B components of the image before processing;

[0020] The Min-Max normalization algorithm is used to normalize the second channel; wherein, the Min-Max normalization algorithm is as follows:

[0021]

[0022] Where max and min are the maximum and minimum values ​​of the i-th channel of the image, respectively. max and V min The maximum and minimum values ​​of the specified scaling range are 255 and 0 respectively, and yi Then it is the original image x i The result after linear mapping.

[0023] Step 2.3, color restoration of the third channel: the sigmoid function is used to restore the color of the image, where the sigmoid function is:

[0024]

[0025] In the formula, the independent variable x is the input pixel value; S(x) is the processed pixel value; k1 and k2 are the standard deviation coefficient and the pixel value coefficient respectively. The standard deviation corresponds to the contrast of the image, and the pixel value corresponds to the brightness of the image. The function presents an S-shaped curve, so the brightness value of most areas can be expanded. The value range is controlled in (0-255], which can avoid over-enhancement.

[0026] The further improvement of the present invention is that: Step 3 is specifically as follows: the custom AXI_DMA_WR write module exchanges data with the DDR3 on the ARM side at high speed through the AXI interface, transmits the processed video data to the DDR3 memory chip for storage, and finally the ARM side sends a read signal to control the AXI_DMA_RD read module to read the image data in the DDR3 and cache it to the FIFO cache module on the FPGA side, and finally displays it on the LCD display device.

[0027] The beneficial effects of the present invention are as follows: 1. The present invention makes various improvements to the Retinex algorithm, such as illumination estimation, color balance, and fusion enhancement, so that the brightness of the processed image is improved, the linear mapping effect of the image data is improved, the original color of the image is effectively restored, the image contrast is increased, and more details are displayed. 2. While ensuring a good defogging effect, it has the advantages of fast computing speed and low power consumption, which can meet the practical requirements of outdoor video systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 : A flow chart of the steps of a real-time image defogging system based on FPGA+ARM.

[0029] Figure 2 : An overall block diagram of a real-time image defogging system based on FPGA+ARM.

[0030] Figure 3 : An original foggy image input by a real-time image defogging system implementation based on FPGA+ARM.

[0031] Figure 4 :An implementation example of a real-time image defogging system based on FPGA+ARM. The final defogging image. DETAILED DESCRIPTION

[0032] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more explicit definition of the protection scope of the present invention. However, these embodiments are illustrative and are only used to specifically describe the present invention, and should not be understood as limiting the present invention.

[0033] like Figure 1-4 As shown, the present invention is a real-time image defogging method based on FPGA+ARM. The specific implementation of the present invention is carried out on a heterogeneous processing platform of FPGA+ARM architecture. The platform can adopt ZYNQ7010 platform. The principle of the real-time image defogging system based on FPGA+ARM is as follows Figure 2 As shown, the whole system is mainly divided into two parts, namely, the FPGA programmable logic part and the ARM processing system. The specific steps of the defogging method of the present invention are as follows:

[0034] Step 1, the system collects foggy images through a 5-megapixel complementary metal oxide semiconductor camera on the FPGA side; that is, the video is collected through a 5-megapixel CMOS camera on the FPGA side, and the ARM side lead pin (EMIO) configures the CMOS camera through the software simulation serial camera control bus (SCCB) protocol, and the resolution of the configured image is set to 1280*720, the pixel data is in 16-bit RGB565 format, and the number of data frames is configured to 30 frames. The collected image data is sent to the image acquisition module in the FPGA using the hardware description language (Verilog) programming package to integrate the data bit width into a 24-bit RGB888 format. Since the frame rate of the image to be processed is 30 frames, the resolution is 1280*720, and the data flow rate is 664Mbps, the data width is 32 bits in the implementation of this defogging method, the AXI clock is set to 150MHz, and the AXI interface data transmission bandwidth is 4.8Gbps, which is sufficient to meet the transmission bandwidth requirements.

[0035] Step 2: After obtaining the foggy image, the HSV space is more in line with the human visual system than the RGB space, and is easier to separate and process. Therefore, this step first converts the color space of the foggy image in the FPGA, converting the image from RGB to HSV space. Then, with the help of the parallel computing characteristics of the FPGA, the foggy image in the HSV space is processed in three channels;

[0036] The first channel mainly realizes the function of optimizing the estimation of illumination components. The filtering function of the traditional Retinex algorithm is a Gaussian function, and the filtering method of this function is spatial domain filtering. The pixel value of the center point is jointly determined by the pixel values ​​of other points in the neighborhood. Because it is a blurring process, it is easy to lose details. In order to solve the problem of image blurring and loss of details after traditional Gaussian filtering, the present invention adopts a bilateral filtering function. Bilateral filtering considers both spatial domain factors and value range factors. The center pixel value is only determined by the pixel values ​​of points with similar pixel values ​​in the neighborhood, and has a good ability to retain details. The specific operation to achieve the optimization of illumination component estimation is: first, bilateral filtering is performed on the V component, and then the lookup table method is used to perform logarithmic operations on the original V component and the V component after bilateral filtering, and the difference between the two results is normalized, and the normalized result is used as the final V component, wherein the normalization process is implemented through the second channel.

[0037] Bilateral filtering is implemented as follows:

[0038]

[0039] Where BF[I] p is the pixel after filtering, s is the filtering window domain, and They are Gaussian filters in the spatial domain and pixel range domain respectively. In this dehazing method, the range of s is set to (0~255).

[0040] The single-scale Retinex algorithm (SSR) can enhance details, but it is easy to lose color, so the SSR algorithm cannot simultaneously meet the two characteristics of detail enhancement and color fidelity. In order to achieve the ideal enhancement effect, the present invention improves the filter. A bilateral filter with a window domain of (0-255) is used, and the first-pass illumination extraction optimization is completed.

[0041] The second channel mainly realizes the functions of saturation stretching and normalization.

[0042] The S component of the HSV space represents the saturation of the image. Saturation corresponds to the vividness of the image color. The higher the saturation, the more vivid the image color. Linear stretching of the S component of the foggy image can solve the whitening problem of the image. Different images have different saturations, and different saturations require different stretching scales. In order to achieve the ideal enhancement effect, this paper adopts an adaptive scale linear stretching algorithm, which can improve the average brightness and visual effect of the image; the adaptive scale linear stretching algorithm is as follows:

[0043]

[0044] In the formula, S is the saturation before treatment; is the saturation after processing; M v is the mean value of the image before processing; max(R,G,B), min(R,G,B) are the maximum and minimum values ​​of the R, G, and B components of the image before processing. In the implementation of this defogging method, after multiple adjustments to the S component, the saturation When the value is 46.8%, the saturation of the image can achieve an ideal effect.

[0045] In order to solve the problem of large amount of digital signal processing (DSP) resource consumption and high timing requirements in the traditional Retinex algorithm when calculating in the exponential domain, the minimum and maximum (Min-Max) normalization proposed in this invention can save a lot of resources compared with the exponential calculation. The Min-Max normalization algorithm is implemented as follows:

[0046]

[0047] Where max and min are the maximum and minimum values ​​of the i-th channel of the image, respectively. max and V min It specifies the maximum and minimum values ​​of the scaling range, which are 255 and 0 respectively. i Then it is the original image x i The result after linear mapping.

[0048] The third channel mainly performs color restoration based on the normalization of the second channel. The algorithm in this paper uses the sigmoid function to restore the color of the image. The sigmoid function for foggy images is:

[0049]

[0050] In the formula, the independent variable x is the input pixel value; S(x) is the processed pixel value; k1 and k2 are the standard deviation coefficient and pixel value coefficient respectively. The standard deviation corresponds to the contrast of the image, and the pixel value corresponds to the brightness of the image. The value of K1 is set to 0.95 and the value of k2 is set to 1.27. At this time, the function presents an S-shaped curve, which can expand the brightness value of most areas. The value range of S(x) is controlled in [0~255], which can avoid over-enhancement.

[0051] Step 3: The system outputs the dehazed image and displays it on a liquid crystal display device.

[0052] The AXI_DMA_WR write module exchanges data with the DDR3 memory on the ARM side through the AXI interface at high speed, and transfers the processed video data to the DDR3 memory chip for storage. Finally, the ARM side sends a read signal to control the AXI_DMA_RD read module to read the image data from the DDR3 and cache it to the FIFO cache module on the FPGA side, and finally displays it on the LCD device.

[0053] The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been discussed above with a preferred embodiment, it is not used to limit the present invention. Any technician familiar with the profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A real-time image defogging method based on FPGA+ARM, characterized by: The steps include: Step 1: Collect foggy images through the camera on the FPGA side; Step 2: Convert the image color space in FPGA, converting the image from RGB to HSV space; defogging the foggy image collected by the system in real time through the improved Retinex defogging algorithm, and introduce the Sigmoid function to correct the color of the defogging image; Step 3, outputting the corrected dehazed image and displaying it on a display screen; The specific steps of step 2 are as follows: Step 2.1, convert the image color space in FPGA, converting the image from RGB to HSV space; Step 2.2, perform algorithm processing on the foggy image in the HSV space; specifically: Step 2.21, optimize the illumination component estimation: improve the Retinex dehazing algorithm, replace the Gaussian function in the original algorithm with the bilateral filtering function, enhance the image V component through the bilateral filtering function, calculate the filtered V component and the original V component, and calculate the difference; Among them, the bilateral filtering function is as follows: Where BF[I] p is the pixel after filtering, s is the filtering window domain, and They are Gaussian filters in spatial domain and pixel range domain respectively; Step 2.22, using the Min-Max normalization algorithm to normalize the result obtained in step 2.21; wherein the Min-Max normalization algorithm is as follows: Where max and min are the maximum and minimum values ​​of the i-th channel of the image, respectively. max and V min The maximum and minimum values ​​of the specified scaling range are 255 and 0 respectively, and y i Then it is the original image x i The result after linear mapping; Step 2.23, using an adaptive linear stretching algorithm to stretch the image saturation; wherein the adaptive linear stretching algorithm is as follows: In the formula, S is the saturation before treatment; is the saturation after processing; M v is the mean value of the image before processing; max(R,G,B), min(R,G,B) are the maximum and minimum values ​​of the R, G, B components of the image before processing; Step 2.3, convert the processed image into RGB format, and use the sigmoid function to restore the color of the image, where the sigmoid function is: In the formula, the independent variable x is the input pixel value; S(x) is the processed pixel value; k1 and k2 are the standard deviation coefficient and pixel value coefficient respectively. The standard deviation corresponds to the contrast of the image, and the pixel value corresponds to the brightness of the image. The function presents an S-shaped curve, so the brightness value of most areas can be expanded, and the value range is controlled in (0-255).

2. The real-time image defogging method based on FPGA+ARM according to claim 1, characterized in that: In step 2.21, a lookup table is used to perform logarithmic operations on the original V component and the V component after bilateral filtering.

Images