Low-latency image defogging system based on FPGA

Abstract

The present application relates to a low-latency image defogging system based on FPGA, which includes: a cache module, an atmospheric light calculation module, a guide filter module, a defogging operation module, and a data processing module. In this system, the image to be defogged is divided into multiple groups of image data by using a filter sliding window in the cache module, so that the subsequent processing module can continuously process the image data like a pipeline, thereby achieving low latency and reducing resource usage.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a low-latency image defogging system based on FPGA. Background Art

[0002] When conducting reconnaissance, remote control or enabling visual navigation, drones or unmanned vehicles rely heavily on images with high imaging quality. Under adverse weather conditions such as fog and haze, the turbid media in the atmosphere seriously absorb and scatter light, which reduces atmospheric visibility and seriously affects the quality and clarity of the captured images, causing degradation and blurring of drone or unmanned vehicle images, distorting and damaging the basic information features in the images, and further affecting the remote control, visual navigation, real-time target tracking and other later tasks of drones or unmanned vehicles. Therefore, real-time defogging of images obtained by drones or unmanned vehicles has important research and application value.

[0003] There are three main platforms for implementing defogging algorithms: 1. Image processing based on PC or embedded computing software. This processing is a CPU algorithm architecture, which is flexible and easy to deploy, but it does not have the ability to process defogging images with low latency. 2. ASIC-based dedicated image processing is mainly an image frame processing method. It uses frame-based image defogging processing and has multi-channel asynchronous processing capabilities, but does not have the processing advantage of low latency. 3. FPGA-based processing method: The FPGA-based processing method is to design the algorithm flow and details by yourself.

[0004] However, the above-mentioned existing technologies all have the problem of computational delay and cannot meet the requirements of strict real-time systems, such as cruise missiles, high-speed drones, etc. Summary of the invention

[0005] Based on this, it is necessary to provide a low-latency image defogging system based on FPGA that can reduce latency to address the above technical problems.

[0006] A low-latency image defogging system based on FPGA, the system comprising: a cache module, an atmospheric light calculation module, a guide filter module, a defogging calculation module and a data processing module;

[0007] The cache module is used to traverse the current frame image of the input image data to be defogged through a filter sliding window of a preset size, generate a set of image data according to the filter sliding window after each movement, and transmit the sequentially generated image data to the atmospheric light calculation module and the guide filtering module;

[0008] The atmospheric light calculation module is used to perform statistical processing on each group of image data in the current frame image in turn, obtain the histogram area and grayscale statistical value of a frame image, and transmit them to the data processing module;

[0009] The data processing module is used to calculate the atmospheric light estimation value of a frame of image according to the histogram area and grayscale statistics of the frame of image, and transmit the atmospheric light estimation value to the defogging operation module;

[0010] The guided filtering module is used to sequentially perform guided filtering on each group of the image data to obtain a corresponding filtered image, and sequentially transmit the filtered image to the defogging operation module;

[0011] The defogging operation module is used to calculate the defogging images of each group of image data in sequence according to the atmospheric light estimation value of the previous frame image and the multiple filtered images, and finally obtain the defogging current frame image.

[0012] In one embodiment, the center of the filtering sliding window starts from the pixel of the first row and first column of the current frame image and moves to the right one column at a time until the center of the filtering sliding window traverses all the pixels in the first row, then moves down one row, starts from the pixel of the second row and first column of the current frame image and moves to the right until the center of the filtering sliding window traverses all the pixels of the current frame image.

[0013] In one embodiment, the system includes a first memory, and a set of image data obtained by each movement of the filtering sliding window is cached in the memory by column for each row;

[0014] When the filtering sliding window moves one column to the right to obtain a new set of image data, the first memory replaces the row of data first cached in the first memory with the latest row of data.

[0015] In one embodiment, the first memory adopts a data cache method of combining data bit width.

[0016] In one embodiment, in the cache module, the pixel data on the column where the center of each group of image data is located is transmitted to the guided filtering module for guided filtering processing, and the center pixel data of each group of image data is transmitted to the atmospheric light calculation module to calculate the histogram area and grayscale statistical value of the current frame image.

[0017] In one embodiment, the guided filtering module includes a dark channel calculation unit, a guided filtering processing unit and a band-guided filtering calculation unit connected in sequence;

[0018] The dark channel calculation unit is used to sequentially calculate multiple groups of dark channel data of the pixel data on the columns where the image data centers are located, and sequentially transmit the multiple groups of dark channel data to the band-directed filtering processing unit;

[0019] The guided filtering processing unit is used to perform mean operation on each group of image data in turn according to the multiple groups of dark channel data, and transmit the mean operation result of each group of image data to the guided filtering calculation unit;

[0020] The band-oriented filtering calculation unit is used to obtain the filtered images corresponding to each group of image data in turn according to the mean calculation results of each group of image data combined with the dark channel data calculated by the dark channel calculation unit.

[0021] In one embodiment, the guided filtering processing unit performs a mean operation on each group of image data, including:

[0022] Calculate the accumulated value of each group of image data in sequence according to each group of dark channel data;

[0023] The corresponding effective mean values ​​are calculated according to the accumulated values ​​of each group of image data, and the central pixels of each group of image data are replaced by each effective mean value accordingly.

[0024] In one embodiment, the calculation of the accumulated value of each group of image data is implemented by using the accumulated value of the bounding box;

[0025] When calculating the effective mean, it is necessary to remove invalid data of each group of image data that is outside the boundary of the current frame image.

[0026] In one embodiment, the image defogging system further includes a second memory, and the cache module transmits the central pixel data of each group of image data to the second memory;

[0027] The second memory transmits the central pixel data of each group of image data to the defogging operation module;

[0028] The defogging operation module sequentially calculates the defogging images of each group of image data according to the atmospheric light estimation value of the previous frame image, the multiple filtered images and the central pixel data corresponding to each filtered image.

[0029] The above-mentioned FPGA-based low-latency image defogging system divides the image to be defogged into multiple groups of image data by adopting a filtering sliding window, so that the subsequent processing module can continuously process the image data like a pipeline, thereby achieving low latency and reducing resource usage. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1Schematic diagram of the structure of a low-latency image defogging system based on FPGA in one embodiment;

[0031] Figure 2 A schematic diagram of data storage in a first memory in an embodiment;

[0032] Figure 3 A schematic diagram of output data of a cache module in one embodiment;

[0033] Figure 4 is a schematic diagram of the structure of a guided filtering module in one embodiment;

[0034] Figure 5 A schematic diagram of bounding box cumulative value calculation in one embodiment;

[0035] Figure 6 A schematic diagram of bounding box boundary calculation in one embodiment;

[0036] Figure 7 A schematic diagram of calculating a histogram area and grayscale statistical values ​​in an embodiment. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0038] In the prior art, when dehazing an image, it is generally necessary to wait for the memory to cache a complete frame of the image, and then read and output the frame of the image to the subsequent module for other algorithm processing. The entire image also needs to be processed during the subsequent module processing. This processing method will result in serious delays, which cannot meet some application scenarios with strict real-time requirements. In addition, the need to cache the complete image is also a consumption of memory.

[0039] In response to the above problems, Figure 1 As shown, the present application provides a low-latency image defogging system based on FPGA, which includes: a cache module, an atmospheric light calculation module, a guided filtering module, a defogging operation module and a data processing module.

[0040] The cache module is used to traverse the current frame image of the input image data to be dehazed through a filter sliding window of a preset size, generate a set of image data according to the filter sliding window after each movement, and transmit the sequentially generated image data to the atmospheric light calculation module and the guide filter module.

[0041] The atmospheric light calculation module is used to perform statistical processing on each group of image data in the current frame image in turn, obtain the histogram area and grayscale statistical value of the current frame image, and transmit them to the data processing module.

[0042] The data processing module is used to calculate the atmospheric light estimation value of a frame (that is, the current frame) according to the histogram area and grayscale statistics of the frame image, and transmit it to the defogging operation module.

[0043] The guided filtering module is used to perform guided filtering on each group of image data in turn to obtain a corresponding filtered image, and transmit the filtered image to the defogging operation module in turn.

[0044] The defogging operation module is used to calculate the defogging images of each group of image data in turn according to the atmospheric light estimation value of the previous frame image and multiple filtered images, and finally obtain the current frame image after defogging.

[0045] In this embodiment, the image data to be defogged in the input buffer module includes multiple frames of images arranged in time sequence, and can also be images acquired in real time by a camera device, and the currently acquired images are continuously input into the low-latency image defogging system to perform defogging in a timely manner. At the same time, each frame of the image is a three-channel image.

[0046] In the cache module, a filtering sliding window is used to traverse the current frame image, and the center of the filtering sliding window starts from the pixel of the first row and first column of the current frame image and moves to the right one column each time until the center of the filtering sliding window traverses all the pixels in the first row, then moves down one row, starting from the pixel of the second row and first column of the current frame image and moves to the right until the center of the filtering sliding window traverses all the pixels of the current frame image.

[0047] Whenever the filtering sliding window moves once in the current frame image, the image data in the filtering sliding window is input into the next module for processing. In this way, the filtering sliding window keeps moving in the current frame image, and the image data in the filtering sliding window is also kept transmitted to the subsequent modules for processing, which effectively improves the processing efficiency and reduces the delay.

[0048] Furthermore, the system includes a first memory, and each time the filtering sliding window moves, a group of image data is obtained, and each row of data is cached in the first memory by column. When the filtering sliding window moves one column to the right, a new group of image data is obtained, and the first memory replaces the row of data first cached in the first memory with the latest row of data.

[0049] Specifically, the first memory is a RAM memory, and data in a column in the filter sliding window is stored in the same address. Next, an example is given with an image resolution of 1920*1080 and a filter sliding window size of 9*9.

[0050] The data bit width of the RAM is preset to 9 in advance according to the column method, that is, the image data in the entire filter sliding window is stored in the RAM. When the filter sliding window moves one column to the right, the column of data on the right outside the previous filter sliding window will be moved into the current filter sliding window, and the leftmost column of data in the previous filter sliding window will be moved out if it is in the current filter sliding window. Similarly, in the RAM memory, when new data is input in the 10th row, 9 vertical data are read out from the RAM memory at a time. After reading the 9 data, it will be shifted upward, and the data of the 10th row of the new data will be shifted up by one row. The data of the first row will be removed and stored in the RAM memory again after the shift and replacement is completed. This design makes the RAM memory not need to have much memory, so as to reduce the consumption of memory.

[0051] In one embodiment, the data bit width of the RAM is the total bit width of 9 data, and the depth of the RAM is greater than or equal to the data depth of one row.

[0052] In one embodiment, the first memory adopts a data cache method of data width merging, which is based on Figure 2 The data bit width merging method shown is that the original pixel width is 24 bits. If a 9*9 filter window is used, the width after splicing is 24*9 bits.

[0053] like Figure 2 As shown, when data C4 is input, the value (A3-A4) of the column where A4 is located is read out from the first memory by column, and after processing, the values ​​of A3, C4, and C0 are transmitted to the subsequent module.

[0054] In this embodiment, when the cache module outputs image data to the atmospheric light calculation module and the guided filtering module, it actually transmits the pixel data on the column where the data of each group of image data is located in sequence to the guided filtering module for guided filtering processing, and transmits the central pixel data of each group of image data to the atmospheric light calculation module for calculating the histogram area and grayscale statistical value of the current frame image.

[0055] like Figure 3As shown in the figure, the initial position of the filter sliding window and the current frame image, the center of the filter sliding window, for a 9*9 sliding window, the center position is at the 5th row and the 5th column, and the center position is the image pixel of the first row and the first column of the current frame image, and the data output by the current cache module is the data of the first row on the center column (A3), the data of the 10th row (C4), and the data of the 5th row (C0) (in the following description, these three data are also used for description). That is to say, the data output by the cache image each time is the image pixel data corresponding to the center position of the current filter sliding window and the data related to the data. Specifically, the cache module reads the column data from the first memory.

[0056] Data (A3), data (C4) and data (C0) are transmitted to the guided filtering module, and data (C0), that is, the image pixel data corresponding to the center position of the current filtering sliding window, is transmitted to the atmospheric light calculation module.

[0057] In the prior art, the system that generally causes delay in the image dehazing system is the guided filtering module. In the present embodiment, in order to further achieve low delay, the processing delay of the key delay module, which is the guided filtering module, is designed in the following manner due to the large size of its filtering sliding window.

[0058] like Figure 4 As shown, the guided filtering module includes a dark channel calculation unit, a guided filtering processing unit and a band-guided filtering calculation unit which are connected in sequence.

[0059] The dark channel calculation unit is used to sequentially calculate multiple groups of dark channel data of pixel data on the central column of each group of image data, and sequentially transmit the multiple groups of dark channel data to the guided filtering processing unit.

[0060] The guided filter processing unit is used to perform mean operation on each group of image data in turn according to multiple groups of dark channel data, and transmit the mean operation result of each group of image data to the guided filter calculation unit. The two guided filter calculation units respectively calculate the dark channel filter and the dark channel square filter to obtain the covariance required for dehazing calculation.

[0061] The guided filter calculation unit is used to calculate the two parameters ak and bk by combining the mean operation results of the dark channel filtering and the dark channel square image data with the dark channel calculation unit. Since the input image I and the guided graph p are the same, the ak in each filter window is the variance of the input image plus the variance ∈∈, and b is equal to (1-ak) multiplied by the average value of the original image in the window. After obtaining ak and bk, it can be calculated according to q i =a k I i +b k , Compute the corresponding filtered image.

[0062] Specifically, in the dark channel calculation unit, since the image is a three-channel image, the pixel data of each image is 24 bits. The darkest channel in the pixel data of each image is retained through the calculation of the dark channel calculation unit, that is, the pixel data of each image is converted into one-channel 8-bit data.

[0063] Specifically, when the guided filtering processing unit performs mean operation on each group of image data, it includes: calculating the accumulated value of each group of image data in turn according to each group of dark channel data, calculating the corresponding effective mean value according to the accumulated value of each group of image data, and replacing the central pixel of each group of image data with each effective mean value accordingly.

[0064] When calculating the accumulated value of each group of image data, the bounding box accumulated value is used. Specifically, because the sliding window of the guided filter, that is, the bounding box, is too large, the mean operation is not suitable for directly adding up all the values ​​to get the average value. In this application, the algorithm for accumulating the value is as follows:

[0065] like Figure 5 As shown, taking the bounding box size of 9*9 as an example, with C0 as the center, C1, C2, C3, C4 (where, Figure 5 The data represented by the letters in Figure 3 The data represented by the mother and son do not have any connection) is the bounding box with the vertex that needs to be accumulated value calculated at present, and the bounding boxes of B1, B2, B3, and B4 with B0 as the center, and the bounding boxes of A1, A2, A3, and A4 with A0 as the center have been calculated before and the accumulated values ​​have been obtained.

[0066] In this way, the cumulative value of the C bounding box = the cumulative value of the B bounding box + the cumulative value of the column where C4 is located - the cumulative value of the column where B2 is located, and the cumulative value of the column where C4 is located (C3~C4) = the cumulative value of the column where A4 is located (A3~A4) + C4 value-A3 value.

[0067] In order to realize the above calculation, two memories are designed in the guided filtering processing unit, namely ram_col and ram_col2 for storing row and column accumulated values.

[0068] When the values ​​of data A3, C4, and C0 are input, the accumulated value of the column where A4 is located is read from ram_col, and the accumulated value of the column where C4 is located is calculated. At the same time, A3 is eliminated, and the value of the column where C4 is located (C3-C4) is stored in the first memory. The accumulated value of the column where C4 is located is stored in ram_col, replacing the accumulated value of the column where A4 is located, for use in the next row. The accumulated value of the column where C4 is located is stored in ram_col2, and the accumulated value of the column where B2 is located is read from ram_col2. Finally, the accumulated value of the bounding box centered on point C0 is calculated.

[0069] After calculating the cumulative value of the bounding box, the effective mean value of the bounding box centered at point C0 = the cumulative value of the C bounding box / N. Generally speaking, N is the number of pixels in the box, taking 9*9 as an example, which is 81. However, when the filter sliding window moves in the current frame image, the data outside the boundary of the current image are invalid data, and the total number of valid pixels N of the bounding box is not a fixed value at the boundary ((2r+1)*(2r+1)), where r is the radius of the filter sliding window. Therefore, it is necessary to calculate the total number of valid pixels N of the bounding box within the boundary point.

[0070] In this application, by adopting a method similar to the implementation of the bounding box accumulation value, the accumulated value of the number of pixels outside the boundary is set to 0, the valid flag flag inside the boundary is set to 1, and the flag value inside the bounding box is accumulated to obtain the total valid pixel number N value of the bounding box, and the bounding box boundary map is obtained, such as Figure 6 shown.

[0071] In this embodiment, a FIFO memory connected between the dark channel calculation unit and the guided filtering unit is further provided in the guided filtering module, and the FIFO memory is used to store the dark channel data obtained by calculating the image pixel data corresponding to the center position of the current filtering sliding window in the dark channel calculation unit. In the last guided filtering calculation unit, the filtered images corresponding to each group of image data are obtained in turn according to the mean operation results of each group of image data combined with the dark channel data in the FIFO memory.

[0072] In the atmospheric light calculation module, the calculation is performed according to the pixel data corresponding to the center position of the filter sliding window after each movement. After the filter sliding window traverses the current frame image, the atmospheric light calculation module can calculate the histogram area and grayscale statistics of the current frame image, and then transmit the histogram area and grayscale statistics of the current frame image to the data processing module. The data processing module calculates the atmospheric estimation value of the current frame. Therefore, when calculating the dehazed image of the current frame image, the atmospheric estimation value calculated for the previous frame image is used for calculation.

[0073] Specifically, in the atmospheric light calculation module, in order to calculate the average value of the brightest 0.1% of pixels, we can start from the highest pixel value and calculate the histogram area and grayscale statistics until the accumulated statistics are greater than or equal to 0.1% of pixels. Then we can estimate the atmospheric value (S / N), such as Figure 7 shown.

[0074] In the data processing module, this module is a software processing module. The software calculates the atmospheric light estimation value according to the atmospheric light statistical results. There are interfaces with the defogging operation module and the atmospheric light calculation module, which are used to write the defogging calculation results and obtain the statistical results respectively.

[0075] In one embodiment, the data processing module adopts an ARM processor.

[0076] In this embodiment, the system further includes a second memory, the cache module transmits the central pixel data of each group of image data to the second memory, and the second memory transmits the central pixel data of each group of image data to the defogging operation module. The defogging operation module sequentially calculates the defogging image of each group of image data according to the atmospheric light estimation value of the previous frame image, multiple filtered images and the central pixel data corresponding to each filtered image.

[0077] In one embodiment, the second memory is a FIFO memory.

[0078] The above-mentioned FPGA-based low-latency image defogging system proposes a method of storing the data in FPGA in RAM according to the bit width set, and finds the guided filter module with the greatest impact on the delay by analyzing the delay of the algorithm. By adopting a special design of the module data flow and the guided filter module, it not only ensures low-latency calculation, but also the minimum delay can reach the guided filter radius r cache delay, while reducing the consumption of multipliers and block RAM. This application is significantly superior to other processing delays based on PC or embedded software and ASIC processing chips in terms of delay indicators, and can meet the requirements of strict real-time systems, such as cruise missiles, high-speed drones, etc., and has a wide range of application value.

[0079] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0080] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. The low-latency image defogging system based on FPGA is characterized by: The system comprises: a cache module, an atmospheric light calculation module, a guide filter module, a defogging calculation module and a data processing module; The cache module is used to traverse the current frame image of the input image data to be defogged through a filter sliding window of a preset size, generate a set of image data according to the filter sliding window after each movement, and transmit the sequentially generated image data to the atmospheric light calculation module and the guide filtering module; The atmospheric light calculation module is used to perform statistical processing on each group of image data in the current frame image in turn, obtain the histogram area and grayscale statistical value of a frame image, and transmit them to the data processing module; The data processing module is used to calculate the atmospheric light estimation value of a frame of image according to the histogram area and grayscale statistics of the frame of image, and transmit the atmospheric light estimation value to the defogging operation module; The guided filtering module is used to sequentially perform guided filtering on each group of the image data to obtain a corresponding filtered image, and sequentially transmit the filtered image to the defogging operation module; The defogging operation module is used to calculate the defogging images of each group of image data in sequence according to the atmospheric light estimation value of the previous frame image and the plurality of filtered images, and finally obtain the defogging current frame image; The system comprises a first memory, wherein a set of image data obtained by each movement of the filtering sliding window is cached in columns for each row of data into the memory; When the filtering sliding window moves one column to the right to obtain a new set of image data, the first memory replaces the row of data first cached in the first memory with the latest row of data; The first memory adopts a data cache method of combining data bit width.

2. The low-latency image defogging system according to claim 1, characterized in that: The center of the filtering sliding window starts from the pixel of the first row and first column of the current frame image and moves to the right one column at a time until the center of the filtering sliding window traverses all the pixels in the first row, then moves down one row, and moves to the right starting from the pixel of the second row and first column of the current frame image until the center of the filtering sliding window traverses all the pixels of the current frame image.

3. The low-latency image defogging system according to claim 1, characterized in that: In the cache module, the pixel data on the column where the data center of each group of image data is located is transmitted to the guided filtering module for guided filtering processing, and the central pixel data of each group of image data is transmitted to the atmospheric light calculation module to calculate the histogram area and grayscale statistical value of the current frame image.

4. The low-latency image defogging system according to claim 1, characterized in that: The guided filtering module includes a dark channel calculation unit, a guided filtering processing unit and a band-guided filtering calculation unit connected in sequence; The dark channel calculation unit is used to sequentially calculate multiple groups of dark channel data of the pixel data on the columns where the image data centers are located, and transmit the multiple groups of dark channel data to the guided filtering processing unit; The guided filtering processing unit is used to perform mean operation on each group of image data in turn according to the multiple groups of dark channel data, and transmit the mean operation result of each group of image data to the guided filtering calculation unit; The band-oriented filtering calculation unit is used to obtain the filtered images corresponding to each group of image data in turn according to the mean calculation results of each group of image data combined with the dark channel data calculated by the dark channel calculation unit.

5. The low-latency image defogging system according to claim 4, characterized in that: The guided filtering processing unit performs mean value calculation on each group of image data, including: Calculate the accumulated value of each group of image data in sequence according to each group of dark channel data; The corresponding effective mean values ​​are calculated according to the accumulated values ​​of each group of image data, and the central pixels of each group of image data are replaced by each effective mean value accordingly.

6. The low-latency image defogging system according to claim 5, characterized in that: When calculating the accumulated value of each group of image data, the bounding box accumulated value is used; When calculating the effective mean, it is necessary to remove invalid data of each group of image data that is outside the boundary of the current frame image.

7. The low-latency image defogging system according to claim 6, characterized in that: The image defogging system further includes a second memory, and the cache module transmits the central pixel data of each group of image data to the second memory; The second memory transmits the central pixel data of each group of image data to the defogging operation module; The defogging operation module sequentially calculates the defogging images of each group of image data according to the atmospheric light estimation value of the previous frame image, the multiple filtered images and the central pixel data corresponding to each filtered image.

Images