Image defogging method and device

Abstract

The present invention provides an image defogging method and device. The image defogging method comprises: obtaining the current frame image of the image to be defogged; obtaining the Gaussian filter image corresponding to the image to be defogged; the Gaussian filter image is generated by the down-sampled image corresponding to the image to be defogged; interpolating the Gaussian filter image to obtain an interpolated image; performing bilateral filtering on the current frame image according to the interpolated image to obtain transmittance; and generating a defogged image corresponding to the current frame image according to the transmittance. The image defogging method only needs to store the Gaussian filter result obtained after the image to be defogged is down-sampled, which is conducive to reducing storage overhead.

Description

Technical Field

[0001] The invention belongs to the field of image processing and relates to a defogging method, in particular to an image defogging method and device. Background Art

[0002] Haze is a common atmospheric phenomenon caused by small floating particles such as dust and smoke in the air. These floating particles greatly absorb and scatter light, resulting in image degradation. Under the influence of haze, many practical applications such as video surveillance, remote sensing, and autonomous driving are easily threatened, and advanced computer vision tasks such as detection and recognition are difficult to complete. Therefore, image dehazing (defog) has become an increasingly important technology.

[0003] The defogging method based on bilateral filtering is a commonly used image defogging method. The defogging method based on bilateral filtering adopts a weighted average method, and uses the weighted average value of the dark channel of the surrounding pixels to represent the fog concentration of a certain pixel. However, when the defogging method based on bilateral filtering is implemented in hardware, it is necessary to store the relevant information of the entire frame image and the Gaussian filtering result, which will lead to an increase in storage resources. Summary of the invention

[0004] In view of the above-mentioned shortcomings of the prior art, an object of the present invention is to provide an image defogging method and device, which are used to solve the problem of excessive storage resource overhead in the existing image defogging methods.

[0005] To achieve the above-mentioned purpose and other related purposes, the present invention provides an image defogging method. The image defogging method comprises: obtaining a current frame image of the image to be defogged; obtaining a Gaussian filter image corresponding to the image to be defogged; the Gaussian filter image is generated by a downsampled image corresponding to the image to be defogged; interpolating the Gaussian filter image to obtain an interpolated image; performing bilateral filtering on the current frame image according to the interpolated image to obtain transmittance; and generating a defogged image corresponding to the current frame image according to the transmittance.

[0006] In one embodiment of the present invention, the Gaussian filtered image corresponding to the image to be defogged is the Gaussian filtered image corresponding to the current frame image.

[0007] In one embodiment of the present invention, the Gaussian filter image corresponding to the image to be defogged is a Gaussian filter image corresponding to the previous frame image.

[0008] In one embodiment of the present invention, the image defogging method also includes: downsampling the current frame image to obtain a downsampled image corresponding to the current frame image; processing the downsampled image corresponding to the current frame image to generate a Gaussian filtered image corresponding to the current frame image; the Gaussian filtered image corresponding to the current frame image is used for defogging the next frame image.

[0009] In one embodiment of the present invention, the image defogging method further includes: reading the Gaussian filter image corresponding to the previous frame image from a first buffer; and storing the Gaussian filter image corresponding to the current frame image in the first buffer.

[0010] In one embodiment of the present invention, for any target pixel in the current frame image, a first Gaussian filter matrix corresponding to the target pixel and its surrounding pixels is obtained; the first Gaussian filter matrix is ​​interpolated to obtain a first interpolation matrix; the first interpolation matrix is ​​expanded to obtain an interpolation image block corresponding to the target pixel; and the interpolation image is obtained based on the interpolation image blocks corresponding to all target pixels.

[0011] The present invention also provides an image defogging device. The image defogging device includes: an image acquisition module, which is used to acquire the current frame image of the image to be defogged; a filter value acquisition module, which is used to acquire the Gaussian filter image corresponding to the image to be defogged; the Gaussian filter image is generated by the down-sampled image corresponding to the image to be defogged; an interpolation module, which is connected to the filter value acquisition module, and is used to interpolate the Gaussian filter image to obtain an interpolated image; a bilateral filtering module, which is connected to the image acquisition module and the interpolation module, and is used to perform bilateral filtering on the current frame image according to the interpolation image to obtain transmittance; a defogging module, which is connected to the bilateral filtering module, and is used to generate a defogged image corresponding to the current frame image according to the transmittance; a control module, which is respectively connected to the image acquisition module, the filter value acquisition module, the interpolation module, the bilateral filtering module and the defogging module, and is used to control the image defogging device.

[0012] In one embodiment of the present invention, the Gaussian filtered image corresponding to the image to be defogged is the Gaussian filtered image corresponding to the current frame image.

[0013] In one embodiment of the present invention, the Gaussian filter image corresponding to the image to be defogged is a Gaussian filter image corresponding to the previous frame image.

[0014] In one embodiment of the present invention, the image defogging device also includes: a downsampling module, connected to the image acquisition module, for downsampling the current frame image to obtain a downsampled image corresponding to the current frame image; a Gaussian filtering module, connected to the downsampling module, for processing the downsampled image corresponding to the current frame image to generate a Gaussian filtered image corresponding to the current frame image; the Gaussian filtered image corresponding to the current frame image is used for defogging the next frame image.

[0015] In one embodiment of the present invention, the image defogging device also includes: a first buffer, connected to the filter value acquisition module and the Gaussian filter module; the filter value acquisition module reads the Gaussian filter image corresponding to the previous frame image from the first buffer; the Gaussian filter module stores the Gaussian filter image corresponding to the current frame image in the first buffer.

[0016] As described above, the image defogging method and device described in the present invention have the following beneficial effects:

[0017] The Gaussian filter image that needs to be stored in the image defogging method is generated after the current frame image is downsampled, and there is no need to store the Gaussian filter result and statistical information of the entire image, which is conducive to reducing storage overhead and saving storage resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Shown is a flow chart of an image defogging method according to a specific embodiment of the present invention.

[0019] Figure 2 Shown is a flow chart of an image defogging method according to a specific embodiment of the present invention.

[0020] Figure 3 Shown is a schematic diagram of downsampling and Gaussian filtering in a specific embodiment of the image defogging method of the present invention.

[0021] Figure 4 Shown is a flow chart of an image defogging method according to a specific embodiment of the present invention.

[0022] Figure 5A Shown is a flow chart of an interpolation method in a specific embodiment of the image defogging method of the present invention.

[0023] Figure 5B FIG. 4 is a schematic diagram showing interpolation of the image defogging method according to a specific embodiment of the present invention.

[0024] Figure 5C FIG. 4 is a schematic diagram showing interpolation of the image defogging method according to a specific embodiment of the present invention.

[0025] Figure 5D Shown is a flow chart of obtaining a first Gaussian filter matrix in a specific embodiment of the image defogging method of the present invention.

[0026] Fig. 6A Shown is a schematic diagram of writing into a buffer area in a specific embodiment of the image defogging method of the present invention.

[0027] Figure 6B Shown is a schematic diagram of buffer area reading in a specific embodiment of the image defogging method of the present invention.

[0028] Figure 6C Shown is a schematic diagram of buffer area reading in a specific embodiment of the image defogging method of the present invention.

[0029] Fig.6D Shown is a schematic diagram of buffer area reading in a specific embodiment of the image defogging method of the present invention.

[0030] Fig. 6E Shown is a schematic diagram of buffer area reading in a specific embodiment of the image defogging method of the present invention.

[0031] Fig. 6F Shown is a schematic diagram of buffer area reading in a specific embodiment of the image defogging method of the present invention.

[0032] Figure 7 Shown is a structural diagram of the image defogging device according to a specific embodiment of the present invention.

[0033] Component number description

[0034] 700 Image Defogging Device

[0035] 710 Image acquisition module

[0036] 720 Filter value acquisition module

[0037] 730 Interpolation Module

[0038] 740 Bilateral Filter Module

[0039] 750 Defogging Module

[0040] 760 Downsampling Module

[0041] 770 Gaussian filter module

[0042] 780 First Buffer

[0043] 790 Second Buffer

[0044] Steps S11 to S15

[0045] Steps S11a to S17a

[0046] Steps S11b to S15b

[0047] Steps S31 to S33

[0048] Steps S51 to S54

[0049] Steps S511 to S513 DETAILED DESCRIPTION

[0050] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0051] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0052] The defogging method based on bilateral filtering is a commonly used image defogging method. The defogging method based on bilateral filtering adopts a weighted average method, and uses the weighted average value of the dark channels of the surrounding pixels to represent the fog concentration of a certain pixel. Its disadvantage is that if only adjacent pixels are used for the bilateral filtering algorithm, when there is a boundary with obvious changes, block effects and bad pixels may appear, and the storage resources will be relatively large. To address this problem, the existing defogging method based on bilateral filtering tends to select the dark channels of pixels in a larger range around the pixel for weighted averaging, for example, the dark channels of pixels in the range of 80×80 around the pixel are selected for weighted averaging. However, when the defogging method based on bilateral filtering is implemented in hardware, it is necessary to store the relevant information of the entire frame image and the Gaussian filtering results, which will lead to an increase in storage resources.

[0053] In view of the above problems, the present invention provides an image defogging method, which includes: obtaining the current frame image of the image to be defogged; obtaining the Gaussian filter image corresponding to the image to be defogged; the Gaussian filter image is generated by the downsampled image corresponding to the image to be defogged; interpolating the Gaussian filter image to obtain an interpolated image; performing bilateral filtering on the current frame image according to the interpolated image to obtain transmittance; generating a defogged image corresponding to the current frame image according to the transmittance. The image defogging method of the present invention only needs to store the Gaussian filter value obtained after downsampling the current frame image, which reduces storage overhead and is conducive to saving storage resources.

[0054] See also Figure 1 In one embodiment of the present invention, the image defogging method includes:

[0055] S11, obtaining a current frame image of the image to be defogged; the image to be defogged is a video image captured by an image acquisition device or a video acquisition device on a foggy or hazy day, and the image to be defogged includes at least one frame image.

[0056] S12, obtaining a Gaussian filter image corresponding to the image to be defogged; the Gaussian filter image is generated by a downsampled image corresponding to the image to be defogged. Preferably, the Gaussian filter image is stored in a memory, and step S12 can directly read the Gaussian filter image from the memory. The Gaussian filter image can be represented by Gaussian filter values ​​of different pixels in the image.

[0057] S13, interpolating the Gaussian filtered image to obtain an interpolated image.

[0058] S14, performing bilateral filtering on the current frame image according to the interpolation image to obtain a transmittance t(x).

[0059] S15, generating a defogging image corresponding to the current frame image according to the transmittance. The formula for generating a defogging image corresponding to the current frame image according to the transmittance is: Wherein, J(x) is the defogging image, I(x) is the image to be defogged, and A is the atmospheric light value.

[0060] The atmospheric light value A can be configured by the user, or obtained by performing bilateral filtering on the previous frame of the image, or obtained by processing the dark channel. The method for obtaining the atmospheric light value by processing the dark channel includes: taking the top 0.1% of the pixel points according to the brightness from the dark channel image, and obtaining the positions of these pixel points; according to these positions, searching for the corresponding value of the point with the highest brightness in the image to be defogged I(x) as the A value.

[0061] In this embodiment, the image defogging method only needs to store the Gaussian filter value obtained after downsampling the current frame image, which reduces storage overhead and is conducive to saving storage resources.

[0062] In one embodiment of the present invention, the Gaussian filter image corresponding to the image to be defogged is the Gaussian filter image corresponding to the current frame image. Figure 2 , the image defogging method includes S11a to S17a. In this embodiment, the method for obtaining the Gaussian filter image corresponding to the current frame image includes:

[0063] S12a, downsample the current frame image to obtain a downsampled image corresponding to the current frame image. Downsampling is the process of reducing an image. For an image with an original size of a×b, downsample it by S times to obtain an image with a size of (a / S)×(b / S). Wherein, S is a positive integer; preferably, S is an integer power of 2 to increase the speed of hardware processing; further preferably, S=16.

[0064] S13a, processing the downsampled image corresponding to the current frame image to generate a Gaussian filter image corresponding to the current frame image. Preferably, the Gaussian filter image corresponding to the current frame image is stored in a memory, and step S14a can directly read the Gaussian filter image corresponding to the current frame image from the memory as the Gaussian filter image corresponding to the image to be defogged. In this embodiment, step S13a only needs to store the Gaussian filter value obtained after the downsampling process, and less storage space is required.

[0065] See also Figure 3 , taking the current frame image BLK with a size of 96×96 as an example, the process of downsampling and Gaussian filtering is as follows:

[0066] The image to be defogged BLK is divided into a plurality of 16×16 pixel blocks, such as blk_00 to blk55; at this time, the downsampling multiple S=16;

[0067] Each pixel block is downsampled into a pixel point, and then a downsampled image is obtained; for example, for a 16×16 pixel block blk_05, it is downsampled to obtain a pixel point s05. Downsampling a pixel block can be achieved using existing technologies, for example, by calculating the weighted average of all the pixels in the pixel block as the downsampled value corresponding to the pixel block;

[0068] Perform Gaussian filtering on each pixel in the downsampled image to obtain the corresponding Gaussian filtered image. Each pixel in the sampled image corresponds to a Gaussian filter value, for example, pixel s05 corresponds to Gaussian filter value g05 and pixel s15 corresponds to Gaussian filter value g15. The Gaussian filter values ​​corresponding to all pixels are combined together to obtain the Gaussian filter value corresponding to the image BLK, and then the Gaussian filter image corresponding to the image BLK can be obtained.

[0069] The Gaussian filter value used in the image defogging method described in this embodiment is generated by the pixel points of the current frame image, which can truly reflect the overall condition of the current frame image, so the defogging effect is better. However, under ideal conditions, steps S12a to S13a in this embodiment require information of all pixel points of the current frame image, so steps S12a to S13a need to be performed after all pixel points in the entire frame image are loaded. In addition, since steps S14a to S17a depend on the Gaussian filter image obtained in step S13a, steps S14a to S17a need to be performed after step S13a is executed. For the current frame image, the single-frame defogging time of the image defogging method described in this embodiment is relatively long, which is easy to cause stuttering when the frame rate is high; wherein, the single-frame defogging time is the time required for defogging a single frame image. Therefore, the image defogging method described in this embodiment is suitable for scenes where the defogging time requirement is not high and / or the difference between two consecutive frames of images is large.

[0070] See also Figure 4 In one embodiment of the present invention, the image defogging method includes steps S11b to S15b. In this embodiment, the Gaussian filter image corresponding to the image to be defogged is the Gaussian filter image corresponding to the previous frame image. At this time, the image to be defogged includes at least two frames of images.

[0071] Preferably, the Gaussian filter image corresponding to the previous frame image is stored in a memory; step S12b only needs to read the Gaussian filter image corresponding to the previous frame image from the memory to obtain the Gaussian filter image corresponding to the image to be defogged, and then steps S13b to S15b can be executed to obtain the defogged image corresponding to the current frame image. Therefore, in the image defogging method described in this embodiment, steps S13b to S15b do not need to wait until all pixels in the current frame are loaded, nor do they need to wait until the Gaussian filter image corresponding to the current frame is acquired. Therefore, the single-frame defogging time of the image defogging method described in this embodiment is short, which can effectively avoid the occurrence of stuttering.

[0072] The Gaussian filter value used in the image defogging method of this embodiment is generated by the pixel points of the previous frame image: when the previous frame image and the current frame image are not much different, the Gaussian filter value can better reflect the status of the current frame, and the defogging effect is good. Therefore, the image defogging method of this embodiment is particularly suitable for scenes such as monitoring.

[0073] In this embodiment, the method for obtaining the Gaussian filter image corresponding to the previous frame image includes: downsampling and Gaussian filtering the previous frame image to obtain the Gaussian filter image corresponding to the previous frame image. The specific process can be referred to in Figure 3 .

[0074] In one embodiment of the present invention, the image defogging method further includes:

[0075] S31, down-sampling the current frame image to obtain a down-sampled image corresponding to the current frame image;

[0076] S32, processing the down-sampled image corresponding to the current frame image to generate a Gaussian filtered image corresponding to the current frame image; the Gaussian filtered image corresponding to the current frame image is used for defogging the next frame image.

[0077] Preferably, steps S31 to S32 and steps S11b to S15b are performed simultaneously, so downsampling and Gaussian filtering of the current frame image will not affect the single-frame defogging duration of the current frame image, thereby avoiding the occurrence of a freeze phenomenon.

[0078] Similarly, the Gaussian filter image corresponding to the previous frame image can be obtained by downsampling and Gaussian filtering the previous frame image; the specific method is referred to steps S31 to S32, which will not be described in detail here.

[0079] In an embodiment of the present invention, the image defogging method further includes: S33, storing the Gaussian filter image corresponding to the current frame image in a memory. Preferably, the memory is a first buffer.

[0080] For a frame of image with size a×b, the storage space of the first buffer only needs to be larger than ceil(a / S)×ceil(b / S) data blocks, where S is the downsampling multiple, the bit width of each data block is 8 bits, and ceil() is the rounding function. For example, for a frame of 1920×1080 image, directly storing the Gaussian filtered image corresponding to the original image requires storing 253.125Kbyte of data; in this embodiment, after 16 times downsampling and then Gaussian filtering, the data to be stored is about 8Kbyte.

[0081] In order to further reduce the cost of storage resources, the Gaussian filter image corresponding to the previous frame image is also stored in the first buffer. At this time, it must be ensured that the write operation of step S33 to the first buffer will not affect the read operation of step S12b to the first buffer, that is, only after step S12b has finished reading the Gaussian filter image corresponding to the previous frame image, step S33 can perform the write operation of the Gaussian filter image corresponding to the current frame image.

[0082] To achieve this goal, the Gaussian filter image corresponding to the current frame image in step S33 can be first stored in the second buffer, and the data in the second buffer can be stored in the first buffer after the data at the corresponding position of the first buffer is read in step S12b. Preferably, the capacity of the second buffer is much smaller than that of the first buffer. For example, the second buffer can be a 4-line buffer, that is, the write operation of the image defogging method to the first buffer is 4 lines slower than the read operation.

[0083] In this embodiment, the introduction of the first buffer is conducive to reducing the single-frame defogging time to avoid freezing. At this time, step S12b only needs to perform a read operation of the buffer to obtain the Gaussian filter image corresponding to the previous frame image. The introduction of the second buffer is conducive to avoiding read-write conflicts and ensuring a good defogging effect.

[0084] See also Figure 5A In one embodiment of the present invention, the Gaussian filter image is interpolated to obtain an interpolated image, and the method includes:

[0085] S51, for any target pixel in the current frame image, obtain a first Gaussian filter matrix corresponding to the target pixel and its surrounding pixels; the target pixel refers to a pixel to be defogged. In particular, all pixels in the current frame image are target pixels.

[0086] S52, interpolating the first Gaussian filter matrix to obtain a first interpolation matrix; the interpolation method may be the nearest neighbor interpolation, bilinear interpolation, mean interpolation, median interpolation, etc. in the existing methods, which are not limited here. Figure 5B , interpolate the first Gaussian filter matrix G to obtain the first interpolation matrix PD. Among them, pd00 is obtained by interpolating the four data g00, g01, g10 and g11, pd01 is obtained by interpolating g01, g02, g11 and g12, and so on.

[0087] S53, expanding the first interpolation matrix to obtain an interpolation image block corresponding to the target pixel point; the size of the interpolation image block is B×C; wherein B and C are both positive integers, indicating that in the bilateral filtering-based defogging method, the dark channels of pixels within the range of B×C around the pixel point are selected for weighted averaging.

[0088] See also Figure 5C In one embodiment of the present invention, B=80, C=80, and the downsampling factor S=16. It should be noted that: Figure 5C The size of the interpolation image block is 80×80 pixels, which is divided into a plurality of sub-blocks sub00 to sub44, each of which has a size of 16×16 pixels; Figure 5COnly the leftmost and topmost pixels of each sub-block are shown for example.

[0089] Taking the sub-block sub00 as an example, one method of expanding the first interpolation matrix is: all pixels in the sub-block sub00 use the value of pd00.

[0090] Taking sub-block sub00 as an example, another method for expanding the first interpolation matrix is: pd00 is used as the upper leftmost pixel of sub-block sub00. For any other pixel in the sub-block, a corresponding weight is set according to the position of the pixel, and the result obtained by multiplying the weight by pd00 is used as the value of the pixel.

[0091] S54, obtaining the interpolated image according to the interpolated image blocks corresponding to all target pixels. Specifically, each target pixel corresponds to an interpolated image block, and the interpolated image blocks corresponding to all target pixels are merged together to obtain the interpolated image.

[0092] In this embodiment, assuming that the image size is 1920×1080, if the interpolated image is directly stored, 1920×1080 data needs to be stored. Preferably, only the Gaussian filter image, that is, the data before interpolation, is stored in the memory; and when the interpolated image is needed, the corresponding Gaussian filter image can be read out for real-time interpolation.

[0093] See also Figure 5D and Figure 3 A method for implementing step S51 includes:

[0094] S511, determine a target pixel block; the target pixel block refers to a pixel block in the current frame image that contains the target pixel point, for example Figure 3 blk_32 in;

[0095] S512, determining a target Gaussian filter value corresponding to the target pixel block; the target Gaussian filter value refers to a Gaussian filter value corresponding to the target pixel block, for example Figure 3 The target Gaussian filter value corresponding to blk_32 is g32;

[0096] S513, selecting a matrix containing the target pixel and having a size of (B / S+1)×(C / S+1) from the Gaussian filter image corresponding to the current frame image as the first Gaussian filter matrix; the first Gaussian filter matrix is, for example Figure 3 The target Gaussian filter value may be located at any position of the first Gaussian filter matrix. Preferably, the target Gaussian filter value is located at the center of the first Gaussian filter matrix.

[0097] The image defogging method of the present invention saves storage resources by downsampling the current frame image, which inevitably leads to information loss during the downsampling process. Since the defogging method based on bilateral filtering needs to select the dark channels of pixels within the B×C range around the pixel point for weighted averaging, for example: B=80, C=80. The larger the value of B×C, the greater the impact of this information loss on the defogging effect. To solve this problem, this embodiment uses the interpolation scheme described in S51~S53 to restore the Gaussian filtered image after downsampling, thereby ensuring that the defogging method based on bilateral filtering can achieve better defogging effect; the interpolation scheme described in this embodiment is particularly suitable for scenes with large B×C values.

[0098] In one embodiment of the present invention, the size of the current frame image is 240×176, the downsampling multiple S=16, and the defogging method based on bilateral filtering needs to select the dark channel of the pixels within the range of B×C around the pixel point for weighted averaging. Fig. 6A , the Gaussian filter image size corresponding to the current frame image is 15×11, and the first buffer needs to store the Gaussian filter image. In this embodiment, B / S+1=6, so when writing to the buffer, 6 lines of data are written each time, specifically as follows: first write the first 6 lines of data g00~g5e, and after the current 6 lines of data are written, write the next 6 lines of data g60~gbe. For images of larger size, write 6 lines of data each time in this order until all data in the Gaussian filter image are written.

[0099] See also Figure 6B to Figure 6F , which shows the reading process of the first buffer. For interpolation, a 5×5 first interpolation matrix needs to be obtained each time, and each point in the first interpolation matrix needs 2 Gaussian filter values ​​to generate, so a 6×6 Gaussian filter value needs to be read each time. Figure 6B , first read g00~g52 and expand the left border three times to form a 6×6 Gaussian filter matrix; please refer to Figure 6C , read g00~g53 for the second time and expand the left border twice to form a 6×6 Gaussian filter matrix; please refer to Fig.6D , read g00~g54 for the third time and expand the left border once to form a 6×6 Gaussian filter matrix; please refer to Fig. 6E , read g00~g55 for the fourth time, and there is no need to expand the edge at this time; please refer to Fig. 6F, g01 to g56 are read for the fifth time, and no edge expansion is required at this time. Similarly, the corresponding Gaussian filter matrices are read from the first buffer in sequence until the Gaussian filter matrices corresponding to all pixels in the current frame image are read. When the left boundary, upper boundary, lower boundary and / or right boundary are encountered, the Gaussian filter value of the boundary needs to be expanded a certain number of times to ensure that the obtained Gaussian filter matrix is ​​6×6.

[0100] The present invention also provides an image defogging device. Figure 7 In one embodiment of the present invention, the image defogging device 700 includes:

[0101] The image acquisition module 710 is used to acquire the current frame image of the image to be defogged; the image to be defogged is a video image captured by an image acquisition device or a video acquisition device on a foggy or hazy day, and the video consists of at least one frame of image.

[0102] The filter value acquisition module 720 is used to acquire the Gaussian filter image corresponding to the image to be defogged; the Gaussian filter image is generated by the downsampled image corresponding to the image to be defogged; preferably, the Gaussian filter image is stored in a memory, and the filter value acquisition module 720 can directly read the Gaussian filter image from the memory. The Gaussian filter image can be represented by the Gaussian filter values ​​of different pixels in the image.

[0103] An interpolation module 730, connected to the filter value acquisition module, is used to interpolate the Gaussian filter image to obtain an interpolated image;

[0104] A bilateral filtering module 740, connected to the image acquisition module 710 and the interpolation module 730, for performing bilateral filtering on the current frame image according to the interpolation image to obtain a transmittance t(x);

[0105] The defogging module 750 is connected to the bilateral filtering module and is used to generate a defogging image corresponding to the current frame image according to the transmittance. The formula for generating the defogging image corresponding to the current frame image according to the transmittance is: Wherein, J(x) is the defogging image, I(x) is the image to be defogged, and A is the atmospheric light value. The atmospheric light value A can be configured by the user, or obtained by performing bilateral filtering on the previous frame of the image, or obtained by processing the dark channel. The method for obtaining the atmospheric light value by processing the dark channel includes: taking the top 0.1% of the pixel points according to the brightness from the dark channel image, and obtaining the positions of these pixel points; according to these positions, searching for the corresponding value of the point with the highest brightness in the image to be defogged I(x) as the A value.

[0106] A control module is connected to the image acquisition module 710 , the filter value acquisition module 720 , the interpolation module 730 , the bilateral filtering module 740 and the defogging module 750 , respectively, and is used to control the image defogging device 700 .

[0107] In this embodiment, the image defogging device only needs to store the Gaussian filter value obtained after downsampling, which reduces storage overhead and is conducive to saving storage resources.

[0108] In one embodiment of the present invention, the Gaussian filter image corresponding to the image to be defogged is the Gaussian filter image corresponding to the current frame image. At this time, the image defogging device 700 further includes:

[0109] The downsampling module 760 is connected to the image acquisition module 710 and is used to downsample the current frame image to obtain a downsampled image corresponding to the current frame image. Downsampling is the process of reducing an image. For an image with an original size of a×b, it is downsampled S times to obtain an image of size (a / S)×(b / S). Wherein, S is a positive integer; preferably, S is an integer power of 2 to increase the speed of hardware processing; further preferably, S=16.

[0110] The Gaussian filter module 770 is connected to the downsampling module 760, and is used to process the downsampling image corresponding to the current frame image to generate a Gaussian filter image corresponding to the current frame image. Preferably, the Gaussian filter image corresponding to the current frame image is stored in the memory in the form of a Gaussian filter value. The filter value acquisition module 720 can directly read the Gaussian filter image corresponding to the current frame image from the memory as the Gaussian filter image corresponding to the image to be defogged. In this embodiment, the image defogging device only needs to store the Gaussian filter value obtained after the downsampling process, and less storage space is required.

[0111] See also Figure 3 , taking the current frame image BLK with a size of 96×96 as an example, the process of downsampling and Gaussian filtering is as follows:

[0112] The downsampling module 760 first divides the defogging image BLK into a plurality of 16×16 pixel blocks, such as blk_00 to blk55; at this time, the downsampling multiple S=16. Thereafter, the downsampling module 760 performs downsampling processing on each pixel block, and each pixel block is downsampled into a pixel point, thereby obtaining a downsampled image; for example, for the 16×16 pixel block blk_05, it is downsampled to obtain the pixel point s05. Downsampling a certain pixel block can be achieved by calculating the weighted average of all the pixel points in the pixel block.

[0113] The Gaussian filter module 770 performs Gaussian filtering on each pixel in the downsampled image to obtain a corresponding Gaussian filtered image. Each pixel in the sampled image corresponds to a Gaussian filter value, for example, pixel s05 corresponds to Gaussian filter value g05. The Gaussian filter values ​​corresponding to all pixels are combined together to obtain the Gaussian filter value corresponding to the image BLK, and then the Gaussian filter image corresponding to the image BLK can be obtained.

[0114] The Gaussian filter value used in the image defogging device described in this embodiment is generated by the pixel points of the current frame image, which can truly reflect the overall condition of the current frame image, so the defogging effect is better. However, under ideal conditions, the downsampling module 760 in this embodiment requires information of all pixel points of the current frame image, so the downsampling processing of the image by the downsampling module 760 needs to be performed after all pixel points in the entire frame image are loaded. In addition, since the filter value acquisition module 720 depends on the Gaussian filter image obtained by the Gaussian filter module 770, the filter value acquisition module 720 needs to be performed after the Gaussian filter module 770 completes the Gaussian filter. Therefore, for the current frame image, the single-frame defogging time of the image defogging device described in this embodiment is relatively long, which is easy to cause stuttering when the frame rate is high. Therefore, the image defogging device described in this embodiment is suitable for scenes where the defogging time requirement is not high and / or the difference between two consecutive frames of images is large.

[0115] In one embodiment of the present invention, the Gaussian filtered image corresponding to the image to be defogged is the Gaussian filtered image corresponding to the previous frame image. Preferably, the Gaussian filtered image corresponding to the previous frame image is stored in a memory. At this time, the filter value acquisition module only needs to perform a read operation of the memory to obtain the Gaussian filtered image corresponding to the image to be defogged. Thereafter, the image defogging device can perform the defogging operation. Therefore, the single-frame defogging time of the image defogging device described in this embodiment is short, which can effectively avoid the occurrence of a jamming phenomenon.

[0116] The Gaussian filter value used in the image defogging device of this embodiment is generated by the pixel points of the previous frame image: when the previous frame image and the current frame image are not much different, the Gaussian filter value can better reflect the status of the current frame, and the defogging effect is good. Therefore, the image defogging device of this embodiment is particularly suitable for scenes such as monitoring.

[0117] In one embodiment of the present invention, the Gaussian filter image generated by the down-sampling module and the Gaussian filter module is used for defogging operation of the next frame of image.

[0118] In this embodiment, the filter value acquisition module acquires the Gaussian filter image corresponding to the previous frame image for defogging. Preferably, the downsampling module and the Gaussian filter module downsample and Gaussian filter the current image, and at the same time, the filter value acquisition module, the interpolation module, the bilateral filter module and the defogging module perform the defogging operation of the current frame image.

[0119] In one embodiment of the present invention, the image defogging device further includes a first buffer 780 for storing a Gaussian filter image corresponding to the current frame image.

[0120] For a frame of image with a size of a×b, the storage space of the first buffer 780 only needs to be greater than ceil(a / S)×ceil(b / S) data blocks, where S is the downsampling multiple and the bit width of each data block is 8 bits. For example, for a frame of 1920×1080 image, directly storing the Gaussian filter image corresponding to the original image requires storing 253.125Kbytes of data; in this embodiment, the data to be stored is about 8Kbytes.

[0121] In order to further reduce the storage resource overhead, the Gaussian filter image corresponding to the previous frame image is also stored in the first buffer. At this time, it must be ensured that the write operation of the Gaussian filter module 770 to the first buffer 780 will not affect the read operation of the filter value acquisition module 720 to the first buffer 780, that is, only after the filter value acquisition module 720 has read the Gaussian filter image corresponding to the previous frame image, the Gaussian filter module 770 can perform the write operation of the Gaussian filter image corresponding to the current frame image.

[0122] To achieve this goal, the Gaussian filter image corresponding to the current frame image generated by the Gaussian filter module 770 can be first stored in the second buffer 790, and after the filter value acquisition module 720 has read the data at the corresponding position of the first buffer 780, the data in the second buffer 790 is stored in the first buffer 780. Preferably, the capacity of the second buffer 790 is much smaller than that of the first buffer. For example, the second buffer can be a 4-line buffer. In this case, the write operation to the first buffer 780 is 4 lines slower than the read operation.

[0123] In this embodiment, the introduction of the first buffer 780 is conducive to reducing the single-frame defogging time. At this time, the filter value acquisition module 720 only needs to perform a read operation of the buffer to obtain the Gaussian filter image corresponding to the previous frame image. The introduction of the second buffer 790 is conducive to avoiding read-write conflicts, thereby ensuring a good defogging effect.

[0124] In one embodiment of the present invention, the interpolation module includes: a filter matrix acquisition unit, a first interpolation unit, a matrix expansion unit and a merging unit. Specifically:

[0125] The filter matrix acquisition unit is connected to the image acquisition module, and is used to obtain the first Gaussian filter matrix corresponding to any target pixel point in the current frame image and its surrounding pixels; in particular, all pixels in the current frame image are target pixels.

[0126] The first interpolation unit is connected to the filter matrix acquisition unit, and is used to interpolate the first Gaussian filter matrix to obtain a first interpolation matrix; the interpolation method can adopt the nearest neighbor interpolation, bilinear interpolation, mean interpolation, median interpolation and other methods in the existing methods, which are not limited here. Figure 5B , interpolating the first Gaussian filter matrix G to obtain a first interpolation matrix PD. Among them, pd00 is obtained by interpolating the four data g00, g01, g10 and g11, pd01 is obtained by interpolating g01, g02, g11 and g12, and so on.

[0127] The matrix expansion unit is connected to the first interpolation unit, and is used to expand the first interpolation matrix to obtain an interpolation image block corresponding to the target pixel point; the size of the interpolation image block is B×C; wherein B and C are both positive integers, indicating that when performing the dehazing based on bilateral filtering, the dark channels of pixels within the range of B×C pixels are selected for weighted averaging.

[0128] See also Figure 5C , where B = 80, C = 80, and the downsampling multiple S = 16. It should be noted that, Figure 5C The size of the interpolation image block is 80×80 pixels, which is divided into a plurality of sub-blocks sub00 to sub44, and the size of each sub-block is 16×16 pixels. Figure 5C Only the leftmost and topmost pixels of each sub-block are shown for example.

[0129] Taking the sub-block sub00 as an example, one method of expanding the first interpolation matrix is: all pixels in the sub-block sub00 use the value of pd00.

[0130] Taking sub-block sub00 as an example, another method for expanding the first interpolation matrix is: pd00 is used as the pixel point at the upper left corner of sub-block sub00. For any other pixel point, a corresponding weight is set according to the position of the pixel point, and the result obtained by multiplying the weight by pd00 is used as the value of the pixel point.

[0131] The merging unit is connected to the matrix expansion unit and is used to obtain the interpolated image according to the interpolated image blocks corresponding to all target pixels. Specifically, each target pixel corresponds to an interpolated image block, and the interpolated image blocks corresponding to all target pixels are merged together to obtain the interpolated image.

[0132] In this embodiment, assuming that the image size is 1920×1080, if the interpolated image is directly stored, 1920×1080 data needs to be stored. Preferably, only the Gaussian filter image, that is, the data before interpolation, is stored in the memory; and when the interpolated image is needed, the corresponding Gaussian filter image can be read out for real-time interpolation.

[0133] The method for the filter matrix acquisition unit to determine the first Gaussian filter matrix includes:

[0134] Determine a target pixel block; the target pixel block refers to a pixel block containing the target pixel point in the current frame image, for example Figure 3 blk_32 in;

[0135] Determine a target Gaussian filter value corresponding to the target pixel block; the target Gaussian filter value refers to the Gaussian filter value corresponding to the target pixel block, for example Figure 3 The target Gaussian filter value corresponding to blk_32 is g32;

[0136] A matrix containing the target pixel and having a size of (B / S+1)×(C / S+1) is selected from the Gaussian filter image corresponding to the current frame image as the first Gaussian filter matrix. The first Gaussian filter matrix is, for example, Figure 3 Preferably, the target Gaussian filter value is located at the center of the first Gaussian filter matrix.

[0137] Since the image defogging device of the present invention saves storage resources by downsampling the current frame image, information loss is inevitable during the downsampling process. The defogging device based on bilateral filtering needs to select the dark channel of pixels within the B×C range around the pixel point for weighted averaging, for example: B=80, C=80. The larger the value of B×C, the greater the impact of this information loss on the defogging effect. The interpolation scheme described in this embodiment can restore the Gaussian filtered image after downsampling, ensuring that the defogging device based on bilateral filtering can achieve better defogging effect. The interpolation scheme described in this embodiment is particularly suitable for scenes with large B×C values.

[0138] In one embodiment of the present invention, the size of the current frame image is 240×176, the downsampling multiple S=16, and the defogging device based on bilateral filtering needs to select the dark channel of the pixels within the range of B×C around the pixel point for weighted averaging. Fig. 6A , the Gaussian filter image size corresponding to the current frame image is 15×11, and the first buffer needs to store the Gaussian filter image. When writing to the buffer, 6 lines of data are written each time, specifically: first write the first 6 lines of data g00~g5e, and after the current 6 lines of data are written, write the next 6 lines of data g60~gbe. For images of larger size, write 6 lines of data each time in this order until all data in the Gaussian filter image are written.

[0139] See also Figure 6B to Figure 6F , which shows the reading process of the first buffer. For interpolation, a 5×5 first interpolation matrix needs to be obtained each time, and each point in the first interpolation matrix needs 2 Gaussian filter values ​​to generate, so a 6×6 Gaussian filter value needs to be read each time. Figure 6B , first read g00~g52 and expand the left border three times to form a 6×6 Gaussian filter matrix; please refer to Figure 6C , read g00~g53 for the second time and expand the left border twice to form a 6×6 Gaussian filter matrix; please refer to Fig.6D , read g00~g54 for the third time and expand the left border once to form a 6×6 Gaussian filter matrix; please refer to Fig. 6E , read g00~g55 for the fourth time, and there is no need to expand the edge at this time; please refer to Fig. 6F , g01-g56 are read for the fifth time, and no edge expansion processing is required at this time. Similarly, the corresponding Gaussian filter matrices are read from the first buffer in sequence until the Gaussian filter matrices corresponding to all pixels in the current frame image are read.

[0140] The Gaussian filter image to be stored in the image defogging method of the present invention is obtained by downsampling the original image. Compared with the prior art that directly stores the Gaussian filter image corresponding to the original image, the amount of data to be stored can be effectively reduced, which is beneficial to saving storage resources and reducing storage overhead.

[0141] The interpolation scheme included in the image defogging method of the present invention can interpolate the Gaussian filter image after downsampling and obtain the interpolated image, and image defogging based on the interpolated image can achieve a good defogging effect. The difference scheme is suitable for an image defogging scheme based on bilateral filtering, especially for an image defogging scheme based on bilateral filtering with a large B×C.

[0142] The image defogging method of the present invention can achieve a shorter single-frame defogging time when using the Gaussian filter image corresponding to the previous frame image, which is beneficial to avoiding the occurrence of a freeze phenomenon.

[0143] When the image defogging method of the present invention uses the Gaussian filter image corresponding to the current frame image, since the Gaussian filter results and related information used in the scheme are all from the current frame image, high-quality image defogging can be achieved.

[0144] When the image defogging method of the present invention uses the Gaussian filter image corresponding to the previous frame image, there is no need to wait for all the pixels of the current frame image to be loaded. No matter how many pixels are currently loaded, the pixel can be defogged, which is beneficial to the pipeline processing of the hardware.

[0145] In summary, the present invention effectively overcomes various shortcomings of the prior art and has high industrial utilization value.

[0146] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. An image defogging method, characterized in that: The image defogging method comprises: Obtain the current frame image of the image to be defogged; Obtaining a Gaussian filter image corresponding to the image to be defogged, wherein the Gaussian filter image is generated by a down-sampled image corresponding to the image to be defogged; Interpolating the Gaussian filtered image to obtain an interpolated image; Performing bilateral filtering on the current frame image according to the interpolation image to obtain transmittance; and Generate a defogging image corresponding to the current frame image according to the transmittance, The interpolation of the Gaussian filtered image to obtain an interpolated image includes: for any target pixel point in the current frame image, obtaining a first Gaussian filter matrix corresponding to the target pixel point and its surrounding pixels; interpolating the first Gaussian filter matrix to obtain a first interpolation matrix; expanding the first interpolation matrix to obtain an interpolated image block corresponding to the target pixel point; and obtaining the interpolated image according to the interpolated image blocks corresponding to all target pixels.

2. The image defogging method according to claim 1, characterized in that: The Gaussian filtered image corresponding to the image to be defogged is the Gaussian filtered image corresponding to the current frame image.

3. The image defogging method according to claim 1, characterized in that: The Gaussian filter image corresponding to the image to be defogged is the Gaussian filter image corresponding to the previous frame image.

4. The image defogging method according to claim 3, characterized in that: The image defogging method further comprises: Down-sampling the current frame image to obtain a down-sampled image corresponding to the current frame image; The down-sampled image corresponding to the current frame image is processed to generate a Gaussian filtered image corresponding to the current frame image; the Gaussian filtered image corresponding to the current frame image is used for defogging the next frame image.

5. The image defogging method according to claim 4, characterized in that: The image defogging method further comprises: Reading a Gaussian filter image corresponding to the previous frame of image from the first buffer; The Gaussian filter image corresponding to the current frame image is stored in the first buffer.

6. An image defogging device, characterized in that: The image defogging device comprises: An image acquisition module is used to acquire a current frame image of an image to be defogged; A filter value acquisition module, used to acquire a Gaussian filter image corresponding to the image to be defogged, wherein the Gaussian filter image is generated by a down-sampled image corresponding to the image to be defogged; An interpolation module, connected to the filter value acquisition module, for interpolating the Gaussian filter image to obtain an interpolated image; A bilateral filtering module, connected to the image acquisition module and the interpolation module, and configured to perform bilateral filtering on the current frame image according to the interpolation image to obtain transmittance; a defogging module, connected to the bilateral filtering module, and configured to generate a defogging image corresponding to the current frame image according to the transmittance; and a control module, connected to the image acquisition module, the filter value acquisition module, the interpolation module, the bilateral filtering module and the defogging module, respectively, and used to control the image defogging device, The interpolation module is used to: for any target pixel in the current frame image, obtain the first Gaussian filter matrix corresponding to the target pixel and its surrounding pixels; interpolate the first Gaussian filter matrix to obtain a first interpolation matrix; expand the first interpolation matrix to obtain an interpolation image block corresponding to the target pixel; and obtain the interpolation image based on the interpolation image blocks corresponding to all target pixels.

7. The image defogging device according to claim 6, characterized in that: The Gaussian filtered image corresponding to the image to be defogged is the Gaussian filtered image corresponding to the current frame image.

8. The image defogging device according to claim 6, characterized in that: The Gaussian filter image corresponding to the image to be defogged is the Gaussian filter image corresponding to the previous frame image.

9. The image defogging device according to claim 8, characterized in that: The image defogging device also includes: A down-sampling module, connected to the image acquisition module, for down-sampling the current frame image to obtain a down-sampled image corresponding to the current frame image; The Gaussian filtering module is connected to the down-sampling module and is used to process the down-sampled image corresponding to the current frame image to generate a Gaussian filtered image corresponding to the current frame image; the Gaussian filtered image corresponding to the current frame image is used for defogging the next frame image.

10. The image defogging device according to claim 9, characterized in that: The image defogging device also includes: The first buffer is connected to the filter value acquisition module and the Gaussian filter module; the filter value acquisition module reads the Gaussian filter image corresponding to the previous frame image from the first buffer; the Gaussian filter module stores the Gaussian filter image corresponding to the current frame image in the first buffer.

Images