Image defogging method and device

Abstract

The present invention discloses an image defogging method and device, and relates to the field of computer technology. A specific implementation of the method includes: obtaining an image to be processed, wherein the image to be processed is a foggy image; obtaining a clear image corresponding to the image to be processed through a pre-constructed image defogging network, wherein the clear image is a fog-free image; wherein the pre-constructed image defogging network is used to obtain a feature image of the image to be processed, and obtain a clear image corresponding to the image to be processed based on the feature image. The defogging image obtained by this implementation is rich in details and has significant defogging performance, and can be widely used in various defogging scenes.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an image defogging method and device. Background Art

[0002] The haze weather caused by various pollution sources such as automobile exhaust, industrial emissions, construction dust, and garbage incineration is not only extremely harmful to the human body, but also the absorption and scattering of light by atmospheric suspended particles, resulting in a decrease in the quality of outdoor images, which seriously affects the effectiveness of outdoor visual systems. Therefore, it is of great practical significance to study how to improve the image quality of foggy days and reduce the adverse effects of haze weather on outdoor imaging systems. In recent years, due to the rise of deep learning algorithms, deep learning defogging methods based on convolutional neural networks have been widely praised by defogging researchers. However, in the process of realizing the present invention, the inventors found that there are at least the following problems in the prior art: the deep learning defogging method based on convolutional neural networks needs to estimate the defogging model parameters or intermediate variables, use the network to solve the transmittance, and then solve the atmospheric light according to the atmospheric scattering model. At the same time, it requires optimization processes such as guided filtering. The above operations will destroy the restoration process of clear images, and separately estimating the two parameters of transmittance and atmospheric light will cause error accumulation. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides an image defogging method and device, by which a defogging image is obtained with rich details and significant defogging performance, and can be widely used in various defogging scenarios.

[0004] To achieve the above object, according to one aspect of an embodiment of the present invention, a method for defogging an image is provided, comprising:

[0005] Acquire an image to be processed, where the image to be processed is a foggy image;

[0006] A clear image corresponding to the image to be processed is obtained through a pre-constructed image dehazing network, and the clear image is a haze-free image; wherein the pre-constructed image dehazing network is used to obtain first feature information of multiple depths and second feature information of multiple scales of the image to be processed, and based on the first feature information of the multiple depths and the second feature information of the multiple scales, a feature image of the image to be processed is obtained, and based on the feature image, a clear image corresponding to the image to be processed is obtained.

[0007] Optionally, the pre-constructed image dehazing network includes at least a densely connected subnetwork and a multi-scale subnetwork, the densely connected subnetwork is used to obtain first feature information of multiple depths of the image to be processed; the multi-scale subnetwork is used to obtain second feature information of multiple scales of the image to be processed.

[0008] Optionally, the densely connected sub-network is used to obtain the first feature information of multiple depths of the image to be processed, including: inputting the image to be processed into the densely connected sub-network to perform convolution operations at multiple depths to obtain a first feature map of the image to be processed;

[0009] The multi-scale sub-network is used to obtain the second feature information of multiple scales of the image to be processed, including: inputting the first feature map into the multi-scale sub-network for multi-scale downsampling operation to obtain multiple second feature maps, and the sizes of the multiple second feature maps are different.

[0010] Optionally, inputting the first feature map into the multi-scale sub-network for a multi-scale downsampling operation to obtain a plurality of second feature maps comprises:

[0011] Inputting the first feature map into the multi-scale sub-network to obtain a plurality of intermediate feature maps; the sizes of the plurality of intermediate feature maps are different;

[0012] A convolution dimension reduction operation is performed on the multiple intermediate feature maps to obtain multiple second feature maps.

[0013] Optionally, the pre-constructed image dehazing network further includes a convolutional activation subnetwork;

[0014] The image defogging network obtains the feature image of the image to be processed based on the first feature information of the multiple depths and the second feature information of the multiple scales, including:

[0015] Performing an upsampling operation on the plurality of second feature maps to obtain a third feature map; the size of the third feature map is the same as the size of the first feature map;

[0016] The third feature map and the first feature map are input into the convolution activation subnetwork to obtain the feature image.

[0017] Optionally, the pre-constructed image dehazing network further includes a reorganization sub-network;

[0018] The pre-constructed image dehazing network obtains a clear image corresponding to the image to be processed based on the feature image, comprising: inputting the feature image and the image to be processed into the recombinant subnetwork to obtain a clear image corresponding to the image to be processed.

[0019] Optionally, the recombinant subnetwork obtains a clear image corresponding to the image to be processed according to the following formula:

[0020] J(x)=(I(x)-V(x)) / (1-V(x))

[0021] Wherein, J(x) represents the clear image, I(x) represents the image to be processed, V(x) represents the feature image, and x represents the pixel coordinates.

[0022] Optionally, the densely connected subnetwork includes N convolutional layers, the convolution kernels of the N convolutional layers are different from each other, and the N convolutional layers form a cascade structure, wherein N is an integer greater than or equal to 3.

[0023] To achieve the above object, according to another aspect of an embodiment of the present invention, an image defogging device is provided, comprising:

[0024] An acquisition module, used for acquiring an image to be processed, wherein the image to be processed is a foggy image;

[0025] A defogging module is used to obtain a clear image corresponding to the image to be processed through a pre-constructed image defogging network, and the clear image is a haze-free image; wherein the pre-constructed image defogging network is used to obtain first feature information of multiple depths and second feature information of multiple scales of the image to be processed, obtain a feature image of the image to be processed based on the first feature information of the multiple depths and the second feature information of the multiple scales, and obtain a clear image corresponding to the image to be processed based on the feature image.

[0026] Optionally, the pre-constructed image dehazing network includes at least a densely connected subnetwork and a multi-scale subnetwork, the densely connected subnetwork is used to obtain first feature information of multiple depths of the image to be processed; the multi-scale subnetwork is used to obtain second feature information of multiple scales of the image to be processed.

[0027] Optionally, the defogging module is further used to: input the image to be processed into the densely connected sub-network to perform convolution operations of multiple depths to obtain a first feature map of the image to be processed;

[0028] The first feature map is input into the multi-scale sub-network for multi-scale downsampling operation to obtain a plurality of second feature maps, where the sizes of the plurality of second feature maps are different.

[0029] Optionally, the dehazing module is further used to: input the first feature map into the multi-scale sub-network to obtain multiple intermediate feature maps; the sizes of the multiple intermediate feature maps are different; and perform convolution dimensionality reduction operations on the multiple intermediate feature maps to obtain multiple second feature maps.

[0030] Optionally, the pre-constructed image dehazing network further includes a convolutional activation subnetwork;

[0031] The defogging module is also used to: perform an upsampling operation on the multiple second feature maps to obtain a third feature map; the size of the third feature map is the same as the size of the first feature map; and input the third feature map and the first feature map into the convolution activation subnetwork to obtain the feature image.

[0032] Optionally, the pre-constructed image dehazing network further includes a reorganization sub-network;

[0033] The defogging module is also used to: input the feature image and the image to be processed into the recombinant subnetwork to obtain a clear image corresponding to the image to be processed.

[0034] Optionally, the recombinant subnetwork obtains a clear image corresponding to the image to be processed according to the following formula:

[0035] J(x)=(I(x)-V(x)) / (1-V(x))

[0036] Wherein, J(x) represents the clear image, I(x) represents the image to be processed, V(x) represents the feature image, and x represents the pixel coordinates.

[0037] Optionally, the densely connected subnetwork includes N convolutional layers, the convolution kernels of the N convolutional layers are different from each other, and the N convolutional layers form a cascade structure, wherein N is an integer greater than or equal to 3.

[0038] To achieve the above-mentioned purpose, according to another aspect of an embodiment of the present invention, there is provided an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the image defogging method according to an embodiment of the present invention.

[0039] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the image defogging method of the embodiment of the present invention is implemented.

[0040] An embodiment of the above invention has the following advantages or beneficial effects: a clear image corresponding to an image to be processed is obtained through a pre-constructed image defogging network, wherein the image to be processed is a foggy image, and the clear image is a fog-free image; wherein the pre-constructed image defogging network is used to obtain first feature information of multiple depths and second feature information of multiple scales of the image to be processed, and based on the first feature information of the multiple depths and the second feature information of the multiple scales, a feature image of the image to be processed is obtained, and based on the feature image, a technical means for obtaining a clear image corresponding to the image to be processed is used, so as to obtain a defogging image with rich details and significant defogging performance, which can be widely used in various defogging scenarios.

[0041] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0043] Figure 1 Schematic diagram of the main process of the image defogging method according to an embodiment of the present invention;

[0044] Figure 2 is a schematic diagram of the main process of an image defogging method according to another embodiment of the present invention;

[0045] Figure 3 is a schematic diagram of an image defogging network in an image defogging method according to an embodiment of the present invention;

[0046] Figure 4 is a schematic diagram of a densely connected subnetwork in an image defogging method according to an embodiment of the present invention;

[0047] Figure 5 is a schematic diagram of a multi-scale sub-network in an image defogging method according to an embodiment of the present invention;

[0048] Figure 6 is a schematic diagram of main modules of an image defogging device according to an embodiment of the present invention;

[0049] Figure 7 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0050] Figure 8 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0052] Figure 1 FIG. 1 is a schematic diagram of the main process of the image defogging method according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0053] Step S101: obtaining an image to be processed, wherein the image to be processed is a foggy image;

[0054] Step S102: Obtain a clear image corresponding to the image to be processed through a pre-constructed image dehazing network, wherein the clear image is a haze-free image; wherein the pre-constructed image dehazing network is used to obtain first feature information of multiple depths and second feature information of multiple scales of the image to be processed, obtain a feature image of the image to be processed based on the first feature information of the multiple depths and the second feature information of the multiple scales, and obtain a clear image corresponding to the image to be processed based on the feature image.

[0055] In order to make the method of the embodiment of the present invention easier to understand, the principle of the image defogging method of the embodiment of the present invention is first described below.

[0056] In foggy weather, the reflected light from the surface of the object is scattered by atmospheric particles and attenuated, and the atmospheric light scattering is superimposed on the reflected light, causing the camera to capture a foggy image, that is, the light reaching the imaging device is actually composed of two parts: the attenuated part of the incident light and the scattered ambient light. Therefore, the imaging model of foggy images is:

[0057] I(x)=E 0 (λ)e -β(λ)d(x) +E ∞ (λ)(1-e -β(λ)d(x) ) (1)

[0058] Where x represents the pixel coordinate, I(x) represents the foggy image, λ represents the wavelength of light, and E 0 (λ) represents the real image, β(λ) represents the atmospheric scattering coefficient, d represents the distance from the object to the camera, and e -β(λ)d(x) Represents the real image E 0 (λ) Transmittance reaching the camera, E ∞ (λ) represents the ambient light.

[0059] Under uniform haze conditions, the suspended particles in the atmosphere scatter light to roughly the same degree, and the atmospheric scattering coefficient β(λ) can be regarded as a constant. In addition, the scattering particles in the atmosphere are evenly distributed during the day, and the ambient light E ∞ (λ) can be approximately regarded as a constant in the global range. Usually, the point at infinity is taken as the ambient light, that is, A ∞ At the same time, the real image E 0 (λ) is replaced by the scene reflected light J(x). Then the above formula (1) can be simplified as:

[0060] I(x)=J(x)t(x)+A ∞ (1-t(x)) (2)

[0061] Where t(x) = e -β(λ)d(x) .

[0062] In the process of restoring images, most physical model-based methods are solved based on local ideas, such as the dark channel idea. Such methods are prone to block effects after defogging, and halo artifacts are prone to appear in places where the depth of field changes dramatically. Therefore, considering that the depth information of foggy days is unknown, the unknown amount of atmospheric light A can be ∞ The transmittance t(x) is combined into the atmospheric light curtain function, that is, V(x) = A(1-t(x)), then the foggy image imaging model can be simplified as:

[0063] I(x)=J(x)t(x)+A(1-t(x))=J(x)t(x)+V(x) (3)

[0064] The above formula can be transformed into:

[0065] I(x)=J(x)(1-V(x) / A)+V(x) (4)

[0066] At this time, compared with solving the atmospheric light A and transmittance t(x), only one variable V(x) needs to be solved. The image is white-balanced before defogging, so the atmospheric light A can be set to (1, 1, 1), and the input image is also normalized to (0, 1). Then formula (4) is further transformed into:

[0067] I(x)=J(x)(1-V(x))+V(x) (5)

[0068] White balance refers to restoring the original color of the fog. The color cast that occurs when shooting under a specific light source is compensated by strengthening the corresponding complementary color. After white balancing, the fog is pure white. If the pixel values ​​of the input image are normalized to the range of 0 to 1, then A can be set to a vector (1, 1, 1).

[0069] Then the clear image J(x) can be obtained by formula (5):

[0070] J(x)=(I(x)-V(x)) / (1-V(x)) (6)

[0071] In Formula 6, V(x) is related to both A and t(x), and A and t(x) are related to I(x), which means that V(x) is correlated with the foggy image I(x) and can be used as the feature image of I(x). Therefore, the feature image V(x) can be obtained from the foggy image I(x). The network can be used to learn the features of the foggy image I(x) to obtain the feature image V(x).

[0072] In the image defogging method of the embodiment of the present invention, a feature image of the image to be processed is first obtained through an image defogging network, and then the feature image and the image to be processed are processed through the image defogging network to obtain a clear image without fog.

[0073] Among them, the image dehazing network can be trained according to the following process:

[0074] The public indoor image dataset can be used as the original dataset. In order to meet the fog image effects of different concentrations, the same clear image in the original dataset is processed into foggy images of different degrees. Specifically, the atmospheric light value can be set to three random values ​​between [0.6, 1.0] and the β value can be set to {0.4, 0.6, 0.8, 1.0, 1.2, 1.4, 1.6} to obtain foggy images of different degrees.

[0075] Set the loss function. In this embodiment, the mean square error MSE can be used as the loss function, as shown in the following formula:

[0076]

[0077] Among them, f(I) is the clear image reconstructed by the image dehazing network, J is the real clear image, N is the batch number, H is the image height, and W is the image width.

[0078] After setting the loss function, the image dehazing network can be trained. When training the model, the weight initialization method uses Gaussian weight initialization, and the activation function uses ReLU to effectively avoid the gradient vanishing problem. The number of iterations is 30, the learning rate is 0.0001, and Adam is used as the optimization function. The Adam optimization method can adaptively calculate the adaptive learning rate of each parameter and is not prone to falling into local minima.

[0079] The image defogging method of the embodiment of the present invention obtains a clear image corresponding to the image to be processed through a pre-constructed image defogging network, wherein the image to be processed is a foggy image, and the clear image is a fog-free image; wherein the pre-constructed image defogging network is used to obtain first feature information of multiple depths and second feature information of multiple scales of the image to be processed, and based on the first feature information of the multiple depths and the second feature information of the multiple scales, a feature image of the image to be processed is obtained, and based on the feature image, a clear image corresponding to the image to be processed is obtained by a technical means, so as to obtain a defogging image with rich details and significant defogging performance, and can be widely used in various defogging scenarios.

[0080] Specifically, in this embodiment, the pre-constructed image dehazing network includes at least a densely connected sub-network and a multi-scale sub-network.

[0081] The densely connected subnetwork is used to obtain first feature information of multiple depths of the image to be processed. Specifically, the image to be processed is input into the densely connected subnetwork for convolution operations of multiple depths to obtain a first feature map of the image to be processed.

[0082] The multi-scale sub-network is used to obtain the second feature information of multiple scales of the image to be processed. Specifically, the first feature map is input into the multi-scale sub-network for multi-scale downsampling operation to obtain multiple second feature maps, and the sizes of the multiple second feature maps are different.

[0083] After obtaining a plurality of second feature maps, the image dehazing network obtains a feature image of the image to be processed based on the second feature maps.

[0084] In an optional embodiment, the pre-constructed image dehazing network may further include a reorganization sub-network, and the reorganization sub-network is used to obtain a clear image corresponding to the image to be processed based on the feature image and the image to be processed.

[0085] In other optional embodiments, the pre-constructed image dehazing network may also include a convolutional activation subnetwork. The convolutional activation subnetwork is located between the multi-scale subnetwork and the reorganization subnetwork. The convolutional activation subnetwork is used to obtain a feature image of the image to be processed based on the first feature information of multiple depths and the second feature information of multiple scales of the image to be processed. The convolutional activation subnetwork includes a convolutional layer and an activation layer, wherein the convolutional layer realizes feature learning and sorting. The activation layer uses a nonlinear activation function commonly used in convolutional neural networks, which is used to alleviate network overfitting and reduce the amount of calculation. The activation function will cause a part of the neuron output to be 0, making the network sparse, which is a commonly used means to avoid overfitting in deep learning.

[0086] The image defogging method of the embodiment of the present invention obtains a clear image corresponding to the image to be processed through a pre-constructed image defogging network, wherein the image to be processed is a foggy image, and the clear image is a fog-free image; wherein the pre-constructed image defogging network is used to obtain first feature information of multiple depths and second feature information of multiple scales of the image to be processed, and based on the first feature information of the multiple depths and the second feature information of the multiple scales, a feature image of the image to be processed is obtained, and based on the feature image, a clear image corresponding to the image to be processed is obtained by a technical means, so as to obtain a defogging image with rich details and significant defogging performance, and can be widely used in various defogging scenarios.

[0087] Figure 2 It is a flowchart diagram of the main steps of the image defogging method according to another embodiment of the present invention. Figure 3 Schematic diagram of the structure of the image defogging network in the image defogging method according to an embodiment of the present invention.

[0088] like Figure 3 As shown, the image dehazing network includes a densely connected subnetwork (Densely Block), a multi-scale subnetwork (Multi-Scale Block), a convolutional activation subnetwork (Conv+Rrelu) and a reorganized subnetwork (Reorganizedlayer). Among them, the densely connected subnetwork includes N convolutional layers, the convolution kernels of the N convolutional layers are different from each other, and the N convolutional layers form a cascade structure, wherein N is an integer greater than or equal to 3. In order to retain the information between all convolutional layers, the embodiment of the present invention connects all convolutional layers in the densely connected subnetwork in pairs, so that the input of each convolutional layer is the feature information of all previous convolutional layers. The densely connected subnetwork uses different convolutional kernels to repeatedly learn previously learned features, so that the image dehazing network has a stronger learning ability. Since the structure repeatedly obtains the output of the previous layer, the model of the densely connected subnetwork should not be too large. Specifically, as Figure 4 As shown in the figure, the densely connected subnetwork is a 5-layer cascade structure. The convolution kernel sizes are 1×1, 3×3, 5×5, 7×7, and 9×9, respectively. The number of filters is 3. Conv is the convolution layer, and the Concat layer is the dimensional lateral connection operation of the convolution layer. The densely connected structure can reuse the previous layer features of each layer.

[0089] The multi-scale sub-network of the embodiment of the present invention uses a multi-scale strategy inside the network to decompose the feature map obtained by the densely connected sub-network into four feature maps of different sizes, and then sends them to the lower sub-network for learning, so that the feature information of different scales can be fully learned and fused and output, which can enhance the generalization ability of the network, improve the defogging effect, and effectively solve the halo effect produced by the defogging result. As a specific example, Figure 5As shown, the multi-scale sub-network downsamples the feature map output by the densely connected sub-network to 1 / 32, 1 / 16, 1 / 8, and 1 / 4 of the size of the original image (i.e., the image to be processed), and after a 1×1 convolution dimensionality reduction operation, the result is restored to its size using an upsampling operation.

[0090] The convolution activation subnetwork includes a convolution layer and an activation layer. The convolution layer is used to learn and integrate the feature graphs of the densely connected subnetwork and the feature graphs of the multi-scale subnetwork. The activation layer makes the output of some neurons 0, so that the network has sparseness and avoids overfitting. The activation function of the activation layer can adopt a nonlinear activation function commonly used in the art.

[0091] The recombinant subnetwork is used to learn the feature map of the image to be processed and the output of the convolution activation layer to obtain a clear image without fog. The recombinant subnetwork obtains the clear image corresponding to the image to be processed based on the following formula:

[0092] J(x)=(I(x)-V(x)) / (1-V(x))

[0093] Wherein, J(x) represents the clear image, I(x) represents the image to be processed, V(x) represents the feature image, and x represents the pixel coordinates.

[0094] In the embodiment of the present invention, a clear fog-free image is obtained by inputting the foggy image to be processed into the above-mentioned image defogging network. Specifically, Figure 2 As shown, the process of defogging the foggy image to be processed by using the image defogging model includes:

[0095] Step S201: obtaining an image to be processed, wherein the image to be processed is a foggy image;

[0096] Step S202: inputting the image to be processed into a densely connected sub-network to perform convolution operations of multiple depths to obtain a first feature map of the image to be processed;

[0097] Step S203: inputting the first feature map into a multi-scale sub-network to obtain a plurality of intermediate feature maps; the sizes of the plurality of intermediate feature maps are different;

[0098] Step S204: the multi-scale sub-network performs a convolution dimensionality reduction operation on the multiple intermediate feature maps to obtain multiple second feature maps;

[0099] Step S205: performing an upsampling operation on the plurality of second feature maps to obtain a third feature map; the size of the third feature map is the same as the size of the first feature map;

[0100] Step S206: inputting the third feature map and the first feature map into the convolution activation subnetwork to obtain the feature image;

[0101] Step S207: input the feature image and the image to be processed into the recombinant sub-network to obtain a clear image corresponding to the image to be processed.

[0102] The image defogging method of the embodiment of the present invention repeatedly learns feature information of different depths of the image to be processed through a densely connected subnetwork in the image defogging network, and performs multi-scale downsampling on the feature map output by the densely connected subnetwork through the multi-scale subnetwork in the image defogging network to learn feature information of different scales of the image to be processed, which can avoid the halo effect caused by insufficient scale information and information loss, and has a better defogging effect on distant scenes and more obvious details; the convolution activation subnetwork learns and organizes the feature information of the densely connected subnetwork and the multi-scale subnetwork to obtain a feature image; the recombinant subnetwork obtains a clear image corresponding to the image to be processed according to the feature image, and the clear image is rich in details and does not contain artifact effects, and the defogging effect of distant scenes is better and the details are obvious.

[0103] The image defogging method of the embodiment of the present invention reconstructs an atmospheric scattering model. The reconstructed atmospheric scattering model concentrates main information in a single variable atmospheric light curtain function. The atmospheric light curtain function is solved by an image defogging network to obtain a clear image without fog.

[0104] Figure 6 is a schematic diagram of the main modules of the image defogging device 600 according to an embodiment of the present invention. Figure 6 As shown, the device 600 includes:

[0105] An acquisition module 601 is used to acquire an image to be processed, where the image to be processed is a foggy image;

[0106] The defogging module 602 is used to obtain a clear image corresponding to the image to be processed through a pre-constructed image defogging network, and the clear image is a haze-free image; wherein the pre-constructed image defogging network is used to obtain a feature image of the image to be processed, and obtain a clear image corresponding to the image to be processed based on the feature image.

[0107] Optionally, the defogging module 602 is also used to: input the image to be processed into a pre-constructed image defogging network to obtain a feature image of the image to be processed; wherein the pre-constructed image defogging network includes at least a densely connected subnetwork and a multi-scale subnetwork, the densely connected subnetwork is used to obtain first feature information of multiple depths of the image to be processed; the multi-scale subnetwork is used to obtain second feature information of multiple scales of the image to be processed, and the image defogging network obtains the feature image of the image to be processed based on the first feature information of multiple depths and the second feature information of multiple scales of the image to be processed.

[0108] Optionally, the defogging module 602 is further used to: input the image to be processed into the densely connected sub-network to perform convolution operations of multiple depths to obtain a first feature map of the image to be processed;

[0109] The first feature map is input into the multi-scale sub-network for multi-scale downsampling operation to obtain a plurality of second feature maps, where the sizes of the plurality of second feature maps are different.

[0110] Optionally, the dehazing module 602 is further used to: input the first feature map into the multi-scale sub-network to obtain multiple intermediate feature maps; the sizes of the multiple intermediate feature maps are different; and perform convolution dimensionality reduction operations on the multiple intermediate feature maps to obtain multiple second feature maps.

[0111] Optionally, the pre-constructed image dehazing network further includes a convolutional activation subnetwork;

[0112] The defogging module 602 is also used to: perform an upsampling operation on the multiple second feature maps to obtain a third feature map; the size of the third feature map is the same as the size of the first feature map; and input the third feature map and the first feature map into the convolution activation subnetwork to obtain the feature image.

[0113] Optionally, the pre-constructed image dehazing network further includes a reorganization sub-network;

[0114] The defogging module 602 is further used to: input the feature image and the image to be processed into the reorganization sub-network to obtain a clear image corresponding to the image to be processed.

[0115] Optionally, the recombinant subnetwork obtains a clear image corresponding to the image to be processed according to the following formula:

[0116] J(x)=(I(x)-V(x)) / (1-V(x))

[0117] Wherein, J(x) represents the clear image, I(x) represents the image to be processed, V(x) represents the feature image, and x represents the pixel coordinates.

[0118] Optionally, the densely connected subnetwork includes N convolutional layers, the convolution kernels of the N convolutional layers are different from each other, and the N convolutional layers form a cascade structure, wherein N is an integer greater than or equal to 3.

[0119] The image defogging device of the embodiment of the present invention obtains a clear image corresponding to the image to be processed through a pre-constructed image defogging network, wherein the image to be processed is a foggy image, and the clear image is a fog-free image; wherein the pre-constructed image defogging network is used to obtain first feature information of multiple depths and second feature information of multiple scales of the image to be processed, obtains a feature image of the image to be processed based on the first feature information of the multiple depths and the second feature information of the multiple scales, and obtains a clear image corresponding to the image to be processed based on the feature image, thereby obtaining a defogging image with rich details and significant defogging performance, and can be widely used in various defogging scenarios.

[0120] The above device can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0121] Figure 7 An exemplary system architecture 700 is shown to which the image defogging method or image defogging device according to the embodiment of the present invention can be applied.

[0122] like Figure 7 As shown, system architecture 700 may include terminal devices 701, 702, 703, network 704 and server 705. Network 704 is used to provide a medium for communication links between terminal devices 701, 702, 703 and server 705. Network 704 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0123] The user can use the terminal devices 701, 702, 703 to interact with the server 705 through the network 704 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 701, 702, 703.

[0124] The terminal devices 701 , 702 , and 703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0125] The server 705 may be a server that provides various services, such as a backend management server that provides support for shopping websites browsed by users using the terminal devices 701, 702, and 703. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information and product information) to the terminal device.

[0126] It should be noted that the image defogging method provided in the embodiment of the present invention is generally executed by the server 705 , and accordingly, the image defogging device is generally disposed in the server 705 .

[0127] It should be understood that Figure 7 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0128] Reference below Figure 8 , which shows a schematic diagram of the structure of a computer system 800 of a terminal device suitable for implementing an embodiment of the present invention. Figure 8 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0129] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the system 800 are also stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0130] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage section 808 as needed.

[0131] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above-mentioned functions defined in the system of the present invention are executed.

[0132] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0133] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0134] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be set in a processor. For example, they may be described as: a processor includes a sending module, an acquisition module, a determination module, and a first processing module. The names of these modules do not, in some cases, constitute a limitation on the unit itself. For example, the sending module may also be described as a "module for sending a picture acquisition request to the connected server."

[0135] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:

[0136] Acquire an image to be processed, where the image to be processed is a foggy image;

[0137] A clear image corresponding to the image to be processed is obtained through a pre-constructed image dehazing network, and the clear image is a haze-free image; wherein the pre-constructed image dehazing network is used to obtain a feature image of the image to be processed, and obtain a clear image corresponding to the image to be processed based on the feature image.

[0138] The technical solution of the embodiment of the present invention obtains a clear image corresponding to the image to be processed through a pre-constructed image defogging network, wherein the image to be processed is a foggy image, and the clear image is a fog-free image; wherein the pre-constructed image defogging network is used to obtain first feature information of multiple depths and second feature information of multiple scales of the image to be processed, and obtains a feature image of the image to be processed based on the first feature information of the multiple depths and the second feature information of the multiple scales, and obtains a clear image corresponding to the image to be processed based on the feature image, thereby obtaining a defogging image with rich details and significant defogging performance, which can be widely used in various defogging scenarios.

[0139] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An image defogging method, characterized in that: include: Acquire an image to be processed, where the image to be processed is a foggy image; Obtain a clear image corresponding to the image to be processed through a pre-constructed image defogging network, wherein the clear image is a haze-free image; wherein the pre-constructed image defogging network is used to obtain first feature information of multiple depths and second feature information of multiple scales of the image to be processed, obtain a feature image of the image to be processed based on the first feature information of the multiple depths and the second feature information of the multiple scales, and obtain a clear image corresponding to the image to be processed based on the feature image; The pre-constructed image dehazing network comprises at least a densely connected sub-network and a multi-scale sub-network, wherein the densely connected sub-network is used to obtain first feature information of multiple depths of the image to be processed; The multi-scale sub-network is used to obtain second feature information of multiple scales of the image to be processed.

2. The method according to claim 1, characterized in that The densely connected sub-network is used to obtain the first feature information of the image to be processed at multiple depths, including: inputting the image to be processed into the densely connected sub-network to perform convolution operations at multiple depths to obtain a first feature map of the image to be processed; The multi-scale sub-network is used to obtain the second feature information of multiple scales of the image to be processed, including: inputting the first feature map into the multi-scale sub-network for multi-scale downsampling operation to obtain multiple second feature maps, and the sizes of the multiple second feature maps are different.

3. The method according to claim 2, characterized in that Inputting the first feature map into the multi-scale sub-network for multi-scale downsampling operation to obtain a plurality of second feature maps comprises: Inputting the first feature map into the multi-scale sub-network to obtain a plurality of intermediate feature maps; the sizes of the plurality of intermediate feature maps are different; A convolution dimension reduction operation is performed on the multiple intermediate feature maps to obtain multiple second feature maps.

4. The method according to claim 3, characterized in that The pre-constructed image dehazing network also includes a convolutional activation sub-network; The image defogging network obtains the feature image of the image to be processed based on the first feature information of the multiple depths and the second feature information of the multiple scales, including: Performing an upsampling operation on the plurality of second feature maps to obtain a third feature map; the size of the third feature map is the same as the size of the first feature map; The third feature map and the first feature map are input into the convolution activation subnetwork to obtain the feature image.

5. The method according to claim 1, characterized in that The pre-constructed image dehazing network also includes a reorganization sub-network; The pre-built image defogging network obtains a clear image corresponding to the image to be processed based on the feature image, comprising: The feature image and the image to be processed are input into the recombinant sub-network to obtain a clear image corresponding to the image to be processed.

6. The method according to claim 5, characterized in that The recombinant subnetwork obtains a clear image corresponding to the image to be processed according to the following formula: J(x)=(I(x)-V(x)) / (1-V(x)) Wherein, J(x) represents the clear image, I(x) represents the image to be processed, V(x) represents the feature image, and x represents the pixel coordinates.

7. The method according to any one of claims 1 to 6, characterized in that: The densely connected subnetwork includes N convolutional layers, the convolution kernels of the N convolutional layers are different from each other, and the N convolutional layers form a cascade structure, wherein N is an integer greater than or equal to 3.

8. An image defogging device, characterized in that: include: An acquisition module, used for acquiring an image to be processed, wherein the image to be processed is a foggy image; A defogging module, used to obtain a clear image corresponding to the image to be processed through a pre-constructed image defogging network, wherein the clear image is a haze-free image; wherein the pre-constructed image defogging network is used to obtain first feature information of multiple depths and second feature information of multiple scales of the image to be processed, obtain a feature image of the image to be processed based on the first feature information of the multiple depths and the second feature information of the multiple scales, and obtain a clear image corresponding to the image to be processed based on the feature image; the pre-constructed image defogging network includes at least a densely connected subnetwork and a multi-scale subnetwork, and the densely connected subnetwork is used to obtain first feature information of multiple depths of the image to be processed; The multi-scale sub-network is used to obtain second feature information of multiple scales of the image to be processed.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images