Method and device for defogging UAV aerial images based on residual detail enhancement

Through the depth residual network and nonlinear regression detail enhancement algorithm, the foggy day image is decomposed into basic components and detailed components in parallel processing, solving the problem of defogging images in foggy day image in drone aerial photography, and realizing the image clarity and detail information The recovery has improved the effectiveness of computer vision algorithms.

CN114331874BActive Publication Date: 2024-12-06XIAN UNIV OF POSTS & TELECOMM
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202111484793.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-07
Publication Date
2024-12-06
Estimated Expiration
2041-12-07

AI Technical Summary

Technical Problem

The prior art is difficult to effectively remove fog in foggy images, resulting in blurred images and distorted colors, affecting the visual effects and the effectiveness of computer vision algorithms, especially in drone aerial photography.

Method used

Based on the detail enhancement method of deep residual network, the foggy image to be processed is decomposed into basic component images and detailed component images, and the fogging process is performed through multi-scale residual network and nonlinear regression detail enhancement algorithm, combined with early vision systems The dual-channel processing mechanism processes the color and detail information of the image respectively.

Benefits of technology

Effectively remove fog, restore detailed information of the image, improve image clarity and visual effects, and enhance the effectiveness of computer vision algorithms.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114331874B_ABST
    Figure CN114331874B_ABST
Patent Text Reader

Abstract

The present invention relates to a method and device for defogging aerial images taken by unmanned aerial vehicles based on residual detail enhancement. The method comprises: obtaining foggy aerial images taken by unmanned aerial vehicles in different scenes as foggy images to be processed; decomposing the foggy images to be processed into a basic component image and a detail component image; defogging the basic component image of the foggy image to be processed to obtain the defogged basic component image, wherein the defogging process is a multi-scale residual network algorithm; performing detail enhancement process on the detail component image of the foggy image to be processed to obtain the enhanced detail component image as the detail component image of the defogged image, wherein the detail enhancement process adopts a nonlinear regression detail enhancement algorithm; and performing sum process on the defogged basic component image and the detail component image after detail enhancement to obtain the defogged image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for defogging unmanned aerial image based on residual detail enhancement. Background Art

[0002] In recent years, air quality has deteriorated seriously, and severe weather such as haze has occurred frequently. The PM2.5 value has attracted more and more attention. Images taken in foggy weather are blurry, lack clarity, lack details, have reduced color fidelity, and have serious color shift and distortion, which cannot achieve satisfactory visual effects. This has limited and affected the effectiveness of systems such as outdoor target recognition and tracking, intelligent navigation, highway visual surveillance, satellite remote sensing monitoring, and military aviation reconnaissance, and has had a great impact on production and life.

[0003] Autumn and winter are usually the seasons with the highest incidence of fog. For drones flying in the air, the greater the distance between them and the ground target, the more serious the impact of fog will be. Due to the influence of fog, the video images taken by the drone will appear white, with reduced contrast and faded colors, which will not only reduce the visual quality of these images, but also seriously affect the performance of computer vision algorithms that use these image data as processing sources.

[0004] Therefore, there is an urgent need for an image defogging method to effectively defog foggy images.

[0005] From the perspective of biological visual systems, in order to achieve efficient information processing, biological visual systems have developed into an effective information processing system that can improve the quality of input visual information. Taking the human visual system as an example, in the early visual system, the input visual image is processed by two types of cells in the retina, namely Midget cells and Parasol cells. Midget cells are small cells with a small receptive field and can be used to encode high-frequency information, such as detail information and noise information in the image, corresponding to the detail component of the image; correspondingly, Parasol cells with a larger receptive field are better at encoding low-frequency information, such as the overall structure of the image, corresponding to the basic component of the image. In this way, the channel processed by Midget cells (M-channel) and the channel processed by Parasol cells (P-channel) serve as two parallel visual channels in the retina, processing the detail component and basic component of the image respectively.

[0006] Inspired by the above-mentioned biological visual system mechanism, this application proposes a detail-enhanced single image defogging algorithm based on a deep residual network. Inspired by the two visual channel mechanism in the early visual system, the foggy image to be processed is decomposed into a basic component image (mainly including the color, contrast and brightness of the foggy image) and a detail component image (mainly including details, textures with sharp edge changes and other features) and processed in parallel to improve the network defogging effect. Summary of the invention

[0007] The present invention provides a method and device for defogging unmanned aerial images based on residual detail enhancement, so as to solve the problem that the prior art is difficult to effectively learn the color information of the image and cannot well achieve good convergence of image restoration.

[0008] The technical solution adopted by the present invention is as follows:

[0009] In a first aspect, the present invention discloses a method for defogging drone aerial images based on residual detail enhancement, comprising:

[0010] Obtain multiple pairs of foggy aerial images taken by drones in the same scene and their corresponding clear images as training data sets, and obtain foggy images in different scenes as foggy images to be processed;

[0011] Decomposing the foggy image to be processed into a basic component image and a detail component image;

[0012] Performing defogging processing on the base component image of the foggy image to be processed to obtain a defogged base component image, wherein the defogging processing is a multi-scale residual network algorithm;

[0013] Performing detail enhancement processing on the detail component image of the foggy image to be processed to obtain an enhanced detail component image and use it as the detail component image of the defogged image, wherein the detail enhancement processing adopts a nonlinear regression detail enhancement algorithm;

[0014] The defogging basic component image and the detail component image after detail enhancement are added together to obtain a defogging image.

[0015] In an achievable implementation, decomposing the foggy image to be processed into a basic component image and a detail component image includes:

[0016] The foggy image to be processed is decomposed into a basic component image and a detail component image through Gaussian kernel function filtering.

[0017] In one achievable implementation, it further includes:

[0018] A plurality of pairs of foggy images and their corresponding clear images in the same scene are obtained as training data sets; and the foggy images and their corresponding clear image samples in the training data sets are respectively decomposed into basic component images and detail component images.

[0019] In an achievable implementation, defogging is performed on a base component image of a foggy image to be processed to obtain a defogged base component image, including:

[0020] Build a dehazing network model based on ResNet;

[0021] Inputting the basic component image of the foggy image and the basic component image of the corresponding clear image as training samples into the ResNet defogging network model for training, thereby obtaining a trained ResNet defogging network model;

[0022] The base component image of the foggy image to be processed is input into the trained ResNet defogging network model to obtain the negative residual between the base component of the foggy image to be processed and the base component of the corresponding defogging image;

[0023] Then, the basic component of the foggy image to be processed is added to the negative residual to obtain the basic component of the defogged image.

[0024] Furthermore, the depth of the ResNet dehazing network model is set to 26 layers, the first 25 layers are composed of Conv, BN and Relu, the 26th layer is composed of Conv and BN, the parameters are set to 16×3×3×3, and the 26th layer is used to estimate the negative residual mapping of the basic component;

[0025] The parameters in the first layer are set to 3×3×3×16, and the parameters in the second to 25th layers are set to 16×3×3×16;

[0026] The size of each layer filter is 16;

[0027] The parameters of the 26th layer are set to 16×3×3×3.

[0028] In an achievable implementation, performing detail enhancement processing on the detail component image of the foggy image to be processed, obtaining the enhanced detail component image and using it as the detail component image of the defogging image, comprises:

[0029] Construct detail enhancement network model;

[0030] Inputting the detail component image of the foggy image and the detail component image of the corresponding clear image as training samples into the detail enhancement network model for training, thereby obtaining a trained detail enhancement network model and a detail enhancement factor;

[0031] The detail component image of the foggy image to be processed is multiplied by the enhancement factor obtained through the trained detail enhancement network to obtain an enhanced detail component image and serve as the detail component image of the defogged image.

[0032] Furthermore, the detail enhancement network sequentially includes two convolutional layers and an average operation layer;

[0033] In the two convolutional layers:

[0034] The first convolutional layer contains 32 filters with a kernel size of 11×11, and reduces the size of the feature channel with a step size of 2×2;

[0035] The second convolutional layer is a filter with a kernel size of 1×1 and a stride of 1×1.

[0036] In a second aspect, the present invention discloses a UAV aerial image defogging device based on residual detail enhancement, comprising:

[0037] An acquisition module is used to acquire multiple pairs of foggy aerial images taken by drones in the same scene and their corresponding clear images as training data sets, and to acquire foggy images in different scenes as foggy images to be processed;

[0038] A decomposition module, used for decomposing the foggy image to be processed into a basic component image and a detail component image;

[0039] A defogging module is used to perform defogging on the basic component image of the foggy image to be processed to obtain a defogged basic component image, wherein the defogging process is a multi-scale residual network algorithm;

[0040] A detail enhancement module is used to perform detail enhancement processing on the detail component image of the foggy image to be processed, obtain the enhanced detail component image and use it as the detail component image of the defogged image, and the detail enhancement processing adopts a nonlinear regression detail enhancement algorithm;

[0041] The adding module is used to add the basic component image after defogging and the detail component image after detail enhancement to obtain a defogging image.

[0042] In a third aspect, the present invention further discloses a computer device, comprising:

[0043] one or more processors;

[0044] a memory for storing one or more programs,

[0045] When the one or more programs are executed by the one or more processors, the one or more processors implement the UAV aerial image defogging method based on residual detail enhancement as described above.

[0046] In a fourth aspect, the present invention further discloses a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method for defogging drone aerial images based on residual detail enhancement as described above.

[0047] The beneficial effects of the technical solution of the present invention are as follows:

[0048] The invention discloses a method, device and computer equipment for defogging drone aerial images based on residual detail enhancement. Inspired by the two visual channel mechanisms in early visual systems, the foggy image to be processed is first decomposed into a basic component image and a detail component image; then the basic component image is defogged to obtain a defogged basic component image, and the defogging process is a multi-scale residual network algorithm; then the detail component image is subjected to detail enhancement to obtain an enhanced detail component image, and the detail enhancement process adopts a nonlinear regression detail enhancement algorithm; finally, the defogged basic component image and the detail component image after detail enhancement are added and processed to obtain a defogged image. Experimental results show that the invention can not only effectively remove fog in an image, but also better restore the detail information of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 Schematic diagram of residual learning unit;

[0051] Figure 2 This is a flow chart of a method for defogging UAV aerial images based on residual detail enhancement according to the present invention;

[0052] Figure 3 It is a ResNet defogging network model in a UAV aerial image defogging method based on residual detail enhancement of the present invention;

[0053] Figure 4 A detail enhancement network in a method for defogging drone aerial images based on residual detail enhancement according to the present invention;

[0054] Figure 5 The present invention is a flow chart of a method for defogging unmanned aerial images based on residual detail enhancement. DETAILED DESCRIPTION

[0055] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following embodiments do not represent all embodiments consistent with the present invention. They are only examples of devices and methods consistent with some aspects of the present invention as detailed in the claims.

[0056] Fog is a common natural weather phenomenon. The images collected by imaging equipment on foggy days have problems such as blurred details, color distortion, and low contrast, which seriously affect the visual effects of the images and increase the difficulty of subsequent image processing.

[0057] From the perspective of biological visual systems, in order to achieve efficient information processing, biological visual systems have developed into an effective information processing system that can improve the quality of input visual information. Taking the human visual system as an example, in the early visual system, the input visual image is processed by two types of cells in the retina, namely Midget cells and Parasol cells. Midget cells are small cells with a small receptive field and can be used to encode high-frequency information, such as detail information and noise information in the image, corresponding to the detail component of the image; correspondingly, Parasol cells with a larger receptive field are better at encoding low-frequency information, such as the overall structure of the image, corresponding to the basic component of the image. In this way, the channel processed by Midget cells (M-channel) and the channel processed by Parasol cells (P-channel) serve as two parallel visual channels in the retina, processing the detail component and basic component of the image respectively.

[0058] Inspired by the mechanism of the biological vision system, the present invention proposes a method for defogging UAV aerial images based on residual detail enhancement inspired by biological vision, which decomposes the foggy image to be processed into a basic component image (mainly including the color, contrast and brightness of the foggy image) and a detail component image (mainly including details, textures with sharp edge changes and other features) and processes them in parallel to improve the network defogging effect. That is, the foggy image to be defogged is first decomposed to obtain the basic component image and the detail component image. The defogging network framework proposed for the basic component image adopts a jump connection method to directly propagate lossless information through the entire network, which is more conducive to network training. The defogging network framework can output the negative residual between the basic component image of the foggy image and the basic component image of the corresponding clear image, and add it to the input basic component image of the foggy image to obtain the basic component of the foggy image to be defogged; further, in order to compensate for the detail texture loss of the foggy image to be defogged, a detail enhancement module is built, which takes the detail component image of the foggy image to be defogged as input information, uses nonlinear regression prediction to obtain the enhancement factor to enhance the image detail component image; the basic component image of the foggy image to be defogged and the detail component image of the foggy image to be defogged are added and processed to obtain the defogged image. Experimental results show that the algorithm of the present invention can not only effectively remove fog in the image, but also better restore the detail information of the image.

[0059] Therefore, in order to solve the problems of image detail information loss, color distortion, low contrast, etc. in foggy scenes, the present invention proposes a method, device, computer equipment and computer-readable medium for defogging drone aerial images based on residual detail enhancement, as described below.

[0060] Among them, the relevant introduction of the residual network mentioned above is as follows:

[0061] In traditional CNN, as the number of network layers increases, the classification accuracy of the network model decreases, which is called the network degradation problem. Based on this, the deep residual network - ResNet can effectively solve this problem. For an ordinary neural network, when the input is x, its output is H(x), and now the residual F(x) = H(x)-x is obtained. The reason is that the residual F(x) is easier to learn than H(x). Even if F(x) = 0, the network performance has not decreased, which is equivalent to the identity mapping. The basic structure of the residual unit is as follows Figure 1 As shown:

[0062] Figure 1It can be seen that by adding skip connections in the residual unit, the network can receive all the information from the input data to the greatest extent. And the residual learning content is less, which is easier than the ordinary CNN network. Next, we analyze the network from the perspective of mathematical theory. First, the residual unit is expressed as:

[0063] y l =h(x l )+F(x l ,W l ) (1)

[0064] and

[0065] x l+1 =f(y l ) (2)

[0066] In the above formula, x l and x l+1 They represent the input information and output information of the lth residual unit respectively; F(·) represents the residual function, which is the residual information learned by the residual unit above; h(x l )=x l represents the identity mapping; f(·) is the ReLU activation function. Through the above formula. Going further, we can write the expression of the residual structure:

[0067] x l+1 =x l +F(x l ,W l ) (3)

[0068] By recursively performing the above formula, we can obtain the learning features from the shallow layer i to the deep layer l in the residual network:

[0069]

[0070] In the above formula (4), x L is the characteristic expression of the unit L at any depth. From the above formulas (1) to (4), we can see that for the characteristic x of the unit L at any depth L It can be expressed as the feature x of the shallow unit l l Add a shape like The residual function shows that there is a residual feature between any unit L and l.

[0071] Another design principle of the residual network is to always ensure the number of image features extracted by the network. If the size of the feature map itself is reduced to half of the original size, its number will double. This operation ensures the richness of the image features extracted by the network, and adding a jump connection mechanism between every two layers forms residual learning.

[0072] About See Figures 1 to 5 shown.

[0073] First, as Figure 1 and Figure 5 As shown, the present invention discloses a method for defogging unmanned aerial image based on residual detail enhancement, comprising:

[0074] S100: Obtain foggy aerial images taken by a drone in different scenes as foggy images to be processed.

[0075] In one achievable implementation, it further includes:

[0076] Multiple pairs of foggy images and their corresponding clear images in the same scene are obtained as training data sets.

[0077] S200: Inspired by the dual-channel processing mechanism in early biological vision systems, the foggy image to be processed is decomposed into a basic component image and a detail component image.

[0078] For any digital image I, inspired by the dual-channel processing mechanism in early biological visual systems, the image can be decomposed into the sum of the basic component image and the detail component image, and its expression is:

[0079] I=I base +I detail (5)

[0080] In an achievable implementation, the method further includes: train haze And its corresponding clear image I train clear To break it down:

[0081] The foggy image I of the training dataset train haze Decompose into basic component image I train haze base and detail component image I train haze detail ;

[0082] The corresponding clear image I train clear Decompose into basic component image I train clear base and detail component image I train clear detail .

[0083] Among them, it should be noted here that the basic component image refers to the low-frequency (Low-Frequency) part of the image, and the detail component image refers to the high-frequency (High-Frequency) part of the image. The most basic information of the image is retained in the low-frequency component image, while the details and other edge and texture information will be contained in the high-frequency component image of the image.

[0084] In step S200, the foggy image to be processed is decomposed into a basic component image and a detail component image, and the decomposition is performed by using a Gaussian kernel function filter.

[0085] For the input foggy image I to be processed haze , perform Gaussian filtering on it to obtain the basic component I haze_base and detail component I haze_detail Therefore, the foggy image I haze The modeling is as follows:

[0086] I haze =I haze_base +I haze_detail (6)

[0087] In order to obtain the basic component (low-frequency information) I haze_base The present invention performs Gaussian kernel blurring on the foggy image to be processed to eliminate the influence of texture and content information, and then extract low-frequency information such as fog, color saturation, contrast and brightness of the image. The action process of the blurring operator is as follows:

[0088] I g (x,y)=∑ a,b I(x+a,y+b)*G(a,b) (7)

[0089] In formula (6), I g (x, y) represents the image after blurring the foggy image I, and the blur operator is represented by G(a, b); a and b in the Gaussian blur operator G(a, b) determine the size of the blur radius, which is The Gaussian blur operator G(a,b) is defined as follows:

[0090]

[0091] In formula (7), σ is the standard deviation of the normal distribution. In order to eliminate high-frequency texture information, the image is Gaussian blurred to maximize the original information of the image while eliminating texture and content information. Through a large amount of data experiments, σ 2 When =3, the decomposition effect of various types of foggy images is the best.

[0092] The foggy image I in the above training dataset train_haze The corresponding clear image I train_clear , decompose it into the basic components I corresponding to the clear image train_clear_base The detail component I corresponding to the clear image train_clear_detail :

[0093] I train clear =I train clear base +I train clear detail(9)

[0094] S300: Base component image I of the foggy image to be processed haze base Perform defogging to obtain the defogging basic component image I dehazed base , the dehazing process is a multi-scale residual network algorithm.

[0095] Specifically, in step S300, the foggy image I to be processed haze The basic component image I haze base Perform defogging to obtain the defogging basic component image I dehazed base ,include:

[0096] Build a ResNet dehazing network model;

[0097] The basic component image I of the foggy image in the training dataset train haze base And its corresponding basic component image I of the clear image train clear base The training samples are input into the ResNet defogging network model to be trained to obtain a trained ResNet defogging network model;

[0098] The foggy image to be processed I haze The basic component image I haze base Input into the trained ResNet defogging network model to obtain the basic component I of the foggy image haze base The corresponding basic component I of the dehazed image dehazed base The negative residual R(x,y) between

[0099] The basis component I of the input foggy image haze base Added to the negative residual R(x,y) above, we get the basic component I of the dehazed image. dehazed base .

[0100] Furthermore, the depth of the ResNet defogging network model is set to 26 layers. The first 25 layers are composed of Conv, BN (BatchNormalization) and ReLU (rectified linear unit), and the 26th layer is composed of Conv and BN, with parameters set to 16×3×3×3; the addition of the BN layer is to solve the problem of excessive difficulty in network training, and it also helps to curb network overfitting. The ReLu activation function reduces the mutual dependence between parameters, making the network sparse and better fitting the nonlinear mapping relationship between the foggy image domain and the clear image. The parameters in the first layer are set to 3×3×3×16, and the parameters in the second to 25th layers are set to 16×3×3×16; the size of the filter in each layer is 16, because a larger filter size can capture more structural information; the parameters of the 26th layer are set to 16×3×3×3, which is used to estimate the basic component image I of the foggy image to be processed. haze base And its corresponding basic component image I after defogging dehazed base The negative residual R(x,y) mapping.

[0101] That is to say, assuming that the final dehazed image is I dehazed The present invention obtains the basic component I corresponding to the foggy image dehazed_base And its corresponding basic component I of the dehazed image dehazed_detail The negative residual between is defined as:

[0102] R(x,y)=I dehazed_base -I haze_base (10)

[0103] Then through the formula:

[0104] R(x,y)+I haze_base (11)

[0105] To solve the basic component I of the dehazed image dehazed_base .

[0106] At the same time, in order to avoid the color distortion problem that may occur during the defogging process, the color information of the input blurred image is maintained by adding jump connections, which can be achieved by embedding the residual learning concept into our defogging framework, because the lossless information can be directly used to update the parameters through the jump connection. In addition, solving the base component image of the image to be defogged by predicting the negative residual of the base component image of the defogged image and the base component of the foggy image to be processed is more conducive to model training, because the above negative residual mapping has lower training and test errors than ResNet. Existing image defogging methods based on ResNet usually adopt an "end-to-end" defogging method, with the image to be defogged as input and the clear image after defogging as output.

[0107] Different from this, the present invention is inspired by the dual-channel processing mechanism of the early biological vision system. First, the foggy image to be processed is decomposed to obtain the basic component image and the detail component image. The defogging network proposed for the basic component image adopts a jump connection method to directly propagate lossless information through the entire network, which is more conducive to network training. The defogging network framework can output the negative residual between the basic component image of the foggy image to be processed and the corresponding basic component image of the defogged image. The negative residual is added to the basic component image of the original foggy image to be processed to obtain the basic component of the defogged image; further, in order to compensate for the detail texture loss of the foggy image to be processed, the present invention builds a detail enhancement network, which takes the detail component image of the foggy image of the training data set as input information, uses nonlinear regression prediction to obtain the enhancement factor to enhance the image detail component image, and obtains the detail component image of the defogged image; the basic component image of the defogged image and the detail component image of the defogged image are added and processed to obtain the defogged image. Experimental results show that the algorithm of the present invention can not only effectively remove fog in the image, but also better restore the detail information of the image.

[0108] The present invention uses the ResNet network structure to predict the negative residual between the basic components of the foggy image to be processed and the basic components of the corresponding defogging image, which reduces the difficulty of network training. At the same time, adding jump connections in the network helps the input information pass through all parameter layers and maximize the extraction of image feature information.

[0109] The ResNet defogging network model proposed in the present invention can better extract the features of foggy images while retaining all the information in the original image. In addition, the jump connection operation in the residual network can effectively solve the gradient vanishing problem existing in the convolutional neural network and improve the propagation efficiency of the image information flow. Among them, the method of the present invention only learns the negative residual between the basic component image of the foggy image to be processed and the basic component of the defogged image corresponding thereto, and adds the above negative residual to the basic component image of the original input foggy image to be processed to obtain the basic component of the defogged image. The overall ResNet defogging network structure is as follows: Figure 3 shown.

[0110] S400: performing detail enhancement processing on the detail component image of the foggy image to be processed, obtaining the enhanced detail component image and using it as the detail component image of the defogged image, wherein the detail enhancement processing adopts a nonlinear regression algorithm to enhance the details.

[0111] In an achievable implementation, performing detail enhancement processing on the detail component image of the foggy image to be processed to obtain an enhanced detail component image as the detail component image of the defogged image includes:

[0112] Construct detail enhancement network model;

[0113] The detail component images of the foggy images and the detail component images of the corresponding clear images in the training data set are input as training samples into the detail enhancement network model to be trained, so as to obtain the trained detail enhancement network model and then obtain the detail enhancement factor;

[0114] The detail component image of the foggy image to be defogged is multiplied by the enhancement factor obtained by the trained detail enhancement network model to obtain an enhanced detail component image as the detail component image of the defogged image.

[0115] Furthermore, the detail enhancement network includes two convolutional layers and an averaging operation layer in sequence, the two convolutional layers are used to extract feature tensors of detail component images of foggy images and their corresponding clear images, the averaging operation layer is used to perform mean operation processing on all elements of the feature tensor, and obtain enhancement factors of detail component images of foggy images and their corresponding clear images for training, among the two convolutional layers: the first layer contains 32 filters with a convolution kernel size of 11×11, and reduces the size of the feature channel with a step size of 2×2; the second layer is a filter with a kernel size of 1×1 and a step size of 1×1.

[0116] Step S400 uses nonlinear regression to enhance the detail component image I of the foggy image to be processed. haze_detailSpecifically, for a given blurred image, the proposed detail enhancement network learns an enhancement factor M. As shown in formula (11):

[0117] E(I dehazed_detail )=M×I haze_detail (12)

[0118] Where M is the same as the input foggy image I haze Related details enhance predictors, I haze detail is the detail image of the foggy image to be defogged. The overall detail enhancement network structure is as follows Figure 4 shown.

[0119] Furthermore, if Figure 4 As shown in the figure, the detail enhancement network model consists of two convolution layers and an average operation layer. The two convolution layers are used to extract the feature tensors of the second detail component image and the third detail component image. The average operation layer is used to perform mean operation on all elements of the feature tensor and obtain the enhancement factors of the second detail component image and the third detail component image. The input of the detail enhancement network model is the detail component corresponding to the foggy image to be defogged. The first layer contains 32 filters with a convolution kernel size of 11×11, and the size of the feature channel is reduced with a step size of 2×2, which will help reduce the computational cost; the second layer is a filter with a kernel size of 1×1 and a step size of 1×1. Finally, by calculating the average value of all elements from the input feature map, the predicted enhancement factor M for detail component image enhancement can be obtained, that is, the enhancement factor M of the foggy image-clear image data pair.

[0120] According to the two visual channel mechanism in the early visual system, any image I can be decomposed into the sum of the basic component image and the detail component image using the expression shown in formula (4). Therefore, the calculation method of the detail component image can be reversed from formula (4) as follows:

[0121] I detail =II base (13)

[0122] For the paired foggy images I in the training dataset train_haze And its corresponding clear image I train_clear , the Gaussian kernel blur function shown in formula (5) is used to decompose the image and obtain the detail component images I and I of the foggy image. train_haze_detail And its corresponding detail component image I of the clear image train_clear_detail Then, the predicted value of the enhancement factor M for the training foggy image-clear image data pair can be calculated and averaged by the following formula, as shown below:

[0123]

[0124] Where (i, j, k) represents the coordinates of the tensor elements; M i,j,k That is the detail component image I of the foggy image train_haze_detail The tensor (I train_heze_detail ) i,j,k and the corresponding detail component image I of the clear image train_clear_detail The tensor (I train_clear_detail ) i,j,k All channels obtain the predicted enhancement factor M of the training image pair by averaging the calculation results of all element directions of the two tensors.

[0125] After training the detail enhancement network model using pairs of foggy images and their corresponding clear images in the training data set, the trained detail enhancement network model can be obtained, and then the detail enhancement factor M value can be obtained. The detail component image of the foggy image to be processed can be directly multiplied by the enhancement factor M value obtained by the trained detail enhancement network model to obtain the enhanced detail component image.

[0126] S500: performing sum processing on the defogging basic component image and the enhanced detail component image to obtain a defogging image.

[0127] The existing deep learning-based defogging algorithms usually adopt an "end-to-end" defogging model, taking the foggy image as input information, using the defogging network to complete the defogging process, and outputting the corresponding clear image. In the network training stage, pairs of foggy image samples and their corresponding clear image samples are used as training data, and the foggy image is directly sent to the network for the model to learn an end-to-end mapping relationship. However, the color information of the image is often not effectively learned during the training process, and good convergence of image restoration cannot be achieved well.

[0128] The present invention selects the basic component as the input information of the defogging network in order to evaluate the differences in brightness, contrast and color saturation between images, and eliminate texture and content information at the same time; the detail enhancement network is to compensate for the detail information lost after the image is defogged. In addition, most of the existing image defogging and enhancement algorithms are end-to-end defogging models, which generally send the entire image (including color contrast and texture) directly into the network for the model to learn an end-to-end mapping relationship. However, the color information of the image may not be well learned during the training process, and the good convergence of image restoration cannot be well achieved. Inspired by the two visual channel mechanisms in the early visual system, the present invention decomposes the foggy image to be processed into a first basic component image and a first detail component image, and proposes a detail enhancement single image defogging algorithm based on a deep residual network inspired by biological vision, which decomposes the image into two parts: a basic component image (color, saturation, contrast) and a detail component image (details, texture information) for parallel processing to improve the network defogging effect.

[0129] The present invention proposes a method for defogging UAV aerial images based on residual detail enhancement inspired by biological vision. First, the foggy image to be processed is decomposed into a basic component image and a detail component image; then the basic component image is defogged to obtain a defogged basic component image, and the defogging process is a multi-scale residual network algorithm; then the detail component image of the foggy image is subjected to detail enhancement to obtain an enhanced detail component image, and the detail enhancement process adopts a nonlinear regression detail enhancement algorithm; finally, the defogged basic component image and the detail component image after detail enhancement are added together to obtain a defogged image. Experimental results show that the present invention can not only effectively remove fog in an image, but also better restore the detail information of the image.

[0130] Specifically, the present invention first decomposes the foggy image to be processed into a basic component image and a detail component image; then a defogging network framework is proposed for the basic component image of the foggy image, and lossless information is directly transmitted through the entire network by using a jump connection method. The defogging network framework can output the negative residual between the basic component image of the foggy image and the basic component image of the corresponding defogged image, and the above negative residual is added to the input basic component image of the foggy image to be processed to obtain the basic component of the defogged image; further, in order to compensate for the detail texture loss of the foggy image to be defogged, a detail enhancement module is built, which uses the second detail component image of the foggy image and the third detail component of the clear image as training network information, uses nonlinear regression prediction to obtain an enhancement factor, and multiplies the detail component image with the enhancement factor M value obtained by the trained detail enhancement network model to enhance the detail component image of the foggy image to be processed; the defogged basic component image and the enhanced detail component image are added and processed to obtain the defogged image. Experimental results show that the algorithm of the present invention can not only effectively remove fog in the image, but also better restore the detail information of the image.

[0131] In a second aspect, the present invention provides a UAV aerial image defogging device based on residual detail enhancement, comprising:

[0132] An acquisition module is used to acquire foggy aerial images taken by drones in different scenes as foggy images to be processed;

[0133] A decomposition module, used for decomposing the foggy image to be processed into a basic component image and a detail component image;

[0134] A defogging module is used to perform defogging on the basic component image of the foggy image to be processed to obtain a defogged basic component image, wherein the defogging process is a multi-scale residual network algorithm;

[0135] A detail enhancement module is used to perform detail enhancement processing on the detail component image of the foggy image to be processed, obtain the enhanced detail component image and use it as the detail component image of the defogged image, and the detail enhancement processing adopts a nonlinear regression detail enhancement algorithm;

[0136] The adding module is used to add the basic component image after defogging and the detail component image after detail enhancement to obtain a defogging image.

[0137] It should be noted that: the UAV aerial image defogging device based on residual detail enhancement provided in the above embodiment only uses the division of the above functional modules as an example when performing image defogging. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the UAV aerial image defogging device based on residual detail enhancement provided in the above embodiment and the UAV aerial image defogging method embodiment based on residual detail enhancement belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0138] Among them, the specific definition of the drone aerial image defogging device based on residual detail enhancement can be found in the above definition of the drone aerial image defogging method based on residual detail enhancement, which will not be elaborated here. In addition, each module in the above-mentioned drone aerial image defogging device based on residual detail enhancement can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0139] As a third aspect, the present invention also discloses a computer device, which may be a server. The computer device includes: one or more processors for providing computing and control capabilities; a memory, the memory is used to store one or more programs, when one or more programs are executed by one or more processors, one or more processors implement the following steps: obtaining aerial foggy images of drones in different scenes as foggy images to be processed; decomposing the foggy images to be processed into basic component images and detail component images; defogging the basic component images of the foggy images to be processed to obtain the defogged basic component images, the defogging process is a multi-scale residual network algorithm; detail enhancement process is performed on the detail component images of the foggy images to be processed, the enhanced detail component images are obtained and used as the detail component images of the defogged images, the detail enhancement process adopts a nonlinear regression detail enhancement algorithm; the defogged basic component images and the detail component images after detail enhancement are summed to obtain the defogged images.

[0140] As a fourth aspect, the present invention also discloses a computer-readable medium having a computer program stored thereon, and the computer-readable medium may be included in the device described in the above embodiment, or may exist independently without being assembled into the device. The above program is executed by the processor in the following steps: obtaining aerial foggy images taken by drones in different scenes as foggy images to be processed; decomposing the foggy images to be processed into basic component images and detail component images; defogging the basic component images of the foggy images to be processed to obtain the defogged basic component images, and the defogging process is a multi-scale residual network algorithm; performing detail enhancement process on the detail component images of the foggy images to be processed to obtain the enhanced detail component images and use them as the detail component images of the defogged images, and the detail enhancement process adopts a nonlinear regression detail enhancement algorithm; adding the defogged basic component images and the detail component images after detail enhancement to obtain the defogged images.

[0141] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that an article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0142] The above description is only a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but shall conform to the widest scope consistent with the principles and novel features disclosed herein.

[0143] It should be understood that the present invention is not limited to what has been described above and shown in the drawings, and various modifications and changes may be made without departing from its scope. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for defogging drone aerial images based on residual detail enhancement, characterized in that: include: Obtain foggy aerial images taken by drones in different scenes as foggy images to be processed; Decomposing the foggy image to be processed into a basic component image and a detail component image; Defogging the basic component image of the foggy image to be processed to obtain a defogged basic component image, The dehazing process is a multi-scale residual network algorithm; Performing detail enhancement processing on the detail component image of the foggy image to be processed to obtain an enhanced detail component image and use it as the detail component image of the defogged image, wherein the detail enhancement processing adopts a nonlinear regression detail enhancement algorithm; Adding the defogging basic component image and the detail component image after detail enhancement to obtain a defogging image; Decomposing the foggy image to be processed into a basic component image and a detail component image, including: Decomposing the foggy image to be processed into a basic component image and a detail component image by Gaussian kernel function filtering; The method further comprises: Obtain multiple pairs of foggy images and their corresponding clear images in the same scene as training data sets; Decomposing the foggy image and its corresponding clear image samples in the training data set into basic component images and detail component images respectively; The basic component image of the foggy image to be processed is defogged to obtain the defogged basic component image, including: Build a dehazing network model based on ResNet; Inputting the basic component image of the foggy image and the basic component image of the corresponding clear image as training samples into the ResNet defogging network model for training, thereby obtaining a trained ResNet defogging network model; The base component image of the foggy image to be processed is input into the trained ResNet defogging network model to obtain the negative residual between the base component of the foggy image to be processed and the base component of the corresponding defogging image; Then, the basic component of the foggy image to be processed is added to the negative residual to obtain the basic component of the defogged image; The ResNet dehazing network model has a depth of 26 layers, the first 25 layers are composed of Conv, BN and Relu, the 26th layer is composed of Conv and BN, and the parameters are set to 16×3×3×3. The 26th layer is used to estimate the negative residual mapping of the basic component; The parameters in the first layer are set to 3×3×3×16, and the parameters in the second to 25th layers are set to 16×3×3×16; The size of each layer filter is 16; The parameters of the 26th layer are set to 16×3×3×3.

2. The method for defogging drone aerial images based on residual detail enhancement according to claim 1 is characterized in that: Performing detail enhancement processing on the detail component image of the foggy image to be processed, obtaining the enhanced detail component image and using it as the detail component image of the defogging image, including: Construct detail enhancement network model; Inputting the detail component image of the foggy image and the detail component image of the corresponding clear image as training samples into the detail enhancement network model for training, thereby obtaining a trained detail enhancement network model and a detail enhancement factor; The detail component image of the foggy image to be processed is multiplied by the enhancement factor obtained through the trained detail enhancement network to obtain an enhanced detail component image and serve as the detail component image of the defogged image.

3. The method for defogging drone aerial images based on residual detail enhancement according to claim 2 is characterized in that: The detail enhancement network sequentially includes two convolutional layers and an average operation layer; In the two convolutional layers: The first convolutional layer contains 32 filters with a kernel size of 11×11, and reduces the size of the feature channel with a step size of 2×2; The second convolutional layer is a filter with a kernel size of 1×1 and a stride of 1×1.

4. A computer device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the UAV aerial image defogging method based on residual detail enhancement as described in any one of claims 1 to 3.

5. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for defogging unmanned aerial images based on residual detail enhancement as described in any one of claims 1 to 3 is implemented.

Citation Information

Patent Citations

  • Image rain removal model training method, device and image rain removal method

    CN108537761A

  • Image defogging method and system, electronic equipment and storage medium

    CN110197469A

  • Image defogging method and device based on combination of priori knowledge and deep learning

    CN111161159A

  • Method and System for Processing an Input Image

    US20160292824A1