Image defogging method for extremely complex environment, medium and program product

Abstract

The invention discloses an image defogging method for an extremely complex environment, a medium and a program product, and the method comprises the steps: inputting an obtained real-time degraded image into a pre-trained clear image extraction network, and generating a defogged clear image; the training method of the clear image extraction network comprises the following steps: inputting a historical degraded image into an atmospheric light extraction module to obtain an atmospheric light value; inputting the historical degraded image into a clear image extraction network to obtain a clear image; inputting the historical degraded image into a transmission image extraction network to obtain a transmission image; inputting the obtained clear image, the projection image and the atmospheric light value into an atmospheric scattering model to obtain a synthetic degraded image; and based on a first error between the synthesized degraded image and the historical degraded image and a second error between the clear image and the standard clear image, adjusting parameters of the clear image extraction network and the transmission image extraction network, and completing training of the clear image extraction network and the transmission image extraction network. According to the invention, the image defogging effect can be effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an image defogging method, a medium and a program product for extremely complex environments. Background Art

[0002] Images are the key to obtaining information in machine vision systems, especially in the field of surveillance images. However, bad weather such as foggy days can cause image color distortion and blurred details, reducing the application value of images and posing challenges to criminal investigations and image evidence collection. Therefore, it is of great significance to study the restoration of degraded images in foggy days.

[0003] Existing image dehazing methods can be divided into two categories: traditional methods combined with physical models and algorithms based on deep learning. Traditional methods are based on atmospheric scattering models, and representative algorithms include the DCP algorithm based on dark channel prior, the adaptive linear transmittance estimation algorithm, the algorithm based on the improved atmospheric scattering model, and the color attenuation prior algorithm (CAP). These algorithms have good interpretability, but the dehazing effect still needs to be improved.

[0004] In terms of deep learning, representative algorithms include DehazeNet, FFA-Net, AOD Net, and closed fusion methods. These algorithms have better dehazing effects than traditional methods, but they require pre-trained models, and deep learning algorithms that use synthetic images as data sets may not get expected results when processing real degraded images.

[0005] In summary, developing an image dehazing method for extremely complex environments is of great significance to the current image and video field. Summary of the invention

[0006] In view of the above problems, the present invention proposes an image defogging method, medium and program product for extremely complex environments, which can effectively improve the image defogging effect.

[0007] In order to achieve the above technical objectives and the above technical effects, the present invention is implemented through the following technical solutions:

[0008] In a first aspect, the present invention provides an image defogging method for extremely complex environments, comprising:

[0009] The acquired real-time degraded image is input into a pre-trained clear image extraction network to generate a clear image after defogging; wherein the training method of the clear image extraction network includes:

[0010] The acquired historical degraded image is input into the atmospheric light extraction module to obtain the atmospheric light value;

[0011] Input the acquired historical degraded image into a clear image extraction network to obtain a clear image;

[0012] Inputting the acquired historical degradation image into a transmission map extraction network to obtain a transmission map;

[0013] The obtained clear image, projection image and atmospheric light value are input into the atmospheric scattering model to obtain a synthetic degraded image;

[0014] Based on a first error between a synthetic degraded image and a historical degraded image and a second error between a clear image and a standard clear image, parameters of a clear image extraction network and a transmission image extraction network are adjusted to complete the training of the clear image extraction network and the transmission image extraction network.

[0015] In combination with the first aspect, optionally, the atmospheric light extraction module adopts a quartering atmospheric light extraction strategy, divides the original image into four equal parts, obtains the atmospheric light value of each part and calculates the average value; the mathematical expression of the atmospheric light value is:

[0016] ;

[0017] in, Indicates the atmospheric light value; , , , They represent the atmospheric light value of each of the four equal parts of the degraded image; Represents spatial coordinates.

[0018] In combination with the first aspect, optionally, the synthesized degraded image is obtained by using the following calculation formula:

[0019] ;

[0020] in, To synthesize the degraded image, For a clear picture, is the transmission diagram, is the atmospheric light value, Represents spatial coordinates.

[0021] In combination with the first aspect, optionally, the clear image extraction network includes a backbone network, a pixel attention branch, a channel attention branch and a fusion layer;

[0022] The backbone network, the pixel attention branch, and the channel attention branch are arranged in parallel, and the inputs of the three are all synthetic degraded images;

[0023] The fusion layer fuses the feature maps output by the backbone network, the pixel attention branch, and the channel attention branch.

[0024] In combination with the first aspect, optionally, the backbone network is used to capture and refine key weights of input features, including a 3×3 convolution kernel, a ReLU activation function, and a 3×3 convolution kernel arranged in sequence;

[0025] The pixel attention branch includes an average pooling layer, a 3×3 convolution kernel, a first residual connection layer, a 3×3 convolution kernel and a Sigmoid activation function arranged in sequence; two inputs of the first residual connection layer come from the pixel attention branch and the channel attention branch respectively;

[0026] The channel attention branch includes a 1×1 convolution kernel, a maximum pooling layer, a 1×1 convolution kernel, a second residual connection layer, a 1×1 convolution kernel and a Sigmoid activation function arranged in sequence; two inputs of the second residual connection layer come from the pixel attention branch and the channel attention branch respectively;

[0027] The fusion layer uses element-wise multiplication to fuse the feature maps output by the backbone network, the pixel attention branch, and the channel attention branch.

[0028] In combination with the first aspect, optionally, the transmission map extraction network includes a plurality of Conv-ReLU convolution units, 1×1 convolution kernels and Leaky-ReLU activation functions arranged in sequence;

[0029] The output of the previous Conv-ReLU convolution unit is connected to the input and output of the next Conv-ReLU convolution unit.

[0030] In combination with the first aspect, optionally, the Conv-ReLU convolution unit includes a 1×1 convolution kernel and a -ReLU activation function connected sequentially.

[0031] In combination with the first aspect, optionally, the first error is calculated using a PSNR loss function; and the second error is calculated using an SSIM loss function.

[0032] In a second aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the distribution network energy storage configuration method for photovoltaic access as described in any one of the first aspects is implemented.

[0033] In a third aspect, the present invention provides a computer program product, including a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, the distribution network energy storage configuration method for photovoltaic access described in any one of the first aspects is implemented.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] The present invention inputs the acquired real-time degraded image into a pre-trained clear image extraction network to generate a clear image after defogging; wherein, the training method of the clear image extraction network includes: inputting the historical degraded image into the atmospheric light extraction module to obtain the atmospheric light value; inputting the historical degraded image into the clear image extraction network to obtain the clear image; inputting the historical degraded image into the transmission image extraction network to obtain the transmission image; inputting the obtained clear image, projection image and atmospheric light value into the atmospheric scattering model to obtain a synthetic degraded image; based on the first error between the synthetic degraded image and the historical degraded image and the second error between the clear image and the standard clear image, adjusting the parameters of the clear image extraction network and the transmission image extraction network to complete the training of the clear image extraction network and the transmission image extraction network. Based on the above processing, the present invention can effectively improve the image defogging effect.

[0036] The improved atmospheric scattering model proposed in the present invention is more suitable for degraded graphics reconstruction and clear image extraction in real severe foggy environment.

[0037] The transmission image extraction network proposed in the present invention utilizes the powerful learning ability of the deep learning model and the flexible processing ability of the multi-attention mechanism to adaptively extract the projection image according to the different input images, thereby improving the practicality of the invention.

[0038] The clear image extraction network proposed in the present invention further improves the integrity of the image by deepening the extraction of image features and proposing a compensation strategy.

[0039] The network training strategy proposed in the present invention can better utilize the various features of the input image in an all-round way and further improve the stability of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor, among which:

[0041] Figure 1 A schematic diagram of the principle of an image defogging method for extremely complex environments according to an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of the result of an atmospheric light extraction module processing a historically degraded image according to an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of a network structure for extracting a clear image according to an embodiment of the present invention;

[0044] Figure 4The figure is a schematic diagram of a transmission map extraction network structure according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0047] Example 1

[0048] The present invention provides an image defogging method for extremely complex environments. Figure 1 As shown, the following steps are included:

[0049] The acquired real-time degraded image is input into a pre-trained clear image extraction network to generate a clear image after defogging; wherein the training method of the clear image extraction network includes:

[0050] The acquired historical degraded image is input into the atmospheric light extraction module to obtain the atmospheric light value;

[0051] Input the acquired historical degraded image into a clear image extraction network to obtain a clear image;

[0052] Inputting the acquired historical degradation image into a transmission map extraction network to obtain a transmission map;

[0053] The obtained clear image, projection image and atmospheric light value are input into the atmospheric scattering model to obtain a synthetic degraded image;

[0054] Based on the first error (Loss1) between the synthetic degraded image and the historical degraded image and the second error (Loss2) between the clear image and the standard clear image, the parameters of the clear image extraction network and the transmission image extraction network are adjusted to complete the training of the clear image extraction network and the transmission image extraction network.

[0055] In the original atmospheric scattering model, atmospheric light refers to the scene brightness of the beam emitted by the current sun reaching the target. Although there is attenuation, the distance is large enough. Therefore, the original atmospheric scattering model usually sets the atmospheric light to a constant value. However, in a real harsh environment, due to on-site environmental problems, the brightness of the image may appear extremely bright or extremely dark, and the atmospheric light estimated in the original atmospheric scattering model is considered to be a constant value. Therefore, the atmospheric light estimated using the original model will not change with the brightness of the scene. As a result, the brightness of the clear image output by the model in a harsh environment may appear to be not full or too full. To this end, in a specific implementation of an embodiment of the present invention, in order to improve the stability of the atmospheric scattering model, combined with Figure 2 The atmospheric light extraction module adopts the quartering atmospheric light extraction strategy. After dividing the original image into four equal parts, the atmospheric light value of each part is obtained and the average value is calculated. The advantage of this is that the atmospheric light approximation value of the degraded image can be obtained more closely and the calculation difficulty of the algorithm can be reduced. The mathematical expression of the atmospheric light value is:

[0056] ;

[0057] in, Indicates the atmospheric light value; , , , Respectively represent the atmospheric light value of each of the four equal parts of the degraded image. For details, see Figure 2 ; Represents spatial coordinates.

[0058] In a specific implementation of the embodiment of the present invention, the synthesized degraded image is obtained by using the following calculation formula:

[0059] ;

[0060] in, To synthesize the degraded image, For a clear picture, is the transmission diagram, is the atmospheric light value.

[0061] In the degraded images obtained in the real foggy environment, due to the different factors such as the fog density at different positions, the outline and color of the objects in the image, and the distance from the shooting device, it is difficult to extract them well using the traditional transmission map extraction method. Therefore, the present invention proposes a transmission map extraction strategy based on a multi-adaptive framework. In a specific implementation of the embodiment of the present invention, as Figure 3 As shown, the clear image extraction network includes a backbone network, a pixel attention branch, a channel attention branch and a fusion layer;

[0062] The backbone network, the pixel attention branch, and the channel attention branch are arranged in parallel, and the inputs of the three are all synthetic degraded images;

[0063] The fusion layer fuses the feature maps output by the backbone network, the pixel attention branch, and the channel attention branch.

[0064] The backbone network is used to capture and refine the key weights of the input features, including a 3×3 convolution kernel, a ReLU activation function, and a 3×3 convolution kernel set in sequence;

[0065] The pixel attention branch includes an average pooling layer, a 3×3 convolution kernel, a first residual connection layer, a 3×3 convolution kernel and a Sigmoid activation function arranged in sequence; two inputs of the first residual connection layer come from the pixel attention branch and the channel attention branch respectively;

[0066] The channel attention branch includes a 1×1 convolution kernel, a maximum pooling layer, a 1×1 convolution kernel, a second residual connection layer, a 1×1 convolution kernel and a Sigmoid activation function arranged in sequence; two inputs of the second residual connection layer come from the pixel attention branch and the channel attention branch respectively;

[0067] The fusion layer uses element-wise multiplication to fuse the feature maps output by the backbone network, the pixel attention branch, and the channel attention branch.

[0068] Based on the above network structure, it can be seen that the clear image extraction network combines the backbone network and two attention branches focusing on different dimensions. The backbone network is the core, responsible for capturing and refining the key weights of the input features. The two branches focus on the pixel and channel levels respectively, and through the innovative cross-structure design, the weight information is complemented and strengthened. Specifically, the pixel attention branch first uses the average pooling (Avgpool) operation to reduce the pixel scale of the feature map and smooth its texture details, and then uses the 3×3 convolution kernel to deeply explore the information between channels. At the same time, the channel attention branch first simplifies the input features to a single channel with a 1×1 convolution, then uses the maximum pooling (Maxpool) technology to highlight the texture features, and then further refines the features through fine 1×1 convolution. In order to fully integrate these two information, the residual connection mechanism is introduced to integrate the output information (i.e., features) of the two branches, and then feed it back to each branch for targeted feature extraction. This process promotes the deep communication and complementarity of information. Finally, the integrated information is nonlinearly mapped through the Sigmoid activation function, and the enhanced target weights are seamlessly integrated into the backbone network using element-wise multiplication operations, achieving automatic enhancement and precise focusing of the target information, thereby significantly improving the overall performance and effect of the model.

[0069] In a specific implementation of the embodiment of the present invention, Figure 4 As shown, the transmission map extraction network includes a plurality of Conv-ReLU convolution units, 1×1 convolution kernels and Leaky-ReLU activation functions arranged in sequence;

[0070] The output of the previous Conv-ReLU convolution unit is connected to the input and output of the next Conv-ReLU convolution unit.

[0071] The Conv-ReLU convolution unit includes a 1×1 convolution kernel and a -ReLU activation function connected in sequence.

[0072] Based on the above network structure, it can be seen that in the transmission map extraction network in the embodiment of the present invention, the first four Conv-ReLU convolution units are connected using dense connections, which deepens the network depth while transmitting the output of each convolution block to the input of the subsequent convolution unit, thereby improving transmission efficiency and avoiding information loss. Finally, a 1×1 convolution kernel is used for dimensionality reduction to reduce network parameters, and a Leaky-ReLU activation function is used for output to prevent the loss of details caused by the necrosis of the output negative half-area neurons, while improving the activity of the transmission, which is helpful for the weight transmission of the deep convolutional neural network.

[0073] In a specific implementation of an embodiment of the present invention, the first error is calculated using a PSNR loss function; the second error is calculated using an SSIM loss function, which is used to further improve the algorithm's accuracy in extracting each parameter, thereby improving network stability.

[0074] Example 2

[0075] In an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the program is executed by a processor, the distribution network energy storage configuration method for photovoltaic access described in any one of Embodiment 1 is implemented.

[0076] Example 3

[0077] A computer program product is provided in an embodiment of the present invention, including a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the distribution network energy storage configuration method for photovoltaic access described in any one of Embodiment 1 is implemented.

[0078] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0079] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0080] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0081] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0082] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

[0083] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. An image defogging method for extremely complex environments, characterized in that: include: The acquired real-time degraded image is input into a pre-trained clear image extraction network to generate a clear image after defogging; wherein the training method of the clear image extraction network includes: The acquired historical degraded image is input into the atmospheric light extraction module to obtain the atmospheric light value; Input the acquired historical degraded image into a clear image extraction network to obtain a clear image; Inputting the acquired historical degradation image into a transmission map extraction network to obtain a transmission map; The obtained clear image, projection image and atmospheric light value are input into the atmospheric scattering model to obtain a synthetic degraded image; Based on a first error between a synthetic degraded image and a historical degraded image and a second error between a clear image and a standard clear image, parameters of a clear image extraction network and a transmission image extraction network are adjusted to complete the training of the clear image extraction network and the transmission image extraction network.

2. The image defogging method for extremely complex environments according to claim 1, characterized in that: The atmospheric light extraction module adopts the quartering atmospheric light extraction strategy, divides the original image into four equal parts, obtains the atmospheric light value of each part and calculates the average value; the mathematical expression of the atmospheric light value is: ； in, Indicates the atmospheric light value; , , , They represent the atmospheric light value of each of the four equal parts of the degraded image; Represents spatial coordinates.

3. The image defogging method for extremely complex environments according to claim 2, characterized in that: The synthetic degraded image is obtained by using the following calculation formula: ； in, To synthesize the degraded image, For a clear picture, is the transmission diagram, is the atmospheric light value, Represents spatial coordinates.

4. The image defogging method for extremely complex environments according to claim 1, characterized in that: The clear image extraction network includes a backbone network, a pixel attention branch, a channel attention branch and a fusion layer; The backbone network, the pixel attention branch, and the channel attention branch are arranged in parallel, and the inputs of the three are all synthetic degraded images; The fusion layer fuses the feature maps output by the backbone network, the pixel attention branch, and the channel attention branch.

5. The image defogging method for extremely complex environments according to claim 4, characterized in that: The backbone network is used to capture and refine the key weights of input features, including a 3×3 convolution kernel, a ReLU activation function, and a 3×3 convolution kernel set in sequence; The pixel attention branch includes an average pooling layer, a 3×3 convolution kernel, a first residual connection layer, a 3×3 convolution kernel and a Sigmoid activation function arranged in sequence; two inputs of the first residual connection layer come from the pixel attention branch and the channel attention branch respectively; The channel attention branch includes a 1×1 convolution kernel, a maximum pooling layer, a 1×1 convolution kernel, a second residual connection layer, a 1×1 convolution kernel and a Sigmoid activation function arranged in sequence; two inputs of the second residual connection layer come from the pixel attention branch and the channel attention branch respectively; The fusion layer uses element-wise multiplication to fuse the feature maps output by the backbone network, the pixel attention branch, and the channel attention branch.

6. The image defogging method for extremely complex environments according to claim 1, characterized in that: The transmission map extraction network includes a plurality of Conv-ReLU convolution units, a 1×1 convolution kernel and a Leaky-ReLU activation function arranged in sequence; The output of the previous Conv-ReLU convolution unit is connected to the input and output of the next Conv-ReLU convolution unit.

7. The image defogging method for extremely complex environments according to claim 6, characterized in that: The Conv-ReLU convolution unit includes a 1×1 convolution kernel and a -ReLU activation function connected sequentially.

8. The image defogging method for extremely complex environments according to claim 1, characterized in that: The first error is calculated using the PSNR loss function; the second error is calculated using the SSIM loss function.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the distribution network energy storage configuration method for photovoltaic access as described in any one of claims 1 to 8 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the distribution network energy storage configuration method for photovoltaic access described in any one of claims 1 to 8 is implemented.

Images