Image defogging method, device, equipment, medium and program product

Abstract

The invention discloses an image defogging method, device and equipment, a medium and a program product, and the method comprises the steps: inputting a to-be-processed haze image obtained through preprocessing into an image defogging model for defogging, calling feature extraction modules in a Unet encoder and a Unet decoder to carry out the feature extraction of the to-be-processed haze image layer by layer, and carrying out the defogging of the to-be-processed haze image, local haze features, image texture features and global haze features in the haze image are extracted through a first branch structure and a second branch structure in the feature extraction module, the atmospheric scattering module is called to perform image restoration processing on the intermediate image, and a target defogged image is output; according to the method, the global features, the local features and the texture features of the haze image are extracted through the double-branch structure, so that image haze can be quickly and accurately removed while image color and texture details are kept, key information in an original image is kept, the influence of haze in the image is removed, and the image quality is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image defogging, and in particular to an image defogging method, device, equipment, medium and program product. BACKGROUND

[0002] Fog is a common natural phenomenon, which often occurs in rainy weather. On the one hand, fog will cause inconvenience to people's travel, life and production activities to a certain extent. On the other hand, with the rapid development of artificial intelligence, especially in the field of computer vision, the quality requirement for pictures is getting higher and higher. However, fog will reduce the clarity of the pictures taken, make the visual effect worse, and reduce the imaging quality, which will cause a certain degree of precision decline in intelligent transportation, remote sensing images, automatic driving and target detection tasks.

[0003] The current image defogging means only processes the pixel equal distribution, image saturation, image contrast and the like of the image, so as to achieve the visual defogging effect, which has many problems such as unsatisfactory defogging effect, image color distortion, and reduced resolution of the image after defogging. SUMMARY

[0004] The main purpose of the present application is to provide an image defogging method, device, equipment, medium and program product, which aims to solve the technical problem that the prior art cannot effectively capture global and local fog information, and cannot effectively retain the texture features in the original image, resulting in low image defogging efficiency, incomplete defogging and loss of original image information.

[0005] To achieve the above-mentioned purpose, the present application provides an image defogging method, which comprises the following steps: Pretreating an original fog image to obtain a to-be-processed fog image; inputting the to-be-processed fog image into an image defogging model for defogging processing, wherein the image defogging model is constructed based on an Unet network, the image defogging model comprises a Unet encoder, a Unet decoder and an atmospheric scattering module, the Unet encoder and the Unet decoder are both composed of a plurality of feature extraction modules, the feature extraction module is composed of a first branch structure, a second branch structure and a feature fusion structure, the first branch structure is used for extracting local fog features and image texture features of an input image of the feature extraction module, the second branch structure is used for extracting global fog features of the input image of the feature extraction module, and the feature fusion structure is used for feature fusion of the local fog features, image texture features and global fog features; calling each feature extraction module in the Unet encoder to perform feature extraction and down-sampling processing on the to-be-processed fog image layer by layer, and outputting a multi-scale feature image; The feature extraction modules in the Unet decoder are called to perform upsampling processing and feature extraction on the multi-scale feature image layer by layer, and output an intermediate image. The atmospheric scattering module is called to perform image restoration processing on the intermediate image to obtain a target defogging image, and the output layer of the image defogging model is called to output the target defogging image.

[0006] Optionally, the first branch structure is constructed based on a CNN network, and the first branch structure comprises: wherein, represents an output feature of the first branch structure, represents an input image of the first branch structure, represents an activation function, represents a batch normalization operation, represents a two-dimensional convolution operation; The second branch structure is constructed based on a Mamba network, and the second branch structure comprises: wherein, represents an output feature of the second branch structure, represents an input image of the second branch structure, represents a feature extraction module; The feature extraction module comprises: wherein, represents layer normalization, represents a linear layer, represents a deep convolution operation, represents an activation function, represents a state space two-dimensional module, represents an element-wise multiplication operation; The feature fusion structure comprises: wherein, represents a fusion feature, represents an input image of a channel.

[0007] Optionally, the atmospheric scattering module is further configured to acquire a defogging parameter based on the intermediate image, and restore the intermediate image to a target defogging image based on the defogging parameter: in, represents the target dehazed image, represents the intermediate image, represents the atmospheric projection matrix, represents the atmospheric light value, represents the correction coefficient, Represents the haze image to be processed.

[0008] Optionally, each feature extraction module in the Unet encoder is connected to a corresponding feature extraction module in the Unet decoder via a skip connection layer; calling each feature extraction module in the Unet decoder to perform upsampling and feature extraction on the multi-scale feature image layer by layer and output an intermediate image includes: Calling each feature extraction module in the Unet decoder to perform upsampling processing on the multi-scale feature image; Calling each feature extraction module in the Unet decoder to fuse the upsampling processing result with the encoder features collected by the skip connection layer to obtain a fused feature; Each feature extraction module in the Unet decoder is called to extract features from the fused features and output an intermediate image.

[0009] Optionally, before inputting the haze image to be processed into an image defogging model for defogging, the method further includes: Build the original model based on the Unet network and determine the first initial learning rate; Freeze the encoder weights of the original model, and train the original model based on the first initial learning rate, the Adam optimizer, and the cosine annealing learning rate adjustment strategy to obtain a first candidate model. The cosine annealing learning rate adjustment strategy refers to the following formula: in, Indicates the The learning rate updated in the iteration, represents the maximum learning rate, represents the minimum learning rate, Indicates the number of cycle steps currently completed. Indicates the total number of steps in a single learning rate cycle; Unfreeze all network parameters of the first candidate model and determine a second initial learning rate, where the encoder learning rate and the decoder learning rate are different; Training the unfrozen first candidate model according to the second initial learning rate, Adam optimizer, and cosine annealing learning rate adjustment strategy to obtain a second candidate model; Determining a third initial learning rate and a decay weight parameter, wherein the encoder learning rate and the decoder learning rate in the third initial learning rate are the same; The second candidate model is trained according to the third initial learning rate, Adamw optimizer and cosine annealing learning rate adjustment strategy to obtain an image defogging model.

[0010] Optionally, preprocessing the original haze image to obtain the haze image to be processed includes: Divide the original haze image into multiple image blocks and calculate the pixel value histogram of each image block; identifying abnormal blocks with abnormal pixel distribution in the image block based on the pixel value histogram, and performing cropping processing on the original haze image based on the abnormal blocks to obtain a first candidate image; performing a distortion analysis on the first candidate image, and performing distortion correction on the first candidate image based on a result of the distortion analysis to obtain a second candidate haze image; Perform sliding average processing on the second candidate haze image based on a multi-scale sliding window to obtain a haze image to be processed: in, Indicates the The sliding average output value of pixels, represents the sliding window size, Indicates the first candidate haze image The image value of each pixel, Indicates the traversal index within the window, Indicates the currently calculated target position index.

[0011] In addition, to achieve the above-mentioned object, the present invention further proposes an image defogging device, which includes: An image preprocessing module is used to preprocess the original haze image to obtain a haze image to be processed; The image defogging module is configured to input the to-be-processed haze image into an image defogging model for defogging processing, the image defogging model is constructed based on an Unet network, the image defogging model comprises a Unet encoder, a Unet decoder and an atmospheric scattering module, the Unet encoder and the Unet decoder are both composed of a plurality of feature extraction modules, the feature extraction module is composed of a first branch structure, a second branch structure and a feature fusion structure, the first branch structure is configured to extract local haze features and image texture features of an input image of the feature extraction module, the second branch structure is configured to extract global haze features of the input image of the feature extraction module, and the feature fusion structure is configured to perform feature fusion on the local haze features, the image texture features and the global haze features; The image encoding module is configured to call each feature extraction module in the Unet encoder to perform feature extraction and down-sampling processing on the to-be-processed haze image layer by layer, and output a multi-scale feature image. The image decoding module is configured to call each feature extraction module in the Unet decoder to perform up-sampling processing and feature extraction on the multi-scale feature image layer by layer, and output an intermediate image. The image recovery module is configured to call the atmospheric scattering module to perform image recovery processing on the intermediate image, obtain a target defogging image, and call an output layer of the image defogging model to output the target defogging image.

[0012] In addition, to achieve the above object, the present application further provides an image defogging device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the image defogging method as described above.

[0013] In addition, to achieve the above object, the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the image defogging method as described above.

[0014] In addition, to achieve the above object, the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the image defogging method as described above.

[0015] The application obtains a to-be-processed haze image by preprocessing an original haze image; inputs the to-be-processed haze image into an image haze removal model for haze removal processing, calls each feature extraction module in the Unet encoder to perform layer-by-layer feature extraction and down-sampling processing on the to-be-processed haze image, and outputs a multi-scale feature image; calls each feature extraction module in the Unet decoder to perform layer-by-layer up-sampling processing and feature extraction on the multi-scale feature image, and outputs an intermediate image, calls the atmospheric scattering module to perform image restoration processing on the intermediate image, obtains a target haze-removed image, and calls the output layer of the image haze removal model to output the target haze-removed image, wherein the image haze removal model is constructed based on a Unet network, the image haze removal model comprises a Unet encoder, a Unet decoder and an atmospheric scattering module, the Unet encoder and the Unet decoder each comprise a plurality of feature extraction modules, each feature extraction module comprises a first branch structure, a second branch structure and a feature fusion structure, the first branch structure is used for extracting local haze features and image texture features of an input image of the feature extraction module, the second branch structure is used for extracting global haze features of the input image of the feature extraction module, and the feature fusion structure is used for fusing the local haze features, the image texture features and the global haze features. Since the first branch structure and the second branch structure in the image haze removal model are used to extract the local haze features, the image texture features and the global haze features in the haze image respectively, the image haze can be quickly and accurately removed while the image color and texture details are maintained, the key information in the original image is retained, the haze influence in the image is removed, and the image quality is improved. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0017] Figure 1 FIG. 1 is a structural schematic diagram of an image haze removal device of a hardware running environment related to an embodiment of the present application; Figure 2 FIG. 2 is a flow schematic diagram of an image haze removal method according to a first embodiment of the present application; Figure 3 FIG. 3 is a structural schematic diagram of an image haze removal module in an embodiment of the image haze removal method of the present application; Figure 4 FIG. 4 is a structural schematic diagram of a feature extraction module in an embodiment of the image haze removal method of the present application; Figure 5Fig. 1 is a structural schematic diagram of a VSS feature extraction module in an embodiment of the image defogging method of the present application; Figure 6 Fig. 2 is a flowchart of a second embodiment of the image defogging method of the present application; Fig. 7(a) is a PSNR curve diagram of the first stage of model training in an embodiment; Fig. 7(b) is a loss curve diagram of the first stage of model training in an embodiment; Fig. 8(a) is a PSNR curve diagram of the second stage of model training in an embodiment; Fig. 8(b) is a loss curve diagram of the second stage of model training in an embodiment; Fig. 9(a) is a PSNR curve diagram of the third stage of model training in an embodiment; Fig. 9(b) is a loss curve diagram of the third stage of model training in an embodiment; Figure 10 Fig. 1 is a structural schematic diagram of a VSS feature extraction module in an embodiment of the image defogging method of the present application;

[0018] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0019] It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

[0020] Reference Figure 1 , Figure 1 Fig. 1 is a structural schematic diagram of a VSS feature extraction module in an embodiment of the image defogging method of the present application;

[0021] As Figure 1As shown in the figure, the image defogging device can include a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize the connection communication between the components. The user interface 1003 can include a display screen, an input unit such as a keyboard, and can also include a standard wired interface, a wireless interface. The network interface 1004 can optionally include a standard wired interface, a wireless interface (such as a wireless fidelity (WI-FI) interface). The memory 1005 can be a high-speed random access memory (RAM), and can also be a stable non-volatile memory (NVM), such as a disk memory. The memory 1005 can also be a storage device independent of the aforementioned processor 1001.

[0022] Those skilled in the art can understand that Figure 1 The structure shown in the figure does not constitute a limitation on the image defogging device, and can include more or fewer components than the figure, or combine certain components, or different component arrangements.

[0023] As Figure 1 As shown, the memory 1005 as a computer readable storage medium can include an operating system, a network communication module, a user interface module, and an image defogging program.

[0024] In Figure 1 In the image defogging device shown in the figure, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the image defogging device of the present application can be arranged in the image defogging device, and the image defogging device calls the image defogging program stored in the memory 1005 through the processor 1001, and executes the image defogging method provided by the embodiment of the present application.

[0025] The embodiment of the present application provides an image defogging method, which refers to Figure 2 , Figure 2 The flowchart of the first embodiment of the image defogging method of the present application.

[0026] In this embodiment, the image defogging method includes the following steps: Step S10: Preprocessing the original haze image to obtain a to-be-processed haze image.

[0027] It should be understood that the execution subject of the embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a terminal electronic device capable of realizing the above functions, etc. The embodiment and the following embodiments are described below taking an image defogging device as an example.

[0028] In some embodiments, the image defogging device can perform random cropping, horizontal flipping, filtering, etc. on the original haze image, so as to improve the quality of the haze image, avoid the interference of invalid pixels and distortion regions, and improve the image defogging efficiency.

[0029] Further, in order to improve the image defogging efficiency and reduce the interference of distortion regions and invalid pixels, the above step S10 can include: Step S101: dividing the original haze image into a plurality of image blocks and calculating a pixel value histogram of each image block.

[0030] It can be understood that the image defogging module can divide the original haze image into a non-overlapping image block set (the set includes a plurality of image blocks) based on the image size, and perform abnormality identification in units of image blocks, so as to reduce the global calculation complexity and focus on the local pixel distribution characteristics.

[0031] Step S102: identifying an abnormal block with abnormal pixel distribution in the image block based on the pixel value histogram, and performing cropping processing on the original haze image based on the abnormal block to obtain a first candidate image.

[0032] It should be understood that the image defogging module can calculate the pixel value histogram of each image block, respectively, so as to obtain the pixel distribution characteristics of each image block, and then identify the abnormal image block through threshold judgment or distance judgment, for example, setting a normal range threshold of histogram statistical characteristics, if the characteristics of a certain block exceed the threshold range, it is marked as an abnormal block; or the distance abnormality can be determined by chi-square distance or KL divergence.

[0033] It should be noted that the image defogging device focuses on the haze dense area and reduces the background interference by identifying the abnormal image block and performing cropping processing on the abnormal block.

[0034] Step S103: performing distortion analysis on the first candidate image, and performing distortion correction on the first candidate image based on the distortion analysis result to obtain a second candidate haze image.

[0035] In some embodiments, the image dehazing device may identify distortion parameters through checkerboard calibration or an edge line-based distortion estimation algorithm (such as Hough transform) to obtain a distortion analysis result, and perform distortion correction on the first candidate image based on the distortion analysis result to obtain a second candidate haze image.

[0036] Step S104: performing sliding average processing on the second candidate haze image based on a multi-scale sliding window to obtain a haze image to be processed: in, Indicates the The sliding average output value of pixels, represents the sliding window size, Indicates the first candidate haze image The image value of each pixel, Indicates the traversal index within the window, Indicates the currently calculated target position index.

[0037] Step S20: inputting the haze image to be processed into an image defogging model for defogging processing.

[0038] It should be noted that the image dehazing model is constructed based on the Unet network. The image dehazing model includes a Unet encoder, a Unet decoder and an atmospheric scattering module. The Unet encoder and the Unet decoder are both composed of a multi-layer feature extraction module. The feature extraction module is composed of a first branch structure, a second branch structure and a feature fusion structure. The first branch structure is used to extract the local haze features and image texture features of the input image of the feature extraction module. The second branch structure is used to extract the global haze features of the input image of the feature extraction module. The feature fusion structure is used to fuse the local haze features, image texture features and global haze features.

[0039] Step S30: calling each feature extraction module in the Unet encoder to perform feature extraction and downsampling processing on the haze image to be processed layer by layer, and outputting a multi-scale feature image.

[0040] In some embodiments, the encoder of the image dehazing model is the downsampling portion of CVMNet, employing a CVM-Net Block (i.e., a feature extraction module) at each layer to extract haze features. After the haze image to be processed is input into the image dehazing model, the encoder performs feature extraction and downsampling on the image four times before inputting it into the decoder. The feature extraction module of the decoder of the image dehazing model is consistent with that of the encoder, with the multi-scale feature maps output by the encoder serving as input to the decoder.

[0041] In some embodiments, each layer of the encoder structure part has a corresponding feature extraction module for feature extraction, and after 4 times of downsampling, it enters the decoder part. After each downsampling, the feature map size is reduced by half, and the channel number is doubled.

[0042] In some embodiments, the image defogging device designs two networks of different sizes for the image defogging model, namely CVMNet-Tiny and CVMNet-Base. The main difference between CVMNet-Tiny and CVMNet-Base is the number of channels and the number of CVM blocks in each layer. The maximum channel number of CVMNet-Tiny is 768, while the maximum channel number of CVMNet-Base is 1024. CVMNet-Tiny has a deeper network structure and higher parameter quantity and computational complexity, and thus has stronger feature extraction capability. Referring to Table 1 below, Table 1 is a network detailed structure parameter table of CVMNet-Tiny and CVMNet-Base. CVM-Net Block represents a feature extraction module.

[0043] Table 1, network structure parameter table Step S40: calling each feature extraction module in the Unet decoder to perform upsampling processing and feature extraction on the multi-scale feature image layer by layer, and outputting an intermediate image.

[0044] In some embodiments, the decoder part is just the opposite of the encoder part. The decoder can include 4 times of upsampling. After each upsampling, the channel number is reduced by half, and the feature map size becomes twice the original size. Until the size of the input picture is restored, in the corresponding same layer of the encoder and the decoder, a long skip connection (Skip Connection) is used to achieve the effect of residual connection. This step can achieve the effect of feature fusion, and also can prevent the problem of gradient explosion and model degradation.

[0045] Further, in order to effectively fuse multi-scale information and combine shallow features and deep features, so as to extract local and global haze features in the haze image, in an embodiment, each feature extraction module in the Unet encoder is connected with the corresponding each feature extraction module in the Unet decoder through a skip connection layer; the above step S40 can include: Step S401: calling each feature extraction module in the Unet decoder to perform upsampling processing on the multi-scale feature image; Step S402: calling each feature extraction module in the Unet decoder to fuse the upsampling processing result with the encoder features collected by the skip connection layer to obtain fused features; Step S403: calling each feature extraction module in the Unet decoder to perform feature extraction on the fused features, and outputting an intermediate image.

[0046] It should be noted that, with reference to Figure 3 , Figure 3 Figure 1 is a structural schematic diagram of an image defogging module in an embodiment. The feature extraction modules of the encoder and the corresponding layers of the decoder are connected through the skip connection layer. The feature extraction modules in the decoder obtain the encoded features through the skip connection layer, and then the feature extraction modules of the decoder fuse the up-sampling results with the encoded features, and then extract through the first branch structure and the second branch structure, effectively fusing multi-scale information, combining shallow features and deep features, so as to extract local and global haze features in the haze image.

[0047] It should be noted that the Unet encoder extracts the features of the image layer by layer. The shallow features contain the detailed information of the image, such as edges and textures; the deep features pay more attention to the semantic information of the image, such as the category and overall structure of the object. The skip connection layer directly transmits the features of the corresponding layers in the encoder to the decoder, so that the decoder can fuse the detailed information of the shallow layer and the semantic information of the deep layer when performing up-sampling and feature extraction. For example, in the image defogging task, the edge information of the shallow layer can help the model more accurately restore the boundary of the object, and the semantic information of the deep layer can help the model understand the overall structure of the object in the image, so as to more effectively remove the haze.

[0048] It should be noted that the decoder fuses the encoder features through the skip connection layer, so that the model can process and fuse features at different scales, so as to better handle objects and details of different sizes in the image. For the scenes of different sizes in the haze image, the fusion of multi-scale information helps to improve the consistency and accuracy of the defogging effect. Moreover, the skip connection layer reduces the analysis complexity of the image defogging model, so that the image defogging model can reuse the learned features to a certain extent without relearning from the bottom. This makes the image defogging model converge to a better state faster, reducing the time and computing resources required for training.

[0049] It should be noted that in the image defogging task, the skip connection layer helps the model more accurately restore the original details and colors of the image. Since it can directly transmit part of the original image information retained in the encoder to the decoder, the decoder can use this information to better reconstruct the defogged image, making the defogged image clearer, more natural, and closer to the real haze-free image.

[0050] Step S50: calling the atmospheric scattering module to perform image restoration processing on the intermediate image, obtaining a target defogging image, and calling the output layer of the image defogging model to output the target defogging image.

[0051] In some embodiments, the decoder finally outputs the intermediate image enhancement factor K(x) generated by the convolution layer with a kernel size of 1x1, applies the atmospheric scattering module to perform restoration processing on the intermediate image enhancement factor K(x), restores a clear image, and calls the output layer of the image defogging model to output the target defogging image.

[0052] As shown in Figure 3 , Fig. 1 is a structural schematic diagram of the image defogging module in an embodiment, Figure 3 Fig. 2 is a structural schematic diagram of the feature extraction module in an embodiment, Figure 3 The to-be-processed haze image is input into the encoder, and after 4 times of feature extraction and down-sampling by the feature extraction modules in each layer of the encoder, the multi-scale feature images are input into the decoder. The feature extraction modules in each layer of the decoder perform up-sampling on the multi-scale feature images, fuse the features of the feature extraction modules in the encoder obtained by the skip connection layer, then perform feature extraction, output an intermediate image to the atmospheric scattering module for image restoration, and output a defogging image.

[0053] Further, in order to accurately capture the global and local haze features in the haze image and effectively preserve the texture features and color details in the original image, in an embodiment, the first branch structure is constructed based on a CNN network, and the second branch structure is constructed based on a Mamba network.

[0054] It should be noted that the feature extraction module (CVM-Net Block) can include a channel split structure (aiming to reduce the parameter amount), a CNN branch structure and a Mamba branch structure. The convolutional neural network CNN branch mainly extracts the local haze features and object texture features of the picture, so as to ensure that the picture after defogging can preserve rich details. The Mamba branch is mainly responsible for capturing global haze information, so that the model can master the dependence between global pixels in a single picture, and defog more effectively. Referring to Figure 4 , Figure 4 Fig. 3 is a structural schematic diagram of the feature extraction module in an embodiment, in which Split is a channel split structure.

[0055] Suppose the input picture of the convolutional neural network CNN branch is , and the output picture is , then the first branch structure constructed based on the convolutional neural network CNN can be expressed by the following formula: , wherein output feature of the first branch structure, input image of the first branch structure, output feature of the first branch structure, activation function, Batch Normalization, represents a batch normalization operation, represents a two-dimensional convolution operation. For the second branch structure constructed based on the Mamba network, assuming that the input is , the output is , the second branch structure can be represented by the following formula: wherein, output feature of the second branch structure, input image of the second branch structure, output feature of the second branch structure, feature extraction module, refer to Figure 5 , Figure 5 is a structural diagram of the feature extraction module, the second branch input data is subjected to layer normalization, which stabilizes the input distribution and is helpful for subsequent calculation. After the data is subjected to layer normalization, it is respectively input to the main branch and the auxiliary branch. The main branch includes a linear layer, a deep convolution layer, an activation function, and layer normalization; the auxiliary branch includes a linear layer and an activation function.

[0056] Main branch: the data after LN (layer normalization) goes through the main branch, first passes through the Linear (linear transformation) layer to adjust the channel dimension, etc.; then passes through the DWConv (depthwise separable convolution) to extract features in the spatial dimension and reduce the amount of calculation; then passes through the SiLU (Sigmoid-weighted Linear Units, activation function) to introduce nonlinearity and enhance the model expression; then passes through the SS2D (state space two-dimensional module for spatial feature optimization); and then passes through the LN layer again for normalization.

[0057] Auxiliary branch: after the initial LN, it is branched off to pass through the Linear layer for linear transformation and then the SiLU activation to generate a path of features.

[0058] The outputs of the main branch and the auxiliary branch in the second branch structure are combined by element-wise multiplication, etc., to complement the information of different paths. The fused data passes through the last Linear layer for further feature transformation, and the output feature of the second branch structure is output.

[0059] The feature extraction module comprises: wherein, Layer Normalization denotes layer normalization, denotes a linear layer, denotes a deep convolution operation, denotes an activation function, denotes a state space two-dimensional module, denotes an element-wise multiplication operation.

[0060] In a specific implementation, the image defogging device calls the feature extraction module to perform shunting according to the number of channels C, and each C / 2 number of channels is subjected to feature extraction through the CNN branch and the Mamba branch respectively, and then subjected to feature fusion through the feature fusion structure, and finally spliced through the residual structure and the initial input. For example, the input of the feature extraction module is , , which is divided into two equal parts according to the channels of the input , so that the output of the entire feature extraction module is , the feature fusion structure comprises: wherein, denotes fused features, denotes an input image of the channel.

[0061] In some embodiments, the feature extraction module is mainly composed of three parts: a shunting structure (i.e. Split in Figure 4 ), a CNN branch and a Mamba branch. Therefore, a plurality of sets of ablation experiments are designed for the three modules to verify the rationality of the design: Experiment one is a feature extraction module with only the CNN branch without the shunting structure; Experiment two is a feature extraction module with only the Mamba branch without the shunting structure; Experiment three is a feature extraction module with the CNN branch combined with the Mamba branch without the shunting structure.

[0062] Experiment four is a feature extraction module combined with the shunting structure, the CNN branch and the Mamba branch in the present embodiment.

[0063] The ablation experiments and the formal training experiments maintain the same experimental environment and hyperparameters, so as to reduce the error influence caused by the environment, etc. Referring to Table 2, Table 2 gives the results of the ablation experiments: Table 2, comparison results of each experimental group in the ablation experiment As can be seen from Table 2, the feature extraction module (Experiment Four) in the image defogging model proposed in the embodiment achieves the optimal trade-off between performance and model complexity. Specifically, Experiment Four achieves 31.81 dB and 0.973 in the PSNR and SSIM indicators, respectively, which is the best performing model among all experimental groups; in terms of model complexity, the parameter amount of Experiment Four is only 24.56M, which is the least among the four experiments; at the same time, its MACs is only 4.56G, which is also at the lowest level, only slightly higher than Experiment Two (4.40G), and significantly lower than Experiment One and Experiment Three.

[0064] Experiment Four achieves the best balance between lightweight design and performance, verifying the effectiveness and practicality of the proposed CVMNet Block (double-branch feature extraction module containing a shunt structure) in the image restoration task.

[0065] Further, in order to accurately restore a clear image and improve the clarity of the output defogging image and the rationality and accuracy of the defogging effect, in an embodiment, the atmospheric scattering module is further configured to obtain a defogging parameter based on the intermediate image, and restore the intermediate image to a target defogging image based on the defogging parameter.

[0066] It should be noted that the channel number of the intermediate image finally output by the decoder is 96 or 24, which is adjusted by a convolution layer with a convolution kernel size of 1x1 to generate a bottom matrix, and then the intermediate image is restored according to the following formula to output a clear target defogging image: wherein, represents the target defogging image, represents the intermediate image, represents the atmospheric projection matrix, represents the atmospheric light value, represents a correction coefficient (for example, the correction coefficient can be 1), represents the to-be-processed haze image.

[0067] This embodiment obtains a haze image to be processed by preprocessing the original haze image; inputs the haze image to be processed into the image defogging model for defogging, calls each feature extraction module in the Unet encoder to perform feature extraction and downsampling processing on the haze image to be processed layer by layer, and outputs a multi-scale feature image; calls each feature extraction module in the Unet decoder to perform upsampling processing and feature extraction on the multi-scale feature image layer by layer, outputs an intermediate image, calls the atmospheric scattering module to perform image restoration processing on the intermediate image to obtain a target defogging image, and calls the output layer of the image defogging model to output the target defogging image, wherein the image defogging model is constructed based on the Unet network, and the image defogging model includes a Unet encoder, a Unet decoder and an atmospheric scattering module, and the Unet encoder and the Un et decoders are composed of a multi-layer feature extraction module, which is composed of a first branch structure, a second branch structure and a feature fusion structure. The first branch structure is used to extract the local haze features and image texture features of the input image of the feature extraction module, and the second branch structure is used to extract the global haze features of the input image of the feature extraction module. The feature fusion structure is used to fuse the local haze features, image texture features and global haze features. Since this embodiment uses the first branch structure and the second branch structure in the image dehazing model to respectively extract the local haze features, image texture features and global haze features in the haze image, it can quickly and accurately remove the haze from the image while maintaining the image color and texture details, thereby retaining the key information in the original image, removing the haze effect in the image, and improving the image quality.

[0068] refer to Figure 6 , Figure 6 FIG. 4 is a flow chart of the second embodiment of the image defogging method of the present invention.

[0069] Based on the first embodiment above, in this embodiment, before step S20, the following steps are further included: Step S21: constructing an original model based on the Unet network and determining a first initial learning rate; Step S22: Freeze the encoder weights of the original model, and train the original model based on the first initial learning rate, Adam optimizer, and cosine annealing learning rate adjustment strategy to obtain a first candidate model. The cosine annealing learning rate adjustment strategy refers to the following formula: in, Indicates the The learning rate updated in the iteration, represents the maximum learning rate, represents the minimum learning rate, represents the number of steps of the current completed cycle, represents the total number of steps of a single learning rate cycle; Step S23: unfreezing all network parameters of the first candidate model, and determining a second initial learning rate, the encoder learning rate and the decoder learning rate in the second initial learning rate being different; Step S24: training the unfreezed first candidate model according to the second initial learning rate, the Adam optimizer and the cosine annealing learning rate adjustment strategy, to obtain a second candidate model; Step S25: determining a third initial learning rate and a decay weight parameter, the encoder learning rate and the decoder learning rate in the third initial learning rate being the same; Step S26: training the second candidate model according to the third initial learning rate, the Adamw optimizer and the cosine annealing learning rate adjustment strategy, to obtain the image defogging model.

[0070] It should be noted that, considering the characteristics of the Mamba network, the encoder part of the image defogging model in the embodiment is weight-initialized using the parameters trained on ImageNet, and the decoder is randomly initialized. In some embodiments, in order to ensure the repeatability of the experiment, all experiments are set with a random seed number 42.

[0071] In some embodiments, the image defogging device can divide the training process of the image defogging model into three stages: In the first stage, the weights of the encoder are frozen, the weights of the decoder are not processed, the learning rate lr=2e-4, the Adam optimizer is used for training, the cosine annealing learning rate adjustment strategy is used, and the initial image defogging model is trained. The training curves are shown in FIGS. 7(a) and 7(b), FIG. 7(a) is the PSNR curve of the first stage of model training, and FIG. 7(b) is the loss curve of the first stage of model training, wherein the abscissa of FIG. 7(a) is the training step number, and the ordinate is the signal-to-noise ratio; the abscissa of FIG. 7(b) is the training step number, and the ordinate is the training loss.

[0072] In the second stage, all network parameters are unfrozen, different learning rates are used for the encoder and the decoder, that is, the learning rate of the encoder part is lr=1e-5, and the learning rate of the decoder part is lr=1e-4, so that the parameter change of the encoder is as small as possible, and the parameters of the decoder can continue to be optimized. The training curves are shown in FIGS. 8(a) and 8(b), FIG. 8(a) is the PSNR curve of the second stage of model training, and FIG. 8(b) is the loss curve of the second stage of model training, wherein the abscissa of FIG. 8(a) is the training step number, and the ordinate is the signal-to-noise ratio; the abscissa of FIG. 8(b) is the training step number, and the ordinate is the training loss.

[0073] In the third stage, the learning rate is unified and a weight decay term is introduced, and the Adamw optimizer is used, and the decay weight 2e-4 is added, which can avoid overfitting when further optimizing the parameters. The training curve is shown in FIG. 9 (a) and FIG. 9 (b), FIG. 9 (a) is the PSNR curve of the third stage of model training, and FIG. 9 (b) is the loss curve of the third stage of model training, wherein the abscissa of FIG. 9 (a) is the training step, and the ordinate is the signal-to-noise ratio; the abscissa of FIG. 9 (b) is the training step, and the ordinate is the training loss.

[0074] The embodiment is based on the Unet network to construct an original model and determine a first initial learning rate, freeze the encoder weight of the original model, train the original model based on the first initial learning rate, the Adam optimizer and the cosine annealing learning rate adjustment strategy, obtain a first candidate model, unfreeze all network parameters of the first candidate model, determine a second initial learning rate, the encoder learning rate and the decoder learning rate in the second initial learning rate are different, train the unfrozen first candidate model according to the second initial learning rate, the Adam optimizer and the cosine annealing learning rate adjustment strategy, obtain a second candidate model, determine a third initial learning rate and a decay weight parameter, the encoder learning rate and the decoder learning rate in the third initial learning rate are the same, train the second candidate model according to the third initial learning rate, the Adamw optimizer and the cosine annealing learning rate adjustment strategy, and obtain an image defogging model; since the embodiment divides the model training into three stages, the problem of model overfitting is effectively prevented, and the model training efficiency and the image defogging ability are improved.

[0075] In addition, the embodiment of the present application also provides a computer readable storage medium, and the computer readable storage medium stores an image defogging program, and the image defogging program is executed by a processor to realize the steps of the image defogging method as described above.

[0076] The computer readable storage medium provided in the application may be, for example, a U disk, but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination of the above. More specific examples of the computer readable storage medium may include, but are not limited to, an electrical connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiment, the computer readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electrical wire, an optical cable, an RF (Radio Frequency), and the like, or any suitable combination of the above.

[0077] The above computer readable storage medium may be contained in the image defogging device, or may exist separately without being assembled into the image defogging device.

[0078] In addition, the embodiment of the application further provides a computer program product comprising an image defogging program, which, when executed by a processor, implements the steps of the image defogging method as described above.

[0079] The computer program product embodiment of the application is basically the same as the above-mentioned image defogging method embodiments, and will not be repeated here.

[0080] Reference Figure 10 , Figure 10 is a structural block diagram of the first embodiment of the image defogging device of the application.

[0081] As Figure 10 shown, the image defogging device provided by the embodiment of the application comprises: An image preprocessing module 10 is configured to pre-process an original haze image to obtain a haze image to be processed. The image defogging module 20 is configured to input the to-be-processed haze image into an image defogging model for defogging processing. The image defogging model is constructed based on a Unet network. The image defogging model comprises a Unet encoder, a Unet decoder and an atmospheric scattering module. The Unet encoder and the Unet decoder each comprise a plurality of feature extraction modules. Each feature extraction module comprises a first branch structure, a second branch structure and a feature fusion structure. The first branch structure is configured to extract local haze features and image texture features of an input image of the feature extraction module. The second branch structure is configured to extract global haze features of the input image of the feature extraction module. The feature fusion structure is configured to fuse the local haze features, the image texture features and the global haze features. The image encoding module 30 is configured to call each feature extraction module in the Unet encoder to perform feature extraction and down-sampling processing on the to-be-processed haze image layer by layer, and output a multi-scale feature image. The image decoding module 40 is configured to call each feature extraction module in the Unet decoder to perform up-sampling processing and feature extraction on the multi-scale feature image layer by layer, and output an intermediate image. The image recovery module 50 is configured to call the atmospheric scattering module to perform image recovery processing on the intermediate image, obtain a target defogging image, and call an output layer of the image defogging model to output the target defogging image.

[0082] The embodiment obtains a to-be-processed haze image by preprocessing an original haze image; inputs the to-be-processed haze image into an image haze removal model for haze removal processing, calls each feature extraction module in the Unet encoder to perform feature extraction and down-sampling processing on the to-be-processed haze image layer by layer, and outputs a multi-scale feature image; calls each feature extraction module in the Unet decoder to perform up-sampling processing and feature extraction on the multi-scale feature image layer by layer, and outputs an intermediate image, calls the atmospheric scattering module to perform image restoration processing on the intermediate image, obtains a target haze-removed image, and calls an output layer of the image haze removal model to output the target haze-removed image, wherein the image haze removal model is constructed based on a Unet network, the image haze removal model comprises a Unet encoder, a Unet decoder and an atmospheric scattering module, the Unet encoder and the Unet decoder each comprise a plurality of feature extraction modules, each feature extraction module comprises a first branch structure, a second branch structure and a feature fusion structure, the first branch structure is used for extracting local haze features and image texture features of an input image of the feature extraction module, the second branch structure is used for extracting global haze features of the input image of the feature extraction module, and the feature fusion structure is used for fusing the local haze features, the image texture features and the global haze features. Since the first branch structure and the second branch structure in the image haze removal model are used to extract the local haze features, the image texture features and the global haze features in the haze image respectively, the image haze can be quickly and accurately removed while the image color and texture details are maintained, the key information in the original image can be retained, the haze influence in the image can be removed, and the image quality can be improved.

[0083] The image haze removal device provided in the present application adopts the image haze removal method in the above embodiment, and can solve the technical problem of image haze removal. Compared with the prior art, the image haze removal device provided in the present application has the same beneficial effects as the image haze removal method provided in the above embodiment, and other technical features in the image haze removal device are the same as the features disclosed in the above embodiment, which will not be repeated here.

[0084] It should be understood that the above is only an example, and does not constitute any limitation on the technical solutions of the present application. In specific applications, those skilled in the art can set it up according to the needs, and the present application does not limit it.

[0085] It should be noted that the above-described workflow is only illustrative and does not limit the scope of protection of the present application. In actual application, those skilled in the art can select part or all of them to achieve the purpose of the embodiment, which is not limited here.

[0086] In addition, technical details not described in detail in the present embodiment can be found in the image defogging method provided by any embodiment of the present application, and will not be described here.

[0087] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, method, article, or system. Without more limitations, an element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or system that includes the element.

[0088] The above-mentioned embodiment numbers of the present application are only for description, not representing the advantages and disadvantages of the embodiments.

[0089] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and necessary general hardware platforms, of course, they can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, an optical disk), and includes a number of instructions for making a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in various embodiments of the present application.

[0090] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields based on the content of the present application specification and drawings, are also included in the patent protection scope of the present application.

Claims

1. An image defogging method, characterized in that: The image defogging method comprises: Preprocessing the original haze image to obtain a haze image to be processed; Inputting the haze image to be processed into an image dehazing model for dehazing processing, wherein the image dehazing model is constructed based on a Unet network, and the image dehazing model includes a Unet encoder, a Unet decoder, and an atmospheric scattering module. The Unet encoder and the Unet decoder are both composed of a multi-layer feature extraction module, and the feature extraction module is composed of a first branch structure, a second branch structure, and a feature fusion structure. The first branch structure is used to extract local haze features and image texture features of the input image of the feature extraction module, and the second branch structure is used to extract global haze features of the input image of the feature extraction module. The feature fusion structure is used to perform feature fusion on the local haze features, image texture features, and global haze features; Calling each feature extraction module in the Unet encoder to perform feature extraction and downsampling processing on the haze image to be processed layer by layer, and outputting a multi-scale feature image; Calling each feature extraction module in the Unet decoder to perform upsampling and feature extraction on the multi-scale feature image layer by layer, and outputting an intermediate image; The atmospheric scattering module is called to perform image restoration processing on the intermediate image to obtain a target defogging image, and the output layer of the image defogging model is called to output the target defogging image.

2. The image defogging method according to claim 1, wherein: The first branch structure is constructed based on a CNN network, and the first branch structure includes: in, represents the output features of the first branch structure, represents the input image of the first branch structure, express activation function, represents the batch normalization operation, Represents a two-dimensional convolution operation; The second branch structure is constructed based on the Mamba network, and the second branch structure includes: in, represents the output features of the second branch structure, represents the input image of the second branch structure, express Feature extraction module; described The feature extraction module includes: in, Representation layer normalization, represents a linear layer, represents the depthwise convolution operation, express activation function, represents a two-dimensional module in the state space, Represents element-wise multiplication operation; The feature fusion structure includes: in, represents the fusion feature, express Channels of the input image.

3. The image defogging method according to claim 2, wherein: The atmospheric scattering module is further configured to obtain defogging parameters based on the intermediate image, and restore the intermediate image to a target defogging image based on the defogging parameters: in, represents the target dehazed image, represents the intermediate image, represents the atmospheric projection matrix, represents the atmospheric light value, represents the correction coefficient, Represents the haze image to be processed.

4. The image defogging method according to claim 3, wherein: Each feature extraction module in the Unet encoder is connected to the corresponding feature extraction modules in the Unet decoder via a skip connection layer; calling each feature extraction module in the Unet decoder to perform upsampling and feature extraction on the multi-scale feature image layer by layer, and outputting an intermediate image, including: Calling each feature extraction module in the Unet decoder to perform upsampling processing on the multi-scale feature image; Calling each feature extraction module in the Unet decoder to fuse the upsampling processing result with the encoder features collected by the skip connection layer to obtain a fused feature; Each feature extraction module in the Unet decoder is called to extract features from the fused features and output an intermediate image.

5. The image defogging method according to any one of claims 1 to 4, characterized in that: Before inputting the haze image to be processed into the image defogging model for defogging, the method further includes: Build the original model based on the Unet network and determine the first initial learning rate; Freeze the encoder weights of the original model, and train the original model based on the first initial learning rate, the Adam optimizer, and the cosine annealing learning rate adjustment strategy to obtain a first candidate model. The cosine annealing learning rate adjustment strategy refers to the following formula: in, Indicates the The learning rate updated in the iteration, represents the maximum learning rate, represents the minimum learning rate, Indicates the number of cycle steps currently completed. Indicates the total number of steps in a single learning rate cycle; Unfreeze all network parameters of the first candidate model and determine a second initial learning rate, where the encoder learning rate and the decoder learning rate are different; Training the unfrozen first candidate model according to the second initial learning rate, Adam optimizer, and cosine annealing learning rate adjustment strategy to obtain a second candidate model; Determining a third initial learning rate and a decay weight parameter, wherein the encoder learning rate and the decoder learning rate in the third initial learning rate are the same; The second candidate model is trained according to the third initial learning rate, Adamw optimizer and cosine annealing learning rate adjustment strategy to obtain an image defogging model.

6. The image defogging method according to any one of claims 1 to 4, characterized in that: The preprocessing of the original haze image to obtain the haze image to be processed includes: Divide the original haze image into multiple image blocks and calculate the pixel value histogram of each image block; identifying abnormal blocks with abnormal pixel distribution in the image block based on the pixel value histogram, and performing cropping processing on the original haze image based on the abnormal blocks to obtain a first candidate image; performing a distortion analysis on the first candidate image, and performing distortion correction on the first candidate image based on a result of the distortion analysis to obtain a second candidate haze image; Perform sliding average processing on the second candidate haze image based on a multi-scale sliding window to obtain a haze image to be processed: in, Indicates the The sliding average output value of pixels, represents the sliding window size, Indicates the first candidate haze image The image value of each pixel, Indicates the traversal index within the window, Indicates the currently calculated target position index.

7. An image defogging device, characterized in that: The image defogging device comprises: An image preprocessing module is used to preprocess the original haze image to obtain a haze image to be processed; An image dehazing module is configured to input the haze image to be processed into an image dehazing model for dehazing processing. The image dehazing model is constructed based on a Unet network and includes a Unet encoder, a Unet decoder, and an atmospheric scattering module. Both the Unet encoder and the Unet decoder are composed of a multi-layer feature extraction module. The feature extraction module is composed of a first branch structure, a second branch structure, and a feature fusion structure. The first branch structure is configured to extract local haze features and image texture features of the input image of the feature extraction module. The second branch structure is configured to extract global haze features of the input image of the feature extraction module. The feature fusion structure is configured to perform feature fusion on the local haze features, image texture features, and global haze features. An image encoding module is used to call each feature extraction module in the Unet encoder to perform feature extraction and downsampling processing on the haze image to be processed layer by layer, and output a multi-scale feature image; An image decoding module is used to call each feature extraction module in the Unet decoder to perform upsampling and feature extraction on the multi-scale feature image layer by layer, and output an intermediate image; An image restoration module is used to call the atmospheric scattering module to perform image restoration processing on the intermediate image to obtain a target defogging image, and call the output layer of the image defogging model to output the target defogging image.

8. An image defogging device, characterized in that: The image defogging device includes: a memory, a processor, and an image defogging program stored in the memory and executable on the processor, wherein the image defogging program is configured to implement the image defogging method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores an image defogging program, and when the image defogging program is executed by the processor, the image defogging method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product comprises an image defogging program, and when the image defogging program is executed by a processor, the steps of the image defogging method according to any one of claims 1 to 6 are implemented.

Images