A channel image defogging method, system, and computing device

Abstract

The application discloses a channel image defogging method, system and computing device, and the method comprises the following steps: acquiring a foggy image of a target channel, performing feature extraction on the foggy image to obtain an initial feature map; performing feature processing on the initial feature map to obtain an initial local feature map and an initial global feature map; performing interactive fusion based on the initial local feature map and the initial global feature map to obtain a local fusion feature map and a global fusion feature map; performing color enhancement on the local fusion feature map and the global fusion feature map respectively to obtain a local enhanced feature map and a global enhanced feature map; and performing fusion reconstruction based on the local enhanced feature map and the global enhanced feature map to obtain a target defogging image of the target channel. The method solves the problem that the defogging effect of the existing method for the channel image is unstable, resulting in poor accuracy of the obtained defogging image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a channel image defogging method and system and a computing device. BACKGROUND

[0002] In recent years, image defogging technology has been widely concerned in the field of computer vision, especially in intelligent transportation, automatic driving and environmental perception and the like. However, most of the existing defogging methods focus on land scenes, especially urban road images, and the processing of channel images is relatively scarce. In the channel environment, there are often complex lighting and reflection conditions. When the existing land image defogging method is used to defog the channel image under complex lighting conditions, the defogging effect is unstable, resulting in poor accuracy of the obtained defogging image. SUMMARY

[0003] In order to overcome the problem that the existing method has unstable defogging effect for the channel image under complex lighting conditions, resulting in poor accuracy of the obtained defogging image, the present application provides a channel image defogging method, system and computing device.

[0004] In a first aspect, in order to solve the above technical problem, the present application provides a channel image defogging method, comprising:

[0005] Obtaining a foggy image of a target channel, performing feature extraction on the foggy image to obtain an initial feature map;

[0006] Performing feature processing on the initial feature map to obtain an initial local feature map and an initial global feature map;

[0007] Performing interactive fusion based on the initial local feature map and the initial global feature map to obtain a local fusion feature map and a global fusion feature map;

[0008] Performing color enhancement on the local fusion feature map and the global fusion feature map respectively to obtain a local enhanced feature map and a global enhanced feature map;

[0009] Performing fusion reconstruction based on the local enhanced feature map and the global enhanced feature map to obtain a target defogging image of the target channel.

[0010] Further, the initial feature map is processed to obtain an initial local feature map and an initial global feature map, comprising:

[0011] Performing local fusion processing on the initial feature map to obtain an intermediate fusion feature map;

[0012] Performing residual connection on the initial feature map and the intermediate fusion feature map to obtain the initial local feature map.

[0013] Further, the initial feature map is subjected to local fusion processing to obtain an intermediate fusion feature map, including:

[0014] The initial feature map is subjected to multi-scale convolution processing to obtain a multi-scale detail feature map, and the multi-scale detail feature map includes a plurality of shallow feature maps of different scales;

[0015] The multi-scale detail feature map is subjected to feature fusion by using a preset star operation to obtain a plurality of initial fusion feature maps;

[0016] The initial fusion feature map is subjected to semantic extraction by using a preset depth separable convolution and a preset fully connected layer to obtain a semantic feature map;

[0017] The plurality of semantic feature maps are subjected to feature fusion by using a star operation to obtain an intermediate fusion feature map.

[0018] Further, the initial feature map is subjected to feature processing to obtain an initial local feature map and an initial global feature map, including:

[0019] The initial feature map is subjected to convolution processing to obtain a local context feature map;

[0020] The foggy image is processed by using a preset self-attention network to obtain a global attention feature map;

[0021] The global attention feature map and the local context feature map are fused to obtain the initial global feature map.

[0022] Further, the self-attention network includes a plurality of residual connection global feature extraction networks; the foggy image is processed by using a preset self-attention network to obtain a global attention feature map, including:

[0023] The foggy image is subjected to window division to obtain a plurality of window images;

[0024] The plurality of window images are subjected to parallel processing by using a plurality of global feature extraction networks to obtain a plurality of window global feature maps; wherein the global feature extraction network includes a multi-head self-attention mechanism and a multi-layer perception machine;

[0025] The plurality of window global feature maps are subjected to residual connection to obtain the global attention feature map.

[0026] Further, the initial local feature map and the initial global feature map are subjected to interactive fusion to obtain a local fusion feature map and a global fusion feature map, including:

[0027] The initial local feature map and the initial global feature map are subjected to interactive fusion by using a preset gating mechanism to obtain an interactive fusion feature map;

[0028] The interaction fusion feature map is connected with the initial local feature map in a residual manner to obtain a local fusion feature map.

[0029] The interaction fusion feature map is connected with the initial global feature map in a residual manner to obtain a global fusion feature map.

[0030] Further, the initial local feature map and the initial global feature map are fused by using a preset gating mechanism to obtain an interaction fusion feature map, including:

[0031] The initial local feature map and the initial global feature map are fused to obtain a rough fusion feature map.

[0032] The initial local feature map and the initial global feature map are fused by using a preset gating mechanism based on the rough fusion feature map to obtain a weighted fusion feature map.

[0033] The weighted fusion feature map is enhanced by using a preset channel attention network to obtain a channel attention enhanced feature map.

[0034] The weighted fusion feature map is enhanced by using a preset spatial attention network to obtain a spatial attention enhanced feature map.

[0035] The spatial attention enhanced feature map, the channel attention enhanced feature map and the rough fusion feature map are connected to obtain the interaction fusion feature map.

[0036] Further, the local fusion feature map and the global fusion feature map are respectively color enhanced to obtain a local enhanced feature map and a global enhanced feature map, including:

[0037] The local fusion feature map is divided into features to obtain a local reserved feature map and a local to-be-transformed feature map.

[0038] The local to-be-transformed feature map is wavelet transformed to obtain a local transformed feature map.

[0039] The local transformed feature map, the local reserved feature map and the initial feature map are connected to obtain the local enhanced feature map.

[0040] The global fusion feature map is divided into features to obtain a global reserved feature map and a global to-be-transformed feature map.

[0041] The global to-be-transformed feature map is wavelet transformed to obtain a global transformed feature map.

[0042] The global transformed feature map, the global reserved feature map and the initial feature map are connected to obtain the global enhanced feature map.

[0043] In a second aspect, the application further provides a channel image defogging system, including:

[0044] The acquisition module is configured to acquire a foggy image of the target channel, and perform feature extraction on the foggy image to obtain an initial feature map;

[0045] The feature processing module is configured to perform feature processing on the initial feature map to obtain an initial local feature map and an initial global feature map;

[0046] The interactive fusion module is configured to perform interactive fusion based on the initial local feature map and the initial global feature map to obtain a local fusion feature map and a global fusion feature map;

[0047] The color enhancement module is configured to perform color enhancement on the local fusion feature map and the global fusion feature map respectively to obtain a local enhanced feature map and a global enhanced feature map;

[0048] The fusion reconstruction module is configured to perform fusion reconstruction based on the local enhanced feature map and the global enhanced feature map to obtain a target dehazed image of the target channel.

[0049] In a third aspect, the present application also provides a computing device, which comprises a memory, a processor, and a program stored in the memory and running on the processor, and the processor implements the steps of the channel image dehazing method as described above when executing the program.

[0050] In a fourth aspect, the present application also provides a computer readable storage medium, which stores instructions, and the instructions make the terminal device execute the steps of the channel image dehazing method when running on the terminal device.

[0051] The beneficial effects of the present application are as follows. Firstly, the initial local feature map containing detailed information and the initial global feature map containing structural features are obtained by performing feature extraction on the foggy image of the target channel and then performing feature processing, and the local fusion feature map and the global fusion feature map are obtained by performing interactive fusion based on the initial local feature map and the initial global feature map. Secondly, the color enhancement is performed on the local fusion feature map and the global enhanced feature map respectively to sufficiently extract the high-frequency detailed information and the structural features of the remote area contained in the foggy image, and the fusion reconstruction is performed based on the local enhanced feature map and the global enhanced feature map obtained by color enhancement to realize the collaborative modeling of the local details and the global structural information of the foggy image, so as to adapt to the foggy channel under complex lighting environment, thereby being capable of significantly improving the visual quality and robustness of the image dehazing in the channel scene, improving the image clarity and reality, and further improving the accuracy of the obtained target dehazed image. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 A flowchart of a channel image dehazing method according to an example embodiment of the present application is shown;

[0053] Figure 2 A local feature processing network structure diagram in an exemplary embodiment of the present application;

[0054] Figure 3 A global feature extraction network structure diagram in an exemplary embodiment of the present application;

[0055] Figure 4 An interactive fusion processing network structure diagram in an exemplary embodiment of the present application;

[0056] Figure 5 A color enhancement processing network structure diagram in an exemplary embodiment of the present application;

[0057] Figure 6 A flowchart of a channel image defogging method provided by an exemplary embodiment of the present application;

[0058] Figure 7 A comparison diagram of defogging effects of the channel image defogging method of the present application and other existing methods;

[0059] Figure 8 A structural diagram of a channel image defogging system according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0060] The following examples are further explanations and supplements of the present application and do not constitute any limitation on the present application.

[0061] The existing mainstream defogging methods mainly include traditional image enhancement algorithms, physical model methods based on atmospheric scattering models, and end-to-end image restoration methods based on deep learning.

[0062] Traditional image enhancement methods, such as histogram equalization and Retinex algorithm (an image processing technology based on the biological visual system), are simple to implement, but are difficult to adapt to complex lighting and reflection conditions, and the defogging effect is unstable. Methods based on physical models such as dark channel prior (DCP) or color line prior often fail when processing water area images due to the existence of strong reflection and low texture areas, and are prone to problems such as color deviation and structure loss. Deep learning methods have achieved good performance, but most model designs do not take into account the particularity of water area scenes, such as large-area water surface, complex light reflection, and long-distance fog distribution.

[0063] In order to solve the above problems, the embodiments of the present application provide a channel image defogging method, system and computing device, which will be described in detail below.

[0064] The channel image defogging method provided by the embodiment of the application can be specifically executed by a server. It should be noted that the server can be a stand-alone server, or a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content delivery network (CDN), and big data and artificial intelligence platform, which is not limited herein.

[0065] Please refer to Figure 1 , Figure 1 The channel image defogging method shown in an exemplary embodiment of the application is as shown in Figure 1 The application provides a channel image defogging method, comprising:

[0066] S11, obtaining a foggy image of a target channel, performing feature extraction on the foggy image to obtain an initial feature map;

[0067] S12, performing feature processing on the initial feature map to obtain an initial local feature map and an initial global feature map;

[0068] S13, performing interactive fusion based on the initial local feature map and the initial global feature map to obtain a local fusion feature map and a global fusion feature map;

[0069] S14, performing color enhancement on the local fusion feature map and the global fusion feature map respectively to obtain a local enhanced feature map and a global enhanced feature map;

[0070] S15, performing fusion reconstruction based on the local enhanced feature map and the global enhanced feature map to obtain a target defogging image of the target channel.

[0071] The channel image defogging method of the embodiment provided by the application first performs feature extraction on the foggy image of the target channel, then performs feature processing to obtain an initial local feature map containing detailed information and an initial global feature map containing structural features, and performs interactive fusion based on the initial local feature map and the initial global feature map to obtain a local fusion feature map and a global fusion feature map. Secondly, the local fusion feature map and the global enhanced feature map are respectively color enhanced to fully extract the high-frequency detailed information and the structural features of the long-distance area contained in the foggy image, and fusion reconstruction is performed based on the local enhanced feature map and the global enhanced feature map obtained by color enhancement to realize collaborative modeling of the local details and the global structural information of the foggy image, so as to adapt to the foggy channel in a complex lighting environment, thereby significantly improving the visual quality and robustness of image defogging in the channel scene, improving the image clarity and realism, and further improving the accuracy of the obtained target defogging image. The method of the application is particularly suitable for water image clarification tasks in ship navigation.

[0072] Optionally, the initial feature map is processed to obtain an initial local feature map and an initial global feature map, including:

[0073] The initial feature map is processed by local fusion to obtain an intermediate fusion feature map.

[0074] The initial feature map and the intermediate fusion feature map are connected by residual connection to obtain the initial local feature map.

[0075] In the embodiment provided in the application, by processing the initial feature map by local fusion to extract the detail information in the initial feature map, the intermediate fusion feature map is obtained, and the initial feature map and the intermediate fusion feature map are connected by residual connection to obtain the initial local feature map, so as to strengthen the detail information in the initial local feature map, thereby retaining more detail information in the subsequent defogging process, and thus the accuracy of the target defogging image finally obtained can be improved.

[0076] Optionally, the initial feature map is processed by local fusion to obtain an intermediate fusion feature map, including:

[0077] The initial feature map is processed by multi-scale convolution to obtain a multi-scale detail feature map, and the multi-scale detail feature map includes a plurality of shallow feature maps of different scales.

[0078] The multi-scale detail feature map is fused by using a preset star operation to obtain a plurality of initial fusion feature maps.

[0079] The initial fusion feature map is processed by using a preset depth separable convolution and a preset full connection layer to obtain a semantic feature map.

[0080] The plurality of semantic feature maps are fused by using the star operation to obtain the intermediate fusion feature map.

[0081] In the embodiment provided in the present application, the initial feature map is subjected to multi-scale convolution processing, which can extract edge, texture and other high-frequency detail information in the initial feature map from multiple scales using convolution kernels of different receptive fields, so as to alleviate the problem of detail loss caused by complex lighting environments such as light scattering and water surface reflection in foggy images. At the same time, the multi-scale detail feature maps are subjected to feature fusion using a preset star operation (Star Operation) to obtain multiple initial fusion feature maps, and the initial fusion feature maps are subjected to semantic extraction using a preset depth separable convolution and a preset fully connected layer to obtain semantic feature maps. The multiple semantic feature maps are subjected to feature fusion again using the star operation to obtain an intermediate fusion feature map, realizing new splicing and fusion of different scales, which can retain original information and strengthen the stability of feature learning, so as to enhance the recovery effect of the water surface boundary and the object contour in the image in the subsequent haze removal processing process, retain more high-frequency detail information, and thus improve the accuracy of the target haze-removed image obtained finally.

[0082] In an example embodiment provided in the present application, first, the initial feature map is subjected to multi-scale convolution processing to obtain a multi-scale detail feature map, and the specific steps are as follows:

[0083] The foggy image is subjected to convolution operation using multiple convolution kernels of different sizes to obtain multiple shallow layer feature maps; wherein the number of shallow layer feature maps can be 4, and the multiple convolution kernels of different sizes include 1x1 convolution kernel, 3x3 convolution kernel, 5x5 convolution kernel and 7x7 convolution kernel.

[0084] The multi-scale detail feature map is formed based on the multiple shallow layer feature maps.

[0085] Secondly, the multi-scale detail feature map is subjected to feature fusion using a preset star operation to obtain multiple initial fusion feature maps, and the specific steps can be as follows:

[0086] The four shallow layer feature maps are subjected to two-by-two fusion using the preset star operation to obtain two groups of initial fusion feature maps.

[0087] The star operation can map the input feature to a very high-dimensional nonlinear feature space, thereby significantly enhancing the expression ability of the feature. The star operation can be expressed as:

[0088]

[0089]

[0090]

[0091]

[0092] wherein, denotes a first weight matrix, denotes a second weight matrix, denotes an input feature tensor, denotes a number of channels, denotes a channel index, denotes a coefficient of each term, i.e.,

[0093]

[0094] The above formula can be expanded as different sub-terms. Therefore, in a dimensional space, the star operation with higher computational efficiency can implicitly construct a dimensional feature space, which significantly improves the feature dimension without increasing the single-layer computational overhead.

[0095] In this way, by introducing the star operation, the feature mapping is projected to a high-dimensional space without increasing the network depth, further enhancing the ability to capture visual details such as color, texture, and shape in the image during the defogging process.

[0096] Please refer to Figure 2 , Figure 2 is a local feature processing network structure diagram in an exemplary embodiment of the present application, as shown in Figure 2 , the local feature processing network structure includes four parallel multi-scale convolutional networks, two-stage feature fusion networks, a deep feature extraction network, and a final output adjustment network. Among them, the types of convolutional layers in each network are shown in the content displayed by the dashed box in Figure 2 .

[0097] First, the initial feature map is convolved by four different sizes of convolution kernels (1x1, 3x3, 5x5, 7x7) to obtain multi-scale shallow feature maps. These feature maps retain rich spatial structure information, which is helpful for subsequent deep modeling.

[0098] In the first stage of the feature fusion network, the star operation is used to fuse the four groups of shallow feature maps two by two to obtain two groups of initial fusion feature maps. Then, each group of fusion features is further extracted through the deep separable convolution and fully connected layer included in the deep feature extraction network to obtain semantic feature maps, thereby enhancing the feature representation ability.

[0099] In the second stage of the feature fusion network, the star operation is used again to fuse the two groups of semantic feature maps to obtain intermediate fusion feature maps.

[0100] The intermediate fused feature map after fusion and the initial feature map of the original input are added through residual connection to obtain an initial local feature map to retain original information and strengthen the stability of feature learning. In order to ensure that the size of the output feature is consistent with the input, an output adjustment network introduces a 3*3 convolution layer for dimension matching to ensure that the initial local feature map of the final output can be seamlessly connected to the subsequent network structure.

[0101] Optionally, the initial feature map is processed to obtain an initial local feature map and an initial global feature map, including:

[0102] The initial feature map is processed by convolution to obtain a local context feature map;

[0103] The foggy image is processed by using a preset self-attention network to obtain a global attention feature map;

[0104] The global attention feature map and the local context feature map are fused to obtain an initial global feature map.

[0105] In the embodiment provided in the application, the initial feature map is processed by convolution to obtain a local context feature map, and the foggy image is processed by using a preset self-attention network to obtain a global attention feature map. The global attention feature map and the local context feature map are fused to obtain an initial global feature map. In this way, since the self-attention network is good at modeling the spatial dependence of long-distance regions in the image and is particularly suitable for water navigation scenes containing vast sky and water surface backgrounds, the long-range target information in the foggy image can be strengthened by the self-attention network, avoiding the loss of long-range target information caused by fog, so that the initial global feature map not only effectively reconstructs the water surface structure of the target channel, but also suppresses the image artifacts caused by complex reflections in the target channel, thereby retaining more overall structural information in the subsequent dehazing process, and further improving the accuracy of the target dehazed image obtained finally. The local context feature map can be captured by an attention mechanism.

[0106] Optionally, the self-attention network includes a plurality of globally feature extraction networks connected by residual connection; the foggy image is processed by using a preset self-attention network to obtain a global attention feature map, including:

[0107] The foggy image is windowed to obtain a plurality of window images;

[0108] The plurality of window images are processed in parallel by using a plurality of global feature extraction networks to obtain a plurality of window global feature maps; wherein the global feature extraction network includes a multi-head self-attention mechanism and a multi-layer perception machine;

[0109] The plurality of window global feature maps are connected by residual connection to obtain a global attention feature map.

[0110] In this embodiment provided by the present application, in the self-attention network, the foggy image is windowed to obtain a plurality of window images, and the plurality of window images are processed in parallel by using a plurality of residual connection global feature extraction networks to obtain a plurality of window global feature maps, and the plurality of window global feature maps are residual connected to obtain a global attention feature map. In this way, by dividing a large foggy image into a plurality of small window images and processing the plurality of small window images in parallel, the processing difficulty of the foggy image can be reduced, the processing speed of the foggy image can be improved, and the efficiency of the foggy image of the target channel can be improved.

[0111] In this embodiment, the self-attention network can be represented as:

[0112] ;

[0113] wherein Q is Query, indicating query, K is Key, indicating key, V is Value, indicating value, d indicates the dimension of the feature, B indicates the sample batch size (batchsize), indicates the original foggy image without windowing, indicates a depth separable convolution (DWConv) or a Conv-Relu-Conv convolution block.

[0114] The self-attention network is used to model the global relationship between the long-distance regions in the image, and the self-attention network includes a plurality of residual connection global feature extraction networks. The structure of the global feature extraction network is as shown in Figure 3 , which includes a multi-head self-attention mechanism (W-MHSA) and a multi-layer perception (MLP). Both of them use a layer normalization layer (LayerNorm) as a preprocessing layer, and are connected by a residual connection to stabilize the training process and enhance the feature expression ability. In this way, by stacking a plurality of global feature extraction networks, the large-scale structure information in the foggy image can be fully extracted, and the accuracy and consistency of the image restoration can be further improved.

[0115] The traditional Swin Transformer (a visual model) adopts a windowing and cyclic shift strategy to realize efficient region modeling, but due to the mask mechanism limitation of the image edge region, it may cause the effective receptive field of the window to shrink, and thus weaken the expression ability of the edge features. To this end, the application uses a self-attention network (a deep learning model architecture based on a multi-head self-attention mechanism) method with a reflection padding strategy to ensure the window size consistency in the entire image range, effectively preserve the image edge information, and improve the dehazing precision. To enhance the information interaction between the windows, a parallel convolution branch is added in the global modeling branch to supplement the deficiencies of the traditional self-attention module in local modeling. The convolution branch performs convolution processing on the input features before window division to extract the local context information across the windows, and fuses the output of the self-attention network to realize joint modeling of global and local information.

[0116] Optionally, the initial local feature map and the initial global feature map are interactively fused based on a preset gating mechanism to obtain a local fusion feature map and a global fusion feature map, including:

[0117] The initial local feature map and the initial global feature map are interactively fused by using a preset gating mechanism to obtain an interactive fusion feature map;

[0118] The interactive fusion feature map and the initial local feature map are connected in residual to obtain a local fusion feature map;

[0119] The interactive fusion feature map and the initial global feature map are connected in residual to obtain a global fusion feature map.

[0120] In the embodiment provided by the application, the initial local feature map and the initial global feature map are interactively fused by using a preset gating mechanism to obtain an interactive fusion feature map, and a local fusion feature map formed by connecting the interactive fusion feature map and the initial local feature map in residual, and a global fusion feature map formed by connecting the interactive fusion feature map and the initial global feature map in residual, further improve the amount of feature information in the local fusion feature map and the global fusion feature map, facilitate the preservation of more detailed information in the subsequent dehazing process, and thus improve the accuracy of the target dehazing image obtained finally.

[0121] Optionally, the initial local feature map and the initial global feature map are interactively fused by using a preset gating mechanism to obtain an interactive fusion feature map, including:

[0122] The initial local feature map and the initial global feature map are fused to obtain a rough fusion feature map;

[0123] The calculation formula of the rough fusion feature map is as follows:

[0124] ;

[0125] in, This represents a coarsely fused feature map. Represents the initial local feature map. Represents the initial global feature map. This indicates a splicing operation along the channel dimension;

[0126] Using a pre-defined gating mechanism, the initial local feature map and the initial global feature map are weighted and fused based on the coarse fused feature map to obtain a weighted fused feature map;

[0127] The formula for calculating the weighted fusion feature map is as follows:

[0128] ;

[0129] ;

[0130] in, This represents the weighted fusion feature map. Indicates the gate value, Represents the initial local feature map. This represents the initial global feature map, and Sigmoid represents the activation function. express Depth-separable convolution, Indicates an expansion rate of 2 Depthwise separable dilated convolution;

[0131] The weighted fusion feature map is enhanced by using a pre-defined channel attention network to obtain a channel attention-enhanced feature map;

[0132] The formula for calculating the channel attention-enhanced feature map is as follows:

[0133] ;

[0134] in, This represents the channel attention enhancement feature map. Indicates the kernel size as Convolution operation, This represents global average pooling along the channel dimension. Represents the ReLU activation function. This represents the weighted fusion feature map;

[0135] The weighted fusion feature map is enhanced using a pre-defined spatial attention network to obtain a spatial attention-enhanced feature map;

[0136] The formula for calculating the spatial attention-enhanced feature map is as follows:

[0137] ;

[0138] wherein, represents a spatial attention enhanced feature map, represents a convolution operation with a kernel size of , represents a global max pooling operation, represents a weighted fusion feature map, represents a global average pooling along the spatial dimension;

[0139] The spatial attention enhanced feature map, the channel attention enhanced feature map and the rough fusion feature map are connected to obtain an interactive fusion feature map.

[0140] The calculation formula of the interactive fusion feature map is as follows:

[0141] ;

[0142] wherein, represents an interactive fusion feature map, represents a convolution operation with a kernel size of , represents a channel attention enhanced feature map, represents a spatial attention enhanced feature map, represents a rough fusion feature map.

[0143] In the embodiment provided in the present application, firstly, based on the rough fusion feature map obtained by fusing the initial local feature map and the initial global feature map, the initial local feature map and the initial global feature map are weighted and fused, and a preset gating mechanism is used to control the fusion weight in the process of weighted fusion, so that the network can adaptively select the optimal fusion strategy to obtain the optimal weighted fusion feature map. Secondly, the weighted fusion feature map is enhanced by using a preset channel attention network and a spatial attention network respectively, so as to enhance the feature representation ability of the key channel and the spatial structure information in the weighted fusion feature map, and the channel attention enhanced feature map and the spatial attention enhanced feature map obtained by feature enhancement, and the rough fusion feature map are connected to obtain an interactive fusion feature map, so that the interactive fusion feature map can contain sufficient feature representation ability and spatial structure information, thereby improving the amount of feature information in the target defogging image obtained by subsequent reconstruction, and further improving the accuracy of the target defogging image.

[0144] In the embodiment, the spatial attention enhanced feature map, the channel attention enhanced feature map and the rough fusion feature map are connected to obtain an interactive fusion feature map, and the specific steps are as follows:

[0145] The channel attention enhanced feature is connected with the spatial attention enhanced feature After splicing, it is combined with coarse fusion features. Perform a residual join, then use a convolution To restore the number of channels, the final interactive fusion feature map is obtained. .

[0146] Please see Figure 4 , Figure 4 This is a diagram of the interactive fusion processing network structure in an exemplary embodiment of this application, such as... Figure 4 As shown, the interactive fusion processing network includes a first feature stitching layer, a gated fusion unit, a channel attention network, a spatial attention network, and a second feature stitching layer. The specific processing logic of the interactive fusion processing network is as follows:

[0147] In the first feature concatenation layer, the initial local feature map and the initial global feature map are fused to obtain a coarse fused feature map;

[0148] In the gated fusion unit, a pre-set gating mechanism is used to perform weighted fusion of the initial local feature map and the initial global feature map based on the coarse fusion feature map to obtain a weighted fusion feature map;

[0149] In the channel attention network, feature enhancement is performed on the weighted fusion feature map to obtain the channel attention enhanced feature map;

[0150] In spatial attention networks, feature enhancement is performed on weighted fusion feature maps to obtain spatial attention-enhanced feature maps;

[0151] In the second feature concatenation layer, the spatial attention-enhanced feature map, the channel attention-enhanced feature map, and the coarse fusion feature map are concatenated to output an interactive fusion feature map.

[0152] Optionally, the local fusion feature map and the global fusion feature map are respectively color-enhanced to obtain a local enhanced feature map and a global enhanced feature map, including:

[0153] The local fused feature map is divided into local preserved feature maps and local feature maps to be transformed.

[0154] The local feature map to be transformed is subjected to wavelet transform to obtain the local transformed feature map;

[0155] The local transformed feature map, the local preserved feature map, and the initial feature map are concatenated to obtain the local enhanced feature map;

[0156] The global fused feature map is divided into a global retained feature map and a global feature map to be transformed.

[0157] Perform wavelet transform on the global feature map to be transformed to obtain the global transformed feature map;

[0158] The global transformed feature map, the global preserved feature map, and the initial feature map are concatenated to obtain the global enhanced feature map.

[0159] In the embodiment provided in this application, a local preserved feature map and a local feature map to be transformed are obtained by feature partitioning the local fused feature map. The local feature map to be transformed is then subjected to wavelet transform to obtain a local transformed feature map. A local enhanced feature map is formed by concatenating the local transformed feature map, the local preserved feature map, and the initial feature map. The global fused feature map is then processed using the same method described above to obtain a global enhanced feature map. In this way, by acquiring high-frequency color-related features and rich detail texture information in the image through wavelet transform and feature concatenation, the color expressiveness of the local and global enhanced feature maps can be enhanced. This improves the color expressiveness of the target dehazed image obtained by fusing and reconstructing based on the local and global enhanced feature maps, thereby improving the accuracy of the target dehazed image.

[0160] Please see Figure 5 , Figure 5 This is a diagram of the color enhancement processing network structure in an exemplary embodiment of this application, such as... Figure 5 As shown, the color enhancement processing network includes a splitting network, a wavelet transform network, a splicing network, a residual connection network, and a convolutional network. The wavelet transform network includes a discrete wavelet transform network, a wavelet domain layer, an inverse wavelet transform network, a connection network, and a 1×1 convolutional layer. The wavelet domain layer includes a channel attention network and a spatial attention network. The specific processing logic of the color enhancement processing network is as follows:

[0161] In the split network, the input feature map (local fusion feature map or global fusion feature map) is divided into a retained feature map (local retained feature map or global retained feature map) and a feature map to be transformed (local feature map to be transformed or global feature map to be transformed).

[0162] In a discrete wavelet transform network, the feature map to be transformed (local feature map or global feature map) is first transformed into the wavelet domain through discrete wavelet transform to obtain the first transformed feature map (first local transformed feature map or first global transformed feature map).

[0163] In the wavelet domain, the first transform feature map after discrete wavelet transform is weighted sequentially using channel attention network and spatial attention network to extract rich contextual information and image details, resulting in the second transform feature map (second local transform feature map or second global transform feature map).

[0164] In the inverse wavelet transform network, the second transform feature map is subjected to inverse wavelet transform to obtain the transform feature map (local transform feature map or global transform feature map).

[0165] In the connection network, the transformed feature map (local transformed feature map or global transformed feature map) is spliced with the reserved feature map (corresponding local reserved feature map or global reserved feature map) to obtain a spliced feature map (local spliced feature map or global spliced feature map);

[0166] The spliced feature map is connected in residual connection with the initial feature map of the original input to obtain an enhanced feature map (local enhanced feature map or global enhanced feature map) to slow down the information loss and promote the stable learning of the network;

[0167] In the 1x1 convolutional layer, the enhanced feature map (local enhanced feature map or global enhanced feature map) is processed and output.

[0168] In this embodiment, the local related features and the corresponding steps constitute a color enhancement scheme of the local fusion feature map, and the global related features and the corresponding steps constitute a color enhancement scheme of the global fusion feature map.

[0169] Please refer to Figure 6 , Figure 6 For an exemplary embodiment of the present application, the flowchart of the channel image defogging method provided by the application is shown in Figure 6 The flow of the channel image defogging method provided by the application is as follows:

[0170] An initial haze image (a haze image of the target channel) is obtained, and the haze image is subjected to feature extraction to obtain an initial feature map;

[0171] In the global feature processing module, the initial feature map is subjected to feature processing to obtain an initial global feature map;

[0172] In the local feature processing module, the initial feature map is subjected to feature processing to obtain an initial local feature map;

[0173] In the feature information interaction module, the initial local feature map and the initial global feature map are interactively fused to obtain a local fusion feature map and a global fusion feature map;

[0174] In the color enhancement module, the local fusion feature map and the global fusion feature map are subjected to color enhancement respectively to obtain a local enhanced feature map and a global enhanced feature map;

[0175] Based on the local enhanced feature map and the global enhanced feature map, a fusion feature map is obtained, and based on the fusion feature map, a target defogging image of the target channel is formed.

[0176] Please refer to Figure 7 , Figure 7 For comparison of the defogging effect of the channel image defogging method of the present application with other existing methods, the comparison chart is as follows:Figure 7 As shown in FIG. 1(a), Figure 7 As shown in FIG. 1(a), Figure 7 As shown in FIG. 1(b), Figure 7 As shown in FIG. 1(c), Figure 7 As shown in FIG. 1(d), through comparative analysis, compared with the dehazed images obtained by the existing FFA-Net method and the Dehazeformer method, the dehazed images obtained by the channel image dehazing method of the present application have higher definition and better dehazing effect, so the target dehazed image obtained by the method of the present application has higher accuracy.

[0177] The channel image dehazing method of the present application effectively restores the near and far targets blocked by haze through the dual-branch structure of local and global feature fusion, improves the perception ability of ships in haze weather conditions. The local branch introduces a multi-scale convolution module and a star operation, which can extract high-frequency details and accurately reconstruct the water surface boundary, object contour and low-texture area. The global branch uses a self-attention network to form an autonomous dependent network, which strengthens the modeling ability of the vast sky and water area in the water image and improves the expression effect of the long-distance information, which is convenient for improving the visibility and definition of the target dehazed image obtained subsequently. At the same time, a gating modulation feature fusion module is used to dynamically adjust the weight of local and global information according to the image content, fully play the complementary advantages of local and global features, and realize the adaptive fusion of local and global features. A color enhancement module based on wavelet transform is introduced to effectively restore the color information degraded by haze in the image and improve the overall visual quality of the image, so as to improve the color restoration effect of the image and thus improve the accuracy of the target dehazed image obtained by final fusion and reconstruction.

[0178] Please refer to Figure 8 , Figure 8 A channel image dehazing system is shown in an exemplary embodiment of the present application, as shown in FIG. 8, Figure 8 The present application provides a channel image dehazing system 800, which comprises:

[0179] The acquisition module 801 is used for acquiring the foggy image of the target channel, performing feature extraction on the foggy image, and obtaining an initial feature map;

[0180] The feature processing module 802 is configured to perform feature processing on the initial feature map to obtain an initial local feature map and an initial global feature map.

[0181] The interactive fusion module 803 is configured to perform interactive fusion based on the initial local feature map and the initial global feature map to obtain a local fusion feature map and a global fusion feature map.

[0182] The color enhancement module 804 is configured to perform color enhancement on the local fusion feature map and the global fusion feature map respectively to obtain a local enhanced feature map and a global enhanced feature map.

[0183] The fusion reconstruction module 805 is configured to perform fusion reconstruction based on the local enhanced feature map and the global enhanced feature map to obtain a target defogging image of the target channel.

[0184] The channel image defogging system 800 provided in the embodiment provided in the present application first performs feature extraction on the obtained foggy image of the target channel by using the acquisition module 801, then performs feature processing on the initial feature map obtained by feature extraction by using the feature processing module 802 to obtain an initial local feature map containing detailed information and an initial global feature map containing structural features, and performs interactive fusion based on the initial local feature map and the initial global feature map by using the interactive fusion module 803 to obtain a local fusion feature map and a global fusion feature map. Secondly, the color enhancement module 804 is used to perform color enhancement on the local fusion feature map and the global enhanced feature map respectively to fully extract the high-frequency detailed information and the structural features of the remote area contained in the foggy image, and the fusion reconstruction module 805 is used to perform fusion reconstruction based on the local enhanced feature map and the global enhanced feature map obtained by color enhancement to realize the collaborative modeling of the local details and the global structural information of the foggy image, so as to adapt to the foggy channel in a complex lighting environment, thereby significantly improving the visual quality and robustness of image defogging in the channel scene, improving the image clarity and realism, and further improving the accuracy of the obtained target defogging image.

[0185] Optionally, the feature processing module 802 is specifically configured to:

[0186] perform local fusion processing on the initial feature map to obtain an intermediate fusion feature map;

[0187] perform residual connection on the initial feature map and the intermediate fusion feature map to obtain the initial local feature map.

[0188] Optionally, the feature processing module 802 is specifically configured to:

[0189] perform multi-scale convolution processing on the initial feature map to obtain a multi-scale detailed feature map, and the multi-scale detailed feature map includes a plurality of shallow feature maps of different scales;

[0190] The multi-scale detail feature maps are fused by using preset star operations to obtain a plurality of initial fusion feature maps.

[0191] The initial fusion feature maps are subjected to semantic extraction by using preset depth separable convolution and preset full connection layers to obtain semantic feature maps.

[0192] The plurality of semantic feature maps are fused by using star operations to obtain intermediate fusion feature maps.

[0193] Optionally, the feature processing module 802 is specifically configured to:

[0194] The initial feature maps are subjected to convolution processing to obtain local context feature maps.

[0195] The foggy image is processed by using a preset self-attention network to obtain a global attention feature map.

[0196] The global attention feature map and the local context feature map are fused to obtain an initial global feature map.

[0197] Optionally, the feature processing module 802 is specifically configured to:

[0198] The foggy image is subjected to window division to obtain a plurality of window images.

[0199] The plurality of window images are subjected to parallel processing by using a plurality of global feature extraction networks to obtain a plurality of window global feature maps; wherein the global feature extraction network comprises a multi-head self-attention mechanism and a multi-layer perception machine.

[0200] The plurality of window global feature maps are subjected to residual connection to obtain the global attention feature map.

[0201] Optionally, the interactive fusion module 803 is specifically configured to:

[0202] The initial local feature map and the initial global feature map are subjected to interactive fusion by using a preset gating mechanism to obtain an interactive fusion feature map.

[0203] The interactive fusion feature map and the initial local feature map are subjected to residual connection to obtain a local fusion feature map.

[0204] The interactive fusion feature map and the initial global feature map are subjected to residual connection to obtain a global fusion feature map.

[0205] Optionally, the interactive fusion module 803 is specifically configured to:

[0206] The initial local feature map and the initial global feature map are fused to obtain a rough fusion feature map.

[0207] The preset gating mechanism is used to perform weighted fusion on the initial local feature map and the initial global feature map based on the coarse fusion feature map, to obtain a weighted fusion feature map.

[0208] The preset channel attention network is used to perform feature enhancement on the weighted fusion feature map, to obtain a channel attention enhanced feature map.

[0209] The preset spatial attention network is used to perform feature enhancement on the weighted fusion feature map, to obtain a spatial attention enhanced feature map.

[0210] The spatial attention enhanced feature map, the channel attention enhanced feature map and the coarse fusion feature map are connected to obtain an interactive fusion feature map.

[0211] Optionally, the color enhancement module 804 is specifically configured to:

[0212] The local fusion feature map is divided in feature to obtain a local reserved feature map and a local to-be-transformed feature map.

[0213] The local to-be-transformed feature map is wavelet transformed to obtain a local transformed feature map.

[0214] The local transformed feature map, the local reserved feature map and the initial feature map are connected to obtain a local enhanced feature map.

[0215] The global fusion feature map is divided in feature to obtain a global reserved feature map and a global to-be-transformed feature map.

[0216] The global to-be-transformed feature map is wavelet transformed to obtain a global transformed feature map.

[0217] The global transformed feature map, the global reserved feature map and the initial feature map are connected to obtain a global enhanced feature map.

[0218] It should be noted that the channel image defogging system provided in the above embodiments and the channel image defogging method provided in the above embodiments belong to the same concept, wherein the specific manner in which each module and unit performs operations has been described in detail in the method embodiments, which will not be described here. In actual application, the above functions can be distributed by different functional modules according to needs, that is, the internal structure of the system is divided into different functional modules to complete all or part of the functions described above, and this is not limited herein.

[0219] A computing device of an embodiment of the present application includes a memory, a processor, and a program stored on the memory and running on the processor. When the processor executes the program, it implements part or all of the steps of the channel image defogging method described above.

[0220] The computing device can be a computer, and the program can be computer software. The parameters and steps of the method can refer to the parameters and steps of the method for removing fog from a channel image.

[0221] The computer readable storage medium stores instructions. When the instructions are executed, the steps of the method for removing fog from a channel image are performed.

[0222] The computer readable storage medium can be a transitory computer readable storage medium or a non-transitory computer readable storage medium.

[0223] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes one or more instructions for causing a computer device (which can be a personal computer, a server, or a network device) to execute all or part of the steps of the method. The computer readable storage medium can be a non-transitory computer readable storage medium, including a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes, or a transitory computer readable storage medium.

[0224] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present disclosure. Each block in the flowcharts or block diagrams can represent a module, a program segment, or a part of code, which includes one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions annotated in the blocks can occur in an order different from that annotated in the drawings. For example, two blocks indicated in succession can actually be executed substantially in parallel, and sometimes can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams or flowcharts, or a combination of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0225] Those skilled in the art know that the present application can be implemented as a system, a method or a computer program product. Therefore, the present disclosure can be embodied in the form of a hardware completely, a software completely (including firmware, resident software, microcode, etc.), or a combination of hardware and software, which is generally referred to as "module" or "system". Furthermore, in some embodiments, the present application can also be embodied in the form of a computer program product in one or more computer readable storage media having computer readable program codes. The computer readable storage media can be, for example but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or apparatus, or any combination thereof.

[0226] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0227] Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and cannot be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A method for removing fog from a waterway image, characterized by, The method comprises the following steps: obtaining a fog image of a target channel, performing feature extraction on the fog image to obtain an initial feature map; performing feature processing on the initial feature map to obtain an initial local feature map and an initial global feature map; using a preset gating mechanism to interactively fuse the initial local feature map and the initial global feature map to obtain an interactive fusion feature map; performing residual connection on the interactive fusion feature map and the initial local feature map to obtain a local fusion feature map; performing residual connection on the interactive fusion feature map and the initial global feature map to obtain a global fusion feature map; performing color enhancement on the local fusion feature map and the global fusion feature map respectively to obtain a local enhanced feature map and a global enhanced feature map; based on the local enhanced feature map and the global enhanced feature map, performing fusion reconstruction to obtain a target defogging image of the target channel.

2. The method of claim 1, wherein, The method comprises the following steps: performing local fusion processing on the initial feature map to obtain an intermediate fusion feature map; performing residual connection on the initial feature map and the intermediate fusion feature map to obtain an initial local feature map.

3. The method of claim 2, wherein, The method comprises the following steps: performing multi-scale convolution processing on the initial feature map to obtain a multi-scale detail feature map, the multi-scale detail feature map comprising a plurality of different scale shallow feature maps; using a preset star operation to perform feature fusion on the multi-scale detail feature map to obtain a plurality of initial fusion feature maps; using a preset depth separable convolution and a preset fully connected layer to perform semantic extraction on the initial fusion feature map to obtain a semantic feature map; using the star operation to perform feature fusion on a plurality of the semantic feature maps to obtain an intermediate fusion feature map.

4. The method of claim 1, wherein, The method comprises the following steps: performing convolution processing on the initial feature map to obtain a local context feature map; using a preset self-attention network to process the fog image to obtain a global attention feature map; fusing the global attention feature map and the local context feature map to obtain an initial global feature map.

5. The method of claim 4, wherein, The self-attention network comprises a plurality of residual connected global feature extraction networks; the method comprises the following steps: performing window division on the fog image to obtain a plurality of window images; using a plurality of the global feature extraction networks to perform parallel processing on a plurality of the window images to obtain a plurality of window global feature maps; wherein the global feature extraction network comprises a multi-head self-attention mechanism and a multi-layer perception machine; performing residual connection on a plurality of the window global feature maps to obtain a global attention feature map.

6. The method of claim 1, wherein, The method comprises the following steps: fusing the initial local feature map and the initial global feature map to obtain a rough fusion feature map; The initial local feature map and the initial global feature map are weightedly fused based on the preset gating mechanism, to obtain a weighted fusion feature map; The weighted fusion feature map is subjected to feature enhancement by using a preset channel attention network, to obtain a channel attention enhanced feature map; The weighted fusion feature map is subjected to feature enhancement by using a preset spatial attention network, to obtain a spatial attention enhanced feature map; The spatial attention enhanced feature map, the channel attention enhanced feature map and the rough fusion feature map are connected, to obtain an interactive fusion feature map.

7. The method according to any one of claims 1 to 6, characterized in that, The local fusion feature map and the global fusion feature map are subjected to color enhancement respectively, to obtain a local enhanced feature map and a global enhanced feature map, including: The local fusion feature map is subjected to feature division, to obtain a local reserved feature map and a local to-be-transformed feature map; The local to-be-transformed feature map is subjected to wavelet transform, to obtain a local transformed feature map; The local transformed feature map, the local reserved feature map and the initial feature map are connected, to obtain a local enhanced feature map; The global fusion feature map is subjected to feature division, to obtain a global reserved feature map and a global to-be-transformed feature map; The global to-be-transformed feature map is subjected to wavelet transform, to obtain a global transformed feature map; The global transformed feature map, the global reserved feature map and the initial feature map are connected, to obtain a global enhanced feature map.

8. A waterway image defogging system, characterized by, including: An acquisition module is configured to acquire a foggy image of a target channel, and perform feature extraction on the foggy image to obtain an initial feature map; A feature processing module is configured to perform feature processing on the initial feature map to obtain an initial local feature map and an initial global feature map; An interactive fusion module is configured to perform interactive fusion on the initial local feature map and the initial global feature map by using a preset gating mechanism, to obtain an interactive fusion feature map; perform residual connection on the interactive fusion feature map and the initial local feature map, to obtain a local fusion feature map; and perform residual connection on the interactive fusion feature map and the initial global feature map, to obtain a global fusion feature map; A color enhancement module is configured to perform color enhancement on the local fusion feature map and the global fusion feature map respectively, to obtain a local enhanced feature map and a global enhanced feature map; A fusion reconstruction module is configured to perform fusion reconstruction based on the local enhanced feature map and the global enhanced feature map, to obtain a target de-fogging image of the target channel.

9. A computing device comprising a memory, a processor, and a program stored on the memory and running on the processor, wherein, The processor executes the program to implement the steps of the channel image de-fogging method according to any one of claims 1 to 7.

Images