Channel image defogging method and system, and computing device

Abstract

The invention discloses a channel image defogging method and system and computing equipment, and the method comprises the steps: obtaining a foggy day image of a target channel, carrying out the feature extraction of the foggy day image, and obtaining an initial feature map; performing feature processing on the initial feature map to obtain an initial local feature map and an initial global feature map; performing interactive fusion based on the initial local feature map and the initial global feature map to obtain a local fusion feature map and a global fusion feature map; performing color enhancement on the local fusion feature map and the global fusion feature map to obtain a local enhanced feature map and a global enhanced feature map; and performing fusion reconstruction based on the local enhanced feature map and the global enhanced feature map to obtain a target defogged image of the target channel. The problem that the accuracy of the obtained defogged image is poor due to the fact that the defogging effect of an existing method for the channel image is unstable is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a channel image defogging method and system and a computing device. BACKGROUND

[0002] In recent years, image defogging technology has been widely concerned in the field of computer vision, especially in intelligent transportation, autonomous driving and environmental perception applications. However, most existing defogging methods focus on land scenes, especially urban road images, and there is relatively little research on channel images. In the channel environment, there are often complex lighting and reflection conditions. When using existing land image defogging methods to defog channel images under complex lighting conditions, the defogging effect is unstable, resulting in poor accuracy of the obtained defogging images. SUMMARY

[0003] In order to overcome the problem that the existing method has unstable defogging effect for channel images under complex lighting conditions, resulting in poor accuracy of the obtained defogging images, the present application provides a channel image defogging method, system and computing device.

[0004] In a first aspect, in order to solve the above technical problems, the present application provides a channel image defogging method, comprising: obtaining a foggy image of a target channel, performing feature extraction on the foggy image to obtain an initial feature map; performing feature processing on the initial feature map to obtain an initial local feature map and an initial global feature map; performing interactive fusion based on the initial local feature map and the initial global feature map to obtain a local fusion feature map and a global fusion feature map; performing color enhancement on the local fusion feature map and the global fusion feature map respectively to obtain a local enhanced feature map and a global enhanced feature map; performing fusion reconstruction based on the local enhanced feature map and the global enhanced feature map to obtain a target defogging image of the target channel.

[0005] Further, performing feature processing on the initial feature map to obtain an initial local feature map and an initial global feature map, comprising: performing local fusion processing on the initial feature map to obtain an intermediate fusion feature map; performing residual connection on the initial feature map and the intermediate fusion feature map to obtain the initial local feature map.

[0006] Further, performing local fusion processing on the initial feature map to obtain an intermediate fusion feature map, comprising: performing multi-scale convolution processing on the initial feature map to obtain a multi-scale detail feature map, the multi-scale detail feature map comprising a plurality of different scale shallow feature maps; The multi-scale detail feature maps are fused by using preset star operations to obtain a plurality of initial fusion feature maps; The initial fusion feature maps are subjected to semantic extraction by using preset depth separable convolution and preset full connection layers to obtain semantic feature maps; The plurality of semantic feature maps are fused by using star operations to obtain intermediate fusion feature maps.

[0007] Further, the initial feature maps are subjected to feature processing to obtain initial local feature maps and initial global feature maps, including: The initial feature maps are subjected to convolution processing to obtain local context feature maps; The foggy image is processed by using a preset self-attention network to obtain a global attention feature map; The global attention feature map and the local context feature map are fused to obtain an initial global feature map.

[0008] Further, the self-attention network includes a plurality of residual connection global feature extraction networks; the foggy image is processed by using a preset self-attention network to obtain a global attention feature map, including: The foggy image is windowed to obtain a plurality of window images; The plurality of window images are processed in parallel by using a plurality of global feature extraction networks to obtain a plurality of window global feature maps; wherein the global feature extraction network includes a multi-head self-attention mechanism and a multi-layer perception machine; The plurality of window global feature maps are residual connected to obtain a global attention feature map.

[0009] Further, the initial local feature maps and the initial global feature maps are interactively fused to obtain local fusion feature maps and global fusion feature maps, including: The initial local feature maps and the initial global feature maps are interactively fused by using a preset gating mechanism to obtain an interactive fusion feature map; The interactive fusion feature map and the initial local feature map are residual connected to obtain a local fusion feature map; The interactive fusion feature map and the initial global feature map are residual connected to obtain a global fusion feature map.

[0010] Further, the initial local feature maps and the initial global feature maps are interactively fused by using a preset gating mechanism to obtain an interactive fusion feature map, including: The initial local feature maps and the initial global feature maps are fused to obtain a rough fusion feature map; The initial local feature maps and the initial global feature maps are weighted fused based on the rough fusion feature map by using a preset gating mechanism to obtain a weighted fusion feature map; The weighted fusion feature map is subjected to feature enhancement by using a preset channel attention network, to obtain a channel attention enhanced feature map. The weighted fusion feature map is subjected to feature enhancement by using a preset spatial attention network, to obtain a spatial attention enhanced feature map. The spatial attention enhanced feature map, the channel attention enhanced feature map and the rough fusion feature map are connected to obtain an interactive fusion feature map.

[0011] Further, the local fusion feature map and the global fusion feature map are subjected to color enhancement respectively to obtain a local enhanced feature map and a global enhanced feature map, including: The local fusion feature map is subjected to feature division to obtain a local reserved feature map and a local to-be-transformed feature map. The local to-be-transformed feature map is subjected to wavelet transform to obtain a local transformed feature map. The local transformed feature map, the local reserved feature map and the initial feature map are connected to obtain the local enhanced feature map. The global fusion feature map is subjected to feature division to obtain a global reserved feature map and a global to-be-transformed feature map. The global to-be-transformed feature map is subjected to wavelet transform to obtain a global transformed feature map. The global transformed feature map, the global reserved feature map and the initial feature map are connected to obtain the global enhanced feature map.

[0012] In a second aspect, the present application further provides a channel image defogging system, including: An acquisition module is configured to acquire a foggy image of a target channel, and extract features from the foggy image to obtain an initial feature map. A feature processing module is configured to process the initial feature map to obtain an initial local feature map and an initial global feature map. An interactive fusion module is configured to perform interactive fusion based on the initial local feature map and the initial global feature map to obtain a local fusion feature map and a global fusion feature map. A color enhancement module is configured to perform color enhancement on the local fusion feature map and the global fusion feature map respectively to obtain a local enhanced feature map and a global enhanced feature map. A fusion reconstruction module is configured to perform fusion reconstruction based on the local enhanced feature map and the global enhanced feature map to obtain a target defogging image of the target channel.

[0013] In a third aspect, the present application further provides a computing device including a memory, a processor and a program stored in the memory and running on the processor, wherein the processor implements the steps of the channel image defogging method as described above when executing the program.

[0014] In a fourth aspect, the present application also provides a computer-readable storage medium, which stores instructions. When the instructions are executed on a terminal device, the terminal device executes the steps of a waterway image defogging method.

[0015] The beneficial effects of the present application are as follows: first, feature extraction is performed on the foggy image of the target waterway, and then feature processing is performed to obtain an initial local feature map containing detail information and an initial global feature map containing structural features, and interactive fusion is performed based on the initial local feature map and the initial global feature map to obtain a local fusion feature map and a global fusion feature map. Secondly, the local fusion feature map and the global enhancement feature map are color enhanced respectively to fully extract the high-frequency detail information and structural features of the long-distance area contained in the foggy image, and the local enhancement feature map and the global enhancement feature map obtained based on the color enhancement are fused and reconstructed to achieve collaborative modeling of the local details and global structural information of the foggy image to adapt to the foggy waterway under complex lighting environment, thereby significantly improving the visual quality and robustness of image defogging in the waterway scene, improving image clarity and realism, and thus improving the accuracy of the obtained target defogging image. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of a method for defogging a waterway image according to an exemplary embodiment of the present application; Figure 2 This is a diagram of a local feature processing network structure in an exemplary embodiment of the present application; Figure 3 A global feature extraction network structure diagram in an exemplary embodiment of the present application; Figure 4 This is a diagram of an interactive fusion processing network structure in an exemplary embodiment of the present application; Figure 5 This is a diagram of a color enhancement processing network structure in an exemplary embodiment of the present application; Figure 6 This is a flowchart of applying the provided channel image defogging method in an exemplary embodiment of the present application; Figure 7 This is a comparison chart of the defogging effect of the channel image defogging method of this application and other existing methods; Figure 8 The figure is a schematic structural diagram of a waterway image defogging system according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0017] The following examples are provided to further explain and supplement the present application and do not constitute any limitation to the present application.

[0018] The existing mainstream defogging methods mainly include a traditional image enhancement algorithm, a physical model method based on an atmospheric scattering model, and an end-to-end image restoration method based on deep learning.

[0019] The traditional image enhancement method, such as histogram equalization, Retinex algorithm (an image processing technology based on a biological vision system), and the like, is simple to implement, but is difficult to adapt to complex lighting and reflection conditions, and the defogging effect is unstable. The physical model-based method such as dark channel prior (DCP) or color line prior, and the like, often fails when processing water area images due to the existence of strong reflection and low texture regions, and is prone to problems such as color deviation and structure loss. The deep learning method achieves good performance, but most model designs do not consider the particularity of the water scene, such as large-area water surface, complex light reflection, and long-distance fog distribution.

[0020] To solve the above problems, the embodiments of the present application provide a channel image defogging method, system and computing device, which will be described in detail below.

[0021] The channel image defogging method provided by the embodiments of the present application can be specifically executed by a server. It should be noted that the server can be an independent server, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms, which are not limited herein.

[0022] Please refer to Figure 1 , Figure 1 A channel image defogging method shown by an exemplary embodiment of the present application, as shown in Figure 1 , the present application provides a channel image defogging method, comprising: S11, obtaining a foggy image of a target channel, performing feature extraction on the foggy image to obtain an initial feature map; S12, performing feature processing on the initial feature map to obtain an initial local feature map and an initial global feature map; S13, performing interactive fusion based on the initial local feature map and the initial global feature map to obtain a local fusion feature map and a global fusion feature map; S14, performing color enhancement on the local fusion feature map and the global fusion feature map respectively to obtain a local enhanced feature map and a global enhanced feature map; S15, performing fusion reconstruction based on the local enhanced feature map and the global enhanced feature map to obtain a target defogging image of the target channel.

[0023] The channel image defogging method of the embodiment provided in the application first performs feature extraction on the foggy image of the target channel, then performs feature processing to obtain an initial local feature map containing detailed information and an initial global feature map containing structural features, and performs interactive fusion based on the initial local feature map and the initial global feature map to obtain a local fusion feature map and a global fusion feature map. Secondly, the local fusion feature map and the global enhancement feature map are respectively subjected to color enhancement to fully extract the high-frequency detailed information and the structural features of the remote area contained in the foggy image, and the local enhancement feature map and the global enhancement feature map obtained based on color enhancement are fused and reconstructed to realize collaborative modeling of the local details and the global structural information of the foggy image, so as to adapt to the foggy channel in a complex lighting environment, thereby significantly improving the visual quality and robustness of image defogging in the channel scene, improving the image clarity and realism, and further improving the accuracy of the obtained target defogging image. The method of the application is particularly suitable for water image clarification tasks in ship navigation.

[0024] Optionally, the initial feature map is subjected to feature processing to obtain an initial local feature map and an initial global feature map, including: The initial feature map is subjected to local fusion processing to obtain an intermediate fusion feature map. The initial feature map and the intermediate fusion feature map are subjected to residual connection to obtain an initial local feature map.

[0025] In the embodiment provided in the application, the initial feature map is subjected to local fusion processing to extract detailed information in the initial feature map and obtain an intermediate fusion feature map, and the initial feature map and the intermediate fusion feature map are subjected to residual connection to obtain an initial local feature map, so as to strengthen the detailed information in the initial local feature map, thereby retaining more detailed information in the subsequent defogging processing process, and further improving the accuracy of the finally obtained target defogging image.

[0026] Optionally, the initial feature map is subjected to local fusion processing to obtain an intermediate fusion feature map, including: The initial feature map is subjected to multi-scale convolution processing to obtain a multi-scale detailed feature map, and the multi-scale detailed feature map includes a plurality of shallow feature maps of different scales. The multi-scale detailed feature map is subjected to feature fusion by using a preset star operation to obtain a plurality of initial fusion feature maps. The initial fusion feature map is subjected to semantic extraction by using a preset depth separable convolution and a preset full connection layer to obtain a semantic feature map. The plurality of semantic feature maps are subjected to feature fusion by using a star operation to obtain an intermediate fusion feature map.

[0027] In the embodiment provided in the present application, the initial feature map is subjected to multi-scale convolution processing, which can extract edge, texture and other high-frequency detail information in the initial feature map from multiple scales using convolution kernels of different receptive fields, so as to alleviate the problem of detail loss caused by complex lighting environments such as light scattering and water surface reflection in foggy images. At the same time, the multi-scale detail feature maps are subjected to feature fusion using a preset star operation (Star Operation) to obtain multiple initial fusion feature maps, and the initial fusion feature maps are subjected to semantic extraction using a preset depth separable convolution and a preset fully connected layer to obtain semantic feature maps. The multiple semantic feature maps are subjected to feature fusion again using the star operation to obtain an intermediate fusion feature map, realizing new splicing and fusion of different scales, which can retain original information and strengthen the stability of feature learning, so as to enhance the recovery effect of the water surface boundary and the object contour in the image in the subsequent dehazing process, retain more high-frequency detail information, and thus improve the accuracy of the target dehazing image obtained finally.

[0028] In an example embodiment provided in the present application, first, the initial feature map is subjected to multi-scale convolution processing to obtain a multi-scale detail feature map, and the specific steps are as follows: The foggy image is subjected to convolution operation using multiple convolution kernels of different sizes to obtain multiple shallow layer feature maps; wherein the number of shallow layer feature maps can be 4, and the multiple convolution kernels of different sizes include 1x1 convolution kernel, 3x3 convolution kernel, 5x5 convolution kernel and 7x7 convolution kernel. The multi-scale detail feature map is formed based on the multiple shallow layer feature maps.

[0029] Secondly, the multi-scale detail feature map is subjected to feature fusion using a preset star operation to obtain multiple initial fusion feature maps, and the specific steps can be as follows: The four shallow layer feature maps are fused two by two using the preset star operation to obtain two groups of initial fusion feature maps.

[0030] The star operation can map the input feature to a very high-dimensional nonlinear feature space, thereby significantly enhancing the expression ability of the feature. The star operation can be expressed as: wherein, represents the first weight matrix, represents the second weight matrix, represents the input feature tensor, represents the number of channels, represents the channel index, The coefficients of each term are denoted as, i.e. The above formula can be expanded as different sub-terms. Therefore, in a dimensional space, the star operation with higher computational efficiency can implicitly construct a dimensional feature space without increasing the single-layer computational overhead, significantly improving the feature dimension.

[0031] In this way, by introducing the star operation, the feature mapping is projected to a high-dimensional space without increasing the network depth, further enhancing the ability to capture visual details such as color, texture, and shape in the image during the defogging process.

[0032] Please refer to Figure 2 , Figure 2 is a local feature processing network structure in an exemplary embodiment of the present application, as shown in Figure 2 , the local feature processing network structure includes four parallel multi-scale convolutional networks, a two-stage feature fusion network, a deep feature extraction network, and a final output adjustment network. Among them, the types of convolutional layers in each network are shown in the dashed box in Figure 2 .

[0033] First, the initial feature map is convolved by four different sizes of convolution kernels (1x1, 3x3, 5x5, 7x7) to obtain multi-scale shallow feature maps. These feature maps retain rich spatial structure information, which is helpful for subsequent deep modeling.

[0034] In the first stage of the feature fusion network, the star operation is used to fuse the four groups of shallow feature maps in pairs to obtain two groups of initial fusion feature maps. Then, each group of fusion features is further extracted through the deep separable convolution and fully connected layer included in the deep feature extraction network to obtain semantic feature maps, thereby enhancing the feature representation ability.

[0035] In the second stage of the feature fusion network, the star operation is used again to fuse the two groups of semantic feature maps to obtain intermediate fusion feature maps.

[0036] The intermediate fusion feature maps after fusion and the initial feature maps of the original input are added through residual connection to obtain initial local feature maps to retain the original information and strengthen the stability of feature learning. In order to ensure that the size of the output feature is consistent with the input, the output adjustment network introduces a 3x3 convolutional layer for dimension matching to ensure that the initial local feature map output can be seamlessly connected to the subsequent network structure.

[0037] Optionally, the initial feature map is processed to obtain an initial local feature map and an initial global feature map, including: perform convolution processing on the initial feature map to obtain a local context feature map; perform processing on the foggy image by using a preset self-attention network to obtain a global attention feature map; fuse the global attention feature map and the local context feature map to obtain an initial global feature map.

[0038] In the embodiment provided in the present application, the initial feature map is subjected to convolution processing to obtain a local context feature map, and the foggy image is processed by using a preset self-attention network to obtain a global attention feature map. The global attention feature map and the local context feature map are fused to obtain an initial global feature map. In this way, since the self-attention network is good at modeling the spatial dependence of distant regions in the image and is particularly suitable for water navigation scenes containing vast sky and water surface backgrounds, the long-range target information in the foggy image can be strengthened by the self-attention network, the loss of long-range target information caused by fog and haze is avoided, the initial global feature map obtained not only effectively reconstructs the water surface structure of the target channel, but also suppresses the image artifacts caused by complex reflections in the target channel, so that more overall structural information is retained in the subsequent dehazing process, and the accuracy of the target dehazed image obtained finally can be improved. The local context feature map can be captured by an attention mechanism.

[0039] Optionally, the self-attention network comprises a plurality of residual connection global feature extraction networks; the processing on the foggy image by using the preset self-attention network to obtain the global attention feature map comprises: perform window division on the foggy image to obtain a plurality of window images; perform parallel processing on the plurality of window images by using a plurality of global feature extraction networks to obtain a plurality of window global feature maps; wherein the global feature extraction network comprises a multi-head self-attention mechanism and a multi-layer perception machine; perform residual connection on the plurality of window global feature maps to obtain the global attention feature map.

[0040] In the embodiment provided in the present application, in the self-attention network, the foggy image is subjected to window division to obtain a plurality of window images, and the plurality of window images are subjected to parallel processing by using a plurality of residual connection global feature extraction networks to obtain a plurality of window global feature maps, and the plurality of window global feature maps are subjected to residual connection to obtain the global attention feature map. In this way, by dividing a large foggy image into a plurality of small window images and performing parallel processing on the plurality of small window images, the processing difficulty of the foggy image can be reduced while the processing speed of the foggy image is improved, and thus the efficiency of the dehazing processing on the foggy image of the target channel can be improved.

[0041] In this embodiment, the self-attention network can be represented as: wherein Q is Query, representing a query, K is Key, representing a key, V is Value, representing a value, d represents the dimension of the feature, and B represents the batch size of the sample, represents an original foggy image without window division, represents a depthwise separable convolution (DWConv) or a Conv-Relu-Conv convolution block.

[0042] The self-attention network is used to model the global relationship between distant regions in the image. The self-attention network includes a plurality of residual connection global feature extraction networks. The structure of the global feature extraction network is as shown in Figure 3 , which includes a multi-head self-attention mechanism (W-MHSA) and a multi-layer perceptron (MLP). Both of them use a layer normalization layer (LayerNorm) as a preprocessing layer and are connected through a residual connection to stabilize the training process and enhance the feature expression capability. In this way, through the stacking of multiple global feature extraction networks, large-scale structural information in the foggy image can be fully extracted, further improving the accuracy and consistency of image restoration.

[0043] The traditional Swin Transformer (a kind of visual model) uses a windowing and cyclic shift strategy to achieve efficient region modeling, but due to the mask mechanism limitation of the image edge region, it may cause the effective receptive field of the window to shrink, and thus weaken the expression ability of the edge features. Therefore, the self-attention network (a deep learning model architecture based on multi-head self-attention mechanism) method using a reflection padding strategy is used to ensure the window size consistency in the entire image range, effectively preserve the image edge information, and improve the dehazing precision. To enhance the information interaction between windows, a parallel convolution branch is added in the global modeling branch to supplement the deficiencies of the traditional self-attention module in local modeling. The convolution branch performs convolution processing on the input features before window division, extracts local context information across windows, and fuses with the output of the self-attention network, thereby realizing joint modeling of global-local information.

[0044] Optionally, the initial local feature map and the initial global feature map are interactively fused to obtain a local fusion feature map and a global fusion feature map, including: The initial local feature map and the initial global feature map are interactively fused by using a preset gating mechanism to obtain an interactive fusion feature map; The interactive fusion feature map is connected in residual connection with the initial local feature map to obtain a local fusion feature map; ​The interactive fusion feature map is connected with the initial global feature map in a residual manner to obtain a global fusion feature map.

[0045] In the embodiment provided in the application, the initial local feature map and the initial global feature map are interactively fused by using a preset gating mechanism to obtain an interactive fusion feature map, and a local fusion feature map formed by connecting the interactive fusion feature map with the initial local feature map in a residual manner and a global fusion feature map formed by connecting the interactive fusion feature map with the initial global feature map in a residual manner are obtained, so that the amount of feature information in the local fusion feature map and the global fusion feature map is further improved, more detailed information is retained in the subsequent haze removal process, and thus the accuracy of the target haze-removed image obtained finally can be improved.

[0046] Optionally, the initial local feature map and the initial global feature map are interactively fused by using a preset gating mechanism to obtain an interactive fusion feature map, including: The initial local feature map and the initial global feature map are fused to obtain a rough fusion feature map; The calculation formula of the rough fusion feature map is as follows: ; wherein, denotes the rough fusion feature map, denotes the initial local feature map, denotes the initial global feature map, denotes a concatenation operation in the channel dimension; The initial local feature map and the initial global feature map are weightedly fused based on the rough fusion feature map by using a preset gating mechanism to obtain a weighted fusion feature map; The calculation formula of the weighted fusion feature map is as follows: ; ; wherein, denotes the weighted fusion feature map, denotes a gating value, denotes the initial local feature map, denotes the initial global feature map, and Sigmoid denotes an activation function, denotes a depth separable convolution of the initial local feature map, denotes a depth separable dilated convolution with an inflation rate of 2, ; The weighted fusion feature map is feature-enhanced by using a preset channel attention network to obtain a channel attention-enhanced feature map; The calculation formula of the channel attention-enhanced feature map is as follows: ; wherein, represents a channel attention enhanced feature map, represents a convolution operation with a kernel size of , represents a global average pooling along the channel dimension, represents a ReLU activation function, represents a weighted fusion feature map; performing feature enhancement on the weighted fusion feature map by using a preset spatial attention network to obtain a spatial attention enhanced feature map; The calculation formula of the spatial attention enhanced feature map is as follows: ; wherein, represents a spatial attention enhanced feature map, represents a convolution operation with a kernel size of , represents a global maximum pooling operation, represents a weighted fusion feature map, represents a global average pooling along the spatial dimension; connecting the spatial attention enhanced feature map, the channel attention enhanced feature map and the rough fusion feature map to obtain an interactive fusion feature map; The calculation formula of the interactive fusion feature map is as follows: ; wherein, represents an interactive fusion feature map, represents a convolution operation with a kernel size of , represents a channel attention enhanced feature map, represents a spatial attention enhanced feature map, represents a rough fusion feature map.

[0047] In the embodiment provided in the present application, firstly, based on the coarse fusion feature map fused from the initial local feature map and the initial global feature map, the initial local feature map and the initial global feature map are weighted fused, and in the process of weighted fusion, the preset gating mechanism is used to control the fusion weight, so that the network can adaptively select the optimal fusion strategy to obtain the optimal weighted fusion feature map. Secondly, the preset channel attention network and the spatial attention network are used to respectively perform feature enhancement on the weighted fusion feature map, so as to enhance the feature representation capability of the key channel and the spatial structure information in the weighted fusion feature map, and the channel attention enhanced feature map and the spatial attention enhanced feature map obtained by feature enhancement, and the coarse fusion feature map are connected to obtain the interactive fusion feature map, so that the interactive fusion feature map can contain sufficient feature representation capability and spatial structure information, thereby the feature information amount in the target defogging image obtained by subsequent reconstruction can be improved, and then the accuracy of the target defogging image can be improved.

[0048] In the embodiment, the spatial attention enhanced feature map, the channel attention enhanced feature map and the coarse fusion feature map are connected to obtain the interactive fusion feature map, and the specific steps are as follows: The channel attention enhanced feature is spliced with the spatial attention enhanced feature , and then residual connection is performed with the coarse fusion feature , and then one convolution is used to restore the channel number to obtain the final interactive fusion feature map .

[0049] Please refer to Figure 4 , Figure 4 , which is the interactive fusion processing network structure diagram in the example embodiment of the present application, as shown in Figure 4 , the interactive fusion processing network includes a first feature splicing layer, a gating fusion unit, a channel attention network, a spatial attention network and a second feature splicing layer. The specific processing logic of the interactive fusion processing network is as follows: In the first feature splicing layer, the initial local feature map and the initial global feature map are fused to obtain a coarse fusion feature map; In the gating fusion unit, based on the coarse fusion feature map, the initial local feature map and the initial global feature map are weighted fused by using the preset gating mechanism to obtain a weighted fusion feature map; In the channel attention network, the weighted fusion feature map is feature enhanced to obtain a channel attention enhanced feature map; In the spatial attention network, the weighted fusion feature map is feature enhanced to obtain a spatial attention enhanced feature map; In the second feature concatenation layer, the spatial attention enhanced feature map, the channel attention enhanced feature map and the coarse fusion feature map are connected to output the interactive fusion feature map.

[0050] Optionally, color enhancement is performed on the local fusion feature map and the global fusion feature map respectively to obtain a local enhanced feature map and a global enhanced feature map, including: Perform feature division on the local fusion feature map to obtain a local retained feature map and a local feature map to be transformed; Perform wavelet transform on the local feature map to be transformed to obtain a local transformed feature map; Connect the local transformed feature map, the local retained feature map and the initial feature map to obtain the local enhanced feature map; Perform feature division on the global fusion feature map to obtain a global retained feature map and a global feature map to be transformed; Perform wavelet transform on the global feature map to be transformed to obtain a global transformed feature map; The global transformed feature map, the global retained feature map and the initial feature map are connected to obtain the global enhanced feature map.

[0051] In the embodiment provided by the present application, a local retained feature map and a local feature map to be transformed are obtained by performing feature division on the local fusion feature map, the local feature map to be transformed is subjected to wavelet transform to obtain a local transformed feature map, and a local enhanced feature map is formed by connecting the local transformed feature map, the local retained feature map and the initial feature map. And the global fusion feature map is processed by the same method as above to obtain a global enhanced feature map. In this way, by obtaining high-frequency features related to color and rich detail texture information in the image through wavelet transform and feature connection, the color expression of the local enhanced feature map and the global enhanced feature map can be enhanced, thereby improving the color expression of the target dehazed image obtained by fusion reconstruction based on the local enhanced feature map and the global enhanced feature map, thereby improving the accuracy of the target dehazed image.

[0052] See also Figure 5 , Figure 5 This is a color enhancement processing network structure diagram in an exemplary embodiment of the present application, such as Figure 5 As shown in the figure, the color enhancement processing network includes a split network, a wavelet transform network, a splicing network, a residual connection network, and a convolutional network. The wavelet transform network includes a discrete wavelet transform network, a wavelet domain, an inverse wavelet transform network, a connection network, and a 1×1 convolutional layer. The wavelet domain includes a channel attention network and a spatial attention network. The specific processing logic of the color enhancement processing network is as follows: In the split network, the input feature map (local fusion feature map or global fusion feature map) is divided to obtain the retained feature map (local retained feature map or global retained feature map) and the feature map to be transformed (local feature map to be transformed or global feature map to be transformed); In the discrete wavelet transform network, the feature map to be transformed (the local feature map to be transformed or the global feature map to be transformed) is first transformed into the wavelet domain through discrete wavelet transform to obtain a first transformed feature map (the first local transformed feature map or the first global transformed feature map); In the wavelet domain, the channel attention network and the spatial attention network are used to sequentially weight the first transform feature map after the discrete wavelet transform to extract rich context information and image details, thereby obtaining a second transform feature map (a second local transform feature map or a second global transform feature map). In the inverse wavelet transform network, the second transform feature map is subjected to an inverse wavelet transform to obtain a transform feature map (a local transform feature map or a global transform feature map).

[0053] In the connection network, the transformed feature map (local transformed feature map or global transformed feature map) and the retained feature map (corresponding local retained feature map or global retained feature map) are spliced ​​to obtain a spliced ​​feature map (local spliced ​​feature map or global spliced ​​feature map); Perform a residual connection between the concatenated feature map and the initial feature map of the original input to obtain an enhanced feature map (local enhanced feature map or global enhanced feature map) to slow down information loss and promote stable learning of the network; In the 1×1 convolution layer, the enhanced feature map (local enhanced feature map or global enhanced feature map) is processed and output.

[0054] In this embodiment, the local correlation features and the corresponding steps constitute a color enhancement scheme for the local fusion feature map, and the global correlation features and the corresponding steps constitute a color enhancement scheme for the global fusion feature map.

[0055] See also Figure 6 , Figure 6 This is a flow chart of an exemplary embodiment of the present application, wherein the provided method for defogging a waterway image is applied. Figure 6 As shown in the figure, the process of applying the provided channel image defogging method is as follows: Obtain an initial haze image (a foggy image of the target waterway), perform feature extraction on the foggy image, and obtain an initial feature map; In the global feature processing module, the initial feature map is subjected to feature processing to obtain an initial global feature map; In the local feature processing module, the initial feature map is subjected to feature processing to obtain an initial local feature map; In the feature information interaction module, the initial local feature map and the initial global feature map are interactively fused to obtain a local fusion feature map and a global fusion feature map; In the color enhancement module, the local fusion feature map and the global fusion feature map are respectively subjected to color enhancement to obtain a local enhanced feature map and a global enhanced feature map; Based on the local enhanced feature map and the global enhanced feature map, a fusion feature map is obtained, and a target dehazed image of the target channel is reconstructed based on the fusion feature map.

[0056] Please refer to Figure 7 , Figure 7 The dehazing effect comparison chart of the channel image dehazing method of the present application and other existing methods is shown in FIG. Figure 7 , in which the three images in (a) of FIG. Figure 7 are the foggy images obtained initially, the FFA-Net (Feature Fusion Attention Network) method is used to dehaze the three images in (a) respectively to obtain the three images shown in (b) of FIG. Figure 7 , the Dehazeformer method is used to dehaze the three images in (a) respectively to obtain the three images shown in (c) of FIG. Figure 7 , and the channel image dehazing method of the present application is used to dehaze the three images in (a) respectively to obtain the three images shown in (d) of FIG. Figure 7 . Through comparative analysis, it can be seen that compared with the dehazed images obtained by the existing FFA-Net method and Dehazeformer method, the dehazed images obtained by the channel image dehazing method of the present application have higher clarity and better dehazing effect, so the target dehazed image obtained by the method of the present application has higher accuracy.

[0057] The channel image defogging method of the application effectively restores the near and far targets blocked by haze through the dual-branch structure of local and global feature fusion, and improves the perception ability of ships in haze weather. The local branch introduces a multi-scale convolution module and star operation, which can extract high-frequency details and accurately reconstruct the water surface boundary, object contour and low-texture area. The global branch uses a self-attention network to form an autonomous dependent network, which strengthens the modeling ability of the vast sky and water area in the water image and improves the expression effect of long-distance information, which is convenient for improving the visibility and clarity of the target defogging image obtained subsequently. At the same time, a gating modulation feature fusion module is used to dynamically adjust the weight of local and global information according to the image content, fully exert the complementary advantages of local and global features, and realize the adaptive fusion of local and global features. A color enhancement module based on wavelet transform is introduced to effectively restore the color information degraded by haze in the image and improve the overall visual quality of the image, so as to improve the color restoration effect of the image and improve the accuracy of the target defogging image obtained by finally fusing and reconstructing.

[0058] Please refer to Figure 8 , Figure 8 An example of a channel image defogging system is shown in the application, as shown in Figure 8 The channel image defogging system 800 provided by the application includes: The acquisition module 801 is configured to acquire a foggy image of a target channel, and perform feature extraction on the foggy image to obtain an initial feature map; The feature processing module 802 is configured to perform feature processing on the initial feature map to obtain an initial local feature map and an initial global feature map; The interactive fusion module 803 is configured to perform interactive fusion based on the initial local feature map and the initial global feature map to obtain a local fusion feature map and a global fusion feature map; The color enhancement module 804 is configured to perform color enhancement on the local fusion feature map and the global fusion feature map respectively to obtain a local enhanced feature map and a global enhanced feature map; The fusion reconstruction module 805 is configured to perform fusion reconstruction based on the local enhanced feature map and the global enhanced feature map to obtain a target defogging image of the target channel.

[0059] The channel image defogging system 800 provided in this embodiment of the present application first extracts features from the obtained foggy image of the target channel through the acquisition module 801, then processes the initial feature map obtained through feature extraction through the feature processing module 802 to obtain an initial local feature map containing detailed information and an initial global feature map containing structural features, and performs interactive fusion based on the initial local feature map and the initial global feature map through the interactive fusion module 803 to obtain a local fusion feature map and a global fusion feature map. Secondly, the color enhancement module 804 is used to perform color enhancement on the local fusion feature map and the global enhancement feature map respectively, so as to fully extract the high-frequency detailed information and the structural features of the remote area contained in the foggy image, and the fusion reconstruction module 805 is used to perform fusion reconstruction based on the local enhancement feature map and the global enhancement feature map obtained through color enhancement, so as to realize collaborative modeling of the local details and the global structural information of the foggy image, adapt to the foggy channel in a complex lighting environment, and thus the visual quality and robustness of image defogging in the channel scene can be significantly improved, the image clarity and the sense of reality are improved, and thus the accuracy of the obtained target defogging image can be improved.

[0060] Optionally, the feature processing module 802 is specifically configured to perform: local fusion processing on the initial feature map to obtain an intermediate fusion feature map; residual connection on the initial feature map and the intermediate fusion feature map to obtain the initial local feature map.

[0061] Optionally, the feature processing module 802 is specifically configured to perform: multi-scale convolution processing on the initial feature map to obtain a multi-scale detailed feature map, the multi-scale detailed feature map including a plurality of shallow feature maps of different scales; feature fusion on the multi-scale detailed feature map through a preset star operation to obtain a plurality of initial fusion feature maps; semantic extraction on the initial fusion feature map through a preset depth separable convolution and a preset fully connected layer to obtain a semantic feature map; feature fusion on the plurality of semantic feature maps through the star operation to obtain the intermediate fusion feature map.

[0062] Optionally, the feature processing module 802 is specifically configured to perform: convolution processing on the initial feature map to obtain a local context feature map; processing on the foggy image through a preset self-attention network to obtain a global attention feature map; fusion on the global attention feature map and the local context feature map to obtain the initial global feature map.

[0063] Optionally, the feature processing module 802 is specifically configured to perform: windowing the foggy image to obtain a plurality of window images; performing parallel processing on the plurality of window images by using a plurality of global feature extraction networks to obtain a plurality of window global feature maps; wherein the global feature extraction network comprises a multi-head self-attention mechanism and a multi-layer perceptron; performing residual connection on the plurality of window global feature maps to obtain a global attention feature map.

[0064] Optionally, the interactive fusion module 803 is specifically configured to: performing interactive fusion on the initial local feature map and the initial global feature map by using a preset gating mechanism to obtain an interactive fusion feature map; performing residual connection on the interactive fusion feature map and the initial local feature map to obtain a local fusion feature map; performing residual connection on the interactive fusion feature map and the initial global feature map to obtain a global fusion feature map.

[0065] Optionally, the interactive fusion module 803 is specifically configured to: performing fusion on the initial local feature map and the initial global feature map to obtain a rough fusion feature map; performing weighted fusion on the initial local feature map and the initial global feature map based on the rough fusion feature map by using a preset gating mechanism to obtain a weighted fusion feature map; performing feature enhancement on the weighted fusion feature map by using a preset channel attention network to obtain a channel attention enhanced feature map; performing feature enhancement on the weighted fusion feature map by using a preset spatial attention network to obtain a spatial attention enhanced feature map; performing connection on the spatial attention enhanced feature map, the channel attention enhanced feature map and the rough fusion feature map to obtain an interactive fusion feature map.

[0066] Optionally, the color enhancement module 804 is specifically configured to: performing feature division on the local fusion feature map to obtain a local reserved feature map and a local to-be-transformed feature map; performing wavelet transform on the local to-be-transformed feature map to obtain a local transformed feature map; performing connection on the local transformed feature map, the local reserved feature map and the initial feature map to obtain a local enhanced feature map; performing feature division on the global fusion feature map to obtain a global reserved feature map and a global to-be-transformed feature map; performing wavelet transform on the global to-be-transformed feature map to obtain a global transformed feature map; performing connection on the global transformed feature map, the global reserved feature map and the initial feature map to obtain a global enhanced feature map.

[0067] It should be noted that the channel image defogging system provided by the above-mentioned embodiments and the channel image defogging method provided by the above-mentioned embodiments belong to the same concept, wherein the specific manner in which each module and unit performs operations has been described in detail in the method embodiments, which will not be repeated here. The channel image defogging system provided by the above-mentioned embodiments can be completed by different functional modules according to the above-mentioned functions in actual application, that is, the internal structure of the system is divided into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0068] The computing device provided by the embodiment of the present application includes a memory, a processor, and a program stored in the memory and running on the processor. When the processor executes the program, it implements part or all of the steps of the above-mentioned channel image defogging method.

[0069] Correspondingly, the program is computer software, and the parameters and steps in the above-mentioned computing device of the present application can refer to the parameters and steps in the above-mentioned embodiment of the channel image defogging method, which will not be repeated here.

[0070] The computer readable storage medium in the embodiment of the present application stores instructions, and when the instructions are executed, the steps of the above-mentioned channel image defogging method are executed.

[0071] The computer readable storage medium can be a transitory computer readable storage medium or a non-transitory computer readable storage medium.

[0072] The technical solutions of the embodiments of the present disclosure can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes one or more instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method of the embodiments of the present disclosure. The aforementioned computer readable storage medium can be a non-transitory computer readable storage medium, including: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, etc. Various media that can store program codes, or a transitory computer readable storage medium.

[0073] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Each box in the flowchart or block diagram can represent a module, program segment or part of the code, and the above-mentioned module, program segment or part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. For example, two boxes shown in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart and the combination of boxes in the block diagram or flowchart can be implemented using a dedicated hardware-based system that performs the specified functions or operations, or can be implemented using a combination of dedicated hardware and computer instructions.

[0074] Those skilled in the art will appreciate that the present application may be implemented as a system, method, or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms: either entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software, generally referred to herein as a "module" or "system." Furthermore, in some embodiments, the present application may also be implemented in the form of a computer program product in one or more computer-readable media, the computer-readable media containing computer-readable program code. Computer-readable storage media may be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof.

[0075] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0076] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for defogging a waterway image, characterized in that: include: Acquire a foggy image of the target waterway, perform feature extraction on the foggy image, and obtain an initial feature map; Performing feature processing on the initial feature map to obtain an initial local feature map and an initial global feature map; Perform interactive fusion based on the initial local feature map and the initial global feature map to obtain a local fusion feature map and a global fusion feature map; Performing color enhancement on the local fusion feature map and the global fusion feature map respectively to obtain a local enhanced feature map and a global enhanced feature map; A fusion reconstruction is performed based on the local enhancement feature map and the global enhancement feature map to obtain a target defogging image of the target channel.

2. The method according to claim 1, characterized in that The step of performing feature processing on the initial feature map to obtain an initial local feature map and an initial global feature map includes: Performing local fusion processing on the initial feature map to obtain an intermediate fusion feature map; Perform a residual connection on the initial feature map and the intermediate fusion feature map to obtain an initial local feature map.

3. The method according to claim 2, characterized in that The local fusion processing of the initial feature map to obtain an intermediate fusion feature map includes: Performing multi-scale convolution processing on the initial feature map to obtain a multi-scale detail feature map, wherein the multi-scale detail feature map includes a plurality of shallow feature maps of different scales; Using a preset star operation, the multi-scale detail feature map is subjected to feature fusion to obtain a plurality of initial fused feature maps; Using a preset depth-wise separable convolution and a preset fully connected layer, semantic extraction is performed on the initial fused feature map to obtain a semantic feature map; The star operation is used to perform feature fusion on the plurality of semantic feature maps to obtain an intermediate fused feature map.

4. The method according to claim 1, wherein The step of performing feature processing on the initial feature map to obtain an initial local feature map and an initial global feature map includes: Performing convolution processing on the initial feature map to obtain a local context feature map; Processing the foggy image using a preset self-attention network to obtain a global attention feature map; The global attention feature map and the local context feature map are fused to obtain an initial global feature map.

5. The method according to claim 4, characterized in that The self-attention network includes a plurality of residual-connected global feature extraction networks; the foggy image is processed using the preset self-attention network to obtain a global attention feature map, including: Performing window division on the foggy image to obtain a plurality of window images; Using multiple global feature extraction networks to process multiple window images in parallel to obtain multiple window global feature maps; wherein the global feature extraction network includes a multi-head self-attention mechanism and a multi-layer perceptron; Perform residual connection on the multiple window global feature maps to obtain a global attention feature map.

6. The method according to claim 1, characterized in that The interactive fusion based on the initial local feature map and the initial global feature map to obtain a local fusion feature map and a global fusion feature map includes: Using a preset gating mechanism, interactively fusing the initial local feature map and the initial global feature map to obtain an interactive fused feature map; Performing a residual connection between the interactive fusion feature map and the initial local feature map to obtain a local fusion feature map; Perform a residual connection between the interactive fusion feature map and the initial global feature map to obtain a global fusion feature map.

7. The method according to claim 6, characterized in that The interactive fusion of the initial local feature map and the initial global feature map by using a preset gating mechanism to obtain an interactive fusion feature map includes: Fusing the initial local feature map and the initial global feature map to obtain a coarse fused feature map; Using a preset gating mechanism, based on the coarse fusion feature map, the initial local feature map and the initial global feature map are weightedly fused to obtain a weighted fusion feature map; Using a preset channel attention network to perform feature enhancement on the weighted fusion feature map to obtain a channel attention enhanced feature map; Using a preset spatial attention network to perform feature enhancement on the weighted fusion feature map to obtain a spatial attention enhanced feature map; The spatial attention enhanced feature map, the channel attention enhanced feature map and the coarse fusion feature map are connected to obtain an interactive fusion feature map.

8. The method according to any one of claims 1 to 7, characterized in that The color enhancement of the local fusion feature map and the global fusion feature map is performed respectively to obtain a local enhanced feature map and a global enhanced feature map, including: Performing feature division on the local fusion feature map to obtain a local retained feature map and a local feature map to be transformed; Performing wavelet transform on the local feature map to be transformed to obtain a local transformed feature map; Connecting the local transformed feature map, the local retained feature map and the initial feature map to obtain a local enhanced feature map; Performing feature division on the global fusion feature map to obtain a global retained feature map and a global feature map to be transformed; Performing wavelet transform on the global feature map to be transformed to obtain a global transformed feature map; The global transformed feature map, the global retained feature map and the initial feature map are connected to obtain a global enhanced feature map.

9. A channel image defogging system, characterized in that: include: an acquisition module, configured to acquire a foggy image of the target waterway, perform feature extraction on the foggy image, and obtain an initial feature map; A feature processing module, configured to perform feature processing on the initial feature map to obtain an initial local feature map and an initial global feature map; An interactive fusion module, configured to perform interactive fusion based on the initial local feature map and the initial global feature map to obtain a local fused feature map and a global fused feature map; A color enhancement module is used to perform color enhancement on the local fusion feature map and the global fusion feature map respectively to obtain a local enhanced feature map and a global enhanced feature map; The fusion reconstruction module is used to perform fusion reconstruction based on the local enhancement feature map and the global enhancement feature map to obtain a target defogging image of the target channel.

10. A computing device comprising a memory, a processor, and a program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the steps of the waterway image defogging method according to any one of claims 1 to 8 are implemented.

Images