Underwater image enhancement method driven by internal and external feature learning

Abstract

The invention relates to an underwater image enhancement method driven by internal and external feature learning. The method comprises the following steps: acquiring an underwater image to be detected; generating a label image according to the underwater image; internal representation learning is carried out on the underwater image, and differential enhancement of different quality areas in the underwater image is realized; external representation learning is carried out on the underwater image after internal representation learning processing, external information and internal information are extracted and fused, and the enhancement effect of the underwater image is improved; minimizing the pixel difference between the enhanced underwater image and the label image by using a loss function, and improving the subjective perception quality of the underwater image; and outputting the processed underwater image. Compared with the prior art, the method has the advantages that rich internal and external information of the underwater image is fully mined and utilized, and more efficient underwater image enhancement is realized.

Description

Technical Field

[0001] The present invention relates to the fields of marine engineering and computer vision, and in particular to an underwater image enhancement method driven by internal and external feature learning. Background Art

[0002] High-quality underwater images are essential for obtaining valuable marine information. However, there are many complex factors in the underwater environment, such as light absorption, scattering, and uneven illumination. The interaction between them leads to the quality degradation of underwater images, such as color cast, blur, low contrast, and uneven brightness. This quality degradation not only has an adverse effect on the observation of underwater images by the human eye, but also brings challenges to the application of underwater machine vision, becoming a major obstacle to underwater exploration, marine research, and applications. Therefore, how to effectively solve the quality problem of underwater images and improve image clarity has become an urgent problem to be solved. Unlike the diverse observation methods on land, due to the limitations of the marine environment, researchers cannot directly go deep underwater for observation or engineering operations. Therefore, researchers usually rely on autonomous underwater vehicles and remotely controlled vehicles to explore underwater space and take underwater images along a predetermined observation path. However, the collected underwater images often have quality degradation problems such as color distortion, low contrast, and blurred details, which greatly limits the subsequent visual perception analysis and exploration of the underwater world. In order to solve this problem, more and more deep learning methods designed specifically for the characteristics of underwater images have been proposed. These methods have made great efforts in designing new sample generation techniques, introducing unique underwater prior knowledge, and exploring more effective learning strategies and network architectures. Although significant progress has been made in this field, there are still two problems that limit the performance of existing methods.

[0003] 1. There are significant regional quality differences within underwater images. Compared with terrestrial imaging, the underwater optical imaging process is more complicated. Land images usually assume that light is uniformly absorbed in the atmosphere and the influence of light scattering effects is negligible. In contrast, the absorption and scattering effects of light in underwater environments are very obvious, and change significantly with changes in water bodies and scenes. These effects cause underwater images to degrade to varying degrees, which is difficult to effectively handle through global correction methods. The unique changing characteristics of underwater light lead to significant regional quality differences within underwater images. Ideally, the enhancement network should give different degrees of response to regions with different quality levels, that is, to achieve differentiated enhancement. However, existing enhancement algorithms ignore this feature and usually use an "average" approach to process the entire image, resulting in very limited enhancement effects.

[0004] 2. A large amount of rich external information can be used to enhance underwater images. Due to the uniqueness of the acquisition method, underwater image acquisition tools (AUVs or ROVs) usually collect multiple images in the same or similar scenes.Figure 1 The underwater images collected by AUVs in practical applications are shown. Figure 2 The UIEB test dataset is commonly used in existing scientific research. By comparing the two, it can be observed that the UIEB dataset fails to fully reflect the characteristics of real underwater test data in practical applications. Figure 1 and 2 As can be seen from the figure, there is a high degree of correlation between the underwater images to be enhanced in practical applications. They are usually related to each other in different ways, such as the same color tone, common objects, and similar scenes. These related underwater images can provide each other with rich complementary information, further improving the enhancement effect. Specifically, multiple underwater images contain two types of complementary information: on the one hand, multiple images contain the same content but with different qualities. Figure 1 As shown in the red box, objects taken at close range may present clearer details than those in long-distance viewing angles, so high-quality content can be used to guide the enhancement of low-quality content. On the other hand, even if all images are of low quality, similar objects may lose or retain different information in different images, such as Figure 1 As shown in the yellow box. This information can complement and combine with each other to build richer features and achieve more complete and accurate content reconstruction. However, existing algorithms only focus on single image information when enhancing underwater images, ignoring the rich external information that can be used, and their performance still has room for improvement. Summary of the invention

[0005] The purpose of the present invention is to provide an underwater image enhancement method driven by internal and external feature learning, which includes internal representation learning and external representation learning, aiming to fully explore and utilize the rich internal and external information of underwater images to achieve more efficient underwater image enhancement.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] An underwater image enhancement method (UIERL) driven by internal and external feature learning, the method comprising:

[0008] Acquire the underwater image to be tested;

[0009] Generate a labeled image based on an underwater image;

[0010] Learn the internal representation of underwater images to achieve differentiated enhancement of different quality areas inside underwater images;

[0011] Perform external representation learning on underwater images processed by internal representation learning, extract and fuse external information and internal information, and improve the enhancement effect of underwater images;

[0012] The loss function is used to minimize the pixel difference between the enhanced underwater image and the labeled image and improve the subjective perception quality of the underwater image;

[0013] Output the processed underwater image.

[0014] Furthermore, the label image is obtained by processing with image editing software and corresponds one-to-one with the underwater image.

[0015] Furthermore, the internal representation learning includes:

[0016] Perform depth estimation on the underwater image to be tested to obtain a scene depth map;

[0017] Performing regional segmentation on the scene depth map, dividing the scene depth map into three or more non-overlapping regions, where images in the same region have similar quality degradation patterns;

[0018] Extract features from the information of each divided area to obtain initial area features;

[0019] The initial regional features are processed through residual connections to further extract regional content features;

[0020] Use graph convolutional networks to process initial regional features, enhance the consistency of regional features in the same region and the differences of regional features in different regions;

[0021] Through pixel-by-pixel addition, the regional content features and regional features of each region are integrated into the global features to achieve differentiated enhancement of different quality areas inside the underwater image.

[0022] Furthermore, when the graph convolutional network is used to process the initial region features, the initial region features are downsampled. The downsampling includes maximum pooling and average pooling operations, which are used to retain important feature information while reducing redundancy.

[0023] Furthermore, the external representation learning includes external information extraction, internal information extraction and effective fusion of external information and internal information.

[0024] Furthermore, the external information extraction includes:

[0025] Perform feature transformation and adaptive dimensionality reduction on underwater images after internal representation learning;

[0026] Extract relevant features of multiple images from the associated underwater images;

[0027] The extracted relevant features are output as external information.

[0028] Furthermore, the internal information extraction includes:

[0029] Perform feature transformation and adaptive dimensionality reduction on underwater images after internal representation learning;

[0030] Perform feature extraction on each underwater image in turn;

[0031] The extracted single image features are output as internal information.

[0032] Furthermore, the effective integration of the external information and the internal information includes:

[0033] When extracting external information and internal information, the external information and the internal information are interactively fused at the feature level, channel level and pixel level respectively to improve the enhancement effect of underwater images.

[0034] Furthermore, the loss function includes content loss and perceptual loss. The content loss is used to minimize the pixel difference between the enhanced underwater image and the label image, and the perceptual loss is used to ensure the consistency of the underwater image and the label image in a high-dimensional space, thereby improving the subjective perceptual quality of the output image.

[0035] Furthermore, the content loss uses the L1 norm to calculate the pixel difference between the enhanced underwater image and the label image, and the perceptual loss uses the output features of the third pooling layer of the VGG-19 network to calculate the consistency between the underwater image and the label image in a high-dimensional space.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] 1. Aiming at the characteristics of significant quality differences in the internal regions of underwater images and high correlation between images, the present invention proposes an underwater image enhancement algorithm based on internal and external representation learning. By combining rich internal and external information, it can more effectively learn underwater scene information and obtain high-quality output images. The algorithm has good generalization ability and universality, and can achieve excellent enhancement effects in various underwater scenes.

[0038] 2. In the internal representation learning stage, the present invention performs region segmentation and feature extraction based on the scene depth map of the underwater image, pays attention to the quality degradation differences inside the image for the underwater scene, provides effective guidance for the global features, and guides the differentiated enhancement inside the image;

[0039] 3. In the external representation learning stage, the present invention extracts internal information and external information simultaneously and integrates them from multiple levels, so that internal and external features fully interact with each other, further improving the effect of image enhancement;

[0040] 4. The present invention sets a label image as a reference for enhancement, provides an optimization target for enhancement, reduces pixel changes during image enhancement by calculating content loss and perceptual loss, and improves the subjective perceptual quality of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Underwater images to be enhanced collected by AUVs in practical applications;

[0042] Figure 2 It is an underwater image to be enhanced in a UIEB test data set commonly used in the prior art;

[0043] Figure 3 It is the overall process structure diagram of the present invention;

[0044] Figure 4 This is a structural diagram of the internal representation learning phase of the present invention;

[0045] Figure 5 This is a structural diagram of the external representation learning phase of the present invention;

[0046] Figure 6 The results of different methods on the SQUID data set in the embodiments of the present invention are compared;

[0047] Figure 7 Comparison of results of different methods on sub-RUIE dataset in the embodiments of the present invention;

[0048] Figure 8 Comparison of results of different methods on sub-UIEB dataset in the embodiments of the present invention;

[0049] Fig. 9 Comparison of results of different methods on sub-EUVP dataset in the embodiments of the present invention;

[0050] Fig.10 Comparison of results of different methods on the UISD-1200 dataset in the embodiments of the present invention. DETAILED DESCRIPTION

[0051] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0052] This embodiment discloses an underwater image enhancement method (UIERL) driven by internal and external feature learning, the method comprising:

[0053] Acquire multiple underwater images of different underwater scenes and construct an underwater image scene dataset.

[0054] Generate label images based on underwater images and build an enhanced reference label set.

[0055] The underwater image scene dataset is input into the underwater image enhancement model for image enhancement. The enhancement process specifically includes:

[0056] Learn the internal representation of underwater images to achieve differentiated enhancement of different quality areas inside underwater images;

[0057] Perform external representation learning on underwater images processed by internal representation learning, extract and fuse external information and internal information, and improve the enhancement effect of underwater images;

[0058] Taking the enhanced reference label set as reference, the loss function is used to minimize the pixel difference between the enhanced underwater image and the label image, and improve the subjective perception quality of the underwater image.

[0059] Output the processed underwater image.

[0060] The existing underwater image test dataset usually contains only one test image per scene, which is different from the characteristics of underwater test data in practical applications. To this end, this embodiment constructs a large-scale real underwater image scene dataset (UISD), each scene containing multiple images. Specifically, a large number of video clips are first obtained from underwater videos shot on the network platform and in the laboratory to cover a variety of underwater scenes. Subsequently, the obtained video clips are strictly screened and optimized. After removing unqualified video clips, 1513 scenes are finally retained, totaling 6064 images, covering a wide range of underwater scene content, color range and contrast changes. There are differences in the number of underwater images contained in different underwater scenes, ranging from 2 to 17. In order to construct an enhanced reference label set as a training reference, this embodiment randomly selects 313 scenes (1571 images) from the UISD dataset, and refers to the commonly used underwater image test dataset (UIEB) dataset, and constructs the corresponding enhanced reference labels by subjective selection. Subsequently, these 313 scenes were randomly divided into training and test sets, of which 253 underwater scenes (1,300 images) were used for training, denoted as Train-R1300; the remaining 60 underwater scenes (271 images) were used for testing, denoted as Test-R271.

[0061] The underwater image enhancement model structure used in this method is implemented based on the PyTorch framework, and all experiments are conducted on a Linux platform equipped with a 48G memory Nvidia Quadro RTX 8000-Ti GPU. The model training uses the Adam optimizer, the learning rate is set to 1e-4, and the default values ​​of β1 and β2 are set to 0.5 and 0.999, respectively. Unlike the fixed batch size in traditional deep learning model training, this study adopts a dynamic batch size strategy. In the training phase, each batch randomly selects an underwater scene and dynamically adjusts the batch size according to the number of images contained in the scene. In the test phase, each underwater scene is taken as a batch, which can contain any number of related images.

[0062] First, the specific content of the internal representation learning design in the invention method will be introduced. Based on the underwater physical imaging model, the degradation of underwater image imaging is closely related to the global background light, attenuation coefficient and scene depth. In the same image, the global background light and attenuation coefficient are usually regarded as constants and remain unchanged. Therefore, it can be inferred that the degree of degradation in the internal area of ​​the underwater image is mainly affected by the scene depth, and different depth areas show different degradation characteristics. Taking this feature into account, this step designs a depth map-based regional feature guidance network (DRFG) to learn regional quality guidance features and provide effective guidance for global enhancement features, thereby achieving differentiated enhancement of different quality areas inside the image. Figure 4 As shown in Figure 1, the regional feature guided network consists of two sub-steps: regional segmentation based on the scene depth map and regional spatial encoding. Specifically, first, the underwater image is converted into a scene depth map, and regional segmentation is performed based on the scene depth map to divide the image into K non-overlapping regions, each with a similar quality degradation pattern. Subsequently, feature learning is performed on different quality regions respectively. Finally, the complete regional quality guided feature of the input image is obtained by spatially combining all local regional features.

[0063] In the region segmentation part, considering that the degradation degree of the internal area of ​​the underwater image is closely related to the scene depth, this module directly performs region segmentation on the scene depth map to distinguish regions of different quality. Specifically, first estimate the input underwater image Depth map Subsequently, the K-means clustering algorithm is used to perform pixel-level clustering on the depth map and divide it into three region masks: (k=1,2,3). Finally, multiple region images are obtained by element-wise multiplication of the input image and the region segmentation mask. (k=1,2,3). Each region corresponds to a specific depth range and has the same quality degradation pattern.

[0064] In the area space coding part, this module designs corresponding enhancement strategies based on two key characteristics of significant differences in the quality of the internal regions of underwater images:

[0065] 1. Regions with different scene depth values ​​usually have different degradation patterns, so differential enhancement is required. To this end, this module designs three encoder paths to encode the information of the three regions respectively and learn regional features in their respective local contexts.

[0066] 2. Regions with similar scene depth values ​​usually show similar quality degradation patterns. This property suggests that information within the same region can cooperate with each other and is conducive to extracting valuable regional information. To this end, a parallel content branch and graph branch are introduced in each encoder path.

[0067] Furthermore, the content branch is specifically represented as: area input I k First, a stacked convolution-instance normalization-activation layer (Conv-IN-ReLU) operation is performed to extract features and obtain the initial regional features F k,o . Subsequently, the feature is fed into the content branch and the graph branch respectively. The content branch contains two stacked residual blocks of Conv-IN-ReLU operations, and further extracts regional content features through residual connections. The specific process can be expressed as:

[0068] F k,c =C conv (F k,o )

[0069] in, Indicates the processing of content branching.

[0070] Furthermore, the graph branch is specifically represented as follows: The graph branch constructs a dense two-layer graph convolution layer, representing all input initial region features as nodes. The edges of the graph connect any pairs of nodes, regardless of their spatial distance, so that richer dependencies can be captured. During the network optimization process, each node is updated according to its neighbor nodes and itself, fully mining and integrating the collaborative information in the same degradation region, and ultimately improving the enhanced consistency of regions with the same quality degradation attributes. In order to reduce the memory and computational burden brought by the nodes, the initial region features F k,o Downsampling is performed. The downsampling part includes maximum pooling and average pooling operations, which aims to retain important feature information and reduce redundancy as much as possible. Finally, the new regional feature F is obtained k,g , the specific operation process can be expressed as:

[0071]

[0072] Where K is the number of regions, represents two trainable parameter matrices used to reduce computational complexity, σ represents the Sigmoid function, represents the processing of graph branches, and Represents the trainable parameter matrices of the first and second layers of the graph convolutional network.

[0073] Finally, the three sub-regions are spatially combined at the pixel level to integrate the local and regional features into the global features obtained from the original resolution. These regional features are significantly heterogeneous and complementary, and can synergistically guide the differential enhancement of uneven quality-degraded regions in the image.

[0074] Next, we will introduce the specific content of the external representation learning design in the invention method. This step consists of two interacting branches, such as Figure 5 As shown in the figure, the first branch is the external information extraction network (EIEN), which aims to extract features from multiple related images, namely external information. The second branch is the internal information extraction network (IIEN), which aims to extract features from a single image, namely internal information. Both the external information extraction network and the internal information extraction network adopt the encoder-decoder network structure based on U-Net. The external information extraction network is used to integrate internal information into the external information extraction process. The internal information extraction network is used to integrate external information into the internal information extraction process. The two branches interact through the designed external auxiliary internal module (EAI) and internal auxiliary external module (IAE) to achieve effective fusion of external and internal information.

[0075] Specifically, given a batch of images Where N is the number of images in the batch, W and H are the width and height of the image, respectively. First, after the internal representation learning phase, each image generates a discriminative feature. Then, in the top branch, these features are concatenated in the channel dimension, such as Figure 5 The feature transformation operation in Then, if Figure 5 The dimension reduction operation in F is performed by adaptive dimension reduction operation. r The channel dimension is reduced from 64N to 64, and these features are input into the external information extraction network to extract multi-image features. In the bottom branch, each image is independently fed into the internal information extraction network to obtain single image features i=1,2…N. Then, the characteristics of the top branch Through external auxiliary internal modules and bottom branches interact, and the bottom branch The internal auxiliary external module and the top branch Through this joint interaction mechanism, the network can make full use of rich internal and external information to effectively improve the enhancement effect of a single image.

[0076] Furthermore, the external auxiliary internal module (EAI) receives external features from an external information extraction network and with features from the internal network (where i = 1, 2…N, h∈[0,H], w∈[0,W]) to interact. The external auxiliary internal module includes three interaction modes: feature-level interaction, channel-level interaction, and pixel-level interaction. Feature-level interaction first converts the external features With each internal feature The concatenated features are then fused using a stacked convolution-normalization-activation layer (Conv-IN-ReLU) operation to obtain the final fused features. The process can be expressed as:

[0077]

[0078] in represents the stacked Conv-IN-ReLU operation, and Concat(·) represents the feature concatenation operation.

[0079] Channel-level interaction first uses stacked Conv-IN-ReLU operations to fused features Perform preliminary feature extraction. Subsequently, use the global average pooling (G) operation to extract the statistical information of the feature map in the channel direction, calculate the interactive dependency between channels through two convolutional layers (C), ReLU activation layer (R) and Sigmoid function (σ), and obtain the weight matrix Finally, the weight matrix is ​​combined with the original features Perform element-by-element multiplication and introduce residual connections to obtain channel-level interaction enhancement features The process can be expressed as:

[0080]

[0081] Pixel-level interaction As input, two convolutional layers (C) are used to combine ReLU (R) and Sigmoid (σ) operations to calculate the interaction weight of each pixel. Finally, the residual is added back to the original input features to obtain pixel-level interaction enhanced features. The process can be expressed as:

[0082]

[0083] Where * represents the multiplication operation, Indicated in With the assistance of , the final output features are obtained through feature-level, channel-level and pixel-level interactions.

[0084] Furthermore, the internal auxiliary external module (IAE) receives internal features from the internal information extraction network i=1,2…N, and combined with features from external networks (h∈[0,H],w∈[0,W]). Similar to the external auxiliary internal module, the internal auxiliary external module also contains three steps: feature level interaction, channel level interaction and pixel level interaction. First, the internal feature i=1,2…N are concatenated along the channel dimension to generate features Afterwards, Apply 1x1 convolution to reduce dimension, and we get Finally, the internal features External features Concatenate along the channel dimension and process through a stacked Conv-IN-ReLU operation. The process can be expressed as:

[0085]

[0086] in, Represents the new features obtained by feature-level interactive update; Denotes the stacked Conv-IN-ReLU operation, and Concat(·) denotes the concatenation operation. Similar to the external-assisted internal, the internal-assisted external module further performs channel-level and pixel-level interactions on the features generated by feature-level interactions, and the final output feature is represented as

[0087] Finally, the loss function used in the invented method will be introduced. The loss function is used after the image has undergone internal representation learning and external representation learning.

[0088] Given a batch of training pairs Where I represents the original underwater image, J represents the corresponding label image, and N represents the number of images in the batch. This algorithm is trained in an end-to-end manner, and the loss function includes content loss (L con ) and perceptual loss (L per ). The content loss function uses the L1 norm, which aims to minimize the pixel difference between the enhanced image and the label image:

[0089]

[0090] Perceptual loss L per It aims to ensure the consistency between the output image and the label image in the high-dimensional space and improve the subjective perception quality of the output image. It is defined as follows:

[0091]

[0092] Here, φ(·) represents the output features of the third pooling layer in the pre-trained VGG-19 network.

[0093] The complete loss function is:

[0094] L total =λ1*L con +λ2*L per

[0095] Among them, λ1 and λ2 represent weight parameters, which are 0.8 and 0.2 respectively.

[0096] This embodiment is experimentally verified on five real underwater image datasets, including SQUID, sub-RUIE, sub-UIEB, sub-EUVP, and the UISD-1200 dataset containing more comprehensive scenes. The SQUID dataset contains 57 pairs of underwater stereo images taken from four different diving sites in Israel, that is, 114 images. These images are divided into 35 scenes, each containing 2 to 10 images. Sub-RUIE, sub-UIEB, and sub-EUVP are image subsets selected from the commonly used real underwater image datasets RUIE, UIEB, and EUVP, and their characteristics are more consistent with underwater test data in practical applications. Among them, the sub-RUIE dataset contains 413 scenes totaling 1524 images, each scene contains 2 to 20 images; the sub-UIEB dataset contains 112 scenes totaling 267 images, each scene contains 2 to 4 images; the sub-EUVP dataset contains 99 scenes totaling 358 images, each scene contains 2 to 5 images. The UISD-1200 dataset is the remaining 1200 scenes in the proposed UISD dataset, containing 4496 images, each scene containing 2 to 17 images. These datasets cover different underwater scenes and image features, and can comprehensively evaluate the generalization ability of the algorithm.

[0097] As shown in Table 1, this embodiment compares six advanced underwater image enhancement methods: FUIEGAN, LCNet, Ucolor, TOPAL, CLUIE, and SGUIE. All comparison algorithms used the test models and parameters released by the original authors to ensure the fairness of the experiment. In addition, in order to more comprehensively evaluate the performance of the underwater image enhancement algorithm driven by internal and external feature learning, this study also compared it with two multi-image-based land image enhancement methods: AHDRNet and LF-IINet. In order to achieve a fair comparison, the last upsampling layer of AHDRNet and the Sigmoid layer of LF-IINet were removed and retrained using the same training data as the UIERL algorithm.

[0098] Table 1 Comparison of objective indicators of different methods on 5 datasets

[0099]

[0100] This embodiment uses four reference-free underwater image quality evaluation indicators to verify the performance of each algorithm, including Underwater Image Quality Measure (UIQM), Underwater Color Image Quality Evaluation (UCIQE), CCF (a linear combination evaluation indicator of color, contrast and fog density) and edge strength (Edge). In order to evaluate the performance more accurately, a subjective evaluation comparison experiment was also conducted. Specifically, 200 images were first randomly selected from each test data set. Then 20 volunteers with different ages, backgrounds and genders were invited to score the enhancement results of each method under the same environment. The scoring adopts a 5-point system, from 5 to 1, representing very good, good, average, poor, and very poor subjective image quality.

[0101] As can be seen from Table 1, the UIERL algorithm proposed in this invention achieves the best performance in most indicators, which shows that the algorithm can achieve enhanced results with good perceptual quality, has stable performance, and can adapt to underwater images with different water quality. Specifically, in terms of CCF, UIQM and Edge indicators, the UIERL algorithm obtains the best quantitative results on all data sets. In terms of UCIQE indicators, the UIERL algorithm performs best on sub-EUVP and SQUID data sets, and is slightly inferior to SGUIE on sub-UIEB, sub-RUIE and UISD-1200 data sets, ranking second. The subjective evaluation results (SA scores) further verify the superiority of the UIERL algorithm, and the proposed algorithm obtains the highest average perceptual scores on all five data sets. Among all the comparison algorithms, AHDRNet and LF-IINet introduce rich underwater external information and perform relatively well. However, they do not fully consider the important feature of significant differences in the quality of the internal areas of underwater images, resulting in limited performance. In contrast, the UIERL algorithm proposed in the present invention effectively explores the internal information of underwater images and further combines the rich external information provided by related images, significantly improving the image enhancement effect.

[0102] Figure 6-Figure 10 Some enhanced examples on five real underwater image datasets are shown. These examples cover a wide range of underwater image characteristics (such as color deviation, low contrast, and blurred details), including blue-toned images, green-toned images, yellow-toned images, cyan-toned images, and some complex images. Figure 6 and Figure 7 Some enhancement examples on the SQUID dataset and the sub-RUIE dataset are shown respectively. For underwater images with obvious blue or green color casts and severe fog, the algorithm proposed in the present invention not only effectively corrects the color cast, but also enhances the texture details and contrast, significantly improving the realism of the results. In contrast, all the comparison algorithms have little effect on the fog effect and color distortion on underwater images. Ucolor and SGUIE cannot restore the true color, and there is still severe color distortion in the foreground. FUIE-GAN, LCNet and CLUIE even introduce additional color deviations in the results. TOPAL enhances the image to a dark tone, which does not look real in the real world. Due to the introduction of richer external information, the enhanced results of AHDRNet and LF-IINet have relatively good visual effects. However, they cannot restore the complete scene color and structure, for example, the details of the coral reefs in the distance are not well restored. In contrast, the algorithm proposed in the present invention can generate more satisfactory results and present clearer details, thanks to the effective design of the internal and external representation learning stages. Figure 8 and Fig. 9Some enhanced examples on the sub-UIEB dataset and the sub-EUVP dataset are shown respectively. For these images with diverse blue-green or yellow tones, the algorithm proposed in the present invention still shows significant advantages in color correction and detail restoration. In contrast, other deep learning-based methods have difficulty in achieving satisfactory results. Almost all comparison methods cannot effectively deal with color deviation and blur problems, and even produce unrealistic colors and additional noise in the results, such as FUIE-GAN and LCNet. Although AHDRNet and LF-IINet correct the color deviation to a certain extent, the contrast of the image is not well improved. Although SGUIE improves the contrast of the input image, it introduces obvious oversaturation and tends to amplify noise and artifacts, resulting in limited visual effects. Fig.10 Some enhanced examples on the UISD-1200 dataset are shown. For underwater images taken in various challenging scenarios, the UIERL algorithm is still able to effectively correct color deviations and improve image clarity. Specifically, thanks to the designed internal representation learning stage, the UIERL algorithm emphasizes the differential enhancement of different degraded regions, which makes the enhanced images look more natural overall (foreground and background regions of the example). In addition, combined with the rich external information in the related images, the UIERL algorithm is able to restore more details of the original image, such as the structure and edges of objects (diver and shark objects of the example). In contrast, the contrast algorithms have limited effects in color correction and contrast enhancement. Ucolor and LCNet even introduce unrealistic colors in the results. The results of AHDRNet and LF-IINet have relatively good visual quality, but they cannot improve the contrast of the foreground and fail to restore clear structures. A large number of quantitative and qualitative results show that the proposed algorithm is able to generate results with higher visual quality, more consistent with human subjective perception, and show stronger robustness in various underwater scenes.

[0103] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An underwater image enhancement method driven by internal and external feature learning, characterized in that: The method comprises: Acquire the underwater image to be tested; Generate a labeled image based on an underwater image; Learn the internal representation of underwater images to achieve differentiated enhancement of different quality areas inside underwater images; Perform external representation learning on underwater images processed by internal representation learning, extract and fuse external information and internal information, and improve the enhancement effect of underwater images; The loss function is used to minimize the pixel difference between the enhanced underwater image and the labeled image and improve the subjective perception quality of the underwater image; Output the processed underwater image.

2. The underwater image enhancement method driven by internal and external feature learning according to claim 1, characterized in that: The label image is obtained by processing with image editing software and corresponds one-to-one with the underwater image.

3. The underwater image enhancement method driven by internal and external feature learning according to claim 1, characterized in that: The internal representation learning includes: Perform depth estimation on the underwater image to be tested to obtain a scene depth map; Performing regional segmentation on the scene depth map, dividing the scene depth map into three or more non-overlapping regions, where images in the same region have similar quality degradation patterns; Extract features from the information of each divided area to obtain initial area features; The initial regional features are processed through residual connections to further extract regional content features; Use graph convolutional networks to process initial regional features, enhance the consistency of regional features in the same region and the differences of regional features in different regions; Through pixel-by-pixel addition, the regional content features and regional features of each region are integrated into the global features to achieve differentiated enhancement of different quality areas inside the underwater image.

4. The underwater image enhancement method driven by internal and external feature learning according to claim 3 is characterized in that: When using the graph convolutional network to process the initial region features, the initial region features are downsampled. The downsampling includes maximum pooling and average pooling operations, which are used to retain important feature information while reducing redundancy.

5. The underwater image enhancement method driven by internal and external feature learning according to claim 1, characterized in that: The external representation learning includes external information extraction, internal information extraction and effective fusion of external information and internal information.

6. The underwater image enhancement method driven by internal and external feature learning according to claim 5, characterized in that: The external information extraction includes: Perform feature transformation and adaptive dimensionality reduction on underwater images after internal representation learning; Extract relevant features of multiple images from the associated underwater images; The extracted relevant features are output as external information.

7. The underwater image enhancement method driven by internal and external feature learning according to claim 5, characterized in that: The internal information extraction includes: Perform feature transformation and adaptive dimensionality reduction on underwater images after internal representation learning; Perform feature extraction on each underwater image in turn; The extracted single image features are output as internal information.

8. The underwater image enhancement method driven by internal and external feature learning according to claim 5, characterized in that: The effective integration of external information and internal information includes: When extracting external information and internal information, the external information and the internal information are interactively fused at the feature level, channel level and pixel level respectively to improve the enhancement effect of underwater images.

9. The underwater image enhancement method driven by internal and external feature learning according to claim 1, characterized in that: The loss function includes content loss and perceptual loss. The content loss is used to minimize the pixel difference between the enhanced underwater image and the label image. The perceptual loss is used to ensure the consistency of the underwater image and the label image in a high-dimensional space and improve the subjective perceptual quality of the output image.

10. The underwater image enhancement method driven by internal and external feature learning according to claim 9, characterized in that: The content loss uses the L1 norm to calculate the pixel difference between the enhanced underwater image and the label image, and the perceptual loss uses the output features of the third pooling layer of the VGG-19 network to calculate the consistency between the underwater image and the label image in a high-dimensional space.

Images