An underwater image super-resolution method based on multi-feature image fusion

Abstract

The application discloses an underwater image super-resolution method based on multi-feature image fusion, and comprises the following steps: obtaining an underwater image, and respectively pre-processing the underwater image by using three methods of white balance, adaptive histogram equalization and dark channel prior; constructing a feature fusion super-resolution network model, inputting the three pre-processed underwater images and an original image into the feature fusion super-resolution network model for training, and obtaining confidence information of each pre-processed image; respectively performing feature optimization on the original image and the three pre-processed images to obtain three feature-optimized images; multiplying the three feature-optimized images and respective confidences to obtain three underwater images to be fused, and obtaining a feature-fused underwater image; and performing super-resolution reconstruction on the feature-fused underwater image to obtain a super-resolution reconstructed image; and the method can better extract useful detail information of the pre-processed image while reducing artifacts.

Description

Technical Field

[0001] This invention relates to the field of underwater image processing technology, and in particular to an underwater image super-resolution method based on multi-feature image fusion. Background Technology

[0002] In recent years, with the deepening of human exploration of marine resources, the development of underwater machine vision has played a crucial role in the exploration of marine resources. However, due to the complex and diverse underwater environment, factors such as light intensity attenuation caused by a large number of suspended particles in seawater, and forward and backscattering of light by the water environment, underwater images often suffer from problems such as color distortion, blurred details, low contrast, and low resolution. Therefore, utilizing underwater image super-resolution reconstruction technology to improve the contrast and resolution of underwater images, in order to solve the problems of missing details and textures and blurred images, has significant research value.

[0003] Image super-resolution reconstruction technology utilizes knowledge from digital image processing and computer vision to restore low-quality, low-resolution images into high-quality, high-resolution images. Compared to low-resolution images, high-resolution images typically contain higher pixel density, richer texture details, and higher reliability. Therefore, high-resolution images greatly improve human image recognition capabilities and accuracy. Image super-resolution reconstruction methods are divided into traditional super-resolution reconstruction algorithms and deep learning-based super-resolution reconstruction algorithms. Traditional super-resolution reconstruction algorithms mainly use digital image processing techniques for image reconstruction, including interpolation-based methods, degradation model-based methods, and learning-based methods. With the rise of deep learning algorithms, more and more deep learning-based super-resolution reconstruction methods have emerged.

[0004] Compared to image super-resolution reconstruction methods on land, existing research on underwater image super-resolution reconstruction methods is relatively limited. This is because underwater imaging is often affected by poor visibility, light absorption, and scattering, resulting in underwater images lacking important details and having indistinct regions of interest, posing significant challenges to underwater image super-resolution reconstruction. Researchers have made some attempts at underwater image super-resolution reconstruction in recent years, focusing on reconstructing higher-quality underwater images from noisy or blurred images. Among these, feature fusion-based image super-resolution methods offer a promising approach to solving underwater image super-resolution reconstruction. The feature fusion module extracts and fuses the most salient features from multiple feature maps into a single image, creating a more discriminative feature image. The super-resolution module employs a deep residual network structure, using skip connections to preserve the label mapping in repeated blocks of convolutional layers, which helps in the stable training of very deep network models. Skip connections in dense residual blocks combine hierarchical features from each layer to improve the performance of image super-resolution reconstruction. Feature fusion-based image super-resolution methods can address the difficulties in feature extraction, the lack of clear details, and the low resolution of underwater images. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention discloses an underwater image super-resolution method based on multi-feature image fusion, which specifically includes the following steps:

[0006] Underwater images were acquired, and three methods—white balance, adaptive histogram equalization, and dark channel prior—were used to preprocess the underwater images.

[0007] A feature fusion super-resolution network model was constructed. The three preprocessed underwater images and the original image were input into the feature fusion super-resolution network model for training, and the confidence information of each preprocessed image was obtained.

[0008] The original image was combined with the three preprocessed images to obtain three feature-optimized images.

[0009] The three feature-optimized images are multiplied by their respective confidence scores to obtain three underwater images to be fused. The three underwater images to be fused are then combined and added together to perform feature fusion, resulting in a feature-fused underwater image.

[0010] Super-resolution reconstruction is performed on underwater images with feature fusion to obtain super-resolution reconstructed images.

[0011] Furthermore, the white balance method is as follows: first, color projection correction is performed on the underwater image to obtain accurate color reproduction; the adaptive histogram equalization method is as follows: first, the local histogram of the underwater image is calculated, and the brightness is redistributed to improve the local contrast of the underwater image, thereby obtaining more details of the image; the dark channel prior method is as follows: the transmittance and atmospheric light component values ​​of the underwater image are calculated, the original image is subjected to Gaussian low-pass filtering and then inverted to obtain a transmittance image, and a locally dehazed underwater image is obtained based on the transmittance image.

[0012] Furthermore, the adaptive histogram equalization feature optimization is applied to image R. HE White balance feature optimization image R WB Optimize image R with prior features of dark channel DCP Each with its own confidence level D HE D WB and D DCP Multiplying the images yields three underwater images to be fused. These three images are then combined and added together to perform feature fusion, resulting in the fused underwater image I. ER The function representation of this process is:

[0013] I ER =D HE ×R HE +D WB ×R WB +D DCP ×R DCP (1)

[0014] In the process of feature fusion of underwater images, a mapping function for feature fusion of underwater images is learned by minimizing the perceptual loss function, thereby reducing artifacts caused by pixel-level loss functions and measuring the feature fusion image I. ER With reference image I RF The difference perception loss function is:

[0015]

[0016] Where N represents the number of batches in each training process, and W k H k C k Let θ represent the width, height, and number of channels of the feature map of the k-th convolutional layer in the network model, respectively. k (x) represents the k-th layer of the pre-trained network model after activation.

[0017] Furthermore, super-resolution reconstruction of underwater images based on feature fusion includes reconstructing both low-level and high-level feature information from the underwater image:

[0018] When reconstructing low-level feature information: firstly, low-level feature information is extracted from the underwater image, and then the low-level feature information in the underwater image is restored;

[0019] When reconstructing high-level feature information: firstly, high-level feature information of the underwater image is extracted, and then the high-level feature information of the high-resolution underwater image is extracted using a pre-trained network model through transfer learning. This high-level information is compared with the high-level feature information extracted in the previous step, and the error between the two is reduced by minimizing the mean square error.

[0020] The deconvolution method is used to perform super-resolution reconstruction of the low-level and high-level feature information of the recovered underwater image.

[0021] Furthermore, when performing super-resolution reconstruction on underwater images with fused features, the difference between the extracted underwater image features and those extracted from the network model is first measured to obtain near-realistic high-resolution underwater images. A content loss function is then used to measure the similarity between the images.

[0022] L C (F)=Ι h,l [||θ(h)-θ(F(l))||2] (3)

[0023] Where F:{l}→h represents a learning function or mapping, l represents the low-resolution image domain, and h represents the high-resolution image domain; in order to adjust the perceptual loss function formula (2) and the content loss function formula (3) so that the two loss functions obtain small losses, a hyperparameter λ related to the loss function is introduced. k With λ C The objective loss function of the feature fusion super-resolution network is expressed as:

[0024]

[0025] By employing the aforementioned technical solutions, this invention provides an underwater image super-resolution method based on multi-feature image fusion. This method effectively addresses issues such as low contrast and low resolution in underwater images. It also helps to resolve color cast problems in underwater images. The method utilizes a feature fusion network, which can better extract useful detail information from the preprocessed image while reducing artifacts. It employs a deep residual block structure, which effectively extracts and combines low-level feature details from underwater images. Transfer learning is then used to supplement high-level feature details in the underwater image, thereby reconstructing a high-quality, high-resolution underwater image. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of the underwater image super-resolution method based on multi-feature image fusion according to the present invention;

[0028] Figure 2 This is a diagram illustrating the overall framework of the underwater image super-resolution method based on multi-feature image fusion according to the present invention. Detailed Implementation

[0029] To make the technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention:

[0030] like Figure 1 This paper presents an underwater image super-resolution method based on multi-feature image fusion. The method first acquires underwater images from different sea areas and depths, and then processes the original images using three algorithms: white balance, adaptive histogram equalization, and dark channel prior, to construct underwater feature maps with multiple discriminative information. Next, the three feature maps, along with the original images, are input into a feature confidence network to obtain the confidence information for each feature image. Furthermore, a feature optimization unit is used to process the feature maps to reduce redundant color mapping and artifacts introduced by the preprocessing algorithms. The optimized feature images are multiplied by their respective confidence scores to obtain the underwater images to be fused. Finally, the different types of images are fused to obtain a feature-fused image. Due to light scattering in water, underwater image feature extraction is difficult. Feature fusion can extract the essential features of the target from underwater feature maps with multiple discriminative information and merge them into a feature image with stronger discriminative power. Then, the feature-fused image is input into the super-resolution module, which uses dense residual blocks to combine low-level feature information from each convolutional layer and uses transfer learning to supplement high-level feature information. Finally, deconvolution is used to increase the image resolution. This method addresses the issues of low contrast and low resolution in underwater images, effectively enhancing the detailed texture features and improving the resolution. The specific steps of this method are as follows:

[0031] S1: Acquire underwater images and preprocess them using three methods: white balance, adaptive histogram equalization, and dark channel prior. The specific methods are as follows:

[0032] S11: Acquire underwater images and integrate them into a dataset. Then, divide the acquired underwater image dataset into a training dataset and a test dataset in a 7:3 ratio.

[0033] S12: Preprocessing underwater images using a white balance method. Addressing the blue-green tint in underwater images, the proportion of red pixels is increased while the proportion of green and blue pixels is decreased, performing color projection correction to achieve accurate color reproduction. Specifically, the histogram distribution of pixel values ​​for each RGB channel is first calculated. The first 0.25% of pixels in the histogram are assigned a value of 0, the next 0.25% are assigned a value of 255, and the remaining values ​​are evenly mapped to a range of 0 to 255, effectively performing histogram equalization on each channel. This results in a more uniform distribution of values ​​across the RGB channels, achieving color balance.

[0034] S13: An adaptive histogram equalization method is used to preprocess the underwater image. The local histogram of the underwater image is calculated, and then brightness is redistributed to improve local contrast and obtain more image details. Specifically, histogram specification is first used to process each local region segmented from the underwater image, and then bilinear interpolation is used to combine adjacent local regions to eliminate artificially introduced boundary effects. In particular, it can limit the contrast of uniform brightness areas to avoid amplifying noise.

[0035] S14: The underwater image is preprocessed using a dark channel prior method. First, the transmittance and atmospheric light composition of the underwater image are calculated. Then, the original image is inverted after being filtered by a Gaussian low-pass filter to obtain a transmittance image. Subsequently, a locally dehazed underwater image is obtained from the transmittance image, which enhances the brightness of dark areas in the underwater image.

[0036] S2: Construct a feature fusion super-resolution network model, specifically using the following method:

[0037] S21: Use PyTorch to build a deep learning framework.

[0038] S22: Compile the model to complete the fitting training process. A gated fusion network architecture is used to design the confidence module, multi-layer convolution and rectified linear units are used to design the feature optimization module, and dense residual blocks and deconvolution operations are used to design the super-resolution module.

[0039] S23: Fit the model on the acquired training dataset, setting the epoch to 400, the batch_size to 16, and the learning rate to 0.001.

[0040] S24: Evaluate the model and make predictions. Test the model with variable data and random tests to verify the effectiveness of the trained model and whether its performance efficiency meets expectations.

[0041] S3: Input the three preprocessed images and the original image together into the feature fusion super-resolution network for training. After passing through the feature confidence module, obtain the confidence of each preprocessed image. Specifically, the following method is used:

[0042] S31: Input the three preprocessed images and the original image in parallel into the feature confidence module of the feature fusion super-resolution network.

[0043] S32: The network is trained using multi-layer convolution and normalized exponential function to obtain confidence information for each preprocessed image.

[0044] S33: Determine the most salient feature in the feature fusion image from the learned confidence information, combine the three preprocessed feature images into a feature image that comprehensively utilizes multiple image features and has stronger discriminative ability, realize the complementary advantages of multiple features, and obtain more robust and salient results.

[0045] S4: Preprocessing the original underwater image using three image processing algorithms enhances the details from different angles, but also introduces unnecessary background noise. The feature optimization module minimizes the color mapping and artifacts introduced by the three image processing algorithms into the original image, thus improving image quality. The original image and the three preprocessed images are input into the feature optimization module for feature optimization, resulting in three optimized feature images, specifically as follows:

[0046] S41: Input the original image and the three preprocessed images into the feature optimization module of the feature fusion super-resolution network.

[0047] S42: To achieve more efficient gradient descent and backpropagation, and to avoid gradient explosion and vanishing problems, a multi-layer convolutional and linear rectified function method is used to train the network, extract and combine the feature information of the input image, reduce the color mapping and artifacts introduced by the three image processing algorithms in the original image, improve image quality, and at the same time reduce the overall computational cost of the neural network by dispersing the activity.

[0048] S5: Due to factors such as light scattering by water, underwater image feature extraction is difficult. Feature fusion, however, can extract the essential features of the target from underwater feature maps with multiple discriminative information and merge them into a feature image with stronger discriminative power. The three optimized feature images are multiplied by their respective confidence scores to obtain the underwater image to be fused. The three underwater images to be fused are then integrated and added together to perform feature fusion, resulting in the fused underwater image. Specifically, the following method is used:

[0049] S51: Using the element-wise multiplication method, the three feature-optimized images obtained by the feature optimization module are multiplied with their respective confidence information to obtain three underwater images to be fused.

[0050] S52: Using the element-wise addition method, the three underwater images to be fused are integrated and added together to perform feature fusion, resulting in a feature-fused underwater image.

[0051] S53: Represent the feature fusion process as a function to facilitate the determination of the loss cost. Optimize the image R using histogram-equalized features. HE White balance feature optimization image R WB Optimize image R with prior features of dark channel DCP Each with its own confidence level D HE D WB and D DCP Multiplying the images yields three underwater images to be fused. These three images are then combined and added together to perform feature fusion, resulting in the fused underwater image I. ER The function representation of this process is:

[0052] I ER =D HE ×R HE +D WB ×R WB +D DCP ×R DCP (1)

[0053] S54: In the process of feature fusion of underwater images, the mapping function of the feature fusion underwater image is learned by minimizing the perceptual loss function, thereby reducing artifacts caused by the pixel-level loss function and measuring the feature fusion image I. ER With reference image I RF The difference perception loss function is:

[0054]

[0055] Where N represents the number of batches in each training process, and W k H k C k Let θ represent the width, height, and number of channels of the feature map of the k-th convolutional layer in the network model, respectively. k (x) represents the k-th layer of the pre-trained network model after activation.

[0056] S6: The underwater image obtained by feature fusion is input into the super-resolution module for super-resolution reconstruction to obtain a super-resolution reconstructed image, specifically using the following method:

[0057] S61: Employing multi-layer convolution and batch normalization, low-level feature information in underwater images is extracted. Dense residual skip connections are used to combine hierarchical features and label mappings from each convolutional layer to restore the low-level feature information in underwater images.

[0058] S62: Employs multi-layer convolution and batch normalization to extract and preserve high-level feature information from underwater images.

[0059] S63: Employ transfer learning to obtain high-level feature information of high-resolution underwater images extracted from pre-trained network models.

[0060] S64: Compare the high-level feature information saved in step two with the high-level feature information obtained in step three, use the minimum mean square error method to reduce the error between the two, and restore the high-level feature information of the underwater image.

[0061] S65: The deconvolution method is used to perform super-resolution reconstruction of the low-level and high-level feature information of the recovered underwater image to obtain a super-resolution underwater image.

[0062] S66: When performing super-resolution reconstruction on underwater images with feature fusion, to measure the difference between the extracted underwater image features and the underwater image features obtained using transfer learning methods, and to obtain a near-realistic high-resolution underwater image, a content loss function is used to measure the similarity between the images.

[0063] L C (F)=Ι h,l [||θ(h)-θ(F(l))||2] (3)

[0064] Where F:{l}→h represents a learning function or mapping, l represents the low-resolution image domain, and h represents the high-resolution image domain; in order to adjust the perceptual loss function formula (2) and the content loss function formula (3) so that the two loss functions obtain small losses, a hyperparameter λ related to the loss function is introduced. k With λ C The objective loss function of the feature fusion super-resolution network is expressed as:

[0065]

[0066] This invention discloses an underwater image super-resolution method based on feature fusion networks, which effectively solves problems such as low contrast and low resolution in underwater images. It also helps address color cast issues in underwater images. By employing feature fusion networks, useful detail information in the preprocessed image can be extracted more effectively while reducing artifacts. Using a deep residual block structure, low-level feature details of the underwater image can be extracted and combined efficiently. Transfer learning is then used to supplement high-level feature details, resulting in a high-quality, high-resolution underwater image reconstructed.

[0067] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A super-resolution method for underwater images based on multi-feature image fusion, characterized in that... include: Underwater images were acquired, and three methods—white balance, adaptive histogram equalization, and dark channel prior—were used to preprocess the underwater images. A feature fusion super-resolution network model was constructed. The three preprocessed underwater images and the original image were input into the feature fusion super-resolution network model for training, and the confidence information of each preprocessed image was obtained. The original image was combined with the three preprocessed images to obtain three feature-optimized images. The three feature-optimized images are multiplied by their respective confidence scores to obtain three underwater images to be fused. The three underwater images to be fused are then combined and added together to perform feature fusion, resulting in a feature-fused underwater image. Super-resolution reconstruction is performed on underwater images with feature fusion to obtain super-resolution reconstructed images; Adaptive histogram equalization feature optimization image White balance feature optimization image Image optimization using prior features of dark channels Each with its own confidence level Multiplying the images yields three underwater images to be fused. These three images are then combined and added together to perform feature fusion, resulting in a fused underwater image. The function representation of this process is: (1) In the process of feature fusion of underwater images, a mapping function for feature fusion of underwater images is learned by minimizing the perceptual loss function, thereby reducing artifacts caused by pixel-level loss functions and measuring the feature fusion image. Compared with reference image The difference perception loss function is: (2) Where N represents the number of batches in each training process. Let represent the width, height, and number of channels of the feature map of the k-th convolutional layer in the network model, respectively. This represents the k-th layer of the pre-trained network model after activation; Super-resolution reconstruction of underwater images with feature fusion involves reconstructing both low-level and high-level feature information from the underwater image. When reconstructing low-level feature information: firstly, low-level feature information is extracted from the underwater image, and then the low-level feature information in the underwater image is restored; When reconstructing high-level feature information: firstly, high-level information of the underwater image is extracted, and then the high-level information of the high-resolution underwater image is extracted using a pre-trained network model through transfer learning. The high-level information is compared with the high-level information extracted in the previous step, and the error between the two is reduced by minimizing the mean square error. The deconvolution method is used to perform super-resolution reconstruction of the low-level and high-level feature information of the recovered underwater image.

2. The underwater image super-resolution method based on multi-feature image fusion according to claim 1, characterized in that: The white balance method is as follows: first, color projection correction is performed on the underwater image to obtain accurate color reproduction; the adaptive histogram equalization method is as follows: first, the local histogram of the underwater image is calculated, and the brightness is redistributed to improve the local contrast of the underwater image, thereby obtaining more details of the image; the dark channel prior method is as follows: the transmittance and atmospheric light component values ​​of the underwater image are calculated, the original image is subjected to Gaussian low-pass filtering and then inverted to obtain the transmittance image, and the locally dehazed underwater image is obtained based on the transmittance image.

3. The underwater image super-resolution method based on multi-feature image fusion according to claim 1, characterized in that: When performing super-resolution reconstruction on underwater images with feature fusion, the first step is to measure the difference between the extracted underwater image features and those extracted from the network model to obtain a near-realistic high-resolution underwater image. A content loss function is then used to measure the similarity between the images. (3) in, Represents a learning function or mapping. Represents the low-resolution image domain. Representing the high-resolution image domain; in order to adjust the perceptual loss function formula (2) and the content loss function formula (3) so that the two loss functions obtain small losses, hyperparameters related to the loss functions are introduced. and The objective loss function of the feature fusion super-resolution network is expressed as: (4)。

Images