A collaborative enhancement and super-resolution method for underwater images

Abstract

This invention discloses an underwater image collaborative enhancement and super-resolution method, belonging to the field of computer vision technology. It includes constructing a training dataset for the model training process using an underwater image dataset with visual enhancement reference images. The training dataset includes low-resolution original underwater degraded images, low-resolution reference ground truth images, high-resolution original degraded underwater images, and high-resolution reference ground truth images. An underwater image collaborative enhancement and super-resolution deep learning network is constructed, using pairs of original underwater degraded images as input to the deep network, and using their corresponding low-resolution reference ground truth images to constrain and optimize the predicted images output by the model. The model training process is constrained and optimized by minimizing pixel value difference loss and structural similarity loss, thereby training the underwater image collaborative enhancement and super-resolution deep learning network. Any pair of original underwater degraded images is used as input to the model, and the output is obtained through model calculation.

Description

Technical Field

[0001] This invention discloses an underwater image collaborative enhancement and super-resolution method, belonging to the field of computer vision technology. Background Technology

[0002] In recent years, Autonomous Underwater Vehicles (AUVs) and Remotely Operated Vehicles (ROVs), as major underwater observation and operation platforms, have experienced rapid development. To acquire underwater visual data, most AUVs are equipped with high-performance cameras. As one of the primary sources of underwater information, vision can provide intuitive observation and guidance for underwater operations. Good visual quality and high resolution are increasingly becoming important requirements for the image acquisition modules of AUVs. However, underwater imaging conditions are dynamically changing and more complex than atmospheric imaging environments. Furthermore, degradation of underwater images is often unavoidable due to wavelength-dependent absorption, forward scattering, backscattering, turbulence, and suspended matter interference. Therefore, Underwater Image Enhancement (UIE), aiming to improve the visual quality of underwater images, has recently received widespread attention. The quality of the enhancement even determines the success or failure of subsequent underwater vision missions.

[0003] Furthermore, although most underwater robots are equipped with high-performance cameras, the transmission of acquired images remains a challenging issue. For cable-free long-range communication underwater robots, transmitting high-resolution images for long-distance underwater missions is extremely difficult due to the high cost and narrow bandwidth of cable-free data transmission. Sending low-resolution images and then transmitting high-resolution images after super-resolution (SR) processing during the decoding stage is an effective compromise. Underwater image enhancement improves both contrast and saturation, while super-resolution technology recovers local details lost due to downsampling operations. Together, these improvements enhance the visual quality of underwater images, resulting in a better underwater observation and operational experience for underwater robots. Therefore, both underwater image enhancement and underwater image super-resolution are crucial for underwater observation and operation.

[0004] Unfortunately, while underwater image enhancement and single-image super-resolution have been extensively studied separately, only a few methods have explored them as a coupled problem. Both tasks share the common goal of improving image visual quality, with enhancement helping super-resolution better capture degradation information in underwater images. Furthermore, we note that most existing underwater image enhancement algorithms are designed for isolated single images, hampered by a severe lack of underwater visual enhancement learning samples, making it difficult to learn robust enhancement models. Few researchers have addressed the cooperative super-resolution task from non-overlapping images of related scenes. Even the few joint enhancement and super-resolution algorithms rely on isolated single images as input. Compared to the rich content diversity of terrestrial scenes, underwater scenes have relatively limited semantic content types, primarily including water bodies, aquatic plants, ruins, reefs, fish, seabed, and rocks. Images taken in similar underwater scenes, i.e., scene-related images, typically contain a wealth of complementary information. Therefore, cooperative processing strategies for scene-related images are more conducive to learning a unified, effective, highly visually consistent, and universal deep model, with very broad application prospects. Based on this, the present invention proposes a method for collaborative enhancement and super-resolution of underwater images. Summary of the Invention

[0005] This invention discloses an underwater image collaborative enhancement and super-resolution method, which solves the problem that existing image enhancement and super-resolution algorithms lack an overall joint calculation method.

[0006] An underwater image collaborative enhancement and super-resolution method includes:

[0007] S1. Using an underwater image dataset with visually enhanced reference images, construct the training dataset required for training the deep learning neural network model. The training dataset includes low-resolution raw underwater degraded images X. l , with X l The corresponding low-resolution reference ground truth image Y l High-resolution raw degraded underwater image X h , with X h The corresponding high-resolution reference ground truth image Y h ;

[0008] S2. Construct an underwater image collaborative enhancement and super-resolution deep learning network, combining paired X... l As input to this deep network, using its corresponding Y h Constraint optimization is performed on the predicted image output by the deep learning neural network model;

[0009] S3. By minimizing pixel value difference loss and structural similarity loss, the training process of the deep learning neural network model is constrained and optimized to train the underwater image collaborative enhancement and super-resolution deep learning network, thereby obtaining the deep learning neural network model.

[0010] S4. Place any pair of X l As input to the model, it outputs Y after passing through a deep learning neural network model. h .

[0011] S1 includes:

[0012] S1.1. Using image downsampling methods, for X h Perform downsampling to obtain the corresponding X l ;

[0013] S1.2. Using image downsampling methods, for Y h Perform downsampling to obtain the corresponding Y l .

[0014] S2 includes:

[0015] S2.1. Construct an underwater image collaborative enhancement and super-resolution deep learning network structure, which consists of a cascaded underwater image collaborative enhancement submodule CoE-M and an underwater image collaborative super-resolution submodule CoSR-M;

[0016] S2.2. Construct CoE-M and CoSR-M based on a dual-branch twin deep network structure. The twin structure is connected by a feature matching association submodule to realize feature information transmission.

[0017] S3 includes:

[0018] S3.1. Place a pair of X l Let it be X l,1 and X l,2 , as input to underwater image collaborative enhancement and super-resolution deep learning networks;

[0019] S3.2. Underwater Image Co-enhancement and Super-resolution Deep Learning Network outputs a pair of model-predicted low-resolution clear underwater images, denoted as... and Calculate separately and Between, Y l,1 and Y l,2 Loss of pixel value difference between them;

[0020] S3.3. Place a pair of X h Let it be X h,1 and X h,2, as input to underwater image collaborative enhancement and super-resolution deep learning networks;

[0021] S3.4. Underwater Image Co-enhancement and Super-resolution Deep Learning Network Outputs a Pair of High-Resolution, Clear Underwater Images Predicted by the Model, denoted as... and Calculate separately and Between, Y h,1 With Y h,2 Pixel value difference loss and structural similarity loss between them;

[0022] S3.5. Input the training dataset obtained in S1 into the underwater image co-enhancement and super-resolution deep learning network in the form of pairing any two sets of samples according to S3.1 and S3.3. Calculate the difference loss between the predicted output and the reference result of the two sub-modules according to steps S3.2 and S3.4 respectively.

[0023] S3.6. Iteratively optimize the model parameters by minimizing the above-mentioned difference loss.

[0024] The beneficial effects of this invention are that it fully utilizes the inherent coupling between underwater image enhancement and image super-resolution tasks, and employs a dual-branch collaborative processing strategy to design a feature association matching module, effectively improving the performance of both the enhancement and super-resolution networks. By optimizing the loss function that fuses the two-stage pixel value difference loss and structural similarity loss, end-to-end training of the underwater image collaborative enhancement and super-resolution model is achieved. During the model deployment phase, any pair of low-resolution original degraded underwater images is used as input to the model, and after passing through the neural network model, a high-resolution clear underwater image is output. This method is innovative, produces excellent image visual quality and resolution restoration, and can be widely applied to operational scenarios requiring visual enhancement of images in harsh underwater imaging environments or when image compression results in low resolution. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating an embodiment of the underwater image collaborative enhancement and super-resolution method of the present invention;

[0026] Figure 2 This is a detailed flowchart illustrating an embodiment of the underwater image collaborative enhancement and super-resolution method of the present invention;

[0027] Figure 3 This is a schematic diagram of the framework of an underwater image collaborative enhancement and super-resolution system according to an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of the training data preparation process for an underwater image enhancement super-resolution network according to an embodiment of the present invention.

[0029] Figure 5 This is a schematic diagram of the underwater image collaborative enhancement and super-resolution depth network according to an embodiment of the present invention. Detailed Implementation

[0030] The present invention will be further described below with reference to specific embodiments.

[0031] like Figure 1 , Figure 2 and Figure 3 As shown, this method comprises three main modules: training data preparation, collaborative super-resolution model training, and model deployment. Data preparation is a prerequisite for collaborative super-resolution model training. The parameter-optimized collaborative super-resolution model obtained through the training process can be deployed to improve underwater imaging quality and resolution.

[0032] An underwater image collaborative enhancement and super-resolution method includes:

[0033] S1. Construct the training dataset required for training the deep learning neural network model using an underwater image dataset with visually enhanced reference images (such as the UIEB dataset, SUIM-E dataset, or other underwater image datasets with visually enhanced reference images). The training dataset includes low-resolution raw underwater degraded images X. l , with X l The corresponding low-resolution reference ground truth image Y l High-resolution raw degraded underwater image X h , with X h The corresponding high-resolution reference ground truth image Y h ;

[0034] S2. Construct an underwater image collaborative enhancement and super-resolution deep learning network, combining paired X... l As input to this deep network, using its corresponding Y h Constraint optimization is performed on the predicted image output by the deep learning neural network model;

[0035] S3. By minimizing pixel value difference loss and structural similarity loss, the training process of the deep learning neural network model is constrained and optimized to train the underwater image collaborative enhancement and super-resolution deep learning network, thereby obtaining the deep learning neural network model.

[0036] S4. Place any pair of X l As input to the model, it outputs Y after passing through a deep learning neural network model. h .

[0037] like Figure 4S1 includes: S1.1. Using an image downsampling method (e.g., the Bicubic method), X... h Perform downsampling to obtain the corresponding X l ;

[0038] S1.2. Using an image downsampling method (e.g., the Bicubic method), the Y... h Perform downsampling to obtain the corresponding Y l .

[0039] like Figure 5 S2 includes: S2.1. Constructing an underwater image collaborative enhancement and super-resolution deep learning network structure, which consists of a cascaded underwater image collaborative enhancement submodule CoE-M and an underwater image collaborative super-resolution submodule CoSR-M;

[0040] S2.2. Construct CoE-M and CoSR-M based on a dual-branch twin deep network structure. The twin structure is connected by a feature matching association submodule to realize feature information transmission.

[0041] The image collaborative enhancement submodule CoE-M consists of a Siamese deep encoder-decoder structure (such as U-Net, Dense-Net, etc.) and a feature association matching module. The Siamese deep encoder-decoder structure is used to obtain the initial enhancement result; the feature association matching module uses cross-correlation to calculate the similarity of bi-branch features and merges feature channels based on nearest neighbor relationships, thereby achieving feature information transfer and collaborative association at the image enhancement level.

[0042] The underwater image collaborative enhancement submodule CoSR-M consists of a Siamese residual module (composed of cascaded ResNet basic units), a feature association matching module, and an upsampling-convolution module. The Siamese residual module acquires deep feature representations for super-resolution tasks; the feature association matching module calculates the similarity of bi-branch features using cross-correlation and merges feature channels based on nearest neighbor relationships, thereby achieving feature information transfer and collaborative association at the super-resolution level; and the upsampling-convolution module improves feature resolution and outputs super-resolution underwater image pairs.

[0043] S3 includes: S3.1. A pair of X l Let it be X l,1 and X l,2 , as input to underwater image collaborative enhancement and super-resolution deep learning networks;

[0044] S3.2. Underwater Image Co-enhancement and Super-resolution Deep Learning Network outputs a pair of model-predicted low-resolution clear underwater images, denoted as... and Calculate separately and Between, Y l,1 and Y l,2 Loss of pixel value difference between them;

[0045] With enhanced image Compared with reference enhancement image Y l,1 Pixel value difference loss Taking the calculation of as an example, the formula is as follows: Where i is the element position index of the output image data, and H, W, and C are the height, width, and number of channels of the output image, respectively. Super-resolution pixel value difference loss. Calculation and The calculation is similar.

[0046] S3.3. Place a pair of X h Let it be X h,1 and X h,2 , as input to underwater image collaborative enhancement and super-resolution deep learning networks;

[0047] S3.4. Underwater Image Co-enhancement and Super-resolution Deep Learning Network Outputs a Pair of High-Resolution, Clear Underwater Images Predicted by the Model, denoted as... and Calculate separately and Between, Y h,1 With Y h,2 Pixel value difference loss and structural similarity loss between them;

[0048] With super-resolution images Compared with the reference super-resolution image Y h,1 Structural similarity loss between For example, the formula for calculation is as follows:

[0049] Where p is the center pixel of an 11×11 pixel image patch, and P is the total number of pixels in the image. SSIM(p) is the image... With Y h,1 The structural similarity between image patches centered at p is denoted as follows: and These are the pixel mean and standard deviation of the former, respectively. and These are the pixel mean and standard deviation of the latter, respectively. C1 and C2 are constants, and a feasible parameter setting combination is C1 = 0.01, C2 = 0.03.

[0050] S3.5. Input the training dataset obtained in S1 into the underwater image co-enhancement and super-resolution deep learning network in the form of pairing any two sets of samples according to S3.1 and S3.3. Calculate the difference loss between the predicted output and the reference result of the two sub-modules according to steps S3.2 and S3.4 respectively.

[0051] S3.6. By minimizing the above-mentioned difference loss, the model parameters are iteratively optimized to obtain the optimal co-enhancement and super-resolution results on the training set, thereby obtaining the trained underwater image co-enhancement and super-resolution model.

[0052] The model's final objective function It is the sum of three constraint terms, as shown in the following formula:

[0053]

[0054] S4 includes: taking any pair of low-resolution original degraded underwater images as input to the model, and outputting a high-resolution clear underwater image after passing through the neural network model; according to the task requirements of the operation, deploying the model on the device terminal, taking the underwater image acquired and transmitted from the front-end device as input, and improving the visual quality and resolution of the input image on the device terminal.

[0055] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A method for collaborative enhancement and super-resolution of underwater images, characterized in that, include: S1. Using an underwater image dataset with visually enhanced reference images, construct the training dataset required for training the deep learning neural network model. The training dataset includes low-resolution original underwater degraded images, and... Corresponding low-resolution reference ground truth image High-resolution original degraded underwater images ,and Corresponding high-resolution reference ground truth image ; S2. Construct an underwater image collaborative enhancement and super-resolution deep learning network, using paired deep learning methods. As input to this deep network, using its corresponding Constraint optimization is performed on the predicted image output by the deep learning neural network model; S3. By minimizing pixel value difference loss and structural similarity loss, the training process of the deep learning neural network model is constrained and optimized to train the underwater image collaborative enhancement and super-resolution deep learning network, thereby obtaining the deep learning neural network model. S4. Place any pair As input to the model, it outputs after passing through the deep learning neural network model. ; S2 includes: S2.

1. Construct an underwater image collaborative enhancement and super-resolution deep learning network structure, which consists of a cascaded underwater image collaborative enhancement submodule CoE-M and an underwater image collaborative super-resolution submodule CoSR-M; S2.

2. Construct CoE-M and CoSR-M based on a dual-branch Siamese deep network structure. The Siamese structure is connected by a feature matching and association submodule to realize feature information transmission. CoE-M consists of a twin-structure deep encoder-decoder structure and a feature association matching module. The twin-structure deep encoder-decoder structure is used to obtain the initial enhancement result. The feature association matching module uses cross-correlation to calculate the similarity of the two-branch feature associations and merges feature channels according to the nearest neighbor relationship, thereby realizing the transmission and collaborative association of feature information at the image enhancement level. CoSR-M consists of a twin residual module, a feature association and matching module, and an upsampling-convolution module. The twin residual module is used to obtain deep feature representations for super-resolution tasks. The feature association and matching module uses cross-correlation to calculate the similarity of bi-branch feature associations and merges feature channels according to nearest neighbor relationships, thereby realizing the transfer and collaborative association of feature information at the image super-resolution level. The upsampling-convolution module is used to improve feature resolution and output super-resolution underwater image pairs. S3 includes: S3.

1. Place a pair Recorded as and , as input to underwater image collaborative enhancement and super-resolution deep learning networks; S3.

2. Underwater Image Co-enhancement and Super-resolution Deep Learning Network outputs a pair of model-predicted low-resolution clear underwater images, denoted as... and Calculate separately and between, and Loss of pixel value difference between them; S3.

3. Place a pair Recorded as and , as input to underwater image collaborative enhancement and super-resolution deep learning networks; S3.

4. Underwater Image Co-enhancement and Super-resolution Deep Learning Network Outputs a Pair of High-Resolution, Clear Underwater Images Predicted by the Model, denoted as... and Calculate separately and between, and Pixel value difference loss and structural similarity loss between them; S3.

5. Input the training dataset obtained in S1 into the underwater image co-enhancement and super-resolution deep learning network in the form of pairing any two sets of samples according to S3.1 and S3.

3. Calculate the difference loss between the predicted output and the reference result of the two sub-modules according to steps S3.2 and S3.4 respectively. S3.

6. Iteratively optimize the model parameters by minimizing the above-mentioned difference loss.

2. The underwater image collaborative enhancement and super-resolution method according to claim 1, characterized in that, S1 includes: S1.

1. Using image downsampling methods, for Perform downsampling to obtain its corresponding ; S1.

2. Using image downsampling methods, for Perform downsampling to obtain its corresponding .

Images