An underwater polarization image restoration system and restoration method based on deep learning

Through the underwater polarized image restoration system based on deep learning, the problem of low imaging quality in traditional underwater optical imaging technology is solved, and high-quality image restoration effect is achieved.

CN113538279BActive Publication Date: 2024-11-12ZHEJIANG SCI-TECH UNIV
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202110815882.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-07-19
Publication Date
2024-11-12
Estimated Expiration
2041-07-19

AI Technical Summary

Technical Problem

Traditional underwater optical imaging technology is affected by scattered light, resulting in low imaging quality, low visibility, low contrast and lack of image details in image displays. The existing underwater image restoration methods have problems with limited recovery capabilities and image distortion.

Method used

Using a deep learning-based underwater polarized image restoration system, the estimation of related parameters K is carried out by introducing circular polarized images and four polarized images, a convolutional neural network is built, and the images are restored using multi-scale features are restored, and the restored images are generated by combining multiplication and addition layers. .

Benefits of technology

It significantly improves the image imaging quality in turbid underwater environments, enhances image contrast and detail recovery capabilities, and improves the overall quality of the image.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113538279B_ABST
    Figure CN113538279B_ABST
Patent Text Reader

Abstract

The present invention provides an underwater polarization image restoration system and restoration method based on deep learning. The method comprises the following steps: Step 1, building an underwater polarization imaging system to shoot clear underwater intensity images and polarization images of water with different turbidity; Step 2, sorting and naming each group of polarization images to establish a large number of data sets. Step 3, dividing the training set, validation set, and test set in a ratio of 0.8:0.1:0.1; Step 4, designing a convolutional neural network suitable for underwater polarization image restoration; Step 5, generating a restored image Step 6 uses the training set to train the above network model; Step 7, using the test set to test the image quality effect after restoration by the network model. Compared with the prior art, the present invention can use a convolutional neural network to restore high turbidity underwater images, and the restored image effect is better.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to polarization imaging technology, and in particular to an underwater polarization image restoration system and a restoration method thereof based on deep learning in a complex underwater environment. Background Art

[0002] As an important method for studying the underwater world, underwater imaging has been widely used in underwater target detection and identification, underwater archaeology, underwater environmental monitoring and other fields. However, in traditional underwater optical imaging, the imaging quality is greatly limited due to the influence of scattered light and the interference of turbid particles. These images usually show low visibility, low contrast and lack of image details. In particular, backscattered light has a significant impact on image contrast.

[0003] At present, underwater image restoration methods are mainly divided into two categories, including digital image processing restoration technology and optical restoration technology. Digital image processing restoration technology usually has a simple algorithm and fast processing speed, but its ability to restore target details is limited, and there is a problem of partial image distortion. Optical restoration technology is mainly based on the inversion of physical degradation models to obtain the light intensity restoration of the real scene, including polarization optical imaging, multi-spectral fusion restoration, etc. Polarization optical imaging has a simple structure and is easy to use. It is one of the most effective optical restoration technologies at present, but the imaging effect still has certain limitations, which affects further application. Summary of the invention

[0004] In view of the shortcomings of the background technology, the present invention provides an underwater polarization image restoration system and restoration method based on deep learning. A circular polarization image is introduced into the input port, and four polarization images are used to estimate the relevant parameter K, thereby restoring the image, which can effectively improve the image quality.

[0005] In view of the shortcomings of the background technology, the present invention provides an underwater polarization image restoration system based on deep learning, comprising an underwater active imaging system; the underwater active imaging system comprises an active light source arranged on the outside of an underwater target (7), a polarization modulation system and an image acquisition system; the active light source comprises a laser (1) aimed at the underwater target (7) for laser irradiation, and the polarization modulation system comprises a first polarizer (2), a first quarter-wave plate (3) and a beam expander (4) arranged on the irradiation beam path of the laser (1), and is used for the light beam emitted by the laser (1) to sequentially pass through the first polarizer (2), the second quarter-wave plate (10) and the beam expander (4) to illuminate the underwater target (7). The laser beam is projected onto an underwater target (7); the image acquisition system comprises a first CMOS camera (11) aimed at the underwater target (7), a second quarter wave plate (10), a second polarizing plate (8) controlled to rotate by a stepping motor, and a computer (13); the second quarter wave plate (10) and the second polarizing plate (8) are respectively arranged on the light beam reflection path between the first CMOS camera (11) and the underwater target (7), so that the light beam emitted by the laser (1) is reflected by the underwater target (7) and then passes through the second polarizing plate (8) and the second quarter wave plate (10) in sequence before reaching the first CMOS camera (11); and the computer (13) is electrically connected to the stepping motor.

[0006] Furthermore, a beam splitter (9) and a second CMOS camera (12) are arranged between the second polarizing plate (8) and the second quarter-wave plate (10), and the second CMOS camera (12) is aligned with the beam splitter (9); a light beam reflected by the underwater target (7) and passing through the second polarizing plate (8) is split into two light beams by the beam splitter (9), one of which passes through the second quarter-wave plate (10) and reaches the first CMOS camera (11), and the other light beam directly reaches the second CMOS camera (12).

[0007] The second polarizer (8) is provided with a polarizer base (not shown) connected thereto, and the polarizer base is controlled to rotate by the stepping motor to automatically adjust the rotation angle of the second polarizer (8).

[0008] Preferably, the laser is a 532nm blue-green laser.

[0009] The computer (13) is used for receiving and processing image information captured by the first CMOS camera (11) and the second CMOS camera (12).

[0010] The present invention also provides a restoration method for an underwater polarization image restoration system based on deep learning:

[0011] The recovery steps include:

[0012] Step 1, constructing an underwater active imaging system; the light beam emitted by the laser (1) passes through the first polarizer (2), the second quarter-wave plate (10) and the beam expander (4) in sequence and then irradiates the underwater target (7); the underwater target (7) reflects the light beam emitted by the laser (1) and then passes through the second polarizer (8), the beam splitter (9) and the second quarter-wave plate (10) in sequence and then reaches the first CMOS camera (11); wherein the beam splitter (9) splits the reflected light beam of the underwater target (7) into another light beam that directly reaches the second CMOS camera (12);

[0013] Step 2, establish a data set; first take a clear intensity image underwater, then add different concentrations of skim milk, use a stepper motor to control the front polarizer of the camera, rotate it to 0°, 45°, and 90°, and add a quarter-wave plate to obtain four images of 0°, 45°, 90°, and circular polarization;

[0014] Step 3: Crop the image; expand the dataset by flipping, and divide the obtained images into training set and validation set in the ratio of 0.8:0.1:0.1;

[0015] Step 4: Design a convolutional neural network. Nine convolutional layers are used to form multi-scale features by fusing filters of different sizes. Four images of 0°, 45°, 90°, and circular polarization taken underwater in turbid water are used as inputs of the convolutional neural network. The K estimation module is obtained to calculate and generate the restored image.

[0016] Step 5: After obtaining the K estimation module, the image generation module consists of a multiplication layer and an addition layer, and generates a restored image through calculation;

[0017] Step 6: Use the convolutional neural network in step 4 to train the training set shot in step 3, and use the verification set in step 3 to verify the image restoration effect of the neural network to obtain a convolutional neural network model;

[0018] Step 7: Use the test set in step 3 to test the convolutional neural network model trained in step 4, and use the image enhancement index to measure the image restoration effect.

[0019] In step 4, the feature information of each image is first refined, and the coarse-scale network features are connected to the middle layer of the fine-scale network; the feature information of the first and second layers is connected using the connection block of the third layer; the feature information is further refined through the convolution layer; the connection block of the fifth layer connects the feature information of the second and fourth layers; the sixth layer connects the feature information of the first, fourth, and fifth layers; and finally the 3*3 convolution layer and the linear activation function Get the K estimation module.

[0020] Beneficial Effects

[0021] The present invention applies multi-scale features to the estimation of the K estimation module, thereby effectively improving the image quality in turbid underwater environments. The circular polarization image is introduced into the input port, and the four polarization images are used to estimate the relevant parameter K, thereby restoring the image, which can effectively improve the image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The present invention is a flowchart of the overall implementation of a restoration method for an underwater polarization image restoration system based on deep learning.

[0023] Figure 2 Schematic diagram of the composition structure of an underwater polarization image restoration system based on deep learning in the present invention.

[0024] Figure 3 Schematic diagram of the convolutional neural network established in the present invention.

[0025] Figure 4 Comparison of the original image (left), the image restored based on the physical model (middle), and the image restored by this method (right).

[0026] Figure Number:

[0027] Laser 1, first polarizer 2, first quarter wave plate 3, beam expander 4, glass water tank 5, simulated underwater environment 6, target object 7, second polarizer 8, beam splitter 9, second quarter wave plate 10, first CMOS camera 11, second CMOS camera 12, computer 13. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and beneficial technical effects of the present invention clearer, the technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be understood that the embodiments described in this specification are only for explaining the present invention, not for limiting the present invention, and the parameters, proportions, etc. of the embodiments can be selected according to local conditions without substantial impact on the results.

[0029] Depend on Figure 1-Figure 4 As shown, the present invention provides an underwater polarization image restoration system based on deep learning, an underwater polarization imaging detection system, including an active light source arranged on the outside of an underwater target 7, a polarization modulation system and an image acquisition system; the active light source includes a laser 1 aimed at the underwater target 7 for laser irradiation, wherein the laser is preferably a 532nm blue-green laser.

[0030] The polarization modulation system includes a first polarizer 2, a first quarter wave plate 3 and a beam expander 4 arranged on the beam path of the laser 1, and is used for the beam emitted by the laser 1 to pass through the first polarizer 2, the second quarter wave plate 10 and the beam expander 4 in sequence and then irradiate the underwater target 7; the image acquisition system includes a first CMOS camera 11 aimed at the underwater target 7, a second quarter wave plate 10, a second polarizer 8 controlled to rotate by a stepping motor and a computer 13; the second quarter wave plate 10 and the second polarizer 8 are respectively arranged on the beam reflection path between the first CMOS camera 11 and the underwater target 7, and is used for the underwater target 7 to reflect the beam emitted by the laser 1 and then pass through the second polarizer 8 and the second quarter wave plate 10 in sequence and then reach the first CMOS camera 11. The computer 13 is used to receive and process the image information captured by the first CMOS camera 11 and the second CMOS camera 12.

[0031] A beam splitter 9 and a second CMOS camera 12 are arranged between the second polarizer 8 and the second quarter-wave plate 10, and the second CMOS camera 12 is aligned with the beam splitter 9; the light beam reflected by the underwater target 7 and passing through the second polarizer 8 is split into two light beams by the beam splitter 9, one of which passes through the second quarter-wave plate 10 to reach the first CMOS camera 11, and the other light beam directly reaches the second CMOS camera 12, so as to simultaneously capture a linear polarization image and a circular polarization image.

[0032] The second polarizer 8 is provided with a polarizer base (not shown) connected thereto, and the polarizer base is controlled to rotate by a stepping motor. The computer 13 is electrically connected to the stepping motor for automatically adjusting the rotation angle of the second polarizer 8 .

[0033] Further preferably, the image processing module built into the computer 13 uses a GeForce RTX3070 graphics card to speed up image processing.

[0034] The present invention also provides a restoration method for an underwater polarization image restoration system based on deep learning:

[0035] The recovery steps include:

[0036] Step 1: Build an underwater active imaging system to collect clear underwater images. Modulate circularly polarized light as the light source 1, select a PMMA (polymethyl methacrylate) glass water tank 5 to obtain a simulated underwater environment 6, place the target 7 in the glass water tank 5, and add different concentrations of skim milk to obtain the experimental sample environment.

[0037] In this embodiment, a total of 124 groups of polarization images are collected, and each group of images includes 0°, 45°, 90° and circular polarization images. In this embodiment, a 532nm blue-green laser is used as an active light source, which passes through the first polarizer 2, the first quarter-wave plate 3, and the beam expander 4 and then irradiates the underwater target 7; after being reflected by the underwater target 7, it passes through the second polarizer 8 of the image acquisition system, passes through the beam splitter 9, and one beam passes through the quarter-wave plate 10 to reach the first CMOS camera 11, and the other beam directly reaches the second CMOS camera 12

[0038] Step 2, establish a data set; first take a clear intensity image underwater, then add different concentrations of skim milk, use a stepper motor to control the front polarizer of the camera, rotate it to 0°, 45°, and 90°, and add a quarter-wave plate to obtain four images of 0°, 45°, 90°, and circular polarization;

[0039] Step 3: Crop the image; expand the dataset by flipping, and divide the obtained images into training set and validation set in the ratio of 0.8:0.1:0.1;

[0040] Step 4: Design a convolutional neural network. Nine convolutional layers are used to form multi-scale features by fusing filters of different sizes. Four images of 0°, 45°, 90°, and circular polarization taken underwater in turbid water are used as inputs of the convolutional neural network. The K estimation module is obtained to calculate and generate the restored image.

[0041] Step 5: After obtaining the K estimation module, the image generation module consists of a multiplication layer and an addition layer, and generates a restored image through calculation;

[0042] Step 6: Use the convolutional neural network in step 4 to train the training set shot in step 3, and use the verification set in step 3 to verify the image restoration effect of the neural network to obtain a convolutional neural network model;

[0043] Step 7: Use the test set in step 3 to test the convolutional neural network model trained in step 4, and use the image enhancement index to measure the image restoration effect.

[0044] Among them, in step 4, the feature information of each image is first refined, and the coarse-scale network features are connected to the middle layer of the fine-scale network; the feature information of the first and second layers is connected by the connection block of the third layer; the feature information is further refined through the convolution layer; the connection block of the fifth layer connects the feature information of the second and fourth layers; the sixth layer connects the feature information of the first, fourth and fifth layers; and finally the 3*3 convolution layer and the linear activation function The K estimation module is obtained. It can be intuitively seen from the experimental results that the present invention significantly improves the turbid underwater image effect.

[0045] In order to quantitatively evaluate the image quality, EME (the value of measure of enhancement) and image contrast (IC) are used as the measurement criteria to evaluate the image enhancement effect. The larger the EME and IC, the higher the image quality. The comparison results are shown in the following table:

[0046] Original image Physical model restoration diagram The present invention EME 0.639 0.666 2.452 IC 0.097 0.176 0.551

[0047] Although the embodiments of the present invention are described in conjunction with the accompanying drawings, the patent owner may make variations or modifications within the scope of the appended claims. As long as they do not exceed the protection scope described in the claims of the present invention, they are all within the protection scope of the present invention.

Claims

1. A method for underwater polarization image restoration based on deep learning, characterized in that: The recovery steps include: Step 1, constructing an underwater active imaging system; a light beam emitted by a laser (1) passes through a first polarizer (2), a first quarter-wave plate (3) and a beam expander (4) in sequence and then irradiates an underwater target (7); the underwater target (7) reflects the light beam emitted by the laser (1) and then passes through a second polarizer (8), a beam splitter (9) and a second quarter-wave plate (10) in sequence and then reaches a first CMOS camera (11); wherein the beam splitter (9) splits the reflected light beam of the underwater target (7) into another light beam that directly reaches a second CMOS camera (12); Step 2, establish a data set; first take a clear intensity image underwater, then add different concentrations of skim milk, use a stepper motor to control the front polarizer of the camera, rotate it to 0°, 45°, and 90°, and add a quarter-wave plate to obtain four images of 0°, 45°, 90°, and circular polarization; Step 3: Crop the image; expand the dataset by flipping it, and divide the obtained images into training set and validation set in the ratio of 0.8:0.1:0.1; Step 4: Design a convolutional neural network. Nine convolutional layers are used to form multi-scale features by fusing filters of different sizes. Four images of 0°, 45°, 90°, and circular polarization taken underwater in turbid water are used as inputs of the convolutional neural network. Get the K estimation module, so as to calculate and generate the restored image; Step 5: After obtaining the K estimation module, the image generation module consists of a multiplication layer and an addition layer to generate a restored image through calculation; Step 6: Use the convolutional neural network in step 4 to train the training set shot in step 3, and use the verification set in step 3 to verify the image restoration effect of the neural network to obtain a convolutional neural network model; Step 7: Use the test set in step 3 to test the convolutional neural network model trained in step 4, and use the image enhancement index to measure the image restoration effect.

2. The method for underwater polarization image restoration based on deep learning according to claim 1, characterized in that: In step 4, the feature information of each image is first refined, and the coarse-scale network features are connected to the middle layer of the fine-scale network; the feature information of the first and second layers is connected using the connection block of the third layer; the feature information is further refined through the convolution layer; the connection block of the fifth layer connects the feature information of the second and fourth layers; the sixth layer connects the feature information of the first, fourth, and fifth layers; and finally the 3*3 convolution layer and the linear activation function , and get the K estimation module.

3. An underwater polarization image restoration system based on deep learning, characterized in that: A method for restoring an underwater polarization image based on deep learning as claimed in any one of claims 1 to 2, comprising an underwater active imaging system; the underwater active imaging system comprises an active light source arranged outside an underwater target (7), a polarization modulation system and an image acquisition system; the active light source comprises a laser (1) aimed at the underwater target (7) for laser irradiation, the polarization modulation system comprises a first polarizer (2), a first quarter wave plate (3) and a beam expander (4) arranged on the irradiation beam path of the laser (1), and is used for irradiating the light beam emitted by the laser (1) through the first polarizer (2), the first quarter wave plate (3) and the beam expander (4) in sequence on the underwater target (7). The image acquisition system comprises a first CMOS camera (11) aimed at the underwater target (7), a second quarter wave plate (10), a second polarizing plate (8) controlled to rotate by a stepping motor, and a computer (13); the second quarter wave plate (10) and the second polarizing plate (8) are respectively arranged on the light beam reflection path between the first CMOS camera (11) and the underwater target (7), so as to allow the light beam emitted by the laser (1) to be reflected by the underwater target (7) and then pass through the second polarizing plate (8) and the second quarter wave plate (10) in sequence before reaching the first CMOS camera (11); and the computer (13) is electrically connected to the stepping motor.

4. The underwater polarization image restoration system based on deep learning according to claim 3, characterized in that: A beam splitter (9) and a second CMOS camera (12) are arranged between the second polarizing plate (8) and the second quarter-wave plate (10), and the second CMOS camera (12) is aligned with the beam splitter (9); the light beam reflected by the underwater target (7) and passing through the second polarizing plate (8) is split into two light beams by the beam splitter (9), one of the light beams passing through the second quarter-wave plate (10) reaching the first CMOS camera (11), and the other light beam directly reaching the second CMOS camera (12).

5. The underwater polarization image restoration system based on deep learning according to claim 3, characterized in that: The second polarizer (8) is provided with a polarizer base connected thereto, and the polarizer base is controlled to rotate by the stepping motor, so as to automatically adjust the rotation angle of the second polarizer (8).

6. The underwater polarization image restoration system based on deep learning according to claim 3, characterized in that: The laser (1) adopts a 532nm blue-green laser.

7. The underwater polarization image restoration system based on deep learning according to claim 3, characterized in that: The computer (13) is used to receive and process image information captured by the first CMOS camera (11) and the second CMOS camera (12).

Citation Information

Patent Citations

  • Turbid underwater gray level image restoration method based on deep neural network and polarization imaging

    CN111429378A

  • Underwater polarization imaging detection system

    CN215986754U

  • Microscopic scattering polarization imaging surface defect measuring device and measuring method

    WO2021003802A1