A polarization recovery imaging method based on transform mueller matrix network

Abstract

The application discloses a polarization restoration imaging method based on a transform Mueller matrix network, and comprises the following steps: step one, shooting a clear polarization image of an underwater target object and a polarization image of the target object under different turbidity; step two, in view of the loss of spatial resolution caused by using a focal plane polarization camera, a targeted interpolation method is used to convert the clear polarization image of the underwater target object into a clear intensity image of the underwater target object; according to the idea of a Mueller matrix, a transform Mueller matrix network is constructed by combining a reverse residual convolution structure and a channel attention mechanism; a data set is established in a simulated environment; the polarization information of the image is fully utilized; the network is trained to construct a network capable of outputting a corresponding transform Mueller matrix according to an input foggy or turbid polarization image; the transform Mueller matrix output by the network is used to restore the image, and good restoration effect can be achieved, and stable performance can be maintained in various environments.

Description

Technical Field

[0001] The present invention relates to the technical field of polarization imaging, and in particular to a polarization restoration imaging method based on a transformation Mueller matrix network. Background Art

[0002] Computer vision technology has greatly benefited various industries. Whether it's traffic monitoring on roads, image cameras in autonomous vehicles, or cameras in underwater exploration equipment, all require capturing images and analyzing them using computer vision to make decisions. However, impurities in both air and underwater media can severely interfere with camera images, resulting in low contrast, low brightness, and blurred object details. This makes image-based object detection and analysis difficult underwater. Currently, methods for restoring foggy and turbid images fall into two main categories: those based on physical models and those based on non-physical models. Physical-based methods consider image degradation processes, mathematically modeling the fog or underwater imaging process, estimating model parameters, and then restoring the image through inverse decomposition. Alternatively, they utilize deep learning techniques to learn mapping functions and then restore the image. Non-physical-based image enhancement methods include histogram equalization, color correction, and fusion-based methods. Both techniques improve image quality to a certain extent. In contrast, image enhancement generally enhances visual perception and is biased towards subjective judgment, leaving lost details unrecovered. Image restoration methods, on the other hand, model the image based on image degradation and design a mapping function to restore the original image. Polarization imaging is a widely researched image restoration technique, but its imaging performance still has certain limitations. The results vary across different environments, resulting in unclear image quality and limited applicability, hindering further application. Summary of the Invention

[0003] In order to overcome the defects in the above-mentioned prior art, the present invention provides a polarization restoration imaging method based on a transformation Mueller matrix network. A data set is established in a simulated environment, and the polarization information of the image is fully utilized. The network is trained to construct a network that can output a corresponding transformation Mueller matrix based on the input foggy or turbid polarized image. Using the transformation Mueller matrix output by the network to repair the image can achieve good restoration effects and maintain stable performance in a variety of environments.

[0004] Technical Solution

[0005] A polarization restoration imaging method based on a transformation Mueller matrix network comprises the following steps:

[0006] Step 1: Capture clear polarization images of underwater targets and polarization images of targets at different concentrations;

[0007] Step 2: To address the spatial resolution loss caused by using a split-focal plane polarization camera, a targeted interpolation method is used to convert the clear polarization image of the underwater target into a clear intensity image of the underwater target;

[0008] Step 3. Based on the idea of ​​transforming the Mueller matrix, an inverted residual convolutional neural network is constructed. The overall network structure is designed according to the U-shaped structure. After the feature map is input, it will first pass through four encoders. The encoder contains an inverted residual polarization attention channel module and a downsampling layer to extract features, and then upsampled through four decoder modules. The decoder module also contains an inverted residual polarization attention channel module and an upsampling layer, which contains a channel attention module for extracting polarization images from different channels and assigning different weights to different channels; the inverted residual structure is used to reduce the loss of high-dimensional information after passing through the activation function. The DW convolution and PW convolution modules are used to reduce the amount of network calculation and optimize network performance. The network loss function contains three parts: edge loss, content loss, and pixel loss. The total loss function is obtained by weighting the edge loss, content loss, and pixel loss.

[0009] Step 4: Train the network. Use the data processed in step 2 to train the inverse residual convolutional neural network constructed in step 3. The trained inverse residual convolutional neural network can output the corresponding repaired Mueller matrix based on the input foggy or turbid polarized image.

[0010] Step 5: Restore the image. Use the restored Mueller matrix output by the inverse residual convolutional neural network trained in step 4 to repair the foggy or turbid polarized image, and finally obtain a significantly improved restored image.

[0011] Furthermore, in step 1, a clear polarization image of an underwater target is captured: the light beam emitted by the light source passes through the polarizer and the beam expander of the polarization modulation system in sequence and then illuminates the target in the water. After being reflected by the target, it reaches the focal plane polarization camera, thereby obtaining a clear polarization image of the target.

[0012] Furthermore, in step 1, polarization images of the target object under different turbidity levels are captured: skim milk is gradually added to the water, and a 5-megapixel polarization Gigabit Ethernet industrial camera is used to capture the target object under different turbidity concentrations, thereby obtaining polarization images of the target object in the turbid underwater environment at different polarization angles and different turbidity concentrations.

[0013] Furthermore, a 532nm blue-green laser is used as the light source.

[0014] Furthermore, the inverted residual polarized attention channel module in step three consists of two parts. The first part is an inverted residual structure consisting of three convolutional layers, which includes a 1*1 convolution to increase the dimension of the feature map, a 3*3 convolution to extract features, and a 1*1 convolution to reduce the dimension of the feature map. Finally, it also includes a BN layer for normalization. The second part is a channel attention structure, which includes an average pooling layer, a one-dimensional convolution layer and a Sigemoid activation function, which avoids dimensionality reduction and effectively performs cross-channel interaction.

[0015] Furthermore, the inverse residual convolutional neural network constructed in step three is designed based on the idea of ​​transforming the Mueller matrix. The inverse residual polarization attention channel module of the network can extract the corresponding feature information so that the network can find the relationship between the Mueller matrix of the turbid image and the Mueller matrix of the clear polarized image based on the corresponding feature information, and output the transformed Mueller matrix corresponding to the turbid image accordingly.

[0016] Furthermore, the repaired Mueller matrix in step five is a physical model proposed based on the idea of ​​the Mueller matrix, which describes the transformation relationship between the Mueller matrices of a clear image and a turbid image.

[0017] Furthermore, the target object is placed in a glass tank filled with water to simulate an underwater environment.

[0018] Furthermore, the glass cylinder is made of PMMA (polymethyl methacrylate).

[0019] Beneficial effects

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] Based on the idea of ​​the Mueller matrix, a transformation Mueller matrix network was constructed by combining the inverted residual convolution structure and the channel attention mechanism. A dataset was established in a simulated environment, and the polarization information of the image was fully utilized. The network was trained to construct a repair Mueller matrix that can output the corresponding output based on the input foggy or turbid polarized image. Using this repair Mueller matrix to repair the image can achieve good restoration effects and maintain stable performance in a variety of environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a schematic diagram of the overall process of a polarization restoration imaging method based on a transform Mueller matrix network according to the present invention;

[0023] Figure 2 Schematic diagram of the experimental device of the present invention;

[0024] Figure 3 To generate a network structure diagram;

[0025] Figure 4 Schematic diagram of the inverted residual polarization channel attention block structure;

[0026] Figure 5 The experimental results of the Mueller matrix network transformation are shown in Figure 1. (a) is a high-turbidity image, (b) is the image restored by the present invention, and (c) is a clear intensity image.

[0027] Figure 6 This is a table of restoration effect evaluation indicators EME, SSIM and PSNR.

[0028] Reference numerals

[0029] Light source 1, polarizer 2, beam expander 3, glass cylinder 4, underwater environment 5, target object 6, focal plane polarization camera 7. DETAILED DESCRIPTION

[0030] To better illustrate the present invention, the following is a detailed description with reference to the accompanying drawings and implementation examples:

[0031] have Figures 1-6 As shown, the present invention discloses a polarization restoration imaging method based on a transformation Mueller matrix network, comprising the following steps:

[0032] Step 1: Capture clear polarization images of underwater targets and polarization images of targets at different turbidity levels;

[0033] Step 2: To address the spatial resolution loss caused by using a split-focal plane polarization camera, a targeted interpolation method is used to convert the clear polarization image of the underwater target into a clear intensity image of the underwater target;

[0034] Step 3. Based on the idea of ​​transforming the Mueller matrix, an inverted residual convolutional neural network is constructed. The overall network structure is designed according to the U-shaped structure. After the feature map is input, it will first pass through four encoders. The encoder contains an inverted residual polarization attention channel module and a downsampling layer to extract features, and then upsampled through four decoder modules. The decoder module also contains an inverted residual polarization attention channel module and an upsampling layer, which contains a channel attention module for extracting polarization images from different channels and assigning different weights to different channels; the inverted residual structure is used to reduce the loss of high-dimensional information after passing through the activation function. The DW convolution and PW convolution modules are used to reduce the amount of network calculation and optimize network performance. The network loss function contains three parts: edge loss, content loss, and pixel loss. The total loss function is obtained by weighting the edge loss, content loss, and pixel loss.

[0035] Step 4: Train the network. Use the data processed in step 2 to train the inverse residual convolutional neural network constructed in step 3. The trained inverse residual convolutional neural network can output the corresponding repaired Mueller matrix based on the input foggy or turbid polarized image.

[0036] Step 5: Restore the image. Use the restored Mueller matrix output by the inverse residual convolutional neural network trained in step 4 to repair the foggy or turbid polarized image, and finally obtain a significantly improved restored image.

[0037] Furthermore, in step one, a clear polarization image of an underwater target is captured: the light beam emitted by the light source 1 passes through the polarizer 2 and the beam expander 3 of the polarization modulation system in sequence and then illuminates the target 6 in the water. After being reflected by the target 6, it reaches the focal plane polarization camera 7, thereby obtaining a clear polarization image of the target 6.

[0038] Furthermore, in step 1, polarization images of the target object at different concentrations are captured: skim milk is gradually added to the water, and a 5-megapixel polarization Gigabit Ethernet industrial camera is used to capture the target object 6 in different turbidity concentration environments, thereby obtaining polarization images of the target object 6 in the turbid underwater environment at different polarization angles and different turbidity concentrations.

[0039] Furthermore, a 532nm blue-green laser is used as the light source 1 .

[0040] Furthermore, the inverted residual polarized attention channel module in step three consists of two parts. The first part is an inverted residual structure consisting of three convolutional layers, which includes a 1*1 convolution to increase the dimension of the feature map, a 3*3 convolution to extract features, and a 1*1 convolution to reduce the dimension of the feature map. Finally, it also includes a BN layer for normalization. The second part is a channel attention structure, which includes an average pooling layer, a one-dimensional convolution layer and a Sigemoid activation function, which avoids dimensionality reduction and effectively performs cross-channel interaction.

[0041] Furthermore, the inverse residual convolutional neural network constructed in step three is designed based on the idea of ​​transforming the Mueller matrix. The inverse residual polarization attention channel module of the network can extract the corresponding feature information so that the network can find the relationship between the Mueller matrix of the turbid image and the Mueller matrix of the clear polarized image based on the corresponding feature information, and output the transformed Mueller matrix corresponding to the turbid image accordingly.

[0042] Furthermore, the repaired Mueller matrix in step five is a physical model proposed based on the idea of ​​the Mueller matrix, which describes the transformation relationship between the Mueller matrices of a clear image and a turbid image.

[0043] Furthermore, the target object 6 is placed in a glass tank 4 filled with water to simulate an underwater environment 5 .

[0044] Furthermore, the glass cylinder 4 is made of PMMA (polymethyl methacrylate).

[0045] Specifically, step 1: using an underwater active imaging system, using linearly polarized light for active illumination, to capture clear polarization images of the underwater target 6 and polarization images of the target 6 at different concentrations. In this embodiment, a 532nm blue-green laser is used as the light source 1, and a PMMA (polymethyl methacrylate) glass cylinder 4 is selected;

[0046] The light beam emitted by the light source 1 passes through the polarizer 2 and the beam expander 3 of the polarization modulation system in sequence and then illuminates the target 6 in the water. After being reflected by the target 6, it reaches the focal plane polarization camera 7, which captures a clear intensity image of the underwater target 6. Skim milk is gradually added to the water, and a 5-megapixel polarization Gigabit Ethernet industrial camera is used to capture the target 6 in different turbidity concentration environments; thereby, polarization images of the target 6 at different polarization angles and different turbidity concentrations in the turbid underwater environment are obtained. In this embodiment, 20 groups of images are captured, and 20 different concentrations of milk are added to each group to simulate underwater environments 5 at 20 different concentrations. For each concentration, one polarization image of the target 6 is captured according to the shooting method in step 1 (including "mosaic" images with polarization directions of 0°, 45°, 90°, and 135°, in which every 4 pixels in the image contain pixels from 4 polarization directions);

[0047] Step 2: Create a dataset. Crop the images obtained in step 1, expand the dataset by flipping and rotating it, and then divide it into a training set, a validation set, and a test set in a ratio of 8:1:1.

[0048] Step 3. Construct an inverted residual polarization channel attention module. The inverted residual polarization attention channel module consists of two parts. The first part is an inverted residual structure consisting of three convolutional layers, which includes an 11 convolution to increase the dimension of the feature map, a 33 convolution to extract the features, and a 1*1 convolution to reduce the dimension of the feature map. Finally, it also includes a BN layer for normalization. The second part is a channel attention structure, which includes an average pooling layer, a one-dimensional convolution layer and a Sigmoid activation function, which avoids dimensionality reduction and effectively performs cross-channel interaction.

[0049] Step 4. Based on the idea of ​​transforming the Mueller matrix, an inverted residual convolutional neural network is constructed. The overall network structure is designed according to a U-shaped structure. After the feature map is input, it will first pass through four encoders. The encoder contains an inverted residual polarization attention channel module and a downsampling layer to extract features, and then upsampled through four decoder modules. The decoder module also contains an inverted residual polarization attention channel module and an upsampling layer, which contains a channel attention module for extracting polarization images from different channels and assigning different weights to different channels. The inverted residual structure is used to reduce the loss of high-dimensional information after passing through the activation function. The DW convolution and PW convolution modules are used to reduce the amount of network calculation and optimize network performance. The network loss function contains three parts: edge loss, content loss, and pixel loss. The total loss function is obtained by weighting the above three losses.

[0050] The experimental results show that the present invention can effectively restore polarization images taken in high-concentration turbid water. Combining the objective evaluation criteria EME (the value of measure of enhancement), SSIM (Structure Similarity Index Measure), PSNR (Peak Signal to Noise Ratio) and subjective perception, the image restoration effect is significant.

[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the technical solutions of the present invention have been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A polarization restoration imaging method based on a transform Mueller matrix network, characterized in that: The following steps are involved: Step 1: Capture clear polarization images of underwater targets and polarization images of targets at different turbidity levels; Step 2: To address the spatial resolution loss caused by using a split-focal plane polarization camera, a targeted interpolation method is used to convert the clear polarization image of the underwater target into a clear intensity image of the underwater target; Step 3. According to the idea of ​​transforming the Mueller matrix, an inverted residual convolutional neural network is constructed. The overall network structure is designed according to the U-shaped structure. After the feature map is input, it will first pass through four encoders. The encoder contains an inverted residual polarization attention channel module and a downsampling layer to extract features, and then upsampled through four decoder modules. The decoder module also contains an inverted residual polarization attention channel module and an upsampling layer, which contains a channel attention module for extracting polarization images from different channels and giving different weights to different channels. The inverted residual structure is used to reduce the loss of high-dimensional information after passing through the activation function. The DW convolution and PW convolution modules are used to reduce the amount of network calculation and optimize network performance. The network loss function contains three parts: edge loss, content loss, and pixel loss. The total loss function is obtained by weighting edge loss, content loss, and pixel loss. Step 4: Train the network. Use the data processed in step 2 to train the inverse residual convolutional neural network constructed in step 3. The trained inverse residual convolutional neural network can output the corresponding repaired Mueller matrix based on the input foggy or turbid polarized image. Step 5: Restore the image. Use the restored Mueller matrix output by the inverse residual convolutional neural network trained in step 4 to repair the foggy or turbid polarized image, and finally obtain a significantly improved restored image.

2. The polarization restoration imaging method based on the transform Mueller matrix network according to claim 1, characterized in that: In step 1, a clear polarization image of an underwater target is captured: a light beam emitted by a light source (1) passes through a polarizer (2) and a beam expander (3) of a polarization modulation system in sequence, and then irradiates a target (6) in the water. After being reflected by the target (6), the light beam reaches a focal plane polarization camera (7), thereby obtaining a clear polarization image of the target (6).

3. The polarization restoration imaging method based on the transformation Mueller matrix network according to claim 2, characterized in that: Step 1: Shooting polarized images of the target under different turbidity levels: gradually adding skim milk to water, and using a 5-megapixel polarized Gigabit Ethernet industrial camera to shoot the target (6) under different turbidity concentration environments, thereby obtaining polarized images of the target (6) in the turbid underwater environment at different polarization angles and different turbidity concentrations.

4. A polarization restoration imaging method based on a transform Mueller matrix network according to claim 2 or 3, characterized in that: A 532nm blue-green laser is used as the light source (1).

5. The polarization restoration imaging method based on the transformation Mueller matrix network according to claim 4, characterized in that: The inverted residual polarized attention channel module in step three consists of two parts. The first part is an inverted residual structure consisting of three convolutional layers, which includes a 1*1 convolution to increase the dimension of the feature map, a 3*3 convolution to extract the features, and a 1*1 convolution to reduce the dimension of the feature map. Finally, it also includes a BN layer for normalization. The second part is a channel attention structure, which includes an average pooling layer, a one-dimensional convolution layer and a Sigmoid activation function, which avoids dimensionality reduction and effectively performs cross-channel interaction.

6. The polarization restoration imaging method based on the transformation Mueller matrix network according to claim 5, characterized in that: The inverse residual convolutional neural network constructed in step three is designed based on the idea of ​​transforming the Mueller matrix. The inverse residual polarization attention channel module of the network can extract the corresponding feature information so that the network can find the relationship between the Mueller matrix of the turbid image and the Mueller matrix of the clear polarized image based on the corresponding feature information, and output the transformed Mueller matrix corresponding to the turbid image accordingly.

7. The polarization restoration imaging method based on the transformation Mueller matrix network according to claim 6, characterized in that: The repaired Mueller matrix in step 5 is a physical model proposed based on the idea of ​​the Mueller matrix, which describes the transformation relationship between the Mueller matrices of a clear image and a turbid image.

8. The polarization restoration imaging method based on the transformation Mueller matrix network according to claim 2 or 3, characterized in that: The target object (6) is placed in a glass tank (4) filled with water to simulate an underwater environment (5).

9. The polarization restoration imaging method based on the transforming Mueller matrix network according to claim 8, characterized in that: The glass cylinder (4) is made of PMMA (polymethyl methacrylate).

Images