Color polarized image restoration method based on three-dimensional convolution-attention joint mechanism

Abstract

The application discloses a color polarized image restoration method based on a three-dimensional convolution-attention joint mechanism. A three-dimensional convolution-attention joint polarization restoration model is adopted, including a space-polarization three-dimensional attention module, a space-polarization three-dimensional convolution module, and each module is connected and fused to form a multi-scale hierarchical network; a polarization restoration loss function based on a Stokes physical model is designed to train the network; three-dimensional features are extracted, fused and polarization information is solved, so that high signal-to-noise ratio and high-resolution color visible light and polarization information are obtained from low signal-to-noise ratio and low-resolution images. The application utilizes three-dimensional feature calculation to perform collaborative feature extraction and fusion on multiple dimensions of space, polarization and color, forms a multi-scale convolution-attention joint mechanism through a hierarchical connection network, combines the advantages of convolution in local detail feature extraction and the advantages of the attention mechanism in long-range dependence modeling, and thus improves the color polarized image restoration effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of polarization imaging, and in particular to a color polarization image restoration method based on a three-dimensional convolution-attention joint mechanism. BACKGROUND

[0002] Polarization, as the fourth important information dimension describing the basic properties of electromagnetic waves in addition to intensity, wavelength and phase, is closely related to the material, shape, texture structure, surface roughness and physicochemical properties of an object. Polarization optical imaging uses the polarization difference between the target and the background to improve the imaging quality, enhance the action distance, improve the detection ability and the identification probability. In recent years, as a new scheme of polarization imaging, the split focal plane polarization sensor is placed in front of the sensor by a micro-polarization array, and each sensor pixel corresponds to four micro-polarization units in four polarization directions, which has the advantages of synchronous imaging, independence of optical system, small volume, etc. However, the system sacrifices the spatial resolution, and the micro-polarization array is easy to introduce noise. Therefore, an effective polarization image restoration method is needed, i.e. a method of simultaneously performing polarization image super-resolution reconstruction and denoising, which converts low signal-to-noise ratio-low resolution images into high signal-to-noise ratio-high resolution polarization images, to obtain accurate visible light and polarization information.

[0003] Polarization image super-resolution reconstruction and denoising technology can be divided into two categories: non-data-driven and data-driven. Non-data-driven methods include spatial domain interpolation, sparse representation super-resolution reconstruction, spatial domain image filtering, transform domain denoising and sparse representation denoising. These methods perform well in some cases, but are still limited by prior models and manual parameter tuning, and lack universality. Data-driven methods, such as methods based on convolutional neural networks and attention mechanisms, can effectively perform polarization image super-resolution reconstruction and remove polarization image noise through deep learning on a large amount of data sets, and are generally superior to non-data-driven methods.

[0004] Although significant progress has been made in polarization restoration methods based on deep learning, existing polarization restoration methods still have deficiencies:

[0005] 1. In terms of feature extraction, current methods mainly focus on spatial dimensions and fail to fully exploit multi-dimensional information such as polarization and color; lack of effective mechanisms to simultaneously focus on local details and global information, with deficiencies in balancing between local details and overall structure; lack of utilization of multi-scale information;

[0006] 2. In terms of model training and testing, there is a lack of polarization information loss function design, which cannot well adapt to the polarization restoration scene; lack of a perfect polarization restoration process, lack of an end-to-end processing flow from the original data of the split focal plane polarization sensor to high-quality visible light and polarization information;

[0007] In practical applications, the existing polarization image restoration method generally lacks the ability to simultaneously perform super-resolution reconstruction and denoising, and has limited application in improving the loss of spatial resolution and the reduction of signal-to-noise ratio of the focal plane polarization sensor. SUMMARY

[0008] The purpose of the present application is to overcome the problems of the prior art, solve the problem that the focal plane color polarization camera is affected by the loss of spatial resolution and the reduction of signal-to-noise ratio, and is difficult to extract color polarization information, and provide a color polarization image restoration method based on a three-dimensional convolution-attention joint mechanism.

[0009] The purpose of the present application is achieved by the following technical solution: a color polarization image restoration method based on a three-dimensional convolution-attention joint mechanism, the method comprising the following steps:

[0010] S1. Establish a pair of polarization restoration data sets composed of four-dimensional color polarization images, the data set being obtained by a combination device of a focal plane polarization camera and a general visible light camera; perform polarization information calculation on the four-dimensional color polarization images to obtain three-dimensional visible light intensity, degree of polarization and polarization angle information, and expand the data dimension; use a data enhancement method to augment the data set, and randomly divide the data set into a training set and a test set according to a specified proportion, for training and detecting network performance;

[0011] S2. Construct a three-dimensional convolution-attention joint polarization restoration network model, including a shallow three-dimensional feature extraction module, a spatial-polarization three-dimensional attention module, a spatial-polarization three-dimensional convolution module, and a module fusion; the shallow three-dimensional feature extraction module is composed of a three-dimensional convolution layer, for extracting shallow three-dimensional features in the spatial and polarization dimensions; the spatial-polarization three-dimensional attention module is based on a sliding window mechanism and a three-dimensional attention mechanism, for modeling global long-range dependencies in the spatial and polarization dimensions; the spatial-polarization three-dimensional convolution module is based on a three-dimensional convolution residual network, for capturing local three-dimensional information; the modules are connected and fused to form a multi-scale hierarchical U-shaped network;

[0012] S3. Design a polarization restoration loss function based on the Stokes physical model, the loss function including four-dimensional color polarization image loss, three-dimensional color light intensity loss, degree of polarization loss and polarization angle loss; use the loss function to train the three-dimensional convolution-attention joint polarization restoration network model;

[0013] S4. Based on the trained three-dimensional convolution-attention joint polarization restoration network model, extract and fuse three-dimensional features, and perform polarization information calculation based on the Stokes physical model, so as to obtain high signal-to-noise ratio and high resolution four-dimensional color polarization images and corresponding three-dimensional visible light intensity, degree of polarization and polarization angle information from low signal-to-noise ratio and low resolution four-dimensional color polarization images.

[0014] Compared with the prior art, the beneficial effects brought by the scheme of the present application are:

[0015] The application discloses a color polarized image restoration network based on a three-dimensional convolution-attention joint mechanism. The network utilizes three-dimensional feature calculation to perform collaborative feature extraction and fusion on multiple dimensions such as space, polarization and color. A multi-scale convolution-attention joint mechanism is formed through a hierarchical connection network, which fully utilizes the advantages of convolution in local detail feature extraction and combines the advantages of the attention mechanism in modeling long-range dependencies. This design effectively improves the color polarized image restoration effect, enabling the network to better understand and restore subtle details and overall information in the image.

[0016] The application discloses a polarization restoration loss function based on a Stokes physical model, which is used for training a model. The loss function mainly includes a four-dimensional color polarized image loss, a three-dimensional color light intensity loss, a polarization degree loss and a polarization angle loss. Among them, the four-dimensional color polarized image loss restricts the learning of the network for super-resolution reconstruction and denoising tasks, which helps the model to converge effectively; the three-dimensional color light intensity loss effectively restricts the learning of the network on color information; the polarization degree loss and the polarization angle loss effectively restrict the learning of the network on polarization information. This loss function enables the model to more comprehensively consider and learn various aspects of the color polarized image restoration problem, thereby improving the restoration effect and the generalization ability of the model.

[0017] The application discloses a polarization image restoration method, which can simultaneously perform super-resolution reconstruction and denoising, realizes an end-to-end process from a low signal-to-noise ratio and low-resolution image to a high signal-to-noise ratio and high-resolution color visible light and polarization information. This method performs well in improving the spatial resolution loss and signal-to-noise ratio reduction of a focal plane polarization sensor, and provides an effective solution for related tasks. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 A color polarized image restoration method flowchart of the present application;

[0019] Figure 2 A color polarized image restoration method input and output data expression form schematic diagram of the present application;

[0020] Figure 3 A color polarized image restoration model overall structure diagram based on a three-dimensional convolution-attention joint mechanism of the present application;

[0021] Figure 4 A space-polarization three-dimensional convolution schematic diagram of the present application;

[0022] Figure 5 A space-polarization three-dimensional-polarization three-dimensional attention module schematic diagram of the present application;

[0023] Figure 6A Color information output of the three-dimensional convolution-attention joint polarization restoration model, from left to right are R, G, B channel light intensity images respectively;

[0024] Figure 6B Color information true value, from left to right are R, G, B channel light intensity true value images respectively;

[0025] Figure 6C Color information true value, from left to right are R, G, B channel light intensity true value images respectively;

[0026] Figure 6D Polarization angle image output by the three-dimensional convolution-attention joint polarization restoration model;

[0027] Figure 6E True value of the polarization angle image;

[0028] Figure 6F Polarization degree image output by the three-dimensional convolution-attention joint polarization restoration model;

[0029] Figure 6G True value of the polarization degree image. DETAILED DESCRIPTION

[0030] In this embodiment, a color polarization image restoration method based on a three-dimensional convolution-attention joint mechanism is proposed, as shown in Figure 1 The following steps are performed:

[0031] S1. Establish a paired polarization restoration dataset composed of four-dimensional color polarization images, which is obtained by a combination device of a focal plane polarization camera and a general visible light camera; polarization information is calculated from the four-dimensional color polarization image to obtain three-dimensional visible light intensity, polarization degree and polarization angle information, and the data dimension is expanded; a data augmentation method is used to augment the dataset, and the dataset is randomly divided into a training set and a test set according to a specified proportion, which is used to train and detect the network performance of the present application; the input and output data expression forms of the color polarization image restoration method of the present application are shown in Figure 2 .

[0032] 1.1 The four-dimensional color polarization image is obtained by a combination device of a general visible light camera and a focal plane polarization camera; a high signal-to-noise ratio high-resolution image is captured by a general visible light camera with a front rotating polarizer, a low signal-to-noise ratio low-resolution image is obtained by pixel recombination of the original image data of the focal plane color polarization camera, and the optical centers of the two cameras are aligned; the four-dimensional color polarization image is paired according to high signal-to-noise ratio high-resolution and low signal-to-noise ratio low-resolution; the four-dimensional color polarization image includes four dimensions of pixel height, pixel width, polarization and color, which are represented in order as .

[0033] 1.2 Four-dimensional color polarized image with high resolution and low noise The polarized information is calculated based on the Stokes physical model to obtain three-dimensional visible light intensity , degree of polarization (linear polarization degree) , polarization angle The Stokes calculation formula is as follows:

[0034] ;

[0035] ;

[0036] ;

[0037] In the above formula, is a four-dimensional color polarized image; is the total light intensity, is the difference in linearly polarized light intensity in the 0° and 90° directions, is the difference in linearly polarized light intensity in the 45° and 135° directions; AOP, DOP are the degree of polarization and the polarization angle;

[0038] 1.3 The data set is expanded using a data enhancement method, and the data set is randomly divided into a training set and a test set according to a specified ratio, which is used to train and detect the network performance of the application.

[0039] S2. Construct a three-dimensional convolution-attention joint polarization restoration network, including a shallow three-dimensional feature extraction module, a spatial-polarization three-dimensional attention module, a spatial-polarization three-dimensional convolution module, and a fusion module; the shallow three-dimensional feature extraction module is composed of a three-dimensional convolution layer, which is used to extract shallow three-dimensional features in the spatial and polarization dimensions; the spatial-polarization three-dimensional attention module is based on a sliding window mechanism and a three-dimensional attention mechanism, and models the global long-range dependence in the spatial and polarization dimensions; the spatial-polarization three-dimensional convolution module is based on a three-dimensional convolution residual network, and captures local three-dimensional information; the modules are connected and fused to form a multi-scale hierarchical U-shaped network; the overall structure of the three-dimensional convolution-attention joint polarization restoration network is as shown in Figure 3 .

[0040] 2.1 Construct a shallow three-dimensional feature extraction module, as shown in Figure 4 , extract shallow three-dimensional features in the spatial and polarization dimensions by a three-dimensional convolution kernel, and the three-dimensional convolution calculation formula is as follows:

[0041] ;

[0042] In the above formula, For input features, including z, x, y, c four dimensions, where z dimension is polarization dimension, including 0°, 45°, 90°, 135° four dimensions, x, y is spatial dimension, c is color channel dimension; For three-dimensional convolution kernel, sliding along x, y, z three dimensions in three-dimensional convolution operation; For shallow three-dimensional features output by shallow feature extraction module;

[0043] 2.2 Constructing spatial-polarization three-dimensional attention module, as shown in Figure 5 The module is based on sliding window mechanism (W-SW), spatial-polarization three-dimensional multi-head attention layer MSPA, Gaussian error linear activated three-dimensional convolution feedforward network GFN, to model global long-range dependencies of spatial-polarization dimensions, spatial-polarization three-dimensional attention module expression is as follows, where is the input of the module, is the intermediate output of the original window spatial-polarization three-dimensional multi-head attention layer W-MSPA, is the intermediate output of the Gaussian error linear activated three-dimensional convolution feedforward network GFN, is the intermediate output of the sliding window spatial-polarization three-dimensional multi-head attention layer W-MSPA, is the output of the module;

[0044] ;

[0045] ;

[0046] ;

[0047] ;

[0048] 2.2.1 Spatial-polarization three-dimensional attention module includes spatial-polarization three-dimensional multi-head attention layer MSPA, which makes input features , through 1×1×1 three-dimensional convolution and 3×3×3 three-dimensional convolution in turn, to get query vector , key vector , value vector , through and matrix multiplication to get spatial-polarization feature map , finally get output feature . MSPA layer calculation formula is as follows:

[0049] ;

[0050] ;

[0051] ;

[0052] ;

[0053] ;

[0054] In the above formula, is an input feature, and the dimension is ; respectively 3x3x3 three-dimensional convolution and 1x1x1 three-dimensional convolution; respectively query vector, key vector, value vector, and the dimension is ; is a space-polarization feature map, and the dimension is ; is an output feature, and the dimension is ;

[0055] 2.2.2 Gaussian error linear activation three-dimensional convolution feedforward network GFN for input feature is divided into two paths, which are sequentially subjected to 1x1x1 three-dimensional convolution and 3x3x3 three-dimensional convolution, and one of the two paths is subjected to Gaussian error linear activation, and the two paths are multiplied to obtain Gaussian error linear activation feature , and finally through 1x1x1 three-dimensional convolution and residual connector to obtain output feature ; the calculation formula of Gaussian error linear activation three-dimensional convolution feedforward network GFN is as follows:

[0056] ;

[0057] ;

[0058] In the above formula, is an input feature; respectively 3x3x3 three-dimensional convolution and 1x1x1 three-dimensional convolution; is a Gaussian error linear activation operation; is a Gaussian error linear activation feature; is an output feature;

[0059] 2.3 Construct a space-polarization three-dimensional convolution module, which takes three-dimensional convolution, i.e., the shallow three-dimensional feature extraction module described in 2.1, as a basic module, and forms a three-dimensional convolution residual network based on residual connection architecture Resnet;

[0060] 2.4 Connecting and fusing the shallow three-dimensional feature extraction modules constructed in 2.1, 2.2, and 2.3, the space-polarization three-dimensional attention module, and the space-polarization three-dimensional convolution module to form a multi-scale hierarchical U-shaped network;

[0061] 2.4.1 The multi-scale hierarchical U-shaped network adopts a typical U-net architecture as shown in FIG. 2, which is composed of a spatial-polarization three-dimensional attention module, and gradually reduces the size of the feature map by down-sampling. The decoder is composed of a spatial-polarization three-dimensional convolution module, and gradually recovers the resolution of the feature map by up-sampling. Figure 3

[0062] 2.4.2 Between the encoder and the decoder, hierarchical skip connections are adopted. These connections ensure the transmission of feature information of corresponding levels, and use 1x1x1 three-dimensional convolution operation to complete channel fusion, realizing feature fusion of corresponding modules of each level.

[0063] S3. Design a polarization restoration loss function based on the Stokes physical model, which is composed of four-dimensional color polarization image loss , three-dimensional color light intensity loss , degree of polarization loss , and polarization angle loss , and the formula is as follows:

[0064] ;

[0065] ;

[0066] ;

[0067] ;

[0068] ;

[0069] In the above formula, , , , , are the total loss, the four-dimensional color polarization image loss, the three-dimensional color light intensity loss, the degree of polarization loss, and the polarization angle loss, respectively. are the proportions of the four-dimensional color polarization image loss, the three-dimensional color light intensity loss, the degree of polarization loss, and the polarization angle loss in the total loss, respectively. are the network four-dimensional color polarization image output and the four-dimensional color polarization image true value, respectively. AOP, DOP are the light intensity, degree of polarization, and polarization angle calculated based on the Stokes physical model, respectively.

[0070] ​The model is trained based on the loss function. According to the requirements of color polarization image restoration and the effects of color information and polarization information restoration, the network model parameters are adjusted, including initial learning rate, learning rate exponential decay rate, batch size, training period, loss function proportion, network module number and channel number and other parameters. Specifically, in the embodiment, the initial learning rate is 2e^-05, the learning rate exponential decay rate is 0.9, the batch size is 16, the training period is 150, the loss function proportion is ; the color polarization image restoration network encoder is composed of 4 space-space-polarization three-dimensional attention modules, the 4 modules are stacked by {1, 2, 4, 4} space-polarization three-dimensional attention submodules and connected by residual connection, and the number of attention heads head in the 4 modules is {4, 4, 8, 8}; the color polarization image restoration network decoder is composed of 3 space-polarization three-dimensional convolution modules, and the 3 modules respectively contain {2, 4, 6} convolution layers; the channel number C of the shallow three-dimensional feature extraction module is 200.

[0071] S4. Based on the trained three-dimensional convolution-attention joint polarization restoration model, three-dimensional features are extracted and fused, and polarization information is calculated based on the Stokes physical model, so that the low signal-to-noise ratio and low resolution four-dimensional color polarization image obtained by the focal plane sensor is obtained. Four-dimensional color polarization image and corresponding three-dimensional visible light intensity, degree of polarization, polarization angle information with high signal-to-noise ratio and high resolution.

[0072] Specifically, the low signal-to-noise ratio and low resolution four-dimensional color polarization image has a dimension of , after pixel reorganization, the network input with a dimension of is obtained; the network input is input into the trained three-dimensional convolution-attention joint polarization restoration model, three-dimensional features are extracted and fused, and a high signal-to-noise ratio and high resolution four-dimensional color polarization image with a dimension of is obtained; the polarization information of the high signal-to-noise ratio and high resolution four-dimensional color polarization image is calculated based on the Stokes physical model, and the corresponding three-dimensional visible light intensity , degree of polarization (linear degree of polarization) , and polarization angle are obtained, and the processing flow is as shown in Figure 2 .

[0073] The present application shows the processing effect of the three-dimensional convolution-attention joint polarization restoration model. Among them, Figure 6A is the original output of the focal plane color polarization camera; Figure 6B is the color information output of the three-dimensional convolution-attention joint polarization restoration model, and the R, G and B channel intensity images from left to right are Figure 6C is the color information true value, and the R, G and B channel intensity true value images from left to right are Figure 6Da polarization angle map output by the three-dimensional convolution-attention combined polarization restoration model, Figure 6E a true value of the polarization angle map; Figure 6F a polarization degree map output by the three-dimensional convolution-attention combined polarization restoration model, Figure 6G a true value of the polarization degree map. As can be seen from the figure, the three-dimensional convolution-attention combined polarization restoration method provided by the present application can effectively perform the color polarization image restoration task, complete super-resolution reconstruction and denoising, obtain high signal-to-noise ratio and high resolution color visible light information, i.e. R, G and B channel light intensity true value images, and polarization information, i.e. a polarization angle map and a polarization degree map, from low signal-to-noise ratio and low resolution original data. The R, G and B channel light intensity true value images, the polarization angle map and the polarization degree map generated by the present example are highly similar to the true images in overall structure, and the recovery effect of image details is good. This shows that the three-dimensional convolution-attention combined polarization restoration model in the present application has a significant effect in the restoration task of color polarization images.

[0074] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, different embodiments or examples described in the present specification and the features of different embodiments or examples can be combined and modified by those skilled in the art without contradiction, within the scope of the present application.

[0075] Although the embodiments of the present application have been shown and described above, it should be understood that the above-described embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-described embodiments within the scope of the present application.

Claims

1. A method for restoring color polarized images based on a three-dimensional convolution-attention joint mechanism, characterized in that, The method includes the following steps: S1. Establish a paired polarization restoration dataset consisting of four-dimensional color polarization images, wherein the dataset is obtained by a combination of a focal plane polarization camera and a regular visible light camera; perform polarization information calculation on the four-dimensional color polarization images to obtain three-dimensional visible light intensity, degree of polarization, and polarization angle information, thereby expanding the data dimension; use data augmentation methods to expand the dataset, and randomly divide the dataset into training set and test set according to a specified ratio for training and testing network performance; S2. Construct a 3D convolutional-attention joint polarization restoration network model, including a shallow 3D feature extraction module, a spatial-polarization 3D attention module, a spatial-polarization 3D convolution module, and module fusion. The shallow 3D feature extraction module consists of 3D convolutional layers to extract shallow 3D features in spatial and polarization dimensions. The spatial-polarization 3D attention module is based on a sliding window mechanism and a 3D attention mechanism to perform global long-range dependency modeling in spatial and polarization dimensions. The spatial-polarization 3D convolution module is based on a 3D convolutional residual network to capture local 3D information. The modules are connected and fused to form a multi-scale hierarchical U-shaped network. S3. Design a polarization restoration loss function based on the Stokes physics model. This loss function includes four-dimensional color polarization image loss, three-dimensional color light intensity loss, degree of polarization loss, and polarization angle loss. Use this loss function to train a three-dimensional convolutional-attention joint polarization restoration network model. S4. Based on the trained 3D convolution-attention joint polarization restoration network model, 3D features are extracted and fused, and polarization information is calculated based on the Stokes physics model. Thus, a high signal-to-noise ratio and high resolution 4D color polarization image and corresponding 3D visible light intensity, polarization degree, and polarization angle information are obtained from a low signal-to-noise ratio and low resolution 4D color polarization image.

2. The color polarization image restoration method based on a three-dimensional convolution-attention joint mechanism according to claim 1, characterized in that, In step S1, the four-dimensional color polarization image includes four dimensions: pixel height, pixel width, polarization, and color, which are represented in order as follows: The dataset contains four-dimensional color polarization images paired according to a high signal-to-noise ratio (SNR) high-resolution model and a low SNR low-resolution model. The paired four-dimensional color polarization images were captured by a combination of a conventional visible light camera and a focal plane polarization camera. The high SNR high-resolution image was captured by a conventional visible light camera with a front-mounted rotating polarizer, while the low SNR low-resolution image was obtained by pixel reconstruction of the original image data from the focal plane color polarization camera. The optical centers of the two cameras were aligned. Polarization information was calculated from the high-resolution, low-noise four-dimensional color polarization images based on the Stokes physics model to obtain the three-dimensional visible light intensity. Linear polarization polarization angle This expands the data dimensions of true values.

3. The color polarization image restoration method based on a three-dimensional convolution-attention joint mechanism according to claim 1, characterized in that, In step S2, the shallow 3D feature extraction module uses 3D convolution. Kernel extraction of shallow 3D features in the spatial-polarization dimension The formula for calculating 3D convolution is: ； In the above formula, The input features contain four dimensions: z, x, y, and c. The z dimension is the polarization dimension, which includes four dimensions: 0°, 45°, 90°, and 135°. The x and y dimensions are spatial dimensions, and the c dimension is the color channel dimension. It is a three-dimensional convolution kernel that slides along the x, y, and z dimensions during the three-dimensional convolution operation; The shallow 3D features output by the shallow feature extraction module; The spatial-polarization 3D attention module is based on a sliding window mechanism (W-SW), a spatial-polarization 3D multi-head attention layer (MSPA), and a Gaussian error linearly activated 3D convolutional feedforward network (GFN) to perform global long-range dependency modeling of the spatial-polarization dimension. The expression for the spatial-polarization 3D attention module is as follows: [Equation omitted for brevity]. For module input, For module output, LN is layer normalization; This is the intermediate output of the original window spatial-polarized 3D multi-head attention layer W-MSPA. This is the intermediate output of a 3D convolutional feedforward network (GFN) linearly activated by Gaussian error. The intermediate output is obtained through a sliding window spatially polarized three-dimensional multi-head attention layer (W-MSPA). ； ； ； ； The spatial-polarization 3D convolution module uses 3D convolution, i.e., the shallow 3D feature extraction module, as the basic module, and constructs a 3D convolution residual network through the ResNet residual connection architecture. The module fusion is achieved through 1×1×1 three-dimensional convolution; the multi-scale hierarchical U-shaped network adopts the U-net architecture, the encoder is composed of spatial-polarization three-dimensional attention modules, which gradually reduce the size of the feature map through downsampling, and the decoder is composed of spatial-polarization three-dimensional convolution modules, which gradually restore the resolution of the feature map through upsampling, with corresponding layers of the encoder and decoder skipping connections.

4. The color polarization image restoration method based on a three-dimensional convolution-attention joint mechanism according to claim 1, characterized in that, In step S3, the polarization restoration loss function based on the Stokes physics model is composed of a four-dimensional color polarization image loss. Three-dimensional color light intensity loss Polarization loss Polarization angle loss The weighted summation is calculated using the following formula: ; ； ； ； ； In the above formula, , , , , These are the total loss, four-dimensional color polarization image loss, three-dimensional color light intensity loss, degree of polarization loss, and polarization angle loss, respectively. These represent the percentages of four-dimensional color polarization image loss, three-dimensional color light intensity loss, degree of polarization loss, and polarization angle loss in the total loss, respectively. Output of four-dimensional color polarization image from network, and true value of four-dimensional color polarization image respectively; AOP and DOP are the light intensity, degree of polarization, and polarization angle calculations based on the Stokes physics model, respectively.

5. The color polarization image restoration method based on a three-dimensional convolution-attention joint mechanism according to claim 1, characterized in that, In step S4, a low signal-to-noise ratio, low-resolution four-dimensional color polarization image with dimensions of... After pixel recombination, the network input is obtained, with dimension [missing information]. The input is then fed into the trained 3D convolutional-attention joint polarization restoration model to extract and fuse 3D features, resulting in a high signal-to-noise ratio, high-resolution 4D color polarization image with dimensions of [missing information]. Polarization information is calculated based on the Stokes physics model from high signal-to-noise ratio, high resolution four-dimensional color polarization images to obtain the corresponding three-dimensional visible light intensity. linear polarization degree polarization angle .

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the color polarization image restoration method based on a three-dimensional convolution-attention joint mechanism as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the color polarization image restoration method based on a three-dimensional convolution-attention joint mechanism as described in any one of claims 1 to 5.

Images