Transform-based underwater polarization image restoration method
Through the multi-branch structure and feature fusion module based on Transformer, the problem of insufficient long-distance dependence and local polarization information capture in underwater polarization imaging technology is solved, efficient underwater image restoration is achieved, and the structural integrity of the image is improved. Details.
Patent Information
- Application Number
- CN202411168331.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-12-17
AI Technical Summary
When existing underwater polarization imaging technology deals with complex environments or non-ideal conditions, it is difficult to effectively capture long-distance dependencies and local polarization information in the image, resulting in insufficient structural integrity and detailed performance during image restoration.
Using the Transformer-based underwater polarization image restoration method, the global and local polarization characteristics are captured through multi-branch Transformer structure and multi-stage network design, combining super token attention, dual adaptive neural blocks and selective nuclear feature fusion modules, and the global and local polarization characteristics are enhanced. Texture information and structural integrity of the image.
It significantly improves the performance of processing long-distance dependencies, generates clear underwater images, improves the structural integrity and detailed performance of the image, and optimizes the computing efficiency and enhances the feature capture ability and adaptability of the model.
Smart Images

Figure CN119151831A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a Transformer-based underwater polarization image restoration method. Background Art
[0002] When light propagates in water, it is scattered and absorbed by particles, resulting in reduced image contrast and color distortion. The polarization image restoration method uses the difference in polarization characteristics between the target reflected signal light and the backscattered light to remove the scattered light, thereby clearly restoring underwater objects and improving the quality of degraded underwater images.
[0003] Polarization imaging is an advanced image restoration method that uses the characteristics of polarized light to enhance the contrast and details of images. Traditional polarization imaging techniques are generally divided into polarization differential imaging and methods based on Stokes vector analysis. They rely on specific physical models and precise parameter estimation, and have limitations when dealing with complex environments or non-ideal conditions. They usually need to rely on specific clues such as background areas, similar objects or depth information to estimate key polarization parameters, so they are difficult to implement in practical applications.
[0004] In recent years, the application of deep learning in underwater polarization imaging has, to a certain extent, solved the problems of low contrast and severe loss of details in underwater images, and has achieved results that are superior to traditional descattering methods based on physical models in various indicators. However, existing methods mainly rely on convolutional neural networks. Convolutional neural networks can effectively capture local features through their local receptive fields, but this locality limits their ability to capture the complex relationships between scattered and distant pixels in the image. In the image restoration task, long-range dependencies, that is, the interconnections between distant pixels in the image, are crucial for a comprehensive understanding of the global structure and contextual information of the image. Existing methods fail to fully utilize these long-range dependencies, resulting in deficiencies in structural integrity and detail representation during image restoration. Therefore, a method is needed that can not only effectively solve long-range dependencies but also better extract local polarization information. Summary of the invention
[0005] In view of the shortcomings of the prior art, the present invention provides a Transformer-based underwater polarization image restoration method. The present invention utilizes the characteristics of polarization imaging to not only handle long-distance dependency problems and capture global contextual polarization information, but also maintain sensitivity to local polarization information. The present invention can enhance the texture information in the image and effectively suppress background scattered light, thereby generating clear underwater images.
[0006] The technical means adopted by the present invention are as follows:
[0007] A Transformer-based underwater polarization image restoration method includes the following steps:
[0008] Acquire an underwater polarization image dataset, wherein the underwater polarization image dataset includes a clear underwater image and a corresponding polarization image group, wherein the polarization image group includes polarization images taken at different polarization angles;
[0009] Constructing a polarization image restoration model, which includes a four-stage encoder-decoder network and a single residual block of a multi-branch Transformer block. The encoder-decoder network of each stage contains a residual block of a multi-branch Transformer block. The input image is subjected to 3*3 convolution for shallow feature extraction, and then the encoder of each stage captures deep polarization information. The decoder features of each stage are then used to restore the details and structure of the image. The information of the encoder and decoder of the same stage is aggregated using jump connections. Finally, a 3*3 convolution compression channel is used to output the reconstructed image.
[0010] Training the polarization image restoration model based on the underwater polarization image dataset;
[0011] The polarization image group to be restored is obtained and input into the trained polarization image restoration model, and the output of the polarization image restoration model is obtained as a restored image.
[0012] Furthermore, the multi-branch Transformer block includes super token attention, dual adaptive neural block and selective kernel feature fusion module; the super token enhances the Transformer block's perception of global context features of polarization information through super token sampling; the dual adaptive neural block effectively captures the global and local context information of the image through a dual-channel structure and a dynamic weighting strategy; the selective kernel feature fusion module fuses the polarization information features generated by different branches.
[0013] Furthermore, the super token attention includes super token sampling, multi-head self-attention mechanism and token upsampling;
[0014] The super token sampling is used to divide the extracted feature map into multiple area units, each area represents a basic token, the tokens are aggregated into super tokens, and then a multi-head self-attention mechanism is performed on the super tokens to capture the global dependencies between them. Finally, a token upsampling operation is performed to retain the local detail information of the image and remap the information of the super token back to the original token. The token is the basic processing unit of the image, and each token contains the pixel information of a specific local area in the image. The super token is a feature unit aggregated from multiple tokens, representing a larger area or a more abstract feature set in the image.
[0015] Furthermore, the super token sampling is configured to perform the following steps: by average pooling, the image is divided into regular grid regions to create an initial super token S 0 , given a visual token Among them, N = H × W is the number of tokens, H is the height of the feature map, and W represents the width of the feature map. Assume that each token Belongs to m super tokens One of them, through an iterative process, calculates the XS sparse association matrix between tokens and super tokens Defined as:
[0016]
[0017] Where C is the number of channels, t is the number of iterations, and T is the transpose of the matrix;
[0018] Update the super token to the weighted sum of the tokens and iterate according to the following formula:
[0019]
[0020] In the formula is the column-normalized Q t , d is the number of channels, t is the number of iterations, and T is the transpose of the matrix.
[0021] Furthermore, the multi-head self-attention mechanism is configured to perform the following steps: each head evaluates the association between super tokens by linear transformation of query, key, and value, calculates the attention weight of each super token by softmax activation function according to the evaluation result, and then performs weighted summation on the value vector, and applies standard self-attention to the sampled super tokens. Defined as:
[0022]
[0023] in is the attention graph, where q, k, and v are query, key, and value respectively, and q(S) = SW q , k(S)=SW k and v(S) = SW v , respectively, the parameters are W q , W k and W v A linear function of q is a learnable parameter matrix corresponding to the query weights, W k is the learnable parameter matrix, corresponding to the key weights, W v is also a learnable parameter matrix, corresponding to the weights of the values.
[0024] Furthermore, the token upsampling is configured to perform the following steps: mapping the super token information processed by the multi-head self-attention mechanism back to the original token through the correlation matrix Q, thereby obtaining superpixel features:
[0025]
[0026] in, Represents superpixel features.
[0027] Further, the dual adaptive neural block is configured to perform the following steps:
[0028] Use point-by-point convolution to mix superpixel features and get the mixed feature representation
[0029]
[0030] in, represents the superpixel feature, P mix (·) represents point-wise convolution;
[0031] Using the dual-channel structure, the mixed features The test paper is divided into two parts, respectively recorded as and
[0032]
[0033] Among them, Φ DW (·) is the group convolution operation, It is a mixed feature The first part of the features obtained after the group convolution operation, Mixed features Another part of the features obtained after the group convolution operation;
[0034] Using dynamic weighting and Combined, including Perform Sigmoid activation to obtain the weights used to readjust the importance of polarization features. Perform GELU activation to obtain the polarization features after nonlinear transformation, and then multiply the two element by element to obtain the weighted features
[0035]
[0036] Among them, σ refers to the Sigmoid activation function, φ is the GELU activation function, and ⊙ represents element-by-element multiplication.
[0037] Finally, the integrated feature map is passed through a 1*1 convolution P final (·) Weighted features Perform refinement to ensure output features Rich in global information and details:
[0038]
[0039] Furthermore, the selective kernel feature fusion module is configured to perform the following steps: dynamically adjust the receptive field through two operations, fusion and selection, wherein the fusion operation generates a global feature description by integrating information from different branches; and the selection operation uses these global descriptions to recalibrate and aggregate feature maps of different branches.
[0040] Furthermore, the loss function during the training of the polarization image restoration model is:
[0041]
[0042] Among them, x n is the predicted value, y n is the true value, and N is the number of samples.
[0043] Compared with the prior art, the present invention has the following advantages:
[0044] The present invention provides an underwater polarization image restoration method, which shows significant performance improvement in processing long-distance dependencies compared to traditional methods. The advantage of the present invention is that it adopts a multi-branch Transformer structure to achieve efficient capture of global and local polarization features, and enhances the clarity and structural integrity of polarization images through a multi-stage network design. At the same time, the present invention strikes a balance between optimizing computational efficiency and maintaining high performance. In particular, the super token attention improves the feature capture capability of the model by reducing the amount of computation while retaining key image information; the dual-channel adaptive neural block enhances the adaptability of the model to complex scenes through a dual-channel structure and a dynamic weighting strategy; and the selective kernel feature fusion module further improves the efficiency and accuracy of feature fusion by dynamically adjusting the receptive field.
[0045] The present invention brings about a breakthrough in the field of underwater polarization image restoration and promotes the development of underwater imaging technology. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0047] Figure 1 This is an overall flow chart of the Transformer-based underwater polarization image restoration method of the present invention.
[0048] Figure 2 2 is a network architecture diagram of a polarization image restoration model in an embodiment of the present invention.
[0049] Figure 3 2 is a model architecture diagram of a multi-branch Transformer block in an embodiment of the present invention.
[0050] Figure 4 Schematic diagram of an experimental device for passive polarization imaging in an underwater environment according to an embodiment of the present invention.
[0051] Figure 5 It is a partial image of the underwater polarization image data set collected in an embodiment of the present invention.
[0052] Figure 6 The figure shows the comparison results of image restoration using the method of the present invention and other methods. DETAILED DESCRIPTION
[0053] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0054] like Figure 1-3 As shown, the present invention discloses a Transformer-based underwater polarization image restoration method, which is characterized by comprising the following steps:
[0055] S1. Acquire an underwater polarization image dataset, where the underwater polarization image dataset includes clear underwater images and corresponding polarization image groups, where the polarization image groups include polarization images taken at different polarization angles.
[0056] Specifically, an experiment of underwater passive polarization imaging was conducted in an underwater environment. In order to enrich the types of objects photographed, models of sea cucumbers and scallops, ink bottles, milk cartons, and cans were placed. Fix the object at the bottom of a transparent water tank. Pour clean water to immerse the object. First, use the polarization camera Blackfly S BFS-U3-51S5P-C in clean water to capture a real image of the object as a reference image. Then drip about 2 ml of milk and capture a set of polarization images of the same scene, which are four polarization images of 0°, 45°, 90°, and 135°. Continue to drip milk and shoot until the water becomes turbid and the object cannot be photographed.
[0057] S2. Construct a polarization image restoration model, which includes a four-stage encoder-decoder network and a separate residual block of a multi-branch Transformer block. The encoder-decoder network of each stage contains a residual block of a multi-branch Transformer block. The input image undergoes shallow feature extraction through 3*3 convolution, and then the deep polarization information is captured by the encoder of each stage. Subsequently, the details and structure of the image are restored through the features of the decoder of each stage. The information of the encoder and decoder of the same stage is aggregated using jump connections. Finally, a 3*3 convolution compression channel is used to output the reconstructed image.
[0058] Specifically, a four-stage encoder-decoder network is used for deep feature extraction. The encoder captures deeper features, and the decoder uses these features to restore the details and structure of the image. Each stage contains a residual block of a multi-branch Transformer block. First, four turbid images with different polarization directions (0°, 45°, 90°, and 135°) are input and subjected to 3*3 convolution for shallow feature extraction. Then, the encoder part is entered, and the encoder is able to capture deeper features. At the end of each stage, a downsampling operation is used to reduce the spatial resolution of the feature map and reduce the number of feature channels to capture more deep polarization information. Next, the decoder part is entered, and the decoder uses these features to restore the details and structure of the image. At the beginning of each stage, an upsampling operation is applied to restore the spatial resolution of the feature map. At the same time, the number of feature channels is increased to gradually reconstruct the details of the image. The information of the encoder and decoder is aggregated using skip connections, and the high-resolution features of the encoder are directly passed to the decoder to prevent the loss of detail information. And 1*1 convolution is used for dimensionality reduction (except for the first stage). A multi-branch Transformer residual block is also used after the encoder-decoder structure to restore fine structure and texture details. Finally, a 3*3 convolution compression channel is used to output the reconstructed image. This method captures and restores the details of the polarization image more efficiently, making the reconstructed image closer to the original scene.
[0059] Furthermore, the multi-branch Transformer block mainly includes super-token attention, dual adaptive neural block and selective kernel feature fusion module. Super-token enhances the model's perception of the global context features of polarization information through super-token sampling; the dual adaptive neural block effectively captures the global and local context information of the image through a dual-channel structure and dynamic weighting strategy. In order to fuse the polarization information features generated by different branches, the selective kernel feature fusion module is introduced at the end of the multi-branch Transformer block.
[0060] Furthermore, super token attention consists of three parts: super token sampling, multi-head self-attention mechanism, and token upsampling. First, tokens are aggregated into super tokens through super token sampling, and then a multi-head self-attention mechanism is performed on these super tokens to capture the global dependencies between them. Since the number of super tokens is less than the original tokens, the computational cost of this step is lower. Finally, in order to retain the local detail information of the image, the model performs a token upsampling operation to remap the information of the super token back to the original token. In this way, the model can capture local detail features while retaining global information while using super tokens to reduce the amount of computation.
[0061] Specifically, a token is the basic processing unit of an image, which is composed of small blocks into which the image is divided. Each token contains the pixel information of a specific local area in the image, which is converted into a feature vector to represent the visual content of the area. A super token is a higher-level feature unit aggregated from multiple tokens. It represents a larger area or a more abstract feature set in the image. By reducing the number of units processed in the self-attention mechanism, it aims to reduce computational costs and improve the ability to capture global context.
[0062] The super token upsampling scheme is as follows: First, the image is divided into regular grid regions through average pooling to create the initial super token S 0 Then, using the idea of soft k-means, given the visual token (N = H × W is the number of tokens), assuming that each token Belongs to m super tokens One of them, through an iterative process, calculates the XS sparse association matrix Q∈R between the token and the super token N×m , defined as:
[0063]
[0064] This matrix reflects the strength of association between a token and a super token, where C is the number of channels, t is the number of iterations, and T is the transpose of the matrix.
[0065] Update the super token to the weighted sum of the tokens. In the iteration, optimize the association of the token with the super token and iteratively update the super token to better represent the characteristics of the related tokens. Limit the token to only establish a connection with the local surrounding super tokens. The iterative formula is as follows:
[0066]
[0067] In the formula is the column-normalized Q t , d is the number of channels, t is the number of iterations, and T is the transpose of the matrix.
[0068] The multi-head self-attention mechanism is set up as follows: after generating super tokens and calculating the sparse association matrix, multi-head self-attention is applied to these super tokens. Focus on capturing different parts of contextual information. In this process, each head evaluates the association between super tokens through linear transformations of query, key, and value. Using these evaluations, the attention weight of each super token is calculated through the softmax activation function, and then the value vector is weighted summed. In this way, the model can learn the complex relationship between local and global features at the same time. Apply standard self-attention to the sampled super tokens Defined as:
[0069]
[0070] in is the attention graph, where q, k, and v are query, key, and value, respectively. q(S) = SW q , k(S)=SW k and v(S) = SW v , respectively, the parameters are W q , W k and W v A linear function of W. q is a learnable parameter matrix corresponding to the weights of the query. k is a learnable parameter matrix corresponding to the key weights. v is also a learnable parameter matrix, corresponding to the weights of the values.
[0071] The super token sampling step has already reduced the number of tokens in advance. Not only are the number of super tokens generated by super token sampling smaller, but each super token also contains a set of information about the original tokens. In this way, the multi-head self-attention mechanism can only consider these more informative and aggregated super tokens during calculation, rather than processing a large number of original tokens, thereby reducing the computational cost.
[0072] The token upsampling process is set as follows: in super token sampling, tokens are aggregated into super tokens; and in token upsampling, the information of super tokens is split back into individual tokens, ensuring the fusion of global polarization information and local polarization information. The multi-head self-attention step provides super tokens enhanced with global context, and token upsampling effectively transfers this information back to the original token, allowing the model to utilize both global information and local details. Specifically, the super token information processed by the multi-head self-attention mechanism is mapped back to the original token through the association matrix Q, so that each token can utilize global context information, ensuring that while capturing global dependencies, detailed information is also retained. The specific mapping formula is as follows:
[0073]
[0074] in, Represents superpixel features.
[0075] Furthermore, the two-channel adaptive neural block is set to first use point-wise convolution (P mix (·)) Mix the superpixel features to get the mixed feature representation
[0076]
[0077] Next, a dual-channel structure is introduced to capture the dependencies of each pixel through the superpixel features in the local range, and extract the pixel features from the superpixel features. Specifically, the dual-channel structure mixes the features The test paper is divided into two parts, respectively recorded as and
[0078]
[0079] Among them, Φ DW (v) is the group convolution operation.
[0080] The above two features are combined by dynamic weighting. Perform Sigmoid activation to obtain weights; Perform GELU activation to obtain features. Finally, multiply the two elements by element to obtain weighted features
[0081]
[0082] Among them, σ refers to the Sigmoid activation function, φ is the GELU activation function, and ⊙ represents element-by-element multiplication.
[0083] Finally, the integrated feature map is passed through a 1*1 convolution P final (·) Weighted features Perform refinement to ensure output features Rich in global information and details:
[0084]
[0085] Furthermore, the selective kernel feature fusion module is configured to dynamically adjust the receptive field through two operations: fusion and selection. The fusion operation generates a global feature description by integrating the information of different branches; the selection operation uses these global descriptions to recalibrate and aggregate the feature maps of different branches.
[0086] Fusion Operation: The selective kernel feature fusion module receives information input at different scales from three parallel convolution streams. First, these multi-scale features are combined using element-wise summation. Then, global average pooling is used on the spatial dimension. Next, a channel-wise dimensionality reduction convolution layer is applied to generate a compact feature representation. Finally, the features are passed through three parallel channel convolutions (one for each resolution stream) to generate three features of dimension 1×1×C.
[0087] Selection operation: This operation applies a softmax activation function to the features of the three parallel channel convolutional layers to generate attention activations, which are used to adaptively recalibrate the features of the three branches.
[0088] In the embodiment of the present application, the loss function of the polarization image restoration model preferably adopts L1 loss, also known as absolute value loss or mean absolute error, which is a commonly used loss function in image restoration. It measures the error by calculating the absolute value of the difference between the model prediction value and the true value.
[0089]
[0090] Among them, x n is the predicted value, y n is the true value and N is the number of samples. By minimizing the L1 loss, the model learns how to effectively restore high-quality images from turbid images. A key advantage of the L1 loss is that it can produce sparse solutions, which helps feature selection and improves the quality of underwater polarization image restoration.
[0091] S3. Training the polarization image restoration model based on the underwater polarization image dataset.
[0092] S4. Obtain the polarization image group to be restored and input it into the trained polarization image restoration model, and obtain the output of the polarization image restoration model as a restored image.
[0093] The scheme and effect of the present invention are further described below based on specific application examples.
[0094] like Figure 4-6 As shown, the present embodiment discloses a Transformer-based underwater polarization image restoration method, and the detailed steps are as follows:
[0095] Step 1: Collect underwater polarization image data
[0096] An underwater passive polarization imaging experiment was conducted in an underwater environment. In order to enrich the types of objects photographed, models of sea cucumbers and scallops, ink bottles, milk cartons, and cans were placed. Figure 4 As shown, the volume of the glass water tank is (100*50*50cm 3 ). Fix the object at the bottom of a transparent water tank. Pour clean water to immerse the object. Use the polarization camera Blackfly SBFS-U3-51S5P-C to collect 4 polarization images at different angles (0°, 45°, 90°, 135°) at one time. First, use the camera to take a real image of the object in clean water as a reference image. Then drip about 2 ml of milk and take a set of polarization images of the same scene, which are four polarization images of 0°, 45°, 90° and 135°. Continue to drip milk and shoot until the water becomes turbid and the object cannot be photographed.
[0097] Step 2: Build an underwater polarization image dataset
[0098] A dataset of 1700 grayscale polarization images was constructed, including 45 scenes. The dataset was divided into training and test sets, of which 1400 sets of images were randomly selected as training sets and the remaining 300 sets were used as test sets. In order to speed up network training and enrich the dataset, the size of all images was unified to 512×512. Figure 5 Some of the acquired images are shown.
[0099] Step 3: Build an underwater polarization image restoration network design, such as Figure 2 As shown in the figure, a four-stage encoder-decoder network is used for deep feature extraction. The encoder captures deeper features, and the decoder uses these features to restore the details and structure of the image. Each stage contains a residual block of a multi-branch Transformer block. The network is mainly composed of multi-branch Transformer blocks.
[0100] The overall network design first uses 3*3 convolution to extract shallow polarization information features. In order to capture deeper features, the encoder uses downsampling operations to reduce the spatial resolution of the feature map, thereby capturing more deep polarization information. Next, the decoder uses upsampling operations to restore the spatial resolution of the feature map in order to use these features to restore the details and structure of the image, so as to gradually reconstruct the details of the image.
[0101] The multi-branch Transformer module mainly includes super token attention and dual-channel adaptive neural blocks. In order to fuse the polarization information features generated by different branches, a selective kernel feature fusion module is introduced at the end of the multi-branch Transformer block. The multi-branch structure can process polarization information in parallel, which not only enhances the model's ability to characterize polarization characteristics, but also improves the efficiency of processing and recovering feature information in polarization images.
[0102] Super token attention consists of super token sampling, multi-head self-attention, and token upsampling. The super token sampling part creates the initial super token by dividing the image into regular grid areas. Then, a method similar to soft k-means clustering is used to aggregate multiple original tokens into a smaller number of super tokens. This step reduces the amount of computation by reducing the number of tokens that need to be processed while retaining the key feature information of the image; after the super token is generated, multi-head self-attention is applied to these aggregated super tokens. Compared with the traditional self-attention mechanism, multi-head self-attention can process the context information focused on capturing different parts in parallel. In this way, the model can learn the complex relationship between local features and global features at the same time; token upsampling: After the super token captures the global information, token upsampling remaps this information back to the original token. The fusion of global polarization information and local polarization information is ensured, so that the model can capture local detail features while utilizing global information. Super token attention reduces the amount of computation through super token sampling, enhances the understanding of global context through multi-head self-attention, and restores local details through token upsampling.
[0103] The dual-channel adaptive neural block uses point-by-point convolution to mix superpixel features to generate a mixed feature representation. Grouped convolution is used to extract two channel slices from each channel. One of the slices introduces nonlinear features through the GELU activation function, while the other slice is converted into weights through the sigmoid activation function. The two feature representations are then integrated through an element-by-element multiplication gated mechanism, where the output of the sigmoid activation function is used as a weight to dynamically adjust the contribution of the other feature. Finally, the integrated feature map is reorganized through a 1x1 convolution. The model's ability to capture local details and global context is enhanced, and the importance of different features is effectively balanced through a dynamic weighting mechanism.
[0104] Selective kernel feature fusion is implemented through two main operations: fusion and selection. In the fusion stage, selective kernel feature fusion first sums the features from three parallel convolution streams element-wise to integrate multi-scale features. Then, global average pooling is performed to capture polarization information, and dimension reduction convolution is used to generate compact feature representations. Next, three parallel convolutions generate feature descriptors for each scale of feature streams, providing a basis for the selection stage. In the selection stage, a softmax activation function is applied to each branch feature to produce attention activations that can adaptively adjust each feature.
[0105] Step 4: Training and testing of underwater polarization image restoration network
[0106] In an embodiment of the present invention, an underwater polarization image restoration network designed in a Transformer-based underwater image polarization restoration method is used for training. The training data set is a collected underwater polarization image data set, and 1400 groups of images are randomly selected for training. In this embodiment, the size of all images is unified to 512×512. Random cropping and random flipping are used to expand the training data, and the initial learning rate is set to 2e-4, and cosine annealing is used to gradually reduce it to 1e-6. L1 loss is used as the loss function. The entire network training process is trained on an NVIDIA GeForce RTX3090Ti GPU using the PyTorch framework. After the training is completed, 300 randomly selected groups of images are used as test sets to verify the effectiveness of the method of the present invention. In addition, four physics-based methods and four deep learning-based methods are compared with the present invention, and the test results Figure 6 The present invention uses three evaluation indicators, namely Mean Squared Error (MSE), Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM), to objectively evaluate Figure 6 The performance of the test results in ,is shown in Table 1. The best and second best results are represented by red and blue, respectively, and "Ours" represents our method.
[0107] Table 1 Comparison of image restoration results of this method and other methods
[0108]
[0109] The Transformer-based underwater polarization image restoration method proposed in the present invention successfully eliminates the blurring caused by scattering in underwater images and effectively restores the polarization information of the image. This method not only solves the limitations of traditional deep learning methods in dealing with long-distance dependency problems, but also performs well in detail restoration, making the restored image closer to the original reference image. Compared with other existing technologies, the present invention has significant advantages in descattering and polarization information restoration of underwater polarization images, thus providing new perspectives and possibilities for the development of underwater polarization imaging technology.
[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A Transformer-based underwater polarization image restoration method, characterized in that: The following steps are involved: Acquire an underwater polarization image dataset, wherein the underwater polarization image dataset includes a clear underwater image and a corresponding polarization image group, wherein the polarization image group includes polarization images taken at different polarization angles; Constructing a polarization image restoration model, which includes a four-stage encoder-decoder network and a single residual block of a multi-branch Transformer block. The encoder-decoder network of each stage contains a residual block of a multi-branch Transformer block. The input image is subjected to 3*3 convolution for shallow feature extraction, and then the encoder of each stage captures deep polarization information. The decoder features of each stage are then used to restore the details and structure of the image. The information of the encoder and decoder of the same stage is aggregated using jump connections. Finally, a 3*3 convolution compression channel is used to output the reconstructed image. Training the polarization image restoration model based on the underwater polarization image dataset; The polarization image group to be restored is obtained and input into the trained polarization image restoration model, and the output of the polarization image restoration model is obtained as a restored image.
2. The Transformer-based underwater polarization image restoration method according to claim 1, characterized in that: The multi-branch Transformer block includes super token attention, dual adaptive neural block and selective kernel feature fusion module; the super token enhances the Transformer block's perception of the global context features of polarization information through super token sampling; the dual adaptive neural block effectively captures the global and local context information of the image through a dual-channel structure and a dynamic weighting strategy; the selective kernel feature fusion module fuses the polarization information features generated by different branches.
3. The Transformer-based underwater polarization image restoration method according to claim 2, characterized in that: The super token attention includes super token sampling, multi-head self-attention mechanism and token upsampling; The super token sampling is used to divide the extracted feature map into multiple area units, each area represents a basic token, the tokens are aggregated into super tokens, and then a multi-head self-attention mechanism is performed on the super tokens to capture the global dependencies between them. Finally, a token upsampling operation is performed to retain the local detail information of the image and remap the information of the super token back to the original token. The token is the basic processing unit of the image, and each token contains the pixel information of a specific local area in the image. The super token is a feature unit aggregated from multiple tokens, representing a larger area or a more abstract feature set in the image.
4. The Transformer-based underwater polarization image restoration method according to claim 3, characterized in that: The super token sampling is configured to perform the following steps: by average pooling, the image is divided into regular grid regions to create an initial super token S 0 , given a visual token Among them, N = H × W is the number of tokens, H is the height of the feature map, and W represents the width of the feature map. Assume that each token Belongs to m super tokens One of them, through an iterative process, calculates the XS sparse association matrix between tokens and super tokens Defined as: Where C is the number of channels, t is the number of iterations, and T is the transpose of the matrix; Update the super token to the weighted sum of the tokens and iterate according to the following formula: In the formula is the column-normalized Q t , d is the number of channels, t is the number of iterations, and T is the transpose of the matrix.
5. The Transformer-based underwater polarization image restoration method according to claim 4, characterized in that: The multi-head self-attention mechanism is configured to perform the following steps: each head evaluates the relevance between super tokens by linear transformation of query, key, and value, calculates the attention weight of each super token by softmax activation function based on the evaluation result, and then performs weighted summation on the value vector, and applies standard self-attention to the sampled super tokens. Defined as: in is the attention graph, where q, k, and v are query, key, and value respectively, and q(S) = SW q , k(S)=SW k and v(S) = SW v , respectively, the parameters are W q , W k and W v A linear function of q is a learnable parameter matrix corresponding to the query weights, W k is the learnable parameter matrix, corresponding to the key weights, W v is also a learnable parameter matrix, corresponding to the weights of the values.
6. The Transformer-based underwater polarization image restoration method according to claim 5, characterized in that: The token upsampling is configured to perform the following steps: mapping the super-token information processed by the multi-head self-attention mechanism back to the original token through the correlation matrix Q, thereby obtaining super-pixel features: in, Represents superpixel features.
7. The Transformer-based underwater polarization image restoration method according to claim 2, characterized in that: The dual adaptive neural block is configured to perform the following steps: Use point-by-point convolution to mix superpixel features and get the mixed feature representation in, represents the superpixel feature, P mix (·) represents point-wise convolution; Using the dual-channel structure, the mixed features The test paper is divided into two parts, respectively recorded as and Among them, Φ DW (·) is the group convolution operation, It is a mixed feature The first part of the features obtained after the group convolution operation, Mixed features Another part of the features obtained after the group convolution operation; Using dynamic weighting and Combined, including Perform Sigmoid activation to obtain the weights used to readjust the importance of polarization features. Perform GELU activation to obtain the polarization features after nonlinear transformation, and then multiply the two element by element to obtain the weighted features Among them, σ refers to the Sigmoid activation function, φ is the GELU activation function, and ⊙ represents element-by-element multiplication. Finally, the integrated feature map is passed through a 1*1 convolution P final (·) Weighted features Perform refinement to ensure output features Rich in global information and details:
8. The Transformer-based underwater polarization image restoration method according to claim 2, characterized in that: The selective kernel feature fusion module is configured to perform the following steps: dynamically adjust the receptive field through two operations: fusion and selection, wherein the fusion operation generates a global feature description by integrating information from different branches; and the selection operation uses these global descriptions to recalibrate and aggregate feature maps of different branches.
9. The Transformer-based underwater polarization image restoration method according to claim 1, characterized in that: The loss function during the training of the polarization image restoration model is: Among them, x n is the predicted value, y n is the true value, and N is the number of samples.
Citation Information
Cited By
Image restoration method
CN119809989A
Ultrahigh-definition image restoration method, device and equipment based on clustering center feature scanning
CN121391673A
Polarization prior double-current collaborative sharpening method for dense fog scene
CN121437319A
A polarized priori dual-flow cooperative dehazing method for thick fog scenes
CN121437319B
