Double-branch network method for polarization low-light image enhancement

Abstract

The invention provides a double-branch network for polarization low-light image enhancement, which belongs to the technical field of image processing, and comprises the following steps: obtaining four original polarization low-light images at different angles, and carrying out linear conversion on a given input polarization low-light image to obtain three images S0, S1 and S2 in a Stokes domain; the image S0 is input into a total light intensity network based on a multi-scale Transform model, an enhanced image # imgabs0 # is obtained, ReLU is used for replacing traditional softmax, and effective attention is reserved in the feature aggregation process; and S1 and S2 are input into a differential light intensity network based on multi-scale RDB to obtain an enhanced image # imgabs 1 # and an enhanced image # imgabs 2 #, and the image # imgabs 3 #, the image # imgabs 4 # and the image # imgabs 5 # are subjected to nonlinear conversion to obtain DoLP and AoP. The invention provides a double-branch network which is used for performing joint denoising on polarization images at different angles through interaction between channels and recovering global structure similarity, edge perception similarity and intensity-polarization information similarity of the images in a Stokes domain.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular, to a dual-branch network method for polarized low-light image enhancement. Background Art

[0002] The polarization imaging system decomposes the input field into four channels, each of which is detected by a photon counting sensor. However, the accuracy of the measurement is affected by the polarization of the light and the main type of noise. Since the polarizer inevitably filters out a portion of the scattered photons, if the scattered photons increase, or if the photon count is insufficient due to insufficient light, the signal-to-noise ratio of the detected image will be greatly reduced. Over the past few decades, researchers have conducted extensive research on low-light color image enhancement to cope with the problem of lower light intensity; one of the feasible methods is single-image low-light enhancement based on deep learning. These methods have become popular techniques for restoring damaged images in recent years, and they have significant advantages over traditional gamma correction and histogram equalization. However, the increase in brightness and contrast of the restored image will also amplify the white noise buried by real electromagnetic waves, which poses a huge challenge to measurement-related applications.

[0003] Denoising color polarization images is more challenging than denoising traditional images for three reasons: (1) limited intensity; (2) both DoLP and AoP are more susceptible to noise; and (3) there are physical correlations between different polarization channels. Therefore, it is particularly important to develop denoising algorithms specifically for polarization images. Summary of the Invention

[0004] In response to the technical problems mentioned in the above background technology, a dual-branch network method for polarization low-light image enhancement is provided. The present invention converts the color polarization image to the Stokes domain to handle the intrinsic error propagation from the same noise source in low-light environments, and proposes a dual-branch network for denoising the polarization image using a multi-scale fusion strategy; in order to restore the global structural similarity of the total intensity image S0, the model capability of the Transformer is used to capture the long-range dependency of the error, and a multi-scale hybrid network is designed to calculate the query while retaining an efficient attention mechanism for feature aggregation. In order to restore the edge perception information in the intensity difference images S1 and S2, a cascaded multi-scale RDB block is used to capture the polarization difference information.

[0005] The technical means adopted in the present invention are as follows:

[0006] A dual-branch network for polarization low-light image enhancement, comprising the following steps:

[0007] S01: Acquire four original polarized low-light images at different angles [I0,I 45 ,I 90,I 135 ], linearly transform the given input polarization low-light image into the Stokes domain, and obtain three images S0, S1 and S2 in the Stokes domain;

[0008] S02: Input S0 into the total light intensity network based on the multi-scale Transformer model and obtain the enhanced image Use ReLU instead of traditional softmax to retain effective attention during feature aggregation;

[0009] S03: Input S1 and S2 into the differential light intensity network based on multi-scale RDB to obtain the enhanced image and images

[0010] S04: Image image and images Perform nonlinear transformation to obtain DoLP and AoP.

[0011] Furthermore, in S01, in order to enable the network to capture the interaction between polarization channels, a Stokes domain linear transformation is performed in the denoising operation:

[0012]

[0013] S1=I 0° -I 90 °;

[0014] S2=I 45° -I 135° ;

[0015] Here, S0 represents the unpolarized intensity image, and S1 and S2 represent similar differential signals that carry the image polarization information.

[0016] Furthermore, in said S02, given the total light intensity image Where H×W represents the spatial resolution of the feature map;

[0017] In each Transformer block, given the l-1th block z l-1 The input features of the Transformer block are then reformulated as:

[0018]

[0019]

[0020] in, and z lThey represent the output features of the MSSA module, MSFM module and MLP of the lth layer respectively; LN represents layer normalization; represents the denoised image;

[0021] Furthermore, the MSSA module aggregates contextual information at multiple scales, initially using 1x1Conv for linear projection; in order to better extract feature information from the image, three convolutions are used to calculate the query Q, key K, and value V of different receptive fields;

[0022] In the process of calculating the query Q, three different Conv sizes are used: 3x3, 5x5 and 7x7; multi-scale data is used to calculate the query and key similarity between any pair of spatial positions; in the process of calculating the self-attention, the query Q, key K and value V are reshaped; after the reshaping Get sparse attention Three different sizes of Conv kernels are used to expand the receptive field size, and the sparse attention is added to the input image S0 to obtain the final denoised output:

[0023]

[0024] Furthermore, the calculation process of the reshaping is:

[0025]

[0026] Here, α represents a learnable parameter.

[0027] Furthermore, in said S03, the enhanced image is obtained in the differential light intensity network based on the multi-scale RDB and images The following steps are involved:

[0028] S031: Apply a 3x3 Conv to images S1 and S2, and perform Conv+ReLU operations to extract shallow features;

[0029] S032: Input shallow features into a series of RDBs and use continuous convolution operations to downsample the feature maps to capture local features at different scales:

[0030] F i =RDB i (F i +Deconv(F p(i) ));

[0031] Among them, RDB i () represents the RDB operation with scale i, 1; Deconv() means deconvolution; F i and Fp(i) Respectively represent the updated local features and prior local features extracted according to the scale;

[0032] S033: Using a U-shaped structure, the final feature fusion is:

[0033] F=fsuion(F i );

[0034] Among them, fusion() represents the fusion function;

[0035] S034: In order to fuse the features extracted at different scales, and The features extracted at the same scale are upsampled;

[0036] S035: and The features extracted at the same scale are upsampled and added to the original image for fusion;

[0037] S036: Use 3x3 Conv to convert enhanced features to image domain.

[0038] Compared with the prior art, the present invention has the following advantages:

[0039] 1. This paper proposes a dual-branch network for jointly denoising polarization images of different angles through inter-channel interaction, and restoring the global structural similarity, edge-aware similarity, and intensity-polarization information similarity of images in the Stokes domain.

[0040] 2. This paper proposes a multi-scale method that combines cross-layer information propagation and feature difference modeling, using the long-range dependency modeling of Transformer and the local feature extraction of RDB blocks to effectively restore polarization information during the denoising process. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0042] Figure 1 Schematic diagram of the architecture of the polarization image denoising model of the present invention.

[0043] Figure 2The figures are qualitative comparison and contrast diagrams of the present invention on the PLIE synthetic dataset, where Figure (a) 1, Figure (a) 2, and Figure (a) 3 are S0, DoLP, and AoP of the original polarization image; Figure (b) 1, Figure (b) 2, and Figure (b) 3 are schematic diagrams after enhancement by the IPLNet method; Figure (c) 1, Figure (c) 2, and Figure (c) 3 are schematic diagrams after enhancement by the ColorPolarNet method; Figure (d) 1, Figure (d) 2, and Figure (d) 3 are schematic diagrams after enhancement by the Polar-aware method; Figure (e) 1, Figure (e) 2, and Figure (e) 3 are schematic diagrams after enhancement by the present invention. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0045] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0046] like Figure 1 As shown, the present invention provides a dual-branch network for polarized low-light image enhancement, comprising the following steps:

[0047] S01: Acquire four original polarized low-light images at different angles [I0,I 45 ,I 90 ,I 135 ], the given input polarized low-light image is linearly transformed into the Stokes domain, and three images S0, S1 and S2 in the Stokes domain are obtained.

[0048] In this application, preferably, in S01, in order to enable the network to capture the interaction between polarization channels, a Stokes domain linear transformation is performed in the denoising operation:

[0049]

[0050] S1=I 0° -I 90° ;

[0051] S2=I 45° -I 135° ;

[0052] Here, S0 represents the unpolarized intensity image, and S1 and S2 represent similar differential signals that carry the image polarization information.

[0053] S02: Input S0 into the total light intensity network based on the multi-scale Transformer model and obtain the enhanced image ReLU is used instead of traditional softmax to retain effective attention in the feature aggregation process. In S02, given the total light intensity image Where H×W represents the spatial resolution of the feature map;

[0054] In each Transformer block, given the l-1th block z l-1 The input features of the Transformer block are then reformulated as:

[0055]

[0056] in, and z l They represent the output features of the MSSA module, MSFM module and MLP of the lth layer respectively; LN represents layer normalization; represents the denoised image.

[0057] Preferably, the MSSA module aggregates context information at multiple scales, initially using 1x1Conv for linear projection; in order to better extract feature information from the image, three convolutions are used to calculate the query Q, key K and value V of different receptive fields;

[0058] In the process of calculating the query Q, three different Conv sizes are used: 3x3, 5x5 and 7x7; multi-scale data is used to calculate the query and key similarity between any pair of spatial positions; in the process of calculating the self-attention, the query Q, key K and value V are reshaped; the calculation process of the reshaping is:

[0059]

[0060] Here, α represents a learnable parameter.

[0061] After remodeling Get sparse attention Three different sizes of Conv kernels are used to expand the receptive field size, and the sparse attention is added to the input image S0 to obtain the final denoised output:

[0062]

[0063] S03: Input S1 and S2 into the differential light intensity network based on multi-scale RDB to obtain the enhanced image and images

[0064] Furthermore, in S03, the enhanced image is obtained in the differential light intensity network based on the multi-scale RDB and images The following steps are involved:

[0065] S031: Apply a 3x3 Conv to images S1 and S2, and perform Conv+ReLU operations to extract shallow features;

[0066] S032: Input shallow features into a series of RDBs and use continuous convolution operations to downsample the feature maps to capture local features at different scales:

[0067] F i =RDB i (F i +Deconv(F p(i) ));

[0068] Among them, RDB i () represents the RDB operation with scale i, Deconv() means deconvolution; F i and F p(i) Respectively represent the updated local features and prior local features extracted according to the scale;

[0069] S033: Using a U-shaped structure, the final feature fusion is:

[0070] F=fusion(F i );

[0071] Among them, fusion() represents the fusion function;

[0072] S034: In order to fuse the features extracted at different scales, and The features extracted at the same scale are upsampled;

[0073] S035: and The features extracted at the same scale are upsampled and added to the original image for fusion;

[0074] S036: Use 3x3 Conv to convert enhanced features to image domain.

[0075] S04: Image image and images Perform nonlinear transformation to obtain DoLP and AoP.

[0076] Example:

[0077] For network training, we used an end-to-end model. The batch size was set to 4, and the initial learning rate was 0.0001. The model was optimized using the Adam optimizer, with β1 and β2 set to 0.9 and 0.999, respectively. The number of epochs was set to 400, and the learning rate was reduced to 0.5 times the original value after 400 epochs. All experiments were conducted on an NVIDIA 2080Ti GPU using the PyTorch framework.

[0078] Table 1 Quantitative comparison on the PLIE synthetic dataset

[0079]

[0080]

[0081] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0082] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0083] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0084] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0085] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0086] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A dual-branch network for polarized low-light image enhancement, characterized in that: The following steps are involved: S01: Acquire four original polarized low-light images at different angles [I0,I 45 ,I 90 ,I 135 ], linearly transform the given input polarization low-light image into the Stokes domain, and obtain three images S0, S1 and S2 in the Stokes domain; S02: Input image S0 into the total light intensity network based on the multi-scale Transformer model and obtain the enhanced image Use ReLU instead of traditional softmax to retain effective attention during feature aggregation; S03: Input images S1 and S2 into the differential light intensity network based on multi-scale RDB to obtain the enhanced image and images S04: Image image and images Perform nonlinear transformation to obtain DoLP and AoP.

2. A dual-branch network for polarization low-light image enhancement according to claim 1, characterized in that: In S01, in order to enable the network to capture the interaction between polarization channels, a Stokes domain linear transformation is performed in the denoising operation: S1=I 0° -IN 90° ; S2=I 45° -I 135° ; Here, S0 represents the unpolarized intensity image, and S1 and S2 represent similar differential signals that carry the image polarization information.

3. The dual-branch network for polarization low-light image enhancement according to claim 1, characterized in that: In said S02, given the total light intensity image Where H×W represents the spatial resolution of the feature map; In each Transformer block, given the l-1th block z l-1 The input features of the Transformer block are then reformulated as: in, and z l They represent the output features of the MSSA module, MSFM module and MLP of the lth layer respectively; LN represents layer normalization; represents the denoised image; 4. The dual-branch network for polarization low-light image enhancement according to claim 3, characterized in that: The MSSA module aggregates contextual information at multiple scales and initially uses 1x1 Conv for linear projection. In order to better extract feature information from the image, three convolutions are used to calculate the query Q, key K, and value V of different receptive fields. In the process of calculating the query Q, three different Conv sizes are used: 3x3, 5x5 and 7x7; multi-scale data is used to calculate the query and key similarity between any pair of spatial positions; in the process of calculating the self-attention, the query Q, key K and value V are reshaped; after the reshaping Get sparse attention Three different sizes of Conv kernels are used to expand the receptive field size, and the sparse attention is added to the input image S0 to obtain the final denoised output:

5. The dual-branch network for polarization low-light image enhancement according to claim 4, characterized in that: The calculation process of the reshaping is: Here, α represents a learnable parameter.

6. The dual-branch network for polarization low-light image enhancement according to claim 1, characterized in that: In the S03, the enhanced image is obtained in the differential light intensity network based on the multi-scale RDB and images The following steps are involved: S031: Apply a 3x3 Conv to images S1 and S2, and perform Conv+ReLU operations to extract shallow features; S032: Input shallow features into a series of RDBs and use continuous convolution operations to downsample the feature maps to capture local features at different scales: F i =RDB i (F i +Deconv(F p(i) )); Among them, RDB i () represents the RDB operation with scale i, 1; Deconv() means deconvolution; F i and F p(i) Respectively represent the updated local features and prior local features extracted according to the scale; S033: Using a U-shaped structure, the final feature fusion is: F=fusion(F i ); Among them, fusion() represents the fusion function; S034: In order to fuse the features extracted at different scales, and The features extracted at the same scale are upsampled; S035: and The features extracted at the same scale are upsampled and added to the original image for fusion; S036: Use 3x3 Conv to convert enhanced features to image domain.

Images