Image enhancement network based on inter-channel and intra-channel attention transformer and perfect flow

Abstract

The application discloses an image enhancement network based on inter-channel and intra-channel attention Transformer and complete flow, which comprises an inter-channel and intra-channel attention Transformer network and a reversible flow network, the inter-channel and intra-channel attention Transformer network comprises four zero elements, an inter-channel and intra-channel joint attention encoding network, four zero elements, an inter-channel and intra-channel joint attention decoding network, a down-sampling module and an up-sampling module; the four zero elements, the inter-channel and intra-channel joint attention encoding network comprises a local-global feature encoding module DSL1, a local-global feature encoding module DSL2, a local-global feature encoding module DSL3 and a local-global feature encoding module DSL4; the four zero elements, the inter-channel and intra-channel joint attention decoding network comprises a local-global feature decoding module UPL1, a local-global feature decoding module UPL2, a local-global feature decoding module UPL3 and a local-global feature decoding module UPL4; the low-light image enhancement network designed in the application can effectively realize the repair of missing information of an image, and the enhanced image generated based on the reversible flow network has excellent visual effect, and a more excellent peak signal-to-noise ratio and structural similarity.

Description

Technical Field

[0001] The present invention relates to the field of low-light image enhancement, and specifically to an image enhancement network designed based on inter-channel and intra-channel attention Transformer and complete flow. Background Art

[0002] In some low-light environments, due to insufficient light, short exposure times, or hardware limitations of the camera's CMOS image sensor, images captured often exhibit low visibility and contrast, severe color distortion, and unpredictable image noise. These low-light images suffer from low aesthetic quality, contain biased and missing information, and present significant challenges for subsequent advanced computer vision tasks such as object detection, object recognition, and scene understanding. Therefore, low-light image enhancement can improve the perceptual and aesthetic quality of images, thereby improving the overall performance of related computer vision tasks.

[0003] Current low-light image enhancement methods often focus on the appearance characteristics of low-light images, such as enhancing image brightness and suppressing noise. Furthermore, existing image enhancement methods rarely consider recovering information lost in low-light conditions. Low-light images often contain a large number of zero elements. Therefore, by marking these zero elements in low-light images and forming a binary zero element map, we can reflect the overall loss of image information.

[0004] In traditional flow reversible networks, the reversible transformation block is generally composed of only simple reversible activation, convolutional layer, affine injection, and affine transformation layer, and multiple convolutional layers (i.e., embedding network) are used to learn affine transformation parameters from conditional features. The lack of transformation diversity leads to inefficient parameter learning and fitting of normal distribution.

[0005] The patent uses a joint zero-element graph to construct inter-channel and intra-channel attention Transformer networks. The intra-channel attention Transformer realizes local feature extraction, and the inter-channel attention Transformer realizes global feature extraction. After cascading the inter-channel and intra-channel attention networks, relatively rich information feature extraction can be achieved. In addition, the zero-element graph can accurately locate the points of information distortion verification in the image, and then accurately guide the constructed attention Transformer network to effectively repair and enhance these distorted areas. Three reversible linear transformation layers, namely linear injection layer, conditional linear coupling layer, and unconditional linear coupling layer, are newly introduced, so that the network can better fit the normal distribution under the condition of linear and coupled superposition. At the same time, a cross-attention network (HWCAN) based on the height axis and width axis is introduced to replace the traditional multi-convolution layer (i.e., embedding network) to learn the affine / linear transformation parameters of the conditional feature driving layer to guide the reversible flow network for better learning and training. Summary of the Invention

[0006] The purpose of this paper is to design an image enhancement network based on inter-channel and intra-channel attention Transformer and complete flow to enhance the network's ability to repair image information loss caused by low-light conditions and improve the visual quality of the enhanced image.

[0007] To achieve the above objectives, the present invention adopts a technical solution: a low-light image enhancement network based on zero-element feature guidance, comprising the following steps:

[0008] Step 1: Data preprocessing;

[0009] Get a low-light image and find its zero element map;

[0010] Step 2: Construct the inter-channel and intra-channel joint zero-element graph attention Transformer-U network;

[0011] The inter-channel and intra-channel joint zero-element graph Transformer-U network includes: 4 zero-element, inter-channel and intra-channel joint attention encoding networks, 4 zero-element, inter-channel and intra-channel joint attention decoding networks, a downsampling module and an upsampling module;

[0012] Step 3: Extract triple conditional features from the inter-channel and intra-channel joint zero-element graph attention Transformer-U network constructed in step 2, and sample the standard normal distribution. Input it together with the triple conditional features into the reversible flow network, and use the height-axis and width-axis cross attention network (HWCAN) to learn the affine / linear transformation parameters of the conditional feature driving layer, guide the reversible transformation to fit the standard normal distribution, and generate the enhanced image by the reversible flow network;

[0013] The reversible flow network includes a scale transformation layer and a reversible transformation block. The reversible transformation block includes a reversible linear transformation block and a reversible coupling transformation block. Based on the existing network, three new reversible linear transformation blocks are introduced, including a linear injection layer, a conditional linear coupling layer, and an unconditional linear coupling layer.

[0014] The image enhancement network based on zero-element, inter-channel and intra-channel attention transformers and complete flow according to claim 1, characterized in that in the image enhancement network based on zero-element, inter-channel and intra-channel attention transformers and complete flow:

[0015] The four zero-element, inter-channel and intra-channel joint attention encoding networks and the four zero-element, inter-channel and intra-channel joint attention decoding networks in step 2 have similar extraction structures.

[0016] The four zero-element, inter-channel and intra-channel joint attention encoding networks include: a local-global feature encoding module DSL1, a local-global feature encoding module DSL2, a local-global feature encoding module DSL3 and a local-global feature encoding module DSL4;

[0017] The four zero-element, inter-channel and intra-channel joint attention decoding networks include: a local-global feature decoding module UPL1, a local-global feature decoding module UPL2, a local-global feature decoding module UPL3 and a local-global feature decoding module UPL4;

[0018] The downsampling module includes a downsampling module D1, a downsampling module D2 and a downsampling module D3;

[0019] The up-sampling module includes an up-sampling module U1, an up-sampling module U2 and an up-sampling module U3.

[0020] The inter-channel and intra-channel joint attention network is composed of a joint attention calculation network and a feedforward network connected in series; the joint attention calculation network is composed of three parts: inter-channel, intra-channel attention modules and zero-element attention modules, among which the intra-channel attention module and the zero-element attention module are connected in parallel to jointly construct the intra-channel attention module, and the inter-channel attention module is connected in series with the parallel result.

[0021] The inter-channel attention module is constructed by the following steps: Input an initial image feature X in , after the normalization layer LN1, three branches are separated, namely q it 、k it and v it, and respectively through 1×1 standard convolution and convolution kernel size 3×3 depth convolution DW, q it 、k it After the two branches perform transposed attention calculations, they obtain a channel attention matrix of C×C dimensions, which is consistent with v it After matrix multiplication, its output continues to serve as the input of the 1×1 standard convolution and is finally combined with the initial image feature X in Accumulate and get the output X of the inter-channel attention module it .

[0022] The intra-channel attention module and the zero-element attention module are constructed by the following steps: the output X of the inter-channel attention module is it As the input of this module, after the normalization layer LN2, four branches are divided, namely q ir 、k ir 、v ir and v zr , and respectively pass through the standard convolution with a convolution kernel size of 1×1 and the depth convolution DW2 with a convolution kernel size of 3×3. ir 、k ir 、v ir Calculate the attention matrix N within the channel separately 2 ×N 2 , v zr The zero-element attention module consists of two branches: q zr 、k zr , are derived from the zero element graph features, and v zr From the image features, after calculation, we get the zero-element attention matrix N 2 ×N 2 The two attention matrices obtained are respectively ir 、v zr After matrix multiplication, they are respectively subjected to 1×1 standard convolution and the output X of the inter-channel self-attention module it Add them separately and get the output X zr With X ir , and output the sum of the two to the feedforward network.

[0023] The feedforward network consists of three 1×1 standard convolutions, two depthwise convolutions with a convolution kernel size of 3×3, and an activation function Gelu.

[0024] The downsampling modules D1, D2, and D3 are respectively composed of convolutions with a kernel size of 4×4, a stride of 2, and a padding of 1;

[0025] The upsampling modules U1, U2, and U3 are respectively composed of convolutions with a kernel size of 2×2, a stride of 2, and a padding of 1;

[0026] The reversible network is divided into 1 to 3 levels, each level is composed of a scale transformation layer and 8 reversible transformation blocks connected in sequence. The reversible transformation block contains 8 cascaded reversible transformation layers, namely: reversible activation layer, 1×1 convolution layer, affine injection layer and cross-affine coupling layer.

[0027] The height-axis and width-axis cross attention network (HWCAN) first extracts triple features from the three feature decoding modules UPL2, UPL3, and UPL4, and then transforms them into output conditional features F through three 1×1 standard convolutions with different channel dimensions. t1 、F t2 、F t3 And merge the concatenated dimension with the stream feature X and input it into the HWCAN network. It is characterized by two reversible activation layers, two 1×1 convolution layers, and two activation functions Gelu, and the output P is obtained. im , then enter the height axis maximum pooling attention (HMA), width axis average pooling attention (WAA). First, along its input feature P im The height and width axes of each channel of size C×H×W are Avgpool / Maxpooled to generate a compressed feature of size C×W (or C×H). Then, the compressed feature is learned through 1×1 standard convolution-3×3 standard convolution-1×1 standard convolution-activation function sigmoid layer to generate a compressed weight of size C×W (or C×H). Then, the size is expanded to C×H×W through broadcasting, and the weights are added to the P im Multiply the weights to get the output P mm , and then P mm Maxpool / Avgpool is performed on the height and width axes of each channel of size C×H×W to generate a compressed feature of size C×H (or C×W). Then, the compressed feature is learned through 1×1 standard convolution-3×3 standard convolution-1×1 standard convolution-activation function sigmoid layer to generate a compressed weight of size C×H (or C×W). Then, the size is expanded to C×H×W by broadcasting, and the weights are added to the P mm Multiply the weights by P im Add up to get the output P om Finally, the final scale and shift (or only scaling) parameters are generated by the 1×1 learnable parameter convolution ConvZero to adapt to different affine / linear transformation layers, and finally the output of the cross attention network (HWCAN) based on the height axis and width axis is sent to the reversible flow network.

[0028] The reversible flow network is divided into 1 to 3 levels, each level is composed of a scale transformation layer and 8 reversible transformation blocks connected in sequence, and the reversible transformation block contains 8 cascaded reversible transformation layers, namely: reversible activation layer, 1×1 convolution layer, affine injection layer, linear injection layer, conditional affine coupling layer, conditional linear coupling layer, unconditional affine coupling layer and unconditional linear coupling layer.

[0029] The inter-channel and intra-channel attention transformer and the complete flow image enhancement network are characterized in that the process of training the inter-channel and intra-channel attention transformer and the complete flow image enhancement network is:

[0030] Furthermore, the training image dataset used includes pairs of low-light and normal-light images. During the data input phase, the low-light images used for training and their corresponding normal-light images are randomly cropped and rotated at the same location to increase the diversity of the training data, resulting in image pairs of size H×W for network training. During the model training phase, a zero-element map is first calculated for the low-light image, and the low-light image and its zero-element map are used as input to the joint-attention Transformer-U network. Simultaneously, the normal-light images are fed into the reversible flow network, where triple conditional features are derived from the joint-attention Transformer-U network and sampled from a standard normal distribution. These features, along with the conditional features, are fed into the reversible flow network. A height-axis and width-axis cross-attention network (HWCAN) is used to learn the affine / linear transformation parameters of the conditional feature-driven layer. The distribution of the normal-light images is mapped to the latent space, and the network parameters are optimized using maximum likelihood estimation to achieve a standard normal distribution fit to the latent space distribution. After the network training is completed, the standard normal distribution is sampled, and the triple conditional features are derived from the joint attention Transformer-U network as the input of the reversible flow network, and its reversible transformation is performed to finally generate an enhanced image.

[0031] Furthermore, based on the low-light image enhancement network guided by zero-element features, the loss function is:

[0032]

[0033] Among them, L(I L ,I GT ) represents the negative log-likelihood loss function, I L and I GT Represent the low-light image and its corresponding normal-light image, P y (y|I L ) represents the distribution of normal light images conditioned on low light images, P z (z) represents the latent space distribution, represents the Jacobian determinant value calculated by the reversible network during the transformation process, φ i and They represent the reversible transform block and the image features processed by it, respectively, and Ft represents the conditional features input to the reversible transform layer.

[0034] Based on the above technical solution, the beneficial effects of the present invention are:

[0035] The designed image enhancement network based on inter-channel and intra-channel attention Transformer and complete flow first transforms the low-light image and the zero element Figure 1 The input is fed into a Transformer-U network based on joint attention between and within channels, participating in different attention computations between and within channels. The intra-channel attention Transformer extracts local features, while the inter-channel attention Transformer extracts global features, enabling richer feature extraction. The zero-element intra-channel attention computation strengthens the network's learning of image regions with severely missing information, thereby accurately guiding the constructed attention Transformer network to effectively repair and enhance these distorted areas. The triple conditional features extracted by the joint attention Transformer-U network are fed into a reversible flow network, where three reversible linear transformation layers—a linear injection layer, a conditional linear coupling layer, and an unconditional linear coupling layer—are introduced. This allows the network to better fit the normal distribution under linear and coupled superposition conditions. Specifically, a height- and width-based cross-attention network (HWCAN) is introduced to replace the traditional multi-convolutional layers (i.e., embedding network) to learn the affine / linear transformation parameters of the conditional feature-driven layer, further improving the normal distribution fitting capability of the overall reversible flow network and providing better guidance for learning and training. In summary, the low-light image enhancement network designed in this paper can effectively repair missing image information. The enhanced images produced by the reversible flow network have excellent visual effects and better peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of the image enhancement network workflow based on inter-channel and intra-channel attention Transformer and complete flow;

[0037] Figure 2 Schematic diagram of the inter-channel and intra-channel attention Transformer-U network constructed for the joint zero-element graph;

[0038] Figure 3 Schematic diagram of the joint attention Transformer structure;

[0039] Figure 4 Schematic diagram of the cross attention network based on the height axis and width axis;

[0040] Figure 5 Schematic diagram of the reversible network structure;

[0041] Figure 6 Schematic diagram of enhanced image. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0043] This embodiment provides an image enhancement network based on inter-channel and intra-channel attention Transformer and complete flow, and its workflow is as follows: Figure 1 As shown, the specific steps include:

[0044] Step 1: Data preprocessing;

[0045] Obtain a low-light image and obtain its zero element map. The process of obtaining the zero element map can be expressed by formula (1):

[0046]

[0047] in Represents color pixel I L (i, j) At coordinates (i, j), the pixel values ​​of the red channel R (Red), green channel G (Green), and blue channel B (Blue) of the pixel point, i and j represent the height index and width index of the image respectively. Formula (1) indicates that by judging the low-light image pixel by pixel, if at least one of the three channels R, G, and B corresponding to a pixel point has a zero pixel value, then the pixel point is marked as a zero element, and the zero element map finally obtained is a binary image, and the pixel point with a value of 1 is a zero element. The zero element map marks the zero elements in the low-light image, which reflects the information missing of the image. The low-light image and the zero element map are used as input to the enhancement network to learn and restore the image information missing area. The low-light image input in this embodiment has a size of (3, 400, 600), that is, the number of channels is 3, the width and height are 400 and 600 respectively. The size of the obtained zero element map is the same as that of the low-light image in terms of the number of channels, width, and height.

[0048] Step 2: Construct the inter-channel and intra-channel joint zero-element graph attention Transformer-U network;

[0049] The inter-channel and intra-channel joint zero-element graph Transformer-U network includes: 4 zero-element, inter-channel and intra-channel joint attention encoding networks, 4 zero-element, inter-channel and intra-channel joint attention decoding networks, a downsampling module and an upsampling module;

[0050] Step 3: The triple conditional features are derived from the inter-channel and intra-channel joint zero-element graph attention Transformer-U network constructed in step 2, and the standard normal distribution is sampled. It is input into the reversible flow network together with the triple conditional features, and the height axis and width axis cross attention network (HWCAN) is used to learn the affine / linear transformation parameters of the conditional feature driving layer, guide the reversible transformation to fit the standard normal distribution, and generate the enhanced image by the reversible flow network.

[0051] The joint attention Transformer-U network includes: 4 zero-element, inter-channel and intra-channel joint attention encoding networks, 4 zero-element, inter-channel and intra-channel joint attention decoding networks, a downsampling module and an upsampling module.

[0052] The four zero-element, inter-channel and intra-channel joint attention encoding networks include: a local-global feature encoding module DSL1, a local-global feature encoding module DSL2, a local-global feature encoding module DSL3 and a local-global feature encoding module DSL4;

[0053] The four zero-element, inter-channel and intra-channel joint attention decoding networks include: a local-global feature decoding module UPL1, a local-global feature decoding module UPL2, a local-global feature decoding module UPL3 and a local-global feature decoding module UPL4;

[0054] The input size of the local-global feature encoding module DSL1 is (3, 256, 256), and the output is (48, 256, 256); the input size of the downsampling module D1 is (48, 256, 256), and the output is (96, 128, 128); the input size of the local-global feature encoding module DSL2 is (96, 128, 128), and the output is (96, 128, 128); the input size of the downsampling module D2 is (96, 12 8, 128), the output is (192, 64, 64); the input size of the local-global feature encoding module DSL3 is (192, 64, 64), the output is (192, 64, 64); the input size of the downsampling module D3 is (96, 128, 128), the output is (384, 32, 32); the input size of the local-global feature encoding module DF4 is (384, 32, 32), the output is (384, 32, 32); local- The input size of the global feature decoding module UPL1 is (384, 32, 32), and the output is (384, 32, 32); the input size of the upsampling module U1 is (384, 32, 32), and the output is (192, 64, 64); the input size of the local-global feature decoding module UPL2 is (384, 64, 64), and the output is (192, 64, 64); the input size of the upsampling module U2 is (192, 64, 64), and the output is is (96, 128, 128); the input size of the local-global feature decoding module UPL3 is (192, 128, 128), and the output is (192, 128, 128); the input size of the upsampling module U3 is (192, 128, 128), and the output is (96, 256, 256); the input size of the local-global feature decoding module UPL4 is (144, 256, 256), and the output is (144, 256, 256);

[0055] The local-global feature encoding / decoding module is as follows Figure 2 As shown, it is composed of a joint attention calculation network and a feedforward network connected in series; the joint attention calculation network is composed of three parts: inter-channel, intra-channel attention modules and zero-element attention modules, among which the intra-channel attention module and the zero-element attention module are connected in parallel to jointly construct the intra-channel attention module, and the inter-channel attention module is connected in series with the parallel result.

[0056] First, the inter-channel attention Transformer block. In the attention network A1, the 1×1 convolution kernel size is 1, and the convolution kernel size of the depth convolution DW1, DW2 and DW3 is 3×3, with a step size of 1. Their outputs will be matrix multiplied to obtain different weights according to different pixels. After the weight matrix is ​​processed by Softmax, it will be normalized to the range of 0 to 1. The more severe the information distortion, the closer the weight value of the pixel is to 0, and the more complete the information, the closer the weight value of the pixel is to 1. In the attention network A1, the image feature X is first input. in , whose size is C×H×W, after the normalization layer LN1, it is divided into 3 branches, namely q it 、k it and v it , and then pass through the depth convolution DW1, DW2, DW3 with a 1×1 convolution kernel size of 1, a convolution kernel size of 3×3, and a step size of 1, and get the output, where q it The branch passes through the built-in function Reshape to size C×HW, k it The branch passes through the built-in function Reshape to HW×C, v it The branch is reshaped to HW×C by the built-in function Reshape, and k it and q it The output after reshape is transposed and multiplied to get the channel attention matrix of size C×C. This channel attention matrix is ​​multiplied with v it The output of the branch after reshape is matrix multiplied to obtain a branch of size HW×C to assign different weights to different pixels of the image. The obtained HW×C size output is finally reshaped to size C×H×W, and then a 1×1 convolution is performed for further feature extraction. Finally, it is added to the initial image features to obtain the output X it .

[0057] In the attention network A2, the output X of the A1 network is it As the initial input, its size is C×H×W, after the normalization layer LN2, it is divided into 4 branches, namely q ir 、k ir 、v ir and v zr , where q ir 、k ir 、v ir Calculate the channel attention matrix N×N separately, v zr The zero-element graph introduced is used to calculate the zero-element-based channel attention matrix N×N. For the three branches q ir 、k ir 、v irAfter the output is obtained by sequentially passing through 1×1 convolution kernel size 1, 3×3 convolution kernel size, and depth convolution DW4, DW5, and DW6, the parameter N is introduced to prepare for the subsequent equal division of the single image block. ir The branch is reshaped by the built-in function Reshape in Indicates that a single image is divided into N equal parts in length and width, k ir The branch is reshaped by the built-in function Reshape v it The branch is reshaped by the built-in function Reshape K ir and q ir The output after reshape is transposed and multiplied to obtain the channel attention matrix of size N2×N2. This channel attention matrix is ​​multiplied with v it The output of the branch after reshape is then matrix multiplied to obtain a size of The branch assigns different weights to different pixels of the image, and the obtained The final reshape size of the output is C×H×W, and then a 1×1 convolution is performed for further feature extraction, and finally added to the initial image features to obtain the output X ir The calculation of the attention matrix of the zero-element graph is similar to the above steps, except that the zero-element graph does not undergo 1×1 convolution and 3×3 depth convolution, where v zr The branch comes from the image feature rather than the zero element map itself, and finally the output X is obtained. zr , output both X ir 、X zr Accumulate and get the output X out1 .

[0058] Finally, the feedforward network N1, which connects to the joint attention network, consists of a normalization layer, a 1×1 convolutional layer, a 3×3 depthwise convolutional layer, a Gelu activation function layer, and a 1×1 convolutional layer, followed by multiplication and accumulation. The feedforward network is designed to perform fully connected transformations and enhance feature extraction.

[0059] The downsampling module consists of a convolution with a kernel size of 4×4, a stride of 2, and a padding of 1. It can make the feature map of (C, H, W) become (2×C, H / 2, W / 2) after a downsampling module, mainly achieving the effect of doubling the number of channels and reducing the length and width by two times.

[0060] The upsampling module consists of a convolution with a kernel size of 2×2, a stride of 2, and a padding of 1. It can achieve expansion of the image size on the H×W scale. That is, a feature map of (C, H, W) will become (C / 2, 2×H, 2×W) after an upsampling module, mainly achieving the effect of doubling the number of channels and doubling the length and width scales.

[0061] As shown in 2, the triple features induced by the joint attention Transformer-U network are subjected to several simple convolution and linear interpolation operations to obtain Ft1, Ft2 and Ft3, with sizes of (288, 128, 128), (288, 64, 64), and (288, 32, 32), respectively. The conditional features will guide the reversible network to convert the standard normal distribution into an enhanced image.

[0062] The reversible network is as follows Figure 5 As shown in the figure, the network includes a scale transformation layer and a reversible transformation block. The reversible transformation block includes a reversible linear transformation block and a reversible coupling transformation block. Based on the existing network, three new reversible linear transformation blocks are introduced, including a linear injection layer, a conditional linear coupling layer, and an unconditional linear coupling layer.

[0063] Specifically, the reversible network uses rescaling layers to gradually resize the features processed by the network. Consequently, the entire network is divided into 1 to 3 levels, each consisting of a rescaling layer and eight reversible transformation blocks connected in sequence. The rescaling layer reversibly adjusts the feature size. During the forward transformation, it resizes the feature width and height by half, quadrupling the number of channels. In this embodiment, because the final normal illumination image size is (3, 256, 256), the feature sizes processed by reversible network levels 1 to 3 are (12, 128, 128), (48, 64, 64), and (192, 32, 32), respectively. The reversible transformation block includes eight cascaded reversible transformation layers: a reversible activation layer, a reversible 1×1 convolutional layer, an affine injection layer, a linear injection layer, a conditional affine coupling layer, a conditional linear coupling layer, an unconditional affine coupling layer, and an unconditional linear coupling layer. The affine injection layer, linear injection layer, conditional affine coupling layer, and conditional linear coupling layer use the conditional feature Ft1, Ft2, or Ft3 as one input of the layer, depending on the size of the currently processed feature. The reversible network achieves the conversion between the normal illumination image distribution and the standard normal distribution through these reversible transformation layers. The following is a further introduction to the eight reversible transformation layers:

[0064] The reversible activation layer implements the affine transformation of features through custom learnable parameters. Formula (6) and formula (7) describe the forward transformation and inverse transformation process of the reversible activation layer:

[0065] Y=S1 e X+T1 (2)

[0066] X=(Y-T1) / S1 (3)

[0067] Where ⊙ represents the element-by-element multiplication operation, X and Y represent the input and output of the flow feature, and S1 and T1 are learnable parameters. The feature undergoes the forward transformation of this layer, and the Jacobian value is calculated using formula (8):

[0068]

[0069] Where f1 and x1 represent the transformation function of formula (6) and its input features, respectively, and h and w represent the width and height of the feature, respectively.

[0070] The reversible 1×1 convolution layer realizes the feature transformation through the convolution layer with a convolution kernel size of 1×1. Formula (9) and Formula (10)

[0071] The forward and inverse transformation processes of the reversible 1×1 convolutional layer are described as follows:

[0072]

[0073] Where W is the convolution kernel matrix. The features are transformed in the forward direction of this layer, and the Jacobian determinant value is calculated by formula (11):

[0074]

[0075] Where f2 and x2 represent the transformation function of formula (9) and its input features, respectively, and h and w represent the width and height of the feature, respectively.

[0076] The affine injection layer takes the conditional features output by the joint attention Transformer encoding network as input and outputs the transformation coefficients to achieve a reversible affine transformation of the features. Formulas (12) and (13) describe the forward and inverse transformation processes of the reversible 1×1 convolutional layer:

[0077]

[0078] Where Ft represents the conditional feature, which outputs the transformation coefficients S3 and T3 through the parameter extraction network HWCAN. The HWCAN feature extraction network is as follows Figure 4 First, the three feature decoding modules UPL2, UPL3, and UPL4 are used to extract triple features, and three 1×1 standard convolutions are used to transform different channel dimensions to obtain the output conditional feature F. t1 、F t2 、F t3And merge the concatenated dimension with the stream feature X and input it into the HWCAN network. It is characterized by two reversible activation layers, two 1×1 convolution layers, and two activation functions Gelu, and the output P is obtained. im , then enter the height axis maximum pooling attention (HMA), width axis average pooling attention (WAA). First, along its input feature P im The height and width axes of each channel of size C×H×W are Avgpool / Maxpooled to generate a compressed feature of size C×W (or C×H). Then, the compressed feature is learned through 1×1 standard convolution-3×3 standard convolution-1×1 standard convolution-activation function sigmoid layer to generate a compressed weight of size C×W (or C×H). Then, the size is expanded to C×H×W through broadcasting, and the weights are added to the P im Multiply the weights to get the output P mm , and then P mm Maxpool / Avgpool is performed on the height and width axes of each channel of size C×H×W to generate a compressed feature of size C×H (or C×W). Then, the compressed feature is learned through 1×1 standard convolution-3×3 standard convolution-1×1 standard convolution-activation function sigmoid layer to generate a compressed weight of size C×H (or C×W). Then, the size is expanded to C×H×W by broadcasting, and the weights are added to the P mm Multiply the weights by P im Add up to get the output P om Finally, the final scale and shift (or only scaling) parameters are generated through the 1×1 learnable parameter convolution ConvZero to adapt to different affine / linear transformation layers. Finally, the output of the cross attention network based on the height axis and width axis is sent to the reversible flow network. The features are transformed in the forward direction of this layer, and the Jacobian determinant value is calculated by formula (14):

[0079]

[0080] Where f3 and x3 represent the transformation function of formula (12) and the characteristics of its input, respectively.

[0081] The linear injection layer takes the conditional features output by the joint attention Transformer encoding network as input and uses the HWCAN learning network to learn the scale parameters of the linear transformation from the conditional features Ft. Formulas (15) and (16) describe the forward and inverse transformation processes of the reversible 1×1 convolutional layer:

[0082]

[0083] Where S4 is a learnable scale parameter. The logarithmic determinant of the Jacobian matrix of the linear injection layer can be calculated as:

[0084]

[0085] Where f4 and x4 represent the transformation function of formula (15) and the characteristics of its input, respectively.

[0086] Conditional affine coupling learns the scale and displacement parameters of the affine transformation from the conditional features Ft and half of the input stream features through the HWCAN learning network. The training and inference process of conditional affine coupling can be expressed as:

[0087]

[0088]

[0089] Among them, Ft represents the conditional feature, (S a ,T a ) and (S b ,T b ) are all learnable transformation coefficients, which are output by the parameter extraction network HWCAN. The features undergo the forward transformation of this layer, and the Jacobian determinant value is calculated by formula (17):

[0090]

[0091] Where f5 and x5 represent the transformation function of formula (15) and its input flow characteristics respectively.

[0092] Conditional linear coupling, using HWCAN to learn the scale parameters of the linear transformation from the conditional features Ft and half of the input stream features. The training and inference of conditional linear coupling can be expressed as:

[0093]

[0094] Where S6 is a learnable scale parameter. The logarithmic determinant of the Jacobian matrix of the conditional linear coupling layer can be calculated as:

[0095]

[0096] Where f6 and x6 represent the transformation function of formula (15) and its input flow characteristics respectively.

[0097] Unconditional affine coupling learns the scale and displacement parameters of the affine transformation from half of the input stream features through HWCAN. The training and inference of unconditional affine coupling can be expressed as:

[0098]

[0099] Where S7 and T7 are learnable scale and shift parameters. The logarithmic determinant of the Jacobian matrix of the unconditional affine coupling layer can be calculated as:

[0100]

[0101] Where f7 and x7 represent the transformation function of formula (15) and its input flow characteristics respectively.

[0102] Unconditional linear coupling uses HWCAN to learn the scale parameters of the linear transformation from half of the input stream features. The training and inference of the unconditional linear coupling layer can be expressed as:

[0103]

[0104] Where S8 is a learnable scale parameter. The logarithmic determinant of the unconditional linear coupling Jacobian matrix can be calculated as:

[0105]

[0106] Where f8 and x8 represent the transformation function of formula (15) and its input flow characteristics respectively.

[0107] For each reversible transformation block φ in the reversible network i (i=1,2,…,K), the input features are sequentially passed through the reversible activation layer, 1×1 convolution layer, affine injection layer, linear injection layer, conditional affine coupling layer, conditional linear coupling layer, unconditional affine coupling layer and unconditional linear coupling layer, and the output features are Based on formula (8), formula (11), formula (14), formula (14), formula (14), formula (14), formula (14), and formula (17), the feature undergoes forward transformation of the reversible transformation block, and the Jacobian value is calculated by formula (18):

[0108]

[0109] The training process of the low-light image enhancement network based on zero-element feature guidance is as follows:

[0110] The training image dataset used consists of low-light and normal-light image pairs. During the data import phase, the low-light images used for training and their corresponding normal-light images are randomly cropped and rotated at the same position to increase the diversity of the training data, resulting in H×W image pairs for network training. During the model training phase, a zero-element map is generated for the low-light images. This map is then combined with the zero-element map to construct a cascaded intra- and inter-channel attention Transformer-U network, extracting triplet conditional features. Simultaneously, the normal-light images are fed into a reversible network. Guided by the extracted triplet conditional features, the network performs a forward transform, mapping the distribution of the normal-light images into a latent space. The network parameters are then optimized using maximum likelihood estimation to fit the latent space distribution to a standard normal distribution. After network training, the standard normal distribution is sampled and combined with the triplet conditional features extracted by the zero-element, intra-, and inter-channel cascaded attention Transformer-U networks as input to the reversible network, which then performs a reversible transform to generate the enhanced image. In this embodiment, the image dataset used is LOL-v2-real, which consists of low-light and normal-light image pairs taken in real scenes. The image size is (3,400,600), of which 689 pairs are used for model training and 100 pairs are used for model testing. In actual training, the image size is randomly cropped to (3,256,256). The loss function is:

[0111]

[0112] Among them, L(I L ,I GT ) represents the negative log-likelihood loss function, I L and I GT Represent the low-light image and its corresponding normal-light image, P y (y|I L ) represents the distribution of normal light images conditioned on low light images, P z (z) represents the latent space distribution, represents the Jacobian determinant value calculated by the reversible network during the transformation process, φ i and Represent the reversible transformation block and the image features processed by it, respectively, and Ft represents the conditional features input to the reversible transformation layer. This loss function optimizes the model parameters based on maximum likelihood estimation, enabling the reversible network to transform the normal illumination image distribution into a standard normal distribution. Based on the reversibility of the network, the standard normal distribution is sampled to ultimately generate an enhanced image.

[0113] The low-light image enhancement network trained in this example based on zero-element guidance is used to enhance 100 test images from the LOL-v2-real dataset. The average PSNR and SSIM of the enhanced 100 images are calculated, and the obtained results are 25.16dB and 0.893, which are better than most existing low-light image enhancement methods. Figure 6 As shown in the figure, the enhanced image has appropriate light intensity, normal colors and rich details, and the serious image information loss caused by low light environment is significantly restored.

Claims

1. Image enhancement network based on inter-channel and intra-channel attention transformer and complete flow, characterized by: The enhanced network includes inter-channel and intra-channel attention Transformer networks and reversible flow networks. The inter-channel and intra-channel attention Transformer network includes 4 zero-element, inter-channel and intra-channel joint attention encoding network, 4 zero-element, inter-channel and intra-channel joint attention decoding network, downsampling module and upsampling module; The four zero-element, inter-channel and intra-channel joint attention encoding networks include: a local-global feature encoding module DSL1, a local-global feature encoding module DSL2, a local-global feature encoding module DSL3 and a local-global feature encoding module DSL4; The four zero-element, inter-channel and intra-channel joint attention decoding networks include: a local-global feature decoding module UPL1, a local-global feature decoding module UPL2, a local-global feature decoding module UPL3 and a local-global feature decoding module UPL4; The reversible flow network includes: a scale transformation layer and a reversible transformation block; The reversible flow network is divided into 1 to 3 levels, each level is composed of a scale transformation layer and 8 reversible transformation blocks connected in sequence, and the reversible transformation block contains 8 cascaded reversible transformation layers, namely: reversible activation layer, 1×1 convolution layer, affine injection layer, linear injection layer, conditional affine coupling layer, conditional linear coupling layer, unconditional affine coupling layer and unconditional linear coupling layer; It also includes a feedforward network, which is composed of three 1×1 standard convolutions, two depthwise convolutions with a convolution kernel size of 3×3, and an activation function Gelu; the downsampling modules D1, D2, and D3 are respectively composed of convolutions with a convolution kernel size of 4×4, a stride of 2, and a padding of 1; the upsampling modules U1, U2, and U3 are respectively composed of convolutions with a convolution kernel size of 2×2, a stride of 2, and a padding of 1; The inter-channel and intra-channel joint attention network is composed of a joint attention calculation network and a feedforward network connected in series; the joint attention calculation network is composed of three parts: inter-channel, intra-channel attention modules and zero-element attention modules, among which the intra-channel attention module and the zero-element attention module are connected in parallel to jointly construct the intra-channel attention module, and the inter-channel attention module is connected in series with the parallel result.

2. The image enhancement network based on inter-channel and intra-channel attention transformer and complete flow according to claim 1, characterized in that The down-sampling module includes a down-sampling module D1, a down-sampling module D2 and a down-sampling module D3, and the up-sampling module includes an up-sampling module U1, an up-sampling module U2 and an up-sampling module U3.

3. The enhancement method based on the image enhancement network of inter-channel and intra-channel attention transformer and complete flow according to claim 2, characterized in that The following steps are included: Step 1: Data preprocessing: Obtain a low-light image and obtain its zero element map; Step 2: Construct the inter-channel and intra-channel joint zero-element graph attention Transformer-U network; Step 3: Extract the triple conditional features from the inter-channel and intra-channel joint zero-element graph attention Transformer-U network constructed in step 2, and sample the standard normal distribution. Input it together with the triple conditional features into the reversible flow network, and use the cross attention network of the height axis and width axis to learn the affine / linear transformation parameters of the conditional feature driving layer, guide the reversible transformation to fit the standard normal distribution, and generate the enhanced image by the reversible flow network.

4. The image enhancement method based on inter-channel and intra-channel attention transformer and complete flow according to claim 3, characterized in that The inter-channel attention module is constructed by the following steps: input an initial image feature , after the normalization layer , divided into 3 branches, namely 、 and , and respectively through 1×1 standard convolution and convolution kernel size 3×3 depth convolution DW, 、 After the two branches perform transposed attention calculations, they obtain a C×C channel attention matrix, which is After matrix multiplication, its output continues to serve as the input of 1×1 standard convolution and is finally combined with the initial image features. Accumulate and get the output of the inter-channel attention module .

5. The image enhancement method based on inter-channel and intra-channel attention transformer and complete flow according to claim 3, characterized in that The intra-channel attention module and the zero-element attention module are constructed by the following steps: the output of the inter-channel attention module As the input of this module, after the normalization layer , divided into 4 branches, namely 、 and , and respectively pass through the standard convolution with a convolution kernel size of 1×1 and the depth convolution DW2 with a convolution kernel size of 3×3, where 、 Calculate the attention matrix within the channel separately × , The attention is calculated by merging the two branches of the zero-element attention module and the zero-element attention module. The zero-element attention module contains two branches, namely 、 , are derived from the zero element graph features, and From the image features, after calculation, we get the zero-element attention matrix × , the two attention matrices obtained are respectively 、 After matrix multiplication, they are respectively subjected to 1×1 standard convolution and the output of the inter-channel self-attention module Add them separately and get the output and , and output the sum of the two to the feedforward network.

6. The image enhancement method based on inter-channel and intra-channel attention transformer and complete flow according to claim 3, characterized in that The cross attention network based on the height axis and the width axis in step 3 includes: First, the three feature decoding modules UPL2, UPL3, and UPL4 are used to extract triple features, and three 1×1 standard convolutions are used to transform different channel dimensions to obtain the output conditional features. 、 、 and with the flow characteristics Perform dimension merging and concatenated input into the HWCAN network.

7. The image enhancement method based on inter-channel and intra-channel attention transformer and complete flow according to claim 6, characterized in that The cross attention network based on height axis and width axis includes: Two reversible activation layers, two 1×1 convolutional layers, and two activation functions Gelu are used to get the output , then enter the maximum pooling attention on the height axis and the average pooling attention on the width axis; First, along its input features Avgpool / Maxpool is performed on the height and width axes of each channel of size C×H×W to generate compressed features of size C×W or C×H; Then, through 1×1 standard convolution - 3×3 standard convolution - 1×1 standard convolution - activation function sigmoid layer, the compression feature is learned to generate the compression weight of size C×W or C×H, and then the size is expanded to C×H×W by broadcasting, and Multiply the weights to get the output , and then Maxpool / Avgpool is performed on the height and width axes of each channel of size C×H×W to generate compressed features of size C×H or C×W. Then, the compressed features are learned through 1×1 standard convolution-3×3 standard convolution-1×1 standard convolution-activation function sigmoid layer to generate compressed weights of size C×H or C×W. The sizes are then expanded to C×H×W through broadcasting. Multiply the weights and Add to get the output ; Finally, the final scale and shift or only scaling parameters are generated through the 1×1 learnable parameter convolution ConvZero to adapt to different affine / linear transformation layers, and finally the output of the cross attention network based on the height axis and width axis is sent to the reversible flow network.

Images