Image enhancement network based on inter-channel and in-channel attention Transformer and complete flow

Abstract

The invention discloses an image enhancement network based on an inter-channel and in-channel attention Transform and a complete flow, and the network comprises an inter-channel and in-channel attention Transform network and a countercurrent network. The inter-channel and in-channel attention Transform network comprises four zero elements, an inter-channel and in-channel joint attention coding network, four zero elements, an inter-channel and in-channel joint attention decoding network, a down-sampling module and an up-sampling module; the inter-channel and intra-channel joint attention coding network comprises a local-global feature coding module DSL1, a local-global feature coding module DSL2, a local-global feature coding module DSL3 and a local-global feature coding module DSL4; the inter-channel and intra-channel joint attention decoding network comprises a local-global feature decoding module UPL1, a local-global feature decoding module UPL2, a local-global feature decoding module UPL3 and a local-global feature decoding module UPL4, and the inter-channel and intra-channel joint attention decoding network comprises a local-global feature decoding module UPL2, a local-global feature decoding module UPL3 and a local-global feature decoding module UPL4; the low-illumination image enhancement network designed by the invention can effectively realize restoration of missing information of the image, and the enhanced image generated based on the countercurrent network has an excellent visual effect and more excellent peak signal-to-noise ratio and structural similarity.

Description

Technical Field

[0001] The present invention relates to the field of low-light image enhancement, and specifically to an image enhancement network designed based on inter-channel and intra-channel attention Transformer and complete flow. Background Art

[0002] In some low-light environments, due to insufficient light, short exposure time or hardware limitations of the camera CMOS, the images taken often have low visibility and contrast, severe color distortion and unpredictable image noise. These low-light images have low aesthetic quality, and the information contained in the images themselves is also biased and missing, which also brings great difficulties to subsequent advanced computer vision tasks such as target detection, target recognition, and scene understanding. Therefore, low-light image enhancement can improve the perceptual quality and aesthetic quality of the image, so as to improve the overall performance of related computer vision tasks.

[0003] Current low-light image enhancement methods mostly consider image enhancement based on the appearance characteristics of low-light images, such as improving the brightness of the image and suppressing the noise in the image. In addition, existing image enhancement methods rarely consider the recovery of information lost in low light. There are often a large number of zero elements in low-light images. Therefore, by marking the zero elements in low-light images to form a binary zero element map, the overall loss of image information can be reflected.

[0004] In traditional flow reversible networks, the reversible transformation block is generally composed of only simple reversible activation, convolutional layer, affine injection, and affine transformation layer, and multiple convolutional layers (i.e., embedding network) are used to learn affine transformation parameters from conditional features. The lack of transformation diversity leads to inefficient parameter learning and fitting of normal distribution.

[0005] The patent uses a joint zero-element graph to construct an inter-channel and intra-channel attention Transformer network. The intra-channel attention Transformer realizes local feature extraction, and the inter-channel attention Transformer realizes global feature extraction. After cascading the inter-channel and intra-channel attention networks, relatively rich information feature extraction can be achieved. In addition, the zero-element graph can accurately locate the points of information distortion verification in the image, and then accurately guide the constructed attention Transformer network to effectively repair and enhance these distorted areas. Three reversible linear transformation layers, namely the linear injection layer, the conditional linear coupling layer, and the unconditional linear coupling layer, are newly introduced, so that the network can better fit the normal distribution in the case of linear and coupled superposition. At the same time, a cross-attention network (HWCAN) based on the height axis and the width axis is introduced to replace the traditional multi-convolutional layer (i.e., the embedding network) to learn the affine / linear transformation parameters of the conditional feature driving layer to guide the reversible flow network for better learning and training. Summary of the invention

[0006] The purpose of this invention is to design an image enhancement network based on inter-channel and intra-channel attention Transformer and complete flow to enhance the network's ability to repair image information loss caused by low-light conditions and improve the visual quality of enhanced images.

[0007] To achieve the above object, the technical solution adopted by the present invention is: a low-light image enhancement network based on zero-element feature guidance, comprising the following steps:

[0008] Step 1: Data preprocessing;

[0009] Get a low-light image and find its zero element map;

[0010] Step 2: Construct the inter-channel and intra-channel joint zero-element graph attention Transformer-U network;

[0011] The inter-channel and intra-channel joint zero-element graph Transformer-U network includes: 4 zero-element, inter-channel and intra-channel joint attention encoding networks, 4 zero-element, inter-channel and intra-channel joint attention decoding networks, a downsampling module and an upsampling module;

[0012] Step 3: The triple conditional features are derived from the inter-channel and intra-channel joint zero-element graph attention Transformer-U network constructed in step 2, and the standard normal distribution is sampled. The triple conditional features are input into the reversible flow network together with the triple conditional features, and the cross attention network (HWCAN) of the height axis and the width axis is used to learn the affine / linear transformation parameters of the conditional feature driving layer, guide the reversible transformation to fit the standard normal distribution, and generate the enhanced image by the reversible flow network;

[0013] The reversible flow network includes: a scale transformation layer and a reversible transformation block. The reversible transformation block includes a reversible linear transformation block and a reversible coupling transformation block. Based on the existing network, three reversible linear transformation blocks are newly introduced, including: a linear injection layer, a conditional linear coupling layer, and an unconditional linear coupling layer;

[0014] According to claim 1, the image enhancement network based on zero-element, inter-channel and intra-channel attention Transformer and complete flow is characterized in that in the image enhancement network based on zero-element, inter-channel and intra-channel attention Transformer and complete flow:

[0015] The 4 zero-element, inter-channel and intra-channel joint attention encoding networks and the 4 zero-element, inter-channel and intra-channel joint attention decoding networks in step 2 have similar extraction structures.

[0016] The four zero-element, inter-channel and intra-channel joint attention encoding networks include: a local-global feature encoding module DSL1, a local-global feature encoding module DSL2, a local-global feature encoding module DSL3 and a local-global feature encoding module DSL4;

[0017] The four zero-element, inter-channel and intra-channel joint attention decoding networks include: a local-global feature decoding module UPL1, a local-global feature decoding module UPL2, a local-global feature decoding module UPL3 and a local-global feature decoding module UPL4;

[0018] The down sampling module includes a down sampling module D1, a down sampling module D2 and a down sampling module D3;

[0019] The up-sampling module includes an up-sampling module U1, an up-sampling module U2 and an up-sampling module U3.

[0020] The inter-channel and intra-channel joint attention network is composed of a joint attention calculation network and a feedforward network connected in series in sequence; the joint attention calculation network is composed of three parts: an inter-channel attention module, an intra-channel attention module and a zero-element attention module, wherein the intra-channel attention module is connected in parallel with the zero-element attention module to jointly construct the intra-channel attention module, and the inter-channel attention module is connected in series with the parallel result.

[0021] The inter-channel attention module is constructed by the following steps: Input an initial image feature X in , after the normalization layer LN1, three branches are separated, namely q it , k it and v it, and respectively through 1×1 standard convolution and convolution kernel size 3×3 depth convolution DW, q it , k it After the two branches perform transposed attention calculations, they obtain a C×C channel attention matrix, which is combined with v it After matrix multiplication, its output continues to be the input of the 1×1 standard convolution and finally combined with the initial image feature X in The output X of the inter-channel attention module is obtained by accumulation it .

[0022] The intra-channel attention module and the zero-element attention module are constructed by the following steps: the output X of the inter-channel attention module it As the input of this module, after the normalization layer LN2, four branches are evenly divided, namely q ir , k ir 、v ir and v zr , and respectively pass through the standard convolution with a convolution kernel size of 1×1 and the depth convolution DW2 with a convolution kernel size of 3×3. ir , k ir 、v ir Calculate the attention matrix N within the channel separately 2 ×N 2 , v zr The zero-element attention module consists of two branches: q zr , k zr , all come from the zero element graph features, and v zr From the image features, after calculation, we get the zero-element attention matrix N 2 ×N 2 The two attention matrices obtained are respectively ir 、v zr After matrix multiplication, they are respectively subjected to 1×1 standard convolution and the output X of the self-attention module between channels it Add them separately to get the output X zr With X ir , and output the sum of the two to the feedforward network.

[0023] The feedforward network consists of three 1×1 standard convolutions, two depth convolutions with a convolution kernel size of 3×3, and an activation function Gelu.

[0024] The downsampling modules D1, D2, and D3 are respectively composed of convolutions with a convolution kernel size of 4×4, a stride of 2, and a padding of 1;

[0025] The upsampling modules U1, U2, and U3 are respectively composed of convolutions with a convolution kernel size of 2×2, a stride of 2, and a padding of 1;

[0026] The reversible network is divided into 1 to 3 levels, each level is composed of 1 scale transformation layer and 8 reversible transformation blocks connected in sequence, and the reversible transformation block contains 8 cascaded reversible transformation layers, namely: reversible activation layer, 1×1 convolution layer, affine injection layer and cross affine coupling layer.

[0027] The height-axis and width-axis-based cross attention network (HWCAN) first extracts triple features from the three feature decoding modules UPL2, UPL3, and UPL4, and then transforms them into output conditional features F through three 1×1 standard convolutions of different channel dimensions. t1 、F t2 、F t3 , and concatenated with the stream feature X and input into the HWCAN network. It is characterized by including two reversible activation layers, two 1×1 convolutional layers, and two activation functions Gelu, and the output P is obtained. im , and then enter the height axis maximum pooling attention (HMA), width axis average pooling attention (WAA). First, along its input feature P im The height and width axes of each channel of size C×H×W are Avgpool / Maxpooled to generate a compressed feature of size C×W (or C×H), and then the compressed feature is learned through 1×1 standard convolution-3×3 standard convolution-1×1 standard convolution-activation function sigmoid layer to generate a compressed weight of size C×W (or C×H), and then expanded to size C×H×W through broadcasting, and P im Multiply the weights to get the output P mm , and then P mm Maxpool / Avgpool is performed on the height and width axes of each channel of size C×H×W to generate a compressed feature of size C×H (or C×W), and then the compressed feature is learned through 1×1 standard convolution-3×3 standard convolution-1×1 standard convolution-activation function sigmoid layer to generate a compressed weight of size C×H (or C×W), and then expanded to size C×H×W through broadcasting, and P mm The weights are multiplied and then added to P im Add and get the output P om , and finally generate the final scale and shift (or just scaling) parameters through the 1×1 learnable parameter convolution ConvZero to adapt to different affine / linear transformation layers, and finally send the output of the cross attention network (HWCAN) based on the height axis and width axis to the reversible flow network.

[0028] The reversible flow network is divided into 1 to 3 levels, each level is composed of 1 scale transformation layer and 8 reversible transformation blocks connected in sequence, and the reversible transformation block contains 8 cascaded reversible transformation layers, namely: reversible activation layer, 1×1 convolution layer, affine injection layer, linear injection layer, conditional affine coupling layer, conditional linear coupling layer, unconditional affine coupling layer and unconditional linear coupling layer.

[0029] The inter-channel and intra-channel attention transformer and the complete flow image enhancement network are characterized in that the process of training the inter-channel and intra-channel attention transformer and the complete flow image enhancement network is:

[0030] Furthermore, the training image dataset used includes low-light and normal-light image pairs. In the data input stage, the low-light images used for training and their corresponding normal-light images are randomly cropped and rotated at the same position to improve the diversity of training data, and obtain image pairs of size H×W for network training. In the model training stage, the zero element map of the low-light image is first obtained, and the low-light image and its zero element map are used as the input of the joint attention Transformer-U network; at the same time, the normal light is input into the reversible flow network, and the triple conditional features are derived from the joint attention Transformer-U network, and the standard normal distribution is sampled, which is input into the reversible flow network together with the conditional features, and the cross attention network (HWCAN) of the height axis and the width axis is used to learn the affine / linear transformation parameters of the conditional feature driving layer, and the distribution of the normal light image is mapped to the latent space, and the network parameters are optimized based on the maximum likelihood estimation method to achieve the purpose of fitting the latent space distribution to the standard normal distribution. After the network training is completed, the standard normal distribution is sampled, and the triple conditional features are derived from the joint attention Transformer-U network as the input of the reversible flow network, and its reversible transformation is performed to finally generate an enhanced image.

[0031] Furthermore, based on the low-light image enhancement network guided by zero-element features, the loss function is:

[0032]

[0033] Among them, L(I L ,I GT ) represents the negative log-likelihood loss function, I L and I GT Represent the low-light image and its corresponding normal-light image, P y (y|I L ) represents the distribution of normal light images conditioned on low light images, P z (z) represents the latent space distribution, represents the Jacobian value calculated by the reversible network during the transformation process, φ i and They represent the reversible transform block and the image features processed by it respectively, and Ft represents the conditional features input to the reversible transform layer.

[0034] Based on the above technical solution, the beneficial effects of the present invention are:

[0035] The designed image enhancement network based on inter-channel and intra-channel attention Transformer and complete flow first transforms low-light images and zero elements Figure 1 The input is fed into the Transformer-U network based on the joint attention between channels and within channels, and participates in different attention calculations between channels and within channels. The intra-channel attention Transformer realizes local feature extraction, and the inter-channel attention Transformer realizes global feature extraction, which can realize relatively rich information feature extraction, and the zero-element intra-channel attention calculation can strengthen the network's learning of the image information missing areas, and then accurately guide the constructed attention Transformer network to effectively repair and enhance these distorted areas; the triple conditional features extracted by the joint attention Transformer-U network are input into the reversible flow network, and three reversible linear transformation layers, namely the linear injection layer, the conditional linear coupling layer, and the unconditional linear coupling layer, are newly introduced into the reversible flow network, so that the network can better fit the normal distribution under the condition of linear and coupled superposition. Among them, the cross attention network (HWCAN) based on the height axis and the width axis is introduced to replace the traditional multi-convolutional layer (i.e., the embedding network) to learn the affine / linear transformation parameters of the conditional feature driving layer, further improving the normal distribution fitting ability of the overall reversible flow network, which can better guide the reversible flow network to learn and train better. In summary, the low-light image enhancement network designed in the present invention can effectively restore the missing information of the image. The enhanced image produced based on the reversible flow network has excellent visual effects and better peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of the workflow of the image enhancement network based on inter-channel and intra-channel attention Transformer and complete flow;

[0037] Figure 2 Schematic diagram of the inter-channel and intra-channel attention Transformer-U network constructed for the joint zero-element graph;

[0038] Figure 3 Schematic diagram of the joint attention Transformer structure;

[0039] Figure 4 Schematic diagram of the cross attention network based on the height axis and width axis;

[0040] Figure 5 Schematic diagram of the reversible network structure;

[0041] Figure 6 Schematic diagram of enhanced image. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0043] This embodiment provides an image enhancement network based on inter-channel and intra-channel attention Transformer and complete flow, and its workflow is as follows: Figure 1 As shown, the specific steps include:

[0044] Step 1: Data preprocessing;

[0045] Get a low-light image and find its zero element map. The process of finding the zero element map can be expressed by formula (1):

[0046]

[0047] in Represents color pixel I L (i, j) At the coordinate (i, j), the pixel values ​​of the red channel R (Red), green channel G (Green) and blue channel B (Blue) of the pixel point, i, j represent the height index and width index of the image respectively. Formula (1) indicates that by judging the low-light image pixel by pixel, if there is at least one channel pixel value of zero in the three channels R, G, and B corresponding to a certain pixel point, the pixel point is marked as a zero element, and the zero element map finally obtained is a binary image, and the pixel point with a value of 1 is a zero element. The zero element map marks the zero elements in the low-light image, which reflects the information loss of the image. The low-light image and the zero element map are used as the input of the enhancement network for the network to learn and restore the image information loss area. The size of the low-light image input in this embodiment is (3, 400, 600), that is, the number of channels is 3, the width and height are 400 and 600 respectively, and the size of the obtained zero element map is the same as that of the low-light image in terms of the number of channels, width and height.

[0048] Step 2: Construct the inter-channel and intra-channel joint zero-element graph attention Transformer-U network;

[0049] The inter-channel and intra-channel joint zero-element graph Transformer-U network includes: 4 zero-element, inter-channel and intra-channel joint attention encoding networks, 4 zero-element, inter-channel and intra-channel joint attention decoding networks, a downsampling module and an upsampling module;

[0050] Step 3: The triple conditional features are derived from the inter-channel and intra-channel joint zero-element graph attention Transformer-U network constructed in step 2, and the standard normal distribution is sampled. It is input into the reversible flow network together with the triple conditional features, and the cross attention network (HWCAN) of the height axis and width axis is used to learn the affine / linear transformation parameters of the conditional feature driving layer, guide the reversible transformation to fit the standard normal distribution, and generate an enhanced image by the reversible flow network.

[0051] The joint attention Transformer-U network includes: 4 zero-element, inter-channel and intra-channel joint attention encoding networks, 4 zero-element, inter-channel and intra-channel joint attention decoding networks, a downsampling module and an upsampling module.

[0052] The four zero-element, inter-channel and intra-channel joint attention encoding networks include: a local-global feature encoding module DSL1, a local-global feature encoding module DSL2, a local-global feature encoding module DSL3 and a local-global feature encoding module DSL4;

[0053] The four zero-element, inter-channel and intra-channel joint attention decoding networks include: a local-global feature decoding module UPL1, a local-global feature decoding module UPL2, a local-global feature decoding module UPL3 and a local-global feature decoding module UPL4;

[0054] The input size of the local-global feature encoding module DSL1 is (3, 256, 256), and the output is (48, 256, 256); the input size of the downsampling module D1 is (48, 256, 256), and the output is (96, 128, 128); the input size of the local-global feature encoding module DSL2 is (96, 128, 128), and the output is (96, 128, 128); the input size of the downsampling module D2 is (96, 12 8, 128), the output is (192, 64, 64); the input size of the local-global feature encoding module DSL3 is (192, 64, 64), the output is (192, 64, 64); the input size of the downsampling module D3 is (96, 128, 128), the output is (384, 32, 32); the input size of the local-global feature encoding module DF4 is (384, 32, 32), the output is (384, 32, 32); local- The input size of the global feature decoding module UPL1 is (384, 32, 32), and the output is (384, 32, 32); the input size of the upsampling module U1 is (384, 32, 32), and the output is (192, 64, 64); the input size of the local-global feature decoding module UPL2 is (384, 64, 64), and the output is (192, 64, 64); the input size of the upsampling module U2 is (192, 64, 64), and the output is is (96, 128, 128); the input size of the local-global feature decoding module UPL3 is (192, 128, 128), and the output is (192, 128, 128); the input size of the upsampling module U3 is (192, 128, 128), and the output is (96, 256, 256); the input size of the local-global feature decoding module UPL4 is (144, 256, 256), and the output is (144, 256, 256);

[0055] The local-global feature encoding / decoding module is as follows Figure 2 As shown, it is composed of a joint attention calculation network and a feedforward network which are connected in series in sequence; the joint attention calculation network is composed of three parts: an inter-channel attention module, an intra-channel attention module and a zero-element attention module, wherein the intra-channel attention module is connected in parallel with the zero-element attention module to jointly construct the intra-channel attention module, and the inter-channel attention module is connected in series with the parallel result.

[0056] First, the inter-channel attention Transformer block. In the attention network A1, the 1×1 convolution kernel size is 1, the convolution kernel size of the deep convolution DW1, DW2 and DW3 is 3×3, and the step size is 1. Their outputs will be matrix multiplied to obtain different weights for different pixels. After the Softmax processing, the weight matrix will be normalized to the range of 0 to 1. The more severely distorted the information, the closer the weight value of the pixel is to 0, and the more intact the information, the closer the weight value of the pixel is to 1. In the attention network A1, the image feature X is first input. in , whose size is C×H×W, after the normalization layer LN1, it is divided into three branches, namely q it , k it and v it , and then pass through the 1×1 convolution kernel size of 1, the convolution kernel size of 3×3, and the depth convolution DW1, DW2, and DW3 with a step size of 1, and get the output, where q it The branch passes through the built-in function Reshape to size C×HW, k it The branch passes through the built-in function Reshape to HW×C, v it The branch is reshaped to HW×C by the built-in function Reshape. it and q it The reshaped output is transposed and matrix multiplied to obtain a C×C channel attention matrix. This channel attention matrix is ​​multiplied by v it The output of the branch reshape is then matrix multiplied to obtain a branch of size HW×C to assign different weights to different pixels of the image. The HW×C size output is finally reshaped to size C×H×W, and then a 1×1 convolution is performed for further feature extraction. Finally, it is added to the initial image features to obtain the output X it .

[0057] In the attention network A2, the output X of the A1 network is it As the initial input, its size is C×H×W. After the normalization layer LN2, it is divided into 4 branches, namely q ir , k ir 、v ir and v zr , where q ir , k ir 、v ir Calculate the channel attention matrix N×N separately, v zr The zero-element graph introduced is used to calculate the zero-element-based channel attention matrix N×N. For the three branches introduced q ir , k ir 、v irAfter the output is obtained by sequentially passing through 1×1 convolution kernel size 1, convolution kernel size 3×3, and depth convolution DW4, DW5, and DW6, the parameter N is introduced to prepare for the subsequent equal division of single image blocks. ir The branch size is Reshape after the built-in function in Indicates that a single image is divided into N equal parts in length and width, k ir The branch size is Reshape after the built-in function v it The branch size is Reshape after the built-in function K ir and q ir The reshaped output is transposed and matrix multiplied to obtain an N2×N2 channel attention matrix. This channel attention matrix is ​​multiplied by v it The output of the branch reshape is then matrix multiplied to obtain a size of The branches are used to assign different weights to different pixels of the image. The size output is finally reshaped to C×H×W size, and then a 1×1 convolution is performed for further feature extraction, and finally added to the initial image features to get the output X ir ; The calculation of the attention matrix of the zero-element graph is similar to the above steps, except that the zero-element graph does not undergo 1×1 convolution and 3×3 depth convolution, where v zr The branch comes from the image features rather than the zero element map itself, and finally the output X is obtained. zr , output both X ir , X zr The output is X. out1 .

[0058] Finally, there is the feedforward network N1 connected to the joint attention network. The feedforward network is composed of a normalization layer, a 1×1 convolution layer, a 3×3 deep convolution layer, a Gelu activation function layer, and a 1×1 convolution layer. The feedforward network is designed to perform full connection transformation and enhance feature extraction.

[0059] The downsampling module consists of a convolution with a kernel size of 4×4, a stride of 2, and a padding of 1. It can make the feature map of (C, H, W) become (2×C, H / 2, W / 2) after a downsampling module, mainly achieving the effect of doubling the number of channels and reducing the length and width by two times.

[0060] The upsampling module consists of a convolution with a kernel size of 2×2, a stride of 2, and a padding of 1. It can achieve expansion of the image size scale H×W, that is, a feature map of (C, H, W) will become (C / 2, 2×H, 2×W) after an upsampling module, mainly achieving the effect of reducing the number of channels by two times and expanding the length and width by two times.

[0061] As shown in 2, the triple features induced by the joint attention Transformer-U network are subjected to several simple convolution and linear interpolation operations to obtain Ft1, Ft2 and Ft3, with sizes of (288, 128, 128), (288, 64, 64), and (288, 32, 32), respectively. The conditional features will guide the reversible network to convert the standard normal distribution into an enhanced image.

[0062] The reversible network is Figure 5 As shown, the network includes: a scale transformation layer and a reversible transformation block. The reversible transformation block includes a reversible linear transformation block and a reversible coupling transformation block. Based on the existing network, three reversible linear transformation blocks are newly introduced, including: a linear injection layer, a conditional linear coupling layer, and an unconditional linear coupling layer.

[0063] Specifically, the reversible network changes the size of the network processing features step by step through the scale transformation layer, so the entire network is divided into 1 to 3 levels, each level is composed of 1 scale transformation layer and 8 reversible transformation blocks connected in sequence. The scale transformation layer can reversibly adjust the size of the feature. In the process of forward transformation, by re-adjusting the feature size, the width and height of the feature are reduced to half of the original, and the number of channels is expanded to 4 times the original. In this embodiment, because the size of the normal illumination image generated at the end is (3,256,256), the feature sizes processed by the reversible network 1 to 3 are (12,128,128), (48,64,64) and (192,32,32) respectively. The reversible transformation block contains 8 cascaded reversible transformation layers, namely: reversible activation layer, reversible 1×1 convolution layer, affine injection layer, linear injection layer, conditional affine coupling layer, conditional linear coupling layer, unconditional affine coupling layer and unconditional linear coupling layer. The affine injection layer, linear injection layer, conditional affine coupling layer and conditional linear coupling layer will use the conditional feature Ft1, Ft2 or Ft3 as one input of the layer according to the size of the currently processed feature. The reversible network realizes the conversion between the normal illumination image distribution and the standard normal distribution through these reversible transformation layers. The following is a further introduction to the 8 reversible transformation layers:

[0064] The reversible activation layer implements the affine transformation of features through custom learnable parameters. Formula (6) and formula (7) describe the forward transformation and inverse transformation process of the reversible activation layer:

[0065] Y=S1 e X+T1 (2)

[0066] X=(Y-T1) / S1 (3)

[0067] Where ⊙ represents the element-by-element multiplication operation, X and Y represent the input and output of the flow feature, and S1 and T1 are learnable parameters. The feature undergoes the forward transformation of this layer, and the Jacobian value is calculated using formula (8):

[0068]

[0069] Where f1 and x1 represent the transformation function of formula (6) and its input features, respectively, and h and w represent the width and height of the feature, respectively.

[0070] The reversible 1×1 convolutional layer realizes the feature transformation through the convolutional layer with a convolution kernel size of 1×1. Formula (9) and Formula (10)

[0071] The process of forward and inverse transformation of a reversible 1×1 convolutional layer is described as follows:

[0072]

[0073] Where W is the convolution kernel matrix. The features are transformed forward in this layer, and the Jacobian determinant is calculated by formula (11):

[0074]

[0075] Where f2 and x2 represent the transformation function of formula (9) and its input features, respectively, and h and w represent the width and height of the feature, respectively.

[0076] The affine injection layer takes the conditional features output by the joint attention Transformer encoding network as input and outputs the transformation coefficients to achieve a reversible affine transformation of the features. Formulas (12) and (13) describe the forward and inverse transformation processes of the reversible 1×1 convolutional layer:

[0077]

[0078] Where Ft represents the conditional feature, which outputs the transformation coefficients S3 and T3 through the parameter extraction network HWCAN. The HWCAN feature extraction network is as follows Figure 4 First, the three feature decoding modules UPL2, UPL3, and UPL4 are used to extract triple features, and then three 1×1 standard convolutions are performed to transform different channel dimensions to obtain the output conditional feature F. t1 、F t2 、F t3, and concatenated with the stream feature X and input into the HWCAN network. It is characterized by including two reversible activation layers, two 1×1 convolutional layers, and two activation functions Gelu, and the output P is obtained. im , and then enter the height axis maximum pooling attention (HMA), width axis average pooling attention (WAA). First, along its input feature P im The height and width axes of each channel of size C×H×W are Avgpool / Maxpooled to generate a compressed feature of size C×W (or C×H), and then the compressed feature is learned through 1×1 standard convolution-3×3 standard convolution-1×1 standard convolution-activation function sigmoid layer to generate a compressed weight of size C×W (or C×H), and then expanded to size C×H×W through broadcasting, and P im Multiply the weights to get the output P mm , and then P mm Maxpool / Avgpool is performed on the height and width axes of each channel of size C×H×W to generate a compressed feature of size C×H (or C×W), and then the compressed feature is learned through 1×1 standard convolution-3×3 standard convolution-1×1 standard convolution-activation function sigmoid layer to generate a compressed weight of size C×H (or C×W), and then expanded to size C×H×W through broadcasting, and P mm The weights are multiplied and then added to P im Add and get the output P om , and finally generate the final scale and shift (or only scaling) parameters through the 1×1 learnable parameter convolution ConvZero to adapt to different affine / linear transformation layers, and finally send the output of the cross attention network based on the height axis and width axis to the reversible flow network. The feature is transformed through the forward transformation of this layer, and the Jacobian determinant value is calculated by formula (14):

[0079]

[0080] Where f3 and x3 represent the transformation function of formula (12) and its input features respectively.

[0081] The linear injection layer takes the conditional features output by the joint attention Transformer encoding network as input and uses the HWCAN learning network to learn the scale parameters of the linear transformation from the conditional features Ft. Formulas (15) and (16) describe the forward and inverse transformation processes of the reversible 1×1 convolutional layer:

[0082]

[0083] Where S4 is a learnable scale parameter. The logarithmic determinant of the Jacobian matrix of the linear injection layer can be calculated as:

[0084]

[0085] Where f4 and x4 represent the transformation function of formula (15) and its input features respectively.

[0086] Conditional affine coupling learns the scale and displacement parameters of affine transformation from the conditional features Ft and half of the input stream features through the HWCAN learning network. The training and inference process of conditional affine coupling can be expressed as:

[0087]

[0088]

[0089] Among them, Ft represents the conditional characteristics, (S a ,T a ) and (S b ,T b ) are all learnable transformation coefficients, which are output by the parameter extraction network HWCAN. The features are transformed through the forward transformation of this layer, and the Jacobian determinant value is calculated by formula (17):

[0090]

[0091] Where f5 and x5 represent the transformation function of formula (15) and its input flow characteristics respectively.

[0092] Conditional linear coupling, through HWCAN, learns the scale parameters of linear transformation from the conditional features Ft and half of the input stream features. The training and inference of conditional linear coupling can be expressed as:

[0093]

[0094] Where S6 is a learnable scale parameter. The logarithmic determinant of the Jacobian matrix of the conditional linear coupling layer can be calculated as:

[0095]

[0096] Where f6 and x6 represent the transformation function of formula (15) and its input flow characteristics respectively.

[0097] Unconditional affine coupling learns the scale and displacement parameters of the affine transformation from half of the input stream features through HWCAN. The training and inference of unconditional affine coupling can be expressed as:

[0098]

[0099] Where S7 and T7 are learnable scale and shift parameters. The logarithmic determinant of the Jacobian matrix of the unconditional affine coupling layer can be calculated as:

[0100]

[0101] Where f7 and x7 represent the transformation function of formula (15) and its input flow characteristics respectively.

[0102] Unconditional linear coupling uses HWCAN to learn the scale parameters of the linear transformation from half of the input stream features. The training and inference of the unconditional linear coupling layer can be expressed as:

[0103]

[0104] Where S8 is a learnable scale parameter. The logarithmic determinant of the unconditional linear coupling Jacobian matrix can be calculated as:

[0105]

[0106] Where f8 and x8 represent the transformation function of formula (15) and its input flow characteristics respectively.

[0107] For each reversible transformation block φ in the reversible network i (i=1,2,…,K), the input features are sequentially passed through the reversible activation layer, 1×1 convolution layer, affine injection layer, linear injection layer, conditional affine coupling layer, conditional linear coupling layer, unconditional affine coupling layer and unconditional linear coupling layer, and the output features are Based on formula (8), formula (11), formula (14), formula (14), formula (14), formula (14), formula (14), and formula (17), the feature undergoes forward transformation of the reversible transformation block, and the Jacobian determinant value is calculated by formula (18):

[0108]

[0109] The training process of the low-light image enhancement network guided by zero-element features is as follows:

[0110] The training image dataset used contains low-light-normal-light image pairs. In the data import stage, the low-light images used for training and their corresponding normal-light images are randomly cropped and rotated at the same position to improve the diversity of training data, and image pairs of size H×W are obtained for network training. In the model training stage, the zero element map is obtained for the low-light image, and the in-channel and out-channel cascaded attention Transformer-U network is constructed with the zero element map, and the triple conditional features are extracted; at the same time, the normal light is input into the reversible network, and under the guidance of the extracted triple conditional features, the forward transformation of the network is performed, the distribution of the normal light image is mapped to the latent space, and the network parameters are optimized based on the maximum likelihood estimation method to achieve the purpose of fitting the latent space distribution to the standard normal distribution. After the network training is completed, the standard normal distribution is sampled, and the triple conditional features extracted by the zero element, in-channel, and inter-channel cascaded attention Transformer-U network are used as the input of the reversible network, and its reversible transformation is performed to finally generate an enhanced image. In this embodiment, the image dataset used is LOL-v2-real, which is a low-light-normal-light image pair taken in a real scene, with an image size of (3,400,600), of which 689 pairs are used for model training and 100 pairs are used for model testing. In actual training, the image size is randomly cropped to (3,256,256). The loss function is:

[0111]

[0112] Among them, L(I L ,I GT ) represents the negative log-likelihood loss function, I L and I GT Represent the low-light image and its corresponding normal-light image, P y (y|I L ) represents the distribution of normal light images conditioned on low light images, P z (z) represents the latent space distribution, represents the Jacobian value calculated by the reversible network during the transformation process, φ i and Represent the reversible transformation block and the image features processed by it, respectively, and Ft represents the conditional features input to the reversible transformation layer. The loss function optimizes the model parameters based on maximum likelihood estimation, so that the reversible network can transform the normal illumination image distribution into a standard normal distribution, and sample the standard normal distribution based on the reversibility of the network, and finally generate an enhanced image.

[0113] The zero-element guided low-light image enhancement network trained in this embodiment enhances 100 test images in the LOL-v2-real dataset, and calculates the average PSNR and SSIM indicators of the 100 enhanced images. The results are 25.16dB and 0.893, which are better than most existing low-light image enhancement methods. Figure 6 As shown, the enhanced image has appropriate light intensity, normal colors and rich details, and the serious image information loss caused by low-light environment has been significantly restored.

Claims

1. Image enhancement network based on inter-channel and intra-channel attention transformer and complete flow, characterized by: The enhanced network includes inter-channel and intra-channel attention Transformer networks and reversible flow networks. The inter-channel and intra-channel attention Transformer network includes 4 zero-element, inter-channel and intra-channel joint attention encoding networks, 4 zero-element, inter-channel and intra-channel joint attention decoding networks, downsampling modules and upsampling modules; The four zero-element, inter-channel and intra-channel joint attention encoding networks include: a local-global feature encoding module DSL1, a local-global feature encoding module DSL2, a local-global feature encoding module DSL3 and a local-global feature encoding module DSL4; The four zero-element, inter-channel and intra-channel joint attention decoding networks include: a local-global feature decoding module UPL1, a local-global feature decoding module UPL2, a local-global feature decoding module UPL3 and a local-global feature decoding module UPL4; The reversible flow network includes: a scale transformation layer and a reversible transformation block; The reversible network is divided into 1 to 3 levels, each level is composed of 1 scale transformation layer and 8 reversible transformation blocks connected in sequence, and the reversible transformation block contains 8 cascaded reversible transformation layers, namely: reversible activation layer, 1×1 convolution layer, affine injection layer, linear injection layer, conditional affine coupling layer, conditional linear coupling layer, unconditional affine coupling layer and unconditional linear coupling layer.

2. The image enhancement network based on inter-channel and intra-channel attention Transformer and complete flow according to claim 1, characterized in that The down sampling module includes a down sampling module D1, a down sampling module D2 and a down sampling module D3, and the up sampling module includes an up sampling module U1, an up sampling module U2 and an up sampling module U3.

3. The enhancement method of the image enhancement network based on inter-channel and intra-channel attention Transformer and complete flow according to claim 2, characterized in that: The following steps are included: Step 1: Data preprocessing: obtain a low-light image and obtain its zero element map; Step 2: Construct the inter-channel and intra-channel joint zero-element graph attention Transformer-U network; Step 3: Extract the triple conditional features from the inter-channel and intra-channel joint zero-element graph attention Transformer-U network constructed in step 2, and sample the standard normal distribution, input it into the reversible flow network together with the triple conditional features, and use the cross attention network of the height axis and width axis to learn the affine / linear transformation parameters of the conditional feature driving layer, guide the reversible transformation to fit the standard normal distribution, and generate an enhanced image by the reversible flow network.

4. The image enhancement method based on inter-channel and intra-channel attention Transformer and complete flow according to claim 3, characterized in that The inter-channel attention module is constructed by the following steps: Input an initial image feature X in , after the normalization layer LN1, three branches are separated, namely q it , k it and v it , and respectively through 1×1 standard convolution and convolution kernel size 3×3 depth convolution DW, q it , k it After the two branches perform transposed attention calculations, they obtain a C×C channel attention matrix, which is combined with v it After matrix multiplication, its output continues to be the input of the 1×1 standard convolution and finally combined with the initial image feature X in The output X of the inter-channel attention module is obtained by accumulation it .

5. The image enhancement method based on inter-channel and intra-channel attention Transformer and complete flow according to claim 1, characterized in that The intra-channel attention module and the zero-element attention module are constructed by the following steps: the output X of the inter-channel attention module it As the input of this module, after the normalization layer LN2, four branches are evenly divided, namely q ir , k ir 、v ir and v zr , and respectively pass through the standard convolution with a convolution kernel size of 1×1 and the depth convolution DW2 with a convolution kernel size of 3×3, where q ir , k ir 、v ir Calculate the attention matrix N within the channel separately 2 ×N 2 , v zr The zero-element attention module contains two branches, namely q zr , k zr , all come from the zero element graph features, and v zr From the image features, after calculation, we get the zero-element attention matrix N 2 ×N 2 , the two attention matrices obtained are respectively ir 、v zr After matrix multiplication, they are respectively subjected to 1×1 standard convolution and the output X of the self-attention module between channels it Add them separately to get the output X zr With X ir , and output the sum of the two to the feedforward network.

6. The image enhancement method based on inter-channel and intra-channel attention Transformer and complete flow according to claim 1, characterized in that The feedforward network consists of three 1×1 standard convolutions, two deep convolutions with a convolution kernel size of 3×3 and an activation function Gelu; the downsampling modules D1, D2, and D3 are respectively composed of convolutions with a convolution kernel size of 4×4, a stride of 2, and a padding of 1; the upsampling modules U1, U2, and U3 are respectively composed of convolutions with a convolution kernel size of 2×2, a stride of 2, and a padding of 1.

7. The image enhancement method based on inter-channel and intra-channel attention Transformer and complete flow according to claim 1, characterized in that The cross attention network based on the height axis and the width axis in step 3 includes: First, the three feature decoding modules UPL2, UPL3, and UPL4 are used to extract triple features, and then three 1×1 standard convolutions are performed to transform different channel dimensions to obtain the output conditional feature F. t1 、F t2 、F t3 , and concatenated with the stream feature X to input into the HWCAN network.

8. The image enhancement method based on inter-channel and intra-channel attention Transformer and complete flow according to claim 7, characterized in that The cross attention network based on the height axis and width axis includes: Two reversible activation layers, two 1×1 convolutional layers, and two activation functions Gelu, to get the output P im , then enter the maximum pooling attention on the height axis and the average pooling attention on the width axis; First, along its input feature P im Avgpool / Maxpool is performed on the height and width axes of each channel of size C×H×W to generate compressed features of size C×W or C×H; Then, the compressed features are learned through 1×1 standard convolution-3×3 standard convolution-1×1 standard convolution-activation function sigmoid layer to generate compressed weights of size C×W or C×H, and then expanded to size C×H×W through broadcasting, respectively, and P im Multiply the weights to get the output P mm , and then P mm Maxpool / Avgpool is performed on the height and width axes of each channel of size C×H×W to generate compressed features of size C×H or C×W, and then the compressed features are learned through 1×1 standard convolution-3×3 standard convolution-1×1 standard convolution-activation function sigmoid layer to generate compressed weights of size C×H or C×W, and then expanded to size C×H×W through broadcasting, respectively, and P mm The weights are multiplied and then added to P im Add and get the output P om ; Finally, the final scale and shift or only scaling parameters are generated through the 1×1 convolution ConvZero with learnable parameters to adapt to different affine / linear transformation layers, and finally the output of the cross attention network based on the height axis and width axis is sent to the reversible flow network.

Images