Image enhancement network method based on compressed self-attention Transform and standardized flow

Abstract

The invention provides an image enhancement network method based on a compressed self-attention Transform and a standardized flow. The image enhancement network method comprises the following steps: step 1, data preprocessing; step 2, constructing a zero element joint height and channel compression self-attention Transform-U network; 3, triple condition features are led out from the zero-element joint height and channel compression self-attention Transform-U network constructed in the step 2, standard normal distribution is sampled, the standard normal distribution and the triple condition features are input into a standardized flow network together, an attention network is used for learning affine / linear transformation parameters of a condition feature driving layer, and the zero-element joint height and channel compression self-attention Transform-U network is constructed; and reversible transformation is guided to fit standard normal distribution, and the image enhancement network based on the compressed self-attention Transform and the standardized flow is generated by the standardized flow network. According to the image enhancement network method based on the compressed self-attention Transform and the standardized flow provided by the invention, the designed low-illumination image enhancement network can effectively realize restoration of missing information of the image, and the produced enhanced image has an excellent visual effect and more excellent peak signal-to-noise ratio and structural similarity.

Description

Technical Field

[0001] This invention relates to the field of low-light image enhancement, specifically to an image enhancement network method based on compressed self-attention Transformer and normalized flow. Background Technology

[0002] Image noise refers to various factors in an image that hinder the perception of information. In low-light environments, due to insufficient light, short exposure times, or hardware limitations of the camera's CMOS sensor, images often exhibit low visibility and contrast, severe color distortion, and unpredictable image noise. These low-light images have poor aesthetic quality, contain biased and missing information, and pose significant challenges to advanced computer vision tasks such as object detection, object recognition, and scene understanding. Therefore, low-light image enhancement can improve the perceptual and aesthetic quality of images, thereby improving the overall performance of related computer vision tasks.

[0003] Current low-light image enhancement methods mostly focus on enhancing the appearance characteristics of low-light images, such as increasing image brightness and suppressing noise. Furthermore, existing image enhancement methods rarely consider the recovery of information lost in low light. Low-light images often contain a large number of zero elements; therefore, by marking these zero elements to form a binary zero-element map, the overall missing information in the image can be reflected.

[0004] In traditional flow reversible networks, reversible transform blocks are generally composed of simple reversible activation, convolutional layers, affine injection, and affine transform layers. They also use multiple convolutional layers (i.e., embedding networks) to learn affine transform parameters from conditional features. This lack of transform diversity leads to low efficiency in parameter learning and fitting normal distributions.

[0005] The patent with announcement number "CN119624830B" discloses an image enhancement network based on inter-channel and intra-channel attention Transformer and complete flow. Based on the traditional inter-channel and intra-channel self-attention mechanism, it is technically an improvement on the attention mechanism used by Restormer, which enables the reversible network to have better feature transformation capabilities. The enhanced images produced by the reversible flow network have excellent visual effects and better peak signal-to-noise ratio and structural similarity. However, it also has the problems of complex reversible network structure, which increases the floating-point computation of the overall network and lacks the ability to process multi-scale information.

[0006] Patent CN116342409B discloses a low-light image enhancement method based on transformer. It constructs a transformer network by using a sliding window-based self-attention module, a feedforward network using 1×1 convolutions, GELU activation layers, and 3×3 depth convolutions, and combines the transformer network with traditional gamma correction. This method reduces the computational complexity of the designed network, improves the ability to extract deep features, enhances the ability to extract global feature parameters, and improves the low-light image enhancement effect. However, it has some drawbacks, such as weakening the network's ability to process multi-scale features and having certain limitations when used alone to deal with images in complex scenes.

[0007] To address the aforementioned issues, this invention aims to provide an image enhancement network method based on compressed self-attention Transformer and normalized flow. This method not only enables richer information feature extraction by cascading height and channel compression self-attention networks, but also allows zero-element maps to accurately locate points in the image that verify information distortion, thereby accurately guiding the constructed attention Transformer network to effectively repair and enhance these distorted regions. Furthermore, it introduces a channel-axis, width-axis, and height-axis-based attention network (CWHF) to replace the traditional multi-convolutional layers (i.e., embedding networks) to learn the affine / linear transformation parameters of the conditional feature-driven layer, thereby guiding the normalized flow network to better learn and train. This is the problem that this invention urgently needs to solve. Summary of the Invention

[0008] The purpose of this invention is to propose an image enhancement network method based on compressed self-attention Transformer and normalized flow, so as to enhance the network's ability to repair image information loss caused by low lighting conditions, while improving the visual quality of the enhanced image.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0010] This invention provides an image enhancement network method based on compressed self-attention Transformer and normalized flow, characterized by the following steps:

[0011] Step 1: Data preprocessing;

[0012] Step 2: Construct a zero-element joint height, channel-compressed self-attention Transformer-U network;

[0013] Step 3: Extract triple conditional features from the zero-element joint height and channel compressed self-attention Transformer-U network constructed in Step 2, and sample the standard normal distribution. Input the standard normal distribution and triple conditional features into the normalized flow network, and use the attention network to learn the affine / linear transformation parameters of the conditional feature-driven layer, guide the invertible transformation to fit the standard normal distribution, and generate an image enhancement network based on compressed self-attention Transformer and normalized flow from the normalized flow network.

[0014] As a further improvement, the data preprocessing in step 1 includes: acquiring a low-light image and obtaining a zero-element map of the low-light image.

[0015] As a further improvement, the zero-element joint height and channel compression self-attention Transformer-U network in step 2 includes: a zero-element joint height and channel compression self-attention encoding network, a height and channel compression self-attention decoding network, a downsampling module, and an upsampling module.

[0016] As a further improvement, the zero-element joint height and channel compression self-attention encoding network consists of 4 networks, and the height and channel compression self-attention decoding network consists of 4 networks.

[0017] As a further improvement, the four zero-element joint height-channel compressed self-attention encoding network and the four height-channel compressed self-attention decoding network in step 2 have similar extraction structures.

[0018] As a further improvement, the standardized flow network in step 3 includes a scaling layer and a reversible transformation block, wherein the reversible transformation block includes an affine injection layer and an affine coupling layer.

[0019] As a further improvement, the attention network is an attention network with channel axis, width axis and height axis.

[0020] As a further improvement, in order to extract global semantic features and understand the global structure of features, the zero-element joint height and channel compression self-attention coding network includes: Local-Global Feature Encoding Module DSL1, Local-Global Feature Encoding Module DSL2, Local-Global Feature Encoding Module DSL3 and Local-Global Feature Encoding Module DSL4;

[0021] To perform feature reconstruction guided by the semantics of the features output by the encoding module, the high-level, channel-compressed self-attention decoding network includes: Local-Global Feature Decoding Module UPL1, Local-Global Feature Decoding Module UPL2, Local-Global Feature Decoding Module UPL3 and Local-Global Feature Decoding Module UPL4;

[0022] The height and channel compressed self-attention network is composed of two parts connected in series: a joint attention computing network and a feedforward network. The joint attention computing network is composed of four parts: a height compressed self-attention module, a channel compressed self-attention module, a zero-element height attention module, and a zero-element channel attention module. The height compressed self-attention module and the zero-element height attention module are connected in parallel, and the channel compressed self-attention module and the zero-element channel attention module are connected in parallel.

[0023] The downsampling module includes downsampling module D1, downsampling module D2 and downsampling module D3. Each downsampling halves the length and width of the input feature, while doubling the number of channels, so that each layer of encoding module can learn features at different resolutions, which is beneficial for understanding the global structure.

[0024] The upsampling module includes upsampling module U1, upsampling module U2, and upsampling module U3. Each upsampling doubles the length and width of the input feature while halving the number of channels, gradually restoring the feature to the input size. The features output from each upsampling layer are concatenated with features of the same size in each downsampling layer in a skip-connection manner along the channel dimension, ensuring that each layer of features retains both semantic and detailed information.

[0025] As a further improvement, the normalized flow is divided into 1 to 3 levels, each level consisting of a scaling layer and eight reversible transformation blocks connected in sequence. The reversible transformation blocks contain four types of cascaded reversible transformation layers: reversible activation layer, 1×1 convolutional layer, affine injection layer and affine coupling layer.

[0026] As a further improvement, the standardized flow network is divided into 1 to 3 levels. Each level consists of a scaling layer and eight reversible transformation blocks connected in sequence. The reversible transformation blocks contain eight cascaded reversible transformation layers, namely: reversible activation layer, 1×1 convolutional layer, affine injection layer and affine coupling layer.

[0027] As a further improvement, the affine / linear transformation parameters of the conditional feature-driven layer, which are learned through an attention network to guide the invertible transformation to fit a standard normal distribution, are constructed by the following steps:

[0028] First, three feature decoding modules—UPL2, UPL3, and UPL4—are used to extract triple features. These features are then transformed into output conditional features by three 1×1 standard convolutions with different channel dimensions. , , Each conditional feature is then processed by a multi-scale fusion channel attention module (MKC) and combined with the streaming features. The input is concatenated and fed into a channel-width-height-axis-based attention network (CWHF). After passing through a combination of two invertible activation layers, two 1×1 convolutional layers, and two Gelu activation functions, the output is obtained. Then, the input features are processed through channel axis fusion attention, width axis fusion attention, and height axis fusion attention. First, let the input features have dimensions C×H×W. After 1×1 convolution and 3×3 depthwise convolution, channel-axis max pooling and channel-axis average pooling are performed respectively. The two results are concatenated along the channel dimension, then passed through 1×1 convolution and sigmoid, and broadcast into features of size C×H×W. Multiplication as the output of SFA Next, let the input feature have dimensions C×H×W. After 1×1 convolution and 3×3 depthwise convolution, width-axis max pooling and width-axis average pooling are performed respectively. The two results are concatenated along the width dimension, then passed through 1×1 convolution and sigmoid, and broadcast into features of size C×H×W. Multiplication as the output of WFA Finally, let the input features have dimensions of C×H×W. After 1×1 convolution and 3×3 depthwise convolution, height-axis max pooling and height-axis average pooling are performed respectively. The two results are concatenated along the height dimension, then passed through 1×1 convolution and sigmoid, and broadcast into features of size C×H×W. Multiplication as the output of HFA , and The summation generates the final scale and shift or scaling parameters through 1×1 learnable parameter convolution ConvZero to adapt to different affine / linear transformation layers. Finally, the output of the attention network based on the channel axis, width axis, and height axis is fed into the normalized flow network, which learns the invertible transformation from the standard normal distribution to the image, using the negative log-likelihood function as the loss function.

[0029] As a further improvement, the highly compressed self-attention module and the zero-element attention module are constructed by the following steps: inputting an initial image feature After normalization layer It splits into three branches, namely , and And through 1×1 standard convolution and depthwise convolution (DW) with a kernel size of 3×3, respectively, the results were analyzed. , After performing height-axis average pooling and height-axis max pooling on the two branches and then performing transposed attention calculation, a W×W dimension height attention matrix is ​​obtained.

[0030] The zero-element high attention module contains two separate branches, namely: , Both originate from zero-element graph features. After channel-average pooling in each branch, transpose attention is calculated to obtain a W×W dimension height attention matrix. The W×W dimension height attention matrices of the highly compressed self-attention module and the zero-element height attention module are added together. After matrix multiplication, the output is used as the input for a 1×1 standard convolution, and finally combined with the initial image features. The output of the highly compressed self-attention module is obtained by accumulating the summations. ;

[0031] The channel compression self-attention module and the zero-element attention module are constructed by the following steps: The output of the highly compressed self-attention module... As input to this module, it passes through the normalization layer. It splits into three branches, namely , and And through 1×1 standard convolution and depthwise convolution (DW) with a kernel size of 3×3, respectively, the results were analyzed. , The two branches perform channel-axis average pooling and channel-axis max pooling respectively, and then perform transposed attention calculation to obtain an H×H dimension channel attention matrix;

[0032] The zero-element channel attention module contains two separate branches, namely: , Both originate from zero-element graph features. After channel average pooling in each branch, transpose attention is calculated to obtain an H×H dimension height attention matrix. The H×H dimension channel attention matrices of the channel compression self-attention module and the zero-element channel attention module are added together. After multiplying the transposed matrices, the output continues to be used as the input to a 1×1 standard convolution, and finally is combined with the output of the highly compressed self-attention module. The output of the channel compression self-attention module is obtained by accumulating the data. ;

[0033] The feedforward network is constructed by the following steps: channel compression from the output of the attention module. As input to the feedforward network, it passes through the normalization layer. The algorithm splits into two branches, each passing through a 1×1 standard convolution and a 3×3 depthwise convolution (DW). One branch passes through a multi-scale convolution module consisting of parallel 1×1, 3×3, 5×5, and 7×7 standard convolutions, a 1×1 standard convolution, and a sigmoid layer. The result is then multiplied by the other branch, which passes through a GeLU layer. The sum is used as the output of the feedforward network;

[0034] The downsampling modules D1, D2, and D3 are each composed of convolutions with a kernel size of 4×4, a stride of 2, and a padding of 1.

[0035] The upsampling modules U1, U2, and U3 are each composed of convolutions with a kernel size of 2×2, a stride of 2, and a padding of 1.

[0036] As a further improvement, the height and channel compression self-attention Transformer and normalized stream image enhancement network are characterized in that the training process for the height and channel compression self-attention Transformer and normalized stream image enhancement network is as follows:

[0037] The training image dataset used contains pairs of low-light and normal-light images. During the data input phase, the low-light images and their corresponding normal-light images used for training are randomly cropped and rotated at the same location to improve the diversity of the training data and obtain images of a size of [size missing]. Image pairs are used for network training. During model training, the zero-element map of the low-light image is first obtained, and the low-light image and its zero-element map are used as inputs to the zero-element joint height, channel compression self-attention Transformer-U network. At the same time, the normal lighting is input to the normalized flow network. Triple conditional features are extracted from the zero-element joint height, channel compression self-attention Transformer-U network, and a standard normal distribution is sampled. This standard normal distribution is then input into the normalized flow network along with the conditional features. The affine / linear transformation parameters of the conditional feature-driven layer are learned through a channel-axis, width-axis, and height-axis-based attention network (CWHF). The distribution of the normal lighting image is mapped to the latent space, and the network parameters are optimized based on maximum likelihood estimation to make the latent space distribution fit the standard normal distribution. After the network training is completed, the standard normal distribution is sampled and used together with the triple conditional features extracted from the zero-element joint height, channel compression self-attention Transformer-U network as inputs to the normalized flow network. Its reversible transformation is then performed to finally generate the enhanced image.

[0038] As a further improvement, the loss function is:

[0039]

[0040] in, This represents the negative log-likelihood loss function. and These represent the low-light image and its corresponding normal-light image, respectively. This represents the distribution of a normal illumination image conditioned on a low-light image. Indicates the potential spatial distribution. This represents the Jacobian determinant value calculated during the transformation process of the normalized flow reversible network. and These represent the reversible transform block and the image features it processes, respectively. This represents the conditional features of the input to the reversible transformation layer.

[0041] This invention, compared with patent CN119624830B, employs a novel compressed self-attention mechanism that performs pooling compression across feature channels, width, and height, significantly differing from patent CN119624830B in its technical approach. This invention greatly reduces the number of parameters and floating-point computations required by traditional self-attention transformer networks while achieving performance comparable to traditional methods. Furthermore, besides the Actnorm and 1×1 reversible convolutional layers, this invention uses only two reversible modules: affine injection and affine coupling, simplifying the structure while preserving performance. The parameter generation module CWHF designed in this invention also employs a spatial compressed attention mechanism, fundamentally different from the CNN-based parameter generation module used in patent CN119624830B. In the feedforward network, this invention designs a multi-scale convolutional module to re-extract features from the attention module's output, a significantly different approach from feedforward networks based on single-scale convolutions.

[0042] This invention, compared with patent CN116342409B, employs a compressed self-attention mechanism based on channels, width, and height. This mechanism reduces computational complexity while preserving performance, a design fundamentally different from that of patent CN116342409B. Building upon a similar simple configuration, this invention uses multi-scale convolutions to construct a feedforward network, resulting in better feature extraction from the attention mechanism. Furthermore, it employs a completely different approach to combining reversible networks with transformers. Since reversible networks are inherently generative modules with learnable parameters, combining them with a transformer network as an encoder enhances the overall model's ability to handle complex scenes and provides better low-light image enhancement.

[0043] Compared with the prior art, the beneficial effects of the present invention are:

[0044] The image enhancement network method based on compressed self-attention Transformer and normalized flow provided by this invention enables the designed low-light image enhancement network to effectively repair missing information in the image, and the produced enhanced image has excellent visual effect as well as better peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). Attached Figure Description

[0045] Figure 1 This is a schematic diagram illustrating the workflow of an image augmentation network method based on compressed self-attention Transformer and normalized flow.

[0046] Figure 2 A schematic diagram of a Transformer-U network with zero-element joint height and channel compression self-attention;

[0047] Figure 3 This is a schematic diagram of a Transformer based on a height- and channel-compression self-attention mechanism.

[0048] Figure 4 This is a channel-axis, width-axis, and height-axis based attention network (CWHF).

[0049] Figure 5 This is a schematic diagram of a standardized flow structure;

[0050] Figure 6 A schematic diagram to enhance the image. Detailed Implementation

[0051] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0052] This embodiment provides an image enhancement network method based on compressed self-attention Transformer and normalized flow, the workflow of which is as follows: Figure 1 As shown, the specific steps include:

[0053] Step 1: Data preprocessing;

[0054] The process of obtaining a low-light image and calculating its zero-element map can be represented by formula (1):

[0055] ;

[0056] in, Represents color pixels In coordinates At this location, the pixel value of the red channel R (Red) of the pixel. Represents color pixels In coordinates At this location, the pixel value of the green channel G (Green) of the pixel. Represents color pixels In coordinates At this location, the pixel value of the blue channel B (Blue) of the pixel. The height index and width index of the image are respectively represented. Formula (1) means that by judging the low light image pixel by pixel, if there is at least one pixel value of zero in the three channels R, G, and B of a certain pixel, the pixel is marked as a zero element. The final zero element image is a binary image. The pixel with a value of 1 is the zero element. The zero element image marks the zero elements in the low light image, which reflects the information missing of the image. The low light image and the zero element image are used as inputs to the enhancement network for the network to learn and recover the image information missing area. The size of the low light image input in this embodiment is (3, 400, 600), that is, the number of channels is 3, the width and height are 400 and 600 respectively. The size of the obtained zero element image is the same as that of the low light image in terms of the number of channels, width and height.

[0057] Step 2: Construct a zero-element joint height, channel-compressed self-attention Transformer-U network;

[0058] The construction of the zero-element joint height and channel compressed self-attention Transformer-U network consists of four zero-element joint height and channel compressed self-attention encoding networks, four height and channel compressed self-attention decoding networks, a downsampling module, and an upsampling module.

[0059] Step 3: Extract triple conditional features from the zero-element joint height and channel compressed self-attention Transformer-U network constructed in Step 2, sample a standard normal distribution, and input it together with the triple conditional features into a normalized flow network. Then, use a channel-axis, width-axis, and height-axis-based attention network (CWHF) to learn the affine / linear transformation parameters of the conditional feature-driven layer, guide the invertible transformation to fit the standard normal distribution, and generate the enhanced image by the normalized flow network.

[0060] The four zero-element joint height, channel compression self-attention coding network includes: Local-Global Feature Encoding Module DSL1, Local-Global Feature Encoding Module DSL2, Local-Global Feature Encoding Module DSL3, and Local-Global Feature Encoding Module DSL4;

[0061] The four height-channel compressed self-attention decoding networks include: Local-Global Feature Decoding Module UPL1, Local-Global Feature Decoding Module UPL2, Local-Global Feature Decoding Module UPL3, and Local-Global Feature Decoding Module UPL4;

[0062] The input size of the local-global feature encoding module DSL1 is (3, 256, 256), and the output is (48, 256, 256); the input size of the downsampling module D1 is (48, 256, 256), and the output is (96, 128, 128); the input size of the local-global feature encoding module DSL2 is (96, 128, 128), and the output is (96, 128, 128); the input size of the downsampling module D2 is (96, 128, 128). The input size of the Local-Global Feature Encoding Module DSL3 is (192, 64, 64), and the output is (192, 64, 64). The input size of the downsampling module D3 is (96, 128, 128), and the output is (384, 32, 32). The input size of the Local-Global Feature Encoding Module DF4 is (384, 32, 32), and the output is (384, 32, 32). The input size of the global feature decoding module UPL1 is (384, 32, 32), and the output is (384, 32, 32); the input size of the upsampling module U1 is (384, 32, 32), and the output is (192, 64, 64); the input size of the local-global feature decoding module UPL2 is (384, 64, 64), and the output is (192, 64, 64); the input size of the upsampling module U2 is (192, 64, 64), and the output is (192, 64, 64). The input dimensions of the local-global feature decoding module UPL3 are (96, 128, 128), and the output is (192, 128, 128). The input dimensions of the upsampling module U3 are (192, 128, 128), and the output is (96, 256, 256). The input dimensions of the local-global feature decoding module UPL4 are (144, 256, 256), and the output is (144, 256, 256).

[0063] The height, channel compression self-attention decoding network is as follows: Figure 3 As shown, it consists of two parts connected in series: a joint attention computing network and a feedforward network; the joint attention computing network consists of four parts: a highly compressed self-attention module, a channel compressed self-attention module, a zero-element highly attention module, and a zero-element channel attention module.

[0064] First, a highly compressed self-attention Transformer block is used. In this attention network A1, the 1×1 convolutional kernel size is 1, and the depthwise convolutions DW1, DW2, and DW3 have 3×3 kernel sizes with a stride of 1. Their outputs are multiplied to obtain different weights for different pixels. This weight matrix is ​​then processed by Softmax and normalized to the range of 0 to 1. The more severe the information distortion of a pixel, the closer its weight value is to 0, and the more intact the information of a pixel, the closer its weight value is to 1. In this attention network A1, the first input image features are... Its dimensions are C×H×W. After passing through a normalized layer LN1, it is then successively passed through 1×1 depthwise convolutions with kernel size 1 and 3×3 kernel size, with a stride of 1, resulting in three branches. , and After obtaining the output, among which The branch is then subjected to height-axis average pooling and reshaped by the built-in function to a size of W×C. The branch undergoes height-axis max pooling and is then reshaped by the built-in function with a size of C×W. and The output after reshaping is multiplied by its transpose to obtain a W×W height attention matrix; the zero-element attention module contains two separate branches, namely... , Both originate from zero-element graph features. After performing high-level average pooling on each branch, transpose attention is calculated to obtain a W×W dimensional high-level attention matrix. The two W×W dimensional attention matrices are then added together and... The branches are multiplied, then a 1×1 convolution is performed for further feature extraction. Finally, the features are added to the initial image features to obtain the output. ;

[0065] In attention network A2, the output of network A1 is... As the initial input, its size is C×H×W. It passes through a normalized layer LN2, then sequentially through 1×1 depthwise convolutions with kernel size 1 and 3×3 kernels with a stride of 1, resulting in three branches. , and After obtaining the output, among which The branch undergoes channel-axis average pooling and is then reshaped by the built-in function to a size of W×H. The branch undergoes channel-axis max pooling and is then reshaped by the built-in function with dimensions H×W. and The reshaped output is multiplied by its transpose to obtain an H×H height attention matrix; the zero-element attention module contains two separate branches, namely... , Both are derived from zero-element graph features. After channel average pooling in each branch, transpose attention is calculated to obtain an H×H channel attention matrix. The two H×H attention matrices are then added together and... The branches are multiplied, then a 1×1 convolution is performed for further feature extraction. Finally, the features are added to the initial image features to obtain the output. ;

[0066] Finally, the feedforward network N1, which connects to the joint attention network, compresses the channels from the output of the attention module. As input to the feedforward network, it passes through the normalization layer. The algorithm splits into two branches, each passing through a 1×1 standard convolution and a 3×3 depthwise convolution (DW). One branch passes through a multi-scale convolutional module (MKC) consisting of parallel 1×1, 3×3, 5×5, and 7×7 standard convolutions, a 1×1 standard convolution, and a sigmoid layer. This result is then multiplied by the other module, which passes through a GeLU layer. The sum is used as the output of the feedforward network. The feedforward network is designed to perform fully connected transformations to enhance feature extraction.

[0067] The downsampling module consists of convolutions with a kernel size of 4×4, a stride of 2, and padding of 1. It transforms a (C, H, W) feature map into a size of (2×C, H / 2, W / 2), effectively doubling the number of channels and halving the aspect ratio.

[0068] The upsampling module consists of convolutions with a kernel size of 2×2, a stride of 2, and padding of 1. It can achieve enlargement on the image size scale H×W. That is, a feature map of (C, H, W) becomes (C / 2, 2×H, 2×W) after passing through an upsampling module. It mainly achieves the effect of reducing the number of channels by half and increasing the length and width scale by half.

[0069] As shown in Figure 2, the triple features derived from the zero-element joint height and channel compression self-attention Transformer-U network are processed through several simple convolutions and linear interpolation operations to obtain Ft1, Ft2, and Ft3, with dimensions of (288, 128, 128), (288, 64, 64), and (288, 32, 32), respectively. These conditional features will guide the normalization flow to transform the standard normal distribution into an enhanced image.

[0070] The standardized flow Figure 5 As shown, the network includes a scaling layer and a reversible transformation block. The reversible transformation block comprises a reversible linear transformation block and a reversible coupled transformation block.

[0071] Specifically, the normalization flow progressively changes the size of the network's processed features through scaling layers. Therefore, the entire network is divided into 1-3 levels, each consisting of a scaling layer and eight reversible transformation blocks connected sequentially. The scaling layer reversibly adjusts the feature size. During the forward transformation, it readjusts the feature size, halving the width and height and quadrupling the number of channels. In this embodiment, since the final generated normal illumination image size is (3, 256, 256), the feature sizes processed by the normalization flow levels 1-3 are (12, 128, 128), (48, 64, 64), and (192, 32, 32), respectively. The reversible transformation block contains four cascaded reversible transformation layers: a reversible activation layer, a reversible 1×1 convolutional layer, an affine injection layer, and an affine coupling layer. The affine injection layer and affine coupling layer adjust the conditional feature F according to the size of the currently processed feature. t1 F t2 or F t3 As one input to this layer, the normalized flow achieves the conversion between the normal illumination image distribution and the standard normal distribution through these reversible transformation layers. The following is a further introduction to the four types of reversible transformation layers:

[0072] The reversible activation layer achieves affine transformation of features through custom learnable parameters. Equations (2) and (3) describe the forward and inverse transformation processes of the reversible activation layer:

[0073] (2);

[0074] (3);

[0075] in This represents element-wise multiplication. and The inputs and outputs represent the characteristics of the stream. and As learnable parameters, the features undergo a positive transformation at this layer, and the Jacobian determinant value is calculated using formula (8):

[0076] (4);

[0077] in and These represent the transformation function of formula (2) and the characteristics of its input, respectively. and These represent the width and height of the feature, respectively.

[0078] A reversible 1×1 convolutional layer transforms features using a convolutional layer with a kernel size of 1×1. Equations (5) and (6) describe the forward and inverse transformation processes of the reversible 1×1 convolutional layer:

[0079] (5);

[0080] (6);

[0081] in Given the convolution kernel matrix, the features undergo a forward transformation at this layer, and the Jacobian determinant is calculated using formula (7):

[0082] (7);

[0083] in and These represent the transformation function of formula (5) and the characteristics of its input, respectively. and These represent the width and height of the feature, respectively.

[0084] The affine injection layer takes the conditional features output by the joint attention Transformer encoding network as input and outputs transformation coefficients to achieve a reversible affine transformation of the features. Equations (12) and (13) describe the forward and inverse transformation processes of the reversible 1×1 convolutional layer:

[0085] (8);

[0086] (9);

[0087] in The representative conditional features are transformed by the CWHF parameter extraction network. and CWHF feature extraction network, such as Figure 4 As shown, three features are first extracted from the three feature decoding modules UPL2, UPL3, and UPL4, and then transformed into output conditional features by three 1×1 standard convolutions with different channel dimensions. , , Each conditional feature is processed by a multi-scale fusion channel attention module (MFS) and then combined with the streaming features. The dimensions are merged and concatenated before being input into the CWHF network. Its key feature is that it includes two reversible activation layers, two 1×1 convolutional layers, and two Gelu activation functions to obtain the output. Then, it enters the channel-axis fusion attention (SFA), width-axis fusion attention (WFA), and height-axis fusion attention (HFA). First, let the input features be of size C×H×W. After 1×1 convolution and 3×3 depthwise convolution, channel-axis max pooling and channel-axis average pooling are performed respectively. The two results are concatenated along the channel dimension, then passed through 1×1 convolution and sigmoid, and broadcast into features of size C×H×W. Multiplication as the output of SFA Next, let the input feature have dimensions C×H×W. After 1×1 convolution and 3×3 depthwise convolution, width-axis max pooling and width-axis average pooling are performed respectively. The two results are concatenated along the width dimension, then passed through 1×1 convolution and sigmoid, and broadcast into features of size C×H×W. Multiplication as the output of WFA Finally, let the input features have dimensions of C×H×W. After 1×1 convolution and 3×3 depthwise convolution, height-axis max pooling and height-axis average pooling are performed respectively. The two results are concatenated along the height dimension, then passed through 1×1 convolution and sigmoid, and broadcast into features of size C×H×W. Multiplication as the output of HFA . and The summation generates the final scale and shift (or scaling-only) parameters through a 1×1 learnable convolution ConvZero to accommodate different affine / linear transformation layers. Finally, the output of the attention network (CWHF) based on the channel axis, width axis, and height axis is fed into the normalized flow network. The features undergo a forward transformation in this layer, and the Jacobian determinant value is calculated using formula (10):

[0088] (10);

[0089] in and These represent the transformation function of formula (8) and the characteristics of its input, respectively.

[0090] Affine coupling learns the scale and displacement parameters of the affine transformation from the conditional feature Ft and half of the input stream features through a CWHF learning network. The training and inference process of affine coupling can be represented as follows:

[0091] (11);

[0092] (12);

[0093] in, Representing conditional features, the input X of the stream feature is divided into channels with the same number of channels. and , These are the corresponding outputs. and All are learnable transformation coefficients, output by the parameter extraction network CWHF. The features undergo a forward transformation at this layer, and the Jacobian determinant value is calculated using formula (13):

[0094] (13);

[0095] in and These represent the transformation function of formula (11) and the flow characteristics of its input, respectively;

[0096] For each reversible transform block in the normalized flow The input features pass sequentially through a reversible activation layer, a 1×1 convolutional layer, an affine injection layer, and an affine coupling layer, and the output features... Based on formulas (4), (7), (10), and (13), the features undergo a forward transformation by the invertible transform block, and the Jacobian determinant value is calculated using formula (14):

[0097] (14);

[0098] The training process of the low-light image enhancement network guided by zero-element features is as follows:

[0099] The training image dataset used contains low-light and normal-light image pairs. During the data import phase, the low-light images and their corresponding normal-light images used for training are randomly cropped and rotated at the same location to improve the diversity of the training data and obtain images of a size of [size missing]. Image pairs are used for network training. During model training, zero-element maps are obtained from low-light images. These zero-element maps are then used to construct cascaded attention Transformer-U networks (intra-channel and extra-channel), and triple conditional features are extracted. Simultaneously, normal lighting is input into the normalization stream. Guided by the extracted triple conditional features, the network's forward transformation is performed, mapping the distribution of the normal lighting image to the latent space. Network parameters are optimized using maximum likelihood estimation to fit the latent space distribution to a standard normal distribution. After network training, a standard normal distribution is sampled and used together with the triple conditional features extracted from the attention Transformer-U network (zero-element, height, and channel compression) as input to the normalization stream. Its reversible transformation is then performed to finally generate the enhanced image. In this embodiment, the image dataset used is LOL-v2-real, which consists of low-light and normal-light image pairs taken in real-world scenes. The image size is (3, 400, 600), of which 689 pairs are used for model training and 100 pairs are used for model testing. In actual training, the image size is randomly cropped to (3, 256, 256). The loss function is:

[0100] (15);

[0101] in, This represents the negative log-likelihood loss function. and These represent the low-light image and its corresponding normal-light image, respectively. This represents the distribution of a normal illumination image conditioned on a low-light image. Indicates the potential spatial distribution. This represents the Jacobian determinant value calculated during the transformation process of the normalized flow. and These represent the reversible transform block and the image features it processes, respectively. The loss function represents the conditional features input to the reversible transformation layer. It optimizes the model parameters based on maximum likelihood estimation, so that the normalized flow can transform the normal illumination image distribution into a standard normal distribution, and samples the standard normal distribution based on the network's reversibility to finally generate an enhanced image.

[0102] The zero-element guided low-light image enhancement network trained in this embodiment enhances 100 test images from the LOL-v2-real dataset, and the average PSNR and SSIM of the 100 enhanced images are calculated. The results are 25.14 dB and 0.878, respectively, which outperform most existing low-light image enhancement methods. An illustration of the enhanced image is shown below. Figure 6As shown, the enhanced image has appropriate lighting intensity, normal colors, and rich details, and the severe loss of image information caused by the low-light environment has been significantly restored.

[0103] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any modifications or equivalent changes made based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. An image enhancement network method based on compressed self-attention Transformer and normalized flow, characterized in that, Includes the following steps: Step 1: Data preprocessing; Step 2: Construct a zero-element joint height, channel-compressed self-attention Transformer-U network; Step 3: Extract triple conditional features from the zero-element joint height and channel compressed self-attention Transformer-U network constructed in Step 2, and sample the standard normal distribution. Input the standard normal distribution and triple conditional features into the normalized flow network, and use the attention network to learn the affine / linear transformation parameters of the conditional feature-driven layer, guide the invertible transformation to fit the standard normal distribution, and generate an image enhancement network based on compressed self-attention Transformer and normalized flow from the normalized flow network.

2. The image enhancement network method based on compressed self-attention Transformer and normalized flow according to claim 1, characterized in that, The data preprocessing in step 1 includes: acquiring a low-light image and obtaining the zero-element map of the low-light image.

3. The image enhancement network method based on compressed self-attention Transformer and normalized flow according to claim 2, characterized in that, The zero-element joint height-channel compressed self-attention Transformer-U network in step 2 includes: a zero-element joint height-channel compressed self-attention encoding network, a height-channel compressed self-attention decoding network, a downsampling module, and an upsampling module.

4. The image enhancement network method based on compressed self-attention Transformer and normalized flow according to claim 3, characterized in that, The normalized flow network in step 3 includes a scaling layer and a reversible transformation block, wherein the reversible transformation block includes an affine injection layer and an affine coupling layer.

5. The image enhancement network method based on compressed self-attention Transformer and normalized flow according to claim 4, characterized in that, The attention network is an attention network with channel axis, width axis and height axis.

6. The image enhancement network method based on compressed self-attention Transformer and normalized flow according to claim 5, characterized in that, The zero-element joint height, channel compression self-attention coding network includes: local-global feature coding module DSL1, local-global feature coding module DSL2, local-global feature coding module DSL3 and local-global feature coding module DSL4; The height-channel compressed self-attention decoding network includes: local-global feature decoding module UPL1, local-global feature decoding module UPL2, local-global feature decoding module UPL3 and local-global feature decoding module UPL4; The height and channel compressed self-attention network is composed of two parts connected in series: a joint attention computing network and a feedforward network. The joint attention computing network is composed of four parts: a height compressed self-attention module, a channel compressed self-attention module, a zero-element height attention module, and a zero-element channel attention module. The height compressed self-attention module and the zero-element height attention module are connected in parallel, and the channel compressed self-attention module and the zero-element channel attention module are connected in parallel. The downsampling module includes downsampling module D1, downsampling module D2 and downsampling module D3; The upsampling module includes upsampling module U1, upsampling module U2 and upsampling module U3.

7. The image enhancement network method based on compressed self-attention Transformer and normalized flow according to claim 4, characterized in that, The normalized flow is divided into 1 to 3 levels. Each level consists of a scaling layer and eight reversible transformation blocks connected in sequence. The reversible transformation blocks contain four types of cascaded reversible transformation layers: reversible activation layer, 1×1 convolutional layer, affine injection layer and affine coupling layer.

8. The image enhancement network method based on compressed self-attention Transformer and normalized flow according to claim 5, characterized in that, The affine / linear transformation parameters of the conditional feature-driven layer, which are learned through an attention network to guide the invertible transformation to fit a standard normal distribution, are constructed by the following steps: First, three feature decoding modules—UPL2, UPL3, and UPL4—are used to extract triple features. These features are then transformed into output conditional features by three 1×1 standard convolutions with different channel dimensions. , , Each conditional feature is then processed by a multi-scale fusion channel attention module and combined with the flow features. The input is concatenated and fed into a channel-width-height-axis-based attention network (CWHF). After passing through a combination of two invertible activation layers, two 1×1 convolutional layers, and two Gelu activation functions, the output is obtained. Then, the input features are processed through channel axis fusion attention, width axis fusion attention, and height axis fusion attention. First, let the input features have dimensions C×H×W. After 1×1 convolution and 3×3 depthwise convolution, channel-axis max pooling and channel-axis average pooling are performed respectively. The two results are concatenated along the channel dimension, then passed through 1×1 convolution and sigmoid, and broadcast into features of size C×H×W. Multiplication as the output of SFA Next, let the input feature have dimensions C×H×W. After 1×1 convolution and 3×3 depthwise convolution, width-axis max pooling and width-axis average pooling are performed respectively. The two results are concatenated along the width dimension, then passed through 1×1 convolution and sigmoid, and broadcast into features of size C×H×W. Multiplication as the output of WFA ; Finally, let the input feature have dimensions C×H×W. After 1×1 convolution and 3×3 depthwise convolution, height-axis max pooling and height-axis average pooling are performed respectively. The two results are concatenated along the height dimension, then passed through 1×1 convolution and sigmoid, and broadcast into features of size C×H×W. Multiplication as the output of HFA , and The summation generates the final scale and shift or scaling parameters through 1×1 learnable parameter convolution ConvZero to adapt to different affine / linear transformation layers. Finally, the output of the attention network based on the channel axis, width axis, and height axis is fed into the normalized flow network, which learns the invertible transformation from the standard normal distribution to the image, using the negative log-likelihood function as the loss function.

9. The image enhancement network method based on compressed self-attention Transformer and normalized flow according to claim 3, characterized in that, The highly compressed self-attention module and the zero-element attention module are constructed by the following steps: inputting an initial image feature. After normalization layer It splits into three branches, namely , and And through 1×1 standard convolution and depthwise convolution (DW) with a kernel size of 3×3, respectively, the results were analyzed. , After performing height-axis average pooling and height-axis max pooling on the two branches and then performing transposed attention calculation, a W×W dimension height attention matrix is ​​obtained. The zero-element high attention module contains two separate branches, namely: , Both originate from zero-element graph features. After high-level average pooling of the two branches, transpose attention is calculated to obtain a W×W dimension high-level attention matrix. The W×W dimension high-level attention matrices of the highly compressed self-attention module and the zero-element high-level attention module are added together. After matrix multiplication, the output is used as the input for a 1×1 standard convolution, and finally combined with the initial image features. The output of the highly compressed self-attention module is obtained by accumulating the summations. ; The channel compression self-attention module and the zero-element attention module are constructed by the following steps: The output of the highly compressed self-attention module... As input to this module, it passes through the normalization layer. It splits into three branches, namely , and And through 1×1 standard convolution and depthwise convolution (DW) with a kernel size of 3×3, respectively, the results were analyzed. , The two branches perform channel-axis average pooling and channel-axis max pooling respectively, and then perform transposed attention calculation to obtain an H×H dimension channel attention matrix; The zero-element channel attention module contains two separate branches, namely: , Both originate from zero-element graph features. After channel average pooling in each branch, transpose attention is calculated to obtain an H×H dimension height attention matrix. The H×H dimension channel attention matrices of the channel compression self-attention module and the zero-element channel attention module are added together. After multiplying the transposed matrices, the output continues to be used as the input to a 1×1 standard convolution, and finally is combined with the output of the highly compressed self-attention module. The output of the channel compression self-attention module is obtained by accumulating the data. ; The feedforward network is constructed by the following steps: channel compression from the output of the attention module. As input to the feedforward network, it passes through the normalization layer. The algorithm splits into two branches, each passing through a 1×1 standard convolution and a 3×3 depthwise convolution (DW). One branch passes through a multi-scale convolution module consisting of parallel 1×1, 3×3, 5×5, and 7×7 standard convolutions, a 1×1 standard convolution, and a sigmoid layer. This result is then multiplied by the other branch, which passes through a GeLU layer. The sum is used as the output of the feedforward network; The downsampling modules D1, D2, and D3 are each composed of convolutions with a kernel size of 4×4, a stride of 2, and a padding of 1. The upsampling modules U1, U2, and U3 are each composed of convolutions with a kernel size of 2×2, a stride of 2, and a padding of 1.

10. The image enhancement network method based on compressed self-attention Transformer and normalized flow according to claim 3, characterized in that, The loss function is: ； in, This represents the negative log-likelihood loss function. and These represent the low-light image and its corresponding normal-light image, respectively. This represents the distribution of a normal illumination image conditioned on a low-light image. Indicates the potential spatial distribution. This represents the Jacobian determinant value calculated during the transformation process of the normalized flow reversible network. and These represent the reversible transform block and the image features it processes, respectively. This represents the conditional features of the input to the reversible transformation layer.

Images