Low-illumination image restoration network containing large number of zero element pixels and restoration method

Abstract

The invention discloses a low-illumination image restoration network based on zero elements, and belongs to the field of low-illumination image restoration. Comprising the following steps of: constructing a zero element mask set through a real-world extremely-low-light night traffic monitoring image; randomly matching the low-illumination image with the mask to generate a low-illumination image containing a large number of zero-element pixels, and solving a zero-element graph of the obtained low-illumination image; inputting the synthesized low-illumination image and the zero-element image of the low-illumination image into an attention Restormer encoder based on a zero-element mask to obtain condition features; and taking the image and the standard normal distribution as the input of a reversible network, and outputting an enhanced and restored image. In an encoder network, a zero element mask attention module is designed on the basis of an original Restormer module. An unconditional affine coupling layer is inserted into the flow step in the reversible network. According to the method, learning of missing information is enhanced based on the zero-element mask graph, and the peak signal-to-noise ratio is remarkably improved while the visual effect is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of low-light image restoration, and specifically to designing a low-light image restoration network based on a large number of zero-element pixels in the low-light image. Background Art

[0002] Low-light images taken under low-light and backlit conditions often lack overall brightness, have high noise, and lose local information, which not only affects visual perception, but also fails to meet the requirements of high-quality images for tasks such as recognition and tracking.

[0003] Under extremely low light conditions in the real world, many low-light RGB (red, green, blue) images may contain a large number of zero-element pixels. A zero-element pixel is defined as a color pixel with three RGB values ​​that contains no less than one zero. There are a large number of zero elements in low-light images, and a large number of zero-element pixels will cause information loss in low-light images. In short, low-light images with a large number of zero-element pixels have problems of weak light and information loss. However, existing low-light enhancement methods aim to amplify low light and improve image brightness, and rarely consider restoring information loss caused by a large number of zero-element pixels. The low-light image restoration network containing a large number of zero-element pixels designed by the present invention generates a low-light image containing a large number of zero-element pixel maps from the zero-element pixel map extracted from the real night traffic image and the low-light image, and uses it and its corresponding zero-element mask map as the input of the restoration network, thereby improving the ability to restore information loss caused by zero-element pixels. Summary of the invention

[0004] The purpose of the present invention is to design a low-light image restoration network containing a large number of zero-element pixels to enhance the network's ability to repair image information loss caused by low-light conditions, while improving the visual quality of the enhanced image.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is: a low-light image restoration network containing a large number of zero-element pixels, the restoration network comprising:

[0006] Zero-element mask-based attention Restormer encoder and reversible network;

[0007] The zero-element mask-based attention Restormer encoder includes: a convolutional layer, a downsampling block, an upsampling block, and a refinement block;

[0008] The downsampling block consists of 4 downsampling layers, the first 3 downsampling layers consist of 4 zero-element mask attention-based Restormer sub-blocks, and the 4th downsampling layer consists of 4 Restormer sub-blocks; the zero-element mask attention-based Restormer sub-block consists of a cascade of three parts: a zero-element mask attention module, a multi-head transposition attention module, and a gated DC feedforward network connected in sequence;

[0009] The upsampling block consists of three upsampling layers, each of which is composed of four Restormer sub-blocks in cascade; the refinement block is composed of four Restormer sub-blocks in cascade; the convolution layer is composed of a depth convolution layer DC1;

[0010] The reversible network includes: a scale transformation layer and a reversible transformation block.

[0011] As an improvement of the present invention, the zero-element mask attention module includes three branches, whose inputs are q, k and v respectively, wherein q and k come from the zero-element mask features, and v comes from the image features. The branch with input q is composed of the reshaping layer R1, the branch with input k is composed of the reshaping layer R2, and the branch with input v is composed of the cascade of the normalization layer LN1, the convolution layer C1, the deep convolution layer DC2, and the reshaping layer R3. The two branches q and k perform attention calculation, and then pass through the channel broadcast layer B1 and v for attention calculation, and the output is used as the cascade input of the reshaping layer R4 and the convolution layer C2.

[0012] As an improvement of the present invention, the multi-head transposition attention module comprises three branches, whose inputs are q l , k l and v l , all come from the image features output by the zero-element mask attention module, and the input is q l The branch consists of a cascade of normalization layer LN2, convolution layer C3, deep convolution layer DC3, and reshaping layer R5, with input k l The branch consists of a cascade of normalization layer LN3, convolution layer C4, deep convolution layer DC4, and reshaping layer R6, with input v l The branch is composed of a cascade of normalization layer LN4, convolution layer C5, deep convolution layer DC4, and reshaping layer R7. The three branches perform attention calculations, and their outputs will be used as the cascade input of the reshaping layer R8 and convolution layer C6.

[0013] As an improvement of the present invention, the gated DC feedforward network includes two branches, whose inputs are F1 and F2, both of which are image features output by the multi-head transposed attention mechanism module. The branch with input F1 is divided into two branches after passing through the normalization layer LN5, namely F 11 and F 12 , input is F11 The branch consists of a convolutional layer C7, a deep convolutional layer DC6, and an activation function layer G1 in cascade, with the input being F 12 The branch consists of a convolutional layer C8 and a deep convolutional layer DC7 cascaded together. 11 and F 12 The dot product is then added to F2 after passing through the convolutional layer C9.

[0014] As an improvement of the present invention, the reversible network is divided into 1 to 3 levels, each level is composed of 1 scale transformation layer and 12 reversible transformation blocks connected in sequence, and the reversible transformation block contains 5 cascaded reversible transformation layers, namely: reversible activation layer, 1×1 convolution layer, affine injection layer, conditional radiation coupling layer and unconditional affine coupling layer.

[0015] As an improvement of the present invention, the convolution kernel size of the deep convolution layer DC1 is 3×3, the input channel is 3, the output channel is 48, the stride is 1, and the padding is 1; the normalization layers LN1~LN5 have the same parameters, and their input dimensions are all 48; the convolution layers C1 and C3~C5 have the same parameters, and their convolution kernel size is 1×1, the input channel is 48, the output channel is 144, and the stride is 1; the deep convolution layers DC2~DC5 have the same parameters, and their convolution kernel size is 3×3, the input channel is 144, the output channel is 144, the stride is 1, and the padding is 1; the reshaping layers R1~R3 ​​have the same parameters, and the input image features H×W are converted into output image features The channel broadcast layer B1 changes the input channel from 1 to the output channel from 48; the reshaping layer R4 changes the input image features from The convolution layers C2 and C6 have the same parameters, with a convolution kernel size of 1×1, 48 input channels, 48 ​​output channels, and a stride of 1. The reshaping layers R5 to R7 have the same parameters, which transform the input image features H×W×C into output image features HW×C. The reshaping layer R8 transforms the input image features from HW×C into output image features H×W×C. The convolution layers C7 to C8 have the same parameters, with a convolution kernel size of 1×1, 48 input channels, 254 output channels, and a stride of 1. The depth convolution layers DC6 to DC7 have the same parameters, with a convolution kernel size of 3×3, 254 input channels, 254 output channels, a stride of 1, and a padding of 1. The activation function layer G1 is the GELU activation function. The convolution kernel size of the convolution layer C9 is 1×1, the input channel is 127, the output channel is 48, and the stride is 1.

[0016] As an improvement of the present invention, a method for restoring a low-light image containing a large number of zero-element pixels based on the restoration network is provided, the method comprising the following steps:

[0017] Step S1, data preprocessing;

[0018] The low-light image I containing a large number of zero-element pixels is obtained by randomly matching the mask ZM in the zero-element mask set we constructed with the low-light image I. in , and find its zero element graph, their definitions are as follows:

[0019] I in =(1-ZM)*I

[0020]

[0021] Respectively represent the pixel values ​​of the red channel R (Red), green channel G (Green), and blue channel B (Blue) of the pixel point at the coordinate (i, j) of the image, and i, j represent the height and width of the image respectively;

[0022] Step S2, constructing the zero-element mask-based attention Restormer encoder and the reversible network respectively to form the overall network and complete the network training;

[0023] Step S3, compare the low light image in step S1 with its zero element Figure 1 The conditional features are extracted from the zero-element mask-based attention Restormer encoder in step S2, and the standard normal distribution is sampled, which is input together with the conditional features into the reversible network in step S2 to generate an enhanced restored image.

[0024] As an improvement of the present invention, the process of training the low-light image restoration network containing a large number of zero-element pixels includes: the training image data set used includes low-light-normal-light image pairs, and in the data import stage, the low-light images containing a large number of zero-element pixels used for training, the new zero-element mask images generated therefrom, and the corresponding normal-light images are randomly cropped and rotated at the same position to obtain image pairs of size H×W for network training.

[0025] As an improvement of the present invention, the process of training the low-light image restoration network containing a large number of zero-element pixels also includes: in the model training stage, the low-light image and its zero-element mask map are used as inputs of the attention Restormer encoder based on the zero-element mask to extract conditional features; at the same time, the normal-light image and the extracted conditional features are input into the reversible network, the distribution of the normal-light image is mapped to the latent space, and the network is trained using the minimized negative log-likelihood function.

[0026] As an improvement of the present invention, the process of training the low-light image restoration network containing a large number of zero-element pixels also includes: in the model inference stage, using a standard normal distribution and the conditional features extracted by the attention Restormer encoder based on the zero-element mask as the input of the reversible network, and performing its reversible transformation to finally generate an enhanced restored image.

[0027] As an improvement of the present invention, the minimized negative log-likelihood function is:

[0028]

[0029] Among them, Loss(I in ,I GT ) represents the negative log-likelihood loss function, I in and I GT They represent low-light images with a large number of zero-element pixels and their corresponding normal-light images, L represents the conditional feature, and F z (z) represents the latent space distribution, φ i and Represent the i-th reversible transform block and input image features respectively.

[0030] Based on the above technical solution, the beneficial effects of the present invention are:

[0031] Aiming at the problem of information loss caused by a large number of zero-element pixels in real nighttime traffic monitoring images, the present invention constructs a mask set containing a large number of zero elements, and generates a low-light image containing a large number of zero-element pixels with a low-light image. The designed low-light image restoration network containing a large number of zero-element pixels combines the generated low-light image containing a large number of zero-element pixels with its zero-element pixels. Figure 1 The conditional features are obtained by inputting the zero-element mask-based attention Restormer encoder, and a zero-element mask attention module is designed in the encoder to restore the information loss caused by zero elements; the conditional features and the standard normal distribution are used as the input of the reversible network, and the enhanced and restored image is output. An unconditional affine coupling layer is added to the reversible network to further improve the performance of the overall network. In summary, the low-light image restoration network designed by the present invention can restore the information loss caused by zero elements and enhance the image, with good visual effects and a high peak signal-to-noise ratio (PSNR). BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of the workflow of the low-light image restoration network containing a large number of zero-element pixels in the present invention;

[0033] Figure 2 Schematic diagram of the structure of a low-light image restoration network containing a large number of zero-element pixels in the present invention;

[0034] Figure 3 for Figure 2 The structural diagram of the attention Restormer sub-block based on zero-element mask is shown;

[0035] Figure 4 for Figure 2 The schematic diagram of the structure of the Restormer sub-block shown;

[0036] Figure 5 for Figure 3 The structural diagram of the zero-element mask attention module shown;

[0037] Figure 6 for Figure 3 The structural diagram of the multi-head transposed attention module shown;

[0038] Figure 7 for Figure 3 The structural diagram of the gated DC feedforward network shown;

[0039] Figure 8 for Figure 2 Schematic diagram of the reversible network structure shown;

[0040] Fig. 9 Schematic diagram of enhanced restored image. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0042] This embodiment provides a low-light image restoration network containing a large number of zero-element pixels, and its workflow is as follows: Figure 1 As shown, the specific steps include:

[0043] Step 1: Data preprocessing;

[0044] Get a low-light image I containing a large number of zero-element pixels in , and find its zero element graph. The zero element graph is expressed by formula (2):

[0045] I in =(1-ZM)*I#(1)

[0046]

[0047] Respectively represent the pixel values ​​of the red channel R (Red), green channel G (Green), and blue channel B (Blue) of the pixel point at the coordinate (i, j) of the image, and i, j represent the height and width of the image respectively;

[0048] Formula (2) indicates that a low-light image containing a large number of zero-element pixels is judged pixel by pixel. If at least one of the three channels R, G, and B corresponding to a certain pixel has a zero pixel value, the pixel is marked as a zero element, the value is marked as 1, and the values ​​of the other non-zero elements are marked as 0, and finally a zero-element map of a binary image is obtained. The zero-element map reflects the information loss of the image. The low-light image containing a large number of zero-element pixels and its zero-element map are used as the input of the enhanced restoration network to enhance the illumination and restore information loss. The size of the low-light image input in this embodiment is (3,600,400), that is, the number of channels is 3, the width and height are 600 and 400 respectively, and the size of the obtained zero-element map is 1 in terms of the number of channels, and the width and height are the same as the low-light image.

[0049] Step 2: Construct the zero-element mask-based attention Restormer encoder and the reversible network respectively to form the overall network and complete the network training;

[0050] Step 3: Compare the low-light image from step 1 with its zero elements Figure 1 The conditional features are extracted from the zero-element mask-based attention Restormer encoder in step 2, and the standard normal distribution is sampled, which is input together with the conditional features into the reversible network in step 2 to generate an enhanced restored image.

[0051] The zero-element mask-based attention Restormer encoder is as follows Figure 2 As shown, the network includes: deep convolutional layer, downsampling block, upsampling block and refinement block.

[0052] Specifically, the convolutional layer is as follows: Figure 2 As shown, it is composed of a depth convolution layer DC1, with a convolution kernel size of 3×3, 3 input channels, 48 ​​output channels, a stride of 1, and a padding of 1. In this embodiment, the feature size of the convolution layer output is (48, 160, 160), that is, the number of channels is 48, and the width and height are 160 and 160 respectively.

[0053] The downsampling block consists of 4 downsampling layers, the first 3 downsampling layers consist of 4 zero-element mask attention Restormer sub-blocks, the 4th downsampling layer consists of 4 Restormer sub-blocks, and the structure of the zero-element mask attention Restormer sub-block is as follows: Figure 3As shown, the input of this sub-block is image features and zero-element features, wherein the image features are obtained by the convolutional layer, and the zero-element mask features are derived from the zero-element map corresponding to the low-light image containing a large number of zero-element pixels obtained in step 1. In this embodiment, the zero-element map is bilinearly interpolated and downsampled to ensure consistency with the image feature size. The zero-element mask features and image features are first processed by a zero-element mask attention Restormer sub-block, which consists of three parts: a zero-element mask attention module, a multi-head transposed attention module, and a gated DC feedforward network. The zero-element mask features and image features are first processed by the zero-element mask attention module. The structure of the zero-element mask attention module is shown in Figure 5 As shown. The zero-element mask feature is copied into two features, Q and K. After the reshaping layer is processed, the output feature Q R and K R , the image feature is V, which is processed by the normalization layer LN1, the convolution layer C1, the deep convolution layer DC2 and the reshaping layer R3 in sequence, and the output feature V R , the above process can be expressed by formula (3):

[0054] Q R =R1(Q)

[0055] K R =R2(K)#(3)

[0056] V R =R3(DC2(C1(LN1(V))))

[0057] The parameters of the reshaping layers R1 to R3 are the same, converting the input image features H×W into output image features The input dimension of the normalization layer LN1 is 48, the convolution kernel size of the convolution layer C1 is 1×1, the input channel is 48, the output channel is 144, and the stride is 1. The convolution kernel size of the depth convolution layer DC2 is 3×3, the input channel is 144, the output channel is 144, the stride is 1, and the padding is 1. Next, the feature Q R , K R and V R The query vector, key vector and numerical vector are respectively used as the input in the zero-element mask attention part to perform self-attention calculation. The calculation formula can be expressed by formula (4):

[0058]

[0059] where α is a learnable scaling parameter that controls Q before applying the softmax function. R and K RThe features obtained by zero-element mask attention calculation are processed by reshaping layer R4 and convolution layer C2 respectively, and then form a residual connection with the original image feature V to obtain the output feature F out , this process can be expressed by formula (5):

[0060] F out =C2(R4(Attention(Q R , K R ,V R )))+V#(5)

[0061] The reshape layer R4 transforms the input image features from The output image feature is H×W, the convolution kernel size of the convolution layer C2 is 1×1, the input channel is 48, the output channel is 48, and the stride is 1. The present invention uses the input of the zero-element mask feature and applies the self-attention mechanism to calculate the cross covariance between local pixels and restore the loss information by learning the zero-element pixel area. The output feature F of the zero-element mask attention module out Input to the multi-head transposition attention module to enhance feature extraction. The structure of the multi-head transposition attention module is as follows Figure 6 As shown. The image features are copied as q l , k l and v l The three features are processed by the normalization layer, convolution layer, depth convolution layer and reshaping layer respectively, and the output features are Q1, K1 and V1. The above processing can be expressed by formula (6):

[0062] Q1=R5(DC3(C3(LN2(q l ))))

[0063] K=R6(DC4(C4(LN3(k l ))))#(6)

[0064] V1=R7(DC5(C5(LN4(v l ))))

[0065] The parameters of the reshape layers R5 to R7 are the same, which transform the input image features H×W×C into the output image features HW×C. The parameters of the deep convolution layers DC3 to DC5 are the same, with a convolution kernel size of 3×3, 144 input channels, 144 output channels, 1 stride, and 1 padding. The parameters of the convolution layers C3 to C5 are the same, with a convolution kernel size of 1×1, 48 input channels, 144 output channels, and 1 stride. Next, the features Q1, K1, and V1 are used as the query vector, key vector, and value vector input in the multi-head transposed attention part, respectively, for self-attention calculation. The calculation formula can be expressed by formula (7):

[0066]

[0067] Where α is a learnable scaling parameter used to control the size of the dot product of Q1 and K1 before applying the softmax function. The features obtained by the zero-element mask attention calculation are then processed by the reshape layer R8 and the convolution layer C6 respectively and then introduced into the original image feature v l Construct a residual connection to obtain the output feature Y out , this process can be expressed by formula (8):

[0068] Y out =C6(R8(Attention(Q1,K1,V1)))+v l #(8)

[0069] The reshape layer R8 transforms the input image features from HW×C to the output image features H×W×C. The convolution kernel size of the convolution layer C6 is 1×1, the input channel is 48, the output channel is 48, and the stride is 1. The output feature Y of the multi-head transposed attention module out Input to the gated DC feedforward network to further enhance feature extraction. The structure of the gated DC feedforward network is as follows: Figure 7 As shown. The image features are copied to F after the normalization layer. 11 and F 12 , F 11 After being processed by the convolution layer, the depth convolution layer and the activation function layer, F 12 After being processed by the convolutional layer and the depth convolutional layer in sequence, the two are multiplied as the input of the convolutional layer. The output of the convolutional layer is connected with the F1 residual to output the image feature F2. The calculation formula can be expressed by formula (9):

[0070] F2=C9((G1(DC6(C7(LN5(F1))))·(DC7(C8(LN5(F1)))))+F1#(9)

[0071] The input dimension of the normalization layer LN5 is 48; the parameters of the convolution layers C7~C8 are the same, the convolution kernel size is 1×1, the input channel is 48, the output channel is 254, and the stride is 1. The parameters of the deep convolution layers DC6~DC7 are the same, the convolution kernel size is 3×3, the input channel is 254, the output channel is 254, the stride is 1, and the padding is 1. The activation function layer G1 is the GELU activation function, and the convolution kernel size of the convolution layer C9 is 1×1, the input channel is 127, the output channel is 48, and the stride is 1. The final output image feature size of the zero-element mask attention Restormer sub-block is (48,160,160) for further processing by subsequent network modules. The structure of the Restormer sub-block is as follows: Figure 4 As shown in Figure 3, this sub-block consists of two parts: a multi-head transposed attention module and a gated DC feedforward network.

[0072] The upsampling block consists of three upsampling layers, each of which is composed of four Restormer sub-blocks in cascade. Figure 2 Shown

[0073] The refinement block is composed of four Restormer sub-blocks cascaded together, such as Figure 2 shown.

[0074] The reversible network structure is as follows Figure 8 As shown, the network includes: a scale transformation layer and a reversible transformation block.

[0075] Specifically, the reversible network changes the size of image features step by step through the scale transformation layer. The entire network is divided into 1 to 3 levels, each of which is composed of a scale transformation layer and 12 reversible transformation blocks connected in sequence. The scale transformation layer can reversibly adjust the size of image features. In the process of forward transformation, the image feature size is adjusted from C×H×W to In this embodiment, because the size of the finally generated normal illumination image is (3,400,600), the feature sizes processed by the reversible network levels 1 to 3 are (12,200,300), (48,100,150) and (192,50,75) respectively. The reversible transformation block contains five cascaded reversible transformation layers, namely: reversible activation layer, 1×1 convolution layer, affine injection layer, affine coupling layer and unconditional affine coupling layer. The affine injection layer and affine coupling layer will use the conditional feature L1, L2 or L3 as one input of the layer according to the size of the currently processed feature. The reversible network realizes the conversion between the normal illumination image distribution and the standard normal distribution through these reversible transformation layers. The following is a further introduction to the five reversible transformation layers:

[0076] The reversible activation layer learns the scale and bias parameters of the element-by-element affine transformation of the flow features. Formula (10) and Formula (11) describe the forward transformation and inverse transformation process of the reversible activation layer:

[0077] Y1=S1⊙X1+T1#(10)

[0078] X1=(Y1-T1) / S1 (11) where ⊙ represents the element-by-element multiplication operation, X1 and Y1 represent the processed features, S1 and T1 are the scale parameter and bias parameter respectively. The features are transformed forward at this layer, and the Jacobian value is calculated by formula (8):

[0079]

[0080] Where f1 and x1 represent the transformation function of formula (10) and its input flow feature, respectively, and h and w represent the width and height of the flow feature, respectively.

[0081] The reversible 1×1 convolutional layer realizes the transformation of flow features through a convolutional layer with a convolution kernel size of 1×1. Formula (13) and formula (14) describe the forward transformation and inverse transformation process of the reversible 1×1 convolutional layer:

[0082] Y2=WX2#(13)

[0083] X2=W -1 Y2#(14)

[0084] Where W is the convolution kernel matrix. The features are transformed forward in this layer, and the Jacobian determinant is calculated by formula (15):

[0085]

[0086] Where f2 and x2 represent the transformation function of formula (13) and its input flow feature, respectively, and h and w represent the width and height of the feature, respectively.

[0087] The affine injection layer generates scale and bias parameters based on the element-by-element affine transformation of the conditional feature stream features output by the attention Restormer encoder based on the zero-element mask. Formulas (16) and (17) describe the forward and inverse transformation processes of the affine injection layer:

[0088]

[0089] Where L represents the conditional feature, which outputs the transformation coefficients S3 and T3 through the parameter extraction network PLN. The feature undergoes the forward transformation of this layer, and the Jacobian determinant value is calculated by formula (18):

[0090]

[0091] Where f3 and x3 represent the transformation function of formula (16) and its input flow characteristics respectively.

[0092] The affine coupling layer connects the conditional features with half of the flow features and generates scale and bias parameters for the unit affine transformation of the other half of the flow features. Formula (19) and Formula (20) describe the forward transformation and inverse transformation process of the affine coupling layer:

[0093]

[0094] Wherein, L represents the conditional feature, (S4, T4) are transformation coefficients, which are output by the parameter extraction network PLN. In this embodiment, the PLN network used in the affine coupling layer is structurally the same as the PLN network used in the affine injection layer. The feature is forward transformed by this layer, and the Jacobian determinant value is calculated by formula (21):

[0095]

[0096] Where f4 and x4 represent the transformation function of formula (15) and its input flow characteristics respectively.

[0097] The unconditional affine coupling layer uses half of the flow features to generate scale and bias parameters for the affine transformation of the other half of the flow features. Formula (22) and Formula (23) describe the forward transformation and inverse transformation process of the unconditional affine coupling layer:

[0098]

[0099] Where (S5, T5) are transformation coefficients, which are output by the parameter extraction network PLN. In this embodiment, the PLN network used in the affine coupling layer is structurally the same as the PLN network used in the affine injection layer. The features are forward transformed by this layer, and the Jacobian determinant value is calculated by formula (24):

[0100]

[0101] Where f4 and x4 represent the transformation function of formula (15) and its input flow characteristics respectively.

[0102] For each reversible transformation block φ in the reversible network i (i=1,2,…,K), the input flow features are sequentially passed through the reversible activation layer, the reversible 1×1 convolution layer, the affine injection layer, the radial coupling layer and the unconditional affine coupling layer, and the output features are Based on formula (12), formula (15), formula (18), formula (21) and formula (24), the feature is transformed by the forward transformation of the reversible transformation block, and the Jacobian determinant value is calculated by formula (25):

[0103]

[0104] The training process of the low-light image restoration network containing a large number of zero-element pixels is as follows:

[0105] The training image dataset used contains low-light-normal-light image pairs. In the data import stage, the low-light image with a large number of zero-element pixels, the new zero-element mask generated by it, and the corresponding normal-light image are randomly cropped and rotated at the same position to obtain image pairs of size H×W for network training. In the model training stage, the low-light image and its zero-element mask are used as the input of the attention Restormer encoder based on the zero-element mask to extract conditional features; at the same time, the normal-light image and the extracted conditional features are input to the reversible network, the distribution of the normal-light image is mapped to the latent space, and the network is trained by minimizing the negative log-likelihood function. In the model inference stage, the standard normal distribution is used, and the conditional features extracted by the attention Restormer encoder based on the zero-element mask are used as the input of the reversible network, and their reversible transformation is performed to finally generate the enhanced restored image. In this embodiment, the image dataset used is LOL-v2-real, which is a low-light-normal-light image pair taken in a real scene, with an image size of (3,400,600), of which 689 pairs are used for model training and 100 pairs are used for model testing. In actual training, the image size is randomly cropped to (3,160,160). The loss function is:

[0106]

[0107] Among them, Loss(I in ,I GT ) represents the negative log-likelihood loss function, I in and I GT Represent the low-light image and its corresponding normal-light image, respectively. z (z) represents the latent space distribution, represents the Jacobian value calculated by the reversible network during the transformation process, φ i and They represent the i-th reversible transform block and the image features processed by it, and L represents the conditional features input to the reversible transform layer.

[0108] The zero-element guided low-light image enhancement network trained in this embodiment enhances 100 test images in the LOL-v2-real dataset and calculates the average PSNR index of the 100 enhanced images. The result is 24.15dB, which is better than most existing low-light image enhancement methods. Fig. 9 As shown, the enhanced image has appropriate light intensity, normal colors and rich details, and the serious image information loss caused by low-light environment has been significantly restored.

[0109] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A low-light image restoration network containing a large number of zero-element pixels, characterized in that: The recovery network includes: Zero-element mask-based attention Restormer encoder and reversible network; The zero-element mask-based attention Restormer encoder includes: a convolutional layer, a downsampling block, an upsampling block, and a refinement block; The downsampling block consists of 4 downsampling layers, the first 3 downsampling layers consist of 4 zero-element mask attention-based Restormer sub-blocks, and the 4th downsampling layer consists of 4 Restormer sub-blocks; the zero-element mask attention-based Restormer sub-block consists of a cascade of three parts: a zero-element mask attention module, a multi-head transposition attention module, and a gated DC feedforward network connected in sequence; The upsampling block consists of three upsampling layers, each of which is composed of four Restormer sub-blocks in cascade; the refinement block is composed of four Restormer sub-blocks in cascade; the convolution layer is composed of a depth convolution layer DC1; The reversible network includes: a scale transformation layer and a reversible transformation block.

2. The low-light image restoration network containing a large number of zero-element pixels according to claim 1, characterized in that: The zero-element mask attention module includes three branches, whose inputs are q, k and v respectively, where q and k come from the zero-element mask features, and v comes from the image features. The branch with input q is composed of the reshaping layer R1, the branch with input k is composed of the reshaping layer R2, and the branch with input v is composed of the cascade of the normalization layer LN1, the convolution layer C1, the deep convolution layer DC2, and the reshaping layer R3. The two branches q and k perform attention calculation, and then pass through the channel broadcast layer B1 and v for attention calculation, and their output is used as the cascade input of the reshaping layer R4 and the convolution layer C2.

3. The low-light image restoration network containing a large number of zero-element pixels according to claim 1, characterized in that: The multi-head transposed attention module contains three branches, whose inputs are q l , k l and v l , all come from the image features output by the zero-element mask attention module, and the input is q l The branch consists of a cascade of normalization layer LN2, convolution layer C3, deep convolution layer DC3, and reshaping layer R5, with input k l The branch consists of a cascade of normalization layer LN3, convolution layer C4, deep convolution layer DC4, and reshaping layer R6, with input v l The branch is composed of a cascade of normalization layer LN4, convolution layer C5, deep convolution layer DC4, and reshaping layer R. The three branches perform attention calculation, and their output will be used as the cascade input of the reshaping layer R8 and convolution layer C6.

4. The low-light image restoration network containing a large number of zero-element pixels according to claim 1, characterized in that: The gated DC feedforward network includes two branches, whose inputs are F1 and F2, both of which are image features output by the multi-head transposed attention mechanism module. The branch with input F1 is divided into two branches after passing through the normalization layer LN5, namely F 11 and F 12 , input is F 11 The branch consists of a convolutional layer C7, a deep convolutional layer DC6, and an activation function layer G1 cascaded, with the input being F 12 The branch consists of a cascade of convolutional layer C8 and deep convolutional layer DC7. 11 and F 12 The dot product is then added to F2 after passing through the convolutional layer C9.

5. The low-light image restoration network containing a large number of zero-element pixels according to claim 1, characterized in that: The reversible network is divided into 1 to 3 levels, each level is composed of 1 scale transformation layer and 12 reversible transformation blocks connected in sequence, and the reversible transformation block contains 5 cascaded reversible transformation layers, namely: reversible activation layer, 1×1 convolution layer, affine injection layer, conditional radiation coupling layer and unconditional affine coupling layer.

6. A method for restoring a low-light image containing a large number of zero-element pixels based on the restoration network according to any one of claims 1 to 5, characterized in that: The method comprises the following steps: Step S1, data preprocessing; Through the mask ZM in the zero-element mask set we constructed and the low-light image, we randomly match the low-light image I containing a large number of zero-element pixels. in , and find its zero element graph, their definitions are as follows: I in =(1-ZM)*I Respectively represent the pixel values ​​of the red channel R (Red), green channel G (Green), and blue channel B (Blue) of the pixel point at the coordinate (i, j) of the image, and i, j represent the height and width of the image respectively; Step S2, constructing the zero-element mask-based attention Restormer encoder and the reversible network respectively to form the overall network and complete the network training; In step S3, the low-light image in step S1 and its zero-element map are input into the zero-element mask-based attention Restormer encoder in step S2 to extract conditional features, and a standard normal distribution is sampled, which is input into the reversible network in step S2 together with the conditional features, and the reversible network generates an enhanced restored image.

7. The method for low-light image restoration in which the restoration network contains a large number of zero-element pixels according to claim 6, characterized in that: The process of training the low-light image restoration network containing a large number of zero-element pixels includes: the training image data set used includes low-light-normal-light image pairs, and in the data import stage, the low-light image containing a large number of zero-element pixels used for training, the new zero-element mask image generated therefrom, and the corresponding normal-light image are randomly cropped and rotated at the same position to obtain an image pair of size H×W for network training.

8. The method for low-light image restoration in which the restoration network contains a large number of zero-element pixels according to claim 6, characterized in that: The process of training the low-light image restoration network containing a large number of zero-element pixels also includes: in the model training stage, the low-light image and its zero-element mask map are used as inputs of the attention Restormer encoder based on the zero-element mask to extract conditional features; at the same time, the normal-light image and the extracted conditional features are input into the reversible network, the distribution of the normal-light image is mapped to the latent space, and the network is trained using the minimized negative log-likelihood function.

9. The method for low-light image restoration in which the restoration network contains a large number of zero-element pixels according to claim 6, characterized in that: The process of training the low-light image restoration network containing a large number of zero-element pixels also includes: in the model inference stage, using the standard normal distribution and the conditional features extracted by the attention Restormer encoder based on the zero-element mask as the input of the reversible network, and performing its reversible transformation to finally generate an enhanced restored image.

10. The method for low-light image restoration in which the restoration network contains a large number of zero-element pixels according to claim 8, characterized in that: The minimized negative log-likelihood function is: Among them, Loss(I in , I GT ) represents the negative log-likelihood loss function, I in and I GT They represent low-light images with a large number of zero-element pixels and their corresponding normal-light images, respectively. L represents the conditional feature, and F z (z) represents the latent space distribution, φ i and Represent the i-th reversible transform block and input image features respectively.

Images