A low-light image enhancement method based on transformer-based structure-aware two-stream network

Abstract

The present invention discloses a low-light image enhancement method based on a transformer-based structure-aware dual-stream network, comprising: the constructed dual-stream network is divided into a mainstream network and a structural stream network, wherein the mainstream network is used to restore the enhanced image and provide structural information, and the structural stream network is used to extract structural features from the mainstream and thus provide rich structural information for the enhancement process; structural information is extracted from the encoder part of the mainstream network through a structural gated transformer; and the features of the mainstream network and the structural stream network are fused using a feature fusion module based on cross-attention. This method not only solves the problem of lack of global feature capture in convolution operations, but also alleviates the loss of image block boundary information, effectively improving the network's information extraction capability, enabling the model to process low-light images in different lighting environments, and restore images with clear visual effects and obvious structures.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a low-illumination image enhancement method based on a transformer-based structure-aware dual-stream network. Background Art

[0002] Low-light image enhancement methods based on convolutional neural networks (CNNs) are superior to previous prior-based methods, making CNNs gradually become the mainstream method for low-light image enhancement. The convolution operation uses convolution kernels with the same parameters to extract image features, but the brightness distribution of many low-light images in the real world is often uneven, which means that different parts of the same image may have different lighting environments. On the other hand, the convolution operation focuses on the extraction of local features, thereby ignoring the important relationship between pixels farther away in the image, however, the structural information in the image is usually throughout.

[0003] In recent years, transformers have achieved remarkable success in many computer vision tasks thanks to their superior ability to model long-range dependencies and extract information about regions of interest. The characteristics of transformers alleviate the problems caused by convolution operations to a certain extent. However, using transformers in low-light image enhancement tasks faces two problems: on the one hand, transformers lack the ability to model local visual structures, and the lack of local visual structures may cause the information of important areas in the image to be ignored; on the other hand, transformers take non-overlapping image blocks as input and perform attention operations on these image blocks independently, which makes it easy for information loss to occur at the boundaries of image blocks.

[0004] Therefore, how to provide an image enhancement method that can restore a low-light image into an image with clear visual effects and obvious structure is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0005] In response to the above research status and existing problems, the present invention provides a low-light image enhancement method based on a transformer-based structure-aware dual-stream network, which effectively improves the information extraction capability of the network, enables the model to process low-light images in different lighting environments, and restore images with clear visual effects and obvious structures.

[0006] The present invention provides a low-light image enhancement method based on a transformer-based structure-aware dual-stream network, comprising the following steps:

[0007] S1: Construct a two-stream network, including a mainstream network and a structural stream network; the low-light image is used as the input of the mainstream network, and passes through the encoder and decoder parts of the mainstream network in sequence. The structural feature map of the low-light image is extracted through smoothed dilated residual convolution as the input of the structural stream network, and passes through the encoder and decoder parts of the structural stream network in sequence;

[0008] S2: Extract structural features from the encoder part of the mainstream network based on the structure-gated transformer to supplement the structural information of the encoder part of the structure flow network;

[0009] S3: A variable-scale window partitioning strategy is used to partition the features from the mainstream network and the decoder part of the structure flow network. The partitioned features of the mainstream network and the structure flow network are fused based on the cross-attention mechanism, and the fusion results are returned to the decoder part of the mainstream network and the structure flow network.

[0010] S4: The fused feature map output by the mainstream network and the fused feature map output by the structured flow network are added element-wise to obtain an enhanced image.

[0011] Preferably, it also includes: obtaining a data set of low-illumination-normal-illumination image pairs, uniformly cropping the data set images to a preset size, using the cropped low-illumination images to execute S1-S4, and using the cropped normal-illumination images to calculate the loss function between the enhanced images, and the loss function is used for training the transformer-based structure-aware dual-stream network.

[0012] Preferably, in S1, a structure feature map of the low-light image is extracted by a structure extraction module based on smoothed dilated residual convolution, and the structure extraction module based on smoothed dilated residual convolution comprises:

[0013] After the low-light image is input into two sets of stacked convolution modules, it enters seven stacked smoothed dilated residual convolution modules, and its output is element-wise added with the output of the fourth smoothed dilated residual convolution module to finally obtain the structural feature map of the low-light image; the convolution module includes a neural network convolution layer, a normalization layer and a Leaky ReLU activation layer connected in sequence.

[0014] Preferably, the smoothed dilated residual convolution module comprises two groups of sequentially connected dilated convolution layers, convolution layers, normalization layers and leaky ReLU layers; the output of the second group of normalization layers is element-wise added to the input of the smoothed dilated residual convolution module and then input to the input of the second group of leaky ReLU layers, and the output of the second group of leaky ReLU layers is used as the output of the smoothed dilated residual convolution module.

[0015] Preferably, the structure-gated transformer based on S2 extracts structural features from the encoder part of the mainstream network to supplement the structural information of the encoder part of the structure flow network, specifically including:

[0016] The encoder part of both the mainstream network and the structured flow network includes several layers of encoders, and the output of the previous layer of encoders serves as the input of the next layer of encoders;

[0017] Extracting the structural information of the intermediate features of each level of the mainstream network encoder, and performing element-wise addition of the structural information and the intermediate features of the mainstream network encoder at the same level as the output of the current level encoder of the mainstream network and the input of the next level encoder;

[0018] The structural information is element-wise added to the intermediate features of the structural flow network encoder at the same level, and used as the output of the encoder at the current layer of the structural flow network and the input of the encoder at the next layer.

[0019] Preferably, the structure-gated transformer based on S2 extracts structural features from the encoder part of the mainstream network, specifically including:

[0020] S21: Perform global detail repair and local structure enhancement on the mainstream features of the mainstream network encoder part, and use the multi-head self-attention mechanism to extract mainstream features for global detail repair; use convolution operation, Leaky ReLU activation function and Sigmoid function to enhance the local structure of mainstream features;

[0021] S22: The global detail restoration and local structure enhancement features extracted by S21 are fused through element-level addition;

[0022] S23: For the features fused in step S23, the normalization of the transformer encoder and the multi-layer perceptron are used to further refine and extract them.

[0023] Preferably, the local structure enhancement feature calculation process based on the convolution operation is as follows:

[0024] L ei =Sigmoid(LeakyReLU(Conv(M ei ))),i=1,2,3,

[0025] Where M ei is the intermediate feature of the i-th layer encoder of the mainstream network;

[0026] The calculation process of the global detail restoration feature based on transformer is as follows:

[0027] G ei=W-MSA(W query M ei ,W key M ei ,W value M ei ),i=1,2,3,

[0028] Where W-MSA represents the multi-head self-attention operation, W value , W key and W query Represent the query mapping matrix, key mapping matrix and value mapping matrix respectively;

[0029] Element-level addition fusion is performed using the following formula:

[0030] P ei =M ei +L ei +G ei ,i=1,2,3,

[0031] The process of further refining and extracting the fusion features using the normalization layer Norm of the transformer encoder and the multi-layer perceptron MLP is as follows:

[0032] MS ei =Norm(MLP(P ei ))+M ei ,i=1,2,3.

[0033] Preferably, the variable scale window partitioning strategy is used in S3 to partition the features from the mainstream network and the structured stream network decoder part, specifically including:

[0034] The feature maps M from the decoders of the same layer of the mainstream network and the structured flow network are di and S di , respectively through three mapping matrices and Projection is query Q mi / Q si , key K mi / K si Sum value V mi / V si ; Splice the projected features together, where the spliced ​​features from the mainstream are recorded as QKV mi , the concatenated features from the structure stream are denoted as QKV si ; i = 1, 2, 3;

[0035] The features from the mainstream and structural streams are divided according to the given window size, and different window sizes are assigned to the concatenated features of the mainstream and structural streams. The calculation process of variable scale window division is as follows:

[0036]

[0037]

[0038] win s ≠win m ,

[0039] In the formula, size indicates the window division size, win s and win m The window partition sizes representing the characteristics of the mainstream network and the structured flow network, respectively.

[0040] Preferably, in S3, the division features of the mainstream network and the structural flow network are integrated based on the cross attention mechanism, specifically including:

[0041]

[0042]

[0043] S d(i+1) =MLP(Norm(I si ))+I si ,i=1,2,3

[0044] M d(i+1) =MLP(Norm(I mi ))+I mi ,i=1,2,3

[0045] In the formula, I si and I mi is the intermediate feature extracted by multi-head self-attention operation, S d(i+1) and M d(i+1) is the output of the cross-attention based feature fusion method.

[0046] Preferably, the l1 loss, MS-SSIM loss and color loss between the enhanced image and the normal illumination image are used for training, and the brightness of the low illumination image is lower than that of the normal illumination image; the loss function is:

[0047] L=L l1 +γ1L MS-SSIM +γ2L color

[0048] Where, L l1 represents the l1 loss between the enhanced image and the normal illumination image; L MS-SSIM represents the MS-SSIM loss between the enhanced image and the normal illumination image; L colorrepresents the color loss between the enhanced image and the normal illumination image; γ1 represents the relative weight of MS-SSIM loss in the overall loss function; γ2 represents the relative weight of color loss in the overall loss function.

[0049] The low-light image enhancement method based on transformer-based structure-aware dual-stream network proposed in the present invention has the following beneficial effects compared with the prior art:

[0050] The method of the present invention obtains a paired low-illumination image enhancement data set, and uniformly crops the data set images into a preset size; the data set images adjusted to the preset size are input into a two-stream network, and the structural features of the input image are extracted as the input of the structural stream through a structure extraction module based on smoothed dilated residual convolution; the two-stream network consists of a mainstream and a structural stream, wherein the mainstream is used to restore the enhanced image and provide structural information for the structural stream, and the structural stream is used to extract the structural features of the image and provide rich structural information for the enhancement process; the structural features are extracted from the encoder part of the mainstream through a structural gated transformer, and the extracted structural features will supplement the structural information of the structural stream during the encoding process; a structure-aware feature fusion module based on cross-attention is used to divide different window sizes at different feature fusion stages, and the size of the window represents the relative scale of the image block. This multi-stage, multi-scale, and multi-type feature fusion method avoids the loss of image block boundary information; the final fused feature map is used as output to obtain an enhanced image.

[0051] This embodiment adopts a dual-stream network with mainstream and structural streams in parallel, and a structural feature extraction mode with convolution and multi-head self-attention in parallel, which can not only well preserve and enhance the structural features of the image, but also fully extract local and global information in the image. It solves the problem of lack of global feature capture in convolution operations and alleviates the loss of information on image block boundaries.

[0052] In the face of the extracted multi-type features, if a simple feature fusion operation is adopted, the expressive ability of the multi-type features for low-light images will be weakened, thereby affecting the final enhancement result. This embodiment adopts a feature fusion method based on cross-attention, which not only reasonably utilizes the multi-type features, but also improves the expressive ability of the multi-type features for low-light images.

[0053] This method not only solves the defects of CNN-based low-light image enhancement methods in modeling image features, but also makes up for the transformer's lack of ability to model local visual structure and the information loss problem of extracting image block boundaries. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work.

[0055] Figure 1 A flow chart of a low-light image enhancement method based on a transformer-based structure-aware dual-stream network provided in an embodiment of the present invention;

[0056] Figure 2 A principle block diagram of a low-light image enhancement method based on a transformer-based structure-aware dual-stream network provided in an embodiment of the present invention;

[0057] Figure 3 A schematic diagram of a structure extraction module based on smoothed dilated residual convolution provided by an embodiment of the present invention;

[0058] Figure 4 A schematic diagram of a structure-aware feature extraction module of parallel convolutional attention provided in an embodiment of the present invention;

[0059] Figure 5 A schematic diagram of a feature fusion module based on cross-attention provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] The present invention provides a low-light image enhancement method based on a transformer-based structure-aware dual-stream network, referring to Figure 1 As shown, the following steps are included:

[0062] S1: Construct a two-stream network, including a mainstream network and a structural stream network; the low-light image is used as the input of the mainstream network, and passes through the encoder and decoder parts of the mainstream network in sequence. The structural feature map of the low-light image is extracted through smoothed dilated residual convolution as the input of the structural stream network, and passes through the encoder and decoder parts of the structural stream network in sequence;

[0063] S2: Extract structural features from the encoder part of the mainstream network based on the structure-gated transformer to supplement the structural information of the encoder part of the structure flow network;

[0064] S3: A variable-scale window partitioning strategy is used to partition the features from the mainstream network and the decoder part of the structure flow network. The partitioned features of the mainstream network and the structure flow network are fused based on the cross-attention mechanism, and the fusion results are returned to the decoder part of the mainstream network and the structure flow network.

[0065] S4: The fused feature map output by the mainstream network and the fused feature map output by the structured flow network are added element-wise to obtain an enhanced image.

[0066] In one embodiment, before S1, it also includes: obtaining a data set of low-illumination-normal-illumination image pairs, uniformly cropping the data set images to a preset size, using the cropped low-illumination images to execute S1-S4, and using the cropped normal-illumination images to calculate the loss function between the enhanced image and the image.

[0067] The deep learning-based no-reference image quality assessment method provided in this embodiment can accurately determine the distortion type of an image when faced with images of various distortion types, and provide a quality assessment prediction score for the distorted image.

[0068] The specific implementation of the low-light image enhancement method S1-S4 is described in detail below:

[0069] After batch loading paired low-light-normal-light dataset images, resize the images to 256*256*3. Figure 2 and Figure 3 As shown, Figure 2 SE in the figure represents the structure extraction module based on smoothed dilated residual convolution; Figure 3 SDRB stands for smoothed dilated residual convolution; the structure extraction module based on smoothed dilated residual convolution extracts the structural feature map from the original image, and then takes the original low-light image and the structural feature map as the input of the mainstream and structural streams, respectively.

[0070] Reference Figure 3As shown in FIG. 1 , the structure extraction module SE based on smoothed dilated residual convolution is: through the stacking of two neural network convolution layers, normalization layers and leaky ReLU activation layers, the first neural network convolution layer is Conv2d(3,16,3) (the first 3 refers to the number of channels of the input convolution layer, 16 refers to the number of channels of the output convolution layer, and the second 3 refers to the size of the convolution kernel), and the second neural network convolution layer is Conv2d(16,16,3) (the first 16 refers to the number of channels of the input convolution layer, the second 16 refers to the number of channels of the output convolution layer, and 3 refers to the size of the convolution kernel). Then, by stacking seven smoothed dilated residual convolution modules, and adding their outputs to the output of the fourth smoothed dilated residual convolution module at the element level, the structural feature map of the original low-light image is finally obtained.

[0071] Reference Figure 2 As shown, given the original low-light image and its structure graph, this embodiment uses them as inputs of the mainstream and structure streams, respectively, where the mainstream is used to restore the enhanced image and provide structural information for the structure stream, and the structure stream is used to extract the structural features of the image and provide rich structural information for the enhancement process. Both the mainstream and structure streams use the U-Net network as the backbone network and are divided into an encoding part and a decoding part. In the encoding part, the SGT module extracts structural information from the intermediate features of the mainstream output to supplement the structure stream; in the decoding part, the SCA module fuses the mainstream and structure stream features and returns the fusion results to the mainstream and structure streams.

[0072] For the structural feature extraction of the encoding part, considering that the feature distribution of low-light images, including brightness, structure, and contrast, is often uneven, using only convolution operations may ignore the details of local extremely dark areas in low-light images. The advantages of transformers in modeling long-term dependencies can alleviate the problems of convolution operations. In this way, the details of local extremely dark areas in low-light images can be compensated by the long-range perception capabilities introduced by the transformer's MSA operation, while the structure of relatively bright areas in the image can be fully utilized by the local enhancement strategy introduced by the convolution operation (SGT module).

[0073] Reference Figure 2 and Figure 4 As shown, Figure 2 SGT stands for the structure-aware feature extraction module of the parallel convolutional MSA; SGT extracts the mainstream intermediate features M e1 ,M e2 ,M e3 Structural information of MS e1 ,MS e2 ,MS e3 , and then the structural information is respectively combined with the corresponding level mainstream intermediate features M e1 ,M e2 ,Me3 and the structure flow intermediate feature S e1 ,S e2 ,S e3 Perform element-wise addition as the output of the current layer and the input of the next layer.

[0074] It is understandable that the structural information MS e1 Compared with the mainstream network intermediate encoding feature M e1 Element-level addition is used as the output of the first layer encoder of the mainstream network and the input of the next layer encoder; structural information MS e1 With the structure flow network intermediate encoding feature S e1 The element-wise addition is used as the output of the first encoder layer of the structured flow network and the input of the next encoder layer.

[0075] SGT for structure information MS e2 ,MS e3 The accumulation process is the same as the above structure information MS e1 The only difference is that the structural information MS e2 ,MS e3 Perform element-wise addition on the intermediate features of the second and third encoder layers respectively and return the corresponding level encoder.

[0076] Reference Figure 4 As shown in the figure, SGT is: global detail restoration and local structure enhancement are performed on the mainstream intermediate features respectively; for global detail restoration, the MSA mechanism is used to extract mainstream features, and for local structure enhancement, convolution operation, LeakyReLU activation function and Sigmoid function are used; global detail restoration features and local structure enhancement features are fused through element-level addition; the features are further refined and extracted through the normalization layer (Norm) of the transformer encoder and the multi-layer perceptron (MLP).

[0077] The local structure enhancement feature calculation process based on convolution operation is as follows:

[0078] L ei =Sigmoid(LeakyReLU(Conv(M ei )))(i=1,2,3)

[0079] The calculation process of the global detail restoration feature based on transformer is as follows:

[0080] G ei =W-MSA(W query M ei ,W key M ei ,W value Mei )(i=1,2,3)

[0081] In the above formula, W-MSA represents the multi-head self-attention operation, W value , W key and W query Represent the query mapping matrix, key mapping matrix and value mapping matrix respectively.

[0082] The convolution operation and MLP operation are parallelized by the following formula:

[0083] P ei =M ei +L ei +G ei (i=1,2,3)

[0084] The normalization layer (Norm) and multi-layer perceptron (MLP) of transformer encoder are used to further refine and extract the fusion features through the following formula:

[0085] MS ei =Norm(MLP(P ei ))+M ei (i=1,2,3)

[0086] For the feature fusion of the decoding part, in order to more reasonably utilize multi-type features and more effectively express the image content, a feature fusion method based on the cross attention mechanism (i.e. SCA) is adopted.

[0087] Reference Figure 2 and Figure 5 As shown, Figure 2 SCA in the middle represents the feature fusion module based on cross attention; SCA fuses the mainstream intermediate features M at the same level d1 ,M d2 ,M d3 and the structure flow intermediate feature S d1 ,S d2 ,S d3 , and then the fused features are element-wise added to the intermediate decoding features of the mainstream or structural stream as the output of the current decoding layer and the input of the next decoding layer.

[0088] It can be understood that the intermediate decoding feature M of the first layer decoder of the mainstream network d1 The intermediate decoding feature S of the first layer decoder of the structured flow network d1 Fusion is performed to combine the fusion features with the intermediate decoding features M of the first layer decoder of the mainstream network d1 Perform element-level addition as the output of the first layer decoder of the mainstream network and the input of the next layer decoder; combine the fusion feature with the intermediate decoding feature S of the first layer decoder of the structure flow networkd1 Perform element-wise addition as the output of the first layer decoder of the structured flow network and the input of the next layer decoder.

[0089] SCA for the intermediate decoding features M of the mainstream network d2 ,M d3 And the intermediate decoding feature S of the structure flow network d2 ,S d3 The processing process is the same as above d1 and S d1 The only difference is that the mainstream network intermediate decoding feature M d2 ,M d3 And the intermediate decoding feature S of the structure flow network d2 ,S d3 They correspond to the input and output of the second layer decoder and the third layer decoder respectively.

[0090] Reference Figure 5 As shown, SCA is: the feature map M from the same layer of the mainstream and structural streams di / S di (i=1,2,3), through three mapping matrices and Projection is query Q mi / Q si (i=1,2,3), key K mi / K si (i=1,2,3) and value V mi / V si (i=1,2,3); The above feature maps are spliced ​​together, where the spliced ​​features from the mainstream are recorded as QKV mi (i=1,2,3), the concatenated features from the structure stream are denoted as QKV si (i=1,2,3); for the spliced ​​feature map, the features from the mainstream and structural streams are divided according to the given window size, and different window sizes are assigned to the spliced ​​features of the mainstream and structural streams; the above-mentioned variable scale window division strategy is adopted to fuse the features of the mainstream and structural streams based on the cross attention mechanism.

[0091] The calculation process of variable scale window division is as follows:

[0092]

[0093]

[0094] win s ≠win m

[0095] In the above formula, size represents the window division size, win sand win m The window sizes representing the mainstream and structural flow characteristics respectively.

[0096] The feature fusion calculation process based on cross attention is as follows:

[0097]

[0098]

[0099] S d(i+1) =MLP(Norm(I si ))+I si (i=1,2,3)

[0100] M d(i+1) =MLP(Norm(I mi ))+I mi (i=1,2,3)

[0101] Among them, I si and I mi is the intermediate feature extracted by multi-head self-attention operation, S d(i+1) and M d(i+1) is the output of the cross-attention based feature fusion method.

[0102] After three feature fusions in the SCA module, the mainstream feature map M of the two-stream network is d4 And the structural flow feature map S d4 The final enhanced image can be obtained by element-wise addition.

[0103] This embodiment uses l1 loss, MS-SSIM loss and color loss between the model output and the normal illumination image for training. The overall loss function can be expressed as:

[0104] L=L l1 +γ1L MS-SSIM +γ2L color

[0105] In the above formula, L l1 represents the l1 loss between the model output and the normal illumination image; L MS-SSIM represents the MS-SSIM loss between the model output and the normal illumination image; L color represents the color loss between the model output and the normal illumination image; γ1 represents the relative weight of MS-SSIM loss in the overall loss function; γ2 represents the relative weight of color loss in the overall loss function.

[0106] L is calculated by the following formula l1 :

[0107]

[0108] In the above formula, N represents the number of images in a batch; It represents the image enhanced by this method for the i-th low-light image in the current batch, and I(i) represents the normal-light image corresponding to the i-th image in the current batch.

[0109] L is calculated by the following formula MS-SSIM :

[0110]

[0111]

[0112]

[0113] in, and μ (I) are the means of the brightened image and the normal illumination image, respectively. represents the covariance, and σ 2 (I) are the standard deviations of the brightened image and the normal illumination image, respectively, and M represents the scale number.

[0114] L is calculated by the following formula color :

[0115]

[0116] Where gauss(·) represents Gaussian blur and S represents the scale level.

[0117] The above is a detailed introduction to the low-illumination image enhancement method based on the transformer-based structure-aware dual-stream network provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0118] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

Claims

1. A low-light image enhancement method based on a transformer-based structure-aware two-stream network, characterized by: The steps include: S1: Construct a two-stream network, including a mainstream network and a structural stream network; the low-light image is used as the input of the mainstream network, and passes through the encoder and decoder parts of the mainstream network in sequence. The structural feature map of the low-light image is extracted through smoothed dilated residual convolution as the input of the structural stream network, and passes through the encoder and decoder parts of the structural stream network in sequence; S2: Extract structural features from the encoder part of the mainstream network based on the structure-gated transformer to supplement the structural information of the encoder part of the structure flow network; S3: Use the variable scale window partitioning strategy to partition the features from the mainstream network and the structure flow network decoder part, fuse the partitioned features of the mainstream network and the structure flow network based on the cross attention mechanism, and return the fusion results to the decoder part of the mainstream network and the structure flow network; specifically include: The feature maps M from the decoders of the same layer of the mainstream network and the structured flow network are di and S di , respectively through three mapping matrices and Projection is query Q mi / Q si , key K mi / K si Sum value V mi / V si ; Splice the projected features together, where the spliced ​​features from the mainstream are recorded as QKV mi , the concatenated features from the structure stream are denoted as QKV si ; i = 1, 2, 3; The features from the mainstream and structural streams are divided according to the given window size, and different window sizes are assigned to the spliced ​​features of the mainstream and structural streams. The calculation process of variable scale window division is as follows: win s ≠win m , In the formula, size indicates the window division size, win s and win m The window partition sizes representing the characteristics of the mainstream network and the structural flow network respectively; S d(i+1) =MLP(Norm(I si ))+I si ,i=1,2,3 M d(i+1) =MLP(Norm(I mi ))+I mi ,i=1,2,3, In the formula, I si and I mi is the intermediate feature extracted by multi-head self-attention operation, S d(i+1) and M d(i+1) is the output of the feature fusion method based on cross attention; S4: The fused feature map output by the mainstream network and the fused feature map output by the structured flow network are added element-wise to obtain an enhanced image.

2. According to claim 1, a low-light image enhancement method based on transformer-based structure-aware dual-stream network is characterized in that: Also includes: Obtain a data set of low-illumination-normal-illumination image pairs, uniformly crop the data set images to a preset size, use the cropped low-illumination images to execute S1-S4, and use the cropped normal-illumination images to calculate the loss function between the enhanced images, and the loss function is used for training the transformer-based structure-aware two-stream network.

3. According to claim 1, a low-light image enhancement method based on transformer-based structure-aware dual-stream network is characterized in that: In S1, a structure feature map of the low-light image is extracted by a structure extraction module based on smooth dilated residual convolution, and the structure extraction module based on smooth dilated residual convolution includes: After the low-light image is input into two sets of stacked convolution modules, it enters seven stacked smoothed dilated residual convolution modules, and its output is element-wise added with the output of the fourth smoothed dilated residual convolution module to finally obtain the structural feature map of the low-light image; the convolution module includes a neural network convolution layer, a normalization layer and a Leaky ReLU activation layer connected in sequence.

4. According to claim 3, a low-light image enhancement method based on transformer-based structure-aware dual-stream network is characterized in that: The smoothed dilated residual convolution module includes two groups of sequentially connected dilated convolution layers, convolution layers, normalization layers and leaky ReLU layers; the output of the second group of normalization layers is element-wise added to the input of the smoothed dilated residual convolution module and then input to the input of the second group of leaky ReLU layers, and the output of the second group of leaky ReLU layers is used as the output of the smoothed dilated residual convolution module.

5. According to claim 1, a low-light image enhancement method based on transformer-based structure-aware dual-stream network is characterized in that: The structure-gated transformer based on S2 extracts structural features from the encoder part of the mainstream network to supplement the structural information of the encoder part of the structure flow network, specifically including: The encoder part of both the mainstream network and the structured flow network includes several layers of encoders, and the output of the previous layer of encoders serves as the input of the next layer of encoders; Extracting the structural information of the intermediate features of each level of the mainstream network encoder, and performing element-wise addition of the structural information and the intermediate features of the mainstream network encoder at the same level as the output of the current level encoder of the mainstream network and the input of the next level encoder; The structural information is element-wise added to the intermediate features of the structural flow network encoder at the same level, and used as the output of the encoder at the current layer of the structural flow network and the input of the encoder at the next layer.

6. According to claim 1, a low-light image enhancement method based on transformer-based structure-aware dual-stream network is characterized in that: The structure-gated transformer based S2 extracts structural features from the encoder part of the mainstream network, including: S21: Perform global detail repair and local structure enhancement on the mainstream features of the mainstream network encoder part, and use the multi-head self-attention mechanism to extract mainstream features for global detail repair; use convolution operation, Leaky ReLU activation function and Sigmoid function to enhance the local structure of mainstream features; S22: The global detail restoration and local structure enhancement features extracted by S21 are fused through element-level addition; S23: For the features fused in step S22, the normalization of transformer encoder and multi-layer perceptron are used to further refine and extract.

7. The low-light image enhancement method based on transformer-based structure-aware dual-stream network according to claim 6, characterized in that: The local structure enhancement feature calculation process based on convolution operation is as follows: L ei =Sigmoid(LeakyReLU(Conv(M ei ))),i=1,2,3, Where M ei is the intermediate feature of the i-th layer encoder of the mainstream network; The calculation process of the global detail restoration feature based on transformer is as follows: G ei =W-MSA(W query M ei ,W key M ei ,W value M ei ),i=1,2,3, Where W-MSA represents the multi-head self-attention operation, W value , W key and W query Represent the value mapping matrix, key mapping matrix and query mapping matrix respectively; Element-level addition fusion is performed using the following formula: P ei =M ei +L ei +G ei ,i=1,2,3, The process of further refining and extracting the fusion features using the normalization layer Norm of the transformer encoder and the multi-layer perceptron MLP is as follows: MS ei =Norm(MLP(P ei ))+M ei ,i=1,2,3。 8. The low-light image enhancement method based on transformer-based structure-aware dual-stream network according to claim 1, characterized in that: The l1 loss, MS-SSIM loss and color loss between the enhanced image and the normal illumination image are used for training. The brightness of the low illumination image is lower than that of the normal illumination image. The loss function is: L=L l1 +γ1L MS-SSIM +γ2L color , Where, L l1 represents the l1 loss between the enhanced image and the normal illumination image; L MS-SSIM represents the MS-SSIM loss between the enhanced image and the normal illumination image; L color represents the color loss between the enhanced image and the normal illumination image; γ1 represents the relative weight of MS-SSIM loss in the overall loss function; γ2 represents the relative weight of color loss in the overall loss function.

Images