Image enhancement method based on masked self-attention feature separation and multi-scale fusion

Abstract

The present invention provides an image enhancement method based on mask self-attention feature separation and multi-scale fusion, comprising: obtaining a first image to be processed; processing the first image using a trained image enhancement network to obtain a high-quality second image, wherein the expression of the image enhancement network is as follows: HQ =H RC (H SF (X LQ )+H DF (H SF (X LQ ))), where X LQ is the first image, X HQ is the second image, H SF is the shallow feature extraction unit, H DF is a deep feature enhancement unit, H RC This is a high-quality image reconstruction unit. By building an image enhancement network based on masked self-attention feature separation and dual-window multi-scale feature fusion, the features of the input image are separated into correlation features and differential features using the masked self-attention mechanism. These correlation features and differential features are then processed and fused using the dual-window multi-scale feature. Combined with shallow features, this method generates a high-quality enhanced image. This method is low in complexity and effectively enhances image quality.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image enhancement method based on mask self-attention feature separation and multi-scale fusion. Background Art

[0002] In modern society, images are widely used in numerous fields, such as digital photography, environmental monitoring, medical imaging, and satellite remote sensing. However, during image acquisition, transmission, and storage, images are often affected by various factors, resulting in image quality degradation, such as blurring, loss of detail, and noise. Single image enhancement technology (SIET) can generate high-quality (HQ) images from low-quality (LQ) images. This technology can be used to address the aforementioned issues, improving the visual quality and usability of images. Therefore, SIET has important research implications for various fields. In recent years, deep learning methods have made significant progress in image enhancement. For example, some methods based on convolutional neural networks (CNNs) enhance images by learning feature representations. However, these methods have limitations. Traditional CNNs are not effective at handling long-range dependencies, and extracting multi-scale features often requires a large number of parameters, resulting in increased computational effort and model complexity. In addition, while some methods improve image quality, they also introduce large computational costs and memory usage, limiting their application in resource-constrained environments.

[0003] In response to the above problems, how to design a lightweight, efficient and effective method to enhance image quality has become the focus of current research. Summary of the Invention

[0004] In view of the above-mentioned defects of the prior art, the present invention provides an image enhancement method based on mask self-attention feature separation and multi-scale fusion to solve the technical problems of high model complexity and large computational complexity.

[0005] To achieve the above-mentioned and other related purposes, the present invention provides an image enhancement method based on mask self-attention feature separation and multi-scale fusion, comprising: obtaining a first image to be processed; processing the first image using a trained image enhancement network to obtain a high-quality second image, wherein the expression of the image enhancement network is as follows: HQ =H RC (H SF (X LQ )+H DF (H SF(X LQ ))), where X LQ is the first image, X HQ is the second image, H SF is the shallow feature extraction unit, H DF is a deep feature enhancement unit, H RC It is a high-quality image reconstruction unit.

[0006] In one embodiment of the present invention, the deep feature enhancement unit includes a mask self-attention feature separation unit and a dual-window multi-scale fusion unit; the mask self-attention feature separation unit separates the shallow features extracted by the shallow feature extraction unit based on the mask self-attention mechanism to obtain associated features and differential features, wherein the associated features are used to characterize the global dependency relationship between different regions in the image, and the differential features are used to characterize local detail information; the dual-window multi-scale fusion unit is used to process the associated features and the differential features to obtain deep features.

[0007] In one embodiment of the present invention, the masked self-attention feature separation unit separates the shallow features according to the following steps: performing a normalization operation on the shallow features to obtain a first feature; using three linear layers to process the first feature respectively to obtain a query matrix, a key matrix and a value matrix; performing a depthwise separable convolution operation on the query matrix and the key matrix and then performing a reshaping operation to obtain a first matrix and a second matrix respectively; performing a permutation operation on the second matrix to obtain a third matrix; performing a matrix multiplication operation on the first matrix and the third matrix and then performing a normalization operation to obtain a fourth matrix; using a trainable binarization layer to process the fourth matrix to obtain a binary matrix; performing a matrix multiplication operation on the value matrix and the binary matrix to obtain the associated feature; subtracting the associated feature from the first feature to obtain the differential feature.

[0008] In one embodiment of the present invention, the dual-window multi-scale fusion unit includes a dual-window self-attention module and a multi-scale convolution module; the dual-window self-attention module processes the associated features to enhance the modeling capability of long-distance dependencies in the image; the multi-scale convolution module enhances the differential features to extract local detail information of different scales; the outputs of the dual-window self-attention module and the multi-scale convolution module are fused to obtain the deep features.

[0009] In one embodiment of the present invention, the expression of the dual-window self-attention module is as follows: X1 = H RWLAB (LN(X lin ))+X lin ;X2=MLP(LN(X1))+X1;X3=H TWLAB(LN(X2))+X2; X4=MLP(LN(X3))+X3; where X lin is the associated feature, X4 is the output of the dual-window self-attention module, LN is normalization, MLP is a multi-layer perceptron, H RWLAB is a rectangular window self-attention block, H TWLAB is the triangular window self-attention block.

[0010] In one embodiment of the present invention, the multi-scale convolution module is formed by connecting convolution layers of multiple scales in parallel.

[0011] In one embodiment of the present invention, the expression of the multi-scale convolution module is as follows: X5 = Conv 3×3 (X dif )+Conv 5×5 (X dif )+Conv 7×7 (X dif )+Conv 9×9 (X dif ), where Conv i×i Represents a convolution operation with a convolution kernel size of i×i.

[0012] In one embodiment of the present invention, the loss function is calculated according to the following formula during the training of the image enhancement network: Where, is the real image corresponding to the input data, is the prediction result of the input data after being enhanced by the image enhancement network. The superscript i is used to indicate the correspondence between the prediction result and the real image, and N is the total number of samples in each batch.

[0013] Beneficial effects of the present invention: The present invention proposes an image enhancement method based on masked self-attention feature separation and multi-scale fusion. This method builds an image enhancement network based on masked self-attention feature separation and dual-window multi-scale feature fusion, separates the features of the input image into associated features and differential features based on the masked self-attention mechanism, and then processes and fuses the associated features and differential features through dual-window multi-scale features, and combines shallow features to generate high-quality enhanced images. This method has low complexity and can effectively enhance image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0015] Figure 1 A flowchart of an image enhancement method provided by one embodiment of the present invention;

[0016] Figure 2 This is an architecture diagram of an image enhancement network provided by one embodiment of the present invention;

[0017] Figure 3 This is an architecture diagram of a masked self-attention feature separation unit provided by one embodiment of the present invention;

[0018] Figure 4 This is an architectural diagram of a dual-window multi-scale fusion unit provided in one embodiment of the present invention;

[0019] Figure 5 This is an architectural diagram of a dual-window self-attention module provided by one embodiment of the present invention;

[0020] Figure 6 This is an architectural diagram of a multi-scale convolution module provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following describes the embodiments of the present invention through specific embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. It should be noted that the following embodiments and the features in the embodiments can be combined with each other unless they conflict. In addition to the specific methods, equipment, and materials used in the embodiments, based on the understanding of the prior art by those skilled in the art and the description of the present invention, any methods, equipment, and materials of the prior art that are similar or equivalent to the methods, equipment, and materials described in the embodiments of the present invention can also be used to implement the present invention.

[0022] It should be understood that the terms used in the examples of the present invention are for describing specific embodiments rather than for limiting the scope of protection of the present invention. Unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those generally understood by those skilled in the art.

[0023] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In some of the embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the embodiments of the present invention.

[0024] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functions and operations that may be implemented by the methods and computer program products of various embodiments disclosed in the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0025] See Figure 1 , Figure 1 An image enhancement method based on mask self-attention feature separation and multi-scale fusion provided by an embodiment of the present invention includes the following two steps: obtaining a first image to be processed; and processing the first image using a trained image enhancement network to obtain a high-quality second image. Figure 2 As shown, its expression is as follows: X HQ =H RC (H SF (X LQ )+H DF (H SF (X LQ ))), where X LQ is the first image, i.e. the image to be processed, X HQ is the second image, i.e. the enhanced image output by the network model, H SF is the shallow feature extraction unit, H DF is a deep feature enhancement unit, H RC It is a high-quality image reconstruction unit.

[0026] The shallow feature extraction unit here is equivalent to the preprocessing before the deep feature enhancement unit. In a specific embodiment of the present invention, it can, for example, use a single 3×3 convolution to extract shallow features. H in the expression SF (X LQ ) is X s The deep feature enhancement unit is the focus of this invention, which is based on the mask self-attention feature separation and multi-scale fusion method to enhance the shallow feature X s Enhance to obtain deep features H in the expressionDF (H SF (X LQ )) That is X d Finally, the high-quality image reconstruction unit will reconstruct the shallow features X s And deep features X d Perform fusion and output the second image X HQ .

[0027] See Figure 2 In a specific embodiment of the present invention, the deep feature enhancement unit includes a masked self-attention feature separation unit and a dual-window multi-scale fusion unit. The masked self-attention feature separation unit separates the shallow features extracted by the shallow feature extraction unit based on the masked self-attention mechanism to obtain correlation features and differential features. The correlation features are used to represent the global dependencies between different regions in the image, and the differential features are used to represent local detail information. The dual-window multi-scale fusion unit is used to process the correlation features and differential features to obtain deep features.

[0028] The masked self-attention feature separation unit is used to decompose shallow features to obtain correlation features and differential features, and then the dual-window multi-scale fusion unit is used to process these two features separately, and then fuse them after processing to obtain deep features.

[0029] See Figure 3 In a specific embodiment of the present invention, the architecture diagram of the mask self-attention feature separation unit is as follows Figure 3 As shown in Figure 2, this unit, as a key component of the image enhancement network, plays an important role in feature extraction and separation operations, achieving efficient computation while improving the network's ability to represent image features. The masked self-attention feature separation unit separates shallow features in the following steps.

[0030] (1) For shallow features X s Perform normalization operation to obtain the first feature X s '. The normalization operation here corresponds to Figure 3 The Norm in it is called Normalization. Before feature extraction, the input features are normalized in order to stabilize the training process and accelerate convergence.

[0031] (2) Use three linear layers to process the first feature respectively to obtain the query matrix, key matrix and value matrix. The linear layer is Figure 3 Linear in, specifically, will use three linear layers L Q , L K , L V For the first feature X s'Processing is performed to obtain the query matrix Q, key matrix K and value matrix V, where the number of channels of the value matrix V is the same as the shallow feature X s Stay consistent.

[0032] (3) Perform a depthwise separable convolution operation on the query matrix and the key matrix and then perform a reshaping operation to obtain the first matrix and the second matrix respectively. The depthwise separable convolution operation is Figure 3 In DWconv, the reshaping operation is Figure 3 After these two operations, the number of channels of the query matrix Q and the key matrix K will be compressed to C / r 2 ,get and Among them, Q' is the first matrix, K' is the second matrix, and C is the shallow feature X s The number of channels, r is a hyperparameter used to control the compression ratio of the number of channels, N is the batch size (that is, the number of samples input to the model at one time), S 2 Indicates the spatial resolution of the feature map (i.e., height × width).

[0033] (4) Perform a permutation operation on the second matrix to obtain the third matrix. The permutation operation is Figure 3 The P (Permutation) in the above formula is to permutate the second matrix K' to obtain the third matrix K", in order to adapt it to the subsequent self-attention operation.

[0034] (5) After performing matrix multiplication on the first matrix and the third matrix, normalization operation is performed to obtain the fourth matrix. Figure 3 ×(Matrix Multiplication), the normalization operation is Figure 3 The σ in is normalized using the Softmax function, which is different from the previous Norm. Norm scales the data to a specific range (usually [0, 1] or [-1, 1]), or standardizes the distribution of the data (mean is 0, variance is 1); Softmax converts a set of values ​​into a probability distribution so that their sum is 1.

[0035] (6) The fourth matrix is ​​processed using a trainable binarization layer to obtain a binary matrix. The trainable binarization layer is Figure 3 The TBL (Trainable Binarization Layer) in the model is used, and the resulting binary matrix can be denoted as A. The binarization process is trainable, and the model can optimize the parameters of the binarization layer through backpropagation to better adapt to different input feature distributions. By learning appropriate thresholds, the values ​​corresponding to the associated features are set to 1, and the values ​​corresponding to the differential features are set to 0, thus obtaining the binary matrix A.

[0036] (7) Perform matrix multiplication on the value matrix and the binary matrix to obtain the associated features. As mentioned above, the binary matrix A obtained by the trainable binarization layer has the associated features corresponding to 1 and the differential features corresponding to 0. Therefore, after multiplying the value matrix V with the binary matrix A, these associated features can be retained. The associated features are X lin (Right now Figure 3 linked features in ), which is equal to V×A.

[0037] (8) Subtract the associated feature from the first feature to obtain the differential feature. By making an identity mapping at the beginning of the unit, the output port is connected to the associated feature X. lin Subtracting them can get the differential features, which are X dif (Right now Figure 3 differencefeatures in X s '-X lin .

[0038] Through the above steps (1) to (8), the input shallow feature X S Divide and obtain the associated feature X lin and differential features X dif .

[0039] See Figure 4 In a specific embodiment of the present invention, the dual-window multi-scale fusion unit includes a dual-window self-attention module and a multi-scale convolution module, wherein the dual-window self-attention module processes the correlation features to enhance the modeling capability of long-distance dependencies in the image; the multi-scale convolution module enhances the differential features to extract local detail information of different scales; finally, the outputs of the dual-window self-attention module and the multi-scale convolution module are fused to obtain deep features.

[0040] See Figure 5 In a specific embodiment of the present invention, the expression of the dual-window self-attention module is as follows:

[0041] X1=H RWLAB (LN(X lin ))+X lin ;

[0042] X2=MLP(LN(X1))+X1;

[0043] X3=H TWLAB (LN(X2))+X2;

[0044] X4=MLP(LN(X3))+X3;

[0045] Among them, X lin is the correlation feature, X1~X3 are the intermediate features, the above four expressions can be written into a general formula, in which these intermediate features can be omitted, and X4 is the output of the dual-window self-attention module. In the above formula, LN is normalized, corresponding to Figure 5 Norm in; MLP is multi-layer perceptron; H RWLAB is a rectangular window self-attention block, corresponding to Figure 5 RWLAB in H TWLAB is the triangular window self-attention block, corresponding to Figure 5 TWLAB in [1]. Because rectangular window local self-attention technology is susceptible to distortion at boundaries, a rectangular window self-attention block and a triangular window self-attention block are added to the dual-window self-attention module to address this issue. The two blocks work synchronously in tandem, first dividing the image into rectangular windows and then into triangular windows. This connection method is most effective in enhancing image quality. The features of adjacent windows are then overlapped and cross-fused, and cross-attention is established between these windows. This design can reduce boundary distortion, capture long-range, multi-scale features, and more effectively enhance various image features.

[0046] The rectangular window self-attention block and the triangular window self-attention block are two local window-based self-attention mechanism variants, primarily used to improve the computational efficiency and performance of the Transformer model. They reduce computational complexity by limiting the attention range while retaining the ability to capture local information.

[0047] See Figure 4 In a specific embodiment of the present invention, the multi-scale convolution module is composed of multiple scale convolution layers in parallel. Its function is to mask the differential feature X output by the self-attention feature separation unit. dif Further feature extraction is performed to achieve better image enhancement effects.

[0048] See Figure 5 In a specific embodiment of the present invention, the expression of the multi-scale convolution module is as follows: X5 = Conv 3×3 (X dif )+Conv 5×5 (X dif )+Conv 7×7 (X dif )+Conv 9×9 (X dif ), where Conv i×i Represents a convolution operation with a convolution kernel size of i×i.

[0049] Finally, the deep feature X extracted by the deep feature enhancement unit d=H DF (X S )=X4+X5. Then the deep feature X d With shallow feature X s After addition, a high-quality second image can be obtained through processing by a high-quality image reconstruction unit.

[0050] In a specific embodiment of the present invention, after obtaining a high-quality result, we use the L1 loss function to calculate the measure of the high-quality result X HQ With the real data X RHQ The error between them is calculated and the network parameters are optimized using the back propagation method. This completes the training of the image enhancement network based on mask self-attention feature separation and dual-window multi-scale fusion. Specifically, the task of training the network using training samples includes the following steps: LQ Substitute into the image enhancement network and get image X HQ ; According to image X HQ , image X in the LQ-HQ image pair RHQ And the following loss function calculation formula is used to calculate the loss loss:

[0051]

[0052] Where, is the real image corresponding to the input data, is the prediction result after the input data is enhanced by the image enhancement network. The superscript i is used to indicate the correspondence between the prediction result and the real image, and N is the total number of samples in each batch. The above steps can easily help us train the parameters of the image enhancement network. After training, the lightweight image enhancement network with dual-window self-attention and feature separation can be directly used to train the first input image X. LQ Perform feature quality enhancement to obtain the corresponding second image X HQ .

[0053] In order to further illustrate the effect of the method of the present invention, the effect of the present invention is further illustrated below in conjunction with experiments.

[0054] 1. Experimental Conditions: The computer hardware environment used in this experiment was an Intel Core i9-10980XE CPU and a GTX 3090 GPU. The software environment was the CentOS 7.6 operating system, the compilation environment was PyCharm, and the deep learning framework was PyTorch. All subsequent training and testing were based on this platform. Table 1 shows a quantitative analysis of the proposed method on the DIV2K and DF2K (DIV2K + Flickr2K) training datasets using scaling factors of ×2, ×3, and ×4. This comprehensive performance test is then compared with other streaming graphics processing models.

[0055] 2. Image enhancement evaluation index: In order to evaluate the enhancement performance of the image feature enhancement method of the present invention, the present invention uses two evaluation indicators, Peak Signal to Noise Ratio (PSNR) and Structural Similarity (SSIM), to evaluate the enhancement results.

[0056] PSNR is an objective evaluation index for evaluating images, and its expression is as follows:

[0057]

[0058] Among them, MAX refers to the maximum value of the color of the image point, and MSE represents X HQ The corresponding X RHQ However, the PSNR value is not consistent with the visual quality of the naked eye. To overcome this shortcoming, we also use SSIM to evaluate the feature enhancement results. Its expression is as follows:

[0059] SSIM(x,y)=[l(x,y)] α [c(x,y)] β [s(x,y)] γ ,

[0060]

[0061] Among them, l(x,y) represents the image brightness comparison, c(x,y) represents the image contrast comparison, s(x,y) represents the image structure comparison, μ represents the mean, σ represents the standard deviation, σ xy represents the covariance, c1, c2, and c3 are constants used to maintain stability and are calculated by the following formula: c1 = (k1L) 2 ,c2=(k2L) 2 Where L is the maximum value of the pixel value in the image, k1 = 0.01, k2 = 0.03. In practical applications, we often use the simplified SSIM formula:

[0062]

[0063] SSIM divides distortion modeling into a combination of three different factors: brightness, contrast, and structure, which can better reflect image quality than PSNR.

[0064] 3. Experimental content and result analysis: The method of the present invention is compared with the most common advanced methods. Here we select CNN-based EDSR, SAN and HAN, and Transformer-based IPT, SwinIR, Swin2SR, ACT, ART, EDT, HAT and other advanced architectures to quantitatively compare the models of the present invention in terms of PSNR and SSIM. HAT is considered to be the best model before, and its results are better than any other SOTA model. However, due to the distortion-free and rich feature exploration capabilities of the image enhancement network in the present invention, it can produce better performance than HAT on all five benchmark datasets and for all scaling factors. These excellent results confirm the importance of using triangular window attention in the present invention. The performance indicators of all compared models are from their respective papers, and the comparison results are shown in Tables 1 to 3. Among them, the best results are indicated in bold and the second best are underlined.

[0065] Table 1: Comparison of various methods on the ×2 scale

[0066]

[0067] Table 2: Comparison of various methods on the ×3 scale

[0068]

[0069]

[0070] Table 3: Comparison of various methods on the ×4 scale

[0071]

[0072] From the quantitative comparison results in Tables 1 to 3, it can be seen that the method proposed in the present invention uses a more flexible method to process information of different features in the image, and achieves the best image enhancement results in all cases.

[0073] To demonstrate that our proposed model strikes a better balance between effectiveness and efficiency, we also qualitatively compare the task performance of various methods when reasoning on a dataset with a ×4 image scale. The results are shown in Table 4.

[0074] Table 4: Quantitative trade-off between model effectiveness and efficiency

[0075]

[0076]

[0077] Here, we selected two currently performing Transformer-based methods and compared their PSNR, SSIM, params (parameter count), FLOPs (Floating Point Operations per second), memory (maximum video memory usage), and latency (inference time) when inferring Set14 at a scale of ×4 images. The results show that our method has a significant advantage in inference time compared to other methods.

[0078] In summary, the present invention proposes an image enhancement method based on masked self-attention feature separation and dual-window multi-scale fusion. A masked self-attention feature separation unit and a dual-window multi-scale fusion unit are constructed. The two units form a deep feature enhancement unit in the image enhancement network. It not only solves the problems of large graphics card memory usage and long calculation time, but also can achieve higher quality feature enhancement tasks.

[0079] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. An image enhancement method based on mask self-attention feature separation and multi-scale fusion, characterized in that: include: Acquire a first image to be processed; The first image is processed using a trained image enhancement network to obtain a high-quality second image, wherein the expression of the image enhancement network is as follows: X HQ =H RC (H SF (X LQ )+H DF (H SF (X LQ ))), Where, X LQ is the first image, X HQ is the second image, H SF is the shallow feature extraction unit, H DF is a deep feature enhancement unit, H RC A high-quality image reconstruction unit; The deep feature enhancement unit includes a mask self-attention feature separation unit and a dual-window multi-scale fusion unit; The masked self-attention feature separation unit separates the shallow features extracted by the shallow feature extraction unit based on the masked self-attention mechanism to obtain correlation features and differential features, wherein the correlation features are used to characterize the global dependency relationship between different regions in the image, and the differential features are used to characterize local detail information; The dual-window multi-scale fusion unit is used to process the correlation features and the differential features to obtain deep features; The masked self-attention feature separation unit separates the shallow features according to the following steps: Normalizing the shallow features to obtain a first feature; Process the first feature using three linear layers to obtain a query matrix, a key matrix, and a value matrix; Performing a depthwise separable convolution operation on the query matrix and the key matrix and then performing a reshaping operation to obtain a first matrix and a second matrix respectively; performing a permutation operation on the second matrix to obtain a third matrix; Performing matrix multiplication on the first matrix and the third matrix, and then performing a normalization operation to obtain a fourth matrix; Processing the fourth matrix using a trainable binarization layer to obtain a binary matrix; Performing matrix multiplication calculation on the value matrix and the binary matrix to obtain the correlation feature; The associated feature is subtracted from the first feature to obtain the differential feature.

2. The image enhancement method based on mask self-attention feature separation and multi-scale fusion according to claim 1, characterized in that: The dual-window multi-scale fusion unit includes a dual-window self-attention module and a multi-scale convolution module; The dual-window self-attention module processes the associated features to enhance the modeling capability of long-range dependencies in the image; The multi-scale convolution module enhances the differential features to extract local detail information of different scales; The outputs of the dual-window self-attention module and the multi-scale convolution module are fused to obtain the deep features.

3. The image enhancement method based on mask self-attention feature separation and multi-scale fusion according to claim 2, characterized in that: The expression of the dual-window self-attention module is as follows: X1=H RWLAB (LN(X lin ))+X lin ; X2=MLP(LN(X1))+X1; X3=H TWLAB (LN(X2))+X2; X4=MLP(LN(X3))+X3; Among them, X lin is the associated feature, X4 is the output of the dual-window self-attention module, LN is normalization, MLP is a multi-layer perceptron, H RWLAB is a rectangular window self-attention block, H TWLAB is the triangular window self-attention block.

4. The image enhancement method based on mask self-attention feature separation and multi-scale fusion according to claim 2, characterized in that: The multi-scale convolution module is composed of convolution layers of multiple scales connected in parallel.

5. The image enhancement method based on mask self-attention feature separation and multi-scale fusion according to claim 4, characterized in that: The expression of the multi-scale convolution module is as follows: X5=Conv 3×3 (X dif )+Conv 5×5 (X dif )+Conv 7×7 (X dif )+Conv 9×9 (X dif ), Where, Conv i×i Represents a convolution operation with a convolution kernel size of i×i.

6. The image enhancement method based on mask self-attention feature separation and multi-scale fusion according to claim 1, characterized in that: The loss function is calculated according to the following formula when the image enhancement network is trained: Where, is the real image corresponding to the input data, is the prediction result of the input data after being enhanced by the image enhancement network. The superscript i is used to indicate the correspondence between the prediction result and the real image, and N is the total number of samples in each batch.

Images