Visual enhancement method based on wavelet transform N-net super-resolution

Abstract

The invention discloses a visual enhancement method based on wavelet transform N-net super-resolution. The visual enhancement method comprises the following steps: S1, carrying out feature extraction to obtain a wavelet transform image; s2, splicing the attention feature graph, the transformation graph and the detail graph, transmitting the spliced images to a next wavelet convolution block, repeating the steps to obtain a coding graph, and transmitting the coding graph to a bottleneck layer; s3, the bottleneck layer carries out feature fusion processing on the coding graph to obtain an up-sampling graph; s4, performing inverse wavelet transform on the reconstructed image, repeating to obtain a high-resolution reconstructed image, and transmitting the high-resolution reconstructed image to an image generation convolution layer; s5, generating a high-resolution image Y from the high-resolution reconstructed image through an image generation convolution layer; by adopting the method, training is performed through the deep learning technology, and the low-resolution image can be output as the high-resolution image, so that the color and details better conform to the visual effect of people.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a visual enhancement method based on wavelet transform N-net super-resolution. Background Art

[0002] Super-resolution SR is an image processing technology that uses algorithms to upgrade low-resolution images to high resolution. It aims to overcome or compensate for the problems of blurred images, poor quality, and unclear areas of interest caused by the limitations of the image acquisition system or the acquisition environment itself, thereby improving the details and clarity of the image. This technology is widely used in many fields, such as medical imaging, satellite remote sensing, video surveillance, and multimedia entertainment. Although image super-resolution methods based on deep learning have made significant progress compared to traditional methods in recent years, this technology still faces a series of challenges.

[0003] In the field of intelligent robots, image clarity is crucial for accurate perception and understanding of the environment, especially when processing images collected by low-quality cameras. By applying the super-resolution method combining wavelet transform with N-net, the details and clarity of the image can be significantly improved. However, most traditional super-resolution models are based on convolutional layers to extract deep features. Since the convolution result is a local feature of a fixed scale, it is generally necessary to gradually increase the receptive field by stacking multiple convolutional layers, resulting in a large amount of computational operations. In addition, the convolutional layer mainly operates in the spatial domain, and it is difficult to directly use frequency domain information, but the high-frequency information in the frequency domain is very important for super-resolution reconstruction. Although there are also technical methods for super-resolution processing from the frequency domain, such as converting image information to the frequency domain for processing, such as wavelet transform, and then converting it back to the spatial domain to achieve super-resolution, such methods may produce boundary effects at the edge of the image, resulting in edge distortion of the reconstructed image. The reason is that the convolution operation in the wavelet transform process involves the local area of ​​the signal. When a convolution operation is performed at the boundary, part of the filter kernel will exceed the boundary range of the signal, resulting in the loss or inaccuracy of the edge information of the signal. This is particularly evident in 2D image processing, especially at the four corners of the image. Summary of the invention

[0004] In view of this, the object of the present invention is to provide a visual enhancement method based on wavelet transform N-net super-resolution.

[0005] The technical solution adopted by the present invention to solve its technical problem is to provide a visual enhancement method based on wavelet transform N-net super-resolution, including a main module and an information supplement module ISM, the main module includes an encoder, a decoder and a bottleneck layer connecting the encoder and the decoder, the encoder includes p wavelet convolution blocks connected in sequence, the decoder includes p upsampling convolution layers connected in sequence, the information supplement module ISM includes an attention calculation module SCM and p attention convolution layers, the input end of each attention convolution layer is connected to the attention calculation module SCM, and the output end of each attention convolution layer corresponds to the output end of a wavelet convolution block;

[0006] The visual enhancement method includes the following steps:

[0007] S1, input a low-resolution image LR, extract features of the low-resolution image LR through a preliminary convolution layer to obtain a wavelet transform image, and transmit it to the wavelet convolution block at the head end;

[0008] The low-resolution image LR is preprocessed by the information supplement module ISM to obtain a preprocessed image, and then transmitted to the attention calculation module SCM;

[0009] S2, the wavelet convolution block performs wavelet transform on the input wavelet transform image to obtain a transform image and a detail image, and jump-connects the detail image to the input end of the corresponding upsampling convolution layer;

[0010] The attention calculation module SCM performs spatial attention SA and / or channel cross attention CCA calculation on the preprocessed image to obtain an attention feature map, and the attention convolution layer performs feature processing on the attention feature map to obtain an attention feature map with the same size and dimension as the transformation map;

[0011] The attention feature map, the transformation map, and the detail map are concatenated and transmitted to the next wavelet convolution block, and step S2 is repeated for p-1 times to obtain the encoding map and transmit it to the bottleneck layer;

[0012] S3, the bottleneck layer performs feature fusion processing on the coded image to obtain an upsampled image and transmits it to the upsampled convolution layer;

[0013] S4, the upsampling convolution layer splices the input upsampling image and the detail image to obtain a reconstructed image, performs an inverse wavelet transform on the reconstructed image and transmits it to the next upsampling convolution layer, repeats step S4, and repeats it p-1 times to obtain a high-resolution reconstructed image and transmits it to the image generation convolution layer;

[0014] S5. Generate a high-resolution image Y from the high-resolution reconstructed image through an image generation convolution layer.

[0015] As a further improvement of the present invention, the specific steps of preprocessing to obtain a preprocessed image are: performing cropping operations on the upper left corner, upper right corner, lower left corner, and lower right corner of the low-resolution image LR respectively to obtain cropping feature maps corresponding to the four corners, and splicing the four cropping feature maps into a preprocessed image according to channels.

[0016] As a further improvement of the present invention, the wavelet transform adopts two-dimensional discrete wavelet transform 2D-DWT, and the inverse wavelet transform adopts two-dimensional discrete inverse wavelet transform 2D-IDWT.

[0017] As a further improvement of the present invention, the specific steps of two-dimensional discrete wavelet transform 2D-DWT include:

[0018] Perform two one-dimensional discrete wavelet transforms, one in the row direction and the other in the column direction;

[0019] In the row direction, let the original image be f[i,j], perform a one-dimensional discrete wavelet transform on the row, and get the intermediate result a row [i,j] and d row [i,j]:

[0020]

[0021] where a row [i,j] represents the low-frequency part in the row direction, d row [i,j] represents the high-frequency part in the row direction, i and j represent the row and column indexes of the image respectively, h[k] represents the filter coefficient of the low-pass filter; g[k] represents the filter coefficient of the high-pass filter, where h[k] and g[k] are expressed as:

[0022]

[0023] A one-dimensional discrete wavelet transform is also performed in the column direction to obtain LL, LH, HL and HH:

[0024]

[0025] Where LL is the low frequency subband, LH is the horizontal high frequency subband, HL is the vertical high frequency subband, and HH is the diagonal high frequency subband;

[0026] In step S2, the transformation map is LL, and the detail maps include LH, HL and HH.

[0027] As a further improvement of the present invention, the bottleneck layer includes a dimensionality reduction convolution layer, a nonlinear activation function ReLU and a dimensionality increase convolution layer. In step S3, the bottleneck layer performs feature fusion processing on the coded image to obtain the calculation process of the up-sampled image as follows:

[0028] X bottleneck =Wexpand *ReLU(W reduce *X+b reduce )+b expand

[0029] Where X is the encoding graph, W reduce is the weight of the dimension reduction convolutional layer, b reduce is the bias of the dimension reduction convolutional layer, W expand is the weight of the up-dimensional convolutional layer, b expand is the bias of the dimension-raising convolutional layer, ReLU is the activation function, and X bottleneck It is an upsampled image.

[0030] As a further improvement of the present invention, in step S2, the attention calculation module SCM performs spatial attention SA and channel cross attention CCA calculations on the preprocessed image.

[0031] As a further improvement of the present invention, the calculation steps of the spatial attention SA include: applying maximum pooling and average pooling to the preprocessed image, performing convolution operations on the pooled feature map after channel splicing, and finally calculating the attention score map F by the sigmoid function. conv , by preprocessing the image and the attention score map F conv Multiply to get the attention feature map F SA .

[0032] As a further improvement of the present invention, the calculation step of the channel cross attention CCA includes: SA The maximum pooling and average pooling are used for the preprocessed images. The fully connected layer FC operation is performed on the feature maps after maximum pooling and average pooling. The fully connected layer FC reduces the dimension, activates ReLU and increases the dimension of the feature maps after maximum pooling and average pooling through the convolution layer, generates two feature tensors, adds them and inputs them into the weights module to calculate the cross attention.

[0033] As a further improvement of the present invention, the image generation convolution layer is a 3*3 convolution layer.

[0034] The beneficial effects of the present invention are at least:

[0035] 1. The traditional downsampling layer is replaced with a wavelet transform layer. After extracting high-frequency domain information at multiple scales in the encoder, the image is reconstructed through inverse wavelet transform in the decoder stage, and jump connections are performed in this process. The feature fusion layer in the upsampling stage fuses the high-frequency information of each encoder layer with the reconstructed information.

[0036] 2. In order to reduce the information loss caused by the boundary effect in wavelet transform, an information supplement module ISM is proposed. This module extracts deep edge information and injects it into the decoder stage through a series of alternating structures of spatial attention and channel cross attention, making up for the information loss in the wavelet transform layer, thereby enhancing edge details and abstract features and achieving better reconstruction results.

[0037] 3. Through training with deep learning technology, low-resolution images can be output as high-resolution images, making the colors and details more in line with people's visual effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is the overall framework diagram of the present invention;

[0039] Figure 2 is the wavelet transform step diagram;

[0040] Figure 3 is the inverse wavelet transform step diagram;

[0041] Figure 4 Visualization of wavelet transform;

[0042] Figure 5 This is the framework diagram of the information supplement module ISM. DETAILED DESCRIPTION

[0043] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings.

[0044] Reference Figure 1-5 The present invention proposes a visual enhancement method based on wavelet transform N-net super-resolution, including a main module and an information supplement module ISM, the main module includes an encoder, a decoder and a bottleneck layer connecting the encoder and the decoder, the encoder includes p wavelet convolution blocks connected in sequence, the decoder includes p upsampling convolution layers connected in sequence, the information supplement module ISM includes an attention calculation module SCM and p attention convolution layers, the input end of each attention convolution layer is connected to the attention calculation module SCM, and the output end of each attention convolution layer corresponds to the output end of a wavelet convolution block;

[0045] The visual enhancement method includes the following steps:

[0046] S1, input a low-resolution image LR, extract features of the low-resolution image LR through a preliminary convolution layer to obtain a wavelet transform image, and transmit it to the wavelet convolution block at the head end;

[0047] The low-resolution image LR is preprocessed by the information supplement module ISM to obtain a preprocessed image, and then transmitted to the attention calculation module SCM;

[0048] S2, the wavelet convolution block performs wavelet transform on the input wavelet transform image to obtain a transform image and a detail image, and jump-connects the detail image to the input end of the corresponding upsampling convolution layer;

[0049] The attention calculation module SCM performs spatial attention SA and / or channel cross attention CCA calculation on the preprocessed image to obtain an attention feature map, and the attention convolution layer performs feature processing on the attention feature map to obtain an attention feature map with the same size and dimension as the transformation map;

[0050] The attention feature map, the transformation map, and the detail map are concatenated and transmitted to the next wavelet convolution block, and step S2 is repeated for p-1 times to obtain the encoding map and transmit it to the bottleneck layer;

[0051] S3, the bottleneck layer performs feature fusion processing on the coded image to obtain an upsampled image and transmits it to the upsampled convolution layer;

[0052] S4, the upsampling convolution layer splices the input upsampling image and the detail image to obtain a reconstructed image, performs an inverse wavelet transform on the reconstructed image and transmits it to the next upsampling convolution layer, repeats step S4, and repeats it p-1 times to obtain a high-resolution reconstructed image and transmits it to the image generation convolution layer;

[0053] S5. Generate a high-resolution image Y from the high-resolution reconstructed image through an image generation convolution layer.

[0054] Frequency domain transformation, i.e. wavelet transform, is a global representation of the entire signal, not just a local area. Therefore, any operation on frequency domain data will affect the entire image. In super-resolution tasks, frequency domain processing is often used to recover lost high-frequency information. If the boundary is not handled properly, the error in the frequency domain will be amplified, resulting in obvious distortion at the edges and corners of the reconstructed image. In the subsequent frequency domain conversion back to the spatial domain, i.e. inverse wavelet transform, it is also a global operation. If the frequency domain data is inaccurate or distorted at the boundary, these errors will be amplified during the inverse transform process and propagate to the entire reconstructed image.

[0055] If wavelet transform is not used and spatial domain convolution is used directly, even if there is some information loss at the boundary, the overall impact is relatively small, because the original intention of convolutional neural networks is to capture local patterns, so they are robust to boundary effects. However, if the resolution of the image is to be improved, it is inevitable to use frequency transform, because in the super-resolution task, the goal is to restore high-frequency details in the frequency domain, such as textures, lines and other fine structures, which are crucial for visual perception because low-resolution images often lose these details. Traditional interpolation methods can only provide smooth estimates and cannot reconstruct the original high-frequency components well.

[0056] The jump connection can directly provide spatial position information from the encoder to the decoder, which is very important for reconstructing high-resolution output, especially for segmentation or restoration tasks. If you only rely on the feature map processed by the bottleneck layer, some details may be lost; even if the feature fusion is performed at the bottleneck layer, it is difficult to ensure that all useful information can be perfectly transmitted. As a supplementary mechanism, the jump connection can help ensure that key features are not lost during the transmission process.

[0057] Since the technical solutions are closely related to each other, the subsequent technical solutions and beneficial effects are fully explained in the final implementation process.

[0058] As a further improvement of the present invention, the specific steps of preprocessing to obtain a preprocessed image are: performing cropping operations on the upper left corner, upper right corner, lower left corner, and lower right corner of the low-resolution image LR respectively to obtain cropping feature maps corresponding to the four corners, and splicing the four cropping feature maps into a preprocessed image according to channels.

[0059] As a further improvement of the present invention, the wavelet transform adopts two-dimensional discrete wavelet transform 2D-DWT, and the inverse wavelet transform adopts two-dimensional discrete inverse wavelet transform 2D-IDWT.

[0060] As a further improvement of the present invention, the specific steps of two-dimensional discrete wavelet transform 2D-DWT include:

[0061] Perform two one-dimensional discrete wavelet transforms, one in the row direction and the other in the column direction;

[0062] In the row direction, let the original image be f[i,j], perform a one-dimensional discrete wavelet transform on the row, and get the intermediate result a row [i,j] and d row [i,j]:

[0063]

[0064] where a row [i,j] represents the low-frequency part in the row direction, d row [i,j] represents the high-frequency part in the row direction, i and j represent the row and column indexes of the image respectively, h[k] represents the filter coefficient of the low-pass filter; g[k] represents the filter coefficient of the high-pass filter, where h[k] and g[k] are expressed as:

[0065]

[0066] A one-dimensional discrete wavelet transform is also performed in the column direction to obtain LL, LH, HL and HH:

[0067]

[0068] Where LL is the low frequency subband, LH is the horizontal high frequency subband, HL is the vertical high frequency subband, and HH is the diagonal high frequency subband;

[0069] In step S2, the transformation map is LL, and the detail maps include LH, HL and HH.

[0070] As a further improvement of the present invention, the bottleneck layer includes a dimensionality reduction convolution layer, a nonlinear activation function ReLU and a dimensionality increase convolution layer. In step S3, the bottleneck layer performs feature fusion processing on the coded image to obtain the calculation process of the up-sampled image as follows:

[0071] X bottleneck =W expand *ReLU(W reduce *X+b reduce )+b expand

[0072] Where X is the encoding graph, W reduce is the weight of the dimension reduction convolutional layer, b reduce is the bias of the dimension reduction convolutional layer, W expand is the weight of the up-dimensional convolutional layer, b expand is the bias of the dimension-raising convolutional layer, ReLU is the activation function, and X bottleneck It is an upsampled image.

[0073] As a further improvement of the present invention, in step S2, the attention calculation module SCM performs spatial attention SA and channel cross attention CCA calculations on the preprocessed image.

[0074] As a further improvement of the present invention, the calculation steps of the spatial attention SA include: applying maximum pooling and average pooling to the preprocessed image, performing convolution operations on the pooled feature map after channel splicing, and finally calculating the attention score map F by the sigmoid function. conv , by preprocessing the image and the attention score map F conv Multiply to get the attention feature map F SA .

[0075] As a further improvement of the present invention, the calculation step of the channel cross attention CCA includes: SA The maximum pooling and average pooling are used for the preprocessed images. The fully connected layer FC operation is performed on the feature maps after maximum pooling and average pooling. The fully connected layer FC reduces the dimension, activates ReLU and increases the dimension of the feature maps after maximum pooling and average pooling through the convolution layer, generates two feature tensors, adds them and inputs them into the weights module to calculate the cross attention.

[0076] As a further improvement of the present invention, the image generation convolution layer is a 3*3 convolution layer.

[0077] The complete implementation process of the present invention is as follows:

[0078] The main part uses an encoder composed of wavelet convolution blocks, a decoder composed of upsampling convolution layers, and a bottleneck layer connecting the two to form a U-shaped network architecture, namely the N-net architecture.

[0079] Given a low-resolution image LR as input, on the one hand, the image enters the information supplement module ISM after preprocessing, and the SCM module passes the processed information to each level of the encoder through skip connections to ensure that the edge and detail information is not lost during the wavelet transform downsampling process.

[0080] On the other hand, the image is subjected to feature extraction through the preliminary convolution layer, and the initial feature map, i.e., the wavelet transform image, is generated as the input of the encoder. In the encoder, the initial feature map passes through multiple wavelet convolution blocks in sequence, and each wavelet convolution block extracts features and downsamples through wavelet transform to extract feature information of different scales. The wavelet convolution blocks pass the features, i.e., the detail map, to the corresponding upsampling convolution layer through skip connections for subsequent feature fusion and upsampling. The feature map further integrates deep features through the bottom bottleneck layer, and the processed feature map is passed to the decoder. In the decoder, the image is gradually reconstructed through the inverse wavelet transform operation in the upsampling convolution layer. Each upsampling convolution layer corresponds to the wavelet convolution block of the same layer, and the features of the previous small convolution wave block are fused through skip connections to ensure the integrity of the features during the upsampling process. The upsampling process will restore the resolution of the image layer by layer. After the decoder is finished, the 3*3 convolution layer is used to generate the final super-resolution image SR.

[0081] The wavelet transform is performed in the wavelet convolution block in the encoder, and the wavelet transform on the image is a two-dimensional discrete wavelet transform DWT, using the Haar wavelet transform. The scale function φ(t) and the wavelet function ψ(t) are:

[0082]

[0083] In discrete wavelet transform, the scaling function and wavelet function correspond to low-pass filter and high-pass filter, respectively, which are used to extract low-frequency information and high-frequency information. Specifically, the filter coefficients of the low-pass filter h[k] and high-pass filter g[k] of the Haar wavelet can be expressed as:

[0084]

[0085] The two-dimensional discrete wavelet transform (2D-DWT) can be viewed as performing two one-dimensional discrete wavelet transforms on the image: one in the row direction and the other in the column direction.

[0086] In the row direction, let the original image be f[i,j], perform a one-dimensional discrete wavelet transform on the row, and get the intermediate result a row [i,j] and d row [i,j]:

[0087]

[0088] where a row [i,j] represents the low-frequency part in the row direction, d row [i,j] represents the high-frequency part in the row direction, i and j represent the row and column indexes of the image respectively, h[k] represents the filter coefficient of the low-pass filter, which is also the scale function. The low-pass filter is used to extract the low-frequency part of the image and retain the overall contour and main features of the image. g[k] represents the filter coefficient of the high-pass filter, which is also the wavelet function. The high-pass filter is used to extract the high-frequency part of the image and retain the details and edge information of the image. k is the translation parameter, and t represents a continuous variable in the time or space domain.

[0089] A one-dimensional discrete wavelet transform is also performed in the column direction to obtain LL, LH, HL and HH:

[0090]

[0091] Among them, LL is the low-frequency sub-band, also called the approximation coefficient, which represents the overall outline and main features of the image; LH is the horizontal high-frequency sub-band, also called the horizontal detail coefficient, which represents the horizontal edges and details of the image; HL is the vertical high-frequency sub-band, also called the vertical detail coefficient, which represents the vertical edges and details of the image; HH is the diagonal high-frequency sub-band, also called the diagonal detail coefficient, which represents the diagonal edges and details of the image.

[0092] like Figure 4 As shown, the overall 2D-DWT result is [LL, LH, HL, HH] = DWT (f). Since only the values ​​of the interval positions are retained when the discrete wavelet transform obtains the corresponding coefficients each time, the size of the four sub-bands is the original image. The four sub-bands obtained after each wavelet transform layer are concatenated according to the channels. At this time, the height and width of the feature map are reduced to the original Figure 4 The number of channels is quadrupled and the feature map is input into the next wavelet transform layer to continue the DWT operation until the result is input into the bottleneck layer after the last wavelet transform layer.

[0093] The bottleneck layer consists of a dimensionality reduction convolution layer, a nonlinear activation function ReLU, and a dimensionality increase convolution layer. Its input is the output result of the wavelet transform layer. The feature map with a high number of channels is reduced in dimension through the dimensionality reduction convolution layer. The bottleneck layer effectively reduces the computational complexity and the number of parameters. The ReLU activation function introduces nonlinear transformation and improves the feature expression ability. Finally, the dimensionality increase convolution layer is used to restore the reduced dimensionality feature map to a high number of channels to maintain the richness of the features. The output of the bottleneck layer is used as the input of the decoder to gradually reconstruct the high-definition image. The process can be expressed as:

[0094] X bottleneck =W expand *ReLU(W reduce *X+b reduce )+b expand

[0095] Where X is the encoding graph, W reduce is the weight of the dimension reduction convolutional layer, b reduce is the bias of the dimension reduction convolutional layer, W expand is the weight of the up-dimensional convolutional layer, b expand is the bias of the dimension-raising convolutional layer, ReLU is the activation function, and X bottleneck It is an upsampled image.

[0096] like Figure 1 As shown in the figure, the decoder takes the output of the bottleneck layer and the jump connection of the encoder and the feature map after channel splicing as input, and performs inverse wavelet transform IDWT in the upsampling convolution layer to gradually reconstruct the image. Contrary to 2D-DWT, the process of 2D-IDWT can be regarded as performing two one-dimensional inverse wavelet transforms on the image: one in the column direction and the other in the row direction.

[0097] IDWT in the column direction yields:

[0098]

[0099] where a col [2i,j] is the approximate part, a col [2i+1,j] is the detail part.

[0100] The IDWT in the row direction yields:

[0101]

[0102] Where h[m], g[m] represent filter coefficients, [2i,2j], [2i+1,2j], [2i,2j+1], [2i+1,2j+1] represent pixel values ​​at each position, which are calculated by combining the coefficients of all subbands through low-pass and high-pass filters.

[0103] The result of the overall two-dimensional discrete inverse wavelet transform is X′=IDWT(LL, LH, HL, HH). Each inverse discrete wavelet transform will perform an upsampling operation to obtain the reconstructed image X′ and gradually increase the image resolution.

[0104] Finally, a 3*3 convolutional layer is used to generate the final high-resolution image Y.

[0105] Y=Conv 3×3 (X′)

[0106] like Figure 1 As shown, in order to reduce the loss of information on the image edges, especially the four corners, during the wavelet transform at the encoder stage, which affects the final reconstruction effect, the present invention designs an information supplement module ISM: the module first performs intercept cropping operations on the four corners of the input image, namely the upper left corner, the upper right corner, the lower left corner, and the lower right corner, with a size of The feature maps of the four corners are concatenated into one feature map by channel and then input into SCM. SCM is composed of N spatial attention SAs and channel cross attention CCAs. The last convolutional layer is used to adjust the size and dimension of the feature map to facilitate concatenation with the feature maps of each level of the encoder. The explanation of SA and CCA of SCM is as follows:

[0107] Spatial attention SA: Input feature map F, apply maximum pooling and average pooling to the feature map respectively, perform convolution operation on the pooled feature map after channel splicing, and finally calculate the attention score map F by sigmoid function conv :

[0108] F max =MaxPool(F)

[0109] F avg =AvgPool(F)

[0110] F conv =σ(Conv([F max ; F avg ]))

[0111] Where “;” represents concatenation in the channel dimension, and σ represents the sigmoid activation function

[0112] By F and attention score map F conv Multiply to get the attention feature map F SA :

[0113] F SA =F⊙F conv

[0114] Where ⊙ represents the point-by-point multiplication of elements

[0115] The overall spatial attention (SA) can be expressed as:

[0116] F SA =F⊙σ(Conv([MaxPool(F);AvgPool(F)]))

[0117] The spatial attention mechanism captures global spatial information by performing maximum pooling and average pooling operations on the input feature map, and fuses this information through convolution operations to generate an enhanced spatial feature map. The spatial attention mechanism can highlight important areas in the image, suppress unimportant areas, improve the spatial feature expression ability of the feature map, and help capture detailed information such as edges and textures.

[0118] Channel Cross Attention CCA: There are two feature map inputs, which are the features after SA Figure 1 (Feature_1) and features that have not undergone SA Figure 2 (Feature_2), for the input feature Figure 1 and Features Figure 2 , perform maximum pooling and average pooling, perform a fully connected layer FC operation on the pooled feature map, and use the convolution layer in the fully connected layer to reduce the dimension of the pooled features, activate ReLU, and increase the dimension. Generate two feature tensors and add them as the input of the weights module and calculate the cross attention. The formula for calculating the sum of feature tensors is:

[0119]

[0120] In the weights module, we first increase the dimensions of the two features to facilitate element-wise multiplication. Each element of the resulting matrix represents the feature Figure 1 A feature and characteristic of Figure 2 The interaction strength of a feature in , and then sum the last dimension of the interaction matrix, at this time each element represents the feature Figure 1 A feature and characteristic of Figure 2 The sum of the interaction strengths of all features in can be regarded as "interactive attention". This process can be expressed as:

[0121]

[0122] The weights module interactively calculates the information features of two different input sources. This module can effectively combine two sets of feature maps to enhance the model's ability to capture key information.

[0123] The channel cross attention mechanism can highlight important channels in the image, suppress unimportant channels, improve the channel feature expression ability of the feature map, and help capture global features such as color and contour.

[0124] By alternating the use of two attention mechanisms in the information supplementation module, the model can capture key information at different scales. Spatial attention can process local details, while channel attention can process global information. This combination enables the model to obtain effective features at different scales, extract and fuse important information in feature maps more effectively, dynamically adjust the importance of different features during feature fusion, enhance the overall feature expression ability of the network, and thus improve the quality of super-resolution image reconstruction.

Claims

1. A visual enhancement method based on wavelet transform N-net super-resolution, characterized in that: It includes a main module and an information supplement module ISM. The main module includes an encoder, a decoder, and a bottleneck layer connecting the encoder and the decoder. The encoder includes p wavelet convolution blocks connected in sequence. The decoder includes p upsampling convolution layers connected in sequence. The information supplement module ISM includes an attention calculation module SCM and p attention convolution layers. The input end of each attention convolution layer is connected to the attention calculation module SCM, and the output end of each attention convolution layer corresponds to the output end of a wavelet convolution block. The visual enhancement method includes the following steps: S1, input a low-resolution image LR, extract features of the low-resolution image LR through a preliminary convolution layer to obtain a wavelet transform image, and transmit it to the wavelet convolution block at the head end; The low-resolution image LR is preprocessed by the information supplement module ISM to obtain a preprocessed image, and then transmitted to the attention calculation module SCM; S2, the wavelet convolution block performs wavelet transform on the input wavelet transform image to obtain a transform image and a detail image, and jump-connects the detail image to the input end of the corresponding upsampling convolution layer; The attention calculation module SCM performs spatial attention SA and / or channel cross attention CCA calculation on the preprocessed image to obtain an attention feature map, and the attention convolution layer performs feature processing on the attention feature map to obtain an attention feature map with the same size and dimension as the transformation map; The attention feature map, the transformation map, and the detail map are concatenated and transmitted to the next wavelet convolution block, and step S2 is repeated for p-1 times to obtain the encoding map and transmit it to the bottleneck layer; S3, the bottleneck layer performs feature fusion processing on the coded image to obtain an upsampled image and transmits it to the upsampled convolution layer; S4, the upsampling convolution layer splices the input upsampling image and the detail image to obtain a reconstructed image, performs an inverse wavelet transform on the reconstructed image and transmits it to the next upsampling convolution layer, repeats step S4, and repeats it p-1 times to obtain a high-resolution reconstructed image and transmits it to the image generation convolution layer; S5. Generate a high-resolution image Y from the high-resolution reconstructed image through an image generation convolution layer.

2. According to claim 1, a visual enhancement method based on wavelet transform N-net super-resolution is characterized in that: The specific steps of preprocessing to obtain the preprocessed image are: cropping the upper left corner, upper right corner, lower left corner, and lower right corner of the low-resolution image LR respectively to obtain cropped feature maps corresponding to the four corners, and splicing the four cropped feature maps into the preprocessed image according to the channel.

3. The visual enhancement method based on wavelet transform N-net super-resolution according to claim 1, characterized in that: The wavelet transform uses two-dimensional discrete wavelet transform 2D-DWT, and the inverse wavelet transform uses two-dimensional discrete inverse wavelet transform 2D-IDWT.

4. The visual enhancement method based on wavelet transform N-net super-resolution according to claim 3 is characterized in that: The specific steps of two-dimensional discrete wavelet transform 2D-DWT include: Perform two one-dimensional discrete wavelet transforms, one in the row direction and the other in the column direction; In the row direction, let the original image be f[i,j], perform a one-dimensional discrete wavelet transform on the row, and get the intermediate result a row [i,j] and d row [i,j]: where a row [i,j] represents the low-frequency part in the row direction, d row [i,j] represents the high-frequency part in the row direction, i and j represent the row and column indexes of the image respectively, h[k] represents the filter coefficient of the low-pass filter; g[k] represents the filter coefficient of the high-pass filter, where h[k] and g[k] are expressed as: A one-dimensional discrete wavelet transform is also performed in the column direction to obtain LL, LH, HL and HH: Where LL is the low frequency subband, LH is the horizontal high frequency subband, HL is the vertical high frequency subband, and HH is the diagonal high frequency subband; In step S2, the transformation map is LL, and the detail maps include LH, HL and HH.

5. The visual enhancement method based on wavelet transform N-net super-resolution according to claim 1, characterized in that: The bottleneck layer includes a dimensionality reduction convolution layer, a nonlinear activation function ReLU, and a dimensionality increase convolution layer. In step S3, the bottleneck layer performs feature fusion processing on the encoded image to obtain the calculation process of the up-sampled image as follows: X bottleneck =W expand *ReLU(W reduce *X+b reduce )+b expand Where X is the encoding graph, W reduce is the weight of the dimension reduction convolutional layer, b reduce is the bias of the dimension reduction convolutional layer, W expand is the weight of the up-dimensional convolutional layer, b expand is the bias of the dimension-raising convolutional layer, ReLU is the activation function, and X bottleneck It is an upsampled image.

6. The visual enhancement method based on wavelet transform N-net super-resolution according to claim 1, characterized in that: Step S2: The attention calculation module SCM performs spatial attention SA and channel cross attention CCA calculations on the preprocessed image.

7. The visual enhancement method based on wavelet transform N-net super-resolution according to claim 6, characterized in that: The calculation steps of spatial attention SA include: applying maximum pooling and average pooling to the preprocessed image, convolving the feature map after pooling according to the channel, and finally calculating the attention score map F by the sigmoid function. conv , by preprocessing the image and the attention score map F conv Multiply to get the attention feature map F SA .

8. The visual enhancement method based on wavelet transform N-net super-resolution according to claim 7, characterized in that: The calculation steps of channel cross attention CCA include: SA The maximum pooling and average pooling are used for the preprocessed images. The fully connected layer FC operation is performed on the feature maps after maximum pooling and average pooling. The fully connected layer FC reduces the dimension, activates ReLU and increases the dimension of the feature maps after maximum pooling and average pooling through the convolution layer, generates two feature tensors, adds them and inputs them into the weights module to calculate the cross attention.

9. The visual enhancement method based on wavelet transform N-net super-resolution according to claim 1, characterized in that: The image generation convolution layer is a 3*3 convolution layer.

Images