A visual enhancement method based on wavelet transform N-net super-resolution

Abstract

The application discloses a kind of visual enhancement methods based on wavelet transform N-net super-resolution, and visual enhancement method includes the following steps: S1, feature extraction is carried out to obtain wavelet transform image;Preprocessing is carried out to obtain preprocessed image S2, attention feature map, transform graph and detail map are spliced and transmitted to next wavelet convolution block, and repeated to obtain encoding graph and transmit to bottleneck layer;S3, bottleneck layer carries out feature fusion processing to encoding graph and obtains up-sampling graph;S4, inverse wavelet transform is carried out to reconstructed image, and repeated to obtain high-resolution reconstructed image and transmit to image generation convolution layer;S5, high-resolution reconstructed image is generated high-resolution image Y by image generation convolution layer;Using the above method, training is carried out by depth learning technology, and low-resolution image can be output as high-resolution image, so that color and detail are more in line with people's visual effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image data processing, and particularly relates to a visual enhancement method based on wavelet transform N-net super resolution. BACKGROUND

[0002] Super resolution (SR) is an image processing technique that enhances the resolution of low-resolution images through algorithms. Its purpose is to overcome or compensate for the limitations of image acquisition systems or the acquisition environment itself, which can result in blurred, low-quality, and unremarkable regions of interest in the resulting images. This technique improves the details and clarity of images and has wide applications in many fields, such as medical imaging, satellite remote sensing, video surveillance, and multimedia entertainment. In recent years, although deep learning-based image super resolution methods have made significant progress compared to traditional methods, this technology still faces a series of challenges.

[0003] In the field of intelligent robots, image clarity is crucial for accurate perception and understanding of the environment, especially when dealing with images captured by low-quality cameras. By applying a wavelet transform combined with N-net super resolution method, the details and clarity of images can be significantly improved. However, traditional super resolution models mostly rely on convolutional layers to extract deep features. Since the convolutional results are fixed-scale local features, multiple convolutional layers are usually stacked to gradually increase the receptive field, resulting in a large amount of computation. In addition, convolutional layers mainly operate in the spatial domain, making it difficult to directly utilize frequency domain information, which is crucial for super resolution reconstruction. Although there are techniques for super resolution processing in the frequency domain, such as converting image information to the frequency domain for processing, like wavelet transform, and then converting back to the spatial domain to achieve super resolution, these methods may produce boundary effects at the image edges, leading to edge distortion in the reconstructed image. The reason is that the convolution operation in wavelet transform involves local regions of the signal. When performing convolution operations at the boundary, part of the filter kernel will exceed the boundary range of the signal, resulting in loss or inaccuracy of edge information. This situation is particularly evident in 2D image processing, especially at the four corners of the image. SUMMARY

[0004] In view of the above, the present application aims to provide a visual enhancement method based on wavelet transform N-net super resolution.

[0005] The technical solution adopted by the present invention to solve its technical problems is to provide a visual enhancement method based on wavelet transform N-net super-resolution, including a main module and an information supplement module ISM, the main module includes an encoder, a decoder and a bottleneck layer connecting the encoder and the decoder, the encoder includes p wavelet convolution blocks connected in sequence, the decoder includes p upsampling convolution layers connected in sequence, the information supplement module ISM includes an attention calculation module SCM and p attention convolution layers, the input end of each attention convolution layer is connected to the attention calculation module SCM, and the output end of each attention convolution layer corresponds to the output end of a wavelet convolution block;

[0006] The visual enhancement method includes the following steps:

[0007] S1, input low-resolution image LR, extract features of the low-resolution image LR through the preliminary convolution layer to obtain a wavelet transformed image, and transmit it to the wavelet convolution block at the head end;

[0008] The low-resolution image LR is preprocessed by the information supplement module ISM to obtain a preprocessed image, and then transmitted to the attention calculation module SCM;

[0009] S2, the wavelet convolution block performs wavelet transform on the input wavelet transform image to obtain the transform image and detail image, and jump-connects the detail image to the input end of the corresponding upsampling convolution layer;

[0010] The attention calculation module SCM performs spatial attention SA and / or channel cross attention CCA calculation on the preprocessed image to obtain an attention feature map, and the attention convolution layer performs feature processing on the attention feature map to obtain an attention feature map with the same size and dimension as the transformation map;

[0011] The attention feature map, transformation map, and detail map are concatenated and transmitted to the next wavelet convolution block. Step S2 is repeated p-1 times to obtain the encoding map and transmitted to the bottleneck layer.

[0012] S3, the bottleneck layer performs feature fusion processing on the encoded image to obtain an upsampled image and transmits it to the upsampling convolution layer;

[0013] S4, the upsampling convolution layer concatenates the input upsampling image and the detail image to obtain a reconstructed image, performs an inverse wavelet transform on the reconstructed image and transmits it to the next upsampling convolution layer, repeats step S4, and repeats it p-1 times to obtain a high-resolution reconstructed image and transmits it to the image generation convolution layer;

[0014] S5. Generate a high-resolution image Y from the high-resolution reconstructed image through an image generation convolution layer.

[0015] As a further improvement of the present application, the specific steps of preprocessing to obtain the preprocessed image are: respectively performing clipping operations on the upper left corner, the upper right corner, the lower left corner and the lower right corner of the low-resolution image LR to obtain the clipping feature maps corresponding to the four corners, and splicing the four clipping feature maps into the preprocessed image according to the channel.

[0016] As a further improvement of the present application, the wavelet transform adopts two-dimensional discrete wavelet transform 2D-DWT, and the inverse wavelet transform adopts two-dimensional discrete inverse wavelet transform 2D-IDWT.

[0017] As a further improvement of the present application, the specific steps of the two-dimensional discrete wavelet transform 2D-DWT include:

[0018] Performing two one-dimensional discrete wavelet transforms, one in the row direction and the other in the column direction;

[0019] In the row direction, assuming that the original image is f[i,j], performing one-dimensional discrete wavelet transform on the row to obtain the intermediate results a row [i,j] and d row [i,j]:

[0020]

[0021] Wherein a row [i,j] represents the low-frequency part in the row direction, d row [i,j] represents the high-frequency part in the row direction, i and j represent the row and column indexes of the image respectively, and h[k] represents the filter coefficient of the low-pass filter; g[k] represents the filter coefficient of the high-pass filter, wherein h[k] and g[k] are represented as:

[0022]

[0023] In the column direction, also performing one-dimensional discrete wavelet transform to obtain LL, LH, HL and HH:

[0024]

[0025] Wherein LL is a low-frequency subband, LH is a horizontal high-frequency subband, HL is a vertical high-frequency subband, and HH is a diagonal high-frequency subband;

[0026] In step S2, the transform image is LL, and the detail image includes LH, HL and HH.

[0027] As a further improvement of the present application, the bottleneck layer includes a dimension reduction convolution layer, a nonlinear activation function ReLU and a dimension increase convolution layer, and the calculation process of the bottleneck layer for performing feature fusion processing on the encoding image to obtain the up-sampling image is represented as:

[0028] X bottleneck =Wexpand ReLU(W reduce X+b reduce )+b expand

[0029] where X is an encoded image, W reduce is a weight of a dimension reduction convolutional layer, b reduce is a bias of the dimension reduction convolutional layer, W expand is a weight of a dimension increase convolutional layer, b expand is a bias of the dimension increase convolutional layer, ReLU is an activation function, and X bottleneck is an up-sampling image.

[0030] As a further improvement of the present application, the step S2 attention calculation module SCM performs spatial attention SA and channel cross attention CCA calculation on the preprocessed image.

[0031] As a further improvement of the present application, the calculation step of the spatial attention SA includes: using maximum pooling and average pooling on the preprocessed image, splicing the feature maps after the pooling by channel, and then performing convolution operation, and finally calculating the attention score map F conv by a sigmoid function. conv The attention feature map F SA is obtained by multiplying the preprocessed image and the attention score map F conv .

[0032] As a further improvement of the present application, the calculation step of the channel cross attention CCA includes: using maximum pooling and average pooling on the attention feature map F SA and the preprocessed image, performing a fully connected layer FC operation on the feature maps after the maximum pooling and average pooling, and performing dimension reduction, ReLU activation and dimension increase on the feature maps after the maximum pooling and average pooling by the convolution layer of the fully connected layer FC to generate two feature tensors, and then adding and inputting the two feature tensors into a weights module to calculate the cross attention.

[0033] As a further improvement of the present application, the image generation convolutional layer is a 3*3 convolutional layer.

[0034] The present application has at least the following beneficial effects:

[0035] 1. The traditional down-sampling layer is replaced by a wavelet transform layer, high-frequency domain information is extracted in the encoder through multi-scale, image reconstruction is performed through inverse wavelet transform in the decoder stage, and a skip connection operation is performed in the process, and the feature fusion layer in the up-sampling stage fuses the high-frequency information of each layer of the encoder and the reconstructed information.

[0036] 2、In order to reduce the information loss caused by boundary effect in wavelet transform reconstruction, an information supplement module ISM is proposed. Through a series of spatial attention and channel cross attention alternating structure, the deep edge information is extracted and injected into the decoder stage, which makes up for the loss of information in the wavelet transform layer, so as to enhance the edge details and abstract features, and achieve better reconstruction results.

[0037] 3、Through the training of deep learning technology, the low resolution image can be output as a high resolution image, so that the color and detail are more in line with people's visual effect. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is the overall framework diagram of the present application;

[0039] Figure 2 is the wavelet transform step diagram;

[0040] Figure 3 is the inverse wavelet transform step diagram;

[0041] Figure 4 is the visualization diagram of wavelet transform;

[0042] Figure 5 is the framework diagram of information supplement module ISM. DETAILED DESCRIPTION

[0043] The technical solutions of the present application will be described in detail below with reference to the drawings.

[0044] Reference Figures 1-5 , the present application proposes a visual enhancement method based on wavelet transform N-net super resolution, which includes a main module and an information supplement module ISM. The main module includes an encoder, a decoder and a bottleneck layer connecting the encoder and the decoder. The encoder includes p wavelet convolution blocks connected in sequence. The decoder includes p up-sampling convolution layers connected in sequence. The information supplement module ISM includes an attention calculation module SCM and p attention convolution layers. The input end of each attention convolution layer is connected to the attention calculation module SCM, and the output end of each attention convolution layer corresponds to the output end of a wavelet convolution block.

[0045] The visual enhancement method includes the following steps:

[0046] S1, input a low resolution image LR, and extract features of the low resolution image LR through a preliminary convolution layer to obtain a wavelet transform image, and transmit the wavelet transform image to a wavelet convolution block at the head end;

[0047] The low resolution image LR is preprocessed by the information supplement module ISM to obtain a preprocessed image, and the preprocessed image is transmitted to the attention calculation module SCM;

[0048] S2, the wavelet convolution block performs wavelet transform on the input wavelet transform image to obtain a transform map and a detail map, and the detail map is jump-connected to the input end of the corresponding up-sampling convolution layer;

[0049] The attention calculation module SCM performs spatial attention SA or / and channel cross-attention CCA calculation on the preprocessed image to obtain an attention feature map, and the attention convolution layer performs feature processing on the attention feature map to obtain an attention feature map with the same size and dimension as the transform map;

[0050] The attention feature map, the transform map and the detail map are spliced and transmitted to the next wavelet convolution block, and step S2 is repeated p-1 times to obtain an encoded map and transmit it to the bottleneck layer;

[0051] S3, the bottleneck layer performs feature fusion processing on the encoded map to obtain an up-sampling map and transmit it to the up-sampling convolution layer;

[0052] S4, the up-sampling convolution layer splices the input up-sampling map and the detail map to obtain a reconstructed image, performs inverse wavelet transform on the reconstructed image and transmits it to the next up-sampling convolution layer, and step S4 is repeated p-1 times to obtain a high-resolution reconstructed image and transmit it to the image generation convolution layer;

[0053] S5, the image generation convolution layer generates a high-resolution image Y from the high-resolution reconstructed image.

[0054] When the frequency domain transform is wavelet transform, it is a global representation of the entire signal, not just a local region, therefore, any operation on the frequency domain data will affect the entire image; in the super-resolution task, frequency domain processing is often used to recover the lost high-frequency information, if the boundary is not handled properly, the error in the frequency domain will be amplified, resulting in obvious distortion of the edges and corners of the reconstructed image, and in the subsequent frequency domain conversion back to the spatial domain, i.e. inverse wavelet transform, is also a global operation, if there is inaccuracy or distortion in the frequency domain data at the boundary, these errors will be amplified in the inverse transform process and propagated to the entire reconstructed image.

[0055] If wavelet transform is not used, and spatial domain convolution is used directly, even if there is some information loss at the boundary, the overall impact is relatively small, because the original design of convolutional neural networks is to capture local patterns, therefore they have a certain robustness to boundary effects, but if the resolution of the image is to be improved, frequency transform is inevitable, because in the super-resolution task, the goal is to recover the high-frequency details in the frequency domain, such as texture, lines and other fine structures, which are crucial to visual perception, because the low-resolution image often loses these details. Traditional interpolation methods can only provide smooth estimates and cannot well reconstruct the original high-frequency components.

[0056] The skip connection can directly provide spatial position information from the encoder to the decoder, which is very important for reconstructing high-resolution output, especially for segmentation or restoration tasks. If only relying on the feature map processed by the bottleneck layer, some details may be lost; even if feature fusion is performed at the bottleneck layer, it is difficult to ensure that all useful information can be perfectly transmitted, and the skip connection as a complementary mechanism can help to ensure that key features are not lost during transmission.

[0057] Since the technical solutions are closely related to each other, the subsequent technical solutions and beneficial effects are fully described in the last implementation process.

[0058] As a further improvement of the present application, the specific steps of preprocessing to obtain the preprocessed image are: respectively performing a clipping operation on the upper left corner, the upper right corner, the lower left corner and the lower right corner of the low-resolution image LR to obtain the clipping feature maps corresponding to the four corners, and splicing the four clipping feature maps into a preprocessed image according to the channel.

[0059] As a further improvement of the present application, the wavelet transform adopts two-dimensional discrete wavelet transform 2D-DWT, and the inverse wavelet transform adopts two-dimensional discrete inverse wavelet transform 2D-IDWT.

[0060] As a further improvement of the present application, the specific steps of the two-dimensional discrete wavelet transform 2D-DWT include:

[0061] Two one-dimensional discrete wavelet transforms are performed, one in the row direction and the other in the column direction;

[0062] In the row direction, let the original image be f[i,j], perform one-dimensional discrete wavelet transform on the row to obtain the intermediate results a row [i,j] and d row [i,j]:

[0063]

[0064] where a row [i,j] represents the low-frequency part in the row direction, d row [i,j] represents the high-frequency part in the row direction, i and j represent the row and column indices of the image respectively, and h[k] represents the filter coefficient of the low-pass filter; g[k] represents the filter coefficient of the high-pass filter, where h[k] and g[k] are represented as:

[0065]

[0066] In the column direction, one-dimensional discrete wavelet transform is also performed to obtain LL, LH, HL and HH:

[0067]

[0068] wherein LL is a low frequency subband, LH is a horizontal high frequency subband, HL is a vertical high frequency subband, and HH is a diagonal high frequency subband;

[0069] The transformed image in step S2 is LL, and the detail image includes LH, HL and HH.

[0070] As a further improvement of the present application, the bottleneck layer includes a dimension reduction convolutional layer, a nonlinear activation function ReLU and a dimension increase convolutional layer, and the calculation process of the bottleneck layer for feature fusion processing of the encoded image to obtain the up-sampled image is represented as:

[0071] X bottleneck =W expand *ReLU(W reduce *X+b reduce )+b expand

[0072] wherein X is the encoded image, W reduce is the weight of the dimension reduction convolutional layer, b reduce is the bias of the dimension reduction convolutional layer, W expand is the weight of the dimension increase convolutional layer, b expand is the bias of the dimension increase convolutional layer, ReLU is the activation function, and X bottleneck is the up-sampled image.

[0073] As a further improvement of the present application, the attention calculation module SCM in step S2 calculates the spatial attention SA and the channel cross attention CCA of the preprocessed image.

[0074] As a further improvement of the present application, the calculation steps of the spatial attention SA include: using maximum pooling and average pooling on the preprocessed image, performing convolution operation on the channel spliced feature map after the pooling, and finally calculating the attention score map F conv by the sigmoid function, and obtaining the attention feature map F conv by multiplying the preprocessed image and the attention score map F SA .

[0075] As a further improvement of the present application, the calculation steps of the channel cross attention CCA include: using maximum pooling and average pooling on the attention feature map F SA and the preprocessed image, performing full connection layer FC operation on the feature maps after the maximum pooling and average pooling, and generating two feature tensors by the full connection layer FC for dimension reduction, ReLU activation and dimension increase on the feature maps after the maximum pooling and average pooling, adding and inputting the two feature tensors into the weights module to calculate the cross attention.

[0076] As a further improvement of the present application, the image generation convolutional layer is a 3*3 convolutional layer.

[0077] The complete implementation process of the present application is as follows:

[0078] The main part adopts an encoder composed of wavelet blocks, a decoder composed of up-convolution layers, and a bottleneck layer connecting the two, forming a U-shaped network architecture, i.e., an N-net architecture,

[0079] Given a low-resolution image LR as input, on the one hand, the image is preprocessed and then input into the information supplement module ISM, and the processed information is transmitted to each level of the encoder through the skip connection of the SCM module, ensuring that the edge and detail information is not lost during the wavelet down-sampling process.

[0080] On the other hand, the image is subjected to feature extraction through the preliminary convolution layer, and the initial feature map, i.e., the wavelet transformed image, is generated as the input of the encoder. In the encoder, the initial feature map is sequentially passed through multiple wavelet convolution blocks, each of which extracts features and down-samples through wavelet transformation to extract feature information of different scales. The features, i.e., the detail maps, are transmitted to the corresponding up-convolution layer through the skip connection between the wavelet convolution blocks, for subsequent feature fusion and up-sampling. The feature map is further integrated with deep-level features through the bottom bottleneck layer, and the processed feature map is transmitted into the decoder. In the decoder, the image is gradually reconstructed through the inverse wavelet transformation operation in the up-convolution layer. Each up-convolution layer corresponds to a wavelet convolution block at the same level, and the features of the previous wavelet convolution block are fused through the skip connection to ensure the integrity of the features during the up-sampling process. The up-sampling process will recover the resolution of the image layer by layer, and the final super-resolution image SR is generated by the 3*3 convolution layer at the end of the decoder.

[0081] In the wavelet convolution block in the encoder, the wavelet transformation is performed, and the wavelet transformation on the image is a two-dimensional discrete wavelet transform (DWT). The haar wavelet transform is used. The scale function φ(t) and the wavelet function ψ(t) are:

[0082]

[0083] In the discrete wavelet transform, the scale function and the wavelet function correspond to the low-pass filter and the high-pass filter, respectively, for extracting low-frequency information and high-frequency information. Specifically, the filter coefficients of the low-pass filter h[k] and the high-pass filter g[k] of the Haar wavelet can be expressed as:

[0084]

[0085] The two-dimensional discrete wavelet transform (2D-DWT) can be regarded as performing one-dimensional discrete wavelet transform on the image twice: once in the row direction and once in the column direction.

[0086] In the row direction, let the original image be f[i,j], a one-dimensional discrete wavelet transform is performed on the rows to obtain an intermediate result a row [i,j] and d row [i,j]:

[0087]

[0088] where a row [i,j] represents the low-frequency part in the row direction, d row [i,j] represents the high-frequency part in the row direction, i and j represent the row and column indices of the image respectively, h[k] represents the filter coefficient of the low-pass filter which is also the scaling function, the low-pass filter is used to extract the low-frequency part of the image, retaining the overall contour and main features of the image. g[k] represents the filter coefficient of the high-pass filter which is also the wavelet function, the high-pass filter is used to extract the high-frequency part of the image, retaining the details and edge information of the image, and k is the translation parameter, t represents a continuous variable in the time or spatial domain.

[0089] A one-dimensional discrete wavelet transform is also performed in the column direction to obtain LL, LH, HL and HH:

[0090]

[0091] where LL is the low-frequency subband also called the approximation coefficient, representing the overall contour and main features of the image; LH is the horizontal high-frequency subband also called the horizontal detail coefficient, representing the horizontal edges and details of the image; HL is the vertical high-frequency subband also called the vertical detail coefficient, representing the vertical edges and details of the image; and HH is the diagonal high-frequency subband also called the diagonal detail coefficient, representing the diagonal edges and details of the image.

[0092] As shown in Figure 4 , the overall 2D-DWT result is [LL, LH, HL, HH] = DWT(f), since only the values at the interval positions are retained when the corresponding coefficients are obtained each time the discrete wavelet transform is performed, the size of each of the four subbands is After each wavelet transform layer, the four subbands obtained are spliced according to the channel, at this time the height and width of the feature map are reduced to Figure Four times, and the number of channels is expanded to four times, the feature map is input to the next wavelet transform layer for continuous DWT operation, until the result is input into the bottleneck layer after the last wavelet transform layer.

[0093] The bottleneck layer consists of a dimensionality reduction convolutional layer, a nonlinear activation function called ReLU, and a dimensionality increase convolutional layer. Its input is the output of the wavelet transform layer. The dimensionality reduction convolutional layer reduces the dimensionality of the high-channel feature map, effectively reducing computational complexity and the number of parameters. The ReLU activation function introduces nonlinear transformations, improving feature expression. Finally, the dimensionality increase convolutional layer restores the reduced dimensionality feature map to a high-channel count, preserving feature richness. The output of the bottleneck layer serves as the input to the decoder, which gradually reconstructs the high-definition image. The process can be expressed as:

[0094] X bottleneck =W expand *ReLU(W reduce *X+b reduce )+b expand

[0095] Where X is the encoding graph, W reduce is the weight of the dimension reduction convolution layer, b reduce is the bias of the dimension reduction convolution layer, W expand is the weight of the dimension-raising convolutional layer, b expand is the bias of the dimensional convolution layer, ReLU is the activation function, X bottleneck It is an upsampled image.

[0096] like Figure 1 As shown in the figure, the decoder takes the output of the bottleneck layer and the jump connection of the encoder and the feature map after channel splicing as input, and performs inverse wavelet transform IDWT in the upsampling convolution layer to gradually reconstruct the image. In contrast to 2D-DWT, the process of 2D-IDWT can be regarded as performing two one-dimensional inverse wavelet transforms on the image: one in the column direction and the other in the row direction.

[0097] The IDWT in the column direction yields:

[0098]

[0099] where a col [2i,j] is the approximate part, a col [2i+1,j] is the detail part.

[0100] The IDWT in the row direction yields:

[0101]

[0102] Where h[m], g[m] represent the filter coefficients, [2i,2j], [2i+1,2j], [2i,2j+1], and [2i+1,2j+1] represent the pixel values ​​at each position, which are calculated by combining the coefficients of all subbands through low-pass and high-pass filters.

[0103] The result of the whole two-dimensional discrete inverse wavelet transform is X' = IDWT(LL, LH, HL, HH), and each inverse discrete wavelet transform is performed by upsampling to obtain a reconstructed image X' and gradually expand the image resolution.

[0104] Finally, a 3*3 convolution layer is used to generate the final high-resolution image Y.

[0105] Y = Conv 3×3 (X')

[0106] As Figure 1 shown, in order to reduce the loss of information of image edges, especially the four corners, during the wavelet transform in the encoder stage, which affects the final reconstruction effect, the present application designs an information supplement module ISM: the module first performs intercept cropping operation on the four corners of the input image, which are the upper left corner, the upper right corner, the lower left corner and the lower right corner, and the size is The feature maps of the four corners are spliced into one feature map according to the channel and then input into the SCM. The SCM is composed of N spatial attention SAs and channel cross attention CCAs, and the last convolution layer is used to adjust the size and dimension of the feature map to facilitate splicing with the feature maps of each level of the encoder. The SA and CCA of the SCM are explained as follows:

[0107] Spatial attention SA: input feature map F, respectively, maximum pooling and average pooling are performed on the feature map, the channel-pasted feature maps are convolved, and finally the attention score map F conv is calculated by the sigmoid function

[0108] F max = MaxPool(F)

[0109] F avg = AvgPool(F)

[0110] F conv = σ(Conv([F max ; F avg ]))

[0111] Where ";" represents channel-wise splicing, and σ represents the sigmoid activation function

[0112] The attention feature map F conv is obtained by multiplying F and the attention score map F SA :

[0113] F SA = F ⊙ F conv

[0114] Where "⊙" represents element-wise multiplication

[0115] The overall spatial attention (SA) can be represented as:

[0116] F SA = F o σ(Conv([MaxPool(F); AvgPool(F)]))

[0117] The spatial attention mechanism captures global spatial information by performing max-pooling and average-pooling operations on the input feature map, and fuses these information through convolution operation to generate enhanced spatial feature map. The spatial attention mechanism can highlight important regions in the image and suppress unimportant regions, enhancing the spatial feature expression ability of the feature map and helping to capture detailed information such as edges and textures.

[0118] Channel cross attention CCA: There are two feature maps as input, which are the features Figure 1 (Feature_1) and the features Figure 2 (Feature_2) that have not been subjected to SA. For the input features Figure 1 and features Figure 2 , max-pooling and average-pooling are performed, and a fully connected layer FC operation is performed on the pooled feature map. In the fully connected layer, the pooled features are reduced in dimension, activated by ReLU, and increased in dimension through a convolution layer to generate two feature tensors, which are added together as the input of the weights module and calculate the cross attention. The formula for calculating the sum of feature tensors is:

[0119]

[0120] In the weights module, the dimensions of the two features are first increased to facilitate element multiplication operations. The resulting matrix has an element representing the interaction strength between a feature in feature Figure 1 and a feature in feature Figure 2 . Then, the last dimension of the interaction matrix is summed, and each element represents the sum of the interaction strength between a feature in feature Figure 1 and all features in feature Figure 2 . This can be regarded as "interaction attention". This process can be represented as:

[0121]

[0122] The weights module calculates the interaction between the information features of the two different input sources, which can effectively combine the two groups of feature maps to enhance the model's ability to capture key information.

[0123] The channel cross attention mechanism can highlight important channels in the image and suppress unimportant channels, enhancing the channel feature expression ability of the feature map and helping to capture global features such as color and contour

[0124] By alternating the use of the two attention mechanisms in the information supplementing module, the model helps to capture key information at different scales. Spatial attention can handle local details, while channel attention can handle global information. This combination enables the model to obtain effective features at different scales, more effectively extract and fuse important information in the feature map, dynamically adjust the importance of different features during the feature fusion process, enhance the overall feature expression ability of the network, and thus improve the quality of the super-resolution image reconstruction.

Claims

1. A visual enhancement method based on wavelet transform N-net super-resolution, characterized in that: The model for implementing the visual enhancement method includes a main module and an information supplement module ISM. The main module includes an encoder, a decoder, and a bottleneck layer connecting the encoder and the decoder. The encoder includes p wavelet convolution blocks connected in sequence. The decoder includes p upsampling convolution layers connected in sequence. The information supplement module ISM includes an attention calculation module SCM and p attention convolution layers. The input end of each attention convolution layer is connected to the attention calculation module SCM, and the output end of each attention convolution layer corresponds to the output end of a wavelet convolution block. The visual enhancement method comprises the following steps: S1, input low-resolution image LR, extract features of the low-resolution image LR through the preliminary convolution layer to obtain a wavelet transformed image, and transmit it to the wavelet convolution block at the head end; The low-resolution image LR is preprocessed by the information supplement module ISM to obtain a preprocessed image, and then transmitted to the attention calculation module SCM; S2, the wavelet convolution block performs wavelet transform on the input wavelet transform image to obtain the transform image and detail image, and jump-connects the detail image to the input end of the corresponding upsampling convolution layer; The attention calculation module SCM performs spatial attention SA and channel cross attention CCA calculations on the preprocessed image to obtain an attention feature map, and the attention convolution layer performs feature processing on the attention feature map to obtain an attention feature map with the same size and dimension as the transformation map; The attention feature map, transformation map, and detail map are concatenated and transmitted to the next wavelet convolution block. Step S2 is repeated p-1 times to obtain the encoding map and transmitted to the bottleneck layer. S3, the bottleneck layer performs feature fusion processing on the encoded image to obtain an upsampled image and transmits it to the upsampling convolution layer; S4, the upsampling convolution layer concatenates the input upsampling image and the detail image to obtain a reconstructed image, performs an inverse wavelet transform on the reconstructed image and transmits it to the next upsampling convolution layer, repeats step S4, and repeats it p-1 times to obtain a high-resolution reconstructed image and transmits it to the image generation convolution layer; S5. Generate a high-resolution image Y from the high-resolution reconstructed image through an image generation convolution layer.

2. The visual enhancement method based on wavelet transform N-net super-resolution according to claim 1, characterized in that: The specific steps of preprocessing to obtain the preprocessed image are: performing cropping operations on the upper left corner, upper right corner, lower left corner, and lower right corner of the low-resolution image LR respectively to obtain cropping feature maps corresponding to the four corners, and splicing the four cropping feature maps into the preprocessed image according to the channel.

3. The visual enhancement method based on wavelet transform N-net super-resolution according to claim 1, characterized in that: The wavelet transform uses two-dimensional discrete wavelet transform 2D-DWT, and the inverse wavelet transform uses two-dimensional discrete inverse wavelet transform 2D-IDWT.

4. The visual enhancement method based on wavelet transform N-net super-resolution according to claim 3, characterized in that: The specific steps of two-dimensional discrete wavelet transform 2D-DWT include: Perform two one-dimensional discrete wavelet transforms, one in the row direction and the other in the column direction; In the row direction, let the original image be , perform one-dimensional discrete wavelet transform on the row and get the intermediate result and : ； ； in represents the low-frequency part in the row direction, Represents the high-frequency part in the row direction, and Represent the row and column indices of the image, respectively. Represents the filter coefficient of the low-pass filter; represents the filter coefficient of the high-pass filter, where and Respectively expressed as: ； ； One-dimensional discrete wavelet transform is also performed in the column direction to obtain , , and : ； ； ； ； in is the low-frequency subband, is the horizontal high frequency subband, is the vertical high frequency sub-band, is the diagonal high frequency subband; The transformation graph in step S2 is , details include 、 and .

5. The visual enhancement method based on wavelet transform N-net super-resolution according to claim 1, characterized in that: The bottleneck layer includes a dimensionality reduction convolution layer, a nonlinear activation function ReLU, and a dimensionality increase convolution layer. In step S3, the bottleneck layer performs feature fusion processing on the encoded image to obtain the upsampled image. The calculation process is expressed as follows: ； in It is a coding diagram. is the weight of the dimension reduction convolution layer, is the bias of the dimension reduction convolution layer, is the weight of the dimension-raising convolutional layer, is the bias of the dimension-raising convolutional layer, is the activation function, It is an upsampled image.

6. The visual enhancement method based on wavelet transform N-net super-resolution according to claim 1, characterized in that: The calculation steps of spatial attention SA include: applying maximum pooling and average pooling to the preprocessed image, performing convolution operation on the feature map after pooling according to the channel, and finally calculating the attention score map by sigmoid function. , by preprocessing the image and attention score map Multiply to get the attention feature map .

7. The visual enhancement method based on wavelet transform N-net super-resolution according to claim 6, characterized in that: The calculation steps of channel cross attention CCA include: The maximum pooling and average pooling are used for the preprocessed images. The fully connected layer FC operation is performed on the feature maps after maximum pooling and average pooling. The fully connected layer FC reduces the dimension, activates ReLU and increases the dimension of the feature maps after maximum pooling and average pooling through the convolution layer, generates two feature tensors, adds them up and inputs them into the weights module to calculate the cross attention.

8. The visual enhancement method based on wavelet transform N-net super-resolution according to claim 1, characterized in that: The image generation convolution layer is a 3*3 convolution layer.

Images