Image Super-Resolution System and Method Based on Wavelet Gated Attention

Abstract

This invention relates to the field of image processing technology, and more particularly to an image super-resolution system and method based on wavelet-gated attention. A multi-scale lightweight encoder performs multi-scale encoding on the input low-resolution image, and then performs attention enhancement on the obtained multi-scale features to obtain encoded features. A frequency-domain gated attention module performs frequency-domain decomposition on the encoded features, and then performs gated attention enhancement operations on the multiple frequency-domain sub-bands obtained. The enhanced frequency-domain sub-bands are then concatenated and transformed into the spatial domain to obtain enhanced features. A multi-granularity lightweight decoder performs positional encoding on the enhanced features, and then performs coarse-grained and fine-grained decoding on the encoded enhanced features. The two types of decoded features are then weighted and fused to obtain the reconstructed image. This invention combines the computational efficiency advantages of wavelet transform with the feature selection capabilities of the gating mechanism, maintaining arbitrary-scale super-resolution capability while significantly reducing the number of parameters and computational complexity.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to an image super-resolution system and method based on wavelet-gated attention. Background Technology

[0002] Super-resolution (SR) image reconstruction technology refers to the technique of recovering high-resolution images from low-resolution images. It has wide applications in fields such as medical imaging, satellite remote sensing, video surveillance, and consumer electronics. Traditional interpolation methods (such as bilinear interpolation and bicubic interpolation), although computationally simple, are difficult to recover high-frequency details of images, resulting in limited reconstruction quality.

[0003] In recent years, deep learning-based super-resolution methods have made significant progress. Convolutional neural network methods, such as SRCNN, EDSR, and RDN, have performed well in super-resolution tasks with fixed magnification, but they still have the following problems: (1) It can only handle specific magnifications and cannot flexibly adapt to different scale requirements; (2) The network parameters are large, making it difficult to deploy on resource-constrained devices such as mobile devices; (3) The receptive field is limited, making it difficult to effectively capture global contextual information.

[0004] To address the fixed-scale problem, researchers have proposed methods based on implicit neural representations, such as LIIF (Local Implicit Image Function) and LTE (Local Texture Estimator), which can achieve super-resolution reconstruction at arbitrary magnification. However, these methods typically have complex network structures, high computational costs, and lack effective feature selection mechanisms, limiting their effectiveness when handling complex texture scenes.

[0005] On the other hand, wavelet transform, due to its excellent time-frequency localization properties and multi-resolution analysis capabilities, has been widely used in image processing. Wavelet transform can decompose an image into different frequency sub-bands, facilitating the targeted processing of low-frequency structural information and high-frequency detail information. However, existing wavelet-based super-resolution methods mainly use wavelet transform for feature extraction or upsampling, failing to fully leverage its advantages in reducing computational complexity.

[0006] Furthermore, attention mechanisms demonstrate powerful feature modeling capabilities in super-resolution tasks, but the complexity of traditional self-attention mechanisms is O(n^2). 2 The computational complexity, where n is the sequence length, limits its application on high-resolution images. Therefore, how to effectively reduce network complexity and improve the targeting of feature selection while maintaining the super-resolution capability at arbitrary scales has become an urgent technical problem to be solved. Summary of the Invention

[0007] In view of this, the present invention aims to provide an image super-resolution system and method based on wavelet-gated attention, which combines the computational efficiency advantage of wavelet transform with the feature selection capability of gating mechanism to achieve lightweight, high-quality arbitrary-scale super-resolution reconstruction.

[0008] To achieve the above objectives, the technical solution created by this invention is implemented as follows: A wavelet-gated attention-based image super-resolution system includes: a multi-scale lightweight encoder that encodes the input low-resolution image at multiple scales, and performs attention enhancement on the obtained multi-scale features to obtain encoded features; a frequency-domain gated attention module that performs frequency-domain decomposition on the encoded features, performs gated attention enhancement on the multiple frequency-domain sub-bands obtained, and concatenates the enhanced frequency-domain sub-bands and converts them into the spatial domain to obtain enhanced features; and a multi-granularity lightweight decoder that performs positional encoding on the enhanced features, performs coarse-grained decoding and fine-grained decoding on the encoded enhanced features, and performs weighted fusion of the two decoded features to obtain the reconstructed image.

[0009] Furthermore, in the multi-scale lightweight encoder: shallow feature extraction is performed on the low-resolution image; different scale depth-separable convolution operations are performed on the shallow features; the feature channels of different scales are concatenated and then channel attention operation is performed; and the processed features are then connected with the shallow feature residuals to obtain the encoded features.

[0010] Furthermore, in the frequency domain gated attention module: the coded features are subjected to discrete wavelet transform to obtain low-frequency sub-bands, horizontal high-frequency sub-bands, vertical high-frequency sub-bands, and diagonal high-frequency sub-bands; gated attention enhancement operations are performed on the four frequency domain sub-bands respectively; and the enhanced four frequency domain sub-bands are subjected to inverse discrete wavelet transform to obtain enhanced features.

[0011] Furthermore, in the gated attention enhancement operation: a linear mapping is performed on each frequency domain sub-band to obtain the corresponding query matrix, key matrix, and value matrix; attention calculation is performed on the query matrix, key matrix, and value matrix using the following formula to obtain the self-attention weight matrix: ; Where A represents the self-attention weight matrix, softmax represents the softmax operation, Q, K, and V represent the query matrix, key matrix, and value matrix, respectively, and d represents the scaling factor. The self-attention weight matrix is ​​pooled, and then the pooled features are continuously subjected to multilayer perceptron operations and sigmoid activation operations to complete the gating operation of the self-attention weight matrix, resulting in the enhanced self-attention weight matrix. The enhanced self-attention weight matrix is ​​multiplied by the corresponding elements of the self-attention weight matrix to obtain the enhanced frequency domain subband.

[0012] Furthermore, the multi-granularity lightweight decoder includes a position encoding module, a coarse-grained decoding module, a fine-grained decoding module, and a decoding and reconstruction module. In the position encoding module, the pixel positions in the target high-resolution image are normalized to obtain the query coordinates. Then, the query coordinates are subjected to sine and cosine position encoding to generate a high-dimensional position encoding vector. In the coarse-grained decoding module, based on the high-dimensional position encoding vector corresponding to the query coordinates, the enhanced features are weighted and predicted based on distance weights for the 4-neighbor features to obtain coarse-grained features. In the fine-grained decoding module, the enhanced features are spatially attention-enhanced. Then, based on the high-dimensional position encoding vector corresponding to the query coordinates, the spatially attention-enhanced features are weighted and predicted based on distance weights for the 4-neighbor features to obtain fine-grained features. In the decoding and reconstruction module, the coarse-grained features and the fine-grained features are weighted and summed to obtain the reconstructed features.

[0013] Furthermore, the coarse-grained features are obtained in the coarse-grained decoding module using the following formula; ; Among them, I coarse (q) represents a coarse-grained feature, where q represents the query coordinate in the query matrix, and N4 represents the 4-neighbor element of the enhanced feature corresponding to the query coordinate q. The weight at position (i,j) in the 4-neighborhood represents the coarse-grained weight, MLP stands for Multilayer Perceptron, and f i,j δ represents the element at position (i,j) in the enhanced feature. i,j This represents the relative coordinates from the query coordinate q to the neighborhood position (i,j), γ represents the high-dimensional position encoding vector corresponding to the query coordinate q, and [;] represents the concatenation operation; The coarse-grained weights are obtained by the following formula: ; Where ε represents a constant to prevent division by zero.

[0014] Furthermore, in the fine-grained decoding module: Spatial attention enhancement is performed on the enhanced features using the following formula: f' i,j =f i,j×softmax(G ssa (f i,j )); Where f' represents the spatial enhancement feature obtained by spatial attention enhancement, f' i,j f represents the element at position (i,j) in the spatial enhancement feature. i,j G represents the element at position (i,j) in the enhanced feature. ssa This indicates the spatial selection attention operation, and softmax indicates the softmax operation. Fine-grained features are obtained using the following formula; ; Among them, I fine (q) represents a coarse-grained feature, where q represents the query coordinate in the query matrix, and N4 represents the 4-neighbor element of the enhanced feature corresponding to the query coordinate q. The δ represents the fine-grained weight at position (i,j) in the 4-neighborhood. MLP stands for Multilayer Perceptron. i,j This represents the relative coordinates from the query coordinate q to the neighborhood position (i,j), γ represents the high-dimensional position encoding vector corresponding to the query coordinate q, and [;] represents the concatenation operation; Fine-grained weights are obtained using the following formula: ; Among them, S i,j S represents the area of ​​the diagonal region formed by the query coordinate q and the neighboring position (i,j). k.l This represents the area of ​​the 4-neighborhood of position (k,l) in the area of ​​the diagonal region.

[0015] An image super-resolution method based on wavelet-gated attention includes: S1: Obtain a dataset including low-resolution images and their corresponding high-resolution images, and preprocess the dataset to obtain the training set; S2: Construct an image super-resolution system based on wavelet-gated attention as provided in this invention; S3: Use the training set obtained in step S1 to train the image super-resolution system constructed in step S2 to obtain the image super-resolution model; S4: Input the low-resolution image to be processed into the super-resolution model of the image obtained in step S3 to obtain the reconstructed image.

[0016] Furthermore, step S3 includes a three-stage training process: in the first stage of training, the constructed image super-resolution system is trained using a fixed magnification factor to obtain a first reconstruction model; in the second stage of training, values ​​are randomly selected within the first magnification factor range, and the first reconstruction model is trained using the randomly selected magnification factor to obtain a second reconstruction model; in the third stage of training, values ​​are randomly selected within the second magnification factor range, and the second reconstruction model is trained using the randomly selected magnification factor to obtain an image super-resolution model; the second magnification factor range is greater than the first magnification factor range.

[0017] Furthermore, in step S3, the following loss function is used for training: L=λ1L pixel +λ2L per +λ3L freq ; Where L represents the loss function for training the image super-resolution model, L pixel L represents pixel loss. per L represents perceived loss. freq Let λ1, λ2, and λ3 represent the frequency domain loss, and let λ1, λ2, and λ3 represent the loss weights. The frequency domain loss is: ; Where s represents the frequency domain subband, LL, LH, HL, and HH represent the low-frequency subband, the horizontal high-frequency subband, the vertical high-frequency subband, and the diagonal high-frequency subband, respectively, DWT represents wavelet transform, and I SR and I GT These represent the reconstructed image output by the image super-resolution model and the corresponding true high-resolution image, respectively.

[0018] Compared with the prior art, the present invention can achieve the following beneficial effects: This invention creates an image super-resolution system and method based on wavelet-gated attention. Through a symmetric multi-branch lightweight encoder and depthwise separable convolution, it significantly reduces the number of network parameters, making it suitable for deployment on mobile and embedded devices. By using wavelet transform to decompose features into low-resolution sub-bands for self-attention calculation, the computational complexity is reduced from O(n^2) to O(n^2). 2 Reduced to O(n) 2 / 4), achieving a 4-fold reduction in complexity and significantly reducing computational overhead; the gating mechanism can adaptively enhance important features and suppress redundant features, and adopt differentiated processing strategies for low-frequency and high-frequency subbands. Low-frequency subbands use global pooling to preserve the structure, while high-frequency subbands use local max pooling to highlight details, thereby improving feature utilization efficiency; the multi-granularity decoding strategy takes into account both global structure and local details, and achieves adaptive fusion through learnable granularity balance coefficients, thereby improving reconstruction results. Attached Figure Description

[0019] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 A schematic diagram of the overall image super-resolution system based on wavelet-gated attention as described in the embodiments of the present invention; Figure 2 A schematic diagram of the feature processing process in the multi-scale lightweight encoder described in the embodiment of the present invention; Figure 3 A schematic diagram illustrating the feature processing of the frequency-domain gated attention module as described in an embodiment of the present invention; Figure 4 A schematic diagram illustrating the feature processing of the gating attention enhancement operation described in the embodiments of the present invention; Figure 5 A schematic diagram illustrating the feature processing of the multi-granularity lightweight decoder as described in an embodiment of the present invention; Figure 6 This is a schematic flowchart of the image super-resolution method based on wavelet-gated attention as described in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0021] In the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0022] The invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0023] like Figure 1As shown in the embodiment of the present invention, the image super-resolution system based on wavelet-gated attention includes a multi-scale lightweight encoder, a frequency-domain gated attention module, and a multi-granularity lightweight decoder. The multi-scale lightweight encoder performs multi-scale encoding on the input low-resolution image, and then performs attention enhancement on the obtained multi-scale features to obtain encoded features. The frequency-domain gated attention module performs frequency-domain decomposition processing on the encoded features, performs gated attention enhancement operations on the multiple frequency-domain sub-bands obtained, and then concatenates the enhanced frequency-domain sub-bands and converts them to the spatial domain to obtain enhanced features. The multi-granularity lightweight decoder performs positional encoding on the enhanced features, performs coarse-grained decoding and fine-grained decoding on the encoded enhanced features, and then performs weighted fusion of the two decoded features to obtain the reconstructed image.

[0024] In some embodiments, the feature processing in the multi-scale lightweight encoder includes: extracting shallow features from a low-resolution image; performing depth-separable convolution operations at different scales on the shallow features; concatenating the feature channels at different scales and performing channel attention operations; and then connecting the processed features with the shallow feature residuals to obtain the encoded features.

[0025] In this embodiment of the invention, the feature processing procedure in the multi-scale lightweight encoder is as follows: Figure 2 As shown, the multi-scale lightweight encoder first uses a convolutional layer with a kernel size of 3×3 to process the input low-resolution RGB image I. LR The image is processed by mapping it from the RGB space to a high-dimensional feature space, resulting in shallow features F0. The formula is as follows: F0=Conv 3×3 (I LR ); Among them, Conv 3×3 This represents a convolution operation with a kernel size of 3×3, a stride of 1, padding of 1, and C output channels. Then, three parallel branches are used to extract multi-scale features from the shallow feature F0. In the first branch, a depthwise separable convolutional layer with a kernel size of 3×3 is used to convolve the shallow feature F0. After batch normalization and ReLU activation, the processed features are then subjected to a 1×1 convolution to obtain features at a scale of 3×3. The feature processing in the second and third branches is the same, except that the kernel size of the depthwise separable convolutional layer in the second branch is 5×5, and the kernel size of the depthwise separable convolutional layer in the third branch is 7×7. The above process is as follows: F1=Conv 1×1 (BN(ReLU(DWConv 3×3 (F0)))); F2=Conv 1×1 (BN(ReLU(DWConv 5×5 (F0)))); F3=Conv 1×1 (BN(ReLU(DWConv 7×7 (F0)))); Where F1, F2, and F3 represent the output features of the first, second, and third branches, respectively, BN represents batch normalization, and DWConv 3×3 DWConv 5×5 and DWConv 7×7 These represent depth-separable convolutional layers with kernels of 3×3, 5×5, and 7×7, respectively.

[0026] The output features of the three branches are concatenated via channels, and then enhanced using the SE channel attention operation. Finally, the enhanced features are concatenated with the shallow feature F0 residual to obtain the encoded feature F. code The process is as follows: F code =F0+Conv 1×1 (SE([F1;F2;F3])); In this context, [;] indicates a channel splicing operation.

[0027] In this embodiment of the invention, the SE channel attention operation specifically includes: The concatenated features are subjected to global average pooling, and then the channel weights z are calculated through two fully connected layers, as shown in the following formula: z=σ(W2×ReLU(W1×GAP([F1;F2;F3]))); Where GAP represents global average pooling, W1 and W2 represent the weights of the two fully connected layers respectively, and σ represents the sigmoid activation operation; Finally, the channel weight z is appended to the feature of the channel concatenation operation to obtain the SE channel attention-enhanced feature, as shown in the following formula: SE([F1;F2;F3])=z×[F1;F2;F3].

[0028] In some embodiments, the feature processing of the frequency domain gating attention module includes: performing discrete wavelet transform on the encoded features to obtain a low-frequency sub-band, a horizontal high-frequency sub-band, a vertical high-frequency sub-band, and a diagonal high-frequency sub-band; performing gating attention enhancement operations on the four frequency domain sub-bands respectively; and performing inverse discrete wavelet transform on the four enhanced frequency domain sub-bands to obtain enhanced features.

[0029] In this embodiment of the invention, the feature processing procedure of the frequency domain gated attention module is as follows: Figure 3 As shown, where: for the encoded feature F enPerforming Discrete Wavelet Transform (DWT) yields the low-frequency subband LL, the horizontal high-frequency subband LH, the vertical high-frequency subband HL, and the diagonal high-frequency subband HH, as shown in the following equation: DWT(F code = [LL,HL,LH,HH]; The low-frequency subband LL contains the main structural information and global semantics of the image; the horizontal high-frequency subband LH contains horizontal edge information; the vertical high-frequency subband HL contains vertical edge information; and the diagonal high-frequency subband HH contains diagonal texture information. After wavelet decomposition, the spatial size of each subband is reduced to 1 / 2 × 1 / 2 of the original feature size. Therefore, the computational complexity of self-attention is reduced from O(n^2) to O(n^2). 2 Reduced to O(n) 2 / 4). Gated attention enhancement operations are performed on the four obtained frequency domain sub-bands respectively, resulting in four enhanced frequency domain sub-bands LL', LH', HL', and HH'. The enhanced four frequency domain sub-bands LL', LH', HL', and HH' are then subjected to inverse discrete wavelet transform (IDWT) to obtain the enhanced feature F. en As shown in the following formula: F en =IDWT(LL',LH',HL',HH').

[0030] In some embodiments, the feature processing of the gated attention enhancement operation is as follows: Figure 4 As shown, this includes: performing a linear mapping on each frequency domain sub-band to obtain the corresponding query matrix, key matrix, and value matrix, as shown in the following equation: Q = s × W Q K=s×W K V=s×W V ; Where Q, K, and V represent the query matrix, key matrix, and value matrix, respectively, and W Q W K and W V These represent the corresponding linear mapping weights; The self-attention weight matrix is ​​obtained by performing the following attention calculation on the query matrix, key matrix, and value matrix: ; Where A represents the self-attention weight matrix, softmax represents the softmax operation, and d represents the scaling factor; A pooling operation is performed on the self-attention weight matrix, followed by continuous multilayer perceptron operations and sigmoid activation operations on the pooled features. This completes the gating operation on the self-attention weight matrix, enabling selective enhancement of the self-attention weight matrix and obtaining an enhanced self-attention weight matrix. In this embodiment of the invention, the self-attention weight matrix A corresponding to the low-frequency subband LL is... LL Global average pooling is employed, followed by continuous multilayer perceptron and sigmoid activation operations on the pooled features to obtain the enhanced self-attention weight matrix g corresponding to the low-frequency subband LL. LL As shown in the following formula: g LL =σ(MLP(GAP(A LL ))); For the self-attention weight matrix A corresponding to the high-frequency subbands LH, HL, and HH LH A HL and A HH Local max pooling is employed, followed by multilayer perceptron operations and sigmoid activation operations on the pooled features to obtain the enhanced self-attention weight matrix g corresponding to the high-frequency subbands LH, HL, and HH. LH g HL and g HH As shown in the following formula: {g LH ,g HL ,g HH}=σ(MLP(LMP({A LL A HL A HH}))); LMP stands for Local Maximum Pooling.

[0031] Multiplying the corresponding elements of the enhanced self-attention weight matrix with the self-attention weight matrix yields the enhanced frequency domain subband, as shown in the following equation: s' = g × A; Where s' represents the enhanced frequency domain subband, and g represents the enhanced self-attention weight matrix.

[0032] In some embodiments, the multi-granularity lightweight decoder includes a position encoding module, a coarse-grained decoding module, a fine-grained decoding module, and a decoding and reassembly module. The feature processing procedure of the multi-granularity lightweight decoder is as follows: Figure 5 As shown.

[0033] In the location encoding module, the pixel positions in the target high-resolution image are normalized to obtain the query coordinates; then, the query coordinates are subjected to sine and cosine location encoding to generate a high-dimensional location encoding vector. In this embodiment of the invention, the high-dimensional coordinates are obtained by the following formula: γ(q)=[sin(20 πq),cos(2 0 πq),sin(2 1 πq),cos(2 1 πq),...,sin(2 L-1 πq),cos(2 L-1 πq)]; Where γ represents the query coordinate q=(x q ,y q The corresponding high-dimensional position encoding vector, where L represents the encoding number, is used in this embodiment of the invention. The query coordinate q = (x q ,y q Specifically, it is obtained from the following formula: ; Where (i,j) represents the pixel position in the target high-resolution image, and Hh and Wh represent the height and width of the target high-resolution image, respectively. Furthermore, the encoding count L is specifically set to 10.

[0034] The coarse-grained decoding module uses the high-dimensional position encoding vector corresponding to the query coordinates to perform weighted prediction of the enhanced features on the 4-neighborhood features based on distance weights, thus obtaining coarse-grained features. In some embodiments, the coarse-grained decoding module obtains the coarse-grained features using the following formula; ; Among them, I coarse (q) represents coarse-grained features, MLP represents multilayer perceptron, and f i,j δ represents the element at position (i,j) in the enhanced feature. i,j This indicates the relative coordinates from coordinate q to the neighboring position (i,j). The coarse-grained weight corresponding to position (i,j) in the 4-neighborhood is obtained by the following formula: ; Wherein, ε represents a constant to prevent division by zero, specifically 10 in this embodiment of the invention. -6 , This represents the Euclidean distance; in this embodiment of the invention, ε = 10. -6 .

[0035] The fine-grained decoding module performs spatial attention enhancement on the enhanced features, and then, based on the high-dimensional position encoding vector corresponding to the query coordinates, performs weighted prediction on the 4-neighborhood features of the spatially attention-enhanced features according to distance weights to obtain the fine-grained features. In some embodiments, the fine-grained decoding module performs spatial attention enhancement on the enhanced features using the following formula: f' i,j =f i,j ×softmax(G ssa(f i,j )); Where f' represents the spatial enhancement feature obtained by spatial attention enhancement, f' i,j G represents the element at position (i,j) in the spatial enhancement feature. ssa This represents the Spatial Selection Attention (SSA) operation, which is used to generate importance weights for corresponding spatial locations based on input features. Fine-grained features are obtained using the following formula; ; Among them, I fine (q) represents the coarse-grained characteristic. The fine-grained weight corresponding to position (i,j) in the 4-neighborhood is obtained by the following formula: ; Among them, S i,j S represents the area of ​​the diagonal region formed by the query coordinate q and the neighboring position (i,j). k.l This represents the area of ​​the 4-neighborhood of position (k,l) in the area of ​​the diagonal region.

[0036] The decoding and reconstruction module performs a weighted summation of the coarse-grained and fine-grained features to obtain the reconstructed features, as shown in the following formula in this embodiment of the invention: I SR =α×I coarse +(1-α)×I fine ; Among them, I SR Let α represent the reconstructed image, and α represent the summation weights.

[0037] This invention also provides an image super-resolution method based on wavelet-gated attention, combined with... Figure 1 and Figure 6 ,include: S1: Obtain a dataset including low-resolution images and their corresponding high-resolution images, and preprocess the dataset to obtain the training set.

[0038] In this embodiment of the invention, the dataset includes at least one of the publicly available high-resolution reconstruction datasets DIV2K, Flickr2K, BSD100, Urban100, Manga109, and Set5 and Set14. High-resolution images in the dataset are used as real image samples, and low-resolution images are generated by downsampling.

[0039] Furthermore, the preprocessing process performed in this embodiment of the invention includes: cropping the high-resolution image to generate image blocks of fixed size, wherein the size of the low-resolution image block is 48×48 and the size of the high-resolution image block is 48r×48r, where r is the magnification factor. Specifically, in the first stage, r=2, 4; in the second stage, r∈(1,4); in the third stage, r∈(1,8); randomly flipping, randomly rotating, and randomly mirroring the image blocks; and using bicubic interpolation downsampling to generate corresponding low-resolution images from the high-resolution images.

[0040] S2: Construct an image super-resolution system based on wavelet-gated attention as provided in this invention.

[0041] S3: Use the training set obtained in step S1 to train the image super-resolution system constructed in step S2 to obtain the image super-resolution model.

[0042] In some embodiments, step S3 includes a three-stage training process: In the first stage of training, the constructed image super-resolution system is trained using a fixed magnification factor to obtain the first reconstruction model. In this embodiment of the invention, the fixed magnification factor r = 2, 4, and the learning rate is 1 × 10⁻⁴. -4 ; In the second stage of training, values ​​are randomly selected from the first magnification range, and the first reconstruction model is fine-tuned using these randomly selected magnification values ​​to obtain the second reconstruction model. In this embodiment of the invention, the first magnification range is (1, 4), and the learning rate at this time is 1 × 10⁻⁶. -5 ; In the third stage of training, values ​​are randomly selected from the range of the second magnification factor, and the randomly selected magnification factor is used to generalize the second reconstruction model to obtain the image super-resolution model. The second magnification range is greater than the first magnification range. In this embodiment of the invention, the second magnification range is (1, 8), and the learning rate at this time is 1 × 10⁻⁶. -5 .

[0043] In some embodiments, the following loss function is used for training in step S3: L=λ1L pixel +λ2L per +λ3L freq ; Where L represents the loss function for training the image super-resolution model, L pixel L represents pixel loss. per L represents perceived loss. freq λ1, λ2, and λ3 represent the frequency domain loss, and in this embodiment of the invention, they are taken as 1.0, 0.1, and 0.5, respectively. The frequency domain loss is: ; Where s represents the frequency domain subband, I SR and I GT These represent the reconstructed image output by the image super-resolution model and the corresponding true high-resolution image, respectively.

[0044] In this embodiment of the invention, the pixel loss L pixel Using L1 loss, the perceived loss L per The feature loss of the VGG19 network is used, and the perceptual loss is L. per As shown in the following formula: ; in, This represents the feature extraction operation of the first layer in the VGG19 network. In this embodiment of the invention, the third and fourth layers of the VGG19 network are specifically used.

[0045] In this embodiment of the invention, the hyperparameters during training include: a batch size of 32, an optimizer named Adam, β1=0.9, β2=0.999 in the optimizer, and an initial learning rate of 1×10⁻⁶. -4 .

[0046] S4: Input the low-resolution image to be processed into the super-resolution model of the image obtained in step S3 to obtain the reconstructed image.

[0047] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0048] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An image super-resolution system based on wavelet-gated attention, characterized in that, include: A multi-scale lightweight encoder encodes the input low-resolution image at multiple scales, and then enhances the attention of the obtained multi-scale features to obtain the encoded features. The frequency domain gated attention module performs frequency domain decomposition on the encoded features, performs gated attention enhancement operations on the multiple frequency domain sub-bands obtained, and then concatenates the enhanced frequency domain sub-bands and transforms them into the spatial domain to obtain the enhanced features. The multi-granularity lightweight decoder performs positional encoding on the enhanced features, performs coarse-grained decoding and fine-grained decoding on the encoded enhanced features, and then performs weighted fusion of the two decoded features to obtain the reconstructed image.

2. The image super-resolution system based on wavelet-gated attention according to claim 1, characterized in that, In a multi-scale lightweight encoder: shallow features are extracted from low-resolution images; shallow features are subjected to depth-separable convolution operations at different scales; the feature channels obtained at different scales are concatenated and then channel attention is performed; the processed features are then concatenated with the shallow feature residuals to obtain the encoded features.

3. The image super-resolution system based on wavelet-gated attention according to claim 1, characterized in that, In the frequency domain gated attention module: Discrete wavelet transform is performed on the encoded features to obtain low-frequency sub-bands, horizontal high-frequency sub-bands, vertical high-frequency sub-bands, and diagonal high-frequency sub-bands; gated attention enhancement operations are performed on the four frequency domain sub-bands respectively; and inverse discrete wavelet transform is performed on the four enhanced frequency domain sub-bands to obtain enhanced features.

4. The image super-resolution system based on wavelet-gated attention according to claim 1, characterized in that, In gated attention enhancement operations: For each frequency domain subband, a linear mapping is performed to obtain the corresponding query matrix, key matrix, and value matrix; The self-attention weight matrix is ​​obtained by performing the following attention calculation on the query matrix, key matrix, and value matrix: ； Where A represents the self-attention weight matrix, softmax represents the softmax operation, Q, K and V represent the query matrix, key matrix and value matrix respectively, and d represents the scaling factor; The self-attention weight matrix is ​​pooled, and then the pooled features are continuously subjected to multilayer perceptron operations and sigmoid activation operations to complete the gating operation of the self-attention weight matrix, thus obtaining the enhanced self-attention weight matrix. Multiplying the corresponding elements of the enhanced self-attention weight matrix with the self-attention weight matrix yields the enhanced frequency domain subband.

5. The image super-resolution system based on wavelet-gated attention according to claim 1, characterized in that, The multi-granularity lightweight decoder includes a position encoding module, a coarse-grained decoding module, a fine-grained decoding module, and a decoding and reassembly module; In the position encoding module, the pixel positions in the target high-resolution image are normalized to obtain the query coordinates; then the query coordinates are subjected to sine and cosine position encoding to generate a high-dimensional position encoding vector. In the coarse-grained decoding module, based on the high-dimensional position encoding vector corresponding to the query coordinates, the enhanced features are weighted and predicted on the 4-neighborhood features according to the distance weight to obtain the coarse-grained features; In the fine-grained decoding module, spatial attention enhancement is performed on the enhanced features. Then, based on the high-dimensional position encoding vector corresponding to the query coordinates, the spatial attention-enhanced features are weighted and predicted on the 4-neighbor features according to the distance weight to obtain the fine-grained features. In the decoding and reconstruction module, the coarse-grained features and fine-grained features are weighted and summed to obtain the reconstructed features.

6. The image super-resolution system based on wavelet-gated attention according to claim 5, characterized in that, The coarse-grained features are obtained in the coarse-grained decoding module using the following formula; ； Among them, I coarse (q) represents a coarse-grained feature, where q represents the query coordinate in the query matrix, and N4 represents the 4-neighbor element of the enhanced feature corresponding to the query coordinate q. The weight at position (i,j) in the 4-neighborhood represents the coarse-grained weight, MLP stands for Multilayer Perceptron, and f i,j δ represents the element at position (i,j) in the enhanced feature. i,j This represents the relative coordinates from the query coordinate q to the neighborhood position (i,j), γ represents the high-dimensional position encoding vector corresponding to the query coordinate q, and [;] represents the concatenation operation; The coarse-grained weights are obtained using the following formula: ； Where ε represents a constant to prevent division by zero.

7. The image super-resolution system based on wavelet-gated attention according to claim 5, characterized in that, In the fine-grained decoding module: Spatial attention enhancement is performed on the enhanced features using the following formula: f’ i,j =f i,j ×softmax(G ssa (f i,j )); Where f' represents the spatial enhancement feature obtained by spatial attention enhancement, f' i,j f represents the element at position (i,j) in the spatial enhancement feature. i,j G represents the element at position (i,j) in the enhanced feature. ssa This indicates the spatial selection attention operation, and softmax indicates the softmax operation. Fine-grained features are obtained using the following formula; ； Among them, I fine (q) represents a coarse-grained feature, where q represents the query coordinate in the query matrix, and N4 represents the 4-neighbor element of the enhanced feature corresponding to the query coordinate q. This represents the fine-grained weight at position (i,j) in the 4-neighborhood elements. MLP stands for Multilayer Perceptron, and δ i,j This represents the relative coordinates from the query coordinate q to the neighborhood position (i,j), γ represents the high-dimensional position encoding vector corresponding to the query coordinate q, and [;] represents the concatenation operation; Fine-grained weights are obtained using the following formula: ； Among them, S i,j S represents the area of ​​the diagonal region formed by the query coordinate q and the neighboring position (i,j). k.l This represents the area of ​​the 4-neighborhood of position (k,l) in the area of ​​the diagonal region.

8. An image super-resolution method based on wavelet-gated attention, characterized in that, include: S1: Obtain a dataset including low-resolution images and their corresponding high-resolution images, and preprocess the dataset to obtain the training set; S2: Construct an image super-resolution system based on wavelet-gated attention as described in any one of claims 1 to 7; S3: Use the training set obtained in step S1 to train the image super-resolution system constructed in step S2 to obtain the image super-resolution model; S4: Input the low-resolution image to be processed into the super-resolution model of the image obtained in step S3 to obtain the reconstructed image.

9. The image super-resolution method based on wavelet-gated attention according to claim 8, characterized in that, Step S3 includes a three-stage training process: In the first stage of training, the constructed image super-resolution system is trained using a fixed magnification factor to obtain a first reconstruction model; in the second stage of training, values ​​are randomly selected within the first magnification factor range, and the first reconstruction model is trained using the randomly selected magnification factor to obtain a second reconstruction model; in the third stage of training, values ​​are randomly selected within the second magnification factor range, and the second reconstruction model is trained using the randomly selected magnification factor to obtain an image super-resolution model; the second magnification factor range is greater than the first magnification factor range.

10. The image super-resolution method based on wavelet-gated attention according to claim 8, characterized in that, In step S3, the following loss function is used for training: L=λ1L pixel +λ2L per +λ3L freq ; Where L represents the loss function for training the image super-resolution model, L pixel L represents pixel loss. per L represents perceived loss. freq Let λ1, λ2, and λ3 represent the frequency domain loss, and let λ1, λ2, and λ3 represent the loss weights. The frequency domain loss is: ； Where s represents the frequency domain subband, LL, LH, HL, and HH represent the low-frequency subband, the horizontal high-frequency subband, the vertical high-frequency subband, and the diagonal high-frequency subband, respectively, DWT represents wavelet transform, and I SR and I GT These represent the reconstructed image output by the image super-resolution model and the corresponding true high-resolution image, respectively.

Images