Hybrid attention and large kernel convolution based lightweight super-resolution method, medium and equipment

Abstract

The application discloses a hybrid attention and large kernel convolution-based lightweight super-resolution method, medium and equipment, and relates to the technical field of image processing. The method comprises the following steps: acquiring a low-resolution image and extracting shallow features; inputting the shallow features into multiple cascaded built-in feature enhancement modules; performing window multi-head self-attention, multi-scale channel attention and global self-attention processing through a hybrid attention module to obtain global enhanced features; performing channel division through a large kernel convolution detail enhancement module, performing partial convolution on the first part of channels, performing large kernel partial convolution on the second part of channels, and retaining transmission for the remaining channels; and obtaining enhanced reconstruction features through point-by-point convolution fusion; and densely fusing multiple enhanced reconstruction features, generating a high-resolution image through convolution and pixel rearrangement. The application enhances the continuity of window structure and cross-scale context expression, improves the image definition and consistency, reduces the large kernel convolution overhead of the full channel, and meets the requirements of reconstruction accuracy and lightweight deployment.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a lightweight super-resolution method, medium, and device based on hybrid attention and large kernel convolution. Background Technology

[0002] Image super-resolution reconstruction refers to image processing techniques that recover corresponding high-resolution images from low-resolution images. It can improve the recognizability of edges, textures, structural contours, and details in images, and has application value in scenarios such as mobile terminal imaging, video enhancement, intelligent visual perception, industrial inspection, remote sensing image processing, and medical image-assisted observation. With the development of deep learning technology, neural network-based image super-resolution methods can improve the clarity and structural consistency of reconstructed images by learning the mapping relationship between low-resolution and high-resolution images.

[0003] Existing image super-resolution models often improve image reconstruction accuracy by increasing the number of network layers, channels, complex attention structures, receptive fields, and window sizes. However, these methods increase the number of model parameters and computational complexity, making them difficult to deploy on mobile devices, embedded systems, and other resource-constrained vision processing platforms. To reduce model complexity, existing lightweight image super-resolution methods typically reduce network depth, decrease the number of channels, and use grouped or partial convolutions to compress model overhead. While these methods reduce computation, they can also weaken the model's feature representation capabilities.

[0004] Specifically, lightweight image super-resolution models, after reducing the number of parameters, often suffer from insufficient global structure modeling capabilities. For images containing long edges, repetitive textures, cross-regional contours, or large-scale structural continuity, relying solely on local windows or local convolutions is insufficient to fully establish the relationships between distant pixels, easily leading to insufficient structural continuity, unstable edge extension, or inconsistent texture orientation in the reconstructed image.

[0005] Meanwhile, existing lightweight models also have shortcomings in local detail restoration. To reduce computational cost, some models oversimplify the local feature extraction process, reducing their ability to represent high-frequency textures, fine edges, and local abrupt changes. On the other hand, directly applying large kernel convolutions or complex convolutional enhancement structures to all channels significantly increases computational cost and parameter count, hindering lightweight deployment. Therefore, how to improve the ability to restore local textures and edge details while maintaining low model complexity is a problem that lightweight image super-resolution methods need to solve.

[0006] Furthermore, in existing methods, global structure enhancement and local detail compensation are typically treated as relatively independent processing steps. Attention modules are used to enhance global or contextual features, while convolutional modules are used to recover local details, but there is a lack of clear technical transfer between the two. In other words, existing methods often fail to determine whether subsequent channels require ordinary partial convolutional compensation, large-kernel partial convolutional compensation, or lightweight preservation processing based on the state of the preceding global structure representation. This results in some channels undergoing redundant computation, while some channels requiring detail compensation may not be adequately enhanced. Summary of the Invention

[0007] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a lightweight super-resolution method, medium and device based on hybrid attention and large kernel convolution to solve the above technical problems.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a lightweight super-resolution method based on hybrid attention and large kernel convolution, comprising: Acquire a low-resolution image, perform shallow feature extraction on the low-resolution image, and obtain a shallow feature map; The shallow feature map is input into multiple cascaded built-in feature enhancement modules, which include a hybrid attention module and a large kernel convolution detail enhancement module. Globally enhanced features are obtained by performing window multi-head self-attention processing, multi-scale channel attention processing, and global self-attention processing on the input features through a hybrid attention module. The global enhancement features are divided into channels by a large kernel convolution detail enhancement module. Partial convolution processing is performed on the first part of the channels after division, large kernel partial convolution processing is performed on the second part of the channels after division, and the remaining channels are preserved and passed on. Cross-channel fusion is performed by point-by-point convolution to obtain the enhanced reconstruction features. Dense feature fusion is performed on the enhanced reconstruction features output by multiple built-in feature enhancement modules to obtain fused features; The fused features are convolutionally processed and pixel rearranged to generate a high-resolution image.

[0009] The present invention is further configured to perform convolution processing on the low-resolution image through a 3×3 convolutional layer to obtain a shallow feature map; The enhanced reconstruction features output by multiple built-in feature enhancement modules are spliced ​​along the channel dimension to obtain the spliced ​​features. Channel compression is performed on the spliced ​​features through a 1×1 convolutional layer, and spatial neighborhood fusion is performed on the channel-compressed features through a 3×3 convolutional layer. The spatially fused features are then residually fused with the shallow feature map to obtain the fused features. The fused features are mapped to upsampled features through convolutional layers, and the resolution of the upsampled features is amplified through pixel rearrangement layers to generate a high-resolution image.

[0010] The present invention is further configured such that the hybrid attention module includes at least two hybrid attention processing layers, each hybrid attention processing layer including a window multi-head self-attention unit, a multi-scale channel attention unit and a global self-attention unit; The multi-head self-attention unit in a window is used to establish inter-pixel relationships within a local window to obtain the window attention response; The multi-scale channel attention unit is used to generate channel attention weights based on pooling responses of different scales. The channel attention weights are then used to enhance the window attention response to obtain a multi-scale channel response. Global self-attention units are used to enhance the global relationships of multi-scale channel responses to obtain global self-attention responses. Globally enhanced features are generated based on window attention response, multi-scale channel response, and global self-attention response.

[0011] The present invention is further configured such that the multi-scale channel attention unit includes a max pooling path and an average pooling path; The max pooling path is used to perform max pooling on the input features at scales of 3×3, 2×2 and 1×1 respectively, to obtain multi-scale max pooling features; The average pooling path is used to perform average pooling processing on the input features at scales of 3×3, 2×2 and 1×1 respectively, to obtain multi-scale average pooling features; The multi-scale max pooling features and multi-scale average pooling features are compressed by 1×1 convolution and expanded by 1×1 convolution to obtain the weights of the two channels. The weights of the two channels are concatenated and merged, and the merged channel weights are obtained through Sigmoid activation. Multiplying the fused channel weights by the window attention response yields the multi-scale channel response.

[0012] The present invention is further configured to include: The global enhanced features obtained by fusing window attention response, multi-scale channel response and global self-attention response are defined as the global semantic extension map. The global semantic extension map is used to characterize the window-outside structural continuity relationship, cross-scale contextual relationship and global semantic response state of the input features. The window attention response, multi-scale channel response, and global self-attention response under the same channel are normalized respectively. Based on the consistency among the normalized window attention response, multi-scale channel response, and global self-attention response, the global extension closure of the channel is determined. The global extension closure is used to characterize the completeness of the global structural expression of the corresponding channel in the global semantic extension graph.

[0013] The present invention is further configured to determine the local texture residual state of the channel based on the high-frequency response, gradient response, edge response, or the difference between the original feature and the smooth feature of the corresponding channel in the global semantic extension map. Based on the global extensional closure and local texture residual state, the channel sparse use compensation amount is determined. The channel sparse use compensation amount is used to determine whether the corresponding channel needs to enter the convolution compensation path, and whether it needs to enter the normal part convolution branch or the large kernel part convolution branch. When the global extension closure is greater than or equal to the preset closure threshold and the local texture residual state is less than the preset residual threshold, the corresponding channel is determined as a lightweight retention candidate channel. When the global extension closure is greater than or equal to the preset closure threshold, and the local texture residual state is greater than or equal to the preset residual threshold, the corresponding channel is determined as a candidate channel for normal partial convolution compensation. When the global extension closure is less than the preset closure threshold and the local texture residual state is greater than or equal to the preset residual threshold, the corresponding channel is determined as a candidate channel for large kernel part convolution compensation. When the global extension closure is less than the preset closure threshold and the local texture residue is less than the preset residue threshold, the corresponding channel is determined as a lightweight retention candidate channel or a normal partial convolution compensation candidate channel according to the preset channel sparse use rule.

[0014] The present invention is further configured to generate a channel sparing mark map based on the channel sparing compensation amount and the preset channel sparing rules. The channel sparing mark map is used to mark the channels in the global enhancement features as lightweight reserved channels, ordinary partial convolution channels or large kernel partial convolution channels, wherein the lightweight reserved channels, ordinary partial convolution channels and large kernel partial convolution channels are non-overlapping channel sets. The preset channel sparse use rules include: the number of channels participating in normal partial convolution processing and large kernel partial convolution processing shall not exceed the preset ratio, and the remaining channels shall be used as lightweight reserved channels to participate in subsequent stitching and pointwise convolution fusion. Based on the channel sparse use labeling map, partial convolution processing is performed on the normal partial convolution channels, 5×5 large kernel partial convolution processing is performed on the large kernel partial convolution channels, and lightweight reserved channels are retained and passed. The results of partial convolution processing, the results of large kernel partial convolution processing, and lightweight preserved channels are concatenated and then fused across channels through 1×1 pointwise convolution to obtain enhanced reconstruction features.

[0015] The present invention is further configured to include a model training phase; During the model training phase, high-resolution images and low-resolution images obtained by downsampling high-resolution images are used to form training sample pairs. Data augmentation is performed on the training sample pairs, including at least one of random horizontal flipping, random vertical flipping, random rotation, random translation, and random cropping; Image patches of a preset size are cropped from low-resolution images and used as training input, while the corresponding high-resolution image patches are used as supervised output. The Adam optimizer was used to train a lightweight super-resolution model consisting of a shallow feature extraction module, multiple cascaded built-in feature enhancement modules, a dense feature fusion module, and a pixel rearrangement layer. The training parameters during the model training phase included: a training patch size of 64×64, 16 image patches per training batch, Adam optimizer parameters β1 of 0.9 and β2 of 0.999, an initial learning rate of 5×10^-4, a total of 1200 training epochs, with the learning rate reduced to half of the previous stage every 200 epochs, an 8×8 window size for multi-head self-attention processing, 5 attention heads, 6 built-in feature enhancement modules, and a 5×5 convolutional kernel for the large kernel part.

[0016] The present invention also provides an electronic device, the electronic device comprising: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, enable the electronic device to implement the lightweight super-resolution method based on hybrid attention and large kernel convolution as described above.

[0017] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer's processor, causes the computer to perform the lightweight super-resolution method based on hybrid attention and large kernel convolution as described in any of the preceding claims.

[0018] This invention provides a lightweight super-resolution method, medium, and device based on hybrid attention and large kernel convolution. The method acquires a low-resolution image, extracts shallow features to obtain a shallow feature map, and inputs this shallow feature map into multiple cascaded built-in feature enhancement modules, including a hybrid attention module and a large kernel convolution detail enhancement module. The hybrid attention module performs window multi-head self-attention processing, multi-scale channel attention processing, and global self-attention processing on the input features to obtain global enhanced features. The large kernel convolution detail enhancement module divides the global enhanced features into channels, performs partial convolution processing on the first part of the divided channels, performs large kernel partial convolution processing on the second part of the divided channels, and preserves and propagates the remaining channels. Cross-channel fusion is then performed through pointwise convolution to obtain enhanced reconstructed features. Dense feature fusion is performed on the enhanced reconstructed features output from the multiple built-in feature enhancement modules to obtain fused features. The fused features are then subjected to convolution and pixel rearrangement processing to generate a high-resolution image. The beneficial effects include: 1. By inputting shallow features of low-resolution images into multiple cascaded built-in feature enhancement modules, and sequentially performing multi-head self-attention processing for windows, multi-scale channel attention processing, and global self-attention processing in the hybrid attention module, the model can further obtain cross-scale contextual information and global structural information based on local window associations. This enhances the expressive ability of long-distance dependencies, out-of-window structural continuity, and global semantic response states in low-resolution images. When generating high-resolution images, it can reduce problems such as edge breakage, inconsistent texture orientation, and insufficient structural continuity, improving the overall clarity and global consistency of the super-resolution reconstructed image. 2. A large-kernel convolution detail enhancement module divides the global enhancement features into channels. Only the first segment of channels undergoes partial convolution and large-kernel partial convolution processing, while the remaining channels are preserved and propagated. Then, pointwise convolution is used for cross-channel fusion, allowing different channels to participate in detail enhancement processing of varying intensities according to their feature carrying state. This approach leverages large-kernel partial convolution to enhance the recovery of local textures, edges, and high-frequency details while avoiding the increased computational cost of performing large-kernel convolution on all channels. This reduces model complexity while maintaining reconstruction accuracy, improving the adaptability of the lightweight super-resolution model in mobile terminals, embedded devices, and other resource-constrained scenarios.

[0019] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 A flowchart illustrating a lightweight super-resolution method based on hybrid attention and large kernel convolution, as shown in an exemplary embodiment of the present invention; Figure 2 The training flowchart of a lightweight super-resolution model based on hybrid attention and large kernel convolution is shown as an exemplary embodiment of the present invention. Figure 3 A schematic diagram of the overall network structure of a lightweight super-resolution model based on hybrid attention and large kernel convolution, as shown in an exemplary embodiment of the present invention; Figure 4 A schematic diagram of the structure of a multi-scale channel attention module is shown as an exemplary embodiment of the present invention; Figure 5 A schematic diagram of the structure of a large kernel convolution detail enhancement module is shown as an exemplary embodiment of the present invention; Figure 6 The figure shows an experimental comparison of lightweight super-resolution models based on hybrid attention and large kernel convolution, which is an exemplary embodiment of the present invention. Detailed Implementation

[0021] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0022] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0023] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0024] Lightweight super-resolution methods based on hybrid attention and large kernel convolution, such as Figure 1 As shown, it includes: Acquire a low-resolution image, perform shallow feature extraction on the low-resolution image, and obtain a shallow feature map; Specifically, low-resolution images can be natural images, video frame images, images captured by mobile terminals, industrial inspection images, remote sensing images, medical observation images, or other images requiring super-resolution reconstruction. After the low-resolution image is input into the lightweight super-resolution model, it first enters the shallow feature extraction module. The shallow feature extraction module is used to extract basic edges, brightness variations, texture directions, contour structures, and low-level spatial features from the low-resolution image, providing basic feature input for subsequent global structure modeling and local detail restoration.

[0025] In one specific implementation, the shallow feature extraction module uses a 3×3 convolutional layer. Let the low-resolution image be the low-resolution input image, and the shallow feature extraction process is represented as follows: , or can be expressed as: ,in, This indicates a low-resolution input image; or This indicates a shallow feature extraction operation; the preferred shallow feature extraction operation is a 3×3 convolution operation. or This represents the extracted shallow feature map. For ease of subsequent description, in this embodiment, the shallow feature map is uniformly referred to as... .

[0026] Input the shallow feature map into multiple cascaded built-in feature enhancement modules; like Figure 3 As shown, the lightweight super-resolution model in this embodiment includes a shallow feature extraction module, multiple cascaded built-in feature enhancement modules, a dense feature fusion module, and an upsampling module. The built-in feature enhancement modules are feature enhancement structures set within the network backbone, not additional external modules. These built-in feature enhancement modules include a hybrid attention module and a large-kernel convolutional detail enhancement module. The hybrid attention module enhances the global structural representation capability of the input features, while the large-kernel convolutional detail enhancement module enhances the recovery capability of local texture and edge details while maintaining the model's lightweight design.

[0027] In one embodiment, the built-in feature enhancement module can correspond to Hybrid Attention and High-Efficiency Large Kernel Block (HAELKB); the large kernel convolution detail enhancement module can correspond to High-Efficiency Large Kernel Convolution (ELKConv) module, or FFEM module. In this embodiment, for consistency, they are uniformly referred to as the built-in feature enhancement module, the hybrid attention module, and the large kernel convolution detail enhancement module.

[0028] Let the number of built-in feature enhancement modules be D, and let the output feature of the d-th built-in feature enhancement module be denoted as . Where 1 ≤ d ≤ D. The cascaded processing of multiple built-in feature enhancement modules is represented as follows: ,in, Indicates the first Feature mapping functions of a built-in feature enhancement module; Indicates the first The enhanced reconstructed features output by each built-in feature enhancement module; D represents the number of built-in feature enhancement modules; the value of d ranges from [1, D].

[0029] Through the cascading processing of multiple built-in feature enhancement modules, the model can acquire feature information at different levels, including shallow texture information, mid-level structural information, high-level contextual information, and detailed information compensated by large kernel convolution.

[0030] Globally enhanced features are obtained by performing window multi-head self-attention processing, multi-scale channel attention processing, and global self-attention processing on the input features through a hybrid attention module. Specifically, the hybrid attention module includes at least two hybrid attention processing layers. Each hybrid attention processing layer includes a window multi-head self-attention unit, a multi-scale channel attention unit, and a global self-attention unit. The window multi-head self-attention unit is used to establish inter-pixel relationships within a local window; the multi-scale channel attention unit is used to extract channel context responses at different scales; and the global self-attention unit is used to further enhance global structural relationships.

[0031] Let the input features of the hybrid attention processing layer be... After performing layer normalization on the input features, the window multi-head self-attention unit establishes inter-pixel correlations within a local window, obtaining the window attention response. The window multi-head self-attention processing process is represented as follows: ,in, This represents the input features of the hybrid attention processing layer; Presentation layer normalization processing; This indicates multi-head self-attention of windows or multi-head self-attention processing of shifted windows; This indicates the window's attention response.

[0032] Multi-head self-attention of windows can capture the relationships between pixels within a local window, but relying solely on window attention is easily limited by the window size and cannot fully express the continuity of structures outside the window. Therefore, this invention sets up a multi-scale channel attention unit after multi-head self-attention of windows to introduce contextual responses at different scales.

[0033] like Figure 4 As shown, the multi-scale channel attention unit includes a max-pooling path and an average-pooling path. The max-pooling path performs max-pooling processing on the input features at scales of 3×3, 2×2, and 1×1, respectively, to obtain multi-scale max-pooled features; the average-pooling path performs average-pooling processing on the input features at scales of 3×3, 2×2, and 1×1, respectively, to obtain multi-scale average-pooled features. Through multi-scale pooling processing, different channels can obtain local, intermediate, and global contextual responses.

[0034] In one specific implementation, window attention response Divided into two partition components along the channel dimension: ,in, This indicates a channel partitioning operation; and The two partition components represent the window attention response.

[0035] The channel weights of the max pooling path are represented as follows: The channel weights of the average pooling path are expressed as follows: ,in, This indicates multi-scale max pooling processing; This indicates multi-scale average pooling processing; This represents a 1×1 pointwise convolution; Represents a non-linear activation function; Indicates Sigmoid activation processing; This represents the channel weights obtained from the max pooling path; This represents the channel weights obtained from the average pooling path.

[0036] The weights of the two channels are concatenated and merged, and then applied to the window attention response to obtain the multi-scale channel response: ,in, Indicates a channel splicing operation; This indicates element-wise multiplication or weighted processing of corresponding channels; This indicates the multi-scale channel response.

[0037] Furthermore, the multi-scale channel responses are fused with the residual input to obtain intermediate features: ,in, This indicates multi-scale channel attention processing; Represents the residual input features in the hybrid attention processing layer; This represents the intermediate features after multi-scale channel enhancement.

[0038] Global self-attention units enhance the global relationships of intermediate features, resulting in globally enhanced features: ,in, This indicates global self-attention unit processing; This represents the global augmentation feature output by the hybrid attention processing layer.

[0039] Thus, the window attention response, multi-scale channel response, and global self-attention response together form global enhancement features, enabling the model to simultaneously obtain local window associations, multi-scale contextual information, and global structural relationships.

[0040] The global augmentation features are defined as a global semantic extension map, and the global extension closure is determined. To enable the global structure perception results of the hybrid attention module to further guide subsequent channel detail compensation, this embodiment defines the global enhanced features obtained by fusing the window attention response, multi-scale channel response, and global self-attention response as the global semantic extension map. The global semantic extension map is an intermediate feature object formed within the hybrid attention module, not a newly added independent network module.

[0041] Let the window attention response in the d-th built-in feature enhancement module be... Multi-scale channel response is The global self-attention response is Then the global semantic extension graph is represented as: ,in, This represents the window attention response in the d-th built-in feature enhancement module; This represents the multi-scale channel response in the d-th built-in feature enhancement module; This represents the global self-attention response in the d-th built-in feature enhancement module; Represents feature fusion mapping; This represents the global semantic extension graph in the d-th built-in feature enhancement module.

[0042] Global semantic extension maps are used to characterize the structural continuity relationships of input features outside the window, cross-scale contextual relationships, and global semantic response states. For example, when long edges, repetitive textures, continuous contours, or cross-regional structures exist in low-resolution images, global semantic extension maps can reflect the continuity state of these structures outside the window.

[0043] To further transform the global structural representation state in the global semantic extension graph into parameters that can be used for channel processing and judgment, this embodiment determines the global extension closure value. The global extension closure value is used to characterize the completeness of the global structural representation of the corresponding channel in the global semantic extension graph.

[0044] Specifically, the window attention response, multi-scale channel response, and global self-attention response under the same channel are normalized respectively: ; ; ,in, This indicates a normalization operation; This represents the normalized window attention response; This represents the normalized multi-scale channel response; d represents the normalized global self-attention response; c represents the sequence number of the built-in feature enhancement module; and d represents the channel sequence number.

[0045] Compute the consistency between the window attention response and the multi-scale channel response: ; Calculate the consistency between the multi-scale channel response and the global self-attention response: ; Compute the consistency between window attention response and global self-attention response: ; in, Indicates inner product operation; Represents the L2 norm; It is a stability constant greater than 0, used to avoid the denominator being 0.

[0046] Determine the global extensional closure based on the pairwise consistency among the three types of responses: ,in, This represents the global extensional closure of the c-th channel in the d-th built-in feature enhancement module. The larger the global extensional closure, the more consistent the response of the channel is across window attention, multi-scale channel attention, and global self-attention processing, and the more complete the global structural representation of the channel. The smaller the global extensional closure, the more insufficient the out-of-window structural continuity or cross-scale contextual representation of the channel.

[0047] Determine the local texture residue state, and determine the channel sparse compensation amount based on the global extensional closure amount and the local texture residue state.

[0048] Local texture remnants are used to characterize high-frequency textures, edge responses, local mutations, or differences between original and smooth features that still exist in the corresponding channel of the global semantic extension map. The stronger the local texture remnant, the more the corresponding channel needs detail compensation through convolutional paths.

[0049] Specifically, for the global semantic extension graph The c-th channel Smoothing is performed to obtain smoothed features. And calculate the high-frequency residual response: ,in, This represents the feature of the c-th channel in the global semantic extension graph; This indicates a smoothing operation, which can be average pooling, low-pass filtering, or learnable smoothed convolution. This indicates a high-frequency residual response.

[0050] Further, calculate the edge response: ,in, Represents the transverse gradient operator; Represents the longitudinal gradient operator; This indicates an edge response.

[0051] Determine the local texture persistence state based on high-frequency persistence response and edge response: ,in, This represents element-wise multiplication; Denotes the norm 1; Indicates the state of residual texture in a local area; It is a stability constant greater than 0. If A larger value indicates that there is still strong high-frequency residue and edge response in the corresponding channel; if A smaller value indicates weaker local texture residue in the corresponding channel.

[0052] Based on the global extensional closure and local texture residual state, the channel sparse compensation amount is determined. The channel sparse compensation amount is used to determine whether the corresponding channel needs to enter the convolution compensation path, and whether it needs to enter the normal partial convolution branch or the large kernel partial convolution branch.

[0053] In one embodiment, the channel sparse compensation amount is expressed as: ,in, Indicates a lightweight, retained component; This represents the ordinary part of the convolution compensation component; This represents the convolution compensation component of the large kernel portion.

[0054] Lightweight retention components are represented as: ; The ordinary part of the convolutional compensation component is represented as:

[0055] The large kernel part of the convolutional compensation component is represented as:

[0056] The above calculation logic indicates that when the global extension closure is high and the local texture residue is weak, the lightweight preservation component is high; when the global extension closure is high and the local texture residue is strong, the convolution compensation component of the ordinary part is high; when the global extension closure is low and the local texture residue is strong, the convolution compensation component of the large kernel part is high.

[0057] In another embodiment, a preset closure threshold and a preset residual threshold can also be used to determine candidate channels. Specifically: when the global extensional closure is greater than or equal to the preset closure threshold and the local texture residual state is less than the preset residual threshold, the corresponding channel is determined as a lightweight retention candidate channel; when the global extensional closure is greater than or equal to the preset closure threshold and the local texture residual state is greater than or equal to the preset residual threshold, the corresponding channel is determined as a normal partial convolution compensation candidate channel; when the global extensional closure is less than the preset closure threshold and the local texture residual state is greater than or equal to the preset residual threshold, the corresponding channel is determined as a large kernel partial convolution compensation candidate channel; when the global extensional closure is less than the preset closure threshold and the local texture residual state is less than the preset residual threshold, the corresponding channel is determined as a lightweight retention candidate channel or a normal partial convolution compensation candidate channel according to the preset channel sparse usage rule.

[0058] Based on the channel sparse use compensation amount and the preset channel sparse use rules, a channel sparse use marker map is generated; The channel sparse labeling map is a control object formed within the large kernel convolution detail enhancement module. It is used to label channels in the global enhancement features as lightweight preservation channels, normal partial convolution channels, or large kernel partial convolution channels. Lightweight preservation channels, normal partial convolution channels, and large kernel partial convolution channels are non-overlapping sets of channels.

[0059] In one embodiment, for the c-th channel in the d-th built-in feature enhancement module, the label value of the channel sparse label map is represented as: This indicates that the corresponding channel is a lightweight reserved channel; This indicates that the corresponding channel is a normal part of the convolution channel; This indicates that the corresponding channel is a convolutional channel of the large kernel.

[0060] In the specific generation process, the candidate label type for each channel is determined based on the lightweight reserved component, the normal partial convolution compensation component, and the large kernel partial convolution compensation component in the channel sparse use compensation amount. The final channel sparse use label map is then determined by combining this with preset channel sparse use rules. These preset channel sparse use rules include: the number of channels participating in normal partial convolution processing and large kernel partial convolution processing does not exceed a preset ratio; the remaining channels are used as lightweight reserved channels for subsequent stitching and point-by-point convolution fusion.

[0061] For example, when the number of candidate channels for normal partial convolution compensation exceeds the preset ratio of normal partial convolution channels, channels within the preset ratio can be selected as normal partial convolution channels in descending order of normal partial convolution compensation components, and the remaining channels can be used as lightweight reserved channels; when the number of candidate channels for large kernel partial convolution compensation exceeds the preset ratio of large kernel partial convolution channels, channels within the preset ratio can be selected as large kernel partial convolution channels in descending order of large kernel partial convolution compensation components, and the remaining channels can be used as lightweight reserved channels.

[0062] Therefore, the channel sparse label map can not only reflect the global structure judgment of the previous stage and the local texture residue state, but also meet the implementation requirements of the preset proportion of channels participating in convolution in the lightweight model, avoiding excessive computational overhead caused by performing large kernel convolution on all channels.

[0063] The global enhancement features are divided into channels by a large kernel convolution detail enhancement module. Partial convolution processing is performed on the first part of the divided channels, large kernel partial convolution processing is performed on the second part of the divided channels, and the remaining channels are preserved and passed on. Cross-channel fusion is performed through pointwise convolution to obtain the enhanced reconstruction features.

[0064] like Figure 5 As shown, the large-kernel convolutional detail enhancement module is used to perform local detail compensation on the global enhanced features output by the hybrid attention module. In the original implementation, the input feature map is divided along the channel dimension: ,in, This represents the input features of the large kernel convolution detail enhancement module; and This represents the two branch features obtained by partitioning along the channel dimension; This indicates a channel partitioning operation.

[0065] The first branch performs partial convolution processing: The second branch performs large-kernel partial convolution processing: ,in, This indicates partial convolution processing; This indicates that the large kernel part of the convolution processing is preferred, and a 5×5 large kernel convolution kernel is preferred. This represents the result of partial convolution processing; This represents the result of convolution processing of the large kernel portion.

[0066] The results from the two branches are then concatenated and fused across channels using a 1×1 pointwise convolution. ,in, This represents a 1×1 pointwise convolution; Indicates a channel splicing operation; This represents the output features of the large kernel convolution detail enhancement module.

[0067] In the implementation combining channel-sparsely used labeling maps, the large-kernel convolutional detail enhancement module divides the global enhancement feature or global semantic extension map into ordinary partial convolutional channel features, large-kernel partial convolutional channel features, and lightweight preserved channel features, denoted as follows: , and .

[0068] Perform partial convolution processing on ordinary partial convolution channels: Perform large kernel partial convolution processing on the large kernel partial convolution channels: ;in, This represents a regular part of convolution; This represents a large kernel convolution, preferably a 5×5 large kernel convolution.

[0069] For lightweight reserved channels The pass-through is preserved, and no additional large-kernel convolution is performed. Then, the partial convolution result, the large-kernel partial convolution result, and the lightweight preserved channel are concatenated: Then, cross-channel fusion is performed using 1×1 pointwise convolution: ,in, Indicates the first Enhanced reconstructed features output by a built-in feature enhancement module.

[0070] Using the above method, the first set of channels is used for general local texture compensation, the second set of channels is used for larger-scale edge continuation and texture direction compensation, and the remaining channels are used for lightweight preservation and subsequent cross-channel fusion. This avoids performing large-kernel convolutions on all channels, improving the recovery capability of local textures, edges, and high-frequency details while maintaining low model complexity.

[0071] Dense feature fusion is performed on the enhanced reconstruction features output by multiple built-in feature enhancement modules to obtain fused features.

[0072] The enhanced reconstruction features output by multiple built-in feature enhancement modules are denoted as follows: These enhanced reconstruction features are then stitched together along the channel dimension to obtain the stitched features. The dense feature fusion module is used to fuse enhanced reconstruction features from different levels, allowing shallow texture information, high-level structural information, global context information, and local detail compensation information to participate in the final reconstruction.

[0073] The dense feature fusion process is represented as: ,in, This indicates the concatenation of output features from multiple built-in feature enhancement modules; This represents the weights of a 1×1 convolutional layer, used for channel compression. This represents the weights of a 3×3 convolutional layer, used for spatial neighborhood fusion. Represents a shallow feature map; This indicates the fusion feature.

[0074] Through the above-mentioned dense feature fusion process, features from different stages in multiple built-in feature enhancement modules can participate in the final reconstruction, and the difficulty of network optimization can be reduced by residual fusion with shallow feature maps.

[0075] The fused features are convolutionally processed and pixel rearranged to generate a high-resolution image.

[0076] Fusion features The input upsampling module consists of convolutional layers and pixel rearrangement layers. The convolutional layers map the fused features to upsampled features, while the pixel rearrangement layers rearrange information from the channel dimension to the spatial dimension, achieving resolution upsampling.

[0077] The high-resolution image generation process is represented as follows: ,in, This represents the generated high-resolution image; This represents the weights of the convolutional layer before upsampling; This indicates a pixel rearrangement operation.

[0078] In one embodiment, the upsampling factor can be 2x, 3x, or 4x. Through convolution and pixel rearrangement processing, the fused features corresponding to the low-resolution image can be rearranged into a high-resolution spatial representation, thereby generating a high-resolution image.

[0079] This embodiment provides a model training process, such as Figure 2 As shown, the model training phase is used to determine the learnable parameters in the shallow feature extraction module, the built-in feature enhancement module with multiple cascaded settings, the dense feature fusion module, and the pixel rearrangement layer.

[0080] During the model training phase, high-resolution images and low-resolution images obtained by downsampling from the high-resolution images constitute training sample pairs. The low-resolution images serve as training inputs, and the corresponding high-resolution images serve as supervised outputs. In one embodiment, the corresponding low-resolution images can be generated from the high-resolution images using a bicubic downsampling method.

[0081] Data augmentation is applied to training sample pairs, including at least one of random horizontal flipping, random vertical flipping, random rotation, random translation, and random cropping. Data augmentation can improve the model's adaptability to different image orientations, different local regions, and different texture distributions.

[0082] In one specific implementation, image patches of a preset size are cropped from the low-resolution image and used as training input, while the corresponding high-resolution image patches are used as supervised output. The training patch size is set to 64×64, and each training batch includes 16 image patches.

[0083] A lightweight super-resolution model consisting of a shallow feature extraction module, multiple cascaded built-in feature enhancement modules, a dense feature fusion module, and a pixel rearrangement layer was trained using the Adam optimizer. Training parameters included: Adam optimizer parameters β1 = 0.9 and β2 = 0.999; initial learning rate of 5 × 10^-4; a total of 1200 training epochs; the learning rate was reduced to half of the previous stage every 200 epochs; the window size for multi-head self-attention processing was 8 × 8; the number of attention heads was 5; the number of built-in feature enhancement modules was 6; and a 5 × 5 convolutional kernel was used for the large kernel part of the convolutional processing.

[0084] In one embodiment, the DIV2K dataset can be used as the training set, and Set5, Set14, BSD100, Urban100, and Manga109 can be used as the test set. Evaluation metrics may include peak signal-to-noise ratio (PSNR) and structural similarity. PSNR is used to evaluate the pixel error between the reconstructed image and the true high-resolution image, while structural similarity is used to evaluate the structural consistency between the reconstructed image and the true high-resolution image.

[0085] Experimental results are as follows Figure 6 As shown. It should be noted that the model in this embodiment does not require achieving the absolute highest value on all datasets, all magnifications, or all individual metrics. Instead, it is designed to achieve good reconstruction accuracy in lightweight image super-resolution scenarios with low parameter count and low computational complexity, thereby achieving a better trade-off between reconstruction accuracy and model complexity. This makes it suitable for resource-constrained scenarios such as mobile terminals, embedded vision devices, and edge computing devices.

[0086] Embodiments of this application also provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the lightweight super-resolution method based on hybrid attention and large kernel convolution provided in the above embodiments.

[0087] Another aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the lightweight super-resolution method based on hybrid attention and large kernel convolution as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0088] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A lightweight super-resolution method based on hybrid attention and large kernel convolution, characterized in that, include: Acquire a low-resolution image, perform shallow feature extraction on the low-resolution image, and obtain a shallow feature map; The shallow feature map is input into multiple cascaded built-in feature enhancement modules, which include a hybrid attention module and a large kernel convolution detail enhancement module. Globally enhanced features are obtained by performing window multi-head self-attention processing, multi-scale channel attention processing, and global self-attention processing on the input features through a hybrid attention module. The global enhancement features are divided into channels by a large kernel convolution detail enhancement module. Partial convolution processing is performed on the first part of the channels after division, large kernel partial convolution processing is performed on the second part of the channels after division, and the remaining channels are preserved and passed on. Cross-channel fusion is performed by point-by-point convolution to obtain the enhanced reconstruction features. Dense feature fusion is performed on the enhanced reconstruction features output by multiple built-in feature enhancement modules to obtain fused features; The fused features are convolutionally processed and pixel rearranged to generate a high-resolution image.

2. The lightweight super-resolution method based on hybrid attention and large kernel convolution as described in claim 1, characterized in that, A shallow feature map is obtained by performing convolution processing on a low-resolution image using a 3×3 convolutional layer. The enhanced reconstruction features output by multiple built-in feature enhancement modules are spliced ​​along the channel dimension to obtain the spliced ​​features. Channel compression is performed on the spliced ​​features through a 1×1 convolutional layer, and spatial neighborhood fusion is performed on the channel-compressed features through a 3×3 convolutional layer. The spatially fused features are then residually fused with the shallow feature map to obtain the fused features. The fused features are mapped to upsampled features through convolutional layers, and the resolution of the upsampled features is amplified through pixel rearrangement layers to generate a high-resolution image.

3. The lightweight super-resolution method based on hybrid attention and large kernel convolution according to claim 1, characterized in that, The hybrid attention module includes at least two hybrid attention processing layers, each of which includes a window multi-head self-attention unit, a multi-scale channel attention unit, and a global self-attention unit. The multi-head self-attention unit in a window is used to establish inter-pixel relationships within a local window to obtain the window attention response; The multi-scale channel attention unit is used to generate channel attention weights based on pooling responses of different scales. The channel attention weights are then used to enhance the window attention response to obtain a multi-scale channel response. Global self-attention units are used to enhance the global relationships of multi-scale channel responses to obtain global self-attention responses. Globally enhanced features are generated based on window attention response, multi-scale channel response, and global self-attention response.

4. The lightweight super-resolution method based on hybrid attention and large kernel convolution according to claim 3, characterized in that, Multi-scale channel attention units include max pooling paths and average pooling paths; The max pooling path is used to perform max pooling on the input features at scales of 3×3, 2×2 and 1×1 respectively, to obtain multi-scale max pooling features; The average pooling path is used to perform average pooling processing on the input features at scales of 3×3, 2×2 and 1×1 respectively, to obtain multi-scale average pooling features; The multi-scale max pooling features and multi-scale average pooling features are compressed by 1×1 convolution and expanded by 1×1 convolution to obtain the weights of the two channels. The weights of the two channels are concatenated and merged, and the merged channel weights are obtained through Sigmoid activation. Multiplying the fused channel weights by the window attention response yields the multi-scale channel response.

5. The lightweight super-resolution method based on hybrid attention and large kernel convolution according to claim 3, characterized in that, Also includes: The global enhanced features obtained by fusing window attention response, multi-scale channel response and global self-attention response are defined as the global semantic extension map. The global semantic extension map is used to characterize the window-outside structural continuity relationship, cross-scale contextual relationship and global semantic response state of the input features. The window attention response, multi-scale channel response, and global self-attention response under the same channel are normalized respectively. Based on the consistency among the normalized window attention response, multi-scale channel response, and global self-attention response, the global extension closure of the channel is determined. The global extension closure is used to characterize the completeness of the global structural expression of the corresponding channel in the global semantic extension graph.

6. The lightweight super-resolution method based on hybrid attention and large kernel convolution according to claim 5, characterized in that, Based on the high-frequency response, gradient response, edge response, or the difference between the original feature and the smooth feature of the corresponding channel in the global semantic extension map, determine the local texture residue state of the channel. Based on the global extensional closure and local texture residual state, the channel sparse use compensation amount is determined. The channel sparse use compensation amount is used to determine whether the corresponding channel needs to enter the convolution compensation path, and whether it needs to enter the normal part convolution branch or the large kernel part convolution branch. When the global extension closure is greater than or equal to the preset closure threshold and the local texture residual state is less than the preset residual threshold, the corresponding channel is determined as a lightweight retention candidate channel. When the global extension closure is greater than or equal to the preset closure threshold, and the local texture residual state is greater than or equal to the preset residual threshold, the corresponding channel is determined as a candidate channel for normal partial convolution compensation. When the global extension closure is less than the preset closure threshold and the local texture residual state is greater than or equal to the preset residual threshold, the corresponding channel is determined as a candidate channel for large kernel part convolution compensation. When the global extension closure is less than the preset closure threshold and the local texture residue is less than the preset residue threshold, the corresponding channel is determined as a lightweight retention candidate channel or a normal partial convolution compensation candidate channel according to the preset channel sparse use rule.

7. The lightweight super-resolution method based on hybrid attention and large kernel convolution according to claim 6, characterized in that, Based on the channel sparse use compensation amount and the preset channel sparse use rules, a channel sparse use label map is generated. The channel sparse use label map is used to label the channels in the global enhancement features as lightweight preservation channels, ordinary partial convolution channels, or large kernel partial convolution channels. Among them, lightweight preservation channels, ordinary partial convolution channels, and large kernel partial convolution channels are non-overlapping sets of channels. The preset channel sparse use rules include: the number of channels participating in normal partial convolution processing and large kernel partial convolution processing shall not exceed the preset ratio, and the remaining channels shall be used as lightweight reserved channels to participate in subsequent stitching and pointwise convolution fusion. Based on the channel sparse use labeling map, partial convolution processing is performed on the normal partial convolution channels, 5×5 large kernel partial convolution processing is performed on the large kernel partial convolution channels, and lightweight reserved channels are retained and passed. The results of partial convolution processing, the results of large kernel partial convolution processing, and lightweight preserved channels are concatenated and then fused across channels through 1×1 pointwise convolution to obtain enhanced reconstruction features.

8. The lightweight super-resolution method based on hybrid attention and large kernel convolution according to claim 1, characterized in that, It also includes the model training phase; During the model training phase, high-resolution images and low-resolution images obtained by downsampling high-resolution images are used to form training sample pairs. Data augmentation is performed on the training sample pairs, including at least one of random horizontal flipping, random vertical flipping, random rotation, random translation, and random cropping; Image patches of a preset size are cropped from low-resolution images and used as training input, while the corresponding high-resolution image patches are used as supervised output. A lightweight super-resolution model consisting of a shallow feature extraction module, multiple cascaded built-in feature enhancement modules, a dense feature fusion module, and a pixel rearrangement layer was trained using the Adam optimizer. The training parameters during the model training phase included: a training patch size of 64×64, 16 image patches per training batch, Adam optimizer parameters β1 of 0.9 and β2 of 0.999, an initial learning rate of 5×10^-4, a total of 1200 training epochs, with the learning rate reduced to half of the previous stage every 200 epochs, an 8×8 window size for multi-head self-attention processing, 5 attention heads, 6 built-in feature enhancement modules, and a 5×5 convolutional kernel for the large kernel part.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the lightweight super-resolution method based on hybrid attention and large kernel convolution as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by the computer's processor, causes the computer to perform any one of the lightweight super-resolution methods based on hybrid attention and large kernel convolution as described in claims 1 to 8.

Images