Frequency-sensing image super-resolution acceleration method and device
Through the frequency-aware image super-resolution acceleration method, using high-frequency and low-frequency features and attention modules, the problem of large computing overhead in the prior art is solved, and efficient image super-resolution recovery under limited resources is achieved, improving the efficiency of image super-resolution recovery under the conditions of limited resources. Performance and detail recovery capabilities.
Patent Information
- Application Number
- CN202411032993.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-29
AI Technical Summary
Existing image super-resolution technology has high computational overhead and long inference time, making it difficult to effectively restore high-resolution images in equipment with limited resources and harsh environments.
Using the frequency-aware image super-resolution acceleration method, the spatial, frequency and channel information are aggregated by extracting high-frequency and low-frequency characteristics in low-resolution images, combining the spatial frequency self-attention module and the channel transposed self-attention module to aggregate the spatial, frequency and channel information. Feedforward network processing, ultimately reconstructing the super-resolution image through the pixel obfuscation layer.
Without adding additional computing overhead, it effectively restores complex details of the image, improves performance, makes up for the loss of long-distance information at attention, enhances high-frequency characteristics, and supplements global information.
Smart Images

Figure CN119048350A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image restoration, and in particular to a frequency-aware image super-resolution acceleration method and device. Background Art
[0002] At present, in real life, people have a strong demand for high-definition images and videos, especially in the experience of smart devices. High-resolution image and video display improves people's visual perception. In the field of computer vision, high-resolution images and videos are conducive to downstream tasks to obtain more detailed information from images and videos, and improve the upper limit of downstream tasks. However, in the process of image acquisition, some backward equipment and harsh environmental factors cause the collected images to have too low resolution and image details to be lost; in the process of network transmission, due to bandwidth limitations, high-resolution images are compressed into low-resolution images for transmission. Image super-resolution reconstruction technology reconstructs low-resolution images into high-resolution images, meeting people's demand for high-resolution HD images.
[0003] Image super-resolution technology has great application value in the fields of mobile devices, medical imaging, and network image transmission. Figure 1 As shown in Figure 1, the super-resolution network input is a low-resolution image generated by mobile devices, medical imaging equipment, and network transmission compression, and the output is a high-resolution image, which is used in 2K display, medical diagnosis, video conferencing, etc. However, most existing single-image super-resolution studies focus on improving performance, and these algorithms have problems such as high computational overhead and long inference time. Summary of the invention
[0004] To this end, the present invention aims to solve the deficiencies in the prior art to at least a certain extent, thereby proposing a frequency-aware image super-resolution acceleration method and device.
[0005] In a first aspect, the present invention provides a frequency-aware image super-resolution acceleration method, which is applied to a frequency-aware image super-resolution reconstruction network model, and the method comprises:
[0006] Extracting shallow features in the low-resolution image, separating a first high-frequency feature and a first low-frequency feature from the shallow features, processing the first high-frequency feature and the first low-frequency feature respectively, and inversely transforming the processed first high-frequency feature and the first low-frequency feature to obtain a first image feature;
[0007] Mapping the first image feature into three branches through a linear layer, inputting the three branches into a spatial frequency self-attention module, obtaining first spatial information, first channel information and frequency information, aggregating and obtaining a second image feature, and performing feedforward network processing on the second image feature;
[0008] Mapping the processed second image feature into two branches through a linear layer, inputting the two branches into a channel transposition self-attention module, obtaining second channel information and second spatial information, aggregating them to obtain a third image feature, and processing the third image feature through a feedforward network;
[0009] The processed third image feature is reconstructed through a pixel confusion layer to obtain a super-resolution image.
[0010] In a second aspect, the present invention provides a frequency-aware image super-resolution acceleration system, which is applied to a frequency-aware image super-resolution reconstruction network model, and the system comprises:
[0011] Frequency segmentation module: used for extracting shallow features in the low-resolution image, separating the first high-frequency features and the first low-frequency features in the shallow features, processing the first high-frequency features and the first low-frequency features respectively, and inversely transforming the processed first high-frequency features and the first low-frequency features to obtain the first image features;
[0012] Frequency-aware self-attention module: used to map the first image feature into three branches through a linear layer, input the three branches into the spatial frequency self-attention module, obtain the first spatial information, the first channel information and the frequency information, aggregate them to obtain the second image feature, and process the second image feature with a feedforward network;
[0013] Channel transposition self-attention module: used for mapping the processed second image feature into two branches through a linear layer, inputting the two branches into the channel transposition self-attention module, obtaining the second channel information and the second spatial information, aggregating them to obtain the third image feature, and performing feedforward network processing on the third image feature;
[0014] Image reconstruction module: used to reconstruct the processed third image feature through a pixel confusion layer to obtain a super-resolution image.
[0015] In a third aspect, the present invention provides a frequency-aware image super-resolution acceleration device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, each step of the frequency-aware image super-resolution acceleration method as described in any one of the first aspects is implemented.
[0016] In a fourth aspect, the present invention further provides a storage medium having a computer program stored thereon, and when the computer program is executed, the various steps in the frequency-aware image super-resolution acceleration method as described in the first aspect are implemented.
[0017] The present invention provides a frequency-aware image super-resolution acceleration method and device, which are applied to a frequency-aware image super-resolution reconstruction network model. The method comprises: extracting shallow features in a low-resolution image, separating a first high-frequency feature and a first low-frequency feature in the shallow features, processing the first high-frequency feature and the first low-frequency feature respectively, and inversely transforming the processed first high-frequency feature and the first low-frequency feature to obtain a first image feature; mapping the first image feature into three branches through a linear layer, inputting the three branches into a spatial frequency self-attention module, obtaining first spatial information, first channel information and frequency information, and aggregating them to obtain a second image feature, and performing feedforward network processing on the second image feature; mapping the processed second image feature into two branches through a linear layer, inputting the two branches into a channel transposition self-attention module, obtaining second channel information and second spatial information, and aggregating them to obtain a third image feature, and performing feedforward network processing on the third image feature; reconstructing the processed third image feature through a pixel confusion layer to obtain a super-resolution image. Through the method provided by the present invention, spatial frequency attention and channel transposed attention are set to capture spatial, frequency and channel information, realize feature aggregation between different dimensions and compensate for the loss of detail information over long distances of attention, and a feedforward network is set to realize the aggregation of frequency information, so that high-frequency features can be enhanced inside the attention block and high-frequency details can be supplemented outside to achieve high-frequency supplementation of the global information output by the attention mechanism; and spatial, frequency and channel information are integrated, and more pixels are activated for reconstruction through multi-branch aggregation; in addition, by mixing frequency and channel information and various frequency aggregations, complex details in the image can be effectively restored, and the performance can be further improved without adding additional overhead. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.
[0019] Figure 1 It is a flow chart of the frequency-aware image super-resolution acceleration method of the present invention;
[0020] Figure 2 A schematic diagram of a sub-process of the frequency-aware image super-resolution acceleration method of the present invention;
[0021] Figure 3 It is a structural schematic diagram of a frequency segmentation module of the frequency-aware image super-resolution acceleration method of the present invention;
[0022] Figure 4 It is a schematic diagram of the structure of the spatial frequency self-attention module of the frequency-aware image super-resolution acceleration method of the present invention;
[0023] Figure 5 It is a structural schematic diagram of the channel transposition self-attention module of the frequency-aware image super-resolution acceleration method of the present invention;
[0024] Figure 6 It is a structural schematic diagram of a space mapping module, a channel mapping module, and a frequency mapping module in the frequency-aware image super-resolution acceleration method of the present invention;
[0025] Figure 7 A schematic diagram of the structure of a feedforward network layer of the frequency-aware image super-resolution acceleration method of the present invention;
[0026] Figure 8 A schematic diagram of the effect of image reconstruction of the frequency-aware image super-resolution acceleration method of the present invention;
[0027] Fig. 9 It is an algorithm flow chart of the frequency-aware image super-resolution acceleration method of the present invention;
[0028] Fig.10 A schematic diagram of a network framework of the frequency-aware image super-resolution acceleration method of the present invention;
[0029] Fig.11 It is a schematic diagram of the program modules of the frequency-aware image super-resolution acceleration system of the present invention. DETAILED DESCRIPTION
[0030] In order to make the purpose, features and advantages of the invention more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0031] Please refer to Figure 1 , Figure 1 : is a flow chart of a frequency-aware image super-resolution acceleration method according to an embodiment of the present application. In this embodiment, the frequency-aware image super-resolution acceleration method is applied to a frequency-aware image super-resolution reconstruction network model, including:
[0032] Step 101: extract shallow features in the low-resolution image, separate first high-frequency features and first low-frequency features from the shallow features, process the first high-frequency features and the first low-frequency features respectively, and inversely transform the processed first high-frequency features and the first low-frequency features to obtain first image features.
[0033] In this embodiment, shallow features are first extracted from the low-resolution image through a convolution layer of size three, that is, a 3*3 convolution process is performed on the low-resolution image to obtain shallow features, so that the low-resolution image can be converted into shallow features with higher dimensions and obtain more information.
[0034] After the shallow features are obtained, the shallow features are separated into the first high-frequency features and the first low-frequency features, and they are processed separately in a divide-and-conquer manner, and then feature aggregation can be performed, that is, the first high-frequency features and the first low-frequency features of different frequencies are processed separately, and the processed first high-frequency features and the first low-frequency features are inversely transformed to obtain the transformed first image features.
[0035] Here, the shallow features are segmented into frequency segmentation modules. The frequency segmentation modules in this embodiment can be repeated, and the number of repetitions of the frequency segmentation modules is preferably 2, so as to obtain better effects.
[0036] Step 102: Map the first image feature into three branches through a linear layer, input the three branches into the spatial frequency self-attention module, obtain the first spatial information, the first channel information and the frequency information, aggregate them to obtain the second image feature, and process the second image feature by a feedforward network.
[0037] In this embodiment, the first image feature obtained in step 101 is mapped into three branches through a linear layer, and the three branches are input into the spatial frequency self-attention module, so that the first spatial information, the first channel information and the frequency information can be obtained, that is, the first image feature is further extracted from the spatial frequency self-attention module, and the extracted first spatial information, the first channel information and the frequency information are aggregated to obtain the second image feature. In which, more pixels are activated for reconstruction through multi-branch aggregation.
[0038] Among them, the obtained second image features are processed by a feedforward network. In order to better combine the advantages of Transformer and CNN in dual-frequency aggregation, a dual-frequency aggregation feedforward network is used, and the second image features are processed by the feedforward network, so that the frequency domain information of the second image features can be extracted.
[0039] Step 103: Map the processed second image feature into two branches through a linear layer, input the two branches into a channel transposition self-attention module, obtain the second channel information and the second spatial information, aggregate them to obtain the third image feature, and process the third image feature through a feedforward network.
[0040] In this embodiment, the second image feature obtained in step 102 is mapped into two branches through a linear layer, and the two branches are input into a channel transposition self-attention module to obtain second channel information and second spatial information, and the obtained second channel information and second spatial information are aggregated to obtain a third image feature. In this step, a channel transposition self-attention module is used, which can compensate for the shortcoming of the spatial self-attention module ignoring channel information through transposition attention, and can effectively restore the second image feature through the second channel information.
[0041] Among them, the order of step 102 and step 103 can be executed interchangeably, that is, the first image feature is first input into the channel transposition self-attention module, and then input into the spatial frequency self-attention module. And step 102 and step 103 can be cycled multiple times, preferably 9 times, for example, the first time is: spatial frequency self-attention module, channel transposition self-attention module; the second time is: spatial frequency self-attention module, channel transposition self-attention module; the third time is: spatial frequency self-attention module, channel transposition self-attention module, ..., the ninth time is: spatial frequency self-attention module, channel transposition self-attention module, that is, the cycle is repeated nine times, and the number of cycles of the two is not limited here, but in this embodiment, it is preferred that the image obtained by 9 times is the clearest.
[0042] Among them, the obtained third image features are processed by a feedforward network. In order to better combine the advantages of Transformer and CNN in dual-frequency aggregation, a dual-frequency aggregation feedforward network is used, and the third image features are processed by the feedforward network, so that the frequency domain information of the third image features can be extracted.
[0043] Step 104: reconstruct the processed third image feature through a pixel confusion layer to obtain a super-resolution image.
[0044] In this embodiment, the third image feature output by the spatial frequency self-attention module and the channel transposition self-attention module is reconstructed through the pixel confusion layer, so that a super-resolution image can be obtained. Among them, by using the spatial frequency self-attention module and the channel transposition self-attention module, the two modules not only work together to restore high-frequency details and extract long-range dependencies, thereby optimizing the output to obtain a more detailed representation, thereby capturing spatial, frequency and channel information, but also can achieve feature aggregation between different dimensions and compensate for the loss of detail information over a long distance of attention, and process the image features after passing through the spatial frequency attention module and the channel transposition self-attention module through a feedforward network to achieve aggregation of frequency information, so that high-frequency features can be enhanced inside the attention module and high-frequency features can be supplemented externally to achieve high-frequency supplementation of global information output by the attention mechanism, and the network model in the embodiment of the present application can further improve performance without adding additional overhead.
[0045] The embodiment of the present application provides a frequency-aware image super-resolution acceleration method, which is applied to a frequency-aware image super-resolution reconstruction network model. The method includes: extracting shallow features in a low-resolution image, separating a first high-frequency feature and a first low-frequency feature from the shallow features, processing the first high-frequency feature and the first low-frequency feature respectively, and inversely transforming the processed first high-frequency feature and the first low-frequency feature to obtain a first image feature; mapping the first image feature into three branches through a linear layer, inputting the three branches into a spatial frequency self-attention module, obtaining first spatial information, first channel information and frequency information, and aggregating them to obtain a second image feature, and performing feedforward network processing on the second image feature; mapping the processed second image feature into two branches through a linear layer, inputting the two branches into a channel transposition self-attention module, obtaining second channel information and second spatial information, and aggregating them to obtain a third image feature, and performing feedforward network processing on the third image feature; reconstructing the processed third image feature through a pixel confusion layer to obtain a super-resolution image. Through the method provided by the present invention, spatial frequency attention and channel transposed attention are set to capture spatial, frequency and channel information, realize feature aggregation between different dimensions and compensate for the loss of detail information over long distances of attention, and a feedforward network is set to realize the aggregation of frequency information, so that high-frequency features can be enhanced inside the attention block and high-frequency details can be supplemented outside to achieve high-frequency supplementation of the global information output by the attention mechanism; and spatial, frequency and channel information are integrated, and more pixels are activated for reconstruction through multi-branch aggregation; in addition, by mixing frequency and channel information and various frequency aggregations, complex details in the image can be effectively restored, and the performance can be further improved without adding additional overhead.
[0046] For further information, see Figure 2 , Figure 2 This is a schematic diagram of a sub-process of the frequency-aware image super-resolution acceleration method according to an embodiment of the present application. In this embodiment, it also includes:
[0047] Step 201: Perform L1 loss calculation on the super-resolution image and the preset high-definition image to obtain a calculation result, and use the calculation result as the super-resolution supervision error;
[0048] Step 202: iteratively optimize the frequency-aware image super-resolution reconstruction network model through the super-resolution supervision error.
[0049] In this embodiment, the super-resolution image actually obtained by the frequency-aware image super-resolution reconstruction network model and the high-definition image that is preset to be obtained by the frequency-aware image super-resolution reconstruction network model are subjected to L1 loss calculation, that is, absolute error loss calculation, and the calculation result is used as the super-resolution supervision error, that is, the loss calculation is performed between the actual image and the expected image, so that the actual error can be obtained. The obtained super-resolution supervision error is input into the frequency-aware image super-resolution reconstruction network model for iterative optimization, so that the super-resolution image actually obtained by the frequency-aware image super-resolution reconstruction network model can be closer to the preset high-definition image that can be obtained. Among them, L1 loss is also called mean absolute error (MAE), which is the average value of the absolute difference between the predicted value and the true value of the calculation model. This loss function is mainly used for regression problems because it directly measures the difference between the predicted value and the actual value without considering the direction.
[0050] For further information, see Figure 3 , Figure 3 This is a structural schematic diagram of a frequency segmentation module of a frequency-aware image super-resolution acceleration method according to an embodiment of the present application. In this embodiment, the first high-frequency feature and the first low-frequency feature in the shallow feature are separated, the first high-frequency feature and the first low-frequency feature are processed respectively, and the processed first high-frequency feature and the first low-frequency feature are inversely transformed to obtain a first image feature, including:
[0051] Step 301, separating the first high-frequency feature and the first low-frequency feature in the shallow layer feature by using a dual-tree complex wavelet;
[0052] Step 302: performing 1*1 convolution processing and 3*3 convolution processing on the first low-frequency features in sequence;
[0053] Step 303: Process the first high-frequency features through two linear layers in sequence;
[0054] Step 304: Perform a dual-tree complex wavelet inverse transform on the processed first high-frequency feature and the first low-frequency feature to obtain the first image feature.
[0055] In this embodiment, the first high-frequency feature and the first low-frequency feature in the shallow feature are first separated by the double-tree complex wavelet, and the first low-frequency feature separated from the shallow feature is first processed in sequence through a group of convolution layers with a size of 1 and a deep convolution layer with a size of 3, namely 1*1 convolution and 3*3 convolution, to converge global features, while the first high-frequency feature is successively processed through two linear layers to aggregate spatial features and channel features, and then the processed first high-frequency feature and the first low-frequency feature are inverse transformed by the double-tree complex wavelet to obtain the first image feature. Among them, the dual-tree complex wavelet transform is a complex wavelet transform with the characteristics of translation invariance, directional selectivity and limited redundancy. Its basic idea is to decompose the input signal into sub-bands of multiple scales and directions to extract information of different frequencies and directions in the signal. The dual-tree complex wavelet transform uses two parallel wavelet trees to decompose and reconstruct the input signal. One wavelet tree is used to extract the approximate component of the signal, and the other wavelet tree is used to extract the detail component of the signal. The two wavelet trees maintain translation invariance through a certain phase relationship, thereby avoiding the shift sensitivity problem in traditional wavelet transform.
[0056] For further information, see Figure 4 , Figure 4 This is a schematic diagram of the structure of the spatial frequency self-attention module of the frequency-aware image super-resolution acceleration method in an embodiment of the present application. In this embodiment, the three branches are input into the spatial frequency self-attention module to obtain the first spatial information, the first channel information and the frequency information and aggregate them, including:
[0057] Step 401, first map the first image feature into three branches through a linear layer, that is, map the input feature F: H*W*C into F2: H*W*3C through a linear layer, so as to obtain three channels, and segment through the channels, so as to obtain three branches. Then convert the three branches into QKV for processing, first input the three branches converted to QKV into the spatial self-attention module for operation, so as to obtain F3 = (Q*K^T)*V, that is, the first spatial information, wherein Q, K and V in the attention mechanism represent query, key and value respectively. The query is used to specify the information of interest, the key and value respectively represent each element in the sequence and the information related thereto, the attention weight is calculated by calculating the similarity between the query and the key, and the output of the attention mechanism is obtained by weighted average. The use of the attention mechanism can improve the performance of the model when processing sequence data;
[0058] Step 402: Input V obtained by dividing the last branch among the three branches into a frequency mapping module to extract the frequency information;
[0059] Step 403: performing 1*1 convolution, 3*3 convolution, and channel feature mapping extraction on V segmented by the last branch of the three branches in sequence, thereby realizing mapping extraction of the first channel information;
[0060] Step 404: Aggregate the first spatial information, the frequency information, and the first channel information by element addition to obtain a first aggregated feature, wherein element addition is to aggregate multiple branches;
[0061] Step 405: spatially map the first spatial information to obtain spatial mapping information, and then cross-aggregate the spatial mapping information with the spatial mapping information processed by the activation function to obtain a second aggregated feature;
[0062] Step 406: cross-aggregate the first channel information with the first activation function processed by the activation function to obtain a third aggregate feature;
[0063] Step 407: perform additive aggregation on the first aggregated feature, the second aggregated feature and the third aggregated feature to obtain a fourth aggregated feature, and then fuse the fourth aggregated feature through a linear layer to obtain the second image feature.
[0064] For further information, see Figure 5 , Figure 5 This is a structural diagram of the channel transposition self-attention module of the frequency-aware image super-resolution acceleration method in the embodiment of the present application. In this embodiment, the second image feature is mapped into two branches through a linear layer, and the two branches are input into the channel transposition self-attention module to obtain the second channel information and the second spatial information and aggregate them, including:
[0065] Step 501, firstly, the second image feature is mapped into two branches through a linear layer, that is, the input feature F: H*W*C is mapped into F2: H*W*3C through a linear layer, thereby obtaining two channels, and the channels are segmented to obtain two branches. Then, the two branches are converted into QKV for processing, and the two branches converted into QKV are first input into the channel self-attention module for operation, thereby obtaining F3 = (Q*K^T)*V, that is, the second channel information;
[0066] Step 502: performing channel mapping on the second channel information to obtain channel mapping information, and then cross-aggregating the channel mapping information with the channel mapping information processed by the activation function to obtain a fifth aggregated feature;
[0067] Step 503: sequentially perform 1*1 convolution, 3*3 convolution and spatial mapping on V split by the last branch of the two branches to obtain second spatial information;
[0068] Step 504: A sixth aggregated feature is generated by cross-aggregating the second spatial information with the second spatial information processed by the activation function;
[0069] Step 505: perform additive aggregation on the fifth aggregate feature and the sixth aggregate feature to obtain a seventh aggregate feature, and then fuse the seventh aggregate feature through a linear layer to obtain the third image feature.
[0070] For further information, see Figure 6 , Figure 6 It is a schematic diagram of the structure of the space mapping module, the channel mapping module, and the frequency mapping module, wherein the process of the space mapping module is as follows:
[0071] The spatial features of the H*W*C input are extracted through a 1*1 convolution and segmented through channels. The extracted spatial features are divided into two branches for processing. One branch extracts global feature information through 1*1 convolution and 3*3 deep convolution. The other branch strengthens the aggregation of spatial features through 1*1 convolution and 3*3 deep convolution and activation function. Finally, the output information of the two branches is aggregated through 1*1 convolution.
[0072] Among them, the process of the channel mapping module is as follows:
[0073] A 1*1 convolution is used to extract spatial features from the input of H*W*C, and the features are segmented through channels. The extracted spatial features are processed into three branches. A residual connection is used to maintain basic features, a 3*3 depth convolution, a 5*5 expansion convolution, and a 1*1 convolution are used to capture a larger receptive field. The last branch processes local channel information through an average pooling layer and a 1*1 convolution. The information of the last three branches is aggregated through a 1*1 convolution and a residual connection.
[0074] Among them, the process of the frequency mapping module is as follows:
[0075] A 1*1 convolution is used to extract spatial features from the input of H*W*C, and the features are segmented through channels. The extracted spatial features are divided into two branches for processing. One branch extracts low-frequency information through 1*1 convolution and activation layer, and the other branch extracts high-frequency information through maximum pooling layer, 1*1 convolution and activation layer to restore the feature details of the low-frequency information branch. Finally, the two branches are converged through channel splicing and 1*1 convolution.
[0076] For further information, see Figure 7 , Figure 7 Schematic diagram of the structure of the feedforward network layer of the frequency-aware image super-resolution acceleration method in the embodiment of the present application. In this embodiment, the feedforward network includes frequency gating, and the feedforward network processing is performed on the second image features in sequence, and further includes:
[0077] Step 601, projecting onto the second image feature through a fully connected layer and processing through an activation function;
[0078] Step 602: Process the second image feature processed by the activation function through the frequency gating, wherein the processed second image feature is first separated into a second low-frequency feature and a second high-frequency feature, and the second high-frequency feature is sequentially subjected to 1*1 convolution and 3*3 depth convolution to obtain the second high-frequency feature after detail feature enhancement;
[0079] Step 603: perform element-wise multiplication on the second low-frequency feature and the second high-frequency feature after the detail feature is enhanced, and fuse them through a linear layer to obtain the processed second image feature.
[0080] In this embodiment, in order to better combine the advantages of Transformer and CNN in dual-frequency aggregation, a dual-frequency aggregation feed-forward network is used to perform feed-forward network processing on the second image features, wherein the feed-forward network also includes a frequency gate, and a frequency gate is introduced in the middle of the fully connected layer to achieve aggregation of frequency information.
[0081] It is first projected onto the second image feature output by the spatial frequency self-attention module through a fully connected layer, then processed by the GELU activation function, and then the frequency domain information is extracted using a frequency gate, where the frequency gate is a submodule composed of two convolutions, which separate the input features into the second low-frequency feature and the second high-frequency feature, respectively. The second low-frequency information remains unchanged, and the second high-frequency information is enhanced through a series of 1*1 convolutions and 3*3 depth convolutions.
[0082] The second low-frequency feature and the second high-frequency feature with enhanced details are then element-wise multiplied and fused through a linear layer to obtain a processed second image feature.
[0083] Furthermore, the processing of the third image feature by a feed-forward network includes:
[0084] Step 701, projecting onto the third image feature through a fully connected layer and processing through an activation function;
[0085] Step 702: Process the third image feature processed by the activation function through the frequency gating, wherein the processed third image feature is first separated into a third low-frequency feature and a third high-frequency feature, and then the third high-frequency feature is sequentially subjected to 1*1 convolution and 3*3 depth convolution to obtain the third high-frequency feature with enhanced detail features;
[0086] Step 703: perform element-wise multiplication on the third low-frequency feature and the third high-frequency feature after enhancing the detail feature, and fuse them through a linear layer to obtain the processed third image feature.
[0087] In this embodiment, in order to better combine the advantages of Transformer and CNN in dual-frequency aggregation, a dual-frequency aggregation feed-forward network is used to perform feed-forward network processing on the third image feature, wherein the feed-forward network also includes a frequency gate, and a frequency gate is introduced in the middle of the fully connected layer to achieve aggregation of frequency information.
[0088] It is first projected onto the third image feature output by the spatial frequency self-attention module through a fully connected layer, then processed by the GELU activation function, and then the frequency domain information is extracted using a frequency gate, where the frequency gate is a submodule composed of two convolutions, which separate the input features into the third low-frequency feature and the third high-frequency feature respectively. The third low-frequency information remains unchanged, and the third high-frequency information is enhanced through a series of 1*1 convolutions and 3*3 depth convolutions.
[0089] The third low-frequency feature and the third high-frequency feature with enhanced details are then element-wise multiplied and fused through a linear layer to obtain a processed third image feature.
[0090] For further information, see Figure 8 , Figure 8 The effect diagram of image reconstruction of the frequency-aware image super-resolution acceleration method in the embodiment of the present application is shown. In the embodiment, the processed third image feature is reconstructed through a pixel confusion layer to obtain a super-resolution image, and further includes:
[0091] Step 801, the shallow feature and the processed third image feature are connected together through residuals to obtain a first residual feature, and the first residual feature is convolved by 3*3 to obtain the processed first residual feature, wherein the third image feature obtained by processing the shallow feature through the spatial frequency self-attention module and the transposed channel self-attention module and the shallow feature are residually connected, so that the initial features of the image can be retained to prevent image distortion;
[0092] Step 802: The shallow feature is connected to the processed first residual feature through residual connection to obtain the second residual feature, and the second residual feature is convolved by 3*3 to obtain the final feature, wherein the shallow feature and the second residual feature after residual connection are residually connected again, so as to better maintain the original features of the image and prevent image distortion;
[0093] Step 803: reconstructing the final feature through the pixel confusion layer to obtain the super-resolution image, wherein the third image feature that passes through the spatial frequency self-attention module and the transposed channel self-attention module and is distorted is converted into a spatial feature through the pixel confusion layer, thereby increasing the resolution of the image, that is, converting the low-resolution multi-channel of the image into a high-resolution normal channel, so that the third image feature is reconstructed from the channel into a super-resolution image through the pixel confusion layer.
[0094] For further information, see Fig. 9 and Fig.10 , Fig. 9 is an algorithm flow chart of the frequency-aware image super-resolution acceleration method in the embodiment of the present application, Fig.10 This is a schematic diagram of the network framework of the frequency-aware image super-resolution acceleration method in the embodiment of the present application. In this embodiment, the specific steps of the frequency-aware image super-resolution acceleration method are as follows:
[0095] 1. Shallow feature extraction: Extract shallow features from low-resolution images through a convolutional layer of size three;
[0096] 2. Frequency segmentation module: The shallow features are separated into the first high-frequency features and the first low-frequency features through the dual-tree complex wavelet transform, and the features of different frequencies are processed respectively. The first low-frequency features are aggregated through a set of convolutional layers with a size of 1 and a deep convolutional layer with a size w of 3 to aggregate the global features; and the first high-frequency features are aggregated through two linear layers for spatial and channel features, and then the processed first low-frequency features and the first high-frequency features are inversely transformed through the dual-tree complex wavelet to obtain the first image features;
[0097] 3. Spatial frequency self-attention module: The first image feature is mapped into three branches through the mapping layer for processing, and the three branches are input into the spatial self-attention module to obtain the first spatial information; the V segmented by the last branch is input into the frequency mapping module to extract the frequency information; the V segmented by the last branch is then mapped and extracted from the first channel information through 1*1 convolution, 3*3 convolution and channel mapping module; the first spatial information, the first channel information and the frequency information are superimposed and aggregated, and the first spatial information and the first channel information are cross-aggregated through the activation function, and all the outputs are additively aggregated to obtain the second image feature;
[0098] 4. Feedforward network layer: The second image feature is processed through a feedforward network, wherein the feedforward network includes a frequency gate to achieve aggregation of frequency information;
[0099] 5. Channel transposition self-attention module: Map the processed second image features into two branches, and convert the two branches into QKV for spatial self-attention operation to obtain the second channel information. Then, the V segmented by the last branch is extracted through 1*1 convolution, 3*3 convolution and spatial mapping module to extract the second spatial information, and finally aggregated to obtain the third image features.
[0100] 6. Feedforward network layer: The third image feature is processed through a feedforward network, wherein the feedforward network includes a frequency gate to achieve aggregation of frequency information;
[0101] 7. Residual connection layer: The processed third image features are connected to the shallow features twice through residual connection and two 3*3 convolutions to obtain the final features;
[0102] 8. Image reconstruction: Reconstruct the final features from the channel into a super-resolution image using a pixel confusion layer;
[0103] 9. Loss calculation: Calculate the absolute error between the super-resolution image and the preset high-definition image to obtain the super-resolution supervision loss, and iteratively optimize the model through the super-resolution supervision loss.
[0104] Among them, the frequency-aware self-attention module includes a spatial frequency self-attention module and a channel transposition self-attention module, and the execution order of the two modules in the frequency-aware self-attention module is not limited. The spatial frequency self-attention module can be executed first, and then the channel transposition self-attention module, or the channel transposition self-attention module can be executed first and then the spatial frequency self-attention module, and both the spatial frequency self-attention module and the channel transposition self-attention module are followed by a dual-frequency domain feedforward network layer. In this embodiment, the frequency-aware self-attention module is preferably cycled 9 times to obtain an image with better resolution; the frequency segmentation module is preferably cycled 2 times in this embodiment.
[0105] In summary, the embodiment of the present application proposes a lightweight frequency-aware image super-resolution network method that combines channel features, frequency domain features, and spatial features, integrates spatial, frequency, and channel information, and activates more pixels for reconstruction through multi-branch aggregation. In addition, by mixing frequency and channel information and various frequency aggregations, it can effectively restore complex details in the image, such as edges, patterns, and small features.
[0106] Among them, the embodiment of the present application first uses the frequency segmentation module to mix the first high-frequency feature and channel information after shallow feature extraction, and performs high-frequency recovery on the information after shallow feature extraction. Then, spatial frequency attention and channel transposition attention are applied alternately in the continuous DTB, wherein the spatial frequency attention and channel transposition attention work together to restore high-frequency details and extract long-range dependencies, thereby optimizing the output to obtain a more detailed representation. Through the combination of multiple attention features, spatial, frequency and channel contexts can be captured, and inter-block feature aggregation between different dimensions and compensation for the loss of detail information over long distances of attention can be achieved. In addition, in order to further integrate feature representation, a dual-domain frequency fusion module is also designed, and a frequency gate is introduced in the middle of the fully connected layer to achieve aggregation of frequency information, wherein the spatial frequency self-attention module and the channel transposition self-attention module are both provided with a dual-frequency domain feedforward network layer. In general, through the above three designs, high-frequency features can be enhanced inside the attention block and high-frequency details can be supplemented outside to achieve high-frequency supplementation of the global information output by the attention mechanism. These combined modules of the model can further improve the performance without adding additional overhead, and can significantly improve the performance with only a small increase in computational cost.
[0107] Furthermore, the embodiment of the present application provides a frequency-aware image super-resolution acceleration system 900. Fig.11 Schematic diagram of a program module of a frequency-aware image super-resolution acceleration system in an embodiment of the present application. In this embodiment, the frequency-aware image super-resolution acceleration system 900 includes:
[0108] Frequency segmentation module 901: used to extract shallow features in the low-resolution image, separate the first high-frequency features and the first low-frequency features in the shallow features, process the first high-frequency features and the first low-frequency features respectively, and inversely transform the processed first high-frequency features and the first low-frequency features to obtain first image features;
[0109] Frequency-aware self-attention module 902: used to map the first image feature into three branches through a linear layer, input the three branches into a spatial frequency self-attention module, obtain first spatial information, first channel information and frequency information, aggregate them to obtain a second image feature, and perform feedforward network processing on the second image feature;
[0110] Channel transposition self-attention module 903: used to map the processed second image feature into two branches through a linear layer, input the two branches into the channel transposition self-attention module, obtain the second channel information and the second spatial information, aggregate them to obtain the third image feature, and process the third image feature through a feedforward network;
[0111] Image reconstruction module 904: used to reconstruct the processed third image feature through a pixel confusion layer to obtain a super-resolution image.
[0112] The embodiment of the present application provides a frequency-aware image super-resolution acceleration system 900, which is applied to a frequency-aware image super-resolution reconstruction network model, and can achieve: extracting shallow features in a low-resolution image, separating a first high-frequency feature and a first low-frequency feature in the shallow features, processing the first high-frequency feature and the first low-frequency feature respectively, and inversely transforming the processed first high-frequency feature and the first low-frequency feature to obtain a first image feature; mapping the first image feature into three branches through a linear layer, inputting the three branches into a spatial frequency self-attention module, obtaining first spatial information, first channel information and frequency information, and aggregating them to obtain a second image feature, and performing feedforward network processing on the second image feature; mapping the processed second image feature into two branches through a linear layer, inputting the two branches into a channel transposition self-attention module, obtaining second channel information and second spatial information, and aggregating them to obtain a third image feature, and performing feedforward network processing on the third image feature; reconstructing the processed third image feature through a pixel confusion layer to obtain a super-resolution image. Through the method provided by the present invention, spatial frequency attention and channel transposed attention are set to capture spatial, frequency and channel information, realize feature aggregation between different dimensions and compensate for the loss of detail information over long distances of attention, and a feedforward network is set to realize the aggregation of frequency information, so that high-frequency features can be enhanced inside the attention block and high-frequency details can be supplemented outside to achieve high-frequency supplementation of the global information output by the attention mechanism; and spatial, frequency and channel information are integrated, and more pixels are activated for reconstruction through multi-branch aggregation; in addition, by mixing frequency and channel information and various frequency aggregations, complex details in the image can be effectively restored, and the performance can be further improved without adding additional overhead.
[0113] Furthermore, the present application also provides a frequency-aware image super-resolution acceleration device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the various steps in the frequency-aware image super-resolution acceleration method described above are implemented.
[0114] Furthermore, the present application also provides a storage medium on which a computer program is stored. When the computer is executed by a processor, each step in the above-mentioned frequency-aware image super-resolution acceleration method is implemented.
[0115] Each functional module in each embodiment of the present invention may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may be stored in a computer-readable storage medium.
[0116] Based on this understanding, the technical solution of the specification of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., various media that can store program codes.
[0117] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present invention is not limited by the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0118] For those skilled in the art, according to the concept of the embodiments of the present application, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.
Claims
1. A frequency-aware image super-resolution acceleration method, It is characterized in that Applied to a frequency-aware image super-resolution reconstruction network model, the method comprises: Extracting shallow features in the low-resolution image, separating a first high-frequency feature and a first low-frequency feature from the shallow features, processing the first high-frequency feature and the first low-frequency feature respectively, and inversely transforming the processed first high-frequency feature and the first low-frequency feature to obtain a first image feature; Mapping the first image feature into three branches through a linear layer, inputting the three branches into a spatial frequency self-attention module, obtaining first spatial information, first channel information and frequency information, aggregating and obtaining a second image feature, and performing feedforward network processing on the second image feature; Mapping the processed second image feature into two branches through a linear layer, inputting the two branches into a channel transposition self-attention module, obtaining second channel information and second spatial information, aggregating them to obtain a third image feature, and processing the third image feature through a feedforward network; The processed third image feature is reconstructed through a pixel confusion layer to obtain a super-resolution image.
2. The method according to claim 1, It is characterized in that Also includes: Perform L1 loss calculation on the super-resolution image and the preset high-definition image to obtain a calculation result, and use the calculation result as the super-resolution supervision error; The frequency-aware image super-resolution reconstruction network model is iteratively optimized through the super-resolution supervision error.
3. The method according to claim 1, wherein the first high-frequency feature and the first low-frequency feature are separated from the shallow features, the first high-frequency feature and the first low-frequency feature are processed respectively, and the processed first high-frequency feature and the first low-frequency feature are inversely transformed to obtain the first image feature, include: Separating the first high-frequency feature and the first low-frequency feature in the shallow layer feature by using a dual-tree complex wavelet; Performing 1*1 convolution processing and 3*3 convolution processing on the first low-frequency feature in sequence; Processing the first high-frequency feature through two linear layers in sequence; The processed first high-frequency feature and first low-frequency feature are then subjected to a dual-tree complex wavelet inverse transform to obtain the first image feature.
4. The method according to claim 1, It is characterized in that The three branches are input into the spatial frequency self-attention module to obtain the first spatial information, the first channel information and the frequency information and aggregate them, including: Convert the three branches into QKV and perform spatial self-attention operation to obtain the first spatial information; Then, the V divided by the last branch among the three branches is input into the frequency mapping module to extract the frequency information; Then, V segmented by the last branch of the three branches is sequentially subjected to 1*1 convolution, 3*3 convolution, and channel feature map extraction to obtain the first channel information; Aggregating the first spatial information, the frequency information, and the first channel information by element addition to obtain a first aggregated feature; Performing spatial mapping on the first spatial information to obtain spatial mapping information, and then cross-aggregating the spatial mapping information with the spatial mapping information processed by the activation function to obtain a second aggregated feature; Then, cross-aggregating the first channel information with the first activation function processed by the activation function to obtain a third aggregated feature; The first aggregate feature, the second aggregate feature and the third aggregate feature are additively aggregated to obtain a fourth aggregate feature, and the fourth aggregate feature is then fused through a linear layer to obtain the second image feature.
5. The method according to claim 1, It is characterized in that The second image feature is mapped into two branches through a linear layer, and the two branches are input into a channel transposition self-attention module to obtain second channel information and second spatial information and aggregate them, including: Convert the two branches into QKV and perform channel self-attention operation to obtain the second channel information; Performing channel mapping on the second channel information to obtain channel mapping information, and then cross-aggregating the channel mapping information with the channel mapping information processed by the activation function to obtain a fifth aggregated feature; Perform 1*1 convolution, 3*3 convolution and spatial mapping on V divided by the last branch of the two branches in sequence to obtain second spatial information; a sixth aggregation feature that cross-aggregates the second spatial information with the second spatial information processed by an activation function; The fifth aggregate feature and the sixth aggregate feature are additively aggregated to obtain a seventh aggregate feature, and the seventh aggregate feature is then fused through a linear layer to obtain the third image feature.
6. The method according to claim 3, It is characterized in that The feedforward network includes frequency gating, and the feedforward network processing is performed on the second image features in sequence, and further includes: Projecting onto the second image feature through a fully connected layer and processing through an activation function; Processing the second image feature processed by the activation function through the frequency gating, wherein the processed second image feature is first separated into a second low-frequency feature and a second high-frequency feature, and the second high-frequency feature is sequentially subjected to 1*1 convolution and 3*3 depth convolution to obtain the second high-frequency feature after enhancing the detail feature; Performing element-wise multiplication on the second low-frequency feature and the second high-frequency feature after enhancing the detail feature, and fusing them through a linear layer to obtain the processed second image feature; And, the processing of the third image feature by a feed-forward network includes: Projecting onto the third image feature through a fully connected layer and processing through an activation function; The third image feature processed by the activation function is processed by the frequency gating, wherein the processed third image feature is first separated into a third low-frequency feature and a third high-frequency feature, and then the third high-frequency feature is sequentially subjected to 1*1 convolution and 3*3 depth convolution to obtain the third high-frequency feature with enhanced detail features; The third low-frequency feature and the third high-frequency feature after the detail feature is enhanced are element-wise multiplied, and the processed third image feature is obtained by fusing them through a linear layer.
7. The method according to claim 3, It is characterized in that The processing of reconstructing the processed third image feature through a pixel confusion layer to obtain a super-resolution image further includes: Connecting the shallow feature and the processed third image feature together through residual to obtain a first residual feature, and performing a 3*3 convolution on the first residual feature to obtain the processed first residual feature; Then, the shallow feature and the processed first residual feature are connected together through the residual to obtain the second residual feature, and the second residual feature is convolved by 3*3 to obtain the final feature; The final feature is then reconstructed through the pixel confusion layer to obtain the super-resolution image.
8. A frequency-aware image super-resolution acceleration system, It is characterized in that Applied to the frequency-aware image super-resolution reconstruction network model, the system comprises: Frequency segmentation module: used for extracting shallow features in the low-resolution image, separating the first high-frequency features and the first low-frequency features in the shallow features, processing the first high-frequency features and the first low-frequency features respectively, and inversely transforming the processed first high-frequency features and the first low-frequency features to obtain the first image features; Frequency-aware self-attention module: used to map the first image feature into three branches through a linear layer, input the three branches into the spatial frequency self-attention module, obtain the first spatial information, the first channel information and the frequency information, aggregate them to obtain the second image feature, and process the second image feature with a feedforward network; Channel transposition self-attention module: used for mapping the processed second image feature into two branches through a linear layer, inputting the two branches into the channel transposition self-attention module, obtaining the second channel information and the second spatial information, aggregating them to obtain the third image feature, and performing feedforward network processing on the third image feature; Image reconstruction module: used to reconstruct the processed third image feature through a pixel confusion layer to obtain a super-resolution image.
9. A frequency-aware image super-resolution acceleration device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, each step of the frequency-aware image super-resolution acceleration method according to any one of claims 1 to 7 is implemented.
10. A storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, each step of the frequency-aware image super-resolution acceleration method according to any one of claims 1 to 7 is implemented.
