Lightweight face super-resolution reconstruction method based on mixed residual attention

By combining local convolution modules and lightweight self-attention modules in super-resolution reconstruction of face images, a hybrid residual attention module is formed, and the reverse order fusion module is used to solve the high computational complexity and feature fusion in the prior art. The problem of insufficient results has achieved high-quality image reconstruction and improved computing efficiency.

CN118941449AActive Publication Date: 2024-11-12BEIJING ELECTRONICS SCI & TECH INST

Patent Information

Application Number
CN202410984595.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-22
Publication Date
2024-11-12
Estimated Expiration
2044-07-22

AI Technical Summary

Technical Problem

The prior art is difficult to effectively combine convolution operation and self-attention mechanism in super-resolution reconstruction of face images, resulting in high computational complexity and the inability to capture global dependencies and local detailed information at the same time.

Method used

Using a lightweight face super-resolution reconstruction method based on mixed residual attention, a local convolution module and a lightweight self-attention module are constructed to form a mixed residual attention module, and combined with a reverse fusion module, high-resolution images are extracted .

Benefits of technology

Improves the quality of face image reconstruction, reduces computing complexity, and is suitable for resource-constrained devices and real-time scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941449A_ABST
    Figure CN118941449A_ABST
Patent Text Reader

Abstract

The invention provides a lightweight face super-resolution reconstruction method based on mixed residual attention, and the method comprises the steps: capturing local detail information and a global dependency relationship of a shallow feature map through a local convolution module and a lightweight self-attention module, and forming local features and global features; constructing a mixed residual attention module by integrating a local convolution module and a lightweight self-attention module, and performing feature fusion on local features and global features through a residual structure; initial features of the low-resolution face image are extracted, a plurality of mixed residual attention modules are connected in series and spliced, multi-layer features are fused through a reverse fusion module, and target deep comprehensive features are obtained; and inputting the deep comprehensive features of the target into an image up-sampling module to generate a high-resolution face reconstruction image. According to the method, the reconstruction quality of the face image is improved by combining the global and local features, the calculation complexity is effectively reduced, and the method is suitable for being applied to resource-limited equipment and real-time scenes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image super-resolution, and in particular to a lightweight face super-resolution reconstruction method based on hybrid residual attention. Background Art

[0002] Image super-resolution reconstruction technology has a wide range of applications in the field of image processing, such as satellite image enhancement, medical image analysis, and video surveillance. In the field of security, especially in facial recognition and monitoring systems, super-resolution reconstruction of facial images is of great significance. By reconstructing low-resolution facial images into high-resolution images, the clarity and details of the image can be significantly improved, thereby improving the accuracy of facial recognition and the effectiveness of the monitoring system.

[0003] Traditional face super-resolution reconstruction methods mainly rely on convolutional neural networks (CNNs) to extract and reconstruct image features. Convolutional neural networks gradually extract multi-level features of images by stacking convolutional layers, pooling layers, and activation functions. However, convolutional neural networks have certain limitations in processing global dependencies and capturing long-distance information. The convolution kernel has a limited receptive field, which makes it less efficient in processing the global information of the image, and it is easy to cause blurring and loss of details in the reconstructed image.

[0004] In recent years, the Transformer model based on the attention mechanism has achieved remarkable results in the fields of natural language processing and computer vision. The self-attention mechanism can effectively capture global dependencies by calculating the similarity between each position in the input feature map. However, the standard Transformer model has high computational complexity, especially when processing high-resolution images, requiring a lot of computing resources and memory space, and is not suitable for resource-constrained devices and real-time scenarios.

[0005] In order to reduce computational complexity, researchers have proposed various lightweight self-attention mechanisms. For example, the Linear Spatial Reduction Attention of the PyramidVision Transformer (PVT) reduces computational and memory requirements by linearly reducing the feature map. However, although the lightweight self-attention mechanism has advantages in capturing global dependencies, it still has shortcomings in capturing local detail information.

[0006] In practical applications, super-resolution reconstruction of face images needs to pay attention to both local detail information and global dependencies. Using convolutional neural networks or self-attention mechanisms alone may not be able to meet both requirements at the same time. Therefore, current technologies face challenges in improving image reconstruction quality and reducing computational complexity. A method that can effectively combine convolution operations with self-attention mechanisms is needed to achieve efficient and high-quality super-resolution reconstruction of face images. Summary of the invention

[0007] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0008] To this end, the purpose of this application is to propose a lightweight face super-resolution reconstruction method based on hybrid residual attention, which is used to solve the problems of ignoring the fusion of local and global dimensional features and high computational complexity in current face super-resolution reconstruction methods based on self-attention mechanism. By constructing a lightweight hybrid residual self-attention module and a reverse fusion module, the face image reconstruction capability can be effectively improved and the parameter complexity and computational complexity can be reduced.

[0009] To achieve the above objectives, the present application proposes a lightweight face super-resolution reconstruction method based on hybrid residual attention, comprising the following steps:

[0010] Constructing a local convolution module and a lightweight self-attention module, wherein the local convolution module and the lightweight self-attention module are used to capture local detail information and global dependency of shallow feature maps, respectively, to form local features and global features;

[0011] Integrate the local convolution module and the lightweight self-attention module to construct a hybrid residual attention module, and perform feature fusion on the local features and the global features through the hybrid residual attention module;

[0012] Extract the initial features of the low-resolution face image, connect multiple hybrid residual attention modules in series, take the initial features as input, fuse multiple layers of features through a reverse fusion module, and obtain the target deep comprehensive features;

[0013] The target deep comprehensive features are input into an image upsampling module to generate a high-resolution face reconstruction image.

[0014] Optionally, the capturing process of the local convolution model includes:

[0015] F local =Conv 1×1 (DWConv 3×3 (SE(Conv 1×1 (F init ))))

[0016] Among them, F init is the initial feature of the low-resolution face image, Conv 1×1 () represents the convolution layer function with a convolution kernel size of 1×1, SE() represents the channel attention mechanism function, DWConv 3×3 () represents the channel-by-channel convolution function with a convolution kernel size of 3×3, F local is the local feature.

[0017] Optionally, the capture process of the lightweight self-attention module includes:

[0018] A lightweight linear space compression layer is constructed to reduce the number of parameters and computational complexity. Its expression is as follows:

[0019] X LLSC =GELU(LN(Conv 1×1 (MaxPool 7×7 (LN(F init ))))

[0020] Among them, MaxPool 7×7 (·) is the global average maximum pooling layer function with a convolution kernel size of 7×7, Conv 1×1 (·) represents the convolution layer function with a kernel size of 1 and a stride of 1, LN(·) represents the layer normalization layer function, which performs normalization on the feature dimension, GELU(·) represents the GELU activation function, and F init is the initial feature, X LLSC ∈R C×H×W is the output feature map, which is used for the subsequent generation of key K and value V. C is the number of feature map channels, H is the feature map length, and W is the feature map width.

[0021] The initial feature F init Input lightweight linear space compression layer to generate lightweight key K feature map and value V feature map, and use linear layer function to generate key value And, the initial feature F init Input to the linear layer function to generate the query Its expression is as follows:

[0022] X kv =f kv (LLSC(F init ))

[0023] X q =f q (LN(F init ))

[0024] Among them, X kv, X q Represents the generated key value and query d k and d v are the dimensions of keys and values, respectively, and f kv (·),f q (·) represents the linear layer function, LN(·) represents the function of the layer normalization layer, and LLSC(·) represents the lightweight linear space compression layer function, which is used to reduce the number of parameters and computational complexity;

[0025] The key will be generated value and query Divided into h heads, the dimension of each head is The self-attention calculation is performed for each head separately, and the expression is:

[0026]

[0027] O i =A i V i

[0028] Among them, Q i , K i denote the query and key matrices of the ith head, respectively, and A i ∈R L×L is the attention score matrix, V i represents the value matrix of the i-th head, is the output of the i-th head;

[0029] Output all headers to O i The matrix is ​​concatenated along the feature dimension to obtain the concatenated matrix The expression is:

[0030] O = concat(O1, O2, ..., O h )

[0031] Among them, concat(·) represents the concatenation function;

[0032] Map the concatenated matrix O back to the original input dimension through a linear transformation, and get O∈R L×d , then the final output Attention(Q, K, V) is expressed as:

[0033] Attention(Q, K, V) = OW O

[0034] in, is the output weight matrix;

[0035] Apply residual connection and layer normalization to the output Attention (Q, K, V) to generate the final global feature, which is expressed as:

[0036] X mid =Attention(Q, K, V)+F init

[0037] F global =MLP(LN(X1))+X mid

[0038] Among them, X mid represents the intermediate calculation result, MLP(·) represents the fully connected layer unit, which is composed of a fully connected layer, a GELU activation function, a Dropout regularization layer and a fully connected layer, and F global is the global feature.

[0039] Optionally, the constructed hybrid residual attention module includes a main path and a residual path, and the local convolution module and the lightweight self-attention module are integrated to construct the hybrid residual attention module, and the local features and the global features are feature-fused through the hybrid residual attention module, including:

[0040] The main path captures the global dependencies of the shallow feature map through the lightweight self-attention module to form a global feature F global ; The residual path captures the local detail information of the shallow feature map through the convolution operation module to form a local feature F local , the expression is:

[0041] F global =LightAttention(F init )

[0042] F local =MBConv(F init )

[0043] Wherein, LightAttention(·) represents the lightweight self-attention module, and MBConv(·) represents the local convolution module;

[0044] The global features of the main path and the local features of the residual path are fused to obtain the deep comprehensive features of the single hybrid residual attention module, which is expressed as:

[0045] F integrate =F global +F local

[0046] Among them, F integrateIt is the deep comprehensive feature of a single hybrid residual attention module.

[0047] The extracting of initial features of the low-resolution face image comprises:

[0048] The initial features of the low-resolution face image are extracted through the initial feature extraction module. The expression is as follows:

[0049] F init =H SF (I Q )

[0050] Among them, I Q Represents the input low-resolution face image, H SF (·) represents the convolution layer function with a convolution kernel of 3×3, a stride of 1, a padding of 1, and 64 output channels. init is the initial feature extracted.

[0051] Optionally, the fusion process of the reverse fusion module includes:

[0052] If there are four hybrid residual attention modules connected in series, start from the output of the last layer of hybrid residual attention modules in series, directly use the features of this layer, and then go forward layer by layer, concatenate the features of the current layer with the fusion results of the previous layer, and then perform feature fusion through convolution operations. By accumulating multi-level feature information, richer context information can be obtained. The expression is as follows:

[0053]

[0054] Among them, F i represents the feature map extracted from the i-th layer hybrid residual attention module, H i represents the feature fusion result at the i-th layer, Conv(·) represents the 1×1 convolutional layer function, and Concat represents the concatenation operation of the feature map.

[0055] Optionally, the multiple hybrid residual attention modules are connected in series, the initial features are used as input, and multiple layers of features are fused through a reverse fusion module to obtain a target deep comprehensive feature, including:

[0056] If there are four mixed residual attention modules in series, the initial feature F init Input four consecutive hybrid residual attention modules and send the output results to the reverse fusion module to extract the target deep comprehensive features. The expression is as follows:

[0057] F deep =RFM(MLT(MLT(MLT(MLT(F init ))))))

[0058] Among them, MLT(·) represents the hybrid residual attention module, RFM(·) represents the reverse fusion module, and F deep It is the deep comprehensive feature of the target finally extracted.

[0059] Optionally, inputting the target deep comprehensive features into an image upsampling module to generate a high-resolution face reconstruction image includes:

[0060] I sr =Conv 3×3 (f up (F deep )

[0061] Among them, Conv 3×3 (·) represents the convolution layer function with a convolution kernel size of 3×3, f up (·) represents the upsampling convolution subelement function, which can be adjusted by f up The number of output channels of (·) determines the amplification factor, I sr Reconstruct high-resolution face images at corresponding magnifications.

[0062] The technical solution provided by the embodiments of the present application brings at least the following beneficial effects:

[0063] The local convolution module and the lightweight self-attention module are used to capture the local detail information and global dependency of the shallow feature map respectively, forming local features and global features; the local convolution module and the lightweight self-attention module are integrated to construct a hybrid residual attention module, and the local features and global features are fused through the residual structure; the initial features of the low-resolution face image are extracted, and multiple hybrid residual attention modules are connected in series, and the multi-layer features are fused through the reverse fusion module to obtain the target deep comprehensive features; the target deep comprehensive features are input into the image upsampling module to generate a high-resolution face reconstruction image. In summary, the combination of global and local features improves the reconstruction quality of the face image. At the same time, the lightweight self-attention module design effectively reduces the computational complexity, which is suitable for resource-constrained devices and real-time scenarios.

[0064] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0066] Figure 1 It is a flowchart of a lightweight face super-resolution reconstruction method based on hybrid residual attention according to an embodiment of the present application;

[0067] Figure 2 is a schematic diagram of a local convolution module according to an embodiment of the present application;

[0068] Figure 3 is a schematic diagram of a lightweight self-attention module according to an embodiment of the present application;

[0069] Figure 4 is a flow chart of a reverse fusion module according to an embodiment of the present application;

[0070] Figure 5 It is a schematic diagram of a face super-resolution reconstruction network based on hybrid residual attention according to an embodiment of the present application. DETAILED DESCRIPTION

[0071] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0072] The purpose of the present invention is to solve the technical problems of the difficulty in effectively fusing global information and local information and the high computational complexity in the existing face super-resolution technology, and to provide a face super-resolution reconstruction method based on a hybrid residual attention mechanism. The method can make full use of the interaction of cross-layer features, better fuse global and local information, significantly improve the quality of face image reconstruction, and effectively reduce the computational complexity, which is suitable for resource-constrained devices and real-time scenarios.

[0073] The following describes a lightweight face super-resolution reconstruction method based on hybrid residual attention in an embodiment of the present application with reference to the accompanying drawings.

[0074] Figure 1 is a flowchart of a lightweight face super-resolution reconstruction method based on hybrid residual attention according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0075] Step 101, construct a local convolution module and a lightweight self-attention module, the local convolution module and the lightweight self-attention module are used to capture the local detail information and global dependency of the shallow feature map, and form local features and global features.

[0076] In an embodiment of the present application, in order to extract initial features from low-resolution face images, a local convolution module and a lightweight self-attention module are constructed to capture local detail information and global dependencies of shallow feature maps to form local features and global features.

[0077] Figure 2 It is a schematic diagram of a local convolution module according to an embodiment of the present application.

[0078] Reference Figure 2 , the specific calculation process of the capture process of the local convolution model is as follows:

[0079] F local =Conv 1×1 (DWConv 3×3 (SE(Conv 1×1 (F init ))))

[0080] Among them, F init is the initial feature of the low-resolution face image, Conv 1×1 () represents the convolution layer function with a convolution kernel size of 1×1, SE() represents the channel attention mechanism function, DWConv 3×3 () represents the channel-by-channel convolution function with a convolution kernel size of 3×3, F local The local features are finally output.

[0081] Figure 3 It is a schematic diagram of a lightweight self-attention module according to an embodiment of the present application.

[0082] Reference Figure 3 , the specific calculation process of the capture process of the lightweight self-attention module is as follows:

[0083] First, a lightweight linear space compression layer is constructed to reduce the number of parameters and computational complexity. Its expression is as follows:

[0084] X LLSC =GELU(LN(Conv 1×1 (MaxPool 7×7 (LN(F init ))))

[0085] Among them, MaxPool 7×7 (·) is the global average maximum pooling layer function with a convolution kernel size of 7×7, Conv 1×1 (·) represents the convolution layer function with a kernel size of 1 and a stride of 1, LN(·) represents the layer normalization layer function, which performs normalization on the feature dimension, GELU(-) represents the GELU activation function, and F init is the initial feature, X LLSC ∈R C×H×W is the output feature map, which is used for the subsequent generation of key K and value V. C is the number of feature map channels, H is the feature map length, and W is the feature map width.

[0086] Next, the initial feature F initInput lightweight linear space compression layer to generate lightweight key K feature map and value V feature map, and use linear layer function to generate key value At the same time, the initial feature F init Input to the linear layer function to generate the query Its expression is as follows:

[0087] X kv =f kv (LLSC(F init ))

[0088] X q =f q (LN(F init ))

[0089] Among them, X kv , X q Represents the generated key value and query d k and d v are the dimensions of keys and values, respectively, and f kv (·),f q (·) represents the linear layer function, LN(·) represents the function of the layer normalization layer, and LLSC(·) represents the lightweight linear space compression layer function, which is used to reduce the number of parameters and computational complexity.

[0090] Again, the key will be generated value and query Divided into h heads, the dimension of each head is And then perform self-attention calculation for each head separately.

[0091] Specifically, we first calculate the scaled dot product attention score of each head, expressed as:

[0092]

[0093] Among them, Q i , K i denote the query and key matrices of the ith head, respectively, and A i ∈R L×L is the attention score matrix.

[0094] Next, calculate the output of the i-th head, the expression is:

[0095] O i =A i V i

[0096] Among them, V irepresents the value matrix of the i-th head, is the output of the i-th head.

[0097] After performing self-attention calculations on each head, the outputs of all heads O i The matrix is ​​concatenated along the feature dimension to obtain the concatenated matrix The expression is:

[0098] O = concat(O1, O2, ..., O h )

[0099] Among them, concat(·) represents the concatenation function.

[0100] Next, the concatenated matrix O is mapped back to the original input dimension through a linear transformation, and we get O∈R L×d , then the final output Attention(Q, K, V) is expressed as:

[0101] Attention(Q, K, V) = OW o

[0102] in, is the output weight matrix;

[0103] Finally, the output Attention (Q, K, V) is applied with residual connection and layer normalization to generate the final global feature, which is expressed as:

[0104] X mid =Attention(Q, K, V)+F init

[0105] F global =MLP(LN(X1))+X mid

[0106] Among them, X mid represents the intermediate calculation result, MLP(·) represents the fully connected layer unit, which is composed of a fully connected layer, a GELU activation function, a Dropout regularization layer and a fully connected layer, and F global is the global feature of the final output.

[0107] Step 102, integrating the local convolution module and the lightweight self-attention module to construct a hybrid residual attention module, and performing feature fusion of local features and global features through the hybrid residual attention module.

[0108] This step involves the construction process of a single-layer hybrid residual attention module.

[0109] In an embodiment of the present application, the constructed hybrid residual attention module includes a main path and a residual path.

[0110] Among them, the main path captures the global dependencies of shallow feature maps through a lightweight self-attention module to form a global feature F global ; The residual path captures the local detail information of the shallow feature map through the convolution operation module to form the local feature F local , the expression is:

[0111] F global =LightAttention(F init )

[0112] F local =MBConv(F init )

[0113] Among them, LightAttention(·) represents the lightweight self-attention module, and MBConv(·) represents the local convolution module.

[0114] Then, the global features of the main path and the local features of the residual path are fused to obtain the deep comprehensive features of a single hybrid residual attention module, which is expressed as:

[0115] F integrate =F global +F local

[0116] Among them, F integrate Deep integrated features for a single hybrid residual attention module.

[0117] Step 103, extract the initial features of the low-resolution face image, connect multiple hybrid residual attention modules in series, take the initial features as input, fuse multiple layers of features through a reverse fusion module, and obtain the target deep comprehensive features.

[0118] This step involves the construction process of a multi-layer hybrid residual attention module, and the fusion of multi-layer features is achieved through a reverse fusion module.

[0119] Figure 4 It is a flowchart of the reverse fusion module according to an embodiment of the present application.

[0120] Reference Figure 4 , if there are four hybrid residual attention modules connected in series, start from the output of the last layer of hybrid residual attention modules in series, directly use the features of this layer, and then go forward layer by layer, concatenate the features of the current layer with the fusion results of the previous layer, and then perform feature fusion through convolution operations, and accumulate multi-level feature information to obtain richer context information. The expression is as follows:

[0121]

[0122] Among them, F i represents the feature map extracted from the i-th layer hybrid residual attention module, H i represents the feature fusion result at the i-th layer, Conv(·) represents the 1×1 convolutional layer function, and Concat represents the concatenation operation of the feature map.

[0123] It can be understood that, referring to the above formula, for i=4, the feature map of this layer is directly used, that is, H4=F4, and for i=3, 2, 1, the current feature map F is first used. i And the fusion result H of the previous layer i+1 Concatenate to obtain the concatenated feature map Concat(F i , H i+1 ), and then perform convolution operation on the concatenated feature map to obtain the final fusion result Conv(Concat(F i , H i+1 )).

[0124] Thus, the entire construction introduction of the lightweight self-attention module, local convolution module, hybrid residual attention module and reverse fusion module is realized, and the application stage will be carried out later.

[0125] First, for low-resolution face images, the initial features are extracted through the initial feature extraction module, and the expression is as follows:

[0126] F init =H SF (I Q )

[0127] Among them, I Q Represents the input low-resolution face image, H SF (·) represents the convolution layer function with a convolution kernel of 3×3, a stride of 1, a padding of 1, and 64 output channels. init is the initial feature extracted.

[0128] Still taking the above embodiment as an example, Figure 5 As shown, if there are four mixed residual attention modules in series, the initial feature F init Input four consecutive hybrid residual attention modules and send the output results to the reverse fusion module to extract the target deep comprehensive features. The expression is as follows:

[0129] F deep =RFM(MLT(MLT(MLT(MLT(F init ))))))

[0130] Among them, MLT(·) represents the hybrid residual attention module, RFM(·) represents the reverse fusion module, and F deepIt is the deep comprehensive feature of the target finally extracted.

[0131] It should be noted that Figure 5 It is a schematic diagram of a face super-resolution reconstruction network based on hybrid residual attention according to an embodiment of the present application. In the present application, the overall system composed of a hybrid residual attention module, a reverse fusion module and an image upsampling module is called a face super-resolution reconstruction network based on hybrid residual attention.

[0132] Step 104, input the target deep comprehensive features into the image upsampling module to generate a high-resolution face reconstruction image.

[0133] Reference Figure 5 The face super-resolution reconstruction network based on hybrid residual attention is shown, which consists of three parts: initial feature extraction, target deep comprehensive feature extraction and final upsampling reconstruction.

[0134] After the initial feature F init After inputting four consecutive hybrid residual attention modules and sending the output results to the reverse fusion module to extract the target deep comprehensive features, in this step, the image upsampling module is finally used to generate high-resolution face reconstruction images.

[0135] Specifically, the image upsampling module is expressed as follows:

[0136] I sr =Conv 3×3 (f up (F deep )

[0137] Among them, Conv 3×3 (·) represents the convolution layer function with a convolution kernel size of 3×3, f up (·) represents the upsampling convolution subelement function, which can be adjusted by f up The number of output channels of (·) determines the amplification factor, I sr Reconstruct high-resolution face images at corresponding magnifications.

[0138] It can be understood that in this application, by adjusting f up (·) The number of output channels can be used to adjust the magnification, which can be 2, 3, 4, 8, etc.

[0139] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution of this application can be achieved, and this document is not limited here.

[0140] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A lightweight face super-resolution reconstruction method based on hybrid residual attention, It is characterized in that include: Constructing a local convolution module and a lightweight self-attention module, wherein the local convolution module and the lightweight self-attention module are used to capture local detail information and global dependency of shallow feature maps, respectively, to form local features and global features; Integrate the local convolution module and the lightweight self-attention module to construct a hybrid residual attention module, and perform feature fusion on the local features and the global features through the hybrid residual attention module; Extract the initial features of the low-resolution face image, connect multiple hybrid residual attention modules in series, take the initial features as input, fuse multiple layers of features through a reverse fusion module, and obtain the target deep comprehensive features; The target deep comprehensive features are input into an image upsampling module to generate a high-resolution face reconstruction image.

2. The method according to claim 1, It is characterized in that The capture process of the local convolution model includes: F local =Conv 1×1 (DWConv 3×3 (SE(Conv 1×1 (F init )))) Among them, F init is the initial feature of the low-resolution face image, Conv 1×1 () represents the convolution layer function with a convolution kernel size of 1×1, SE() represents the channel attention mechanism function, DWConv 3×3 () represents the channel-by-channel convolution function with a convolution kernel size of 3×3, F local is the local feature.

3. The method according to claim 2, It is characterized in that The capture process of the lightweight self-attention module includes: A lightweight linear space compression layer is constructed to reduce the number of parameters and computational complexity. Its expression is as follows: X LLsC =GELU(LN(Conv 1×1 (MaxPool 7×7 (LN(F init )))) Among them, MaxPool 7×7 (·) is the global average maximum pooling layer function with a convolution kernel size of 7×7, Conv 1×1 (·) represents the convolution layer function with a kernel size of 1 and a stride of 1, LN(·) represents the layer normalization layer function, which performs normalization on the feature dimension, GELU(·) represents the GELU activation function, and F init is the initial feature, X LLSC ∈R C×H×W is the output feature map, which is used for the subsequent generation of key K and value V. C is the number of feature map channels, H is the feature map length, and W is the feature map width. The initial feature F init Input lightweight linear space compression layer to generate lightweight key K feature map and value V feature map, and use linear layer function to generate key value And, the initial feature F init Input to the linear layer function to generate the query Its expression is as follows: X kv =f kv (LLSC(F init )) X q =f q (LN(F init )) Among them, X kv , X q Represents the generated key value and query d k and d v are the dimensions of keys and values, respectively, and f kv (·),f q (·) represents the linear layer function, LN(·) represents the function of the layer normalization layer, and LLSC(·) represents the lightweight linear space compression layer function, which is used to reduce the number of parameters and computational complexity; The key will be generated value and query Divided into h heads, the dimension of each head is The self-attention calculation is performed for each head separately, and the expression is: About i =A i In i Among them, Q i , K i denote the query and key matrices of the ith head, respectively, and A i ∈R L×L is the attention score matrix, V i represents the value matrix of the i-th head, is the output of the i-th head; Output all headers to O i The matrix is ​​concatenated along the feature dimension to obtain the concatenated matrix The expression is: O=concat(O 1 ,THE 2 ,…,THE h ) Among them, concat(·) represents the concatenation function; Map the concatenated matrix O back to the original input dimension through a linear transformation, and get O∈R L×d , then the final output Attention(Q,K,V) is expressed as: Attention(Q,K,V)=OW O in, is the output weight matrix; Apply residual connection and layer normalization to the output Attention (Q, K, V) to generate the final global feature, which is expressed as: X mid =Attention(Q,K,V)+F init F global =MLP(LN(X1))+X mid Among them, X mid represents the intermediate calculation result, MLP(·) represents the fully connected layer unit, which is composed of a fully connected layer, a GELU activation function, a Dropout regularization layer and a fully connected layer, and F global is the global feature.

4. The method according to claim 3, It is characterized in that The constructed hybrid residual attention module includes a main path and a residual path, and the local convolution module and the lightweight self-attention module are integrated to construct a hybrid residual attention module, and the local features and the global features are feature-fused through the hybrid residual attention module, including: The main path captures the global dependencies of the shallow feature map through the lightweight self-attention module to form a global feature F global ; The residual path captures the local detail information of the shallow feature map through the convolution operation module to form a local feature F locdl , the expression is: F global =LightAttention(F init ) F local =MBConv(F init ) Wherein, LightAttention(·) represents the lightweight self-attention module, and MBConv(·) represents the local convolution module; The global features of the main path and the local features of the residual path are fused to obtain the deep comprehensive features of the single hybrid residual attention module, which is expressed as: F integrate =F global +F local Among them, F integrate It is the deep comprehensive feature of a single hybrid residual attention module.

5. The method according to claim 4, It is characterized in that The extracting of initial features of the low-resolution face image comprises: The initial features of the low-resolution face image are extracted through the initial feature extraction module. The expression is as follows: F init =H SF (I Q ) Among them, I Q Represents the input low-resolution face image, H SF (·) represents the convolution layer function with a convolution kernel of 3×3, a stride of 1, a padding of 1, and 64 output channels. init is the initial feature extracted.

6. The method according to claim 5, It is characterized in that The fusion process of the reverse fusion module includes: If there are four hybrid residual attention modules connected in series, start from the output of the last layer of hybrid residual attention modules in series, directly use the features of this layer, and then go forward layer by layer, concatenate the features of the current layer with the fusion results of the previous layer, and then perform feature fusion through convolution operations. By accumulating multi-level feature information, richer context information can be obtained. The expression is as follows: Among them, F i represents the feature map extracted from the i-th layer hybrid residual attention module, H i represents the feature fusion result at the i-th layer, Conv(·) represents the 1×1 convolutional layer function, and Concat represents the concatenation operation of the feature map.

7. The method according to claim 6, It is characterized in that The multiple hybrid residual attention modules are connected in series, the initial features are used as input, and the multi-layer features are fused through the reverse fusion module to obtain the target deep comprehensive features, including: If there are four mixed residual attention modules in series, the initial feature F init Input four consecutive hybrid residual attention modules and send the output results to the reverse fusion module to extract the target deep comprehensive features. The expression is as follows: F deep =RFM(MLT(MLT(MLT(MLT(F init )))))) Among them, MLT(·) represents the hybrid residual attention module, RFM(·) represents the reverse fusion module, and F deep It is the deep comprehensive feature of the target finally extracted.

8. The method according to claim 7, It is characterized in that The step of inputting the target deep comprehensive features into an image upsampling module to generate a high-resolution face reconstruction image comprises: I sr =Conv 3×3 (f up (F deep ) Among them, Conv 3×3 (·) represents the convolution layer function with a convolution kernel size of 3×3, f up (·) represents the upsampling convolution subelement function, which can be adjusted by f up The number of output channels of (·) determines the amplification factor, I sr Reconstruct high-resolution face images at corresponding magnifications.

Citation Information

Patent Citations

  • Lightweight double-branch convolutional neural network for image target detection and detection method thereof

    CN114648684A

  • Image super-resolution reconstruction model and method based on residual mixed attention network

    CN115222601A

  • Transform and convolutional neural network combination-based face super-resolution reconstruction method and system

    CN115953296A

  • Face super-resolution reconstruction method based on progressive training and face semantic segmentation

    CN116091315A

  • Single-frame image super-resolution method and system based on cross-layer mixed attention Transform

    CN117173025A

Cited By

  • CNN-Transform-based hierarchical information fusion module and fusion method

    CN119152334A

  • Image super-resolution method based on convolutional neural network and Transform hybrid network structure

    CN119693236A

  • Image super-resolution method and system, computer equipment and storage medium

    CN119784593A

  • Face super-resolution method and system based on double-branch dynamic collaborative network

    CN119809930A

  • Artificial intelligence-based plum blossom growth cycle disease image processing method and system

    CN119919814A