Face super-resolution method and system based on double-branch dynamic collaborative network

Abstract

The invention provides a face super-resolution method and system based on a double-branch dynamic collaborative network, and relates to the technical field of computer vision face image processing. The method comprises the following steps: extracting shallow layer features from a low-resolution face image through a first convolutional layer; performing global feature extraction on the shallow features through a plurality of multi-path attention aggregation modules to obtain face global features; local feature extraction is carried out on the shallow layer features through a plurality of multi-scale dynamic calibration modules, and face local features are obtained; splicing the face global features and the face local features through a dynamic feature fusion module to obtain fusion features; and performing reconstruction operation on the shallow features and the fusion features through a reconstruction module to obtain a super-resolution face reconstruction image. By fusing features of different hierarchical stages and different branches of the network, the capability of restoring fine facial details is improved.

Description

Technical Field

[0001] The present invention mainly relates to the technical field of computer vision face image processing, and in particular to a face super-resolution method and system based on a dual-branch dynamic collaborative network. Background Art

[0002] In real-life scenarios, differences in facial image acquisition equipment, such as hardware, shooting angles, and lighting conditions, can result in poor quality facial images. These low-quality images can hinder tasks like facial analysis and recognition. Therefore, facial super-resolution requires not only improving the number of pixels and image quality, but also maintaining the accuracy and authenticity of facial structure during the super-resolution restoration process.

[0003] In recent years, with the rapid development of deep learning networks, super-resolution facial image reconstruction methods based on convolutional neural networks (CNNs) are mainly divided into image reconstruction methods based on local facial features and image reconstruction methods based on global facial features. However, the image reconstruction methods based on local facial features use a blocking strategy that damages the overall structural coherence of the facial image, resulting in blurring in local areas of the reconstructed image. Image reconstruction methods based on global facial features are not only unsuitable for obtaining global information from the entire facial image, but also for technologies that introduce additional facial prior information to learn the global facial structure, since the prior information usually comes from low-resolution facial images, it will lead to inaccurate prior information, thereby affecting the accuracy of the generated super-resolution facial image. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to address the deficiencies of the existing technology and provide a face super-resolution method and system based on a dual-branch dynamic collaborative network.

[0005] The technical solution of the present invention to solve the above technical problems is as follows:

[0006] The face super-resolution method based on a dual-branch dynamic collaborative network includes the following steps:

[0007] Constructing a two-branch dynamic collaborative network, the two-branch dynamic collaborative network including a first convolutional layer, a plurality of multi-path attention aggregation modules connected in series, a plurality of multi-path attention aggregation modules connected in series, a dynamic feature fusion module, and a reconstruction module;

[0008] Extracting shallow features from the imported low-resolution face image through the first convolutional layer;

[0009] Performing global feature extraction on the shallow features through multiple multi-path attention aggregation modules to obtain global features of the face;

[0010] Performing local feature extraction on the shallow features through a plurality of the multi-scale dynamic calibration modules to obtain local features of the face;

[0011] splicing the global face features and the local face features through the dynamic feature fusion module to obtain a fused feature;

[0012] The reconstruction module performs upsampling processing on the shallow features and the fusion features, and performs a reconstruction operation on the upsampling processing results to obtain a super-resolution face reconstructed image.

[0013] Another technical solution of the present invention to solve the above technical problems is as follows:

[0014] The face super-resolution system based on a dual-branch dynamic collaborative network includes:

[0015] A network construction unit, configured to construct a two-branch dynamic collaborative network, the two-branch dynamic collaborative network comprising a first convolutional layer, a plurality of multi-path attention aggregation modules connected in series, a plurality of multi-path attention aggregation modules connected in series, a dynamic feature fusion module, and a reconstruction module;

[0016] A network processing unit is used to extract shallow features from the imported low-resolution face image through the first convolutional layer, perform global feature extraction on the shallow features through multiple multi-path attention aggregation modules to obtain global face features, perform local feature extraction on the shallow features through multiple multi-scale dynamic calibration modules to obtain local face features, splice the global face features and the local face features through the dynamic feature fusion module to obtain fused features, upsample the shallow features and the fused features through the reconstruction module, and perform reconstruction on the upsampling results to obtain a super-resolution face reconstructed image.

[0017] The beneficial effects of the present invention are as follows: the present invention reduces the overall computational complexity through a multi-path attention aggregation module, and explores the feature representation of different channels and position information to obtain global features. The multi-scale dynamic calibration module can capture and optimize local details of facial images at multiple scale levels, adaptively establish information interaction between multi-dimensional spaces and channels, and thus achieve accurate reconstruction of facial features. The dynamic feature fusion module integrates local features and global features to ensure that the network can fully capture the multi-dimensional information of the image. It can adaptively fuse features at different hierarchical stages and different branches, further improving the ability to restore fine facial details, and can generate higher quality super-resolution facial images while maintaining efficient operation, providing an efficient and high-performance solution for the field of super-resolution reconstruction of facial images. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of a face super-resolution method provided by an embodiment of the present invention;

[0019] Figure 2 A structural diagram of a face super-resolution method provided by an embodiment of the present invention;

[0020] Figure 3 A structural diagram of a dual-branch dynamic collaboration network provided by an embodiment of the present invention;

[0021] Figure 4 A structural diagram of a multi-path attention aggregation module provided by an embodiment of the present invention;

[0022] Figure 5 A structural diagram of a dynamic residual module provided by an embodiment of the present invention;

[0023] Figure 6 A structural diagram of a multi-scale dynamic calibration module provided in an embodiment of the present invention;

[0024] Figure 7 This is a module block diagram of the face super-resolution system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0026] In the super-resolution of facial images, it is crucial not only to increase the number of pixels and image quality, but also to maintain the accuracy and authenticity of facial structure during the super-resolution process. This involves effectively enhancing facial details in low-quality facial images to produce clearer and more natural super-resolution facial images. Successful execution of this task is crucial for improving facial image quality and, consequently, the performance of subsequent high-level vision tasks.

[0027] In recent years, with the rapid development of deep learning networks, methods based on convolutional neural networks (CNNs) have gradually emerged as a promising approach for facial image super-resolution reconstruction. This trend stems from CNNs' ability to leverage the superior performance of convolutional layers to achieve local perception of facial image data. By sliding convolution kernels across the image and sharing weights, the network can effectively capture local patterns within the image.

[0028] Similar to traditional face super-resolution reconstruction methods, CNN-based face image super-resolution reconstruction methods are mainly divided into two categories: image reconstruction methods based on local facial features and image reconstruction methods based on global facial features. Image reconstruction methods based on local facial features divide the face image into multiple sub-regions and input these image blocks into CNN to reconstruct the lost high-frequency facial details. However, this blocking strategy may damage the overall structural coherence of the face image, resulting in blurring of the reconstructed image in local areas. In view of this, researchers have begun to turn to building global methods to preserve and utilize the overall facial structure information to improve the reconstruction quality.

[0029] Image reconstruction methods based on global facial features can be mainly divided into two categories: holistic image-based reconstruction methods and reconstruction methods based on facial structure priors. Holistic image-based reconstruction methods aim to input the entire face image into a CNN to focus on global facial structure. However, due to the limited receptive field of a CNN, these methods are not very effective at handling long-range dependencies and are therefore not suitable for extracting global information from the input face image. In contrast, reconstruction methods based on facial structure priors utilize facial parsing maps and facial landmarks as additional facial prior information. Leveraging the facial structure information and the locations of facial components provided by this prior information, they can better learn the global structure of the face. However, these methods also have limitations. Since the additional facial prior information introduced is often derived from low-resolution or coarse-resolution face images, it leads to inaccurate prior information, which in turn affects the quality of the generated high-resolution face images. Secondly, the encoding and decoding components in these methods are often constructed in tandem. This connection fails to fully utilize low-level features, and low-level features cannot fully guide the learning of high-level features, which in turn affects the performance of face super-resolution tasks.

[0030] Therefore, this paper investigates how to effectively balance long-range dependencies, global structure, and local details in facial image super-resolution reconstruction, proposing an innovative dual-branch model architecture to capture both global structure and local details. To further enhance the ability to restore fine facial features, this paper proposes a facial super-resolution technique based on a CNN-Transformer dynamic collaborative network.

[0031] like Figure 1-Figure 3 As shown, the face super-resolution method based on a two-branch dynamic collaborative network provided by an embodiment of the present invention includes the following steps:

[0032] Constructing a two-branch dynamic collaborative network, the two-branch dynamic collaborative network including a first convolutional layer, a plurality of multi-path attention aggregation modules connected in series, a plurality of multi-path attention aggregation modules connected in series, a dynamic feature fusion module, and a reconstruction module;

[0033] Extracting shallow features from the imported low-resolution face image through the first convolutional layer;

[0034] Performing global feature extraction on the shallow features through multiple multi-path attention aggregation modules to obtain global features of the face;

[0035] Performing local feature extraction on the shallow features through a plurality of the multi-scale dynamic calibration modules to obtain local features of the face;

[0036] splicing the global face features and the local face features through the dynamic feature fusion module to obtain a fused feature;

[0037] The reconstruction module performs upsampling processing on the shallow features and the fusion features, and performs a reconstruction operation on the upsampling processing results to obtain a super-resolution face reconstructed image.

[0038] Specifically, a dual-branch CNN-Transformer Dynamic Collaboration Network (CTDCNet), a heterogeneous fusion network model based on dynamic collaboration, is constructed based on the CNN and Transformer architectures for face image super-resolution. This model employs a dual-branch architecture consisting of two main parallel paths: a dynamic feature fusion module and a reconstruction module. One path utilizes the Efficient Multi-Path Attention Aggregation Block (EMAAB) based on the Transformer architecture to capture global facial structure. The other path utilizes the Multi-Scale Dynamic Calibration Block (MDCB) based on the CNN architecture to restore local facial details through local representation. The Dynamic Fusion Module (DFM) aligns Transformer and CNN architecture features, adaptively fusing features from different layers and branches of the network to further enhance the ability to restore fine facial details. The reconstruction module performs feature splicing to achieve residual learning, and uses an upsampling module for upscaling to produce super-resolution face reconstructions.

[0039] It should be understood that the downsampling process of the imported super-resolution face image to obtain a low-resolution face image includes: intercepting a face image from video surveillance, performing image cropping and face alignment operations on the face image to obtain a target face image (i.e., a super-resolution face image), and performing a data processing module on the original high-resolution face image I with a size of Ht×Wt×3. HR (i.e. target face image) uses bicubic interpolation algorithm to obtain its corresponding low-resolution face image I LR , where H and W are width and height respectively, 3 is the number of channels of the image, and t is the downsampling multiple. The training set is constructed by super-resolution face images and low-resolution face images, and the dual-branch dynamic collaborative network is trained using the training set. The first convolution layer is a normal convolution with a convolution kernel of 3×3. A 3×3 normal convolution is used to extract the low-resolution face image I LR Obtain shallow features F s .

[0040] In the embodiment of the present invention, the acquired face image is processed to convert the original high spatial resolution face image I HR ∈R Ht×Wt×3 (i.e. super-resolution face image) downsampled by t times to obtain low spatial resolution face image I LR ∈R H×W×3 (i.e. low-resolution face images). The low-spatial-resolution face images are input into the multi-path attention aggregation module, and the face images are processed in blocks to reduce the overall computational complexity of the network model and explore feature representations involving different channels and position information to extract global facial features. The low-spatial-resolution face images are simultaneously input into the multi-scale dynamic calibration module to learn features of different local positions of the face images, adaptively establish information interaction between multi-dimensional space and channels, and extract local facial features. In the dynamic feature fusion module, global facial features and local facial features are spliced ​​along the channel dimension, and global structure and local detail representations are learned simultaneously to output fused features. The reconstruction module is used to perform feature splicing on the fused features and shallow features to realize residual learning, and the upsampling operation is used to perform a scale magnification process to obtain upsampled features and reconstruct a super-resolution face reconstruction image.

[0041] Preferably, if Figure 4 As shown, the global feature extraction of the shallow features by multiple multi-path attention aggregation modules to obtain the global features of the face includes:

[0042] Each of the multi-path attention aggregation modules includes a block module, a fully connected layer, a self-attention mechanism module, a multi-channel segmentation attention module and a multi-layer perceptron; the steps of extracting global features through any multi-path attention aggregation module include:

[0043] The global input feature is divided into blocks by the block module to obtain multiple local blocks, the multiple local blocks are linearly aligned by the fully connected layer to obtain linear alignment features, the attention weights of the linear alignment features are calculated by the self-attention mechanism module to obtain weighted global features, the linear alignment features are channel-segmented by the multi-channel segmentation attention module to obtain segmented global features, the weighted global features and the segmented global features are weightedly fused by the multi-layer perceptron, and the weighted fusion features are summed with the shallow features to obtain initial global face features, wherein the initial global face features are represented as follows:

[0044] F EMAAB,G =H MLP (Concat(F MCSAM,G ,F g ))+F s ,

[0045] Among them, F EMAAB,G is the initial local face feature, H MLP (·) is a multi-layer perceptron, F MCSAM,G is the global feature for segmentation, F g is the weighted global feature, F s is a shallow feature, Concat(·) is a weighted fusion;

[0046] The initial global face features output by the current multi-path attention aggregation module are used as the input of the next multi-path attention aggregation module, and the steps of feature extraction of any multi-path attention aggregation module are executed until all multi-path attention aggregation modules have completed the feature extraction operation to obtain the global face features. The global face features are expressed as:

[0047] F G =H EMAAB (F s )=H EMAAB,G (H EMAAB,G-1 (…(H EMAAB,1 (F s ))),

[0048] Among them, F G is the local feature of the face, H EMAAB,G (·) is the Gth multi-scale dynamic calibration module, F s The global input features of the first multi-path attention aggregation module are shallow features.

[0049] Specifically, the shallow feature F s Divided into multiple local blocks, the tensor size of each local block is adjusted to (P 2,C), in order to adapt to the subsequent processing flow, the fully connected layer is used to accurately align the embedding dimension of each local block to obtain the linear alignment feature To ensure the consistency and comparability of features among different local blocks.

[0050] Preferably, the calculating of attention weights on the linear alignment features by the self-attention mechanism module to obtain weighted global features includes:

[0051] The linear alignment features are respectively mapped into a query matrix, a key matrix, and a value matrix. The mapped query matrix, key matrix, and value matrix are respectively divided into multiple heads along the channel dimension of the linear alignment features to obtain a head query matrix, a head key matrix, and a head value matrix. Correlation is calculated between the head query matrix and the head key matrix to obtain a global attention weight. The global attention weight is multiplied by the head value matrix to obtain a weighted global feature.

[0052] Specifically, the linear alignment feature Mapped to the query matrix (Q), key matrix (K) and value matrix (V) of the self-attention mechanism module Transformer, the query matrix Q is obtained i , bond matrix K i Sum matrix V i , expressed as:

[0053]

[0054] Among them, W Q , W K , W K are all learnable weight parameters;

[0055] Then along the feature The channel dimension of the query matrix Q i , bond matrix K i Sum matrix V i Split into p heads, we get and Each head processes the input data independently, thereby capturing information in a different representation subspace. The channel dimension of each head is C / p. The query matrix Q after splitting into p heads is calculated i and bond matrix K i The correlation between The information of different positions is obtained to obtain the global attention weight, and finally the global attention weight is combined with the value matrix V after splitting into p heads. i Multiply them together to get the weighted global feature F g , expressed as:

[0056]

[0057] in, d is the size of the learned feature map.

[0058] In an embodiment of the present invention, the self-attention mechanism module can dynamically allocate attention between different local blocks and capture global context information. The features processed by the self-attention mechanism are fused to form higher-level representations. These representations can capture complex patterns and relationships in the image, thereby achieving excellent performance in various visual tasks.

[0059] Preferably, performing channel segmentation processing on the linear alignment features by the multi-channel segmentation attention module to obtain segmentation global features includes:

[0060] Segmenting the linear alignment feature into multiple channel segmentation features along the channel dimension of the linear alignment feature, performing spatial pyramid pooling operations on the multiple channel segmentation features to obtain multiple context features, performing point-by-point convolution operations on the multiple context features to obtain multiple context channel features, performing point-by-point convolution operations on the multiple context channel features to obtain multiple context refinement features, normalizing the multiple context refinement features using a first activation function, and performing element-by-element summation operations on the multiple normalized features to obtain an initial segmentation global feature;

[0061] The initial segmentation global feature is segmented into multiple new channel segmentation features along the channel dimension of the linear alignment feature, and the multiple channel segmentation features and the multiple new channel segmentation features are element-wise multiplied to obtain a segmentation global feature.

[0062] Specifically, we construct a Multi-Channel Segmentation Attention Module (MCSAM). First, we linearly align the features Split into N channel segmentation features Seg along the channel dimension, where each linear alignment feature The number of output channels of is defined as C, so each channel segmentation feature Seg i The number of output channels is C / N. Then for each segmentation feature Seg i Perform spatial pyramid pooling operations and use pooling windows of different scales to effectively capture segmentation features Seg i The local details and global context information of the segmentation feature Seg are used to enhance the robustness of the network to scale changes. iAfter spatial pyramid pooling, it is first sent to the first point-by-point convolution layer, which fine-tunes the feature channels by independently applying convolution kernels at each spatial position. Then, the features processed by the first layer are sent to the second point-by-point convolution layer to further refine the feature representation. This operation not only improves the feature representation capability, but also increases the computational efficiency of the network model. Finally, the output of the second point-by-point convolution layer is normalized by the Softmax activation function of the first activation function layer. In order to integrate the feature information from different groups, an element-by-element summation operation is used to accumulate the output of each group on an element-by-element basis to fuse the results of N groups to obtain the new channel segmentation feature Seg′, which is expressed as:

[0063] Seg′=H Sum (Seg1,Seg2,...,Seg N ),

[0064] Among them Seg N Represents the feature grouping partitioning of the Nth channel splitting operation.

[0065] Then the new channel segmentation feature Seg′ is divided into N subgroups (Seg1′, Seg2′, ..., Seg i ′,…,Seg N ′), each new channel segmentation feature Seg i ′ are all related to the corresponding N segmentation features Seg i Multiply, using element-wise product operation to calculate. New N grouping features Seg i ′ and the corresponding previous N grouping features Seg i Element-wise multiplication is performed, and then the element-wise summation operation is used to merge each path group into the output of the multi-channel segmentation attention module MCSAM, that is, the segmentation global feature F MCSAM The output of MCSAM is F MCSAM，1 The definition is as follows:

[0066] F MCSAM，G =H Sum (H Ep (Seg1,Seg1′),…,H Ep (Seg N ,Seg N ′)),

[0067] Among them, F MCSAM，G represents the output of the Gth MCSAM, H Ep (·) represents an element-wise product operation.

[0068] In an embodiment of the present invention, the multi-channel segmentation attention module can enhance the diversity of feature representation, increase the ability of the network model to capture multi-scale information and different feature channels, and improve the internal correlation of features through multi-path channel information, so that the network model can adaptively focus on the information of different channel groups in the input, thereby enhancing the network model's perception of key feature information.

[0069] Specifically, finally, the multi-layer perceptron is used to MCSAM，1 With F g The features are weighted and fused with the shallow residual features F s The output features of the first multi-path attention aggregation module EMAAB are added together. The output features of the first EMAAB block are expressed as:

[0070] F EMAAB,1 =H EMAAB,1 (F s )=H MLP (Concat(F MCSAM,1 ,F g ))+F s ,

[0071] F EMAAB,G =H EMAAB,G (F EMAAB,G-1 )=H MLP (Concat(F MCSAM,G ,F g ))+F s ,

[0072] Among them, H MLP (·) represents a multilayer perceptron as a feedforward network, consisting of two fully connected layers.

[0073] In summary, the EMAAB architecture of the efficient multi-path attention aggregation module for global feature expression is expressed as:

[0074] F G =H EMAAB (F s )=H EMAAB,G (H EMAAB,G-1 (…(H EMAAB,1 (F s ))),

[0075] Among them, H EMAAB (·) represents the EMAAB architecture for global feature expression, F G H is the extracted global features of the face, EMAAB,G (·) denotes the G-th multi-path attention aggregation module.

[0076] It should be understood that when multiple multi-path attention aggregation modules extract global features of the face, the first multi-path attention aggregation module H EMAAB,1 The input is the shallow feature F s , the output is the initial face global feature F EMAAB,1 , the second multi-path attention aggregation module H EMAAB,2 The input is the initial face global feature F EMAAB,1 , the output is the second face global feature F EMAAB,2 , and so on, the last multi-path attention aggregation module H EMAAB,G The input is the G-1th face global feature F EMAAB,G-1 , the output is the G-th face global feature F EMAAB,G , that is, the global face feature F G .

[0077] Preferably, if Figure 5 and Figure 6 As shown, the local feature extraction of the shallow features by multiple multi-scale dynamic calibration modules to obtain local features of the face includes:

[0078] Each of the multi-scale dynamic calibration modules includes multiple dynamic residual blocks, a first full-dimensional dynamic convolution layer, a second activation function layer, a second full-dimensional dynamic convolution layer, and a second convolution layer. The step of extracting local features through any multi-scale dynamic calibration module includes:

[0079] Performing depth feature extraction on local input features through multiple dynamic residual blocks to obtain depth features, obtaining dynamic calibration weights based on the first full-dimensional dynamic convolution layer and the second activation function layer, multiplying the dynamic calibration weights by the local input features element-by-element to obtain dynamic calibration features, performing a dynamic convolution operation on the local input features and the dynamic calibration features through the second full-dimensional dynamic convolution layer to obtain detail features, splicing the dynamic calibration features, the depth features, and the detail features, performing a normal convolution operation on the spliced ​​feature map through the second convolution layer to obtain a fusion feature, summing the input features, the depth features, the detail features, and the fusion features element-by-element to obtain initial local facial features, wherein the initial local facial features are represented as follows:

[0080] F MDCB,S =H Sum (F MDCB,S-1 ,F DRB,S ,F Dm ,F F )),

[0081] Among them, F MDCB,S is the initial local face feature, H Sum (·) is the element-by-element sum, FMDCB,S-1 is the output of the S-1th multi-scale dynamic calibration module, F DRB,S is the depth feature, F Dm is the detail feature, F F It is a fusion feature;

[0082] The initial local facial features output by the current multi-scale dynamic calibration module are used as the input of the next multi-scale dynamic calibration module, and the step of feature extraction is executed in any multi-scale dynamic calibration module until all multi-scale dynamic calibration modules complete the feature extraction operation to obtain local facial features. The local facial features are expressed as:

[0083] F L =H MDCB (F s )=H MDCB,S (H MDCB,S-1 ( … (H MDCB,1 (F s ))),

[0084] Among them, F L is the local feature of the face, H MDCB,S (·) is the Sth multi-scale dynamic calibration module, F s The local input features of the first multi-scale dynamic calibration module are shallow features.

[0085] Specifically, by using the dynamic residual block H DRB,S (·) While acquiring deep features, a parallel path calculates the dynamic calibration weight W through a 1×1 full-dimensional dynamic convolution layer and a Sigmiod activation function Dw , and the dynamic calibration weight W Dw With shallow features F S Multiply element by element to get the dynamic calibration feature F Dc After obtaining the dynamic calibration feature F Dc Then, a path is made to pass through a 5×5 full-dimensional dynamic convolution layer to the shallow feature F S and the dynamic calibration feature F Dc Perform dynamic convolution operations to consider more local detail information through a larger receptive field to obtain a more refined feature representation F Dm . The depth feature F is transformed along the channel dimension DRB,S , dynamic calibration feature F Dc and detail features F Dm Splice along the channel dimension and use the ordinary convolution with a convolution kernel size of 3×3 to learn the fused feature representation. F Expressed as:

[0086] F F =Conv 3×3 (Concat(F DRB,S ,F Dc ,F Dm )),

[0087] The shallow feature F S , deep features F DRB,S Detail feature F Dm and fusion feature F F Perform element-by-element addition to obtain the final output feature F of the Multi-Scale Dynamic Calibration Block (MDCB) MDCB,S , expressed as:

[0088] F MDCB,1 =H MDCB,1 (F S )=H Sum (F S ,F DRB,S ,F Dm ,F F )),

[0089] F MDCB,S =H MDCB,S (F MDCB,S-1 )=H Sum (F MDCB,S-1 ,F DRB,S ,F Dm ,F F )),

[0090] Among them, H MDCB,S (·) represents the Sth multi-scale dynamic calibration module MDCB operation.

[0091] In summary, the architecture of the multi-scale dynamic calibration module MDCB for local feature expression is expressed as:

[0092] F L =H MDCB (F s )=H MDCB,S (H MDCB,S-1 ( … (H MDCB,1 (F s ))),

[0093] Among them, H MDCB (·) represents the multi-scale dynamic calibration module for local feature expression, F L Local features of the face extracted by local path.

[0094] It should be understood that when multiple multi-scale dynamic calibration modules extract local features of the face, the first multi-scale dynamic calibration module H MDCB,1 The input is the shallow feature F s , the output is the initial face local feature F MDCB,1 , the second multi-path attention aggregation module H MDCB,2 The input is the initial face local feature F MDCB,1 , the output is the second face local feature F MDCB,2 , and so on, the last multi-path attention aggregation module H MDCB,S The input is the S-1th face local feature F MDCB,S-1 , the output is the Sth face local feature F MDCB,S , that is, the local face feature F L .

[0095] Preferably, the extracting deep features from local input features using a plurality of the dynamic residual blocks to obtain deep features comprises:

[0096] The dynamic residual block includes a third full-dimensional dynamic convolution layer, an average pooling layer, a fourth full-dimensional dynamic convolution layer and a third activation function layer;

[0097] A dynamic convolution operation is performed on the local input features through the third full-dimensional dynamic convolution layer, an adaptive pooling operation is performed on the third dynamic convolution feature through the average pooling layer, a dynamic convolution operation is performed on the adaptive pooling feature through the fourth full-dimensional dynamic convolution layer, the fourth dynamic convolution feature is normalized through the third activation function layer, and an element-by-element multiplication operation is performed on the third dynamic convolution feature and the normalized feature to obtain a deep feature.

[0098] Specifically, the shallow feature F s Input into the path composed of S dynamic residual blocks (DRB), introduce residual learning to avoid the degradation of network performance and extract deep features F DRB,S , expressed as:

[0099]

[0100] Among them, each dynamic residual block H DRB,S (·) Contains multiple 3×3 full-dimensional dynamic convolution layers ODConv 3×3 , an average pooling layer F Avg ,δ represents the Sigmoid activation function operation, H Ep Represents an element-wise product operation.

[0101] The dynamic convolution operation of the full-dimensional dynamic convolution layer gradually applies multiple attentions to the multiplication calculation of the convolution kernel. The calculation expression of the full-dimensional dynamic convolution layer is:

[0102] y=(α w1 ⊙α f1 ⊙α c1 ⊙α s1 ⊙W1+…+α wn ⊙α fn ⊙α cn ⊙α sn ⊙W n )*x, where and Respectively represent c in The input features of the channels are out The output features of the channels have a feature size of h×w, ⊙ represents the multiplication operations on different dimensions in the convolution kernel space, W n Indicated by c out output filter m=1,...,c out The nth convolution kernel composed of * represents the convolution operation, α wn ∈R represents the entire convolution kernel W n The attention coefficient, Assign different attention coefficients to the output convolution filters, For each convolution filter The input channels in the CNN are assigned different attention coefficients, α sn ∈R k×k For each convolution filter Different attention coefficients are assigned to k×k spatial locations.

[0103] In the embodiment of the present invention, the proposed efficient dynamic residual module can adaptively adjust network parameters, thereby enhancing the network model's ability to express complex facial features.

[0104] Preferably, the step of splicing the global face features and the local face features by the dynamic feature fusion module to obtain fused features includes:

[0105] The dynamic feature fusion module includes a fifth full-dimensional dynamic convolution layer, a third convolution layer and a sixth full-dimensional dynamic convolution layer;

[0106] Reshaping the global face features according to the channel dimensions of the shallow features, and performing a dynamic convolution operation on the reshaped features through the fifth full-dimensional dynamic convolution layer to obtain fine-tuned global features;

[0107] Performing a convolution operation on the local features of the face through the third convolution layer to obtain fine-tuned local features;

[0108] The fine-tuned global features and the fine-tuned local features are spliced ​​together, and a dynamic convolution operation is performed on the spliced ​​features through the sixth full-dimensional dynamic convolution layer to obtain a fusion feature.

[0109] Specifically, the dynamic fusion module (DFM) combines the global facial features F G Reshape into the shallow feature F of the input s The tensor size is consistent with the dimension and is passed through a 1×1 full-dimensional dynamic convolution ODConv 1×1 (·) to dynamically fine-tune the features and obtain the fine-tuned global features F G ′, expressed as:

[0110] F G ′=ODConv 1×1 (F G ),

[0111] Among them, F G ′ represents fine-tuning of global features;

[0112] The local features F of the face are obtained through a 3×3 convolutional layer. L Fine-tune and output the fine-tuned local feature F L ';

[0113] Splice and fine-tune the global feature F along the channel dimension G ′ and fine-tuned local features F L ′, learn global and local representations simultaneously, using a 1×1 full-dimensional dynamic convolution ODConv 1×1 (·) Output fusion feature F Mix , expressed as:

[0114] F Mix =ODConv 1×1 (Concat(F G ′,F L ′)),

[0115] Among them, Concat(·) represents the feature concatenation operation along the channel dimension.

[0116] Preferably, the upsampling process is performed on the shallow features and the fusion features by the reconstruction module, and the reconstruction operation is performed on the upsampling process result to obtain a super-resolution face reconstructed image, including:

[0117] The reconstruction module includes a sub-pixel convolution layer and a fourth convolution layer;

[0118] The shallow features and the fusion features are fused to obtain residual features, the residual features are upsampled by the sub-pixel convolution layer to obtain upsampled features, and the upsampled features are convolved by the fourth convolution layer to obtain a super-resolution face reconstructed image.

[0119] Specifically, the fusion feature F Mix With shallow features F s Perform fusion to achieve residual learning and obtain residual feature F R , using efficient sub-pixel convolutional layers to train the residual features F R Upsample to obtain the upsampled feature F Up , expressed as:

[0120] F Up =H Up (F R ),

[0121] Among them, H Up (·) indicates upsampling module;

[0122] The upsampled features F are processed by a reconstruction module composed of 3×3 convolutional layers. Up Reconstruct and output the target SR face image I Sr (i.e. super-resolution face reconstruction image), expressed as:

[0123] I Sr =H Re (F Up )=H CTDCNet (I l ),,

[0124] Among them, H Re (·) represents the reconstruction module, H CTDCNet (·) represents a two-branch dynamic collaboration network.

[0125] Preferably, after the step of obtaining the super-resolution face reconstructed image, the method further comprises:

[0126] The super-resolution face reconstruction image and the super-resolution face image are calculated using a mean absolute error loss function to obtain a mean absolute error loss. The mean absolute error loss function is:

[0127]

[0128] Among them, L(Θ) is the mean absolute error loss, is the super-resolution face reconstruction image, N is the number of images, ‖·‖1 is the L1 paradigm, H CTDCNet (·) is a two-branch dynamic collaborative network, Θ represents the parameter set of the training network model, and the training set Contains N low-resolution face images and corresponding super-resolution face images, is a low-resolution face image, is a super-resolution face image;

[0129] The parameters of the two-branch dynamic collaboration network are optimized by mean absolute error loss.

[0130] It should be understood that when the two-branch dynamic collaborative network is trained through the training set, the L1 paradigm loss function is used to optimize the network depth error and correct the network parameters until the set number of training times is reached, and the network outputs the final super-resolution face reconstructed image.

[0131] In an embodiment of the present invention, the network model structure is optimized by using the loss value, and the weights and bias terms in the network are updated, thereby significantly reducing the number of parameters of the network model while maintaining high reconstructed image quality.

[0132] like Figure 7 As shown, the face super-resolution system based on a two-branch dynamic collaborative network provided by the embodiment of the present invention includes:

[0133] A network construction unit, configured to construct a two-branch dynamic collaborative network, the two-branch dynamic collaborative network comprising a first convolutional layer, a plurality of multi-path attention aggregation modules connected in series, a plurality of multi-path attention aggregation modules connected in series, a dynamic feature fusion module, and a reconstruction module;

[0134] A network processing unit is used to extract shallow features from the imported low-resolution face image through the first convolutional layer, perform global feature extraction on the shallow features through multiple multi-path attention aggregation modules to obtain global face features, perform local feature extraction on the shallow features through multiple multi-scale dynamic calibration modules to obtain local face features, splice the global face features and the local face features through the dynamic feature fusion module to obtain fused features, upsample the shallow features and the fused features through the reconstruction module, and perform reconstruction on the upsampling results to obtain a super-resolution face reconstructed image.

[0135] Compared to existing methods, the face super-resolution method based on the CNN-Transformer dynamic collaborative network in this invention includes an efficient multi-path attention aggregation module for learning global representations and a multi-scale dynamic calibration module for learning local representations, simultaneously focusing on the global structure and local details of the face. Furthermore, to further improve the reconstruction effect, the dual-branch dynamic collaborative network CTDCNet introduces a dynamic fusion module to adaptively and finely fuse features from different branches, dimensions, and depths. This effectively alleviates the problem of static convolutional neural networks ignoring global facial structure information. Furthermore, the dual-branch dynamic collaborative network model in this invention has only 78,300 parameters.

[0136] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0137] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0138] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of units is merely a logical functional division. In actual implementation, other division methods may be used, such as combining or integrating multiple units or modules into another system, or ignoring or not implementing certain features.

[0139] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected based on actual needs to achieve the objectives of the embodiments of the present invention.

[0140] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A face super-resolution method based on a dual-branch dynamic collaborative network, characterized in that: The steps include: Constructing a dual-branch dynamic collaborative network, the dual-branch dynamic collaborative network comprising a first convolutional layer, a plurality of multi-path attention aggregation modules connected in series, a plurality of multi-path attention aggregation modules connected in series, a dynamic feature fusion module, and a reconstruction module; Extracting shallow features from the imported low-resolution face image through the first convolutional layer; Performing global feature extraction on the shallow features through a plurality of the multi-path attention aggregation modules to obtain global features of the face; Performing local feature extraction on the shallow features through a plurality of the multi-scale dynamic calibration modules to obtain local features of the face; The dynamic feature fusion module is used to combine the global face feature and the local face feature to obtain a fused feature; The shallow features and the fusion features are upsampled by the reconstruction module, and a reconstruction operation is performed on the upsampling result to obtain a super-resolution face reconstructed image.

2. The face super-resolution method according to claim 1, characterized in that: The extracting global features of the shallow features by using the multiple multi-path attention aggregation modules to obtain global features of the face includes: Each of the multi-path attention aggregation modules includes a block module, a fully connected layer, a self-attention mechanism module, a multi-channel segmentation attention module and a multi-layer perceptron; the steps of extracting global features through any multi-path attention aggregation module include: The global input feature is processed by the block module to obtain multiple local blocks, the multiple local blocks are linearly aligned by the fully connected layer to obtain linear alignment features, the attention weights of the linear alignment features are calculated by the self-attention mechanism module to obtain weighted global features, the linear alignment features are processed by the multi-channel segmentation attention module to obtain segmentation global features, the weighted global features and the segmentation global features are weighted fused by the multi-layer perceptron, and the weighted fusion features are summed with the shallow features to obtain initial global face features, and the initial global face features are expressed as: F EMAAB,G =H MLP (Concat(F MCSAM,G ,F g ))+F s , Among them, F EMAAB,G is the initial local face feature, H MLP (·) is a multi-layer perceptron, F MCSAM,G is the global feature for segmentation, F g is the weighted global feature, F s is a shallow feature, Concat(·) is a weighted fusion; The initial global face feature output by the current multi-path attention aggregation module is used as the input of the next multi-path attention aggregation module, and the step of feature extraction of any multi-path attention aggregation module is executed until all multi-path attention aggregation modules have completed the feature extraction operation to obtain the global face feature, which is represented by: F G =H EMAAB (F s )=H EMAAB,G (H EMAAB,G-1 (…(H EMAAB,1 (F s ))), Among them, F G is the local feature of the face, H EMAAB,G (·) is the Gth multi-scale dynamic calibration module, F s The global input feature of the first multi-path attention aggregation module is a shallow feature.

3. The face super-resolution method according to claim 2, characterized in that: The step of calculating the attention weight of the linear alignment feature through the self-attention mechanism module to obtain a weighted global feature includes: The linear alignment features are respectively mapped into a query matrix, a key matrix and a value matrix, and the mapped query matrix, key matrix and value matrix are respectively divided into multiple heads along the channel dimension of the linear alignment features to obtain a head query matrix, a head key matrix and a head value matrix, and correlation calculation is performed on the head query matrix and the head key matrix to obtain a global attention weight, and the global attention weight is multiplied by the head value matrix to obtain a weighted global feature.

4. The face super-resolution method according to claim 2, characterized in that: The multi-channel segmentation attention module is used to perform channel segmentation processing on the linear alignment feature to obtain a segmentation global feature, including: The linear alignment feature is segmented into a plurality of channel segmentation features along the channel dimension of the linear alignment feature, a spatial pyramid pooling operation is performed on the plurality of channel segmentation features respectively to obtain a plurality of context features, a point-by-point convolution operation is performed on the plurality of context features to obtain a plurality of context channel features, a point-by-point convolution operation is performed on the plurality of context channel features to obtain a plurality of context refinement features, the plurality of context refinement features are normalized by a first activation function, an element-by-element summation operation is performed on the plurality of normalized features to obtain an initial segmentation global feature; The initial segmentation global feature is segmented into a plurality of new channel segmentation features along the channel dimension of the linear alignment feature, and the plurality of channel segmentation features and the plurality of new channel segmentation features are element-wise multiplied to obtain a segmentation global feature.

5. The face super-resolution method according to claim 1, characterized in that: The extracting local features of the shallow features by using a plurality of the multi-scale dynamic calibration modules to obtain local features of the face includes: Each of the multi-scale dynamic calibration modules includes a plurality of dynamic residual blocks, a first full-dimensional dynamic convolution layer, a second activation function layer, a second full-dimensional dynamic convolution layer, and a second convolution layer. The step of extracting local features through any multi-scale dynamic calibration module includes: Performing deep feature extraction on local input features through multiple dynamic residual blocks to obtain deep features, obtaining dynamic calibration weights based on the first full-dimensional dynamic convolution layer and the second activation function layer, multiplying the dynamic calibration weights by local input features element by element to obtain dynamic calibration features, performing dynamic convolution operations on local input features and the dynamic calibration features through the second full-dimensional dynamic convolution layer to obtain detail features, splicing the dynamic calibration features, the deep features and the detail features, performing ordinary convolution operations on the spliced ​​feature maps through the second convolution layer to obtain fused features, summing the input features, the deep features, the detail features and the fused features element by element to obtain initial local face features, wherein the initial local face features are represented as follows: F MDCB,S =H Sum (F MDCB,S-1 ,F DRB,S ,F Dm ,F F )), Among them, F MDCB,S is the initial local face feature, H Sum (·) is the element-by-element sum, F MDCB,S-1 is the output of the S-1th multi-scale dynamic calibration module, F DRB,S is the deep feature, F Dm is the detail feature, F F For fusion features; The initial local face features output by the current multi-scale dynamic calibration module are used as the input of the next multi-scale dynamic calibration module, and the step of extracting features by any multi-scale dynamic calibration module is executed until all multi-scale dynamic calibration modules complete the feature extraction operation to obtain local face features, wherein the local face features are expressed as: F L =H MDCB (F s )=H MDCB,S (H MDCB,S-1 (…(H MDCB,1 (F s ))), Among them, F L is the local feature of the face, H MDCB,S (·) is the Sth multi-scale dynamic calibration module, F s The local input features of the first multi-scale dynamic calibration module are shallow features.

6. The face super-resolution method according to claim 5, characterized in that: The extracting deep features from the local input features by using the plurality of dynamic residual blocks to obtain the deep features comprises: The dynamic residual block includes a third full-dimensional dynamic convolution layer, an average pooling layer, a fourth full-dimensional dynamic convolution layer and a third activation function layer; The local input features are dynamically convolved through the third full-dimensional dynamic convolution layer, the third dynamic convolution features are adaptively pooled through the average pooling layer, the adaptive pooling features are dynamically convolved through the fourth full-dimensional dynamic convolution layer, the fourth dynamic convolution features are normalized through the third activation function layer, and the third dynamic convolution features and the normalized features are element-by-element multiplied to obtain deep features.

7. The face super-resolution method according to claim 1, characterized in that: The step of splicing the global face feature and the local face feature through the dynamic feature fusion module to obtain a fused feature includes: The dynamic feature fusion module includes a fifth full-dimensional dynamic convolution layer, a third convolution layer and a sixth full-dimensional dynamic convolution layer; Reshaping the global face feature according to the channel dimension of the shallow feature, and performing a dynamic convolution operation on the reshaped feature through the fifth full-dimensional dynamic convolution layer to obtain a fine-tuned global feature; Performing a convolution operation on the local features of the face through the third convolution layer to obtain fine-tuned local features; The fine-tuned global features and the fine-tuned local features are spliced, and a dynamic convolution operation is performed on the spliced ​​features through the sixth full-dimensional dynamic convolution layer to obtain fused features.

8. The face super-resolution method according to claim 1, characterized in that: The upsampling process is performed on the shallow features and the fusion features by the reconstruction module, and the upsampling process result is reconstructed to obtain a super-resolution face reconstructed image, including: The reconstruction module includes a sub-pixel convolution layer and a fourth convolution layer; The shallow features and the fusion features are fused to obtain residual features, the residual features are upsampled by the sub-pixel convolution layer to obtain upsampled features, and the upsampled features are convolved by the fourth convolution layer to obtain a super-resolution face reconstructed image.

9. The face super-resolution method according to claim 1, characterized in that: After the step of obtaining the super-resolution face reconstruction image, the method further includes: The super-resolution face reconstruction image and the super-resolution face image are calculated by using a mean absolute error loss function to obtain a mean absolute error loss, wherein the mean absolute error loss function is: Among them, L(Θ) is the mean absolute error loss, Reconstruct images for super-resolution faces, is a low-resolution face image, is a super-resolution face image, N is the number of images, ‖·‖1 is the L1 paradigm; The parameters of the two-branch dynamic collaboration network are optimized by mean absolute error loss.

10. A face super-resolution system based on a dual-branch dynamic collaborative network, characterized in that: include: A network construction unit, used to construct a dual-branch dynamic collaboration network, wherein the dual-branch dynamic collaboration network includes a first convolutional layer, a plurality of multi-path attention aggregation modules connected in series, a plurality of multi-path attention aggregation modules connected in series, a dynamic feature fusion module, and a reconstruction module; The network processing unit is used to extract shallow features from the imported low-resolution face image through the first convolution layer, perform global feature extraction on the shallow features through multiple multi-path attention aggregation modules to obtain global face features, perform local feature extraction on the shallow features through multiple multi-scale dynamic calibration modules to obtain local face features, splice the global face features and the local face features through the dynamic feature fusion module to obtain fused features, upsample the shallow features and the fused features through the reconstruction module, and reconstruct the upsampling results to obtain a super-resolution face reconstructed image.

Images