A Face Super-Resolution Method and System Based on Two-Branch Dynamic Collaborative Networks

Abstract

This invention provides a face super-resolution method and system based on a dual-branch dynamic collaborative network, belonging to the field of computer vision face image processing technology. The method includes: extracting shallow features from a low-resolution face image through a first convolutional layer; performing global feature extraction on the shallow features using multiple multi-path attention aggregation modules to obtain global face features; extracting local features from the shallow features using multiple multi-scale dynamic calibration modules to obtain local face features; concatenating the global and local face features using a dynamic feature fusion module to obtain fused features; and performing reconstruction operations on the shallow and fused features using a reconstruction module to obtain a super-resolution face reconstruction image. By fusing features from different levels and branches of the network, the ability to restore fine facial details is improved.

Description

Technical Field

[0001] This invention mainly relates to the field of computer vision face image processing technology, specifically to a face super-resolution method and system based on a dual-branch dynamic cooperative network. Background Technology

[0002] In real-life scenarios, differences in hardware, shooting angles, and lighting conditions among facial image acquisition devices often result in poor-quality facial images. Low-quality facial images negatively impact processing tasks such as facial analysis and recognition. Therefore, in facial image super-resolution processing, it is crucial not only to increase the number of pixels and image quality but also to emphasize maintaining the accuracy and realism of facial structure during the super-resolution restoration process.

[0003] In recent years, with the rapid development of deep learning networks, face image super-resolution reconstruction methods based on convolutional neural networks (CNNs) have been mainly divided into image reconstruction methods based on local face features and image reconstruction methods based on global face features. However, image reconstruction methods based on local face features use a block-based strategy, which can damage the overall structural coherence of the face image, resulting in blurring in local areas of the reconstructed image. Image reconstruction methods based on global face features are not only unsuitable for obtaining global information from the entire face image, but also, for techniques that introduce additional face prior information to learn the global facial structure, the prior information is usually derived from low-resolution face images, leading to inaccurate prior information and thus affecting the accuracy of the generated super-resolution face image. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a face super-resolution method and system based on a dual-branch dynamic cooperative network.

[0005] The technical solution of the present invention to solve the above-mentioned technical problems is as follows:

[0006] The face super-resolution method based on a dual-branch dynamic cooperative network includes the following steps:

[0007] A dual-branch dynamic collaborative network is constructed, which includes a first convolutional layer, multiple multi-path attention aggregation modules connected in series, multiple multi-path attention aggregation modules connected in series, a dynamic feature fusion module, and a reconstruction module.

[0008] The first convolutional layer extracts shallow features from the imported low-resolution face image.

[0009] Global features of the face are obtained by extracting global features from the shallow features through multiple multi-path attention aggregation modules.

[0010] Local features of the face are obtained by extracting local features from the shallow features through multiple multi-scale dynamic calibration modules.

[0011] The dynamic feature fusion module combines the global and local facial features to obtain a fused feature.

[0012] The reconstruction module performs upsampling on the shallow features and the fused features, and then performs a reconstruction operation on the upsampling results to obtain a super-resolution face reconstruction image.

[0013] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows:

[0014] A face super-resolution system based on a dual-branch dynamic cooperative network includes:

[0015] A network construction unit is used to construct a dual-branch dynamic collaborative network, which includes a first convolutional layer, multiple multi-path attention aggregation modules connected in series, multiple multi-path attention aggregation modules connected in series, a dynamic feature fusion module, and a reconstruction module.

[0016] The network processing unit is configured to extract shallow features from the imported low-resolution face image through the first convolutional layer, perform global feature extraction on the shallow features through multiple multi-path attention aggregation modules to obtain global face features, perform local feature extraction on the shallow features through multiple multi-scale dynamic calibration modules to obtain local face features, concatenate the global face features and the local face features through the dynamic feature fusion module to obtain fused features, and perform upsampling processing on the shallow features and the fused features through the reconstruction module, and perform reconstruction operation on the upsampling processing result to obtain a super-resolution face reconstruction image.

[0017] The beneficial effects of this invention are as follows: This invention reduces overall computational complexity through a multi-path attention aggregation module and explores feature representations of different channels and location information to obtain global features. Through a multi-scale dynamic calibration module, it can capture and optimize local details of face images at multiple scale levels, adaptively establishing information interaction between multi-dimensional spaces and channels, thereby achieving accurate reconstruction of facial features. Through a dynamic feature fusion module, it integrates local and global features to ensure that the network can comprehensively capture multi-dimensional information of the image. It can adaptively fuse features from different levels and branches, further improving the ability to restore fine facial details. While maintaining high efficiency, it can generate higher-quality face super-resolution images, providing an efficient and high-performance solution in the field of face image super-resolution reconstruction. Attached Figure Description

[0018] Figure 1 A flowchart of a face super-resolution method provided in an embodiment of the present invention;

[0019] Figure 2 This is a structural diagram of the face super-resolution method provided in an embodiment of the present invention;

[0020] Figure 3 This is a structural diagram of a dual-branch dynamic cooperative network provided in an embodiment of the present invention;

[0021] Figure 4 This is a structural diagram of the multi-path attention aggregation module provided in an embodiment of the present invention;

[0022] Figure 5 This is a structural diagram of the dynamic residual module provided in an embodiment of the present invention;

[0023] Figure 6 This is a structural diagram of the multi-scale dynamic calibration module provided in an embodiment of the present invention;

[0024] Figure 7 This is a block diagram of a face super-resolution system provided in an embodiment of the present invention. Detailed Implementation

[0025] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0026] In face image super-resolution processing, it is not only necessary to increase the number of pixels and image quality, but also to emphasize maintaining the accuracy and realism of facial structure during the super-resolution restoration process. This involves effectively enhancing facial details in low-quality face images to produce clearer and more natural super-resolution face images. Successfully executing this task is crucial for improving face image quality, thereby enhancing the performance of subsequent advanced vision tasks.

[0027] In recent years, with the rapid development of deep learning networks, methods based on convolutional neural networks (CNNs) have gradually emerged in the task of super-resolution reconstruction of face images. This trend stems from the fact that CNNs can leverage the superior performance of convolutional layers to achieve local perception of face image data. By sliding convolutional kernels across the image and sharing weights, the network can effectively capture local patterns in the image.

[0028] Similar to traditional face super-resolution reconstruction methods, CNN-based face image super-resolution reconstruction methods are mainly divided into two categories: image reconstruction methods based on local facial features and image reconstruction methods based on global facial features. Image reconstruction methods based on local facial features divide the face image into multiple sub-regions and input these image patches into a CNN to reconstruct lost high-frequency facial details. However, this block-based strategy can impair the overall structural coherence of the face image, leading to blurring in local areas of the reconstructed image. Therefore, researchers have begun to develop global methods to preserve and utilize overall facial structural information to improve reconstruction quality.

[0029] Image reconstruction methods based on global facial features are mainly divided into two types: reconstruction methods based on the entire image and reconstruction methods based on facial structure priors. Reconstruction methods based on the entire image aim to input the entire face image into a CNN to focus on the global facial structure. However, due to the limitations of the receptive field of CNNs, these methods are not very effective in handling long-range dependencies and are therefore not suitable for obtaining global information from the input face image. In contrast, reconstruction methods based on facial structure priors utilize facial parsing maps and facial keypoints as additional facial prior information. By leveraging the facial structure information and facial component locations provided by the prior information, they can better learn the global structure of the face. However, these methods also have some limitations. Since the introduced additional facial prior information usually comes from low-resolution or coarse-resolution face images, the prior information is inaccurate, thus affecting the quality of the generated high-resolution face image. Secondly, the encoding and decoding parts in these methods are usually concatenated. This connection method fails to fully utilize low-level features, and low-level features cannot comprehensively guide the learning of high-level features, thus affecting the performance of face super-resolution tasks.

[0030] Therefore, this invention studies how to effectively balance long-range dependencies, global structure, and local details in face image super-resolution reconstruction methods, proposing an innovative dual-branch model architecture to capture both the global structure and local detail information of face images. To further enhance the ability to restore fine facial features, this invention proposes a face super-resolution technique based on a CNN-Transformer dynamic cooperative network.

[0031] like Figures 1-3 As shown, the face super-resolution method based on a dual-branch dynamic cooperative network provided in this embodiment of the invention includes the following steps:

[0032] A dual-branch dynamic collaborative network is constructed, which includes a first convolutional layer, multiple multi-path attention aggregation modules connected in series, multiple multi-path attention aggregation modules connected in series, a dynamic feature fusion module, and a reconstruction module.

[0033] The first convolutional layer extracts shallow features from the imported low-resolution face image.

[0034] Global features of the face are obtained by extracting global features from the shallow features through multiple multi-path attention aggregation modules.

[0035] Local features of the face are obtained by extracting local features from the shallow features through multiple multi-scale dynamic calibration modules.

[0036] The dynamic feature fusion module combines the global and local facial features to obtain a fused feature.

[0037] The reconstruction module performs upsampling on the shallow features and the fused features, and then performs a reconstruction operation on the upsampling results to obtain a super-resolution face reconstruction image.

[0038] Specifically, a dual-branch dynamic collaboration network (CTDCNet, i.e., a heterogeneous fusion network model based on dynamic collaboration) is constructed based on CNN and Transformer architectures for face image super-resolution. This model adopts a dual-branch architecture, including two main parallel paths, a dynamic feature fusion module, and a reconstruction module. One path uses an efficient multi-path attention aggregation block (EMAAB) based on the Transformer architecture to capture global facial structure information. The other path uses a multi-scale dynamic calibration block (MDCB) based on the CNN architecture, focusing on recovering local facial details through local representations. The dynamic feature fusion module (DFM) aligns Transformer architecture features with CNN architecture features, adaptively fusing features from different network levels and branches to further improve the ability to restore fine facial details. The reconstruction module performs feature concatenation to achieve residual learning and uses an upsampling module for scale magnification to obtain the super-resolution face reconstruction image.

[0039] Understandably, downsampling the imported super-resolution face image to obtain a low-resolution face image includes: extracting face images from video surveillance, performing image cropping and face alignment operations on the face images to obtain the target face image (i.e., the super-resolution face image), and then processing the original high-resolution face image I with a size of Ht×Wt×3 using the data processing module. HR (i.e., the target face image) uses a bicubic interpolation algorithm to obtain its corresponding low-resolution face image I. LR Where H and W are the width and height, respectively, 3 is the number of image channels, and t is the downsampling factor. A training set is constructed from super-resolution and low-resolution face images, and this training set is used to train the dual-branch dynamic collaborative network. The first convolutional layer is a regular convolution with a 3×3 kernel, using a 3×3 regular convolution from the low-resolution face image I... LR Obtaining shallow features F s .

[0040] In this embodiment of the invention, the acquired face image is processed to convert the original high spatial resolution face image I... HR ∈R Ht×Wt×3 (i.e., the super-resolution face image) is downsampled by t times to obtain a low spatial resolution face image I. LR ∈R H×W×3 (i.e., low-resolution face images). The low-spatial-resolution face image is input into a multi-path attention aggregation module, which segments the face image to reduce the overall computational complexity of the network model and explore feature representations involving different channels and location information, extracting global face features. Simultaneously, the low-spatial-resolution face image is input into a multi-scale dynamic calibration module to learn different local location features of the face image, adaptively establishing information interaction between multi-dimensional space and channels, and extracting local face features. In the dynamic feature fusion module, global and local face features are concatenated along the channel dimension, simultaneously learning global structure and local detail representations, outputting fused features. The reconstruction module uses feature concatenation with shallow features to achieve residual learning, and upsampling is used for scale magnification to obtain upsampled features and reconstruct a super-resolution face image.

[0041] Preferably, such as Figure 4 As shown, the step of extracting global features from the shallow features through multiple multi-path attention aggregation modules to obtain global facial features includes:

[0042] Each of the multi-path attention aggregation modules includes a block segmentation module, a fully connected layer, a self-attention mechanism module, a multi-channel segmentation attention module, and a multilayer perceptron; the steps for global feature extraction through any one of the multi-path attention aggregation modules include:

[0043] The global input features are divided into blocks by the block segmentation module to obtain multiple local blocks. These local blocks are then linearly aligned using the fully connected layer to obtain linearly aligned features. Attention weights are calculated on these linearly aligned features using the self-attention mechanism module to obtain weighted global features. These features are then segmented using the multi-channel segmentation attention module to obtain segmented global features. Finally, the weighted global features and the segmented global features are weighted and fused using the multilayer perceptron. The weighted and fused features are then summed with the shallow features to obtain initial global face features. The initial global face features are represented as follows:

[0044] F EMAAB,G =H MLP (Concat(F MCSAM,G ,F g ))+F s ,

[0045] Among them, F EMAAB,G H is the initial local facial feature. MLP (·) represents a multilayer perceptron, F MCSAM,G To segment global features, F g For weighted global features, F s These are shallow features; Concat(·) is a weighted fusion method.

[0046] The initial global face features output by the current multi-path attention aggregation module are used as the input to the next multi-path attention aggregation module, and the feature extraction steps of any one of the multi-path attention aggregation modules are executed until all multi-path attention aggregation modules have completed the feature extraction operation, resulting in the global face features. The global face features are represented as follows:

[0047] F G =H EMAAB (F s )=H EMAAB,G (H EMAAB,G-1 (…(H EMAAB,1 (F s ))),

[0048] Among them, F G H is a local facial feature. EMAAB,G (·) represents the Gth multi-scale dynamic calibration module, F s The global input features of the first multi-path attention aggregation module are shallow features.

[0049] Specifically, the shallow features F s The structure is divided into multiple local blocks, and the tensor size of each local block is adjusted to (P). 2To adapt to subsequent processing, a fully connected layer is used to precisely align the embedding dimensions of each local block, resulting in linear alignment features. This is to ensure the consistency and comparability of features across different local blocks.

[0050] Preferably, the step of calculating attention weights for the linearly aligned features through the self-attention mechanism module to obtain weighted global features includes:

[0051] The linear alignment features are mapped onto the query matrix, key matrix, and value matrix, respectively. The mapped query matrix, key matrix, and value matrix are divided into multiple headers along the channel dimension of the linear alignment features to obtain a header query matrix, a header key matrix, and a header value matrix. The relevance of the header query matrix and the header key matrix is ​​calculated to obtain the global attention weight. The global attention weight is multiplied by the header value matrix to obtain the weighted global feature.

[0052] Specifically, linear alignment features Mapping these to the query matrix (Q), key matrix (K), and value matrix (V) of the self-attention mechanism module Transformer, we obtain the query matrix Q. i Key matrix K i Sum matrix V i , is represented as:

[0053]

[0054] Among them, W Q W K W K All of these are learnable weight parameters;

[0055] Then along the features The channel dimension will query matrix Q i Key matrix K i Sum matrix V i Break it down into p headers, and get and Each head processes the input data independently, thereby capturing information in different representation subspaces. The channel dimension of each head is C / p, and the query matrix Q is calculated after splitting the data into p heads. i Bond matrix K i The correlation between them is aggregated Information from different positions is used to obtain global attention weights. Finally, the global attention weights are combined with the value matrix V after splitting into p heads. i Multiply by the product to obtain the weighted global feature F. g , is represented as:

[0056]

[0057] in, d represents the size of the learned feature map.

[0058] In this embodiment of the invention, the self-attention mechanism module can dynamically allocate attention among different local blocks, capture global contextual information, and the features processed by the self-attention mechanism are fused to form higher-level representations. These representations can capture complex patterns and relationships in the image, thereby achieving excellent performance in various visual tasks.

[0059] Preferably, the step of performing channel segmentation processing on the linearly aligned features through the multi-channel segmentation attention module to obtain segmented global features includes:

[0060] The linear alignment feature is segmented into multiple channel segmentation features along the channel dimension. Spatial pyramid pooling is then performed on each of the multiple channel segmentation features to obtain multiple context features. Pointwise convolution is then performed on the multiple context features to obtain multiple context channel features. Pointwise convolution is then performed on the multiple context channel features to obtain multiple context refinement features. The multiple context refinement features are then normalized using a first activation function. Finally, element-wise summation is performed on the multiple normalized features to obtain the initial global segmentation features.

[0061] The initial global segmentation feature is segmented into multiple new channel segmentation features along the channel dimension of the linear alignment feature. The multiple channel segmentation features and the multiple new channel segmentation features are then multiplied element-wise to obtain the global segmentation feature.

[0062] Specifically, a Multi-Channel Segmentation Attention Module (MCSAM) is constructed. First, linearly aligned features are... The data is segmented along the channel dimension into N channel segmentation features Seg, where each feature is linearly aligned. The number of output channels is defined as C, therefore each channel segmentation feature Seg i The number of output channels is C / N. Then, for each segmentation feature Seg... i Performing spatial pyramid pooling operations, utilizing pooling windows of different scales, can effectively capture segmentation features (Seg). i This enhances the network's robustness to scale changes by incorporating local details and global contextual information. Segmentation features (Seg) iAfter spatial pyramid pooling, the data is first fed into the first pointwise convolutional layer. This layer finely adjusts the feature channels by applying convolutional kernels independently at each spatial location. Next, the features processed by the first layer are fed into the second pointwise convolutional layer to further refine the feature representation. This operation not only improves the feature representation capability but also increases the computational efficiency of the network model. Finally, the output of the second pointwise convolutional layer is normalized using the Softmax activation function of the first activation function layer. To integrate feature information from different groups, an element-wise summation operation is used, accumulating the outputs of each group on an element-wise basis to fuse the results of N groups, resulting in the new channel segmentation feature Seg′, represented as:

[0063] Seg′=H Sum (Seg1,Seg2,...,Seg N ),

[0064] Seg N This represents the feature grouping for the Nth channel segmentation operation.

[0065] The new channel segmentation feature Seg′ is then divided into N subgroups (Seg1′, Seg2′, ..., Seg′). i ′,…,Seg N Seg , each new channel segmentation feature i ′ are all related to the corresponding N previous segmentation features Seg i Multiplication is performed using element-wise multiplication. The new N grouped features Seg i ′ and the corresponding previous N grouping features Seg i Element-wise multiplication is performed, followed by element-wise summation to group and merge each path into the output of the Multi-Channel Segmentation Attention Module (MCSAM), which is the segmentation global feature F. MCSAM The output F of MCSAM MCSAM，1 The definition is as follows:

[0066] F MCSAM，G =H Sum (H Ep (Seg1,Seg1′),…,H Ep (Seg N Seg N ′)),

[0067] Among them, F MCSAM，G H represents the output of the Gth MCSAM. Ep (·) represents element-wise multiplication.

[0068] In this embodiment of the invention, the multi-channel segmentation attention module can enhance the diversity of feature representation, increase the network model's ability to capture multi-scale information and different feature channels, and improve the internal correlation of features through multi-path channel information, enabling the network model to adaptively focus on information from different channel groups in the input, thereby enhancing the network model's ability to perceive key feature information.

[0069] Specifically, F is finally processed through a multilayer perceptron. MCSAM，1 With F g The features are weighted and fused, and then combined with the shallow residual features F. s The features of the first multipath attention aggregation module (EMAAB) are summed. The output features of the first EMAAB block are represented as:

[0070] F EMAAB,1 =H EMAAB,1 (F s )=H MLP (Concat(F MCSAM,1 ,F g ))+F s ,

[0071] F EMAAB,G =H EMAAB,G (F EMAAB,G-1 )=H MLP (Concat(F MCSAM,G ,F g ))+F s ,

[0072] Among them, H MLP (·) indicates a multilayer perceptron as a feedforward network, containing two fully connected layers.

[0073] In summary, the architecture of the efficient multi-path attention aggregation module EMAAB for global feature representation is as follows:

[0074] F G =H EMAAB (F s )=H EMAAB,G (H EMAAB,G-1 (…(H EMAAB,1 (F s ))),

[0075] Among them, H EMAAB (·) denotes the EMAAB architecture used for global feature representation, F G To extract global facial features, H EMAAB,G (·) represents the Gth multipath attention aggregation module.

[0076] It should be understood that when multiple multi-path attention aggregation modules extract global facial features, the first multi-path attention aggregation module H... EMAAB,1 The input is shallow features F s The output is the initial global face features F. EMAAB,1 The second multi-path attention aggregation module H EMAAB,2 The input is the initial global face features F EMAAB,1 The output is the global feature F of the second face. EMAAB,2 And so on, until the last multi-path attention aggregation module H... EMAAB,G The input is the global face feature F of the G-1th face. EMAAB,G-1 The output is the global face feature F of the Gth face. EMAAB,G That is, the global facial features F G .

[0077] Preferably, such as Figure 5 and Figure 6 As shown, the process of extracting local features from the shallow features using multiple multi-scale dynamic calibration modules to obtain local facial features includes:

[0078] Each of the multi-scale dynamic calibration modules includes multiple dynamic residual blocks, a first full-dimensional dynamic convolutional layer, a second activation function layer, a second full-dimensional dynamic convolutional layer, and a second convolutional layer. The steps for local feature extraction through any one of the multi-scale dynamic calibration modules include:

[0079] Deep features are extracted from local input features using multiple dynamic residual blocks to obtain deep features. Dynamic calibration weights are obtained based on the first full-dimensional dynamic convolutional layer and the second activation function layer. The dynamic calibration weights are multiplied element-wise with the local input features to obtain dynamic calibration features. Dynamic convolution operations are performed on the local input features and the dynamic calibration features using the second full-dimensional dynamic convolutional layer to obtain detail features. The dynamic calibration features, the deep features, and the detail features are concatenated. A normal convolution operation is performed on the concatenated feature map using the second convolutional layer to obtain fused features. The input features, the deep features, the detail features, and the fused features are summed element-wise to obtain initial local facial features. The initial local facial features are represented as follows:

[0080] F MDCB,S =H Sum (F MDCB,S-1 ,F DRB,S ,F Dm ,F F )),

[0081] Among them, F MDCB,S H is the initial local facial feature. Sum (·) represents element-wise summation, FMDCB,S-1 For the output of the (S-1)th multi-scale dynamic calibration module, F DRB,S For depth features, F Dm For detailed features, F F Features of fusion;

[0082] The initial local facial features output by the current multi-scale dynamic calibration module are used as the input to the next multi-scale dynamic calibration module, and the feature extraction step of any one of the multi-scale dynamic calibration modules is executed until all multi-scale dynamic calibration modules have completed the feature extraction operation, resulting in local facial features. The local facial features are represented as follows:

[0083] F L =H MDCB (F s )=H MDCB,S (H MDCB,S-1 ( … (H MDCB,1 (F s ))),

[0084] Among them, F L H is a local facial feature. MDCB,S (·) represents the S-th multi-scale dynamic calibration module, F s As shallow features, the local input features of the first multi-scale dynamic calibration module are shallow features.

[0085] Specifically, through the dynamic residual block H DRB,S (·) While acquiring deep features, a parallel path calculates the dynamic calibration weights W through a 1×1 full-dimensional dynamic convolutional layer and a Sigmoid activation function. Dw And the dynamic calibration weight W Dw With shallow features F S Element-wise multiplication yields the dynamically calibrated feature F. Dc After obtaining the dynamically calibrated feature F Dc Then, one path is passed through a 5×5 fully dynamic convolutional layer to process the shallow features F. S and the dynamically calibrated feature F Dc Dynamic convolution operations are performed to consider more local details through a larger receptive field, resulting in a more refined feature representation F. Dm Depth features F along the channel dimension DRB,S Dynamic calibration feature F Dc and detailed features F Dm The concatenation is performed along the channel dimension, and the fused feature representation is learned using a regular convolution with a kernel size of 3×3. The fused feature F is then used to... F Represented as:

[0086] F F =Conv 3×3 (Concat(F DRB,S ,F Dc ,F Dm )),

[0087] shallow features F S Deep features F DRB,S Detail features F Dm and fusion feature F F Element-wise summation is performed to obtain the final output characteristic F of the Multi-Scale Dynamic Calibration Block (MDCB). MDCB,S , is represented as:

[0088] F MDCB,1 =H MDCB,1 (F S )=H Sum (F S ,F DRB,S ,F Dm ,F F )),

[0089] F MDCB,S =H MDCB,S (F MDCB,S-1 )=H Sum (F MDCB,S-1 ,F DRB,S ,F Dm ,F F )),

[0090] Among them, H MDCB,S (·) indicates the operation of the S-th multiscale dynamic calibration module (MDCB).

[0091] In summary, the architecture of the multi-scale dynamic calibration module (MDCB) for local feature representation is as follows:

[0092] F L =H MDCB (F s )=H MDCB,S (H MDCB,S-1 ( … (H MDCB,1 (F s ))),

[0093] Among them, H MDCB (·) represents a multi-scale dynamic calibration module used for local feature representation, F L Local facial features extracted using local paths.

[0094] It should be understood that when multiple multi-scale dynamic calibration modules extract local facial features, the first multi-scale dynamic calibration module H... MDCB,1 The input is shallow features F s The output is the initial local facial features F. MDCB,1 The second multi-path attention aggregation module H MDCB,2 The input is the initial local facial features F MDCB,1 The output is the second face local feature F. MDCB,2 And so on, until the last multi-path attention aggregation module H... MDCB,S The input is the local facial feature F of the (S-1)th face. MDCB,S-1 The output is the local facial features F of the Sth face. MDCB,S That is, the local facial features F L .

[0095] Preferably, the step of extracting deep features from local input features using multiple dynamic residual blocks to obtain deep features includes:

[0096] The dynamic residual block includes a third full-dimensional dynamic convolutional layer, an average pooling layer, a fourth full-dimensional dynamic convolutional layer, and a third activation function layer;

[0097] The local input features are dynamically convolved by the third full-dimensional dynamic convolutional layer, adaptively pooled by the average pooling layer, dynamically convolved by the adaptively pooled features by the fourth full-dimensional dynamic convolutional layer, normalized by the fourth dynamic convolutional features by the third activation function layer, and then element-wise multiplied by the third dynamic convolutional features and the normalized features to obtain the deep features.

[0098] Specifically, the shallow features F s The input is fed into a path consisting of S dynamic residual blocks (DRBs). Residual learning is introduced to avoid network performance degradation in order to extract deep features F. DRB,S , is represented as:

[0099]

[0100] Among them, each dynamic residual block H DRB,S (·) contains multiple 3×3 full-dimensional dynamic convolutional layers ODConv 3×3 An average pooling layer F Avg δ represents the Sigmoid activation function operation, H Ep This indicates an element-wise multiplication operation.

[0101] The dynamic convolution operation of a fully-dimensional dynamic convolutional layer applies multiple attention levels progressively to the multiplication calculation of the convolution kernel. The calculation expression for a fully-dimensional dynamic convolutional layer is as follows:

[0102] y=(α w1 ⊙α f1 ⊙α c1 ⊙α s1 ⊙W1+…+α wn ⊙α fn ⊙α cn ⊙α sn ⊙W n )*x, where, and They respectively represent having c in The input characteristics of each channel and c out The output features of each channel have a size of h×w, where ⊙ represents the multiplication operation in different dimensions of the convolution kernel space, and W n Indicated by c out Output filters m = 1, ..., c out The nth convolutional kernel is formed, where * represents the convolution operation, and α wn ∈R represents the entire convolution kernel W n Attention coefficient Assign different attention coefficients to the output convolutional filter. For each convolution filter Different attention coefficients, α, are assigned to the input channels. sn ∈R k×k For each convolution filter Different attention coefficients are assigned to locations in a k×k spatial space.

[0103] In this embodiment of the invention, the proposed high-efficiency dynamic residual module can adaptively adjust network parameters, thereby enhancing the network model's ability to express complex facial features.

[0104] Preferably, the step of concatenating the global facial features and the local facial features through the dynamic feature fusion module to obtain the fused features includes:

[0105] The dynamic feature fusion module includes a fifth full-dimensional dynamic convolutional layer, a third convolutional layer, and a sixth full-dimensional dynamic convolutional layer;

[0106] The global facial features are reshaped according to the channel dimension of the shallow features. The reshaped features are then dynamically convolved through the fifth full-dimensional dynamic convolutional layer to obtain fine-tuned global features.

[0107] The third convolutional layer performs a convolution operation on the local facial features to obtain fine-tuned local features.

[0108] The fine-tuned global features and the fine-tuned local features are concatenated, and the concatenated features are dynamically convolved through the sixth full-dimensional dynamic convolutional layer to obtain the fused features.

[0109] Specifically, the Dynamic Fusion module (DFM) integrates global facial features into the F... G Reshape into shallow features F of the input s Tensors of uniform dimension are used, and a 1×1 full-dimensional dynamic convolution ODConv is applied. 1×1 (·) is used to dynamically fine-tune the features, resulting in the fine-tuned global feature F. G ′, represented as:

[0110] F G ′=ODConv 1×1 (F G ),

[0111] Among them, F G ′ indicates fine-tuning of global features;

[0112] F is applied to local facial features using a 3×3 convolutional layer. L Perform fine-tuning and output the fine-tuned local features F. L ′;

[0113] Fine-tuning global features F along the channel dimension G ′ and fine-tuning local features F L Simultaneously learning global and local representations, using a 1×1 full-dimensional dynamic convolution ODConv 1×1 (·) Output fusion feature F Mix , is represented as:

[0114] F Mix =ODConv 1×1 (Concat(F G ′,F L ′)),

[0115] Concat(·) represents the connection feature operation along the channel dimension.

[0116] Preferably, the step of upsampling the shallow features and the fused features through the reconstruction module, and then performing a reconstruction operation on the upsampling result to obtain a super-resolution face reconstruction image includes:

[0117] The reconstruction module includes subpixel convolutional layers and a fourth convolutional layer;

[0118] The shallow features and the fused features are fused to obtain residual features. The residual features are then upsampled through the subpixel convolutional layer to obtain upsampled features. Finally, the upsampled features are convolved through the fourth convolutional layer to obtain a super-resolution face reconstruction image.

[0119] Specifically, the fusion feature F Mix With shallow features F s By performing fusion to achieve residual learning, residual features F are obtained. R Using efficient sub-pixel convolutional layers to process residual features F R Upsampling is performed to obtain the upsampled feature F. Up , is represented as:

[0120] F Up =H Up (F R ),

[0121] Among them, H Up (·) indicates the upsampling module;

[0122] The upsampled feature F is processed by a reconstruction module consisting of 3×3 convolutional layers. Up Reconstruct the image and output the target SR face image I. Sr (i.e., super-resolution face reconstruction image), represented as:

[0123] I Sr =H Re (F Up )=H CTDCNet (I l ),

[0124] Among them, H Re (·) indicates a refactoring module, H CTDCNet (·) indicates a two-branch dynamic cooperative network.

[0125] Preferably, after the step of obtaining the super-resolution face reconstruction image, the method further includes:

[0126] The mean absolute error loss is calculated by applying the mean absolute error loss function to both the super-resolution face reconstruction image and the super-resolution face image. The mean absolute error loss function is as follows:

[0127]

[0128] Where L(Θ) is the mean absolute error loss, For super-resolution face reconstruction images, N is the number of images, ||·||1 is the L1 paradigm, and H... CTDCNet (·) represents a two-branch dynamic cooperative network, where Θ represents the parameter set for training the network model, and the training set is... It contains N low-resolution face images and their corresponding super-resolution face images. For low-resolution face images, For super-resolution face images;

[0129] The parameters of the two-branch dynamic cooperative network are optimized using mean absolute error loss.

[0130] It should be understood that when training the dual-branch dynamic collaborative network using the training set, the L1 paradigm loss function is used to optimize the network depth error and correct the network parameters until the set number of training iterations is reached, and the network outputs the final super-resolution face reconstruction image.

[0131] In this embodiment of the invention, the network model structure is optimized by loss value, and the weights and biases in the network are updated, thereby significantly reducing the number of network model parameters while maintaining high reconstructed image quality.

[0132] like Figure 7 As shown, the face super-resolution system based on a dual-branch dynamic cooperative network provided in this embodiment of the invention includes:

[0133] A network construction unit is used to construct a dual-branch dynamic collaborative network, which includes a first convolutional layer, multiple multi-path attention aggregation modules connected in series, multiple multi-path attention aggregation modules connected in series, a dynamic feature fusion module, and a reconstruction module.

[0134] The network processing unit is configured to extract shallow features from the imported low-resolution face image through the first convolutional layer, perform global feature extraction on the shallow features through multiple multi-path attention aggregation modules to obtain global face features, perform local feature extraction on the shallow features through multiple multi-scale dynamic calibration modules to obtain local face features, concatenate the global face features and the local face features through the dynamic feature fusion module to obtain fused features, and perform upsampling processing on the shallow features and the fused features through the reconstruction module, and perform reconstruction operation on the upsampling processing result to obtain a super-resolution face reconstruction image.

[0135] Compared to existing methods, this invention's face super-resolution based on a CNN-Transformer dynamic collaborative network includes an efficient multi-path attention aggregation module that learns global representations and a multi-scale dynamic calibration module that learns local representations, simultaneously focusing on both the global structure and local details of the face. Furthermore, to further improve reconstruction results, the dual-branch dynamic collaborative network CTDCNet introduces a dynamic fusion module for adaptively and finely fusing features from different branches, dimensions, and depths, effectively alleviating the problem of static convolutional neural networks ignoring global facial structure information. Moreover, the dual-branch dynamic collaborative network model of this invention has only 78,300 parameters.

[0136] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0137] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0138] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0139] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0140] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A face super-resolution method based on a dual-branch dynamic cooperative network, characterized in that, Includes the following steps: A dual-branch dynamic collaborative network is constructed, which includes a first convolutional layer, multiple multi-path attention aggregation modules connected in series, multiple multi-scale dynamic calibration modules connected in series, a dynamic feature fusion module, and a reconstruction module. The first convolutional layer extracts shallow features from the imported low-resolution face image. Global features of the face are obtained by extracting global features from the shallow features through multiple multi-path attention aggregation modules. Local features of the face are obtained by extracting local features from the shallow features through multiple multi-scale dynamic calibration modules. The dynamic feature fusion module combines the global and local facial features to obtain a fused feature. The reconstruction module performs upsampling on the shallow features and the fused features, and then performs a reconstruction operation on the upsampling results to obtain a super-resolution face reconstruction image. The step of extracting global features from the shallow features using multiple multi-path attention aggregation modules to obtain global facial features includes: Each of the multi-path attention aggregation modules includes a block segmentation module, a fully connected layer, a self-attention mechanism module, a multi-channel segmentation attention module, and a multilayer perceptron; the steps for global feature extraction through any one of the multi-path attention aggregation modules include: The global input features are divided into blocks by the block segmentation module to obtain multiple local blocks. These local blocks are then linearly aligned using the fully connected layer to obtain linearly aligned features. Attention weights are calculated on these linearly aligned features using the self-attention mechanism module to obtain weighted global features. These features are then segmented using the multi-channel segmentation attention module to obtain segmented global features. Finally, the weighted global features and the segmented global features are weighted and fused using the multilayer perceptron. The weighted and fused features are then summed with the shallow features to obtain initial global face features. The initial global face features are represented as follows: ， in, For initial local facial features, It is a multilayer perceptron. To segment global features, For weighted global features, These are shallow features. For weighted fusion; The initial global face features output by the current multi-path attention aggregation module are used as the input to the next multi-path attention aggregation module, and the feature extraction steps of any one of the multi-path attention aggregation modules are executed until all multi-path attention aggregation modules have completed the feature extraction operation, resulting in the global face features. The global face features are represented as follows: ， in, For local facial features, For the Gth multi-scale dynamic calibration module, The global input features of the first multi-path attention aggregation module are shallow features.

2. The face super-resolution method according to claim 1, characterized in that, The step of calculating attention weights for the linearly aligned features through the self-attention mechanism module to obtain weighted global features includes: The linear alignment features are mapped onto the query matrix, key matrix, and value matrix, respectively. The mapped query matrix, key matrix, and value matrix are divided into multiple headers along the channel dimension of the linear alignment features to obtain a header query matrix, a header key matrix, and a header value matrix. The relevance of the header query matrix and the header key matrix is ​​calculated to obtain the global attention weight. The global attention weight is multiplied by the header value matrix to obtain the weighted global feature.

3. The face super-resolution method according to claim 1, characterized in that, The step of performing channel segmentation processing on the linearly aligned features through the multi-channel segmentation attention module to obtain segmented global features includes: The linear alignment feature is segmented into multiple channel segmentation features along the channel dimension. Spatial pyramid pooling is then performed on each of the multiple channel segmentation features to obtain multiple context features. Pointwise convolution is then performed on the multiple context features to obtain multiple context channel features. Pointwise convolution is then performed on the multiple context channel features to obtain multiple context refinement features. The multiple context refinement features are then normalized using a first activation function. Finally, element-wise summation is performed on the multiple normalized features to obtain the initial global segmentation features. The initial global segmentation feature is segmented into multiple new channel segmentation features along the channel dimension of the linear alignment feature. The multiple channel segmentation features and the multiple new channel segmentation features are then multiplied element-wise to obtain the global segmentation feature.

4. The face super-resolution method according to claim 1, characterized in that, The step of extracting local features from the shallow features using multiple multi-scale dynamic calibration modules to obtain local facial features includes: Each of the multi-scale dynamic calibration modules includes multiple dynamic residual blocks, a first full-dimensional dynamic convolutional layer, a second activation function layer, a second full-dimensional dynamic convolutional layer, and a second convolutional layer. The steps for local feature extraction through any one of the multi-scale dynamic calibration modules include: Deep features are extracted from local input features using multiple dynamic residual blocks to obtain deep features. Dynamic calibration weights are obtained based on the first full-dimensional dynamic convolutional layer and the second activation function layer. The dynamic calibration weights are multiplied element-wise with the local input features to obtain dynamic calibration features. Dynamic convolution operations are performed on the local input features and the dynamic calibration features using the second full-dimensional dynamic convolutional layer to obtain detail features. The dynamic calibration features, the deep features, and the detail features are concatenated. A normal convolution operation is performed on the concatenated feature map using the second convolutional layer to obtain fused features. The input features, the deep features, the detail features, and the fused features are summed element-wise to obtain initial local facial features. The initial local facial features are represented as follows: ， in, For initial local facial features, To sum element by element, For the first The output of a multi-scale dynamic calibration module For depth features, For detailed features, Features of fusion; The initial local facial features output by the current multi-scale dynamic calibration module are used as the input to the next multi-scale dynamic calibration module, and the feature extraction step of any one of the multi-scale dynamic calibration modules is executed until all multi-scale dynamic calibration modules have completed the feature extraction operation, resulting in local facial features. The local facial features are represented as follows: ， in, For local facial features, For the Sth multi-scale dynamic calibration module, As shallow features, the local input features of the first multi-scale dynamic calibration module are shallow features.

5. The face super-resolution method according to claim 4, characterized in that, The process of extracting deep features from local input features using multiple dynamic residual blocks to obtain deep features includes: The dynamic residual block includes a third full-dimensional dynamic convolutional layer, an average pooling layer, a fourth full-dimensional dynamic convolutional layer, and a third activation function layer; The local input features are dynamically convolved by the third full-dimensional dynamic convolutional layer, adaptively pooled by the average pooling layer, dynamically convolved by the adaptively pooled features by the fourth full-dimensional dynamic convolutional layer, normalized by the fourth dynamic convolutional features by the third activation function layer, and then element-wise multiplied by the third dynamic convolutional features and the normalized features to obtain the deep features.

6. The face super-resolution method according to claim 1, characterized in that, The process of concatenating the global facial features and the local facial features through the dynamic feature fusion module to obtain fused features includes: The dynamic feature fusion module includes a fifth full-dimensional dynamic convolutional layer, a third convolutional layer, and a sixth full-dimensional dynamic convolutional layer; The global facial features are reshaped according to the channel dimension of the shallow features. The reshaped features are then dynamically convolved through the fifth full-dimensional dynamic convolutional layer to obtain fine-tuned global features. The third convolutional layer performs a convolution operation on the local facial features to obtain fine-tuned local features. The fine-tuned global features and the fine-tuned local features are concatenated, and the concatenated features are dynamically convolved through the sixth full-dimensional dynamic convolutional layer to obtain the fused features.

7. The face super-resolution method according to claim 1, characterized in that, The step of upsampling the shallow features and the fused features using the reconstruction module, and then performing a reconstruction operation on the upsampling result to obtain a super-resolution face reconstruction image, includes: The reconstruction module includes subpixel convolutional layers and a fourth convolutional layer; The shallow features and the fused features are fused to obtain residual features. The residual features are then upsampled through the subpixel convolutional layer to obtain upsampled features. Finally, the upsampled features are convolved through the fourth convolutional layer to obtain a super-resolution face reconstruction image.

8. The face super-resolution method according to claim 1, characterized in that, After the step of obtaining the super-resolution face reconstruction image, the method further includes: The mean absolute error loss is calculated by applying the mean absolute error loss function to both the super-resolution face reconstruction image and the super-resolution face image. The mean absolute error loss function is as follows: ， in, For the average absolute error loss, To reconstruct super-resolution face images, For low-resolution face images, For super-resolution face images, N For the number of images, for Paradigm; The parameters of the two-branch dynamic cooperative network are optimized using mean absolute error loss.

9. A face super-resolution system based on a dual-branch dynamic cooperative network, characterized in that, include: A network construction unit is used to construct a dual-branch dynamic collaborative network, which includes a first convolutional layer, multiple multi-path attention aggregation modules connected in series, multiple multi-scale dynamic calibration modules connected in series, a dynamic feature fusion module, and a reconstruction module. The network processing unit is configured to extract shallow features from the imported low-resolution face image through the first convolutional layer, perform global feature extraction on the shallow features through multiple multi-path attention aggregation modules to obtain global face features, perform local feature extraction on the shallow features through multiple multi-scale dynamic calibration modules to obtain local face features, stitch the global face features and the local face features through the dynamic feature fusion module to obtain fused features, and perform upsampling processing on the shallow features and the fused features through the reconstruction module, and perform reconstruction operation on the upsampling processing result to obtain a super-resolution face reconstruction image. The step of extracting global features from the shallow features using multiple multi-path attention aggregation modules to obtain global facial features includes: Each of the multi-path attention aggregation modules includes a block segmentation module, a fully connected layer, a self-attention mechanism module, a multi-channel segmentation attention module, and a multilayer perceptron; the steps for global feature extraction through any one of the multi-path attention aggregation modules include: The global input features are divided into blocks by the block segmentation module to obtain multiple local blocks. These local blocks are then linearly aligned using the fully connected layer to obtain linearly aligned features. Attention weights are calculated on these linearly aligned features using the self-attention mechanism module to obtain weighted global features. These features are then segmented using the multi-channel segmentation attention module to obtain segmented global features. Finally, the weighted global features and the segmented global features are weighted and fused using the multilayer perceptron. The weighted and fused features are then summed with the shallow features to obtain initial global face features. The initial global face features are represented as follows: ， in, For initial local facial features, It is a multilayer perceptron. To segment global features, For weighted global features, These are shallow features. For weighted fusion; The initial global face features output by the current multi-path attention aggregation module are used as the input to the next multi-path attention aggregation module, and the feature extraction steps of any one of the multi-path attention aggregation modules are executed until all multi-path attention aggregation modules have completed the feature extraction operation, resulting in the global face features. The global face features are represented as follows: ， in, For local facial features, For the Gth multi-scale dynamic calibration module, The global input features of the first multi-path attention aggregation module are shallow features.

Images