Non-aligned face image super-resolution method based on layered structure perception space conversion network

Abstract

The application discloses a non-aligned face image super-resolution method based on a layered structure perception space conversion network, and comprises the following steps: performing random angle rotation, face region cutting and size adjustment on a public face dataset, constructing a non-aligned low-resolution face / aligned high-resolution face image pair dataset, and dividing a training set and a test set; constructing a non-aligned face image super-resolution model based on a layered structure perception space conversion network; training the non-aligned face image super-resolution model based on the layered structure perception space conversion network on the training set; and performing super-resolution on non-aligned low-resolution face images in the test set. The non-aligned face image super-resolution model constructed by the application can overcome the problems of structure distortion and details accompanying artificial artifacts of a generated face image under the influence of a non-aligned factor in an existing face super-resolution method, so that the application can reconstruct an aligned and high-fidelity high-resolution face image.

Description

Technical Field

[0001] This invention belongs to the field of image processing and face super-resolution technology, specifically relating to a method for super-resolution of unaligned face images based on a hierarchical perceptual spatial transformation network. Background Technology

[0002] Face super-resolution technology is an important research direction in the field of deep learning in recent years. Specifically targeting the characteristics of facial images, it restores low-resolution facial images into high-quality, detailed, high-resolution images to meet the high image quality requirements in various practical applications. With the widespread use of social media and mobile devices, facial images, as a special type of data containing rich information, have become increasingly important. High-definition, high-detail facial images play a crucial role in various applications such as facial recognition, security surveillance, virtual reality, augmented reality, entertainment selfies, and beauty filters. It can be said that facial images have become integrated into every aspect of our lives. Especially in these applications, the clarity and detail of the image directly affect the accuracy of recognition, the smoothness of the user experience, and the robustness of the system.

[0003] High-quality face images are a prerequisite for the successful development and application of face image processing research. However, in real-world imaging scenarios, factors such as shooting angle, pose variations, and device imaging quality easily lead to the acquisition of low-resolution face images that are misaligned. Existing deep learning-based face image super-resolution methods rely on pre-aligned images whose interocular distance (IOL) matches the sample distribution in the dataset during training. This results in a significant decrease in reconstruction quality when processing misaligned face images with large IOL deviations or significant pose variations, making it difficult to obtain high-quality restoration results. Therefore, designing a super-resolution method suitable for misaligned low-resolution face images, effectively improving its reconstruction capability for vertical, high-quality face images, is a research topic of significant value.

[0004] Existing face image super-resolution methods can be broadly categorized into two main types: traditional super-resolution algorithms and deep learning-based super-resolution algorithms, which have emerged in recent years with the rapid development of deep learning technology. Traditional super-resolution algorithms primarily rely on image processing, statistical models, and prior knowledge. These methods achieved some success in early research, their core idea being to utilize statistical information in the image and space to recover high-quality, high-resolution images through specific operations or optimization strategies. Deep learning-based super-resolution algorithms are characterized by using deep neural networks to learn the mapping relationship from low-resolution to high-resolution. Although current face super-resolution tasks have made some progress, and related algorithms have shown good reconstruction performance on aligned face images, they still face many key challenges when processing unaligned low-resolution face images, especially in terms of inconsistencies in facial geometry. Most existing face super-resolution methods are highly dependent on the alignment preprocessing of face images, assuming that the input image and training data have high consistency in facial key point positions and interocular distances. Therefore, when the input face image contains misaligned features, existing methods struggle to accurately locate facial structural regions, leading to the following problems: ① Accumulation of facial geometric alignment errors: Because the eyes are not on the same horizontal line, traditional alignment algorithms fail to adequately model information such as interocular distance and facial contours, resulting in structural distortion during inference. ② Failure of spatial attention mechanisms: Misalignment causes significant fluctuations in the positions of local facial regions (such as eyes, nose, and mouth), weakening the ability to extract semantic region features based on spatial priors or attention mechanisms. ③ Inconsistent training and testing distributions: Existing datasets mostly consist of aligned faces, making it difficult for models to generalize their learned representations to real-world images of misaligned faces with varying poses and angles. Therefore, effectively modeling and adapting to the complex and variable facial geometry in misaligned face images to improve the perception of key regions and texture restoration during reconstruction is a crucial challenge for face image super-resolution methods. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, this invention provides a method for super-resolution of unaligned face images based on a hierarchical perceptual spatial transformation network. Based on facial structure priors and spatial transformation networks, this invention studies a more accurate, efficient, and universal face super-resolution method to achieve super-resolution of unaligned face images.

[0006] To achieve the above objectives, the technical solution of the present invention is: a method for super-resolution of unaligned face images based on a hierarchical perceptual spatial transformation network, comprising the following steps:

[0007] S1: Construct the structure-aware spatial transformation module, the channel-space attention module, and the upsampling module respectively;

[0008] S2: A cascaded perceptual spatial transformation module and an upsampling module are used to construct a coarse alignment super-resolution module and a fine alignment super-resolution module; a cascaded perceptual spatial transformation module and a channel-space attention module are used to construct a recursive alignment enhancement module; then, a cascaded input convolutional layer, a coarse alignment super-resolution module, a recursive alignment enhancement module, a fine alignment super-resolution module, and an image reconstruction module are used to construct a hierarchical perceptual spatial transformation face super-resolution network;

[0009] S3: Randomly rotate and align the high-resolution face images in the public dataset according to the feature points, then crop out the corresponding face regions and perform Bicubic downsampling to construct unaligned low-resolution / aligned high-resolution face image pairs, and then divide them into training set and test set;

[0010] S4: Train a face super-resolution model based on the training set;

[0011] S5: Input the images in the test set into the trained unaligned face image super-resolution model based on the hierarchical perceptual spatial transformation network, and output the reconstructed high-resolution face image.

[0012] Furthermore, the construction process of the structure-aware spatial transformation module in step S1 is as follows:

[0013] A1: Constructing a face prior prediction module: An input convolutional layer (7×7 kernel size, stride 2, padding 3) is used for initial feature encoding of the input features. Subsequently, multiple hourglass network modules are stacked to achieve top-down and bottom-up multi-scale feature flow. Within each hourglass module, global semantic information is extracted through downsampling, spatial resolution is restored through upsampling, and features are fused element-wise with those from high-resolution paths to achieve joint modeling of local details and global contextual information. The output features of each hourglass network module undergo channel adjustment via convolutional layers and normalization operations. Finally, a 1×1 convolution is used to generate heatmap features of facial key points, and ground truth supervision is applied to these heatmap features. To enhance cross-module information flow, the output of each hourglass module is accumulated with the current features and the channel-mapped heatmap prediction before being fed into the next module, thus forming a deep supervision mechanism and multi-level residual connections.

[0014] A2: Construct a prior-guided localization network: Stack one input convolutional layer, one face prior prediction module, and one output convolutional layer. Concatenate the face features encoded by the input convolutional layer and the facial structure prior features predicted by the face prior prediction module in the channel dimension, and input them into the output convolutional layer together.

[0015] A3: Constructing a mesh generator: Stack one input linear transformation layer, one PReLU activation function, and one output linear transformation layer to generate an affine transformation matrix;

[0016] A4: Applying an affine transformation matrix to the input face features achieves feature alignment. Specifically, the target output feature pixels are obtained by sampling sub-pixels at specific locations on the input feature map. These sampling locations typically correspond to non-integer pixel coordinates, thereby achieving a continuous spatial mapping of the input features.

[0017] Furthermore, the construction process of the channel-space attention module in step S1 is as follows:

[0018] B1: Construct a channel attention module, which receives input features and performs average pooling and max pooling respectively to obtain two channel attention features; input the two channel attention features into a multilayer perceptron module (containing one receiving convolutional layer, one PReLU layer and one output convolutional layer) to output channel attention enhancement features; sum the two channel attention enhancement features and pass them through a sigmoid function to obtain channel attention weights, which are then multiplied pixel by pixel with the input features to construct effective channel enhancement features;

[0019] B2: Construct a spatial attention module, perform average pooling and max pooling on the effective channel enhancement features along the channel dimension to obtain spatial enhancement features; after concatenation along the channels, pass through the output convolutional layer (convolutional kernel size is 7×7, stride is 2, padding is 3) and pass through Sigmoid to obtain a spatial attention map, multiply it element-wise with the effective channel enhancement features to further realize spatial enhancement, and serve as the final output of the module.

[0020] Furthermore, the construction process of the face super-resolution network based on the hierarchical perceptual spatial transformation network in step S2 is as follows:

[0021] C1: Constructing the image input module: cascaded convolutional layers (3×3 kernel size, stride 1, padding 1, output channels 64), batch processing layer (GroupNorm), activation function (PRELU), and upsampling layer (nearest neighbor interpolation, 2x magnification).

[0022] C2: Constructing a coarse-aligned super-resolution module: a cascaded perceptual spatial transformation module and an upsampling layer;

[0023] C3: Constructing a recursive alignment enhancement module: This module consists of three cascaded sub-modules. Each sub-module comprises a structure-aware spatial transformation module and a channel-spatial attention module, connected end-to-end. The output of the i-th sub-module serves as the input of the (i+1)-th sub-module, thus achieving recursive feature alignment and enhancement. To ensure consistency between input and output features in the channel dimension, each module is terminated by a channel adjustment convolutional layer.

[0024] C4: Constructing a fine-aligned super-resolution module: a cascaded perceptual spatial transformation module and an upsampling layer;

[0025] C5: Constructing an image reconstruction module: cascading three consecutive residual modules and one output convolutional layer (convolutional kernel size is 3×3, stride is 1, padding is 1, and output channels are 3).

[0026] Furthermore, the construction process in step S3 is specifically as follows:

[0027] E1: Preprocess the published dataset by dividing all face images in the dataset into [-20] . o +20 o Randomly rotate to generate unaligned face images, while retaining the corresponding aligned face images;

[0028] E2: Detect and crop the face regions of the unaligned and aligned face images respectively, and construct a high-resolution face image in which the face region occupies more than 60% of the entire image;

[0029] E3: For each high-resolution unaligned face image, use the Bicubic interpolation method to downsample and obtain the corresponding low-resolution face image;

[0030] E4: Pair the generated unaligned low-resolution face images with aligned high-resolution face images to construct training or test sample pairs;

[0031] E5: Store the training and test sets separately according to the training and test set division rules (90% training and 10% test) to ensure that each unaligned low-resolution face image corresponds to a unique high-resolution aligned face image.

[0032] Furthermore, the construction process in step S4 is specifically as follows:

[0033] F1: Construct the mean squared error as the loss function;

[0034] F2: Construct a perceptual loss function based on the VGG19 model;

[0035] F3: Construct a face landmark heatmap loss function based on a 2D-FAN model;

[0036] F4: Constructing the adversarial loss function;

[0037] F5: Initialize the parameters of the hierarchical perceptual spatial transformation network;

[0038] F6: Combine all the above loss functions to perform multi-level supervised training on the output of the hierarchical perceptual spatial transformation network;

[0039] F7: Set the learning rate, number of model training iterations, and number of batch training samples;

[0040] F8: Train the hierarchical perceptual spatial transformation network using unaligned low-resolution images and corresponding aligned high-resolution images from the training dataset, and update the network parameters using an optimization algorithm;

[0041] F9: Train the super-resolution model for unaligned face images until the number of iterations reaches a preset value.

[0042] Furthermore, the perceptual loss function based on the VGG19 model in step F2... for:

[0043]

[0044] in, This represents the joint distribution of the super-resolution results and the corresponding ground truth faces of each sub-network. This represents the feature map extracted by the pre-trained VGG19 model from the convolutional layer (after activation) before all its max-pooling layers. Represents a true high-resolution image. This represents the generated super-resolution image.

[0045] Furthermore, the face landmark heatmap loss function based on the 2D-FAN network in step F3... for:

[0046]

[0047] Where K represents the number of key points (e.g., 68 facial key points). This represents the super-resolution result generated by the structure prior prediction module acting on the fine-grained complex atom network and the global optimization sub-network, showing the predicted heatmap of the k-th key point. The ground truth heatmap of the k-th key point is generated by applying a 2D-FAN model to a ground truth high-resolution face image.

[0048] Furthermore, the adversarial loss function in step F4 is:

[0049]

[0050]

[0051] in, To constrain the adversarial loss function of the hierarchical perceptual spatial transformation network, Let E be the adversarial loss function constraining the discriminator, used to train the discriminator to distinguish between real and generated images, and let D be the average value over the samples. This represents the output of the discriminator model, indicating the probability that the input is "true" (typically in the range [0, 1]).

[0052] Compared with the prior art, the significant advantages of the present invention are: the face super-resolution model constructed by the present invention overcomes the problems of image structure distortion and artificial artifacts under the influence of non-alignment factors in existing face super-resolution methods, enabling the present invention to reconstruct aligned high-resolution face images with complete structure, clear details and high realism. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0054] Figure 2 This is a schematic diagram of the structure of an unaligned face image super-resolution model based on a hierarchical perceptual spatial transformation network.

[0055] Figure 3 This is a schematic diagram of the structure of a structure-aware spatial transformation network.

[0056] Figure 4 This is a schematic diagram of the channel-space attention module structure.

[0057] Figure 5 This is a diagram illustrating the comparison of test results with those of other similar methods. Detailed Implementation

[0058] This invention proposes a method for super-resolution of unaligned face images based on a hierarchical perceptual spatial transformation network. The method includes: performing random angle rotation, face region cropping, and size adjustment on a publicly available face dataset to construct a dataset of unaligned low-resolution face images / aligned high-resolution face images, dividing it into training and test sets; constructing an unaligned face image super-resolution model based on a hierarchical perceptual spatial transformation network; training the unaligned face image super-resolution model based on the hierarchical perceptual spatial transformation network on the training set; and performing super-resolution on the unaligned low-resolution face images in the test set. This invention constructs a face image super-resolution model based on a hierarchical perceptual spatial transformation network, whose super-resolution network consists of a coarse alignment super-resolution module, a recursive alignment enhancement module, and a fine alignment super-resolution module. The unaligned face image super-resolution model constructed in this invention overcomes the problems of structural distortion and artificial artifacts in face images generated under the influence of unalignment factors in existing face super-resolution methods, enabling this invention to reconstruct aligned and high-fidelity high-resolution face images.

[0059] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments.

[0060] This invention provides a method for super-resolution of unaligned face images based on a hierarchical perceptual spatial transformation network, referring to... Figure 1 It includes the following steps:

[0061] Step 1: Construct a structure-aware spatial transformation module, which includes steps 1.1-1.3:

[0062] Step 1.1: Construct a face prior prediction module based on the stacked hourglass network module, generate heat map features of face key points, and set ground truth supervision for the heat map features.

[0063] Step 1.1: Constructing the Prior-Guided Localization Network: The prior-guided localization network consists of one input convolutional layer, one face prior estimation module, and one output convolutional layer, as shown in the specific structure below. Figure 3 As shown;

[0064] Step 1.2: Constructing the mesh generator: The mesh generator consists of an input linear transformation layer, a PReLU activation function, and an output linear transformation layer;

[0065] Step 1.3: Based on the affine transformation matrix generated by the mesh generator, perform spatial transformation on the input face features. Specifically, each output pixel is obtained by sampling sub-pixel locations at specific positions on the input feature map. These sampling locations typically correspond to non-integer pixel coordinates, thereby achieving continuous spatial mapping of the input features.

[0066] Step 2: Construct the channel-space attention module, which includes steps 2.1-2.2:

[0067] Step 2.1: Construct a channel attention module to receive features. Perform average pooling and max pooling respectively to obtain the dimension as The channel attention features are obtained; they are input into a shared multilayer perceptron to output channel attention enhancement features; the two channel attention enhancement features are summed and passed through a sigmoid function to obtain the channel attention weights, which are then multiplied pixel-by-pixel with the input features to construct effective channel enhancement features;

[0068] Step 2.2: Construct a spatial attention module, performing average pooling and max pooling on the effective channel enhancement features along the channel dimension to obtain a dimension of Features; after concatenation along the channels, the spatial attention map is obtained through the output convolutional layer and then through the sigmoid function. This attention map is then multiplied element-wise with the effective channel enhancement features to further enhance spatial attention, and serves as the final output of the module. The specific structure is as follows: Figure 4 As shown.

[0069] Step 3: Construct a coarsely aligned super-resolution module, which includes step 3.1:

[0070] Step 3.1: Stack the structure-aware spatial transformation module and the upsampling layer. The structure-aware spatial transformation module is cascaded with three consecutive sets of convolutional layers, a batch processing layer (GroupNorm), a pooling layer (MaxPool2d), an activation function (PRELU), and a grid generator. It outputs an affine transformation matrix (3 rows * 2 columns), which is applied to the input face features. Feature alignment is achieved through grid sampling. The specific structure is as follows: Figure 2 As shown.

[0071] Step 4: Construct the recursive alignment enhancement module, which includes step 4.1:

[0072] Step 4.1: Construct a recursive alignment enhancement module based on the structure-aware spatial transformation module, the channel-spatial attention module, and the intermediate output convolutional layer. The recursive alignment enhancement module consists of three cascaded sub-modules, each consisting of a structure-aware spatial transformation module and a channel-spatial attention module sequentially. The three sub-modules are connected end-to-end, with the output of the i-th group serving as the input of the (i+1)-th group, thus achieving recursive feature alignment and enhancement. To ensure consistency between input and output features in the channel dimension, each module is terminated by a channel adjustment convolutional layer, as shown in the specific structure below. Figure 2 As shown.

[0073] Step 5: Construct a fine-grained super-resolution decoder, which includes step 5.1:

[0074] Step 5.1: Stack the structure-aware spatial transformation module and the upsampling layer. The structure-aware spatial transformation module cascades the input convolutional layer, the face prior prediction module, five consecutive sets of convolutional layers, the batch processing layer (GroupNorm), the pooling layer (MaxPool2d), the activation function (PRELU), and the grid generator, outputting an affine transformation matrix (3 rows * 2 columns). Further feature alignment is achieved through grid sampling. The specific structure is as follows: Figure 2 As shown.

[0075] Step 6: Construct an unaligned face image super-resolution model based on a hierarchical perceptual spatial transformation network, which includes step 6.1:

[0076] Step 6.1: Construct the input convolutional layer and upsampling layer, cascade a coarse alignment super-resolution module, a recursive alignment enhancement module, a fine alignment super-resolution module, a residual module, and an output convolutional layer to generate the final high-quality face image.

[0077] Step 7: Preprocess the dataset, splitting it into the dataset and the test set, which includes steps 7.1-7.5:

[0078] Step 7.1: Preprocess the published dataset by dividing all face images in the dataset into [-20] . o +20o Randomly rotate to generate unaligned face images, while retaining the corresponding aligned face images;

[0079] Step 7.2: Detect and crop the face regions of the unaligned face image and the aligned face image respectively, and construct a high-resolution face image in which the face region occupies more than 60% of the entire image;

[0080] Step 7.3: For each high-resolution unaligned face image, use the Bicubic interpolation method to downsample it to obtain the corresponding low-resolution face image;

[0081] Step 7.4: Pair the generated unaligned low-resolution face images with aligned high-resolution face images to construct training or test sample pairs;

[0082] Step 7.5: Store the training and test sets separately according to the training and test set division rules (90% training and 10% test) to ensure that each non-aligned low-resolution face image corresponds to a unique high-resolution aligned face image.

[0083] Step 8: Train an unaligned face image super-resolution model based on a hierarchical perceptual spatial transformation network, which includes steps 8.1-8.8:

[0084] Step 8.1: Construct a loss function based on the mean square error to impose pixel-level constraints on the final generated high-resolution aligned face image;

[0085] Step 8.2: Construct the perceptual loss function based on the VGG19 model, the specific structure of which is as follows:

[0086]

[0087] in, This represents the joint distribution of the super-resolution results and the ground truth face. This represents the feature map extracted by the pre-trained VGG19 model from the convolutional layer (after activation) before all its max-pooling layers. Represents a true high-resolution image. This represents the generated super-resolution image.

[0088] Step 8.3: Construct a face landmark heatmap loss function based on a 2D-FAN model, the specific structure of which is as follows:

[0089]

[0090] Where K represents the number of key points (e.g., 68 facial key points). This represents the super-resolution result generated by the structure prior prediction module acting on the fine-grained complex atom network and the global optimization sub-network, showing the predicted heatmap of the k-th key point. The ground truth heatmap of the k-th key point is generated by applying a 2D-FAN model to a ground truth high-resolution face image.

[0091] Step 8.4: The adversarial loss function is:

[0092]

[0093]

[0094] in, To constrain the adversarial loss function of the hierarchical perceptual spatial transformation network, Let E be the adversarial loss function constraining the discriminator, used to train the discriminator to distinguish between real and generated images, and let D be the average value over the samples. This represents the output of the discriminator model, indicating the probability that the input is "true" (typically in the range [0, 1]).

[0095] Step 8.5: Initialize the parameters of each module in the unaligned face image super-resolution model based on the hierarchical perceptual spatial transformation network;

[0096] Step 8.6: Set the learning rate, the number of model training iterations, and the number of training samples per batch;

[0097] Step 8.7: Train the network using unaligned low-resolution images and corresponding aligned high-resolution images from the training dataset, and update the parameters in the unaligned face image super-resolution network based on the hierarchical perceptual spatial transformation network using an optimization algorithm;

[0098] Step 8.8: Train an unaligned face image super-resolution network based on a hierarchical perceptual spatial transformation network until the number of iterations reaches a preset value.

[0099] Step 9: Input the images from the test dataset into the trained unaligned face image super-resolution model based on a hierarchical perceptual spatial transformation network, and output the reconstructed aligned high-resolution face images.

[0100] In this embodiment, the face image finally output by the above method is compared with the face images obtained by other methods. The comparison result is as follows: Figure 5As shown, from left to right, the images are: (a) unaligned low-resolution image (16 * 16 pixels), (b) img2pose+SFMNet, (c) img2pose+PGDiff, (d) img2pose+WFEN, (e) img2pose+DifFace, (f) SFMNet+img2pose, (g) PGDiff+img2pose, (h) WFEN+img2pose, (i) DifFace+img2pose, (j) MTDN, (k) the result of this invention, and (l) the original high-resolution image (128 * 128 pixels). The comparison demonstrates that the aligned face image reconstructed by this invention has a more complete structure, richer texture details, and higher realism in visual perception.

[0101] The specific embodiments of the present invention have been described in detail above. It should be noted that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for super-resolution of unaligned face images based on a hierarchical perceptual spatial transformation network, characterized in that, Includes the following steps: S1: Construct the structure-aware spatial transformation module, the channel-space attention module, and the upsampling module respectively; S2: A cascaded perceptual spatial transformation module and an upsampling module are used to construct a coarse alignment super-resolution module and a fine alignment super-resolution module; a cascaded perceptual spatial transformation module and a channel-space attention module are used to construct a recursive alignment enhancement module; then, a cascaded input convolutional layer, a coarse alignment super-resolution module, a recursive alignment enhancement module, a fine alignment super-resolution module, and an image reconstruction module are used to construct a hierarchical perceptual spatial transformation face super-resolution network; S3: Randomly rotate and align the high-resolution face images in the public dataset according to the feature points, then crop out the corresponding face regions and perform Bicubic downsampling to construct unaligned low-resolution / aligned high-resolution face image pairs, and then divide them into training set and test set; S4: Train a face super-resolution model based on the training set; S5: Input the images in the test set into the trained unaligned face image super-resolution model based on the hierarchical perceptual spatial transformation network, and output the reconstructed high-resolution face image.

2. The method for super-resolution of unaligned face images based on a hierarchical perceptual spatial transformation network according to claim 1, characterized in that, The construction process of the structure-aware spatial transformation module in step S1 is as follows: A1: Constructing a face prior prediction module: Input features are initially encoded using an input convolutional layer with a kernel size of 7×7, a stride of 2, and padding of 3. Multiple hourglass network modules are then stacked to achieve top-down and bottom-up multi-scale feature flow. Within each hourglass module, global semantic information is extracted through downsampling, spatial resolution is restored through upsampling, and features from high-resolution paths are fused element-wise to achieve joint modeling of local details and global contextual information. The output features of each hourglass network module undergo channel adjustment via convolutional layers and normalization operations. Finally, a 1×1 convolution is used to generate heatmap features of facial key points, and ground truth supervision is applied to these heatmap features. To enhance cross-module information flow, the output of each hourglass module is accumulated with the current features and the channel-mapped heatmap prediction before being fed into the next module, thus forming a deep supervision mechanism and multi-level residual connections. A2: Construct a prior-guided localization network: Stack one input convolutional layer, one face prior prediction module, and one output convolutional layer. Concatenate the face features encoded by the input convolutional layer and the facial structure prior features predicted by the face prior prediction module in the channel dimension, and input them into the output convolutional layer together. A3: Constructing a mesh generator: Stack one input linear transformation layer, one PReLU activation function, and one output linear transformation layer to generate an affine transformation matrix; A4: Apply the affine transformation matrix to the input face features to achieve feature alignment; obtain the target output feature pixels by sampling sub-pixels at specific locations on the input feature map. These sampling locations correspond to non-integer pixel coordinates, thereby achieving continuous spatial mapping of the input features.

3. The method for super-resolution of unaligned face images based on a hierarchical perceptual spatial transformation network according to claim 1, characterized in that, The construction process of the channel-space attention module in step S1 is as follows: B1: Construct a channel attention module, receive input features and perform average pooling and max pooling respectively to obtain two channel attention features; input the two channel attention features into the multilayer perceptron module respectively, and output channel attention enhancement features; The attention enhancement features of the two channels are summed and passed through a Sigmoid function to obtain the channel attention weights. These weights are then multiplied pixel-by-pixel with the input features to construct effective channel enhancement features. B2: Construct a spatial attention module to perform average pooling and max pooling on the channel dimension of the effective channel augmentation features to obtain spatial augmentation features; After concatenation along the channels, the spatial attention map is obtained through the output convolutional layer and the Sigmoid function. It is then multiplied element-wise with the effective channel enhancement features to further enhance the spatial attention map, which is then used as the final output of the module.

4. The method for super-resolution of unaligned face images based on a hierarchical perceptual spatial transformation network according to claim 1, characterized in that, The construction process of the unaligned face image super-resolution model based on the hierarchical perceptual spatial transformation network in step S2 is as follows: C1: Constructing the image input module: cascaded convolutional layers, batch processing layers, activation functions, and upsampling layers; C2: Constructing a coarse-aligned super-resolution module: a cascaded perceptual spatial transformation module and an upsampling layer; C3: Constructing a recursive alignment enhancement module: It consists of three cascaded sub-modules. Each sub-module is composed of a structure-aware spatial transformation module and a channel-spatial attention module in sequence. The three sub-modules are connected end-to-end, and the output of the i-th group is used as the input of the i+1-th group, thereby realizing recursive feature alignment and enhancement. To ensure consistency between input and output features in the channel dimension, each module group is connected to a channel adjustment convolutional layer at the end; C4: Constructing a fine-aligned super-resolution module: a cascaded perceptual spatial transformation module and an upsampling layer; C5: Constructing an image reconstruction module: cascade three consecutive residual modules and one output convolutional layer, with a kernel size of 3×3, a stride of 1, padding of 1, and 3 output channels.

5. The method for super-resolution of unaligned face images based on a hierarchical perceptual spatial transformation network according to claim 1, characterized in that, The process of step S3 is as follows: E1: Preprocess the published dataset by dividing all face images in the dataset into [-20] . o +20 o Randomly rotate to generate unaligned face images, while retaining the corresponding aligned face images; E2: Detect and crop the face regions of the unaligned and aligned face images respectively, and construct a high-resolution face image in which the face region occupies more than 60% of the entire image; E3: For each high-resolution unaligned face image, use the Bicubic interpolation method to downsample and obtain the corresponding low-resolution face image; E4: Pair the generated unaligned low-resolution face images with aligned high-resolution face images to construct training or test sample pairs; E5: Store the data according to the rules for dividing the training set and the test set, ensuring that each non-aligned low-resolution face image corresponds to a unique high-resolution aligned face image.

6. The method for super-resolution of unaligned face images based on a hierarchical perceptual spatial transformation network according to claim 1, characterized in that, The process of step S4 is as follows: F1: Construct the mean squared error as the loss function; F2: Construct a perceptual loss function based on the VGG19 model; F3: Construct a face landmark heatmap loss function based on a 2D-FAN model; F4: Constructing the adversarial loss function; F5: Initialize the parameters of the hierarchical perceptual spatial transformation network; F6: Combine all the above loss functions to perform multi-level supervised training on the output of the hierarchical perceptual spatial transformation network; F7: Set the learning rate, number of model training iterations, and number of batch training samples; F8: Train the hierarchical perceptual spatial transformation network using unaligned low-resolution images and corresponding aligned high-resolution images from the training dataset, and update the network parameters using an optimization algorithm; F9: Train the super-resolution model for unaligned face images until the number of iterations reaches a preset value.

7. The method for super-resolution of unaligned face images based on a hierarchical perceptual spatial transformation network according to claim 6, characterized in that, The perceptual loss function based on the VGG19 model in step F2 for: ； in, This represents the joint distribution of the super-resolution results and the corresponding ground truth faces of each sub-network. This represents the feature map extracted by the pre-trained VGG19 model from the convolutional layers before all its max-pooling layers. Represents a true high-resolution image. This represents the generated super-resolution image.

8. The method for super-resolution of unaligned face images based on a hierarchical perceptual spatial transformation network according to claim 6, characterized in that, The face landmark heatmap loss function based on the 2D-FAN network in step F3 for: ； Where K represents the number of key points, This represents the super-resolution result generated by the structure prior prediction module acting on the fine-grained complex atom network and the global optimization sub-network, showing the predicted heatmap of the k-th key point. The ground truth heatmap of the k-th key point is generated by applying a 2D-FAN model to a ground truth high-resolution face image.

9. A method for super-resolution of unaligned face images based on a hierarchical perceptual spatial transformation network according to claim 6, characterized in that, The adversarial loss function in step F4 is: ； ； in, To constrain the adversarial loss function of the hierarchical perceptual spatial transformation network, Let E be the adversarial loss function constraining the discriminator, used to train the discriminator to distinguish between real and generated images, and let D be the average value over the samples. This represents the output of the discriminator model, which indicates the probability that the input is "true".

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1-9.

Images