Face image super-resolution reconstruction method and device

Abstract

The invention provides a face image super-resolution reconstruction method and device, and the method comprises the steps: carrying out the N times of down-sampling of an initial face feature of a low-resolution face image, and enabling the face feature obtained through any time of down-sampling to be obtained through the fusion of an enhanced first low-frequency feature and a plurality of enhanced first high-frequency features; performing N times of up-sampling on the facial features output by the Nth time of down-sampling to output enhanced facial features, wherein the facial features obtained by any time of up-sampling are obtained by inverse wavelet transformation of the low-frequency fusion features and a plurality of enhanced second high-frequency features; and fusing the initial facial features and the enhanced facial features to obtain a face image after reconstruction of the low-resolution face image. According to the embodiment of the invention, the loss of a key face structure can be reduced during down-sampling, and extra features can be obtained during up-sampling to enhance and improve the face contour, so that the problem of distortion of the face contour of the reconstructed face image is solved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image processing, and in particular to a method and device for super-resolution reconstruction of facial images. Background Art

[0002] Face super-resolution (FSR), also known as face hallucination, aims to convert low-resolution (LR) face images into high-resolution (HR) face images.

[0003] Traditional image super-resolution techniques, such as nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation, can improve the resolution of images to a certain extent, but these methods reduce the number of image pixels and may result in the loss of key facial details required for facial feature reconstruction, resulting in distortion of the reconstructed facial contour.

[0004] Therefore, there is an urgent need for a facial image super-resolution reconstruction method to solve the above technical problems. Summary of the invention

[0005] In view of this, the present disclosure proposes a facial image super-resolution reconstruction method and device to solve the problem in the related art that the reconstructed facial contour is distorted due to the loss of key facial details.

[0006] The first embodiment of the present disclosure provides a method for super-resolution reconstruction of a face image, comprising:

[0007] The initial facial features of the low-resolution face image are downsampled N times successively according to a preset sampling rate, and the facial features obtained by any downsampling are obtained by fusion of the enhanced first low-frequency features and the enhanced multiple first high-frequency features, wherein the first low-frequency features and the multiple first high-frequency features are obtained by wavelet transforming the input facial features, and the input facial features are the initial facial features or the facial features output by the last downsampling;

[0008] The facial features output by the Nth downsampling are upsampled N times successively according to the preset sampling rate to output enhanced facial features, the facial features obtained by any upsampling are obtained by inverse wavelet transform of low-frequency fusion features and multiple enhanced second high-frequency features, the low-frequency fusion features are obtained by fusion of the second low-frequency features and the third low-frequency features, the second low-frequency features and the multiple second high-frequency features are obtained by wavelet transform of the facial features output by the downsampling corresponding to the upsampling, and the third low-frequency features refer to the low-frequency features in the facial features output by the last upsampling or the last downsampling;

[0009] The initial facial features and the enhanced facial features are fused to obtain a facial image reconstructed from the low-resolution facial image.

[0010] The disclosed embodiment obtains a first low-frequency feature and multiple first high-frequency features by wavelet transforming the initial facial features of a low-resolution facial image or the facial features output from the last downsampling, which can simulate the result of downsampling after low-pass filtering, and helps to prevent the occurrence of frequency domain aliasing; by first decomposing into low-frequency features and high-frequency features, and then enhancing the features separately, and finally fusing the enhanced high-frequency features with the enhanced low-frequency features, it is possible to enhance the edge information of the face, alleviate the problem of facial structure loss caused by downsampling, and make the facial contour clearer.

[0011] The disclosed embodiment obtains a second low-frequency feature and a plurality of second high-frequency features by wavelet transforming the second facial feature outputted by the downsampling corresponding to the upsampling, and can obtain additional features through wavelet decomposition to enhance and improve the facial contour; and obtains a low-frequency fused feature by fusing the second low-frequency feature and the third low-frequency feature, and obtains a target facial feature by inverse wavelet transforming the low-frequency fused feature and the plurality of second high-frequency features, and can effectively utilize features from different scales and enhance facial details by fusing high-frequency and low-frequency features respectively.

[0012] In the embodiment of the present disclosure, for any downsampling in the N downsamplings, the downsampling step includes:

[0013] Performing wavelet transformation on the initial facial features of the low-resolution facial image or the facial features output by the last downsampling to obtain the first low-frequency features and the plurality of first high-frequency features;

[0014] Performing feature enhancement on the first low-frequency feature and the plurality of first high-frequency features respectively to obtain an enhanced first low-frequency feature and an enhanced plurality of first high-frequency features;

[0015] The enhanced first low-frequency feature and the enhanced multiple first high-frequency features are fused to obtain a fused feature; the fused feature is used as the facial feature of the down-sampling output.

[0016] In an embodiment of the present disclosure, for any upsampling in the N upsamplings, the upsampling step includes:

[0017] Acquire a third low-frequency feature in the first facial feature outputted by the last upsampling or the last downsampling, and acquire a second facial feature outputted by the downsampling corresponding to the upsampling; the resolution of the second facial feature is greater than the resolution of the first facial feature;

[0018] Performing wavelet transformation on the second facial feature to obtain the second low-frequency feature and the plurality of second high-frequency features;

[0019] Fusing the second low-frequency feature and the third low-frequency feature to obtain the low-frequency fusion feature;

[0020] The low-frequency fusion feature and the plurality of second high-frequency features are subjected to inverse wavelet transformation to obtain a target facial feature; the target facial feature is used as the facial feature of the up-sampling output.

[0021] In the embodiment of the present disclosure, the first low-frequency feature is a low-frequency sub-band; the multiple first high-frequency features include a horizontal sub-band, a vertical sub-band, and a diagonal sub-band; wavelet transform is performed on the initial facial features of the low-resolution face image or the facial features output by the last downsampling to obtain the first low-frequency feature and the multiple first high-frequency features, including:

[0022] Decompose the initial facial features or the facial features output by the last downsampling into horizontal sub-bands, vertical sub-bands, diagonal sub-bands and low-frequency sub-bands; the horizontal sub-bands refer to the high-frequency features in the horizontal direction of the initial facial features or the facial features output by the last downsampling; the vertical sub-bands refer to the high-frequency features in the vertical direction of the initial facial features or the facial features output by the last downsampling; the diagonal sub-bands refer to the high-frequency features in the diagonal direction of the initial facial features or the facial features output by the last downsampling; the low-frequency sub-bands refer to the low-frequency features in the initial facial features or the facial features output by the last downsampling.

[0023] In the disclosed embodiment, the N downsamplings correspond one-to-one to N global transformers and one-to-one to N residual blocks; the first low-frequency features and the plurality of first high-frequency features are feature enhanced to obtain enhanced first low-frequency features and enhanced plurality of first high-frequency features, including:

[0024] Performing feature enhancement on the first low-frequency feature by a global transformer corresponding to the downsampling to obtain the enhanced first low-frequency feature;

[0025] The plurality of first high-frequency features are feature enhanced by using a residual block corresponding to the downsampling to obtain the plurality of enhanced first high-frequency features.

[0026] The disclosed embodiment can greatly improve the processing efficiency of high-frequency facial features and low-frequency facial features by enhancing the first low-frequency feature of the downsampled output through a global transformer that is more sensitive to low-frequency features, and by enhancing the multiple first high-frequency features of the downsampled output through a residual block that is more sensitive to high-frequency features.

[0027] In the embodiment of the present disclosure, before downsampling the initial facial features of the low-resolution facial image N times successively according to a preset sampling rate, the method further includes:

[0028] Extracting original facial features of the low-resolution face image through a convolutional layer;

[0029] The original facial features are extracted by a global transformer to obtain the initial facial features; the initial facial features include local facial features, regional facial features and global facial features; the initial facial features are used as input for the first downsampling in the N downsamplings.

[0030] In the disclosed embodiment, the initial facial features include local facial features and regional facial features; the initial facial features are obtained by extracting features from the original facial features through a global transformer, including:

[0031] Performing feature segmentation on the original facial features to obtain multiple window features;

[0032] Filtering out at least one target window feature from the multiple window features, wherein the target window feature refers to a window feature among the multiple window features for enhancing local facial details;

[0033] For any one of the at least one target window feature, calculating a self-attention weight of the target window feature;

[0034] Using the self-attention weight, weighted sum is performed on the value vector of the target window feature to obtain a first local facial feature;

[0035] At least one first local facial feature corresponding to the at least one target window feature is spliced ​​to obtain a regional facial feature of the original facial feature.

[0036] In the disclosed embodiment, after obtaining the regional facial features of the original facial features, the method further includes:

[0037] Performing random window movement on the multiple window features to obtain multiple new window features;

[0038] According to the new multiple window features, the step of selecting at least one target window feature from the multiple window features is performed multiple times to obtain multiple regional facial features.

[0039] The disclosed embodiments can promote the communication between adjacent window features through the window shift mechanism, thereby effectively enhancing the regional facial features and local facial features of the face image.

[0040] In the embodiment of the present disclosure, the facial features further include global facial features; and the method further includes:

[0041] Fusing the regional facial features with the original facial features to obtain fused facial features;

[0042] Setting a plurality of self-attention heads, each of which is used to calculate the self-attention weight of a partial feature in the fused facial feature; using the self-attention weight to perform weighted summation on the value vectors of the partial features to obtain a second local facial feature;

[0043] The multiple second local facial features output by the multiple self-attention heads are spliced ​​to obtain the global facial feature.

[0044] The disclosed embodiments can extract local facial features and regional facial features through a regional self-attention mechanism, and can extract local facial features and regional facial features through a global self-attention mechanism. It can explore different receptive fields and discover deeper associations in facial features, thereby extracting more comprehensive information to enhance facial details.

[0045] In an embodiment of the present disclosure, the method further includes:

[0046] The partial features that the multiple self-attention heads are responsible for are shuffled so that each self-attention head is responsible for different partial features each time, thereby enhancing the information exchange between the multiple self-attention heads.

[0047] The disclosed embodiment can enhance the information exchange between multiple self-attention heads through a shuffling multi-head mechanism, thereby effectively enhancing the local facial features and global facial features of a face image.

[0048] In the disclosed embodiment, calculating the self-attention weight of the target window feature includes:

[0049] Performing a linear transformation on the target window feature to generate a query vector, a key vector and the value vector;

[0050] Calculating an attention score of the target window feature based on the query vector and the key vector;

[0051] The attention score is normalized to obtain the self-attention weight.

[0052] The second aspect of the present disclosure provides a facial image super-resolution reconstruction device, comprising:

[0053] A downsampling module, used for downsampling the initial facial features of the low-resolution face image N times in sequence according to a preset sampling rate, wherein the facial features obtained by any downsampling are obtained by fusion of the enhanced first low-frequency features and the enhanced multiple first high-frequency features, wherein the first low-frequency features and the multiple first high-frequency features are obtained by wavelet transforming the input facial features, and the input facial features are the initial facial features or the facial features output by the last downsampling;

[0054] An upsampling module is used to perform N upsampling on the facial features output by the Nth downsampling according to the preset sampling rate to output enhanced facial features, wherein the facial features obtained by any upsampling are obtained by inverse wavelet transformation of low-frequency fusion features and multiple enhanced second high-frequency features, wherein the low-frequency fusion features are obtained by fusion of the second low-frequency features and the third low-frequency features, wherein the second low-frequency features and the multiple second high-frequency features are obtained by wavelet transformation of the facial features output by the downsampling corresponding to the upsampling, and the third low-frequency features refer to the low-frequency features in the facial features output by the last upsampling or the last downsampling;

[0055] An image reconstruction module is used to fuse the initial facial features with the enhanced facial features to obtain a facial image reconstructed from the low-resolution facial image.

[0056] An embodiment of the third aspect of the present disclosure provides an electronic device, which includes a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the facial image super-resolution reconstruction method described in the first aspect by executing the computer instructions.

[0057] An embodiment of the fourth aspect of the present disclosure provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to enable a computer to execute the facial image super-resolution reconstruction method described in the first aspect.

[0058] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will become apparent from the following description or be learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] By reading the detailed description of the preferred embodiment below, various other advantages and benefits will become clear to those of ordinary skill in the art. The accompanying drawings are only used for the purpose of illustrating the preferred embodiment and are not considered to be limitations of the present disclosure. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings.

[0060] In the attached picture:

[0061] Figure 1 A schematic diagram showing the difference between the related prior art provided by an embodiment of the present disclosure and the present technical solution in reconstructing a face image;

[0062] Figure 2 A schematic diagram of a process for super-resolution reconstruction of a face image provided by an embodiment of the present disclosure is shown;

[0063] Figure 3 A schematic diagram showing a flow chart of another facial image super-resolution reconstruction method provided by an embodiment of the present disclosure;

[0064] Figure 4 A schematic diagram of a process of down-sampling based on wavelet transform provided by an embodiment of the present disclosure is shown;

[0065] Figure 5 A schematic diagram of a process of up-sampling based on wavelet transform provided by an embodiment of the present disclosure is shown;

[0066] Figure 6 A schematic diagram of the process of regional self-attention in a global transformer provided by an embodiment of the present disclosure is shown;

[0067] Figure 7 A schematic diagram of RSA window shifting provided by an embodiment of the present disclosure is shown;

[0068] Figure 8 A schematic diagram of the process of global self-attention in a global transformer provided by an embodiment of the present disclosure is shown;

[0069] Fig. 9 A schematic diagram of a GSA shuffle multi-head provided by an embodiment of the present disclosure is shown;

[0070] Fig.10 A schematic diagram showing a comparison between the technical solution provided by an embodiment of the present disclosure and other prior art super-resolution structures;

[0071] Fig.11 A schematic diagram of the structure of a facial image super-resolution reconstruction device provided by an embodiment of the present disclosure is shown;

[0072] Fig.12 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure is shown;

[0073] Fig.13 A schematic diagram of a storage medium provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0074] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0075] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in the present disclosure should have the common meanings understood by those skilled in the art to which the present disclosure belongs.

[0076] The following describes the technical scenarios involved in the embodiments of the present disclosure.

[0077] Face super-resolution (FSR), also known as face hallucination, aims to convert low-resolution (LR) face images into high-resolution (HR) face images. Different from image super-resolution, FSR focuses on reconstructing the basic structural information of the face, including the facial contour and the shape of facial components.

[0078] Existing FSR methods usually adopt an encoder-decoder structure. This is because this structure helps the model grasp the overall facial structure with a larger receptive field in the encoder stage and enhance facial details in the decoder stage. Specifically, the encoder first downsamples the LR input to extract facial features of different scales. Subsequently, the decoder gradually upsamples the features from the encoder output, refines the details, and finally generates HR output. However, previous methods have ignored the impact of the selected downsampling technique on the restoration results. For example, methods such as SPARNet, AD-GNN, and LAAT use bicubic interpolation or strided convolution for downsampling. These operations reduce the number of image pixels and may cause the loss of key facial details required for reconstruction. Figure 1 As shown in (b) and (c), bicubic interpolation and piecewise convolution cause a large loss of overall facial structure texture, resulting in distortion of the facial contours subsequently reconstructed. Another example from EIPNet is downsampling through average pooling operations. Figure 1 As shown in (d), this will cause aliasing artifacts in high-frequency and low-frequency facial features. This phenomenon is particularly obvious in the eye features, which seriously affects the accurate presentation of facial components.

[0079] To solve the above problems, the present invention decomposes facial features through wavelet transform for downsampling at the encoder stage, which can reduce the distortion of key facial structures during downsampling; and at the decoder stage, the additional features obtained through wavelet decomposition can be used to enhance and improve the facial contour, thereby solving the problem of distortion of the reconstructed facial contour due to the loss of key facial details in the related art.

[0080] According to an embodiment of the present disclosure, an embodiment of a method for super-resolution reconstruction of a facial image is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0081] In this embodiment, a method for super-resolution reconstruction of a face image is provided. Figure 2 is a flowchart of a method for super-resolution reconstruction of a face image according to an embodiment of the present disclosure. Figure 2 As shown, the process includes the following steps:

[0082] Step S101, down-sampling the initial facial features of the low-resolution face image N times in succession according to a preset sampling rate.

[0083] Specifically, the facial features obtained by any downsampling are obtained by fusion of the enhanced first low-frequency features and the enhanced multiple first high-frequency features, the first low-frequency features and the multiple first high-frequency features are obtained by wavelet transform of the input facial features, and the input facial features are the initial facial features or the facial features output by the previous downsampling.

[0084] More specifically, the number of downsampling times can be set according to actual conditions and is not specifically limited here. Figure 3 As shown: In the encoding stage, the dimension of the facial feature map can be adjusted from H×W×C to That is: the first downsampling: adjusted from H×W×C to The second downsampling: Adjust to The third downsampling: Adjust to Among them, H refers to the height of the image, that is, the number of vertical pixels of the image; W refers to the width of the image, that is, the number of horizontal pixels of the image; C refers to the number of channels of the image, that is, the number of color channels in the image.

[0085] In some specific embodiments, for any downsampling in the N downsamplings, the downsampling step includes:

[0086] Step S1011, performing wavelet transformation on the initial facial features of the low-resolution facial image or the facial features output by the last downsampling to obtain the first low-frequency features and the plurality of first high-frequency features.

[0087] In some specific embodiments, Figure 4 As shown: For the input facial feature F1∈R H×W×C , the input facial feature F1 can be decomposed into four wavelet sub-bands through discrete wavelet transform WT: low-frequency feature and high-frequency features in horizontal, vertical, and diagonal directions and

[0088] Among them, the discrete wavelet transform can simulate the standard downsampling (including low-pass filter and downsampling) based on the Nyquist sampling theorem, and decompose the input image into four components of different frequencies, including one low-frequency component and three high-frequency components. The low-frequency component can be approximated as the result of low-pass filtering and then downsampling. At the same time, the obtained high-frequency component can also be fused with the low-frequency features to enhance information such as the edge of the face, thereby alleviating the problem of facial structure loss caused by downsampling; Figure 1 As shown in (e), after wavelet feature decomposition and fusion, the facial contour is significantly clearer and there is no frequency domain aliasing. This result shows that the strategy of using wavelet transform decomposition features for downsampling and further using the decomposed high-frequency features to enhance the facial contour is very effective.

[0089] Step S1012: performing feature enhancement on the first low-frequency feature and the plurality of first high-frequency features respectively to obtain an enhanced first low-frequency feature and an enhanced plurality of first high-frequency features.

[0090] Specifically, considering that the transformer is more sensitive to low-frequency features and the residual block CNN is more sensitive to high-frequency features, the high-frequency features can be extracted by the residual block CNN to enhance the high-frequency features, for example Figure 3 The high-frequency features of the mid- and down-sampled WFD-a1 output can be enhanced through the residual block CNN to obtain the low-frequency features F high ; and extracting low-frequency features through a global transformer to enhance low-frequency features, such as Figure 3 The low-frequency features of the down-sampled WFD-a1 output can be enhanced by the global transformer to obtain the enhanced low-frequency features F low .

[0091] More specifically, since the low-frequency part mainly carries the basic information in the image, we focus on enhancing the low-frequency facial details on the main path, while paying attention to the high-frequency facial texture on the residual path:

[0092]

[0093] Among them, F low Indicates the enhanced low-frequency features, F high represents the enhanced high-frequency features, T represents the global transformer, and R represents the residual block.

[0094] Step S1013, fusing the enhanced first low-frequency feature and the enhanced multiple first high-frequency features to obtain a fused feature; the fused feature is used as the facial feature of the down-sampling output.

[0095] Specifically, Figure 3 As shown: By enhancing the low-frequency features F low and the enhanced high frequency features F high The fusion is performed to obtain a fused feature F2; the fused feature F2 can be used as the input for the next downsampling or the input for the corresponding upsampling.

[0096] In the disclosed embodiment, downsampling is usually used to reduce the size of the feature map during the encoder processing. However, the existing methods ignore the irreversible distortion caused by downsampling, resulting in unclear edges of the FSR results. The reason for this is that the traditional downsampling operation reduces the resolution by merging adjacent pixels, but due to the reduction in sampling points, it will cause feature distortion, especially in areas with obvious gradient changes. In this case, we introduce downsampling WFD using wavelet transform to alleviate this phenomenon, specifically refer to steps S1011-step S1013.

[0097] Step S102, upsampling the facial features output by the Nth downsampling successively N times according to the preset sampling rate to output enhanced facial features.

[0098] Specifically, the facial features obtained by any upsampling are obtained by inverse wavelet transform of low-frequency fusion features and multiple enhanced second high-frequency features, the low-frequency fusion features are obtained by fusion of the second low-frequency features and the third low-frequency features, the second low-frequency features and the multiple second high-frequency features are obtained by wavelet transform of the facial features output by the downsampling corresponding to the upsampling, and the third low-frequency features refer to the low-frequency features in the facial features output by the last upsampling or the last downsampling.

[0099] More specifically, the number of upsampling is the same as the number of downsampling in the above encoding stage. Figure 3 As shown: In the stage stage, the dimension of the facial feature map can be increased from 1 to 2 by upsampling 3 times. Adjust to H×W×C. That is, the first upsampling: Adjust to The second upsampling: Adjust to The third upsampling: Adjust to H×W×C.

[0100] In some specific embodiments, for any upsampling in the N upsamplings, the upsampling step includes steps S1021 to S1024:

[0101] Specifically, in order to obtain more details, several methods (AD-GNN, LAAT, CTCNet) propose to use residual concatenation to enable the decoder to utilize the information of the encoder. Since the feature resolutions of different scales are different, upsampling is usually used to adjust them to the same resolution before feature fusion. However, the direct fusion operation is not the best choice because it may cause a certain degree of high-frequency and low-frequency aliasing. In order to better fuse the features from the encoder, the embodiment of the present disclosure uses wavelet transform for image scale transformation, and provides an upsampling WFU based on wavelet transform, which effectively utilizes features from different scales in the decoder stage, and enhances facial details by fusing high-frequency and low-frequency features separately. The specific implementation steps refer to steps S1021-step S1024.

[0102] Step S1021, obtaining the third low-frequency feature in the first facial feature output by the last upsampling or the last downsampling, and obtaining the second facial feature output by the downsampling corresponding to the upsampling; the resolution of the second facial feature is greater than the resolution of the first facial feature.

[0103] Specifically, the input required for each upsampling includes two categories, namely: one category is the third low-frequency feature in the first facial feature outputted by the last upsampling or the last downsampling, such as Figure 3 As shown in , when the upsampling is WFU-b3, the input is the facial features of the last downsampling WFD-a3 output; when the upsampling is WFU-b2, the input is the facial features of the last upsampling WFU-b3 output. The other type is the second facial features or initial facial features of the downsampling output corresponding to the upsampling, such as Figure 3 As shown, when the upsampling is WFU-b3, the input is the facial features output by the downsampling WFD-a2; when the upsampling is WFU-b1, the input is the initial facial features F1.

[0104] More specifically, “a second facial feature of the downsampled output corresponding to the upsampled” can be understood as: Figure 3As shown, for example: the input of the upsampling WFU-b2 includes the facial features output by the last upsampling WFU-b3 and the facial features output by the downsampling WFD-a1, that is: the upsampling WFU-b2 has a corresponding relationship with the downsampling WFD-a1; for another example: the input of the upsampling WFU-b3 includes the facial features output by the last downsampling WFD-a3 and the facial features output by the downsampling WFD-a2, that is: the upsampling WFU-b3 has a corresponding relationship with the downsampling WFD-a2.

[0105] More specifically, the resolution of facial features output by upsampling WFU-b3 is The resolution of facial features output by downsampling WFD-a2 is That is, in the upsampling and downsampling having a corresponding relationship, the scale of the facial features output by the downsampling is larger than the scale of the facial features output by the upsampling.

[0106] Step S1022: performing wavelet transform on the second facial feature to obtain the second low-frequency feature and the plurality of second high-frequency features.

[0107] Specifically, the second facial feature can be understood as the facial feature F of the downsampling output with a larger scale in the upsampling and downsampling having the corresponding relationship above. s In order to solve the problem that the scales of the facial features output by upsampling and downsampling with a corresponding relationship are different, the embodiment of the present disclosure applies wavelet transform to the second facial feature F output by downsampling with a larger scale. s , decompose it into facial features F with upsampled output s+1 Four wavelet subbands of the same scale, namely:

[0108]

[0109] in Indicates F s The low frequency part, Indicates F s The three high frequency parts.

[0110] Step S1023: Fusing the second low-frequency feature and the third low-frequency feature to obtain the low-frequency fusion feature.

[0111] Specifically, the third low-frequency feature can be understood as the facial feature F of the upsampling output with a smaller scale in the upsampling and downsampling with the corresponding relationship above. s+1 The low frequency part of .

[0112] Specifically, considering the small-scale feature F s+1 It mainly contains low-frequency information, so F s The low frequency part With Fs+1 The low-frequency features of are fused to obtain low-frequency fusion features, such as Figure 5 shown.

[0113] Step S1024, performing inverse wavelet transform on the low-frequency fusion feature and the plurality of second high-frequency features to obtain a target facial feature; the target facial feature is used as the facial feature of the up-sampling output.

[0114] Specifically, after decomposing F s After obtaining multiple second high-frequency features, the residual block can be used to enhance the features; then the enhanced second high-frequency features and frequency fusion features are used to perform inverse wavelet transform to obtain the target facial features, as shown below:

[0115]

[0116] Among them, F' s represents the target facial features, IWT represents inverse wavelet transform, C represents fusion, and R represents the standard residual block.

[0117] More specifically, the target facial feature can be used as the input for the next upsampling, or can be used as an enhanced facial feature to be fused with the initial facial feature.

[0118] Step S103, fusing the initial facial features with the enhanced facial features to obtain a facial image reconstructed from the low-resolution facial image.

[0119] Specifically, Figure 3 As shown, the initial facial feature F0 and the enhanced facial feature F d Perform element-by-element summation and process the summation result according to the 3*3 convolutional layer to obtain a high-resolution face image reconstructed from the low-resolution face image.

[0120] In some specific embodiments, before downsampling the initial facial features of the low-resolution facial image N times successively according to a preset sampling rate, the method further includes:

[0121] Extracting original facial features of the low-resolution face image through a convolutional layer;

[0122] The original facial features are extracted through a global transformer to obtain the initial facial features.

[0123] Specifically, for example Figure 3As shown in the figure, the low-resolution face image can be feature extracted through 3*32D convolution to obtain the original facial feature F0, and then the original facial feature F0 is feature extracted based on the global transformation, thereby extracting local facial features, regional facial features and global facial features. Among them, local facial features include skin detail features, regional facial features include regional features such as eyes and mouth, and global facial features include the overall contour features of the face.

[0124] More specifically, the embodiments of the present disclosure are mainly used to process low-frequency information, so it is more advantageous to use a transformer structure that is more sensitive to low-frequency information to extract facial features. In order to improve the restoration effect of facial images, it is crucial to effectively utilize facial features at local, regional, and global levels. Specifically, the local area includes multiple pixels, and it is most effective to model it using a small kernel convolution (1×1 or 3×3), which can capture typical features such as local facial details. Regional features include dozens of pixels, such as eyes, noses, and other facial components. Due to its large spatial range, it is better to model it using a large kernel convolution or a window-based transformer. Global features involve structural correlations of the entire face, such as the overall facial contour, and are best modeled using a global transformer. However, many methods (Faceformer, CTCNet) and others only focus on utilizing local and global features or local and regional features. Therefore, as Figure 6 and Figure 8 As shown, the disclosed embodiment proposes a full domain transformer (FDT) as the main module for feature extraction, which mainly consists of two parts: the regional self-attention (RSA) focuses on extracting local and regional facial features, while the global self-attention (GSA) is responsible for extracting local and global facial features.

[0125] In some specific embodiments, the initial facial features include local facial features and regional facial features; extracting the original facial features through a global transformer to obtain the initial facial features includes:

[0126] Step c1, performing feature segmentation on the original facial features to obtain multiple window features;

[0127] Step c2: Filter out at least one target window feature from the multiple window features.

[0128] Specifically, the target window feature refers to a window feature among the multiple window features used to enhance local facial details.

[0129] Step c3: for any one of the at least one target window features, calculate the self-attention weight of the target window feature.

[0130] Specifically, the self-attention weight of the target window feature can be calculated in the following manner: linearly transform the target window feature to generate a query vector, a key vector and the value vector; based on the query vector and the key vector, calculate the attention score of the target window feature; normalize the attention score to obtain the self-attention weight.

[0131] Step c4, using the self-attention weight to perform weighted summation on the value vector of the target window feature to obtain a first local facial feature.

[0132] Step c5, splicing at least one first local facial feature corresponding to the at least one target window feature to obtain a regional facial feature of the original facial feature.

[0133] In the above steps c1 to c5, if Figure 6 As shown:

[0134] First, from the original facial features X∈R H×W×C Extract a set of window features:

[0135] {X1,X2,…,X n-1 ,X n}=Split(X)

[0136] in represents the window size, and N is the dimension of the window feature, such as the side length; n is the number of window features.

[0137] Then, local facial details are captured to enhance the contextual information of the network. Local details are captured by 1×1 point convolution and 3×3 depthwise separable convolution. For each window feature patch that enhances the local context, it is projected onto the query key value In , the process is:

[0138] {Q i ,K i ,V i =RS(D(P(X i )))

[0139] Among them, RS represents the reshaping operator, D represents the depth convolution layer, and P represents the point convolution layer. On this basis, for each window feature patch, the regional self-attention can be expressed as:

[0140]

[0141] Among them, Attention(Q i ,K i ,Vi ) represents the self-attention weight, and α represents the learnable parameter. In order to avoid (HW) 2 In order to reduce the computational complexity of , we choose to implicitly encode the global features across channels when calculating the feature covariance A. Specifically, we convert the traditional size of A∈R HW×HW The attention map is replaced by a size of A∈R C×C Regional attention map.

[0142] After obtaining the self-attention weight Attention, the value vector V of the target window feature can be i A weighted sum is performed to obtain the first local facial feature.

[0143] In addition, to address the problem of lack of connectivity between different windows, the disclosed embodiment also provides an RSA window shift mechanism, such as Figure 7 As shown, that is, the multiple window features are randomly moved to obtain multiple new window features; according to the new multiple window features, the step of selecting at least one target window feature from the multiple window features is performed multiple times to obtain multiple regional facial features. The disclosed embodiment promotes communication between adjacent windows by moving windows, so that RSA can effectively enhance regional and local facial features.

[0144] In some specific embodiments, the facial features further include global facial features; after the above step c5, the method further includes:

[0145] Step c6, fusing the regional facial features with the original facial features to obtain fused facial features, for example Figure 6 The last step in .

[0146] Step c7, setting multiple self-attention heads, each self-attention head is used to calculate the self-attention weight of some features in the fused facial features, and using the self-attention weight to perform weighted summation on the value vectors of the some features to obtain a second local facial feature.

[0147] Step c8, splicing the multiple second local facial features output by the multiple self-attention heads to obtain the global facial features.

[0148] In steps c6 to c8, if Figure 8 As shown: For the facial feature X'∈R obtained by the above step c6 H ×W×C First, 1×1 point convolution and 3×3 depth-separable convolution are used to extract local information to ensure accurate recovery of facial details; then, the channel is subdivided into multiple heads h and different self-attentions are learned simultaneously, including: generating queries in the multi-head channel key value Based on the projection of the overall facial features after enhancing the local details, is the number of channels per head. Next, we create a vector of size This process emphasizes the relationship between channels and implicitly encodes global facial features. In summary, the complete process of global self-attention can be described as follows:

[0149] Attention(Q,K,V)=VReLU(QK T / β)

[0150] Among them, β represents a learnable parameter, and Attention(Q,K,V) represents the self-attention weight of some features in the fusion facial features.

[0151] Finally, after obtaining the self-attention weight, the value vectors of some features are weighted summed using the self-attention weight to obtain the second local facial feature; multiple second local facial features output by multiple self-attention heads are spliced ​​to obtain the global facial feature.

[0152] In some specific embodiments, Fig. 9 As shown, the embodiment of the present disclosure provides a shuffling multi-head feature mechanism, including: shuffling the partial features that the multiple self-attention heads are responsible for, so that the partial features that each self-attention head is responsible for are different each time, thereby enhancing the information exchange between the multiple self-attention heads, and further effectively enhancing the local and global features of the input face image.

[0153] Compared with the prior art, the embodiments of the present disclosure propose a wavelet-based feature enhancement network (WEFN) for efficient FSR. The method integrates wavelet feature downsampling (WFD) and wavelet feature upsampling (WFU) in the encoder-decoder structure, which can solve the feature distortion caused by direct downsampling in the encoder-decoder structure. In addition, by further utilizing FDT to comprehensively extract low-frequency facial features, the method can achieve more accurate facial structure reconstruction. The effectiveness of WFD and WFU in minimizing facial structure distortion during reconstruction, as well as the comprehensive facial feature perception capability provided by FDT, are verified. Extensive experiments (including face matching in surveillance scenarios) show that the method provided by the embodiments of the present disclosure can effectively achieve higher-fidelity FSR, achieving an excellent balance between performance, model size, and inference speed.

[0154] The present invention also provides a specific embodiment, as follows:

[0155] like Figure 3 As shown, given a degraded face image I LR ∈RH×W×3 The goal of this scheme (wavelet-based feature enhancement network) is to reconstruct a clean face by using a wavelet-based encoder-decoder structure combined with a residual block and the global transformer proposed in this scheme. SR ∈R H×W×3 The wavelet-based encoder-decoder structure includes a wavelet feature downsampling module that downsamples features at the encoder stage, and a wavelet feature upsampling module that upsamples features at the decoder stage. Specifically:

[0156] First, from I LR Extract shallow features F0∈R H×W×C , where H×W represents the spatial resolution of the image and C represents the number of channels.

[0157] Subsequently, F0 is processed step by step by the wavelet feature downsampling module WFD and gradually transformed into a low-resolution potential representation F l ∈R H / 8×W / 8×4C At each level, the low-frequency part obtained by wavelet transform is fed into the global transformer, while the high-frequency part is fed into the residual block for better feature extraction.

[0158] In the bottleneck stage (between the encoder and decoder stages), a series of global transformers are used to refine F l , to obtain F e ∈R H / 8×W / 8×4C ; Among them, F l and F e One or more global transformers may be set between them, preferably 6 transformers. Then, a wavelet feature upgrading module WFU is added before each decoding stage to effectively perform cross-scale feature fusion to obtain accurate deep features F d ∈R H×W×C Finally, the decoder stage outputs F d After residual concatenation and dimensionality reduction processing, a clean face image I is restored. SR The structure of the global converter proposed in this scheme is as follows: Figure 6 and Figure 8 As shown.

[0159] In the above scheme, the number of channels C is 40. H×W×C In the stage, the number of global transformers is set to 2, and in each R H / 8×W / 8×4C In the first stage, the number of global transformers was set to 6, while in all other stages, the number of global transformers was set to 1. During the training phase, the model was optimized using an L1 loss with a coefficient of 1, and the Adam optimizer was used: β1 = 0.9, β2 = 0.99. The initial learning rate was set to 0.0002, and the batch size of the training was 12.

[0160] Fig.10 The figure shows the comparison of the super-resolution visual results of this scheme with other advanced methods. LR represents the low-resolution face image input, and HR represents the ground-truth face image. The visual results of this scheme are significantly better than those of other methods, especially in key areas such as the eyes. The quantitative results are obtained by comparing the output SR face image with the HR face image using LR as input.

[0161] The disclosed embodiment also provides a comparison of the super-resolution quantitative results of this solution and other advanced methods, such as shown in Table 1 and Table 2:

[0162] Table 1

[0163]

[0164] Table 2

[0165]

[0166]

[0167] The above Table 1 refers to the experimental results on the CelebA test dataset, and the above Table 2 refers to the experimental results on the Helen test dataset.

[0168] Corresponding to the implementation of the above face image super-resolution reconstruction method, the present disclosure also provides a face image super-resolution reconstruction device for performing the above Figures 1 to 10 The method for super-resolution reconstruction of a face image described in any of the embodiments shown in the figure. Fig.11 As shown, the facial image super-resolution reconstruction device comprises:

[0169] A downsampling module, used for downsampling the initial facial features of the low-resolution face image N times in sequence according to a preset sampling rate, wherein the facial features obtained by any downsampling are obtained by fusion of the enhanced first low-frequency features and the enhanced multiple first high-frequency features, wherein the first low-frequency features and the multiple first high-frequency features are obtained by wavelet transforming the input facial features, and the input facial features are the initial facial features or the facial features output by the last downsampling;

[0170] An upsampling module is used to perform N upsampling on the facial features output by the Nth downsampling according to the preset sampling rate to output enhanced facial features, wherein the facial features obtained by any upsampling are obtained by inverse wavelet transformation of low-frequency fusion features and multiple enhanced second high-frequency features, wherein the low-frequency fusion features are obtained by fusion of the second low-frequency features and the third low-frequency features, wherein the second low-frequency features and the multiple second high-frequency features are obtained by wavelet transformation of the facial features output by the downsampling corresponding to the upsampling, and the third low-frequency features refer to the low-frequency features in the facial features output by the last upsampling or the last downsampling;

[0171] An image reconstruction module is used to fuse the initial facial features with the enhanced facial features to obtain a facial image reconstructed from the low-resolution facial image.

[0172] Optionally, the downsampling module is also used to: perform wavelet transform on the initial facial features of the low-resolution face image or the facial features output from the last downsampling to obtain the first low-frequency features and the multiple first high-frequency features; perform feature enhancement on the first low-frequency features and the multiple first high-frequency features to obtain enhanced first low-frequency features and enhanced multiple first high-frequency features; fuse the enhanced first low-frequency features and the enhanced multiple first high-frequency features to obtain fused features; and use the fused features as the facial features output from the downsampling.

[0173] Optionally, the upsampling module is also used to: obtain the third low-frequency feature in the first facial feature outputted by the last upsampling or the last downsampling, and obtain the second facial feature outputted by the downsampling corresponding to the upsampling; the resolution of the second facial feature is greater than the resolution of the first facial feature; perform a wavelet transform on the second facial feature to obtain the second low-frequency feature and the multiple second high-frequency features; fuse the second low-frequency feature and the third low-frequency feature to obtain the low-frequency fused feature; perform an inverse wavelet transform on the low-frequency fused feature and the multiple second high-frequency features to obtain a target facial feature; and use the target facial feature as the facial feature outputted by the upsampling.

[0174] Optionally, the downsampling module is also used to: decompose the initial facial features or the facial features output by the last downsampling into horizontal sub-bands, vertical sub-bands, diagonal sub-bands and low-frequency sub-bands; the horizontal sub-bands refer to the high-frequency features in the horizontal direction of the initial facial features or the facial features output by the last downsampling; the vertical sub-bands refer to the high-frequency features in the vertical direction of the initial facial features or the facial features output by the last downsampling; the diagonal sub-bands refer to the high-frequency features in the diagonal direction of the initial facial features or the facial features output by the last downsampling; the low-frequency sub-bands refer to the low-frequency features in the initial facial features or the facial features output by the last downsampling.

[0175] Optionally, the downsampling module is also used to: perform feature enhancement on the first low-frequency feature through a global transformer corresponding to the downsampling to obtain the enhanced first low-frequency feature; perform feature enhancement on the multiple first high-frequency features through a residual block corresponding to the downsampling to obtain the enhanced multiple first high-frequency features.

[0176] Optionally, the device also includes: a feature extraction module, used to extract the original facial features of the low-resolution facial image through a convolution layer before the initial facial features of the low-resolution facial image are successively down-sampled N times according to a preset sampling rate; feature extraction of the original facial features is performed through a global transformer to obtain the initial facial features; the initial facial features include local facial features, regional facial features and global facial features; the initial facial features are used as input for the first downsampling in the N downsamplings.

[0177] Optionally, the feature extraction module is also used to perform feature segmentation on the original facial features to obtain multiple window features; filter out at least one target window feature from the multiple window features, and the target window feature refers to a window feature among the multiple window features used to enhance local facial details; for any one of the at least one target window feature, calculate the self-attention weight of the target window feature; use the self-attention weight to perform weighted summation on the value vector of the target window feature to obtain a first local facial feature; and splice at least one first local facial feature corresponding to the at least one target window feature to obtain a regional facial feature of the original facial feature.

[0178] Optionally, the feature extraction module is also used to: after obtaining the regional facial features of the original facial features, perform random window movement on the multiple window features to obtain new multiple window features; and based on the new multiple window features, perform the step of screening out at least one target window feature from the multiple window features multiple times to obtain multiple regional facial features.

[0179] Optionally, the feature extraction module is also used to: fuse the regional facial features and the original facial features to obtain a fused facial feature; set multiple self-attention heads, each self-attention head is used to calculate the self-attention weight of some features in the fused facial feature; use the self-attention weight to perform weighted summation on the value vectors of the partial features to obtain a second local facial feature; splice multiple second local facial features output by the multiple self-attention heads to obtain the global facial feature.

[0180] Optionally, the feature extraction module is further used to: shuffle the partial features that the multiple self-attention heads are responsible for, so that each self-attention head is responsible for a different partial feature each time, thereby enhancing information exchange between the multiple self-attention heads.

[0181] The facial image super-resolution reconstruction device provided by the above-mentioned embodiment of the present disclosure and the facial image super-resolution reconstruction method provided by the embodiment of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0182] The present disclosure also provides an electronic device to perform the above-mentioned facial image super-resolution reconstruction method. Fig.12 , which shows a schematic diagram of an electronic device provided by some embodiments of the present disclosure. Fig.12 As shown, the electronic device 12 includes: a processor 1200, a memory 1201, a bus 1202 and a communication interface 1203, wherein the processor 1200, the communication interface 1203 and the memory 1201 are connected via the bus 1202; the memory 1201 stores a computer program that can be run on the processor 1200, and the processor 1200 executes the aforementioned Figures 1 to 10 A method for super-resolution reconstruction of facial images provided by any of the illustrated embodiments.

[0183] The memory 1201 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 1203 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0184] The bus 1202 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 1201 is used to store programs. After receiving the execution instruction, the processor 1200 executes the program. Figures 1 to 10 The facial image super-resolution reconstruction method disclosed in any of the illustrated embodiments may be applied to the processor 1200 or implemented by the processor 1200 .

[0185] The processor 1200 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 1200 or the instruction in the form of software. The above processor 1200 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present disclosure may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1201, and the processor 1200 reads the information in the memory 1201 and completes the steps of the above method in combination with its hardware.

[0186] The electronic device provided by the embodiment of the present disclosure and the facial image super-resolution reconstruction method provided by the embodiment of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by them.

[0187] The present disclosure also provides a computer-readable storage medium corresponding to the facial image super-resolution reconstruction method provided in the above embodiment. Fig.13 The computer-readable storage medium shown is a CD 30 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the method for super-resolution reconstruction of a facial image provided by any of the aforementioned embodiments will be executed.

[0188] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0189] The computer-readable storage medium provided by the above-mentioned embodiments of the present disclosure and the facial image super-resolution reconstruction method provided by the embodiments of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0190] It should be noted that:

[0191] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present disclosure can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0192] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present disclosure, the various features of the present disclosure are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following schematic diagram: the claimed disclosure requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims below, the inventive aspects lie in less than all the features of the single embodiment disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present disclosure.

[0193] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of this disclosure and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0194] The above is only a preferred specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for super-resolution reconstruction of facial images, characterized in that: The method comprises: The initial facial features of the low-resolution face image are downsampled N times successively according to a preset sampling rate, and the facial features obtained by any downsampling are obtained by fusion of the enhanced first low-frequency features and the enhanced multiple first high-frequency features, wherein the first low-frequency features and the multiple first high-frequency features are obtained by wavelet transforming the input facial features, and the input facial features are the initial facial features or the facial features output by the last downsampling; The facial features output by the Nth downsampling are upsampled N times successively according to the preset sampling rate to output enhanced facial features, the facial features obtained by any upsampling are obtained by inverse wavelet transform of low-frequency fusion features and multiple enhanced second high-frequency features, the low-frequency fusion features are obtained by fusion of the second low-frequency features and the third low-frequency features, the second low-frequency features and the multiple second high-frequency features are obtained by wavelet transform of the facial features output by the downsampling corresponding to the upsampling, and the third low-frequency features refer to the low-frequency features in the facial features output by the last upsampling or the last downsampling; The initial facial features and the enhanced facial features are fused to obtain a facial image reconstructed from the low-resolution facial image.

2. The method according to claim 1, characterized in that For any downsampling in the N downsamplings, the downsampling step includes: Performing wavelet transformation on the initial facial features of the low-resolution facial image or the facial features output by the last downsampling to obtain the first low-frequency features and the plurality of first high-frequency features; Performing feature enhancement on the first low-frequency feature and the plurality of first high-frequency features respectively to obtain an enhanced first low-frequency feature and an enhanced plurality of first high-frequency features; The enhanced first low-frequency feature and the enhanced multiple first high-frequency features are fused to obtain a fused feature; the fused feature is used as the facial feature of the down-sampling output.

3. The method according to claim 1 or 2, characterized in that: For any upsampling in the N upsamplings, the upsampling step includes: Acquire a third low-frequency feature in the first facial feature outputted by the last upsampling or the last downsampling, and acquire a second facial feature outputted by the downsampling corresponding to the upsampling; the resolution of the second facial feature is greater than the resolution of the first facial feature; Performing wavelet transformation on the second facial feature to obtain the second low-frequency feature and the plurality of second high-frequency features; Fusing the second low-frequency feature and the third low-frequency feature to obtain the low-frequency fusion feature; The low-frequency fusion feature and the plurality of second high-frequency features are subjected to inverse wavelet transformation to obtain a target facial feature; the target facial feature is used as the facial feature of the up-sampling output.

4. The method according to claim 2, characterized in that: The first low-frequency feature is a low-frequency sub-band; the multiple first high-frequency features include a horizontal sub-band, a vertical sub-band and a diagonal sub-band; the initial facial features of the low-resolution face image or the facial features output by the last downsampling are subjected to wavelet transformation to obtain the first low-frequency feature and the multiple first high-frequency features, including: Decompose the initial facial features or the facial features output by the last downsampling into horizontal sub-bands, vertical sub-bands, diagonal sub-bands and low-frequency sub-bands; the horizontal sub-bands refer to the high-frequency features in the horizontal direction of the initial facial features or the facial features output by the last downsampling; the vertical sub-bands refer to the high-frequency features in the vertical direction of the initial facial features or the facial features output by the last downsampling; the diagonal sub-bands refer to the high-frequency features in the diagonal direction of the initial facial features or the facial features output by the last downsampling; the low-frequency sub-bands refer to the low-frequency features in the initial facial features or the facial features output by the last downsampling.

5. The method according to claim 2, characterized in that: The N downsamplings correspond one-to-one to N global transformers and one-to-one to N residual blocks; the first low-frequency feature and the plurality of first high-frequency features are respectively enhanced to obtain an enhanced first low-frequency feature and an enhanced plurality of first high-frequency features, including: Performing feature enhancement on the first low-frequency feature by a global transformer corresponding to the downsampling to obtain the enhanced first low-frequency feature; The plurality of first high-frequency features are feature enhanced by using a residual block corresponding to the downsampling to obtain the plurality of enhanced first high-frequency features.

6. The method according to claim 1 or 2, characterized in that: Before downsampling the initial facial features of the low-resolution facial image N times in succession according to a preset sampling rate, the method further includes: Extracting original facial features of the low-resolution face image through a convolutional layer; The original facial features are extracted by a global transformer to obtain the initial facial features; the initial facial features include local facial features, regional facial features and global facial features; the initial facial features are used as input for the first downsampling in the N downsamplings.

7. The method according to claim 6, characterized in that The initial facial features include local facial features and regional facial features; extracting the original facial features through a global transformer to obtain the initial facial features includes: Performing feature segmentation on the original facial features to obtain multiple window features; Filtering out at least one target window feature from the multiple window features, wherein the target window feature refers to a window feature among the multiple window features for enhancing local facial details; For any one of the at least one target window feature, calculating a self-attention weight of the target window feature; Using the self-attention weight, weighted sum is performed on the value vector of the target window feature to obtain a first local facial feature; At least one first local facial feature corresponding to the at least one target window feature is spliced ​​to obtain a regional facial feature of the original facial feature.

8. The method according to claim 7, characterized in that After obtaining the regional facial features of the original facial features, the method further includes: Performing random window movement on the multiple window features to obtain multiple new window features; According to the new multiple window features, the step of selecting at least one target window feature from the multiple window features is performed multiple times to obtain multiple regional facial features.

9. The method according to claim 7 or 8, characterized in that: The facial features also include global facial features; The method further comprises: Fusing the regional facial features with the original facial features to obtain fused facial features; Setting a plurality of self-attention heads, each of which is used to calculate the self-attention weight of a partial feature in the fused facial feature; using the self-attention weight to perform weighted summation on the value vectors of the partial features to obtain a second local facial feature; The multiple second local facial features output by the multiple self-attention heads are spliced ​​to obtain the global facial feature.

10. The method according to claim 9, characterized in that The method further comprises: The partial features that the multiple self-attention heads are responsible for are shuffled so that each self-attention head is responsible for different partial features each time, thereby enhancing the information exchange between the multiple self-attention heads.

11. The method according to claim 7, characterized in that Calculating the self-attention weight of the target window feature includes: Performing a linear transformation on the target window feature to generate a query vector, a key vector and the value vector; Calculating an attention score of the target window feature based on the query vector and the key vector; The attention score is normalized to obtain the self-attention weight.

12. A facial image super-resolution reconstruction device, characterized in that: The device comprises: A downsampling module, used for downsampling the initial facial features of the low-resolution face image N times in sequence according to a preset sampling rate, wherein the facial features obtained by any downsampling are obtained by fusion of the enhanced first low-frequency features and the enhanced multiple first high-frequency features, wherein the first low-frequency features and the multiple first high-frequency features are obtained by wavelet transforming the input facial features, and the input facial features are the initial facial features or the facial features output by the last downsampling; An upsampling module is used to perform N upsampling on the facial features output by the Nth downsampling according to the preset sampling rate to output enhanced facial features, wherein the facial features obtained by any upsampling are obtained by inverse wavelet transformation of low-frequency fusion features and multiple enhanced second high-frequency features, wherein the low-frequency fusion features are obtained by fusion of the second low-frequency features and the third low-frequency features, wherein the second low-frequency features and the multiple second high-frequency features are obtained by wavelet transformation of the facial features output by the downsampling corresponding to the upsampling, and the third low-frequency features refer to the low-frequency features in the facial features output by the last upsampling or the last downsampling; An image reconstruction module is used to fuse the initial facial features with the enhanced facial features to obtain a facial image reconstructed from the low-resolution facial image.

Images