Face image reconstruction method, system and device based on adaptive texture and frequency domain perception and medium

Abstract

The application discloses a face image reconstruction method, system and device based on adaptive texture and frequency domain perception, and a medium. A self-attention calculation is performed on coarse features by an adaptive texture perception module, then the attention matrix is used to search for a part with strong correlation in fine features, and then fine self-attention calculation is performed. A multi-dimensional perception module is used to enhance the expression ability of features in cross-space and cross-channel. A multi-frequency fusion module based on wavelet transform is used to fuse the middle and low frequency features from the encoder and the high frequency features from the decoder. The adaptive texture perception module enables the model to more finely restore complex regions in the image, the multi-dimensional perception module enhances important channel information in the features, and the multi-frequency fusion module eliminates high frequency noise in the original features, and fuses the effective middle and low frequency features with the restored high frequency details, thereby improving the restoration ability and generalization ability of the model for face images.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and deep learning technology, and relates to a method, system, device and medium for face image reconstruction based on adaptive texture and frequency domain perception. Background Technology

[0002] Deep convolutional neural networks (CNNs) are a type of neural network used in image processing, computer vision, and pattern recognition. They are feedforward neural networks that use techniques such as convolutional layers and pooling layers to extract features from images, enabling tasks such as image classification, object detection, and face recognition. Face image super-resolution reconstruction is an image processing technique that converts low-resolution images into high-resolution images. In applications such as face recognition and video surveillance, it is necessary to convert low-resolution images to high-resolution images to obtain better image quality and higher accuracy. Traditional face image super-resolution reconstruction methods mainly include interpolation methods and edge-based methods. Interpolation methods obtain high-resolution images by interpolating pixels in a low-resolution image, but this method can lead to image blurring and distortion. Edge-based methods reconstruct low-resolution images based on the structural information of face image edges, but this method also has some limitations in some complex situations.

[0003] In recent years, deep learning-based super-resolution reconstruction methods for face images have gradually become mainstream. Among them, methods using deep convolutional neural networks (CNNs) are widely used. CNNs can extract high-level features from low-resolution images by stacking multiple convolutional blocks with different functions to extract shallow-to-deep features from the original image, and then use these extracted features to achieve low-resolution to high-resolution image conversion. However, in some complex situations, CNNs are limited by the size of the convolutional kernels and cannot effectively establish dependencies on global features, making it difficult for the network to accurately identify and reconstruct image details and textures, resulting in image distortion and blurring.

[0004] Due to its effectiveness in natural language processing, the Transformer model has recently been applied to computer vision. Compared to traditional deep convolutional neural networks, Transformer-based models offer advantages such as long-range dependency modeling capabilities, global awareness, scalability, and spatial independence, achieving better performance in various computer vision tasks and bringing new possibilities to research and applications in the field. The Transformer model divides an image into blocks of equal size and then establishes global dependencies through a self-attention mechanism, but this leads to high computational costs. The Swin Transformer model proposes a method to compute self-attention within a local window, using a sliding window to achieve interaction of adjacent information. This method has achieved excellent results in several computer vision tasks. However, when dealing with face super-resolution reconstruction, the Swin Transformer model cannot effectively model non-local dependencies in face images. Furthermore, the fixed rectangular window may result in unrelated textures within the window, leading to artifacts in the reconstruction results and affecting the recovery of complex regions in the face. Summary of the Invention

[0005] The purpose of this invention is to solve the problems of high computational cost of Transformer models, the inability of SwingTransformer models to model non-local dependencies in face images well, and the potential lack of correlation of textures within fixed rectangular windows in existing technologies. This invention provides a face image reconstruction method, system, device, and medium based on adaptive texture and frequency domain awareness.

[0006] To achieve the above objectives, the present invention employs the following technical solution:

[0007] Face image reconstruction methods based on adaptive texture and frequency domain awareness include:

[0008] The cropped face image dataset to be reconstructed is obtained and divided into training and test sets.

[0009] An adaptive texture perception module is constructed to perform fine feature recovery on complex texture regions in human faces;

[0010] Based on a multidimensional perception module composed of alternating deep convolutions and channel attention, features are enhanced from both spatial and channel dimensions to obtain features with multidimensional perception.

[0011] A multi-frequency fusion module is designed. Based on wavelet transform, the low- and mid-frequency information in the features are extracted from the encoder stage. The low- and mid-frequency information and the high-frequency information in the decoder stage are fused using inverse wavelet transform to obtain the output features with full frequency domain information after fusion.

[0012] Based on the U-shaped stacked adaptive texture perception module, multi-dimensional perception module and multi-frequency fusion module, a face image super-resolution reconstruction model based on adaptive texture and frequency domain perception is obtained.

[0013] The adaptive texture and frequency domain-aware face super-resolution reconstruction model is trained based on the training set to obtain the optimal model;

[0014] Based on the optimized model, face image super-resolution reconstruction is performed on the test set to obtain the restored high-resolution face images.

[0015] A further improvement of the present invention is that:

[0016] Furthermore, the ratio of the training set to the test set can be randomly divided or divided according to a preset ratio; the preset ratio is set manually.

[0017] Furthermore, an adaptive texture perception module is constructed to perform fine feature recovery on complex texture regions in the face, specifically:

[0018] The algorithm searches for regions with complex textures within a coarse texture and performs fine-grained self-attention calculations on these regions to achieve texture-aware restoration. The adaptive texture-aware module includes self-attention branches for coarse features and fine features. The formula for calculating the self-attention branch for coarse features is as follows:

[0019] Q C ,K C V C =Linear(Down(F) input ))

[0020] TSM=Q c K C T

[0021]

[0022] Among them, F input The input feature map is represented by Down and Linear, which are downsampling and linear projection operations, respectively. Softmax is the activation function, and TSM is the texture self-attention search map. Then, the k regions with the largest attention values ​​in the search map are selected and mapped to the fine feature map. The fine features of the corresponding regions are then searched to calculate multi-head self-attention.

[0023] F fine =FindByRegion(F input Topk(Sum(TSM),k))

[0024] Attention fine =MHSA(F fine )

[0025] Among them, F fine This represents the refined features found through the search. Sum indicates accumulation along the second-to-last dimension, Topk represents the k largest values ​​in TSM, and FindByRegion searches for the corresponding region based on the coordinates of the maximum value; k represents the number of regions to search.

[0026]

[0027] H and W are the input features F, respectively. input The height and width; MHSA stands for Multi-Head Self-Attention Computation;

[0028] The calculation results of the two branches are combined as follows:

[0029] F output =Up(Attention) coarse +Attention fine

[0030] Here, Up represents the upsampling operation.

[0031] Furthermore, a multidimensional perception module based on alternating deep convolutions and channel attention enhances features across spatial and channel dimensions, resulting in features with multidimensional perception. Specifically, the multidimensional perception module aggregates important information from both spatial and channel dimensions. The multidimensional perception module consists of two branches: alternating deep convolutions and channel attention. Deep convolutions aggregate and fuse features in the space, while channel attention uses a gating mechanism to evaluate the weights of each channel and multiplies the corresponding weights by the features of each channel. The alternating deep convolutions and channel attention input and output important information across multiple dimensions.

[0032] Furthermore, wavelet transform is used to extract mid-to-low frequency information from the features in the encoder stage, and inverse wavelet transform is used to fuse the mid-to-low frequency information with the high frequency information in the decoder stage to obtain features with rich frequency domain information, including:

[0033] The multi-frequency fusion module performs wavelet transform on the output features of each stage of the encoder to extract the corresponding mid- and low-frequency information; based on the mid- and low-frequency information and the output features of each stage of the decoder, it performs inverse wavelet transform to obtain the fused output features with full-frequency domain information; the specific process expression is as follows:

[0034] LL i ,LH i HL i =Wavelet(Fi )

[0035]

[0036] Among them, F i , Let represent the output features of the encoder, decoder, and multi-frequency fusion module at the i-th stage, respectively. Wavelet and InverseWavelet represent wavelet transform and inverse wavelet transform, respectively. LL i LH i and HL i They represent the features F respectively i The three types of low-to-medium frequency features extracted are shown in Conv, which represents a 1×1 convolution operation.

[0037] Furthermore, based on the U-shaped stacked adaptive texture perception module, multi-dimensional perception module, and multi-frequency fusion module, a face image super-resolution reconstruction model based on adaptive texture and frequency domain perception is obtained. Specifically, the encoder and decoder are each composed of four stacked adaptive texture perception modules, and the encoder and decoder are connected by an adaptive texture perception module with residuals. The output features of the encoder and decoder at the same level are connected through the multi-frequency fusion module.

[0038] Furthermore, the adaptive texture and frequency domain-aware face super-resolution reconstruction model is trained based on the training set to obtain the optimal model. Specifically, during the training process, it is determined whether the loss function of the adaptive texture and frequency domain-aware face super-resolution reconstruction model is lower than the set threshold, or whether the number of iterations of the adaptive texture and frequency domain-aware face super-resolution reconstruction model has reached the maximum value. If the loss function is lower than the set threshold or the number of iterations has reached the maximum value, training is stopped and the optimal model is obtained.

[0039] A face image reconstruction system based on adaptive texture and frequency domain awareness includes:

[0040] A partitioning module, which acquires and divides the cropped face image dataset into a training set and a test set;

[0041] The first construction module performs fine feature recovery on complex texture regions in the face;

[0042] A multidimensional perception module enhances features from the dimensions of cross-space and cross-channel to obtain features with multidimensional perception.

[0043] The multi-frequency fusion module extracts low- and mid-frequency information from the features of the encoder stage based on wavelet transform, and fuses the low- and mid-frequency information with the high-frequency information of the decoder stage using inverse wavelet transform to obtain the fused output features with full-frequency domain information.

[0044] The second construction module is based on a U-shaped stacked adaptive texture perception module, multi-dimensional perception module and multi-frequency fusion module to obtain a face image super-resolution reconstruction model based on adaptive texture and frequency domain perception.

[0045] The training module trains the adaptive texture and frequency domain-aware face super-resolution reconstruction model based on the training set to obtain the optimal model.

[0046] The reconstruction module performs face image super-resolution reconstruction on the test set based on an optimized model to obtain the restored high-resolution face images.

[0047] A terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the steps of the method described above.

[0048] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] This invention utilizes an adaptive texture perception module to model complex image regions in a coarse-to-fine manner, restoring finer textures and enhancing the model's ability to handle complex textures. Furthermore, a multi-dimensional perception module allows features to interact across multiple dimensions, enhancing the global perception capability of features. In addition, this method can aggregate important information, improving the model's utilization of features. Through a multi-frequency fusion module, high-frequency noise in the original features is effectively eliminated, and effective high-frequency information is recovered in the decoder stage, enhancing the effective information of features across the entire frequency domain and improving the model's generalization ability. Attached Figure Description

[0051] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1This is a flowchart illustrating a face image reconstruction method based on adaptive texture and frequency domain awareness according to the present invention.

[0053] Figure 2 This is another flowchart illustrating the face image reconstruction method based on adaptive texture and frequency domain awareness of the present invention.

[0054] Figure 3 This is a schematic diagram of the adaptive texture perception module structure according to an embodiment of the present invention;

[0055] Figure 4 This is a schematic diagram of the multi-dimensional sensing module structure according to an embodiment of the present invention;

[0056] Figure 5 This is a schematic diagram of the structure of the multi-frequency fusion module according to an embodiment of the present invention;

[0057] Figure 6 This is a structural diagram of an adaptive texture and frequency domain-aware face image super-resolution reconstruction model provided for embodiments of the present invention;

[0058] Figure 7 These are comparison images of face images reconstructed by LR, RCAN, DIC, SwinIR, SRDD, HGSRCNN, ATFDP Net, and HR according to embodiments of the present invention.

[0059] Figure 8 This is a schematic diagram of the structure of a face image reconstruction system based on adaptive texture and frequency domain perception according to an embodiment of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0061] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0062] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0063] In the description of the embodiments of the present invention, it should be noted that if terms such as "upper," "lower," "horizontal," or "inner" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of the invention is in use, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Furthermore, terms such as "first" and "second" are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0064] Furthermore, the use of the term "horizontal" does not imply that the component must be absolutely horizontal, but rather that it can be slightly tilted. For example, "horizontal" simply means that its direction is more horizontal than "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0065] In the description of the embodiments of the present invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention according to the specific circumstances.

[0066] The present invention will now be described in further detail with reference to the accompanying drawings:

[0067] See Figure 1 This invention discloses a face image reconstruction method based on adaptive texture and frequency domain awareness, comprising:

[0068] S101, Obtain and divide the cropped face image dataset into training and test sets.

[0069] The ratio of training set to test set can be randomized or set according to a preset ratio; the preset ratio is manually set.

[0070] S102, Construct an adaptive texture perception module to perform fine feature recovery on complex texture regions in the face;

[0071] The algorithm searches for regions with complex textures within a coarse texture and performs fine-grained self-attention calculations on these regions to achieve texture-aware restoration. The adaptive texture-aware module includes self-attention branches for coarse features and fine features. The formula for calculating the self-attention branch for coarse features is as follows:

[0072] Q C,K C V C =Linear(Down(F) input ))

[0073] TSM=Q C K C T

[0074]

[0075] Among them, F input The input feature map is represented by Down and Linear, which are downsampling and linear projection operations, respectively. Softmax is the activation function, and TSM is the texture self-attention search map. Then, the k regions with the largest attention values ​​in the search map are selected and mapped to the fine feature map. The fine features of the corresponding regions are then searched to calculate multi-head self-attention.

[0076] F fine =FindByRegion(F input Topk(Sum(TSM),k))

[0077] Attention fine =MHSA(F fine )

[0078] Among them, F fine This represents the refined features found through the search. Sum indicates accumulation along the second-to-last dimension, Topk represents the k largest values ​​in TSM, and FindByRegion searches for the corresponding region based on the coordinates of the maximum value; k represents the number of regions to search.

[0079]

[0080] H and W are the input features F, respectively. input The height and width; MHSA stands for Multi-Head Self-Attention Computation;

[0081] The calculation results of the two branches are combined as follows:

[0082] F output =Up(Attention) coarse +Attention fine

[0083] Here, Up represents the upsampling operation.

[0084] S103, based on a multidimensional perception module composed of alternating deep convolutions and channel attention, enhances features from both spatial and channel dimensions to obtain features with multidimensional perception.

[0085] The multidimensional perception module gathers important information from both spatial and channel dimensions. The multidimensional perception module consists of two branches: alternating deep convolution and channel attention. Deep convolution gathers and fuses features in space, while channel attention evaluates the weights of each channel through a gating mechanism and multiplies the corresponding weights by the features of each channel itself. The alternating deep convolution and channel attention input and output important information in multiple dimensions.

[0086] S104, design a multi-frequency fusion module, which extracts mid- and low-frequency information from the features in the encoder stage based on wavelet transform, and uses inverse wavelet transform to fuse the mid- and low-frequency information with the high-frequency information in the decoder stage to obtain the fused output features with full-frequency domain information.

[0087] The multi-frequency fusion module performs wavelet transform on the output features of each stage of the encoder to extract the corresponding mid- and low-frequency information; based on the mid- and low-frequency information and the output features of each stage of the decoder, it performs inverse wavelet transform to obtain the fused output features with full-frequency domain information; the specific process expression is as follows:

[0088] LL i ,LH i HL i =Wavelet(F i )

[0089]

[0090] Among them, F i , Let represent the output features of the encoder, decoder, and multi-frequency fusion module at the i-th stage, respectively. Wavelet and InverseWavelet represent wavelet transform and inverse wavelet transform, respectively. LL i LH i and HL i They represent the features F respectively i The three types of low-to-medium frequency features extracted are shown in Conv, which represents a 1×1 convolution operation.

[0091] S105, based on the U-shaped stacked adaptive texture perception module, multi-dimensional perception module and multi-frequency fusion module, obtains a face image super-resolution reconstruction model based on adaptive texture and frequency domain perception.

[0092] The encoder and decoder are each composed of four stacked adaptive texture perception modules. The encoder and decoder are connected by an adaptive texture perception module with residuals. The output features of the encoder and decoder at the same level are connected through a multi-frequency fusion module.

[0093] S106, based on the training set, train the adaptive texture and frequency domain-aware face super-resolution reconstruction model to obtain the optimal model.

[0094] During training, it is determined whether the loss function of the adaptive texture and frequency domain aware face super-resolution reconstruction model is lower than the set threshold, or whether the number of iterations of the adaptive texture and frequency domain aware face super-resolution reconstruction model has reached the maximum value. If the loss function is lower than the set threshold or the number of iterations has reached the maximum value, training is stopped and the optimal model is obtained.

[0095] S107, perform face image super-resolution reconstruction on the test set based on the optimized model to obtain the restored high-resolution face image.

[0096] Example:

[0097] The following describes some embodiments of this application in detail with reference to the accompanying drawings:

[0098] See Figure 2 The present invention provides a flowchart of a face image reconstruction method based on adaptive texture and frequency domain awareness, comprising:

[0099] Step S1: Obtain and divide the cropped face image dataset into a training set and a test set; the training set is used for training the model, and the test set is used to test the trained model. The above dataset division can be random or divided according to a preset ratio; this embodiment does not impose any limitations.

[0100] Step S2: The adaptive texture perception module is used to perform fine feature recovery on complex regions of the face. By calculating coarse attention and performing region search, regions with complex textures are adaptively found, and fine attention is further calculated for these regions.

[0101] The texture restoration feature representation obtained above can be: Let X be the input feature of the i-th adaptive texture awareness module. i , using ATPM i (·) represents the i-th adaptive texture perception module in the adaptive texture and frequency domain aware face super-resolution reconstruction model. Then, the feature output by the i-th adaptive texture perception module in the adaptive texture and frequency domain aware face super-resolution reconstruction model is: Y i =ATPM i (X i ).

[0102] The process of the adaptive texture perception module can be represented as follows:

[0103] Q C ,K C VC =Linear(Down(X) i ))

[0104] TSM=Q C K C T

[0105]

[0106] Where X i The input feature map is represented by the symbols: Down and Linear represent downsampling and linear projection operations, respectively; Softmax is the activation function; and TSM is the texture self-attention search map. Then, the k regions with the highest attention values ​​in the search map are selected and mapped to the fine feature map. The fine features of the corresponding regions are then searched to calculate multi-head self-attention.

[0107] F fine =FindByRegion(F ibput Topk(Sum(TSM),k))

[0108] Attention fine =MHSA(F fine )

[0109] Where F fine This represents the refined features found through the search. `Sum` indicates accumulation along the second-to-last dimension, `Topk` represents extracting the k largest values ​​in the TSM, and `FindByRegion` searches for the corresponding region based on the coordinates of the maximum value. `k` represents the number of regions to search, which is set here to [value missing]. H and W are the input features F, respectively. input The height and width. MHSA represents multi-head self-attention calculation. Finally, we combine the calculation results of the two branches, the formula is Y. i =Up(Attention) coarse +Attention fine , where Up represents the upsampling operation.

[0110] Figure 3 The adaptive texture perception module provided in this embodiment of the invention can model complex regions of an image from coarse to fine, enabling these regions to recover more refined textures and enhancing the model's ability to process complex textures.

[0111] Step S3: Employ a multi-dimensional perception module to aggregate useful information. This module can operate across multiple dimensions, both spatially and across channels. Specifically, it includes alternating channel attention and depthwise convolution, which extracts and enhances multi-dimensional representations of input features, thereby obtaining a globally perceptive feature representation. The resulting globally perceptive feature representation can be: Let X be the input feature of the multi-dimensional perception module in the i-th stage. i Use MDPM i (·) represents the i-th multidimensional sensing module. Then the feature output by the i-th multidimensional sensing module is: Y i =MDPM i (X i ).

[0112] like Figure 4 As shown, the multidimensional perception module consists of two information extraction branches and a residual connection. The information extraction branches include alternating deep convolutional channel attention modules, representing interactive processing of information from spatial and channel dimensions, respectively, allowing features to be fused from multiple dimensions and enhancing the global perception capability of features. Furthermore, this method can also aggregate important information, enhancing the model's utilization of features.

[0113] Step S4: Employ the multi-frequency fusion module, see [link / reference] Figure 5 The low- and mid-frequency information extracted from the encoder stage using wavelet transform is fused with the high-frequency information from the decoder stage to obtain features with rich frequency domain information. The fusion process is as follows:

[0114] LL i ,LH i HL i =Wavelet(F i ),

[0115]

[0116] Where F i , Let represent the output features of the encoder, decoder, and multi-frequency fusion module at the i-th stage, respectively. Wavelet and InverseWavelet represent wavelet transform and inverse wavelet transform, respectively. LL i LH i and HL i They represent the features F respectively i The three types of low-to-medium frequency features extracted are shown in Conv, which represents a 1×1 convolution operation.

[0117] By using a multi-frequency fusion module, high-frequency noise is effectively eliminated in the encoder stage, while high-frequency information is effectively recovered in the decoder stage. This enhances the effective information of the original features in the full frequency domain, thereby improving the generalization ability of the model.

[0118] Step S5: Stack multiple adaptive texture perception modules, multi-dimensional perception modules, and multi-frequency fusion modules in a U-shaped structure. Figure 6 This invention provides a structural diagram of an adaptive texture and frequency domain-aware face image super-resolution reconstruction model. The network uses a 3×3 convolution and a LeakyReLU activation function for shallow feature extraction, which is then input into the encoder. Each module in the encoder stage consists of a multi-dimensional perception module, an adaptive texture perception module, a wavelet transform, and a downsampling operation. The low-to-mid-frequency information after the wavelet transform is directly input into the multi-frequency fusion module of the corresponding stage in the decoder. After each encoder stage, the height and width of the feature map are reduced to half of their original values, while the number of channels is doubled. During the transition from encoder to decoder, each module contains only a multi-dimensional perception module and an adaptive texture perception module for deep feature extraction and processing. In the decoder stage, each module consists of a multi-frequency fusion module, a multi-dimensional perception module, and an adaptive texture perception module. The multi-frequency fusion module fuses the low-to-mid-frequency information from the encoder stage with the current high-frequency information, doubling the feature map resolution and halving the number of channels. Finally, a 3×3 convolution and a LeakyReLU activation function are used to recover the features, and the original input image is added to form a residual connection to obtain the output high-resolution face image.

[0119] Step S6: Using the face image training set, train the adaptive texture and frequency domain-aware face image super-resolution reconstruction model from step S5 using the gradient descent method based on the Adam optimizer.

[0120] Step S7: Using the trained adaptive texture and frequency domain-aware face image super-resolution reconstruction model, super-resolution reconstruction is performed on the face image test set to obtain the reconstructed high-resolution face image.

[0121] This invention utilizes an adaptive texture perception module to model complex image regions in a coarse-to-fine manner, restoring finer textures and enhancing the model's ability to handle complex textures. A multi-dimensional perception module allows features to interact across multiple dimensions, enhancing the global perception capability of features. Furthermore, this module can aggregate important information, improving the model's utilization of features. A multi-frequency fusion module effectively eliminates high-frequency noise in the original features and recovers effective high-frequency information in the decoder stage, enhancing the effective information of features across the entire frequency domain and improving the model's generalization ability.

[0122] The invention was tested on two mainstream face image datasets, the Helen dataset and the CelebA dataset. The experimental results show that the invention significantly improves the image reconstruction capability of the model.

[0123] The table showing the comparison of PSNR, SSIM, and LPIPS values ​​on the Helen and CelebA face datasets is presented. This invention compares PSNR, SSIM, and LPIPS values ​​with other benchmark methods on the Helen and CelebA face datasets. The benchmark methods compared include Bicubic, RCAN, DIC, SwinIR, SRDD, and HGSRCNN. The quantitative results in Table 1 show that the ATFDP Net adaptive texture and frequency domain-aware face image super-resolution reconstruction model of this invention significantly outperforms other methods in all three metrics.

[0124] Table 1: Comparison of PNSR, SSIM, and LPIPS values ​​on the Helen and CelebA face datasets.

[0125]

[0126] Figure 7 This paper showcases partial face images reconstructed using the aforementioned benchmark methods and the proposed adaptive texture and frequency domain awareness model ATFDPNet (where LR and HR represent the input low-resolution image and the corresponding high-resolution image, respectively). Figure 7 It can be seen that the high-resolution face image reconstructed using the ATFDP Net super-resolution reconstruction model based on adaptive texture and frequency domain awareness proposed in this invention can recover more accurate details in complex areas of the face, such as the nose and eyes.

[0127] See Figure 8 This invention discloses a face image reconstruction system based on adaptive texture and frequency domain awareness, comprising:

[0128] A partitioning module, which acquires and divides the cropped face image dataset into a training set and a test set;

[0129] The first construction module performs fine feature recovery on complex texture regions in the face;

[0130] A multidimensional perception module enhances features from the dimensions of cross-space and cross-channel to obtain features with multidimensional perception.

[0131] The multi-frequency fusion module extracts low- and mid-frequency information from the features of the encoder stage based on wavelet transform, and fuses the low- and mid-frequency information with the high-frequency information of the decoder stage using inverse wavelet transform to obtain the fused output features with full-frequency domain information.

[0132] The second construction module is based on a U-shaped stacked adaptive texture perception module, multi-dimensional perception module and multi-frequency fusion module to obtain a face image super-resolution reconstruction model based on adaptive texture and frequency domain perception.

[0133] The training module trains the adaptive texture and frequency domain-aware face super-resolution reconstruction model based on the training set to obtain the optimal model.

[0134] The reconstruction module performs face image super-resolution reconstruction on the test set based on an optimized model to obtain the restored high-resolution face images.

[0135] An embodiment of the present invention provides a terminal device. This terminal device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the various method embodiments described above. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in the various device embodiments described above.

[0136] The computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention.

[0137] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0138] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0139] The memory can be used to store the computer program and / or module. The processor implements various functions of the terminal device by running or executing the computer program and / or module stored in the memory and calling the data stored in the memory.

[0140] If the modules / units integrated into the terminal device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0141] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A face image reconstruction method based on adaptive texture and frequency domain awareness, characterized in that, include: The cropped face image dataset to be reconstructed is obtained and divided into training and test sets. An adaptive texture perception module is constructed to perform fine feature recovery on complex texture regions in human faces; Based on a multidimensional perception module composed of alternating deep convolutions and channel attention, features are enhanced from both spatial and channel dimensions to obtain features with multidimensional perception. A multi-frequency fusion module is designed. Based on wavelet transform, the low- and mid-frequency information in the features are extracted from the encoder stage. The low- and mid-frequency information and the high-frequency information in the decoder stage are fused using inverse wavelet transform to obtain the output features with full frequency domain information after fusion. Based on the U-shaped stacked adaptive texture perception module, multi-dimensional perception module and multi-frequency fusion module, a face image super-resolution reconstruction model based on adaptive texture and frequency domain perception is obtained. The adaptive texture and frequency domain-aware face super-resolution reconstruction model is trained based on the training set to obtain the optimal model; Based on the optimized model, face image super-resolution reconstruction is performed on the test set to obtain the restored high-resolution face images; The multidimensional perception module, based on alternating deep convolutions and channel attention, enhances features across spatial and channel dimensions to obtain features with multidimensional perception. Specifically, the multidimensional perception module aggregates important information from both spatial and channel dimensions. The multidimensional perception module consists of two branches: alternating deep convolutions and channel attention. Deep convolutions aggregate and fuse features in the space, while channel attention uses a gating mechanism to evaluate the weights of each channel and multiplies the corresponding weights by the features of each channel itself. The alternating deep convolutions and channel attention input and output important information across multiple dimensions.

2. The face image reconstruction method based on adaptive texture and frequency domain awareness according to claim 1, characterized in that, The division ratio between the training set and the test set is either random or based on a preset ratio; the preset ratio is set manually.

3. The face image reconstruction method based on adaptive texture and frequency domain awareness according to claim 1, characterized in that, The adaptive texture perception module is constructed to perform fine feature recovery on complex texture regions in the face, specifically as follows: The system searches for regions with complex textures within coarse textures and performs fine self-attention calculations on these regions to achieve texture-aware restoration. The adaptive texture-aware module includes self-attention branches for coarse features and fine features. The formula for calculating the self-attention branch of coarse features is: in, Represents the input feature map, and These are downsampling and linear projection operations, respectively. For activation function, This is a texture self-attention search graph; then, the graph is selected from the graphs with the highest attention values. Each region is mapped to a fine feature map, and the fine features of the corresponding region are searched to calculate multi-head self-attention: in, This represents the refined features found during the search. This indicates accumulation along the penultimate dimension. Indicates in The largest One value, To search for the corresponding region based on the coordinates of the maximum value; Represents the number of regions searched. and Input features Height and width; Represents multi-head self-attention calculation; The calculation results of the two branches are combined as follows: in, This represents an upsampling operation.

4. The face image reconstruction method based on adaptive texture and frequency domain awareness according to claim 1, characterized in that, The process of extracting mid-to-low frequency information from features in the encoder stage based on wavelet transform, and fusing the mid-to-low frequency information with the high frequency information in the decoder stage using inverse wavelet transform to obtain fused output features with full-frequency domain information, includes: The multi-frequency fusion module performs wavelet transform on the output features of each stage of the encoder to extract the corresponding mid- and low-frequency information; based on the mid- and low-frequency information and the output features of each stage of the decoder, it performs inverse wavelet transform to obtain the fused output features with full-frequency domain information; the specific process expression is as follows: in, , , These represent the output features of the encoder, decoder, and multi-frequency fusion module at the i-th stage, respectively. and These represent wavelet transform and inverse wavelet transform, respectively. , and They represent the features respectively The three mid-to-low frequency features extracted from them This represents a 1×1 convolution operation.

5. The face image reconstruction method based on adaptive texture and frequency domain awareness according to claim 1, characterized in that, The U-shaped stacked adaptive texture perception module, multi-dimensional perception module, and multi-frequency fusion module are used to obtain a face image super-resolution reconstruction model based on adaptive texture and frequency domain perception. Specifically, the encoder and decoder are each composed of 4 stacked adaptive texture perception modules. The encoder and decoder are connected by an adaptive texture perception module with residuals. The output features of the encoder and decoder at the same level are connected through the multi-frequency fusion module.

6. The face image reconstruction method based on adaptive texture and frequency domain awareness according to claim 1, characterized in that, The process of training an adaptive texture and frequency domain-aware face super-resolution reconstruction model based on a training set to obtain the optimal model involves: during the training process, determining whether the loss function of the adaptive texture and frequency domain-aware face super-resolution reconstruction model is lower than a set threshold, or whether the number of iterations of the adaptive texture and frequency domain-aware face super-resolution reconstruction model has reached its maximum value. If the loss function is lower than the set threshold or the number of iterations has reached its maximum value, then training is stopped and the optimal model is obtained.

7. A face image reconstruction system based on adaptive texture and frequency domain awareness, characterized in that, include: A partitioning module, which acquires and divides the cropped face image dataset into a training set and a test set; The first construction module performs fine feature recovery on complex texture regions in the face; A multidimensional perception module enhances features from the dimensions of cross-space and cross-channel to obtain features with multidimensional perception. The multi-frequency fusion module extracts low- and mid-frequency information from the features of the encoder stage based on wavelet transform, and fuses the low- and mid-frequency information with the high-frequency information of the decoder stage using inverse wavelet transform to obtain the fused output features with full-frequency domain information. The second construction module is based on a U-shaped stacked adaptive texture perception module, multi-dimensional perception module and multi-frequency fusion module to obtain a face image super-resolution reconstruction model based on adaptive texture and frequency domain perception. The training module trains the adaptive texture and frequency domain-aware face super-resolution reconstruction model based on the training set to obtain the optimal model. The reconstruction module performs face image super-resolution reconstruction on the test set based on an optimized model to obtain the restored high-resolution face images. The multidimensional perception module enhances features across spatial and channel dimensions to obtain features with multidimensional perception. Specifically, the multidimensional perception module aggregates important information from both spatial and channel dimensions. The multidimensional perception module consists of two branches: alternating deep convolution and channel attention. Deep convolution aggregates and fuses features in the space, while channel attention evaluates the weights of each channel through a gating mechanism and multiplies the corresponding weights by the features of each channel. The alternating deep convolution and channel attention input and output important information across multiple dimensions.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-6.

Images