A face image restoration method based on state space model

Abstract

The present invention discloses a method for restoring a face image based on a state space model, the method comprising: inputting a face image to be restored into a restoration model to obtain a restored face image; wherein the restoration model comprises an encoder and a decoder; the encoder comprises a first image fusion module, a first multi-scale state space module, a second image fusion module, a second multi-scale state space module, a third image fusion module, and a third multi-scale state space module in sequence; the decoder comprises a fourth multi-scale state space module, a first multi-scale attention fusion module, a fifth multi-scale state space module, a second multi-scale attention fusion module, a sixth multi-scale state space module, and a third multi-scale attention fusion module in sequence. The method ensures the consistency of face semantic information while restoring detail textures, and can achieve a better restoration effect for low-quality face images in real degraded scenes.

Description

Technical Field

[0001] The invention relates to a human face image restoration method based on state space, belonging to the technical field of image processing. Background Art

[0002] In scenarios such as audio-visual entertainment and security monitoring, high-quality and clear facial images can not only provide users with a good visual experience, but also help law enforcement officers search for suspects and missing persons. Due to the many unstable factors in shooting conditions, such as imaging equipment focus failure, camera shake; low light, high exposure, and motion interference of the object in the imaging environment; lossy compression of channel transmission, encoding and decoding formats, etc., multiple interferences will cause the image to degrade to varying degrees. The effective identity information provided by indistinguishable low-quality facial images is very limited. Therefore, facial image restoration aims to restore clear and high-quality facial images from degraded low-quality facial images, which helps to improve the robustness of downstream tasks, such as: face super-resolution and recognition, old photo restoration, virtual digital human image editing, etc.

[0003] At present, the model architectures used in face restoration methods are mainly based on convolutional neural networks (CNN) and Transformer networks. Due to the local characteristics of the convolution operation, the existing methods based on convolutional neural networks often fail to mine global information; the existing methods based on Transformer networks divide the image into several blocks and use the self-attention mechanism with large computational complexity to grasp the global information, but the ability to capture pixel-level detail information is limited.

[0004] The method has achieved certain positive results in the task of face restoration. However, when faced with face images that are larger in size, more complex and more severely degraded, it is unable to simultaneously mine and efficiently integrate local detail information and global geometric information, resulting in its restoration performance being unable to meet existing industry needs. Summary of the invention

[0005] In order to overcome the deficiencies in the prior art, the present invention provides a face image restoration method based on a state space model.

[0006] Compared with convolutional neural networks, the state space model architecture has global information mining capabilities; compared with the Transformer model architecture, it not only reduces computational overhead, but also has stronger temporal memory capabilities. The present invention further introduces multi-scale technology to alleviate the problem of insufficient fusion of local and global information, and can maintain the consistency of facial identity information while ensuring the restoration of detailed textures and geometric contours. When faced with complex real-world degraded scenes, the restoration method proposed in the present invention can not only achieve higher indicator scores, such as peak signal-to-noise ratio PSNR, but also restore high-quality clear facial images.

[0007] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0008] In a first aspect, the present invention discloses a method for restoring a face image based on a state space model, comprising the following steps:

[0009] Obtaining a face image to be restored;

[0010] Inputting the face image to be restored into a restoration model to obtain a restored face image;

[0011] Wherein, the restoration model includes an encoder and a decoder;

[0012] The encoder comprises a first image fusion module, a first multi-scale state space module, a second image fusion module, a second multi-scale state space module, a third image fusion module, and a third multi-scale state space module in sequence; the face image to be restored is fused with a 2-fold downsampled low-quality face image by the first image fusion module, and then extracted by the first multi-scale state space module to obtain first-stage features; the first-stage features are fused with a 4-fold downsampled low-quality face image by the second image fusion module, and then extracted by the second multi-scale state space module to obtain second-stage features; the second-stage features are fused with an 8-fold downsampled low-quality face image by the third image fusion module, and then extracted by the third multi-scale state space module to obtain third-stage features;

[0013] The decoder includes a fourth multiscale state space module, a first multiscale attention fusion module, a fifth multiscale state space module, a second multiscale attention fusion module, a sixth multiscale state space module, and a third multiscale attention fusion module in sequence; the third stage feature is subjected to the fourth multiscale state space module to obtain a 2-fold up-sampled restored face image, the 2-fold up-sampled restored face image is fused with the first stage feature, the second stage feature and the third stage feature through the first multiscale attention fusion module, and then is subjected to the fifth multiscale state space module to obtain a 4-fold up-sampled restored face image, the 4-fold up-sampled restored face image is fused with the first stage feature, the second stage feature and the third stage feature through the second multiscale attention fusion module, and then is subjected to the sixth multiscale state space module to obtain an 8-fold up-sampled restored face image, the 8-fold up-sampled restored face image is fused with the first stage feature, the second stage feature and the third stage feature through the third multiscale attention fusion module, and then is decoded and output, and is aligned and fitted with the input low-quality face image to obtain a restored high-quality face image.

[0014] In some embodiments, the network structures of the first image fusion module, the second image fusion module and the third image fusion module are the same, and all include a 2x downsampling unit, a feature extraction unit and a channel attention unit. Low-quality face images of different scales are input, and the shallow features mined by the feature extraction unit and the deep features output by different stages of the encoder are fused with key features using the channel attention unit to promote mutual learning between image pixels and semantic features.

[0015] Furthermore, the feature extraction unit includes a residual convolution block and an activation function; wherein the residual convolution block includes 2 convolution layers and 2 activation functions, a rectified linear function is used as the activation function, and the output features after the input features are extracted through the 2 convolution layers to extract deep features are added to the input features to obtain the final output;

[0016] Furthermore, the channel attention unit includes a dimensionality reduction unit, an attention mechanism operation for assigning weights to different channels, and a dimensionality increase unit.

[0017] In some embodiments, the network structures of the first multiscale state space module, the second multiscale state space module, the third multiscale state space module, the fourth multiscale state space module, the fifth multiscale state space module and the sixth multiscale state space module are the same, and all include multiple multiscale state space models and 1 feature extraction unit, each of the multiscale state space models includes 2 state space branches, 1 dimension reduction unit, 2 linear layers, 1 layer normalization unit and 1 dimension increase unit, and the state space branch includes 1 depthwise separable convolution, 1 layer normalization unit and 1 state space model; the output of the first state space branch is added to the output of the dimension reduction unit and then passes through the first linear layer and the second linear layer respectively, the output of the second state space branch is added to the output of the dimension reduction unit, and then feature multiplied with the output of the second linear layer, and then feature added with the output of the first linear layer, and finally passed through the layer normalization unit and dimension increased output, and the module feature mining and learning capabilities are enhanced by utilizing the long sequence deep feature extraction capability of the state space model.

[0018] Further, in some embodiments, the state space model includes 3 linear layers, 1 convolutional layer, 2 activation layers, and 1 selection state space model, and is connected in a residual manner; the first output branch of the layer normalization unit sequentially passes through the third linear layer, the convolutional layer, the first activation layer, and the selection state space model to obtain a first feature, and the second output branch of the layer normalization unit sequentially passes through the fourth linear layer and the second activation layer to obtain a second feature, and the first feature and the second feature are connected in a residual manner and then output through the fifth linear layer;

[0019] The expression of the selected state space model is:

[0020]

[0021] In the formula, u(t) represents the input signal, x(t) represents the historical state, and x ′ (t) represents the current state, y(t) represents the output signal; A represents the state transfer matrix, B is the matrix from input to state, C is the matrix from state to output, and D is the parameter from input to output.

[0022] Furthermore, since the input data of image processing is often discrete, by selecting a suitable discrete time step Δt, the ordinary differential equation of the state space model is converted into a discrete time difference equation according to the bilinear transformation method. The difference equation is:

[0023]

[0024] In the formula, x k represents the current k-th state, u k represents the kth discrete value of the input sequence, x k-1 Represents the k-1th historical state, by calculating the current input For the previous state The impact of x k , and predict the output Cx k ;in, represents the discretized state transfer matrix, A matrix representing the discretized input to the state;

[0025] By adjusting the step size Δt, the state-space model's attention to the current input is controlled, thereby achieving selective forgetting or retention of the state; when Δt is increased, the model tends to focus on the current input and forget the previous state; when Δt is reduced, the model tends to retain more historical states, thereby achieving a selective state-space model.

[0026] In some embodiments, the network structures of the first multi-scale attention fusion module, the second multi-scale attention fusion module and the third multi-scale attention fusion module are the same, and all include 1 universal interpolation unit, 1 2x downsampling unit, 1 local attention stage and 1 global attention stage; the universal interpolation unit unifies the size of the first-stage features, the second-stage features and the third-stage features output by the encoder to keep them consistent with the size of the output features of the current decoder stage after passing through the 2x downsampling unit, and then fuses the local and global features through the local attention stage and the global attention stage.

[0027] Furthermore, the 2x downsampling unit includes a pooling layer and a convolution layer, the feature extraction unit includes 2 residual convolution blocks and 1 activation function, the channel attention unit includes 1 dimensionality reduction unit, an attention mechanism and a dimensionality reduction unit, the universal interpolation unit includes 1 image fusion unit and a convolution layer, the local attention stage includes 2 residual convolution blocks and 2 activation functions, and the global attention stage includes 1 hourglass-shaped attention fusion unit and 1 channel attention unit.

[0028] In some embodiments, the training method of the restoration model comprises the following steps:

[0029] Acquire a training set, wherein the training set includes a public high-definition face data set to simulate a real degradation process, and generate a low-quality face training image to be restored and a corresponding high-definition face image;

[0030] Inputting the low-quality face training image to be restored into the state space restoration model, performing model training and obtaining the restored face training image in an iterative process;

[0031] Calculating a restoration model loss function according to the restored face training image and the corresponding high-definition face restoration image;

[0032] The restoration model is iteratively updated and trained based on the gradient descent method until an iteration stop condition is reached to obtain a trained restoration model.

[0033] Furthermore, obtaining a training set includes the following steps:

[0034] Acquire a high-quality face image, and use the high-quality face image as a high-definition real face image;

[0035] Performing pixel adjustment operation on the high-quality face image to obtain a degraded face image; using the degraded face image as a face training image to be restored;

[0036] In a second aspect, the present application provides a face image restoration system based on a state space model, including a processor and a storage medium;

[0037] The storage medium is used to store instructions;

[0038] The processor is configured to operate according to the instructions to execute the method according to the first aspect.

[0039] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect.

[0040] In a fourth aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the first aspect when executing the computer program.

[0041] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] The present invention proposes a facial image restoration method based on a state-space model. First, based on the image fusion module, input images of different scales are fused at different stages of the encoder, so that image pixels and semantic features learn from each other; secondly, a multi-scale state-space model is introduced based on the traditional state-space model, and processing branches with different receptive field sizes are used to mine local and global information and assist the model in deep feature extraction; finally, the multi-scale attention fusion module uses a jump connection encoding and decoding structure compared to the general connection method, which fully considers the characteristics of the features of different stages of the encoder and effectively improves the learning ability of the decoder. The present invention can ensure the consistency of facial identity information while ensuring the detailed texture and geometric contour of the restoration result, further improving the generalization performance of the face restoration model in real scenes, thereby meeting various tasks and application requirements related to facial images. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of the process of a facial image restoration method based on a state space model provided by an embodiment of the present invention;

[0045] Figure 2 is a schematic diagram of the overall structure of the restoration model provided in the embodiment;

[0046] Figure 3 is a schematic diagram of the structure of an image fusion module provided in an embodiment;

[0047] Figure 4 is a schematic diagram of the residual convolution block structure provided in an embodiment;

[0048] Figure 5 is a schematic diagram of the channel attention unit structure provided by the embodiment;

[0049] Figure 6 is a schematic diagram of the structure of a multi-scale attention fusion module provided in an embodiment;

[0050] Figure 7 is a schematic diagram of the structure of a multi-scale state space module provided in an embodiment;

[0051] Figure 8is a schematic diagram of the structure of a multi-scale state space model provided in an embodiment;

[0052] Fig. 9 It is a schematic diagram of the state space model structure provided by the embodiment. DETAILED DESCRIPTION

[0053] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0054] Example 1

[0055] This embodiment provides a face image restoration method based on a state space model, including:

[0056] Obtaining a face image to be restored;

[0057] Inputting the face image to be restored into a restoration model to obtain a restored face image;

[0058] Wherein, the restoration model includes an encoder and a decoder;

[0059] The encoder comprises a first image fusion module, a first multi-scale state space module, a second image fusion module, a second multi-scale state space module, a third image fusion module, and a third multi-scale state space module in sequence; the face image to be restored is fused with a 2-fold downsampled low-quality face image by the first image fusion module, and then extracted by the first multi-scale state space module to obtain first-stage features; the first-stage features are fused with a 4-fold downsampled low-quality face image by the second image fusion module, and then extracted by the second multi-scale state space module to obtain second-stage features; the second-stage features are fused with an 8-fold downsampled low-quality face image by the third image fusion module, and then extracted by the third multi-scale state space module to obtain third-stage features;

[0060] The decoder includes a fourth multiscale state space module, a first multiscale attention fusion module, a fifth multiscale state space module, a second multiscale attention fusion module, a sixth multiscale state space module, and a third multiscale attention fusion module in sequence; the third stage feature is subjected to the fourth multiscale state space module to obtain a 2-fold up-sampled restored face image, the 2-fold up-sampled restored face image is fused with the first stage feature, the second stage feature and the third stage feature through the first multiscale attention fusion module, and then is subjected to the fifth multiscale state space module to obtain a 4-fold up-sampled restored face image, the 4-fold up-sampled restored face image is fused with the first stage feature, the second stage feature and the third stage feature through the second multiscale attention fusion module, and then is subjected to the sixth multiscale state space module to obtain an 8-fold up-sampled restored face image, the 8-fold up-sampled restored face image is fused with the first stage feature, the second stage feature and the third stage feature through the third multiscale attention fusion module, and then is decoded and output, and is aligned and fitted with the input low-quality face image to obtain a restored high-quality face image.

[0061] In some embodiments, Figure 1 and Figure 2 As shown, a face image restoration method based on a state space model comprises the following steps:

[0062] Acquire a training set; the training set includes a public high-definition face data set to simulate a real degradation process, generate a low-quality face training image to be restored and a corresponding high-definition face image;

[0063] The restoration model is iteratively trained using the training set to obtain a trained restoration model.

[0064] Input a low-quality face image to be restored in a real scene or generated by a simulated degradation process, and obtain a restored face image inferred based on the model.

[0065] The encoder is used to mine multi-scale deep semantic features based on the face image to be restored and its downsampled images of different scales; the decoder is used to generate the restored face image based on the multi-scale deep semantic features generated by the encoder;

[0066] In the present application, the image fusion module is used to, according to the feature extraction unit, the downsampling unit and the channel attention unit, input low-quality face images of different scales, the shallow features mined by the feature extraction unit and the deep features output by the encoder at different stages, and use the channel attention mechanism to perform key feature fusion, so as to promote the mutual learning of image pixels and semantic features;

[0067] The multi-scale state space module is used to perform feature multiplication and addition according to the output of the state space branch, and then perform dimension-upgrading output after passing through the layer normalization unit. It uses the long-sequence deep feature extraction characteristics of the state space model to enhance the module's feature mining and learning capabilities.

[0068] The multi-scale attention fusion module is used to mine local and global information based on processing branches with different receptive field sizes and adopt skip connections to improve the learning ability of the decoder.

[0069] The technical concept of the present invention is as follows: based on the image fusion module, multi-scale input images are fused at different stages of the encoder so that image pixels and semantic features learn from each other; secondly, based on the traditional state-space model, a multi-scale state-space model is introduced, and processing branches with different receptive field sizes are used to mine local and global information and assist the model in deep feature extraction; the multi-scale attention fusion module adopts a jump connection encoding and decoding structure compared to the general connection method, fully considering the characteristics of features at different stages of the encoder, improving the learning ability of the decoder, and improving the problem that the existing method is difficult to grasp long-distance global feature information and has low reasoning efficiency, thereby improving the generalization performance of the face restoration model and achieving higher evaluation index scores and high-quality visualization effects in real scenarios.

[0070] like Figure 3 As shown, the network structures of the first image fusion module, the second image fusion module and the third image fusion module are the same, all of which include a 2x downsampling unit, a feature extraction unit and a channel attention unit. Low-quality face images of different scales are input, and the shallow features mined by the feature extraction unit and the deep features output by different stages of the encoder are fused with key features using the channel attention unit, thereby promoting mutual learning between image pixels and semantic features.

[0071] The feature extraction unit includes a residual convolution block and an activation function; the residual convolution block includes 2 convolution layers and 2 activation functions, a modified linear function is used as the activation function, and the output features after the input features are extracted through 2 convolution layers with deep features are added to the input features to obtain the final output.

[0072] Furthermore, if Figure 4 As shown, the residual convolution block includes 2 convolution layers and 2 activation functions, and uses the rectified linear function (ReLU) as the activation function. The output features of the input features after the deep features are extracted by the two convolution layers are added to the input features to obtain the final output;

[0073] like Figure 5 As shown, the channel attention unit includes, in sequence, a dimensionality reduction unit, a dimensionality increase unit, and an attention mechanism operation for assigning weights to different channels.

[0074] The image fusion module is used to make image pixels and semantic features learn from each other. Specifically, the semantic features of low-quality face image input are initially extracted based on the convolutional neural network, and the image features from different scales are effectively combined by using multi-scale fusion.

[0075] In some embodiments, Figure 6 As shown, the network structures of the first multi-scale attention fusion module, the second multi-scale attention fusion module and the third multi-scale attention fusion module are the same, all of which include 1 universal interpolation unit, 1 2x downsampling unit, 1 local attention stage and 1 global attention stage. The universal interpolation unit unifies the size of the first-stage features, the second-stage features and the third-stage features output by the encoder to keep them consistent with the size of the output features of the current decoder stage after passing through the 2x downsampling unit. Then, the local attention stage and the global attention stage are used to fully consider the feature characteristics of different stages of the encoder, and efficiently fuse local and global features.

[0076] Furthermore, in some embodiments, the 2x downsampling unit includes a pooling layer and a convolution layer, the feature extraction unit includes 2 residual convolution blocks and 1 activation function, the channel attention unit includes 1 dimensionality reduction unit, an attention mechanism and a dimensionality reduction unit, the universal interpolation unit includes 1 image fusion unit and a convolution layer, the local attention stage includes 2 residual convolution blocks and 2 activation functions, and the global attention stage includes 1 hourglass attention fusion unit and 1 channel attention unit.

[0077] The multi-scale attention fusion module is used to combine low-level details of different scale features of face images with high-level semantics. Specifically, the encoder is divided into three stages of different scales to extract low-level semantic features of face images, and the output features of the decoder in the i-th stage are jump-connected through the concat splicing function. The local semantic features are enhanced based on residual convolution and rectified linear function (ReLU). The hourglass-shaped attention fusion mechanism based on the staged dimensionality reduction encoding input fusion and dimensionality increase decoding output fusion enhances the global semantic features, retains the important features and identity information of the global face image, improves the learning ability of the decoder, and makes the restored face image more realistic.

[0078] like Figure 7As shown, the network structures of the first multi-scale state space module, the second multi-scale state space module, the third multi-scale state space module, the fourth multi-scale state space module, the fifth multi-scale state space module and the sixth multi-scale state space module are the same, and all include multiple multi-scale state space models and one feature extraction unit. Specifically, multi-scale state space processing branches with different receptive field sizes are used to mine local and global information of features to assist the model in deep feature extraction.

[0079] Furthermore, in some embodiments, Figure 8 As shown, each multiscale state space module includes multiple multiscale state space models and a feature extraction unit, each multiscale state space model includes 2 state space branches, 1 dimension reduction unit, 2 linear layers, 1 layer normalization unit and 1 dimension increase unit, each state space branch includes 1 depthwise separable convolution, 1 layer normalization unit and 1 state space model, the output of the first state space branch is added to the output of the dimension reduction unit and then passes through the first linear layer and the second linear layer respectively, the output of the second state space branch is added to the output of the dimension reduction unit, feature multiplied with the output of the second linear layer, and then feature added with the output of the first linear layer, and finally passed through the layer normalization unit for dimension increase output, and the module feature mining and learning capabilities are enhanced by using the long sequence deep feature extraction capability of the state space model.

[0080] like Fig. 9 As shown, the state space model includes 3 linear layers, 1 convolutional layer, 2 activation layers, and 1 selection state space model, and is connected in a residual manner; the output first branch of the layer normalization unit passes through the third linear layer, the convolutional layer, the first activation layer, and the selection state space model in sequence to obtain a first feature, and the output second branch of the layer normalization unit passes through the fourth linear layer and the second activation layer in sequence to obtain a second feature. The first feature and the second feature are connected in a residual manner and then output through the fifth linear layer.

[0081] The multi-scale state space model is based on the traditional state space theory. The expression of the state space model is:

[0082]

[0083] In the formula, u(t) represents the input signal, x(t) represents the historical state, and x ′ (t) represents the current state, y(t) represents the output signal; A represents the state transfer matrix, B is the matrix from input to state, C is the matrix from state to output, and D is the parameter from input to output.

[0084] Since the input data of image processing is discrete, by selecting a suitable discrete time step Δt, the ordinary differential equation of the state space model is converted into a discrete time difference equation that is convenient for processing face image pixels according to the bilinear transformation method. The expression of the difference equation is as follows:

[0085]

[0086] In the formula, x k represents the current k-th state, u k represents the kth discrete value of the input sequence, x k-1 Represents the k-1th historical state, by calculating the current input For the previous state The impact of x k , and predict the output Cx k ;in, represents the discretized state transfer matrix, A matrix representing the discretized input to the state;

[0087] By adjusting the step size Δt, the state-space model's attention to the current input is controlled, thereby achieving selective forgetting or retention of the state; when Δt is increased, the model tends to focus on the current input and forget the previous state; when Δt is reduced, the model tends to retain more historical states, thereby achieving a selective state-space model.

[0088] The feature sequence processing method based on the state space model can capture long-distance global information according to the selective mechanism. Compared with other feature sequence analysis models, it has greater computational throughput and model inference speed, high execution efficiency, and can retain the consistency of facial semantic information as much as possible while ensuring the details and texture of the restored results. It can also achieve better restoration effects in real scenes.

[0089] The training method for the restoration model in this embodiment includes:

[0090] 1. Obtain a training set, which includes low-quality face training images I to be restored lq And the corresponding high-definition face real image I hq ;

[0091] Specifically, obtaining a training set includes the following steps:

[0092] From a public face dataset, such as the Flickr-Faces-Hight-Quality (Flickr-Faces-HQ, FFHQ) high-definition face dataset, high-quality face images are obtained and used as the real high-definition face images I hq ;

[0093] Perform pixel adjustment on the high-quality face image to obtain a degraded face image; use the degraded face image as the face training image to be restored I lq .

[0094] Specifically, each high-quality face image is extracted from the FFHQ dataset and its length and width are adjusted to 512 pixels to obtain the degraded face image. The expression of the pixel adjustment operation is as follows:

[0095] I lq ={JPEG q ((I hq *k σ )↓ s +n δ )}↑ s

[0096] In the formula, I lq Represents the degraded face image, that is, the face training image to be restored; JPEG q Indicates JPEG compression with compression quality q; I hq represents a high-quality face image, i.e., a real high-definition face image; * represents a convolution operation; k σ represents the blur kernel with sigma = σ;↓ s Indicates downsampling s times operation; n δ Indicates Gaussian noise with sigma = δ; ↑ s Indicates an upsampling operation by a factor of s.

[0097] The specific parameters of the pixel adjustment operation can be adjusted according to the actual image and are not limited here.

[0098] 2. The multi-scale downsampled face training image I to be restored lq Input into the pre-built restoration model to obtain the face training image after multi-scale upsampling and restoration.

[0099] 3. Training images based on restored faces and the corresponding high-definition face image I hq , calculate the loss function value L of the restoration model.

[0100] The loss function expression for the restoration model is as follows:

[0101] L=L l1 +λ per L per +λ adv L adv

[0102] In the formula, L represents the total loss function value of the restoration model; l1 represents the L1 loss function value; λper represents the perceptual loss weight, in this embodiment, λ per The value is 0.1; L per Represents the perceptual loss function value based on the VGG network; λ adv Represents the adversarial loss weight, in this embodiment, λ adv The value is 0.01; L adv Represents the adversarial loss function value based on adversarial training.

[0103] L1 loss function value L l1 The expression is as follows:

[0104]

[0105] In the formula, I hq Represents a collection of high-definition real face images; represents the restored face training image set; ‖·‖1 represents the mean absolute error.

[0106] Perceptual loss function value L based on VGG network per The expression is as follows:

[0107]

[0108] In the formula, Represents the feature maps of the 1st to 5th convolutional layers in the pre-trained VGG model; Represents the square of the 2-norm.

[0109] The adversarial loss function value L based on adversarial training adv The expression is as follows:

[0110]

[0111] Where D(·) represents the output of the discriminator in adversarial training; Indicates about The expected value of the distribution; softplus represents the softplus function, and the expression of the softplus function is:

[0112] softplus(x)=ln(1+e x ).

[0113] 4. Perform iterative update training on the restoration model based on the gradient descent method, and select the restoration model with the smallest total loss function value as the pre-trained restoration model.

[0114] Example 2

[0115] Based on embodiment 1, this embodiment provides a face image restoration system based on a state space model, including a processor and a storage medium;

[0116] The storage medium is used to store instructions;

[0117] The processor is used to operate according to the instructions to execute the steps of the method according to embodiment 1.

[0118] Example 3

[0119] Based on Example 1, this embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in Example 1 are implemented.

[0120] Example 4

[0121] Based on Example 1, this embodiment provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in Example 1 when executing the computer program.

[0122] Example 5

[0123] Based on Example 1, this embodiment provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the method described in Example 1 are implemented.

[0124] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0125] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0128] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A face image restoration method based on a state space model, characterized in that: include: Obtaining a face image to be restored; Inputting the face image to be restored into a restoration model to obtain a restored face image; Wherein, the restoration model includes an encoder and a decoder; The encoder comprises a first image fusion module, a first multi-scale state space module, a second image fusion module, a second multi-scale state space module, a third image fusion module, and a third multi-scale state space module in sequence; the face image to be restored is fused with a 2-fold downsampled low-quality face image by the first image fusion module, and then extracted by the first multi-scale state space module to obtain first-stage features; the first-stage features are fused with a 4-fold downsampled low-quality face image by the second image fusion module, and then extracted by the second multi-scale state space module to obtain second-stage features; the second-stage features are fused with an 8-fold downsampled low-quality face image by the third image fusion module, and then extracted by the third multi-scale state space module to obtain third-stage features; The decoder includes a fourth multiscale state space module, a first multiscale attention fusion module, a fifth multiscale state space module, a second multiscale attention fusion module, a sixth multiscale state space module, and a third multiscale attention fusion module in sequence; the third stage feature is subjected to the fourth multiscale state space module to obtain a 2-fold up-sampled restored face image, the 2-fold up-sampled restored face image is subjected to the first multiscale attention fusion module to fuse with the first stage feature, the second stage feature and the third stage feature, and then subjected to the fifth multiscale state space module to obtain a 4-fold up-sampled restored face image, the 4-fold up-sampled restored face image is subjected to the second multiscale attention fusion module to fuse with the first stage feature, the second stage feature and the third stage feature, and then subjected to the sixth multiscale state space module to obtain an 8-fold up-sampled restored face image, the 8-fold up-sampled restored face image is subjected to the third multiscale attention fusion module to fuse with the first stage feature, the second stage feature and the third stage feature, and then decoded and output, and aligned and fitted with the input low-quality face image to obtain a restored high-quality face image; The network structures of the first multiscale state space module, the second multiscale state space module, the third multiscale state space module, the fourth multiscale state space module, the fifth multiscale state space module and the sixth multiscale state space module are the same, and all include multiple multiscale state space models and a feature extraction unit. Each of the multiscale state space models includes two state space branches, one dimension reduction unit, two linear layers, one layer normalization unit and one dimension increase unit. The state space branch includes one depthwise separable convolution, one layer normalization unit and one state space model. The output of the first state space branch is added to the output of the dimension reduction unit and then passes through the first linear layer and the second linear layer respectively. The output of the second state space branch is added to the output of the dimension reduction unit, and then feature multiplied with the output of the second linear layer, and then feature added with the output of the first linear layer, and finally passes through the layer normalization unit and then dimensionally increased output.

2. The method according to claim 1, characterized in that The network structures of the first image fusion module, the second image fusion module and the third image fusion module are the same, and all include a 2x downsampling unit, a feature extraction unit and a channel attention unit. The shallow features mined by the feature extraction unit for inputting low-quality face images of different scales and the deep features output by different stages of the encoder are fused with key features using the channel attention unit.

3. The method according to claim 2, characterized in that The feature extraction unit includes a residual convolution block and an activation function; wherein the residual convolution block includes 2 convolution layers and 2 activation functions, a rectified linear function is used as the activation function, and the output features after the input features are extracted through the 2 convolution layers with deep features are added to the input features to obtain the final output.

4. The method according to claim 2, characterized in that: The channel attention unit includes a dimensionality reduction unit, an attention mechanism operation for assigning weights to different channels, and a dimensionality increase unit.

5. The method according to claim 1, characterized in that The state space model includes 3 linear layers, 1 convolutional layer, 2 activation layers, and 1 selection state space model, and is connected in a residual manner; the first output branch of the layer normalization unit passes through the third linear layer, the convolutional layer, the first activation layer, and the selection state space model in sequence to obtain a first feature, and the second output branch of the layer normalization unit passes through the fourth linear layer and the second activation layer in sequence to obtain a second feature, and the first feature and the second feature are connected in a residual manner and then output through the fifth linear layer; The expression of the state space model is: In the formula, u(t) represents the input signal, x(t) represents the historical state, and x ′ (t) represents the current state, y(t) represents the output signal; A represents the state transfer matrix, B is the matrix from input to state, C is the matrix from state to output, and D is the parameter from input to output.

6. The method according to claim 5, characterized in that By selecting a suitable discrete time step Δt, the ordinary differential equation of the state space model is converted into a discrete time difference equation according to the bilinear transformation method. The difference equation is: In the formula, x k represents the current k-th state, u k represents the kth discrete value of the input sequence, x k-1 Represents the k-1th historical state, by calculating the current input For the previous state The impact of x k , and predict the output Cx k ;in, represents the discretized state transfer matrix, A matrix representing the discretized input to the state; By adjusting the step size Δt, the attention of the state space model to the current input is controlled.

7. The method according to claim 2, characterized in that The network structures of the first multi-scale attention fusion module, the second multi-scale attention fusion module and the third multi-scale attention fusion module are the same, and all include 1 universal interpolation unit, 1 2x downsampling unit, 1 local attention stage and 1 global attention stage; the universal interpolation unit unifies the size of the first-stage features, the second-stage features and the third-stage features output by the encoder to keep them consistent with the size of the output features of the current decoder stage after passing through the 2x downsampling unit, and then fuses the local and global features through the local attention stage and the global attention stage.

8. The method according to claim 7, characterized in that The 2x downsampling unit includes a pooling layer and a convolution layer, the feature extraction unit includes 2 residual convolution blocks and 1 activation function, the channel attention unit includes 1 dimensionality reduction unit, an attention mechanism and 1 dimensionality reduction unit, the universal interpolation unit includes 1 image fusion unit and a convolution layer, the local attention stage includes 2 residual convolution blocks and 2 activation functions, and the global attention stage includes 1 hourglass-shaped attention fusion unit and 1 channel attention unit.

9. The method according to claim 1, characterized in that: The training method of the restoration model comprises the following steps: Acquire a training set, wherein the training set includes a public high-definition face data set to simulate a real degradation process, generate a low-quality face training image to be restored and a corresponding high-definition face image; Inputting the low-quality face training image to be restored into the state space restoration model, performing model training and obtaining the restored face training image in an iterative process; Calculating a restoration model loss function according to the restored face training image and the corresponding high-definition face restoration image; The restoration model is iteratively updated and trained based on the gradient descent method until an iteration stop condition is reached to obtain a trained restoration model.

10. A face image restoration system based on a state space model, characterized in that: including processor and storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 9.

Images