Facial-image restoration method based on state space model

Abstract

Disclosed in the present invention is a facial-image restoration method based on a state space model. The method comprises: inputting into a restoration model a facial image to be restored, so as to obtain a restored facial image, wherein the restoration model comprises an encoder and a decoder, the encoder sequentially comprising a first image fusion module, a first multi-scale state space module, a second image fusion module, a second multi-scale state space module, a third image fusion module and a third multi-scale state space module, and the decoder sequentially comprising a fourth multi-scale state space module, a first multi-scale attention fusion module, a fifth multi-scale state space module, a second multi-scale attention fusion module, a sixth multi-scale state space module and a third multi-scale attention fusion module. The method restores details and textures while ensuring the consistency of facial semantic information, and can achieve a better restoration effect particularly for a low-quality facial image in a real degradation scenario.

Description

A face image restoration method based on a state-space model Technical Field

[0001] This invention relates to a face image restoration method based on state space, belonging to the field of image processing technology. Background Technology

[0002] In scenarios such as audio-visual entertainment and security monitoring, high-quality, clear facial images not only provide users with a good visual experience but also assist law enforcement officers in searching for suspects and missing persons. However, due to numerous unstable factors in shooting conditions, such as focusing failure of imaging equipment, camera shake; low light, high exposure, and motion interference from the subject in the imaging environment; and lossy compression and encoding / decoding formats in channel transmission, images can suffer from multiple interferences, leading to varying degrees of degradation. Low-quality facial images that are difficult to distinguish provide very limited effective identity information. Therefore, facial image restoration aims to recover clear, high-quality facial images from degraded, low-quality facial images, which helps improve the robustness of downstream tasks such as facial super-resolution and recognition, old photo restoration, and virtual digital human image editing.

[0003] Currently, face restoration methods primarily employ two model architectures: Convolutional Neural Networks (CNNs) and Transformer networks. Existing CNN-based methods often fail to capture global information due to the local nature of convolution operations. Similarly, existing Transformer-based methods, after dividing the image into blocks, employ computationally intensive self-attention mechanisms to grasp global information, resulting in limited ability to capture pixel-level details.

[0004] The method has achieved certain positive results in face restoration tasks, but when faced with larger face images that are more complex and have more severe degradation, it cannot simultaneously mine and efficiently integrate local detail information and global geometric information, resulting in its restoration performance failing to meet current industry needs.

[0005] Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention provides a face image restoration method based on a state-space model.

[0007] Compared to convolutional neural networks, state-space model architectures possess global information mining capabilities; compared to Transformer model architectures, they not only reduce computational overhead but also have stronger temporal memory capabilities. This invention further introduces multi-scale techniques to alleviate the problem of insufficient fusion between local and global information, and can maintain the consistency of facial identity information while ensuring the restoration of detailed textures and geometric contours. When facing complex real-world degradation scenarios, the restoration method proposed in this invention not only achieves high performance metrics such as Peak Signal-to-Noise Ratio (PSNR) but also restores high-quality, clear facial images.

[0008] To achieve the aforementioned objective, the present invention employs the following technical solution:

[0009] In a first aspect, the present invention discloses a face image restoration method based on a state-space model, comprising the following steps:

[0010] Obtain the face image to be restored;

[0011] The face image to be restored is input into the restoration model to obtain the restored face image;

[0012] The restoration model includes an encoder and a decoder;

[0013] The encoder sequentially includes a first image fusion module, a first multi-scale state space module, a second image fusion module, a second multi-scale state space module, a third image fusion module, and a third multi-scale state space module. The face image to be restored and the 2x downsampled low-quality face image are fused by the first image fusion module, and then the first multi-scale state space module extracts the first-stage features. The first-stage features are fused with the 4x downsampled low-quality face image by the second image fusion module, and then the second multi-scale state space module extracts the second-stage features. The second-stage features are fused with the 8x downsampled low-quality face image by the third image fusion module, and then the third multi-scale state space module extracts the third-stage features.

[0014] The decoder sequentially includes a fourth multi-scale state space module, a first multi-scale attention fusion module, a fifth multi-scale state space module, a second multi-scale attention fusion module, a sixth multi-scale state space module, and a third multi-scale attention fusion module. The third-stage features are processed by the fourth multi-scale state space module to obtain a 2x upsampled restored face image. The 2x upsampled restored face image is then fused with the first, second, and third-stage features by the first multi-scale attention fusion module and processed by the fifth multi-scale state space module to obtain a 4x upsampled restored face image. The 4x upsampled restored face image is then fused with the first, second, and third-stage features by the second multi-scale attention fusion module and processed by the sixth multi-scale state space module to obtain an 8x upsampled restored face image. The 8x upsampled restored face image is then fused with the first, second, and third-stage features by the third multi-scale attention fusion module and decoded and output. This output is then aligned and fitted with the input low-quality face image to obtain the restored high-quality face image.

[0015] In some embodiments, the network structures of the first image fusion module, the second image fusion module, and the third image fusion module are the same, each including a 2x downsampling unit, a feature extraction unit, and a channel attention unit. The shallow features mined by the feature extraction unit from low-quality face images of different scales are fused with the deep features output from different stages of the encoder using the channel attention unit to promote mutual learning between image pixels and semantic features.

[0016] Furthermore, the feature extraction unit includes a residual convolutional block and an activation function; wherein the residual convolutional block includes two convolutional layers and two activation functions, using a modified linear function as the activation function. The input features are processed through two convolutional layers to extract deep features, and the output features are added to the input features to obtain the final output.

[0017] Furthermore, the channel attention unit includes a dimensionality reduction unit, an attention mechanism operation for assigning weights to different channels, and a dimensionality increase unit.

[0018] In some embodiments, the network structures of the first, second, third, fourth, fifth, and sixth multi-scale state space modules are identical, each including multiple multi-scale state space models and one feature extraction unit. Each multi-scale state space model includes two state space branches, one dimensionality reduction unit, two linear layers, one layer normalization unit, and one dimensionality increase unit. Each state space branch includes one depthwise separable convolution, one layer normalization unit, and one state space model. The output of the first state space branch is added to the output of the dimensionality reduction unit and then passed through the first and second linear layers. The output of the second state space branch is added to the output of the dimensionality reduction unit, multiplied by the output of the second linear layer, and then added by the output of the first linear layer. Finally, it is passed through the layer normalization unit and output in increased dimensionality. The feature mining and learning capabilities of the module are enhanced by utilizing the long-sequence deep feature extraction capability of the state space model.

[0019] Furthermore, in some embodiments, the state space model includes three linear layers, one convolutional layer, two activation layers, and one selected state space model, which are connected by residuals; the first branch of the output of the fourth linear layer passes through the selected state space model, the first activation layer, the convolutional layer, and the fifth linear layer in sequence to obtain the first feature, and the second branch of the output of the fourth linear layer passes through the second activation layer and the sixth linear layer in sequence to obtain the second feature. The first feature and the second feature are connected by residuals for output.

[0020] The expression for the selected state-space model is:

[0021] In the formula, u(t) represents the input signal, x(t) represents the historical state, x'(t) represents the current state, and y(t) represents the output signal; A represents the state transition matrix, B is the matrix from input to state, C is the matrix from state to output, and D is the parameter from input to output.

[0022] Furthermore, since the input data for image processing is often discrete, by selecting an appropriate discrete time step Δt, the ordinary differential equation of the state-space model is transformed into a discrete-time difference equation using the bilinear transform method. The difference equation is as follows:

[0023] In the formula, x k Represents the current k-th state, u k Let x represent the k-th discrete value of the input sequence. k-1 This represents the (k-1)th historical state, calculated by the current input. For the previous state The impact x kAnd predict the output Cx k ;in, This represents the discretized state transition matrix. A matrix representing the discretized input to the state;

[0024] By adjusting the step size Δt, the focus of the state-space model on the current input can be controlled, thereby achieving selective forgetting or retention of states. When Δt is increased, the model tends to focus on the current input and forget previous states; when Δt is decreased, the model tends to retain more historical states, thus achieving a selective state-space model.

[0025] In some embodiments, the network structures of the first multi-scale attention fusion module, the second multi-scale attention fusion module, and the third multi-scale attention fusion module are the same, each including a general interpolation unit, a 2x downsampling unit, a local attention stage, and a global attention stage. The general interpolation unit unifies the size of the first-stage features, the second-stage features, and the third-stage features output by the encoder, making them consistent with the size of the current decoder stage output features after passing through the 2x downsampling unit. Then, the local and global features are fused through the local attention stage and the global attention stage.

[0026] Furthermore, the 2x downsampling unit includes a pooling layer and a convolutional layer; the feature extraction unit includes two residual convolutional blocks and one activation function; the channel attention unit includes two residual convolutional blocks and two activation functions; the general interpolation unit includes one image fusion unit and a convolutional layer; the local attention stage includes one dimensionality reduction unit, an attention mechanism, and one dimensionality reduction unit; and the global attention stage includes one hourglass-shaped attention fusion unit and one channel attention unit.

[0027] In some embodiments, the training method of the restoration model includes the following steps:

[0028] Obtain a training set, which includes a publicly available high-definition face dataset to simulate a real degradation process, and generate low-quality face training images to be restored and corresponding high-definition face images.

[0029] The low-quality face training image to be restored is input into the state space restoration model to train the model and obtain the restored face training image in the iterative process.

[0030] The loss function of the restoration model is calculated based on the restored face training image and the corresponding high-definition face restoration image.

[0031] The restored model is iteratively updated and trained using the gradient descent method until the iteration stops, resulting in a well-trained restored model.

[0032] Furthermore, obtaining the training set includes the following steps:

[0033] Acquire high-quality face images and use these high-quality face images as high-definition real face images;

[0034] The high-quality face image is subjected to pixel adjustment to obtain a degraded face image; the degraded face image is used as a training image for the face to be restored.

[0035] Secondly, this application provides a face image restoration system based on a state-space model, including a processor and a storage medium;

[0036] The storage medium is used to store instructions;

[0037] The processor is configured to operate according to the instructions to execute the method according to the first aspect.

[0038] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0039] Fourthly, this application provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in the first aspect.

[0040] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0041] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0042] This invention proposes a face image restoration method based on a state-space model. First, an image fusion module fuses input images of different scales at different stages of the encoder, enabling image pixels and semantic features to learn from each other. Second, a multi-scale state-space model is introduced based on the traditional state-space model, using processing branches with different receptive field sizes to mine local and global information, assisting in deep feature extraction. Finally, a multi-scale attention fusion module, compared to general connection methods, adopts a skip-connection encoder-decoder structure, fully considering the characteristics of features at different stages of the encoder, effectively improving the learning ability of the decoder. This invention can ensure the consistency of facial identity information while maintaining the detail texture and geometric contour of the restored result, further improving the generalization performance of the face restoration model in real-world scenarios, thus meeting the needs of various face image-related tasks and applications. Attached Figure Description

[0043] Figure 1 is a schematic diagram of the face image restoration method based on the state space model provided in an embodiment of the present invention;

[0044] Figure 2 is a schematic diagram of the overall structure of the restoration model provided in the embodiment;

[0045] Figure 3 is a schematic diagram of the image fusion module structure provided in the embodiment;

[0046] Figure 4 is a schematic diagram of the channel attention unit structure provided in the embodiment;

[0047] Figure 5 is a schematic diagram of the residual convolution block structure provided in the embodiment;

[0048] Figure 6 is a schematic diagram of the multi-scale attention fusion module structure provided in the embodiment;

[0049] Figure 7 is a schematic diagram of the multi-scale state space module structure provided in the embodiment;

[0050] Figure 8 is a schematic diagram of the multi-scale state-space model structure provided in the embodiment;

[0051] Figure 9 is a schematic diagram of the state-space model structure provided in the embodiment. Detailed Implementation

[0052] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0053] Example 1

[0054] This embodiment provides a face image restoration method based on a state-space model, including:

[0055] Obtain the face image to be restored;

[0056] The face image to be restored is input into the restoration model to obtain the restored face image;

[0057] The restoration model includes an encoder and a decoder;

[0058] The encoder sequentially includes a first image fusion module, a first multi-scale state space module, a second image fusion module, a second multi-scale state space module, a third image fusion module, and a third multi-scale state space module. The face image to be restored and the 2x downsampled low-quality face image are fused by the first image fusion module, and then the first multi-scale state space module extracts the first-stage features. The first-stage features are fused with the 4x downsampled low-quality face image by the second image fusion module, and then the second multi-scale state space module extracts the second-stage features. The second-stage features are fused with the 8x downsampled low-quality face image by the third image fusion module, and then the third multi-scale state space module extracts the third-stage features.

[0059] The decoder sequentially includes a fourth multi-scale state space module, a first multi-scale attention fusion module, a fifth multi-scale state space module, a second multi-scale attention fusion module, a sixth multi-scale state space module, and a seventh multi-scale attention fusion module. The third-stage features are processed by the fourth multi-scale state space module to obtain a 2x upsampled restored face image. The 2x upsampled restored face image is then fused with the first, second, and third-stage features by the first multi-scale attention fusion module and processed by the fifth multi-scale state space module to obtain a 4x upsampled restored face image. The 4x upsampled restored face image is then fused with the first, second, and third-stage features by the second multi-scale attention fusion module and processed by the sixth multi-scale state space module to obtain an 8x upsampled restored face image. The 8x upsampled restored face image is then fused with the first, second, and third-stage features by the third multi-scale attention fusion module and decoded and output. This output is then aligned and fitted with the input low-quality face image to obtain the restored high-quality face image.

[0060] In some embodiments, as shown in Figures 1 and 2, a face image restoration method based on a state-space model includes the following steps:

[0061] Obtain a training set; the training set includes a publicly available high-definition face dataset to simulate the real degradation process, generating low-quality face training images to be restored and corresponding high-definition face images;

[0062] The restoration model is iteratively trained using the training set to obtain a well-trained restoration model.

[0063] Input a low-quality face image to be restored from a real scene or generated by simulating a degradation process, and obtain a face restoration image inferred based on the model.

[0064] The encoder is used to mine multi-scale deep semantic features based on the face image to be restored and its downsampled images at different scales; the decoder is used to generate the restored face image based on the multi-scale deep semantic features generated by the encoder.

[0065] In this application, the image fusion module is used to fuse shallow features mined by the feature extraction unit and deep features output from different stages of the encoder into low-quality face images of different scales using the channel attention mechanism, based on the feature extraction unit, the downsampling unit and the channel attention unit, thereby promoting mutual learning between image pixels and semantic features.

[0066] The multi-scale state space module is used to perform feature multiplication and addition based on the output of the state space branch, and then output the increased dimension after passing through the layer normalization unit. It utilizes the long sequence deep feature extraction characteristics of the state space model to enhance the module's feature mining and learning capabilities.

[0067] The multi-scale attention fusion module is used to mine local and global information based on processing branches with different receptive field sizes, and to improve the learning ability of the decoder by using skip connections.

[0068] The technical concept of this invention is as follows: Based on the image fusion module, multi-scale input images are fused at different stages of the encoder, enabling image pixels and semantic features to learn from each other; secondly, a multi-scale state space model is introduced based on the traditional state space model, using processing branches with different receptive field sizes to mine local and global information and assist in deep feature extraction of the model; the multi-scale attention fusion module, compared with the general connection method, adopts a skip connection encoding and decoding structure, which fully considers the characteristics of features at different stages of the encoder, improves the learning ability of the decoder, and improves the problem that existing methods have difficulty in grasping long-distance global feature information and have low inference efficiency, thereby improving the generalization performance of the face restoration model and achieving high evaluation index scores and high-quality visualization effects in real-world scenarios.

[0069] As shown in Figure 3, the network structures of the first image fusion module, the second image fusion module, and the third image fusion module are the same, each including a 2x downsampling unit, a feature extraction unit, and a channel attention unit. The shallow features mined by the feature extraction unit and the deep features output from different stages of the encoder are fused using the channel attention unit to promote mutual learning between image pixels and semantic features.

[0070] The feature extraction unit includes a residual convolutional block and an activation function; the residual convolutional block includes two convolutional layers and two activation functions, using a modified linear function as the activation function. The input features are processed through two convolutional layers to extract deep features, and the output features are added to the input features to obtain the final output.

[0071] Further, as shown in Figure 4, the residual convolutional block includes two convolutional layers and two activation functions, using the modified linear function (ReLU) as the activation function. The input features are processed by the two convolutional layers to extract deep features, and the output features are added to the input features to obtain the final output.

[0072] As shown in Figure 5, the channel attention unit includes a dimensionality reduction unit, a dimensionality increase unit, and an attention mechanism operation for assigning weights to different channels.

[0073] The image fusion module enables image pixels and semantic features to learn from each other. Specifically, it initially extracts semantic features from low-quality face image input using a convolutional neural network, and then effectively combines image features from different scales using a multi-scale fusion approach.

[0074] In some embodiments, as shown in FIG6, the network structures of the first multi-scale attention fusion module, the second multi-scale attention fusion module, and the third multi-scale attention fusion module are the same, each including a general interpolation unit, a 2x downsampling unit, a local attention stage, and a global attention stage. The general interpolation unit unifies the size of the first-stage features, the second-stage features, and the third-stage features output by the encoder, and keeps them consistent with the size of the current decoder stage output features after passing through the 2x downsampling unit. Then, the local attention stage and the global attention stage respectively fully consider the feature characteristics of different stages of the encoder to perform efficient fusion of local and global features.

[0075] Furthermore, in some embodiments, the 2x downsampling unit includes a pooling layer and a convolutional layer, the feature extraction unit includes two residual convolutional blocks and one activation function, the channel attention unit includes two residual convolutional blocks and two activation functions, the general interpolation unit includes one image fusion unit and a convolutional layer, the local attention stage includes one dimensionality reduction unit, an attention mechanism, and one dimensionality reduction unit, and the global attention stage includes one hourglass-shaped attention fusion unit and one channel attention unit.

[0076] The multi-scale attention fusion module combines low-level details and high-level semantics of facial image features at different scales. Specifically, the encoder extracts low-level semantic features of the facial image in three stages at different scales. These features are then connected to the output features of the i-th stage decoder via a concat concatenation function. Local semantic features are enhanced based on residual convolution and the corrected linear function (ReLU). Global semantic features are enhanced by an hourglass-shaped attention fusion mechanism that combines staged dimensionality reduction encoding input fusion with dimensionality increase decoding output fusion. This preserves important global features and identity information of the facial image, improves the learning ability of the decoder, and makes the reconstructed facial image more realistic.

[0077] As shown in Figure 7, the network structures of the first, second, third, fourth, fifth, and sixth multi-scale state space modules are the same, each including multiple multi-scale state space models and one feature extraction unit. Specifically, multi-scale state space processing branches with different receptive field sizes are used to mine local and global information of features to assist in deep feature extraction of the model.

[0078] Furthermore, in some embodiments, as shown in Figure 8, each multi-scale state space module includes multiple multi-scale state space models and one feature extraction unit. Each multi-scale state space model includes two state space branches, one dimensionality reduction unit, two linear layers, one layer normalization unit, and one dimensionality increase unit. Each state space branch includes one depthwise separable convolution, one layer normalization unit, and one state space model. The output of the first state space branch is added to the output of the dimensionality reduction unit and then passed through the first linear layer and the second linear layer, respectively. The output of the second state space branch is added to the output of the dimensionality reduction unit, then multiplied by the output of the second linear layer, and then multiplied by the output of the first linear layer. Finally, it is passed through the layer normalization unit and output in increased dimensionality. The feature mining and learning capabilities of the module are enhanced by utilizing the long sequence deep feature extraction capability of the state space model.

[0079] As shown in Figure 9, the state space model includes 3 linear layers, 1 convolutional layer, 2 activation layers, and 1 selected state space model, which are connected by residuals. The first branch of the output of the fourth linear layer passes through the selected state space model, the first activation layer, the convolutional layer, and the fifth linear layer in sequence to obtain the first feature. The second branch of the output of the fourth linear layer passes through the second activation layer and the sixth linear layer in sequence to obtain the second feature. The first feature and the second feature are connected by residuals for output.

[0080] The multi-scale state-space model is based on traditional state-space theory, and its expression is as follows:

[0081] In the formula, u(t) represents the input signal, x(t) represents the historical state, x'(t) represents the current state, and y(t) represents the output signal; A represents the state transition matrix, B is the matrix from input to state, C is the matrix from state to output, and D is the parameter from input to output.

[0082] Since the input data for image processing is discrete, by selecting an appropriate discrete time step Δt, the ordinary differential equation of the state-space model is transformed into a discrete-time difference equation that is easier to process for face image pixels using the bilinear transform method. The expression of the difference equation is as follows:

[0083] In the formula, x k Represents the current k-th state, u k Let x represent the k-th discrete value of the input sequence. k-1 This represents the (k-1)th historical state, calculated by the current input. For the previous state The impact x k And predict the output Cx k ;in, This represents the discretized state transition matrix. A matrix representing the discretized input to the state;

[0084] By adjusting the step size Δt, the focus of the state-space model on the current input can be controlled, thereby achieving selective forgetting or retention of states. When Δt is increased, the model tends to focus on the current input and forget previous states; when Δt is decreased, the model tends to retain more historical states, thus achieving a selective state-space model.

[0085] The feature sequence processing method based on the state-space model can capture long-range global information through a selective mechanism. Compared with other feature sequence analysis models, it has greater computational throughput and model inference speed, resulting in high execution efficiency. While ensuring the details and texture of the restored result, it preserves the consistency of facial semantic information as much as possible and achieves good restoration results in real-world scenarios.

[0086] The training method for the restoration model in this embodiment includes:

[0087] 1. Obtain the training set, which includes the low-quality face training image Ilq to be restored and the corresponding high-resolution real face image I. hq ;

[0088] Specifically, obtaining the training set includes the following steps:

[0089] High-quality face images are obtained from publicly available face datasets, such as the Flickr-Faces-Hight-Quality (Flickr-Faces-HQ, FFHQ) high-definition face dataset, and these high-quality face images are used as high-definition real face images I. hq ;

[0090] Pixel adjustments are performed on a high-quality face image to obtain a degraded face image; this degraded face image is then used as the training image I for face restoration. lq .

[0091] Specifically, each high-quality face image is extracted from the FFHQ dataset and its dimensions are adjusted to 512 pixels to obtain the degraded face image. The expression for the pixel adjustment operation is as follows: I lq ={JPEG q ((I hq *k σ )↓ s +n δ )}↑ s

[0092] In the formula, I lq This represents a degraded face image, i.e., a training image of the face to be restored; JPEG q This indicates JPEG compression with a compression quality of q; I hq * represents a high-quality face image, i.e., a high-resolution, realistic face image; * represents a convolution operation; k σ Represents the fuzzy kernel of sigma = σ; ↓ s This indicates a downsampling operation of factor s; n δ This represents Gaussian noise with sigma = δ; ↑ s This indicates an upsampling operation of s times.

[0093] The specific parameters for pixel adjustment can be adjusted according to the actual image, and are not limited here.

[0094] II. Training images of the face to be restored, sampled at multiple scales (I) lq The input is fed into a pre-built restoration model to obtain multi-scale upsampled and restored face training images.

[0095] III. Training based on the restored face images and corresponding high-resolution facial images I hq Calculate the loss function value L of the restoration model.

[0096] The expression for the loss function of this restoration model is as follows: L = L l1 +λ per L per +λ advL adv

[0097] In the formula, L represents the total loss function value of the restoration model; where L l1 λ represents the L1 loss function value. per λ represents the perceptual loss weight, in this embodiment per The value is 0.1; L per λ represents the perceptual loss function value based on the VGG network. adv In this embodiment, λ represents the adversarial loss weight. adv The value is 0.01; L adv This represents the adversarial loss function value based on adversarial training.

[0098] L1 loss function value L l1 The expression is as follows:

[0099] In the formula, I hq This represents a collection of high-definition, realistic human face images. This represents the set of training images of the restored face; ||·||1 represents the mean absolute error.

[0100] The perceptual loss function value L based on VGG network per The expression is as follows:

[0101] In the formula, This represents the feature maps of the 1st to 5th convolutional layers in the pre-trained VGG model; This represents the square of the 2-norm.

[0102] Adversarial loss function value L based on adversarial training adv The expression is as follows:

[0103] In the formula, D(·) represents the output of the discriminator during adversarial training; Indicates about The expected value of the distribution; softplus represents the softplus function, and the expression of the softplus function is: softplus(x)=ln(1+e x ).

[0104] Fourth, the restoration model is iteratively updated and trained based on the gradient descent method, and the restoration model with the smallest total loss function value is selected as the pre-trained restoration model.

[0105] Example 2

[0106] Based on Embodiment 1, this embodiment provides a face image restoration system based on a state-space model, including a processor and a storage medium;

[0107] The storage medium is used to store instructions;

[0108] The processor is configured to operate according to the instructions to perform the steps of the method according to Embodiment 1.

[0109] Example 3

[0110] Based on Embodiment 1, this embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in Embodiment 1.

[0111] Example 4

[0112] Based on Embodiment 1, this embodiment provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in Embodiment 1.

[0113] Example 5

[0114] Based on Embodiment 1, this embodiment provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in Embodiment 1.

[0115] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0116] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0117] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0118] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0119] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A face image restoration method based on a state-space model, characterized in that, include: Obtain the face image to be restored; The face image to be restored is input into the restoration model to obtain the restored face image; The restoration model includes an encoder and a decoder; The encoder sequentially includes a first image fusion module, a first multi-scale state space module, a second image fusion module, a second multi-scale state space module, a third image fusion module, and a third multi-scale state space module. The face image to be restored and the 2x downsampled low-quality face image are fused by the first image fusion module, and then the first multi-scale state space module extracts the first-stage features. The first-stage features are fused with the 4x downsampled low-quality face image by the second image fusion module, and then the second multi-scale state space module extracts the second-stage features. The second-stage features are fused with the 8x downsampled low-quality face image by the third image fusion module, and then the third multi-scale state space module extracts the third-stage features. The decoder sequentially includes a fourth multi-scale state space module, a first multi-scale attention fusion module, a fifth multi-scale state space module, a second multi-scale attention fusion module, a sixth multi-scale state space module, and a third multi-scale attention fusion module. The third-stage features are processed by the fourth multi-scale state space module to obtain a 2x upsampled reconstructed face image. This 2x upsampled face image is then fused with the first, second, and third-stage features by the first multi-scale attention fusion module, and then processed by the fifth multi-scale state space module to obtain a 4x upsampled reconstructed face image. This 4x upsampled face image is then fused with the first, second, and third-stage features by the second multi-scale attention fusion module, and then processed by the sixth multi-scale state space module to obtain an 8x upsampled face image. The sampled and restored face image is then fused with the first-stage features, second-stage features, and third-stage features by the third multi-scale attention fusion module. The resulting image is then decoded and aligned with the input low-quality face image to obtain the restored high-quality face image.

2. The method according to claim 1, characterized in that, The first image fusion module, the second image fusion module, and the third image fusion module have the same network structure, each including a 2x downsampling unit, a feature extraction unit, and a channel attention unit. The shallow features mined by the feature extraction unit from low-quality face images of different scales are fused with the deep features output from different stages of the encoder using the channel attention unit to promote mutual learning between image pixels and semantic features.

3. The method according to claim 2, characterized in that, The feature extraction unit includes a residual convolutional block and an activation function; wherein the residual convolutional block includes two convolutional layers and two activation functions, using a modified linear function as the activation function. The input features are processed through two convolutional layers to extract deep features, and the output features are added to the input features to obtain the final output. And / or, the channel attention unit includes a dimensionality reduction unit, an attention mechanism operation for assigning weights to different channels, and a dimensionality increase unit.

4. The method according to claim 1, characterized in that, The network structures of the first, second, third, fourth, fifth, and sixth multi-scale state space modules are identical, each including multiple multi-scale state space models and one feature extraction unit. Each multi-scale state space model contains two state space branches, one dimensionality reduction unit, two linear layers, one layer normalization unit, and one dimensionality increase unit. Each state space branch contains one depthwise separable convolution, one layer normalization unit, and one state space model. The output of the first state space branch is added to the output of the dimensionality reduction unit and then passed through the first and second linear layers. The output of the second state space branch is added to the output of the dimensionality reduction unit, multiplied by the output of the second linear layer, and then multiplied by the output of the first linear layer. Finally, it is passed through the layer normalization unit and output in increased dimensionality. The feature mining and learning capabilities of the module are enhanced by utilizing the long-sequence deep feature extraction capability of the state space model.

5. The method according to claim 4, characterized in that, The state space model comprises three linear layers, one convolutional layer, two activation layers, and one selected state space model, which are connected by residuals. The first branch of the output of the fourth linear layer passes through the selected state space model, the first activation layer, the convolutional layer, and the fifth linear layer in sequence to obtain the first feature. The second branch of the output of the fourth linear layer passes through the second activation layer and the sixth linear layer in sequence to obtain the second feature. The first feature and the second feature are connected by residuals for output. The expression for the state-space model is: In the formula, u(t) represents the input signal, x(t) represents the historical state, x'(t) represents the current state, and y(t) represents the output signal; A represents the state transition matrix, B is the matrix from input to state, C is the matrix from state to output, and D is the parameter from input to output.

6. The method according to claim 5, characterized in that, By selecting an appropriate discrete time step Δt, the ordinary differential equations of the state-space model are transformed into discrete-time difference equations using the bilinear transformation method. The difference equations are as follows: In the formula, x k Represents the current k-th state, u k Let x represent the k-th discrete value of the input sequence. k-1 Indicates the k-th One historical state, calculated by the current input. For the previous state The impact x k And predict the output Cx k ;in, This represents the discretized state transition matrix. A matrix representing the discretized input to the state; By adjusting the step size Δt, the attention given to the current input by the state-space model can be controlled, thereby achieving selective forgetting or retention of the state, thus realizing a selective state-space model.

7. The method according to claim 1, characterized in that, The network structures of the first, second, and third multi-scale attention fusion modules are identical, each including a general interpolation unit, a 2x downsampling unit, a local attention stage, and a global attention stage. The general interpolation unit unifies the size of the first-stage, second-stage, and third-stage features output by the encoder, ensuring that the size is consistent with the size of the current decoder stage output features after passing through the 2x downsampling unit. Then, the local and global features are fused through the local and global attention stages.

8. The method according to claim 7, characterized in that, The 2x downsampling unit includes a pooling layer and a convolutional layer; the feature extraction unit includes two residual convolutional blocks and one activation function; the channel attention unit includes two residual convolutional blocks and two activation functions; the general interpolation unit includes one image fusion unit and a convolutional layer; the local attention stage includes one dimensionality reduction unit, an attention mechanism, and one dimensionality reduction unit; and the global attention stage includes one hourglass-shaped attention fusion unit and one channel attention unit.

9. The method according to claim 1, characterized in that, The training method for the restoration model includes the following steps: Obtain a training set, which includes a publicly available high-definition face dataset to simulate a real degradation process, and generate low-quality face training images to be restored and corresponding high-definition face images. The low-quality face training image to be restored is input into the state space restoration model to train the model and obtain the restored face training image in the iterative process. The loss function of the restoration model is calculated based on the restored face training image and the corresponding high-definition face restoration image. The restored model is iteratively updated and trained using the gradient descent method until the iteration stops, resulting in a well-trained restored model.

10. A face image restoration system based on a state-space model, characterized in that, Including processor and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1 to 9.

Images