A face image processing method that satisfies multiple degradation models

Abstract

The present invention discloses a facial image processing method that satisfies a variety of degradation models, which is used to improve image quality and reconstruct resolution. When constructing high- and low-resolution image training sets, a degradation model composed of a variety of blur kernel parameters and a variety of noise parameters is used to replace the existing bicubic downsampling model to degrade the high-resolution image. The obtained high- and low-resolution image pairs are used as training sets, and are input into a neural network with excellent performance at present for training. The high-resolution reconstruction result is output, and the obtained neural network can perform super-resolution reconstruction on low-resolution images obtained by a variety of degradation models. The innovative ideas proposed by the present invention can be extended to any existing deep learning neural network framework with excellent performance in the field of super-resolution reconstruction, and have strong scalability. In addition, the method proposed by the present invention has more engineering practical value.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a face image processing method that satisfies multiple degradation models, which is used for improving image quality and reconstructing resolution. Background Art

[0002] At present, the quality and clarity of facial images obtained by monitoring equipment will affect the judgment of relevant information. Therefore, it is of great significance to improve and reconstruct facial image resolution. The technology of improving image resolution through software methods is called super-resolution reconstruction technology. While saving costs, it restores the realistic subjective visual effect of the image and improves the clarity of the image. It has a wide range of application scenarios in video reconnaissance, database retrieval, face recognition, license plate recognition, prohibited items detection and other fields.

[0003] like Figure 1 As shown in the figure, SRCNN adopts a three-layer neural network structure. The first layer is the image block extraction and feature representation layer, the second layer is the feature nonlinear mapping layer, and the third layer is the final reconstruction layer. First, the high-resolution image is downsampled by bicubic downsampling to obtain a low-resolution image with the corresponding multiple scaling, and then the low-resolution image is interpolated by bicubic upsampling to obtain a low-resolution image of the same size as the original image. The low-resolution image obtained after interpolation is used as the input of the neural network for training, and nonlinear mapping is performed through a three-layer convolutional network. The output of the network is a high-resolution image result.

[0004] In recent years, super-resolution reconstruction technology based on deep learning algorithms, represented by SRCNN, has achieved excellent performance and solved many technical problems of traditional algorithms in this field. Low-resolution images are usually obtained by a series of degradation processes of high-resolution images. In actual scenes, it is usually impossible to determine the blur kernel parameters and noise types of the low-resolution images that need to be processed. However, the high-resolution and low-resolution training sets of images constructed by the current deep learning-based super-resolution reconstruction algorithm are all degraded by using a degradation model of bicubic downsampling of high-resolution images. When the low-resolution image actually obtained does not meet the bicubic downsampling degradation process, the reconstruction result is very poor. This is also the biggest technical problem currently faced by the field of super-resolution reconstruction. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention aims to propose a face image processing method that satisfies multiple degradation models, while meeting the needs of practical applications and taking into account the accuracy and performance of reconstruction.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A method for processing face images that meets multiple degradation models includes the following steps:

[0008] S1. Degrade the original image:

[0009] The original high-resolution image is degraded to obtain a corresponding low-resolution image. The degradation process uses a degradation model based on a combination of multiple blur kernel parameters and multiple noise parameters; the degradation model is expressed by the following formula:

[0010]

[0011] Among them, I HR ,I LR , s, k and n represent the high-resolution original image, the degraded low-resolution image, the sampling operator, the blur kernel parameter and the additive Gaussian white noise respectively;

[0012] S2. Construct a training dataset of high and low resolution image pairs:

[0013] The low-resolution image obtained by the degradation process in step S1 is matched with its original high-resolution image to form a high-resolution and low-resolution image pair as a training data set for the super-resolution reconstruction neural network model;

[0014] S3. Preprocess the input data of the super-resolution reconstruction neural network model through the dimension extension strategy:

[0015] The vectorized blur kernel parameters are subjected to PCA dimensionality reduction, and then merged with the noise parameters to obtain a vector, and then the vector is stretched into a tensor, which is called a degradation parameter map. The degradation parameter map has the same dimension as the low-resolution image.

[0016] S4. Input the processed data into the super-resolution reconstruction neural network model for training:

[0017] The degradation parameter map obtained after dimensional extension and the low-resolution image are combined and input into the super-resolution reconstruction neural network model for training. The original high-resolution image is used as a label. During the training process, the neural network model is trained by reducing the value of the loss function so that the output super-resolution result and the label are as close as possible.

[0018] S5. After training, the super-resolution reconstruction neural network model has the ability to reconstruct and restore low-resolution images of various types of degradation. By inputting a low-resolution test image obtained through a complex degradation model, the corresponding high-resolution reconstruction result can be obtained.

[0019] Furthermore, in step S1, the blur kernel parameter k of the degradation process is set to (Kernel width = 0.2, 1.3, 2.6), and the standard deviation parameter σ of the Gaussian white noise is set to (σ = 0, 15, 50),↓ s The downsampling method is bicubic downsampling, and corresponding low-resolution images of various degradation types are obtained.

[0020] Furthermore, the super-resolution reconstruction neural network model uses a recursive residual neural network as a basic model. In step S4, the data obtained in step S3 is input into the recursive residual neural network for training.

[0021] Furthermore, the recursive residual neural network includes 1 global cross-layer connection, 7 residual units, each residual unit includes a 2-layer convolution recursive unit, and the network has a total of 16 convolution layers.

[0022] Furthermore, in step S4, a random Gaussian distribution is used to initialize parameters, and the initial learning rate is set to 0.01, which is reduced by half every 10 rounds; the Adam algorithm is used to optimize the super-resolution reconstruction neural network model.

[0023] The beneficial effect of the present invention is that when constructing a high-resolution and low-resolution image training set, the present invention uses a degradation model composed of a variety of blur kernel parameters and a variety of noise parameters instead of the existing bicubic downsampling model to degrade the high-resolution image, and uses the high-resolution and low-resolution image pairs obtained through the above process as a training set, inputs a neural network that currently performs well in super-resolution reconstruction to train the model, and outputs a high-resolution reconstruction result. The obtained neural network can perform super-resolution reconstruction on low-resolution images obtained by a variety of degradation models. Compared with the current deep learning-based algorithm that can only reconstruct low-resolution images obtained by a degradation model such as bicubic downsampling, the innovative idea proposed by the present invention can be extended to any existing deep learning neural network framework with excellent performance in the field of super-resolution reconstruction, has strong scalability, and the method proposed by the present invention has more engineering practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of the existing SRCNN network structure;

[0025] Figure 2 Schematic diagram of the process of the method of Example 1 of the present invention;

[0026] Figure 3 This is a process diagram of the dimensional extension strategy in the method of Example 1 of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described below in conjunction with the accompanying drawings. It should be noted that this embodiment is based on the technical solution and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to this embodiment.

[0028] Example 1

[0029] In practical applications, images that need to be super-resolution reconstructed often have complex and difficult to infer degradation models. Blind super-resolution has always been the biggest technical challenge in the field of image super-resolution reconstruction. Currently, there is limited research on algorithms based on blind super-resolution. Most solutions decompose the blind super-resolution problem into two steps: one is to estimate the blur kernel based on a given low-resolution image, and the other is to restore the super-resolution reconstructed image based on the estimated blur kernel. These two-step solutions involve two independently trained models that are not compatible with each other. Even if the estimation error in the first step is small, the performance of the second step will be degraded. On the other hand, the first step can only use limited information in the low-resolution image, which makes the prediction of the blur kernel very difficult.

[0030] Therefore, the goal of this embodiment is to introduce a non-blind super-resolution reconstruction neural network structure that can meet the needs of various degradation models, while taking into account the accuracy and performance of its reconstruction. Figure 2 As shown, the face image processing method that satisfies multiple degradation models in this embodiment includes the following steps:

[0031] S1. Degrade the original image:

[0032] In order to construct the high-resolution and low-resolution image pair dataset required for super-resolution reconstruction neural network model training, the original high-resolution image needs to be degraded to obtain the corresponding low-resolution image. The degradation process no longer uses the degradation method of simple bicubic downsampling of the original high-resolution image, but uses a degradation model based on a combination of multiple blur kernel parameters and multiple noise parameters to degrade the original image. The degradation model is expressed as follows:

[0033]

[0034] Among them, I HR ,I LR , s, k and n represent the high-resolution original image, the degraded low-resolution image, the sampling operator, the blur kernel parameter and the additive Gaussian white noise, respectively.

[0035] In this embodiment, the blur kernel parameter k of the degradation process is set to (Kernel width = 0.2, 1.3, 2.6), and the standard deviation parameter σ of the Gaussian white noise is set to (σ = 0, 15, 50),↓ sThe downsampling method is bicubic downsampling, and corresponding low-resolution images of various degradation types are obtained.

[0036] S2. Construct a training dataset of high and low resolution image pairs:

[0037] The low-resolution image obtained by the degradation process in step S1 is matched with its original high-resolution image to form a high-resolution and low-resolution image pair, which serves as a training data set for the super-resolution reconstruction neural network model.

[0038] S3, preprocessing the input data of the super-resolution reconstruction neural network model through a dimensional extension strategy;

[0039] The method of this embodiment uses the blur kernel parameters and noise parameters used in the degradation model in step S1 and the low-resolution image processed in step S1 as the input of the neural network model. However, the dimensions of the blur kernel parameters, noise parameters and low-resolution image are different, so they cannot be directly used as the input of the neural network model. For this reason, the method of this embodiment proposes a dimensional extension strategy, such as Figure 3 As shown in FIG. 1 , before inputting data into the super-resolution reconstruction neural network model, the vectorized blur kernel parameters are subjected to PCA dimensionality reduction, and then merged with the noise parameters to obtain a vector, and then the vector is stretched into a tensor, which is called a degradation parameter map. The degradation parameter map has the same dimension as the low-resolution image. In this way, the dimensions of the blur kernel parameters, noise parameters and low-resolution image are unified.

[0040] S4. Input the processed data into the super-resolution reconstruction neural network model for training:

[0041] The degradation parameter map and the low-resolution image obtained after dimensional extension are combined and input into the super-resolution reconstruction neural network model for training. The original high-resolution image is used as the label. During the training process, the neural network model is trained by reducing the value of the loss function so that the output result (super-resolution image) and the label (original high-resolution image) are as close as possible.

[0042] In order to verify the ability of the method of this embodiment to reconstruct images of various types of degradation, this embodiment does not select a particularly complex and time-consuming neural network, but uses a recursive residual neural network with excellent performance as the basic model, and inputs the data obtained in step S3 into the recursive residual neural network for model training. It should be noted that the recursive residual neural network can also be replaced by other neural networks with excellent performance in the field of super-resolution reconstruction.

[0043] The recursive residual neural network includes 1 global cross-layer connection, 7 residual units, each residual unit includes a 2-layer convolution recursive unit, and the network has a total of 16 convolution layers.

[0044] The method of this embodiment uses random Gaussian distribution to initialize parameters, and the initial learning rate is set to 0.01, which is reduced by half every 10 rounds. The Adam algorithm is used to optimize the network.

[0045] S5. After training, the super-resolution reconstruction neural network model has the ability to reconstruct and restore low-resolution images of various types of degradation. By inputting a low-resolution test image obtained through a complex degradation model, the corresponding high-resolution reconstruction result can be obtained.

[0046] Example 2

[0047] This embodiment verifies the accuracy of the super-resolution reconstruction result by calculating the PSNR value and SSIM value of the test image. The unit of PSNR is dB, which is an objective indicator for evaluating image quality. It can be expressed as the mean square error between the original image and the reconstructed image relative to (2 n -1) 2 The higher the PSNR value, the better the quality of the reconstruction result. The calculation formula of PSNR is as follows:

[0048]

[0049]

[0050] Among them, MSE is the mean square error loss function; m and n represent the width and height of the image respectively; p and q represent the values ​​of the horizontal and vertical coordinate axes respectively; X and H represent the original image and the super-resolution reconstruction result respectively.

[0051] SSIM is generally used to evaluate the similarity of the context structure of an image. The closer SSIM is to 1, the better the reconstruction result is and the higher the structural similarity between the image contexts is. The following two formulas are the calculation process of SSIM:

[0052] SSIM(X,H)=l(X,H)c(X,H)s(X,H)

[0053]

[0054]

[0055]

[0056] Among them, C 1 , C 2 and C 3 is a constant, σ Xis the variance of the high-resolution image, σ H is the variance of the reconstructed image; σ XH represents the covariance of the original high-resolution image and the reconstructed image, μ X is the mean value of the high-resolution image, μ H is the mean of the reconstructed image, X and H represent the original image and the super-resolution reconstruction result, respectively.

[0057] This embodiment selects an image with rich high-frequency texture details as a test image, sets the blur kernel parameter of the degradation process to (0.2, 1.3, 2.6), sets the standard deviation parameter of the Gaussian white noise to (0, 15, 50), uses multiple degradation models to degrade the image quality of the high-resolution original image, inputs the degraded low-resolution image into the model trained by the method of Embodiment 1 for reconstruction, calculates the PSNR and SSIM values ​​of the output high-resolution reconstruction result, and obtains the reconstruction objective index results shown in Table 1. It can be seen that by comparing with the SRCNN algorithm and VDSR algorithm, which currently perform well in the field of super-resolution reconstruction, the method of Embodiment 1 can obtain the best PSNR and SSIM results and restore more vivid and rich texture details.

[0058] Table 1 PSNR and SSIM test results of test images

[0059]

[0060] For those skilled in the art, various corresponding changes and modifications can be made according to the above technical solutions and concepts, and all of these changes and modifications should be included in the protection scope of the claims of the present invention.

Claims

1. A facial image processing method that satisfies multiple degradation models, characterized in that: The steps include: S1. Degrade the original image: The original high-resolution image is degraded to obtain a corresponding low-resolution image. The degradation process uses a degradation model based on a combination of multiple blur kernel parameters and multiple noise parameters; the degradation model is expressed by the following formula: Among them, I HR ,I LR , s, k and n represent the high-resolution original image, the degraded low-resolution image, the sampling operator, the blur kernel parameter and the additive Gaussian white noise respectively; S2. Construct a training dataset of high and low resolution image pairs: The low-resolution image obtained by the degradation process in step S1 is matched with its original high-resolution image to form a high-resolution and low-resolution image pair as a training data set for the super-resolution reconstruction neural network model; S3. Preprocess the input data of the super-resolution reconstruction neural network model through the dimension extension strategy: The vectorized blur kernel parameters are subjected to PCA dimensionality reduction, and then merged with the noise parameters to obtain a vector, and then the vector is stretched into a tensor, which is called a degradation parameter map. The degradation parameter map has the same dimension as the low-resolution image. S4. Input the processed data into the super-resolution reconstruction neural network model for training: The degradation parameter map obtained after dimensional extension and the low-resolution image are combined and input into the super-resolution reconstruction neural network model for training. The original high-resolution image is used as a label. During the training process, the neural network model is trained by reducing the value of the loss function so that the output super-resolution result and the label are as close as possible. S5. After training, the super-resolution reconstruction neural network model has the ability to reconstruct and restore low-resolution images of various types of degradation. By inputting a low-resolution test image obtained through a complex degradation model, the corresponding high-resolution reconstruction result can be obtained.

2. The method according to claim 1, characterized in that In step S1, the blur kernel parameter k of the degradation process is set to (Kernel width = 0.2, 1.3, 2.6), and the standard deviation parameter σ of the Gaussian white noise is set to (σ = 0, 15, 50). s The downsampling method is bicubic downsampling, and corresponding low-resolution images of various degradation types are obtained.

3. The method according to claim 1, characterized in that: The super-resolution reconstruction neural network model uses a recursive residual neural network as a basic model. In step S4, the data obtained in step S3 is input into the recursive residual neural network for training.

4. The method according to claim 3, characterized in that The recursive residual neural network includes 1 global cross-layer connection, 7 residual units, each residual unit includes a 2-layer convolution recursive unit, and the network has a total of 16 convolution layers.

5. The method according to claim 3, characterized in that: In step S4, a random Gaussian distribution is used to initialize the parameters, the initial learning rate is set to 0.01, and it is reduced by half every 10 rounds; the Adam algorithm is used to optimize the super-resolution reconstruction neural network model.

Images