A face super-resolution method and system based on meta-learning optimization network

Abstract

The present invention provides a face super-resolution method and system based on a meta-learning optimization network. The meta-learning optimization network connects the intrinsic features of multiple degradation processes learned in the training phase and uses them in the unknown degradation process of the test, thereby solving the technical problem that the face image reconstruction performance of the current face super-resolution reconstruction algorithm has certain limitations when the image degradation type is unknown, and improving the reconstruction performance of the face image. The present invention uses meta-learning to quickly learn the characteristics of a new task, so that the network can achieve good super-resolution performance on the new test task through empirical learning of the training task. The present invention can address the problem of reduced super-resolution performance of face images when the degradation process is unknown, and generate higher quality high-resolution face images.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision face super-resolution, and in particular relates to a face super-resolution method and system based on a meta-learning optimization network. Background Art

[0002] Face super-resolution (FSR), also known as face hallucination, aims to extract high-frequency details from low-resolution (LR) face images to obtain high-resolution (HR) face images. In recent years, scholars have proposed a large number of face super-resolution algorithms based on deep learning. The subjective and objective reconstruction performance of images is significantly better than traditional shallow learning methods, so it has become a new research hotspot.

[0003] From the perspective of degradation type, deep learning-based FSR methods can be divided into single degradation type and multiple degradation type. Under the ideal conditions of bicubic downsampling, single degradation methods can achieve significant quantitative or qualitative performance, but if the degradation of the test image deviates from bicubic degradation, it usually produces sharp or blurred results. Therefore, scholars began to study multiple degradation FSR methods.

[0004] Multiple degradation FSR methods are divided into non-blind FSR methods and blind FSR methods. Non-blind FSR methods usually have information about the actual degradation (e.g. blur kernel, noise), which can be used as relevant inputs to the network to help simulate the actual degradation. In blind SR methods, only LR images are used as input, and ground truth images are not used as references. Typically, for FSR in the real world, the quality of LR images is affected by multiple factors (such as weather, motion blur caused by camera shake, etc.), which leads to unknown complex degradations of real LR images, which cannot be simulated by known blur kernels and noise. To solve this problem, scholars have proposed many unsupervised or self-supervised FSR methods.

[0005] Although the above non-blind FSR methods have achieved satisfactory results, most of them reconstruct the super-resolution network by estimating the degradation blur kernel, which increases the number of parameters of the overall network and greatly increases the memory overhead and computing power required during training. However, when the simulated degradation blur kernel deviates from the actual degradation, unnecessary artifacts will appear in the reconstructed SR image. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a face super-resolution method and system based on a meta-learning optimization network, so as to improve the reconstruction performance of face images.

[0007] The technical solution adopted by the present invention to solve the above technical problems is: a face super-resolution method based on a meta-learning optimization network, comprising the following steps:

[0008] S0: Build a face super-resolution system based on a meta-learning optimization network, including an image processing module, a meta-training module, a parameter optimization module and a super-resolution result module connected in sequence;

[0009] S1: The image processing module performs different types of degradation processes on the high-resolution face images to obtain the corresponding target low-resolution face images, thereby dividing the training set and the test set;

[0010] S2: The meta-training module divides the low-resolution face images in the training set into blocks, and uses the meta-training network to learn the feature mapping from low-resolution face images to high-resolution face images under various degradation processes, and obtains an initial super-resolution model with transferable and sensitive parameters.

[0011] S3: The parameter optimization module sends the low-resolution face images of the test set to the initial super-resolution model for training, iterates the model parameters for a set number of times to adapt to the new degradation process, and obtains a meta-parameter optimization network;

[0012] S4: The super-resolution result module sends the low-resolution face images of the test set into the meta-parameter optimization network for learning to obtain the final high-resolution face images.

[0013] According to the above scheme, in step S1, the specific steps are:

[0014] S11: high-resolution face image y i Perform blur convolution to obtain a blurred image x with the same scale but reduced clarity i ;

[0015] S12: downsampling the blurred image to a required scale to obtain a target low-resolution face image x;

[0016] S13: All (x, y) pairs constitute a data set D; randomly select from the data set D to divide the training set D train and the test set D test , the quantity ratio is 9:1.

[0017] Furthermore, in the step S11, the specific steps are: blur convolution uses anisotropic Gaussian blur kernel and isotropic Gaussian blur kernel to blur the high-resolution face image to obtain a blurred face image; assuming that the value range of the random angle number θ is [0, π], the downsampling scale factor is s, and the eigenvalue ranges of λ1 and λ2 are [1, 2.5s] and [1, λ1] respectively, then the blur kernel formula is expressed as:

[0018]

[0019] Furthermore, in the step S12, the specific steps are: downsampling the blurred face image to a target low-resolution face image using a bicubic interpolation method; assuming that the high-resolution face image is HR, the low-resolution face image of the target size is LR, and the blur kernel is K, then the degradation process formula without considering noise is:

[0020]

[0021] Furthermore, in step S2, the specific steps are:

[0022] S21: The entire training set D train Divide into N types of tasks according to the type of degradation process The samples in each task maintain the same degradation process and quantity;

[0023] S22: The samples in a single task are further divided into two sub-training sets D of equal size. sub -train and subtest sets D sub -test;

[0024] S23: Use the sub-training set D of N tasks sub-train Perform task-level training to learn and update the base learner base-learner; set the task-level learning rate to α, the task-level training to tr, and sub-train One or more gradient descent updates are calculated for the parameters θ to obtain new parameters θ'. The calculation formula for the gradient update is:

[0025]

[0026] S24: Use subtest set D of N tasks sub-test Perform task-level testing and use the test error to learn and update the meta-learner; let the task-level test be te, T i Subject to the task distribution p(T), the subtest set D sub-test The test error on the update parameter θ' is minimized by the loss, and the formula is as follows:

[0027]

[0028] S25: After multiple rounds of training, the parameter θ is updated to θ T ; Then the update rule of the parameter θ of stochastic gradient descent is:

[0029]

[0030] Furthermore, in step S3, the specific steps are:

[0031] The test set D test All (x i ,y i ) The initial super-resolution model is trained for 10 iterations, and the super-resolution face image obtained is Minimize the error between the high-resolution face image y and the updated parameter is θ k , and obtain the meta-parameter optimization network.

[0032] Furthermore, in step S4, the specific steps are:

[0033] The test set D test Send it to the optimized network for super-resolution reconstruction, which is implemented by convolution. Let the i-th low-resolution face image be x i , the optimization model is f θ , get the i-th super-resolution face image It is expressed as:

[0034]

[0035] Furthermore, the following steps are included: let the network parameter be θ, the data set be D, the current number of samples be m, and the i-th high-resolution face image in the data set be y i , the super-resolution face image generated by the i-th low-resolution face image is Then the loss function of the entire network is It is expressed as:

[0036]

[0037] A human face super-resolution system based on a meta-learning optimization network comprises an image processing module, a meta-training module, a parameter optimization module and a super-resolution result module which are connected in sequence; the image processing module is used for subjecting a high-resolution human face image to different degradation processes to obtain a target low-resolution human face image, and dividing it into a training set and a test set; the meta-training module is used for sending the low-resolution image of the training set into a meta-training network for training, and obtaining an initial super-resolution model for learning the intrinsic feature connection of multiple types of degradation processes; the parameter optimization module is used for sending the low-resolution image of the test set into the initial super-resolution model for training, and performing a small amount of iterative update on the parameters, and obtaining an optimization model with good adaptability to the new degradation process; the super-resolution result module is used for placing the low-resolution image of the test set into the optimization model for processing, and obtaining a final super-resolution image.

[0038] Furthermore, the meta-training network is an 8-layer convolutional network with single-skip connections, which is used to extract shallow feature maps and connect them with deep features.

[0039] The beneficial effects of the present invention are:

[0040] 1. The present invention discloses a face super-resolution method and system based on a meta-learning optimization network. The meta-learning optimization network connects the intrinsic features of multiple degradation processes learned in the training phase and applies them to the unknown degradation process under test. This solves the technical problem that the current face super-resolution reconstruction algorithm has certain limitations in the reconstruction performance of face images when the image degradation type is unknown, thereby improving the reconstruction performance of face images.

[0041] 2. The present invention utilizes meta-learning to quickly learn the characteristics of a new task, so that the network can achieve good super-resolution performance on new test tasks through empirical learning of training tasks.

[0042] 3. The present invention can address the problem of degradation in super-resolution performance of facial images when the degradation process is unknown, and generate higher quality high-resolution facial images. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of an embodiment of the present invention.

[0044] Figure 2 It is a network structure diagram of an embodiment of the present invention.

[0045] Figure 3 It is a functional block diagram of an embodiment of the present invention.

[0046] Figure 4 2 is a result comparison diagram of the embodiments of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0048] See also Figure 1 and Figure 2 , a face super-resolution method based on a meta-learning optimization network of the present invention comprises the following steps:

[0049] S1: Perform different types of degradation processes on the high-resolution face images to obtain the corresponding target low-resolution face images, thereby dividing the training set and the test set;

[0050] S11: high-resolution face image y i Perform blur convolution to obtain a blurred image x with the same scale but reduced clarityi ;

[0051] S12: downsampling the blurred image to a required scale to obtain a target low-resolution face image x;

[0052] S13: All (x, y) pairs constitute a data set D; randomly select from the data set D to divide the training set D train and the test set D test , the quantity ratio is 9:1.

[0053] In an embodiment of the present invention, the degradation process includes blur convolution and downsampling. Blurred convolution uses anisotropic Gaussian blur kernel and isotropic Gaussian blur kernel to blur the high-resolution face image to obtain a blurred face image. Downsampling uses bicubic interpolation to downscale the blurred face image to the target low-resolution face image. Let θ be a random angle number belonging to [0, π], the eigenvalue ranges of λ1 and λ2 are [1, 2.5s] and [1, λ1] respectively, s is the downsampling scale factor, and the blur kernel formula is expressed as:

[0054]

[0055] In some optional implementation schemes, in step S1, the degradation process only includes the blur kernel and the downsampling process, and the noise is not considered. The formula is as follows:

[0056]

[0057] The HR face image y is first convolved with a blur kernel and then downsampled to obtain the LR face image x of the target size.

[0058] In the embodiment of the present invention, the FEI face dataset is preferably used as the training set and the test set, which includes 360 images as the training dataset and 40 images as the test dataset. The image size in the dataset is 360×260 pixels. The present invention uses a bicubic degradation model to downsample the dataset, where the downsampling factor is 4, so the size of the downsampled low-resolution image is 90×65 pixels.

[0059] S2: Divide the low-resolution images of the training set into blocks, and use the meta-training network to learn the feature mapping from low-resolution face images to high-resolution images under multiple types of degradation processes, and obtain an initial super-resolution model with transferable and sensitive parameters;

[0060] The meta-training phase in step S2 is mainly to train a large number of different tasks (with different blur kernels) to obtain internal knowledge connections across tasks. Different from the usual setting, the training is based on tasks rather than samples. A task contains multiple samples.

[0061] S21: The entire training set D train Divide into N types of tasks according to the type of degradation process The samples in each task maintain the same degradation process and quantity;

[0062] S22: The samples in a single task are further divided into two sub-training sets D of equal size. sub-train and subtest set D sub-test ;

[0063] S23: Use the sub-training set D of N tasks sub-train Perform task-level training to learn and update the base learner; set the task-level learning rate to be α, the task-level training to be tr, and sub-train One or more gradient descent updates are calculated for the parameters θ to obtain new parameters θ'. The calculation formula for the gradient update is:

[0064]

[0065] S24: Use subtest set D of N tasks sub-test Perform task-level testing and use the test error to learn and update the meta-learner; let the task-level test be te, T i Subject to the task distribution p(T), the subtest set D sub-test The test error on the update parameter θ' is minimized by the loss, and the formula is as follows:

[0066]

[0067] S25: After multiple rounds of training, the parameter θ is updated to θ T ; Then the update rule of the parameter θ of stochastic gradient descent is:

[0068]

[0069] S3: Send the low-resolution face test set to the initial super-resolution model for training, update the model parameters a few times to adapt to the new degradation process, and obtain the meta-parameter optimization network.

[0070] The test set D test All (x i ,y i ) is fed into the initial super-resolution model, the network is iterated for 10 trainings and the parameters are updated to obtain an optimized network. In the embodiment of the present invention, the test set D is directly test The obtained SR image The error between the HR image y is minimized and the updated parameter is θ k, and get the optimized model.

[0071] S4: Send the low-resolution face test set to the meta-parameter optimization network for learning to obtain the final high-resolution face image.

[0072] The test set D test Send it to the optimized network for super-resolution reconstruction, which is implemented by convolution. Let the i-th low-resolution face image be x i , the optimization model is f θ , get the i-th super-resolution face image It is expressed as:

[0073]

[0074] S5: Let the network parameters be θ, the data set be D, the current number of samples be m, and the i-th high-resolution face image in the data set be y i , the super-resolution face image generated by the i-th low-resolution face image is Then the loss function of the entire network is It is expressed as:

[0075]

[0076] The network used in steps S2, S3, and S4 is a simple 8-layer single-hop convolutional network, which first extracts shallow feature maps and connects them with deep features. Compared with densely connected convolutional networks, it has fewer parameters and lower computational cost.

[0077] See also Figure 3 According to another aspect of the present invention, a face super-resolution system based on a meta-learning optimization network is provided, comprising:

[0078] An image processing module 101 is used to perform different degradation processes on the high-resolution face image to obtain a target low-resolution face image, and divide it into a training set and a test set;

[0079] A meta-training module 102, used for sending the training set low-resolution images into the network for training, so as to obtain an initial super-resolution model capable of learning the intrinsic feature connections of multiple types of degradation processes;

[0080] The parameter optimization module 103 is used to send the low-resolution images of the test set to the initial super-resolution model for training, perform a small amount of iterative updates on the parameters, and obtain an optimized model with good adaptability to the new degradation process;

[0081] The super-resolution result module 104 is used to process the low-resolution image of the test set into the optimization model to obtain the final super-resolution image;

[0082] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the face super-resolution method based on the meta-learning optimization network described in any one of the above is implemented.

[0083] The present invention uses the FEI face dataset to verify the algorithm. 360 images are used as the training dataset and 40 images are used as the test dataset. The HR image size is 360×260 pixels, and the downsampling factor is 4, so the size of the LR image (using the bicubic degradation model) is 90×65 pixels. It should be noted that all training and testing are based on the brightness channel in the YCbCr color space, and a 4-fold magnification factor is used for training and testing. The SR reconstruction results are evaluated using two evaluation indicators, Peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), to verify the performance of SR reconstruction under the brightness channel. During the training process, each iteration includes 5 tasks (corresponding to 5 different blur kernels), and each task samples 8×2 face images for training. The model of the present invention is trained by the Adam optimizer, α=0.01, β=0.0001. The initial learning rate is set to 10 -4 The model of the present invention is trained using the Tensorflow platform and a GTX1080Ti graphics card. Table 1 shows the comparison results under the condition of a reconstruction multiple of 4 using the above three evaluation indicators. Figure 4 The comparison diagram of the reconstruction results of 4 times the face image, where (a) represents the Bicubic result, (b) represents the MSRN result, (c) represents the RCAN result, (d) represents the ZSSR result, (e) represents the DBPN result, (f) represents the RDN result, (g) SAN result, (h) represents the result of the present invention, and (i) represents the original high-resolution face reference image. It can be found that the present invention has achieved excellent visual effects at the visual level. It should be noted that the comparison method uses the same experimental settings as the present invention, and the blur kernels used in the training and testing stages are the same. In particular, it is pointed out that the blur kernel in the testing stage has not been used in the training stage.

[0084] Table 1 Comparison results between the present invention and seven excellent algorithms

[0085] method Bicubic MSRN RCAN ZSSR DBP RDN SAN The present invention PSNR / dB 36.29 37.57 38.26 37.83 38.13 36.45 37.10 38.62 SSIM 0.9416 0.9524 0.9517 0.9490 0.9522 0.9273 0.9434 0.9529

[0086] It can be seen from the experimental index data in the above table that the present invention has achieved both performance indicators compared with the other seven methods, which can prove that the present invention is superior to the other methods.

[0087] The method according to the present invention described above can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and will be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the face super-resolution method based on a hierarchical multi-scale residual fusion network described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the processing shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for performing the processing shown herein.

[0088] The parts not described in the specification are all prior art or common knowledge. This embodiment is only used to illustrate the invention, and is not used to limit the scope of the invention. The equivalent substitution and other modifications made by those skilled in the art to the invention are considered to fall within the scope of protection of the claims of the invention.

[0089] It should be pointed out that, according to the needs of implementation, the various steps / components described in this application can be split into more steps / components, and two or more steps / components or partial operations of steps / components can be combined into new steps / components to achieve the purpose of the present invention.

[0090] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A face super-resolution method based on a meta-learning optimization network, characterized by: The following steps are involved: S0: Build a face super-resolution system based on a meta-learning optimization network, including an image processing module, a meta-training module, a parameter optimization module and a super-resolution result module connected in sequence; S1: The image processing module performs different types of degradation processes on the high-resolution face images to obtain the corresponding target low-resolution face images, thereby dividing the training set and the test set; S2: The meta-training module divides the low-resolution face images in the training set into blocks, and uses the meta-training network to learn the feature mapping from low-resolution face images to high-resolution face images under various degradation processes, and obtains an initial super-resolution model with transferable and sensitive parameters. S3: The parameter optimization module sends the low-resolution face images of the test set to the initial super-resolution model for training, iterates the model parameters for a set number of times to adapt to the new degradation process, and obtains a meta-parameter optimization network; S4: The super-resolution result module sends the low-resolution face images of the test set into the meta-parameter optimization network for learning to obtain the final high-resolution face images.

2. The face super-resolution method based on meta-learning optimization network according to claim 1, characterized in that: In the step S1, the specific steps are: S11: high-resolution face image y i Perform blur convolution to obtain a blurred image x with the same scale but reduced clarity i ; S12: downsampling the blurred image to a required scale to obtain a target low-resolution face image x; S13: All (x, y) pairs constitute a data set D; randomly select from the data set D to divide the training set D train and the test set D test , the quantity ratio is 9:

1.

3. The face super-resolution method based on meta-learning optimization network according to claim 2, characterized in that: In the step S11, the specific steps are: blur convolution uses anisotropic Gaussian blur kernel and isotropic Gaussian blur kernel to blur the high-resolution face image to obtain a blurred face image; assuming that the value range of the random angle number θ is [0, π], the downsampling scale factor is s, and the eigenvalue ranges of λ1 and λ2 are [1, 2.5s] and [1, λ1] respectively, then the blur kernel formula is expressed as:

4. The face super-resolution method based on meta-learning optimization network according to claim 3, characterized in that: In the step S12, the specific steps are: downsampling using bicubic interpolation to downscale the blurred face image to a target low-resolution face image; assuming that the high-resolution face image is HR, the low-resolution face image of the target size is LR, and the blur kernel is K, then the degradation process formula without considering noise is:

5. The face super-resolution method based on meta-learning optimization network according to claim 3, characterized in that: In the step S2, the specific steps are: S21: The entire training set D train Divide into N types of tasks according to the type of degradation process The samples in each task maintain the same degradation process and quantity; S22: The samples in a single task are further divided into two sub-training sets D of equal size. sub-train and subtest set D sub-test ; S23: Use the sub-training set D of N tasks sub-train Perform task-level training to learn and update the base learner base-learner; set the task-level learning rate to α, the task-level training to tr, and sub-train One or more gradient descent updates are calculated for the parameters θ to obtain new parameters θ'. The calculation formula for the gradient update is: S24: Use subtest set D of N tasks sub-test Perform task-level testing and use the test error to learn and update the meta-learner meta-leamer; let the task-level test be te, T i Subject to the task distribution p(T), the subtest set D sub-test The test error on the update parameter θ' is minimized by the loss, and the formula is as follows: S25: After multiple rounds of training, the parameter θ is updated to θ T ; Then the update rule of the parameter θ of stochastic gradient descent is:

6. The face super-resolution method based on meta-learning optimization network according to claim 5, characterized in that: In the step S3, the specific steps are: The test set D test All (x i ,y i ) The initial super-resolution model is trained for 10 iterations, and the super-resolution face image obtained is Minimize the error between the high-resolution face image y and the updated parameter is θ k , and obtain the meta-parameter optimization network.

7. The face super-resolution method based on meta-learning optimization network according to claim 6, characterized in that: In the step S4, the specific steps are: The test set D test Send it to the optimized network for super-resolution reconstruction, which is implemented by convolution. Let the i-th low-resolution face image be x i , the optimization model is f θ , get the i-th super-resolution face image It is expressed as:

8. The face super-resolution method based on meta-learning optimization network according to claim 7, characterized in that: The following steps are also included: Suppose the network parameters are θ, the data set is D, the current number of samples is m, and the i-th high-resolution face image in the data set is y i , the super-resolution face image generated by the i-th low-resolution face image is Then the loss function of the entire network is It is expressed as:

9. A system for the face super-resolution method based on meta-learning optimization network according to any one of claims 1 to 8, characterized in that: It includes an image processing module, a meta-training module, a parameter optimization module and a super-resolution result module which are connected in sequence; The image processing module is used to perform different degradation processes on the high-resolution face images to obtain the target low-resolution face images, and divide them into training sets and test sets; The meta-training module is used to send the low-resolution images of the training set to the meta-training network for training, so as to obtain an initial super-resolution model for learning the intrinsic feature connections of multiple types of degradation processes; The parameter optimization module is used to feed the low-resolution images of the test set into the initial super-resolution model for training, and perform a small number of iterative updates on the parameters to obtain an optimized model that is well adaptable to the new degradation process; The super-resolution result module is used to process the low-resolution images of the test set into the optimization model to obtain the final super-resolution image.

10. The system according to claim 9, characterized in that: The meta-training network is an 8-layer convolutional network with single-skip connections, which is used to extract shallow feature maps and connect them with deep features.

Images