Super-resolution repair network model training method and device and video repair method and device

Abstract

The invention relates to a super-resolution repair network model training method and device and a video repair method and device, and the model training method comprises the steps: determining a face image in each video frame in a video work as a low-quality face image, and determining a corresponding high-quality face image; determining a degradation parameter of the face image by using the low-quality face image and the corresponding high-quality face image, and applying the degradation parameter to a high-quality natural image in a high-quality natural image set to obtain a corresponding low-quality natural image; and training a super-resolution repair network model by adopting the low-quality natural image and the corresponding high-quality natural image to obtain a trained super-resolution repair network model, and accurately and efficiently repairing the video work through the network model, so that the overall consistency of the images in the video work can be improved, and the video work quality is improved. And the repairing effect can be improved.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a super-resolution restoration network model training, video restoration method and device. Background Art

[0002] At present, in the process of restoring some video works, due to the age of transmission media and other reasons, various problems will occur in the video quality, such as noise, blur, distortion, etc., which seriously affect the user's perception and viewing experience. Among them, the video works can be classic film and television works, etc. In order to solve the above problems, super-resolution technology is usually used to restore the video works.

[0003] There are a large number of video works in the video work library. When using traditional super-resolution technology to restore video works, while improving the resolution, it is often difficult to ensure the overall consistency of the image in the video works. Summary of the invention

[0004] The present application provides a super-resolution restoration network model training, video restoration method and device, which can improve the overall consistency of images in video works and improve the restoration effect.

[0005] In a first aspect, the present application provides a super-resolution restoration network model training method, comprising:

[0006] Determine a face image in each video frame in the video work as a low-quality face image, and determine a corresponding high-quality face image;

[0007] Determine a degradation parameter of the face image using the low-quality face image and the corresponding high-quality face image, and apply the degradation parameter to a high-quality natural image in a high-quality natural image set to obtain a corresponding low-quality natural image;

[0008] The low-quality natural images and the corresponding high-quality natural images are used to train a super-resolution restoration network model to obtain a trained super-resolution restoration network model.

[0009] In a second aspect, the present application provides a video repair method, comprising:

[0010] Obtain a video work, and use the trained super-resolution restoration network model to process video frames in the video work frame by frame;

[0011] The processed video frames are synthesized into a new video file in sequence to obtain a restored video work; wherein the super-resolution restoration network model is trained by the method provided in the embodiment of the present application.

[0012] In a third aspect, an embodiment of the present application provides a super-resolution restoration network model training device, comprising:

[0013] A face image determination module, used to determine a face image in each video frame in the video work as a low-quality face image, and determine a corresponding high-quality face image;

[0014] A natural image determination module, configured to determine a degradation parameter of a facial image using the low-quality facial image and the corresponding high-quality facial image, and apply the degradation parameter to a high-quality natural image in a high-quality natural image set to obtain a corresponding low-quality natural image;

[0015] The training module is used to train the super-resolution restoration network model using the low-quality natural images and the corresponding high-quality natural images to obtain a trained super-resolution restoration network model.

[0016] In a fourth aspect, an embodiment of the present application provides a video repair device, comprising:

[0017] The video frame processing module is used to obtain video works and use the trained super-resolution restoration network model to process the video frames in the video works frame by frame;

[0018] The video frame synthesis module is used to synthesize the processed video frames into a new video file in sequence to obtain a restored video work; wherein the super-resolution restoration network model is trained by the super-resolution restoration network model training method provided in the present application.

[0019] In a fifth aspect, an embodiment of the present application provides an electronic device, including:

[0020] Memory for storing computer programs;

[0021] The processor is used to execute the computer program stored in the memory, and when the computer program is executed, the method provided in the embodiment of the present application is implemented.

[0022] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method provided in the above-mentioned embodiment of the present application is implemented.

[0023] The technical solution provided by the embodiment of the present application is to identify the face image in each video frame from the video work as a low-quality face image, and obtain the corresponding high-quality face image; determine the degradation parameters of the face image through the low-quality face image and the corresponding high-quality face image, apply the degradation parameters of the face image to the high-quality natural image set, and obtain the low-quality natural image corresponding to each high-quality natural image; finally, train the super-resolution restoration network model by using the high-quality natural images and the corresponding low-quality natural images to obtain the trained super-resolution restoration network model.

[0024] The present application processes face images in video works, applies the face image degradation method to high-quality natural images, obtains corresponding low-quality natural images, forms a training data set of natural images, and trains a super-resolution restoration network model through the training data set of natural images to obtain a corresponding trained model, so as to process each frame of the video work in sequence according to the model, and synthesize the processed video frames into a new video file to complete the restoration of the video work. Since the degradation parameters of the face images are applied to high-quality natural images to obtain a training data set of natural images, and the super-resolution restoration network model is trained, a trained super-resolution restoration network model can be obtained based on the face structure prior, so that the overall consistency of the image of the face area and the non-face area in the video work can be improved, and the problem of artifacts or loss of details in the detail and complex texture area, which reduces the picture quality, can be solved, and the video work can be restored efficiently and accurately; each trained super-resolution restoration network model has a corresponding video work, so that each video work can use the trained super-resolution restoration network model separately, which can overcome the difficulty of the video work to adapt to the diverse situations such as noise and degradation, and can ensure that the restored image keeps the original details and realism as much as possible, and improve the restoration effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0027] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0028] Figure 1 It is a flow chart of a super-resolution restoration network model training method provided in an embodiment of the present application;

[0029] Figure 2 It is a schematic diagram of the process of determining a high-quality face image;

[0030] Figure 3a It is a high-quality natural image;

[0031] Figure 3b yes Figure 3a Low-quality natural images corresponding to medium-high-quality natural images;

[0032] Figure 4a It is another high-quality natural image;

[0033] Figure 4b yes Figure 4a Low-quality natural images corresponding to medium-high-quality natural images;

[0034] Figure 5 It is a flow chart of a super-resolution restoration network model training method provided in an embodiment of the present application;

[0035] Figure 6 It is a schematic diagram of the training of the degraded parameter network model;

[0036] Figure 7 is a flow chart of a video repair method provided by an embodiment of the present application;

[0037] Figure 8 is a flow chart of a video repair method provided by an embodiment of the present application;

[0038] Fig. 9 This is a structural block diagram of a super-resolution restoration network model training device provided in an embodiment of the present application;

[0039] Fig.10 It is a structural block diagram of a video repair device provided in an embodiment of the present application;

[0040] Fig.11 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0042] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0043] In the related art, the traditional super-resolution technology used to restore video works can improve the resolution, but it is often difficult to ensure the consistency of the overall image in the video work, especially in the details and complex texture areas, which are prone to artifacts or loss of details, reducing the picture quality. In order to solve the above problems, this application provides the following technical solutions, as follows:

[0044] Figure 1 A flow chart of a super-resolution restoration network model training method provided in an embodiment of the present application, wherein the method is performed by a super-resolution restoration network model training device, the device can be implemented by software and / or hardware, the device can be configured in electronic devices such as computers, and the method can be applied to scenarios for restoring video works.

[0045] like Figure 1 As shown, the technical solution provided by the embodiment of the present application includes:

[0046] S110: Determine a face image in each video frame in the video work as a low-quality face image, and determine a corresponding high-quality face image.

[0047] In the embodiment of the present application, the process of obtaining a high-quality face image by processing the video work can be specifically referred to in Figure 2 ,like Figure 2 As shown, by extracting video frames from the video work, the video frames are formed into an image sequence, and then face recognition is performed to obtain a low-quality face image. The low-quality face image is restored to a high-quality face image through a trained face super-resolution restoration network model.

[0048] In one implementation of the embodiments of the present application, optionally, the determining of the facial images in each video in the video work includes: extracting video frames frame by frame from the video file of the video work, performing facial recognition on the video frames, and processing the recognized facial images. Specifically, the video work may be a classic film and television work, etc., and the classic film and television work may include a TV series, a movie, etc. Optionally, the processing of video frames and facial recognition may use methods in the prior art, such as opencv tools, etc. Optionally, each recognized facial image may be aligned and preprocessed to facilitate the accuracy of subsequent data. Among them, the alignment process specifically aligns the faces in each facial image to ensure that the faces have consistent postures and positions in different video frames, so as to facilitate the subsequent acquisition of accurate data.

[0049] In one embodiment of the present application, determining the corresponding high-quality face image includes: inputting the low-quality face image into a trained face super-resolution restoration network model to obtain a high-quality face image. Optionally, the trained face super-resolution restoration network model can be an existing model, using an existing open source network. Alternatively, the face super-resolution restoration network model can be trained using a training data set of face images to obtain a trained face super-resolution restoration network model. Among them, the face super-resolution restoration network model can be a model of a generative adversarial network (GAN).

[0050] S120: Determine degradation parameters of the facial image using the low-quality facial image and the corresponding high-quality facial image, and apply the degradation parameters to the high-quality natural image in the high-quality natural image set to obtain the corresponding low-quality natural image.

[0051] In the embodiment of the present application, each low-quality face image and the corresponding high-quality face image form a low-quality (LQ)-high-quality (HQ) face image pair, i.e., an LQ-HQ face image pair. By summarizing and analyzing each face image pair, the degradation characteristics of the high-quality-low-quality face image can be captured and the degradation parameters can be determined. Alternatively, each low-quality face image and the corresponding high-quality face image can be used to train a degradation parameter network to learn the degradation parameters of the HQ-LQ face image. Alternatively, other reasonable methods can be used.

[0052] In this embodiment, optionally, the degradation parameter may include resolution, noise information, etc. The noise information may include noise intensity, noise type, etc. In this embodiment, when a high-quality face image is degraded to a low-quality face image, each pixel unit in the high-quality face image may be degraded to a corresponding pixel unit in the low-quality face image.

[0053] In the embodiment of the present application, the degradation parameters of the face image are applied to the high-quality natural image to obtain the corresponding low-quality natural image. Specifically, the high-quality natural image is degraded based on the degradation parameters to obtain the corresponding low-quality natural image. The low-quality natural image and the corresponding high-quality natural image are the LQ-HQ natural image pair, where: Figure 3a and Figure 3b are high-quality natural images and corresponding low-quality natural images, Figure 4a and Figure 4b They are another high-quality natural image and a corresponding low-quality natural image. The LQ-HQ natural image pair ensures that the image data covers a wide enough range of natural image content to improve the generalization ability of the super-resolution restoration network model. Optionally, the high-quality natural image set includes images from different data sources, such as images containing faces, natural scenery images, etc. The high-quality natural image set contains a sufficient number, and the current number can reach 1 million.

[0054] S130: Using the low-quality natural image and the corresponding high-quality natural image to train a super-resolution restoration network model to obtain a trained super-resolution restoration network model.

[0055] In the embodiment of the present application, each low-quality natural image and the corresponding high-quality natural image are used to train the super-resolution restoration network model, thereby obtaining a trained super-resolution restoration network model. Optionally, the super-resolution restoration network model can be an enhanced super-resolution network (Enhanced Deep Super-ResolutionNetwork) architecture or a deep residual channel attention network (Residual Channel Attention Network, RCAN) architecture.

[0056] In one implementation of the present application, optionally, the use of low-quality natural images and corresponding high-quality natural images to train a super-resolution restoration network model to obtain a trained super-resolution restoration network model includes: inputting a low-quality natural image into the super-resolution restoration network model, outputting a corresponding high-quality natural image as an output high-quality natural image; optimizing the super-resolution restoration network model through a second loss function, the output high-quality natural image, and the high-quality natural image, repeating the optimization steps and obtaining a trained super-resolution restoration network model. Specifically, the output high-quality natural image and the original high-quality natural image are substituted into the loss function to calculate the loss value. If the loss value does not meet the preset conditions or the iteration does not reach the preset number of times, etc., the parameters of the super-resolution restoration network model are adjusted, and the above optimization steps are repeated to continuously adjust the parameters of the super-resolution restoration network model until a trained super-resolution restoration network model is obtained.

[0057] The technical solution provided by the embodiment of the present application is to identify the face image in each video frame from the video work as a low-quality face image, and obtain the corresponding high-quality face image; determine the degradation parameters of the face image through the low-quality face image and the corresponding high-quality face image, apply the degradation parameters of the face image to the high-quality natural image set, obtain the low-quality natural image corresponding to each high-quality natural image, and finally use each high-quality natural image and the corresponding low-quality natural image to train the super-resolution restoration network model to obtain a trained super-resolution restoration network model. That is, by processing the face image in the video work, applying the degradation method of the face image to the high-quality natural image, obtaining the corresponding low-quality natural image, forming a training data set of natural images, and training the super-resolution restoration network model through the training data set of natural images to obtain a corresponding trained model, that is, the super-resolution restoration network model obtained by the face structure prior can efficiently and accurately restore the video work, and the use of this model can improve the overall consistency of the image of the face area and the non-face area in the video work, and solve the problem of artifacts or loss of details in the detail and complex texture area, which reduces the picture quality.

[0058] In the related art, there are a large number of video works in the video work library. Each video work may have different noises, degradation characteristics, and different conditions during shooting, storage, and digitization. The unified super-resolution and denoising models used in the related art are difficult to adapt to diverse situations, resulting in unsatisfactory restoration effects. Since each trained super-resolution restoration network model is based on the corresponding video work, each trained super-resolution restoration network model has a corresponding video work, so that each video work can use a trained super-resolution restoration network model separately, which can overcome the difficulty of video works to adapt to diverse situations such as noise and degradation, ensure that the restored image maintains the original details and realism as much as possible, and improve the restoration effect.

[0059] Figure 5 It is a flow chart of a super-resolution restoration network model training method provided in an embodiment of the present application. In this embodiment, "using low-quality face images and corresponding high-quality face images to determine degradation parameters of face images, applying the degradation parameters to high-quality natural images in a high-quality natural image set to obtain corresponding low-quality natural images" is specifically defined, that is, the degradation parameters are learned by model learning, so that the model is applied to a high-quality natural image set.

[0060] like Figure 5 As shown, the technical solution provided by the embodiment of the present application includes:

[0061] S510: Determine a face image in each video frame in the video work as a low-quality face image, and determine a corresponding high-quality face image.

[0062] S520: Using the low-quality face image and the corresponding high-quality face image to train the degraded parameter network model to obtain a trained degraded parameter network model.

[0063] In an embodiment of the present application, a degraded parameter network model can be designed and constructed, which can be a convolutional neural network model of deep learning. Optionally, the degraded parameter network model can be based on a variational autocoder (VAE) or other suitable model architecture. Among them, VAE is a generative model that learns the probability distribution of data by mapping data to a latent space. In image processing tasks, VAE can be used to learn a compressed representation of an image, which is beneficial for image degradation modeling because the complex characteristics of the image can be captured, and the latent space can provide a way to control image generation. Specifically, VAE can help learn a degraded representation of an image without paired data, thereby allowing real-world low-quality images to be processed more effectively in blind image super-resolution tasks. In this embodiment, each low-quality face image and the corresponding high-quality face image are input into the degraded parameter model for training (refer to Figure 6 ), the model can learn the degradation parameters from high-quality face images to low-quality face images, so that the model has corresponding degradation characteristics during use, so that the trained degradation parameter model can degrade the face image based on the degradation parameters.

[0064] In one implementation of the present application, optionally, the high-quality face image is input into the degraded parameter network model, and the corresponding low-quality face image is output as the output low-quality face image; the degraded parameter network model is optimized by the first loss function, the output low-quality face image and the low-quality face image, and the optimization steps are repeated to obtain a trained degraded parameter network model. Specifically, the output low-quality face image and the original low-quality face image are substituted into the loss function to calculate the loss value. If the loss value does not meet the set conditions or the iteration does not reach the set number of times, the model parameters are adjusted, and the above optimization steps are repeated to continuously adjust the parameters of the model until a trained degraded parameter network is obtained.

[0065] In the related art, the degradation method used for image training data is often manually designed, which makes it difficult to capture the complex noise of the real world in the image. In the embodiment of the present application, a degradation parameter network model is designed and constructed, and is trained using high-quality face images and corresponding low-quality face images. The model can learn the image degradation process and accurately degrade the image. Therefore, it can accurately capture the complex noise of the real world in the image and avoid the influence of complex noise.

[0066] S530: Inputting the high-quality natural image in the high-quality natural image set into the trained degradation parameter network model to obtain a corresponding low-quality natural image.

[0067] In an embodiment of the present application, the degradation parameter network model can learn the degradation parameters of a high-quality face image to a corresponding low-quality face image. The high-quality natural images in the high-quality natural image dataset are input into the degradation parameter network model, and the degradation parameter network model can degrade the high-quality natural images based on the degradation parameters to obtain the corresponding low-quality natural images, thereby making the overall image in the video work restored by the trained super-resolution restoration network model consistent.

[0068] Therefore, the embodiment of the present application constructs a degradation parameter network model, trains it, and inputs a high-quality natural image into the model to obtain a corresponding low-quality natural image. The degradation method is learned by the model. Compared with the manual design method, it can capture the complex noise of the real world in the image and avoid the influence of noise. It can accurately degrade the image, thereby making the training of the super-resolution restoration network model more accurate.

[0069] S540: Using the low-quality natural image and the corresponding high-quality natural image to train a super-resolution restoration network model to obtain a trained super-resolution restoration network model.

[0070] Figure 7 It is a flow chart of a video repair method provided in an embodiment of the present application. The method can be implemented by a video repair device, the device can be implemented by software and / or hardware, and the device can be configured in electronic devices such as computers.

[0071] like Figure 7 As shown, the technical solution provided by the embodiment of the present application includes:

[0072] S710: Obtain a video work, and use the trained super-resolution restoration network model to process video frames in the video work frame by frame.

[0073] In the embodiment of the present application, the super-resolution restoration network model can be trained by the super-resolution restoration network model training method provided in the embodiment of the present application. In the process of obtaining the trained super-resolution restoration network model, the face image in the video work is applied, and the trained super-resolution restoration network model is used to process the video frames in the video work frame by frame, including the face area and non-face area in the video frame, to ensure the consistency of the overall image (or overall picture) quality.

[0074] In an embodiment of the present application, after processing the video frame, the consistency of the face area and the non-face area in the restored video frame can also be verified, so as to ensure that the image details and the overall visual effect are better. Optionally, the verification of the restored video frame can be a subjective verification, or the restoration effect can be quantified by image quality indicators. For example, the difference between the image before and after restoration can be evaluated by indicators such as the structural similarity index and the peak signal-to-noise ratio. Thus, by verifying the processed video frame, it is possible to evaluate whether the face area and the non-face area in the restored image are consistent, thereby evaluating the quality of the overall restored image.

[0075] S720: Synthesize the processed video frames into a new video file in sequence to obtain a restored video work.

[0076] In an embodiment of the present application, the processed high-quality video frames are synthesized into a new video file in the original order, which can ensure smoothness and consistency during playback.

[0077] The technical solution provided by the present application is to process the face image, apply the degradation method of the face image to the high-quality natural image, obtain the corresponding low-quality natural image, form a training data set of natural images, and train the super-resolution restoration network model through the training data set of natural images to obtain the corresponding trained model, that is, to restore the video work through the super-resolution restoration network model obtained by the face structure prior, so as to efficiently and accurately restore the video work. The use of this model can improve the overall consistency of the image in the video work, and solve the problem that artifacts or loss of details are easy to occur in the details and complex texture areas, thereby reducing the picture quality. Since each trained super-resolution restoration network model is obtained based on the corresponding video work, each trained super-resolution restoration network model has a corresponding video work, and each video work can use a separately trained super-resolution restoration network model, which can overcome the difficulty of the video work in adapting to the diverse situations such as noise and degradation, and can make the image in the restored video work keep the original details and realism as much as possible, improve the restoration effect, and can significantly enhance the visual experience of film and television works, providing an efficient and comprehensive solution.

[0078] The overall flow chart of the video repair method provided in the embodiment of the present application can be referred to Figure 8 In order to more clearly describe the technical solution of the embodiment of the present application, the technical solution of the present application may specifically include the following steps. In this embodiment, the video work may be a TV series.

[0079] 1. Face data collection and preprocessing

[0080] 1.1 Video frame extraction:

[0081] Video frames are extracted frame by frame from the video file of the target TV series, and these video frames will be used in subsequent processing steps.

[0082] 1.2 Face detection and alignment:

[0083] Perform face detection on each extracted video frame and identify the face area therein.

[0084] The detected faces are aligned to ensure that they have consistent posture and position in different frames.

[0085] 2. Super-resolution restoration of face images

[0086] 2.1 Use pre-trained network to repair face images:

[0087] Apply a pre-trained face super-resolution restoration network, such as a GAN-based model, to restore the LQ face image to the HQ face image.

[0088] Generate LQ-HQ face image pairs for subsequent steps.

[0089] 3. Degraded parameter learning

[0090] 3.1 Constructing the degraded parameter network model:

[0091] Design and build a degradation parameter network model, input LQ-HQ face image pairs, and learn the degradation parameters from HQ face images to LQ face images. The network model can be based on VAE or other suitable model architectures.

[0092] 3.2 Training the degraded parameter network model:

[0093] The degradation parameter network model is trained using the LQ-HQ face image pairs generated above, so that it can effectively capture the degradation parameters from HQ face images to LQ face images.

[0094] 4. Degraded parameter migration

[0095] 4.1 High-quality natural image degradation:

[0096] The learned degradation parameters are applied to a high-quality natural image set (e.g., DF2K) to generate corresponding low-quality (LQ) natural images, forming LQ-HQ natural image pairs.

[0097] 4.2LQ-HQ Natural Image Pair Generation:

[0098] The generated LQ-HQ natural image pairs are sorted and saved to form a dataset for subsequent training.

[0099] 5. Super-resolution restoration network model training

[0100] 5.1 Training data preparation:

[0101] The generated LQ-HQ natural image pairs are used to ensure that the dataset covers a wide enough range of natural image content to improve the generalization ability of the model.

[0102] 5.2 Training super-resolution restoration network model:

[0103] Design and train a super-resolution restoration network model (such as EDSR or RCAN architecture) to learn the mapping relationship from LQ natural images to HQ natural images.

[0104] The model parameters are optimized by training the loss function (the second loss function mentioned above) to ensure that the image quality can be effectively improved.

[0105] 6. Film and TV drama image restoration

[0106] 6.1 Super-resolution processing of video frames:

[0107] Use the trained super-resolution restoration network model to process the original TV series video frames frame by frame, including face and non-face areas, to ensure consistent overall picture quality.

[0108] 6.2 Ensure consistency and detail fidelity:

[0109] Verify the consistency of the restored face area and non-face area to ensure that the image details and overall visual effects meet the standards.

[0110] 6.3 Generate repaired video:

[0111] The processed high-quality image frames are synthesized into a new video file in the original order to ensure smoothness and consistency during playback.

[0112] Through the above technical solution, the present application can accurately and efficiently restore old film and television works. When the technical solution provided by the embodiment of the present application is applied to old film and television works, it can be specifically applied to the following scenarios:

[0113] Film and television production and distribution companies: aim to produce and distribute high-quality film and television works, and may use facial restoration technology to restore old film and television works.

[0114] Video streaming platforms: May use face restoration technology to improve the quality of the content they offer, enhance user experience and attract more viewers.

[0115] Film and television post-production company: focuses on providing post-processing services for film and television works, and uses facial restoration technology to provide customers with better restoration services.

[0116] Artificial intelligence and computer vision technology companies: may research and improve face restoration technology, develop new products, or apply it to real-world projects.

[0117] Film and television archives, film museums and archives: For institutions that need to restore, store and display historical content, face restoration technology becomes a viable restoration tool.

[0118] Therefore, the technical solution provided by this application can be applied to many specific scenarios. When applied to old film and television works, it aims to use advanced technology to repair and enhance the faces in old film and television works to improve their visual quality and audience experience, and has the following uses and values:

[0119] Improve visual quality: Old movies and TV series may be affected by factors such as shooting conditions, storage and transmission degradation during production, resulting in blurred image quality and loss of details. The face restoration technology provided by this application (face-based restoration technology, i.e., the video restoration method provided by this application) can effectively improve the visual quality of the face, allowing the audience to see the performer's facial expressions more clearly.

[0120] Enhance artistic value: Through face restoration technology, old film and television works can better convey the performance of the performers and more fully display the artistic value of the works. To a certain extent, it can stimulate the audience's interest in old film and television works and help promote the inheritance of cultural heritage.

[0121] Expand the audience group: With the development of technology, modern audiences have higher and higher requirements for video quality. Through the face restoration technology of this application, the viewing experience of old movies and old TV series is improved, thereby attracting more audiences and expanding the audience range of film and television works.

[0122] Providing a basis for reproduction and re-production: When re-producing old film and television works, the restored faces can provide directors, designers and producers with clearer and more realistic character images. This helps them better understand the original work and provide better reference and basis for subsequent re-production.

[0123] Restoration and protection of film and television archives: The face restoration technology of this application can effectively restore the quality of old film and television works, which is helpful for the restoration of old film and television archives. This is of great significance for protecting human cultural heritage.

[0124] In summary, the technical solution provided by this application has high practical value and significance. It can effectively restore old film and television works, improve their visual quality and artistic value, and provide a basis for subsequent re-production and dissemination.

[0125] Fig. 9is a structural block diagram of a super-resolution restoration network model training device provided in an embodiment of the present application, wherein the device is used to execute a corresponding method, such as Fig. 9 As shown, the device comprises:

[0126] The face image determination module 910 is used to determine the face image in each video frame in the video work as a low-quality face image, and determine the corresponding high-quality face image;

[0127] A natural image determination module 920, configured to determine a degradation parameter of a facial image using the low-quality facial image and the corresponding high-quality facial image, and apply the degradation parameter to the high-quality natural image in the high-quality natural image set to obtain a corresponding low-quality natural image;

[0128] The training module 930 is used to train the super-resolution restoration network model using the low-quality natural images and the corresponding high-quality natural images to obtain a trained super-resolution restoration network model.

[0129] Optionally, the using a low-quality face image and a corresponding high-quality face image to determine a degradation parameter of a face image, and applying the degradation parameter to a high-quality natural image in a high-quality natural image set to obtain a corresponding low-quality natural image includes:

[0130] Using the low-quality face image and the corresponding high-quality face image to train the degraded parameter network model to obtain a trained degraded parameter network model;

[0131] The high-quality natural images in the high-quality natural image set are input into the trained degradation parameter network model to obtain corresponding low-quality natural images.

[0132] Optionally, the adopting of low-quality face images and corresponding high-quality face images to train the degraded parameter network model to obtain a trained degraded parameter network model includes:

[0133] Inputting the high-quality face image into the degradation parameter network model, and outputting a corresponding low-quality face image as an output low-quality face image;

[0134] The degraded parameter network model is optimized by using the first loss function, the output low-quality face image and the low-quality face image, and the optimization steps are repeated to obtain a trained degraded parameter network model.

[0135] Optionally, the adopting of low-quality natural images and corresponding high-quality natural images to train a super-resolution restoration network model to obtain a trained super-resolution restoration network model includes:

[0136] Inputting a low-quality natural image into a super-resolution restoration network model, and outputting a corresponding high-quality natural image as an output high-quality natural image;

[0137] The super-resolution restoration network model is optimized through a second loss function, the output high-quality natural image and the high-quality natural image, and the optimization steps are repeated to obtain a trained super-resolution variability restoration network model.

[0138] Optionally, determining the facial image in each video in the video work includes:

[0139] Video frames are extracted frame by frame from a video file of a video work, face recognition is performed on the video frames, and the recognized face images are processed.

[0140] Optionally, determining a corresponding high-quality face image includes:

[0141] Input the low-quality face image into the trained face super-resolution restoration network model to obtain a high-quality face image.

[0142] The device provided in the embodiments of the present application can execute the method provided in any embodiment of the present application, and has the corresponding functional modules and beneficial effects of the execution method.

[0143] Fig.10 is a structural block diagram of a video repair device provided in an embodiment of the present application, such as Fig. 9 As shown, the device includes a video frame processing module 101 and a video frame synthesis module 102 .

[0144] The video frame processing module 101 is used to obtain a video work and process the video frames in the video work frame by frame using a trained super-resolution restoration network model;

[0145] The video frame synthesis module 102 is used to synthesize the processed video frames into a new video file in sequence to obtain a restored video work; wherein the super-resolution restoration network model is trained by the training method of the super-resolution restoration network model provided in the present application.

[0146] The device provided in the embodiments of the present application can execute the method provided in any embodiment of the present application, and has the corresponding functional modules and beneficial effects of the execution method.

[0147] like Fig.11 As shown, an embodiment of the present application provides an electronic device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0148] Memory 113, used for storing computer programs;

[0149] In one embodiment of the present application, the processor 111 is used to execute the program stored in the memory 113 to implement the method provided by any of the above method embodiments, including:

[0150] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps provided in any of the aforementioned method embodiments are implemented.

[0151] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0152] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0153] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0154] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A super-resolution restoration network model training method, characterized in that: include: Determine a face image in each video frame in the video work as a low-quality face image, and determine a corresponding high-quality face image; Determine a degradation parameter of the face image using the low-quality face image and the corresponding high-quality face image, and apply the degradation parameter to a high-quality natural image in a high-quality natural image set to obtain a corresponding low-quality natural image; The low-quality natural images and the corresponding high-quality natural images are used to train a super-resolution restoration network model to obtain a trained super-resolution restoration network model.

2. The method according to claim 1, characterized in that The method of using a low-quality face image and a corresponding high-quality face image to determine a degradation parameter of the face image, and applying the degradation parameter to a high-quality natural image in a high-quality natural image set to obtain a corresponding low-quality natural image includes: Using the low-quality face image and the corresponding high-quality face image to train the degraded parameter network model to obtain a trained degraded parameter network model; The high-quality natural images in the high-quality natural image set are input into the trained degradation parameter network model to obtain corresponding low-quality natural images.

3. The method according to claim 2, characterized in that The method of using low-quality face images and corresponding high-quality face images to train the degraded parameter network model to obtain a trained degraded parameter network model includes: Inputting the high-quality face image into the degradation parameter network model, and outputting a corresponding low-quality face image as an output low-quality face image; The degraded parameter network model is optimized by using the first loss function, the output low-quality face image and the low-quality face image, and the optimization steps are repeated to obtain a trained degraded parameter network model.

4. The method according to claim 1, characterized in that: The method uses low-quality natural images and corresponding high-quality natural images to train a super-resolution restoration network model to obtain a trained super-resolution restoration network model, including: Inputting the low-quality natural image into a super-resolution restoration network model, and outputting a corresponding high-quality natural image as an output high-quality natural image; The super-resolution restoration network model is optimized through a second loss function, the output high-quality natural image and the high-quality natural image, and the optimization steps are repeated to obtain a trained super-resolution variability restoration network model.

5. The method according to any one of claims 1 to 4, characterized in that: The determining of the facial image in each video in the video work includes: Video frames are extracted frame by frame from a video file of a video work, face recognition is performed on the video frames, and the recognized face images are processed.

6. The method according to any one of claims 1 to 4, characterized in that: The determining of the corresponding high-quality face image comprises: The low-quality face image is input into a trained face super-resolution restoration network model to obtain a high-quality face image.

7. A video repair method, characterized in that: include Obtain a video work, and use the trained super-resolution restoration network model to process video frames in the video work frame by frame; The processed video frames are synthesized into a new video file in sequence to obtain a restored video work; wherein the super-resolution restoration network model is trained by the method described in any one of claims 1-6.

8. A super-resolution restoration network model training device, characterized in that: include: A face image determination module, used to determine a face image in each video frame in the video work as a low-quality face image, and determine a corresponding high-quality face image; A natural image determination module, configured to determine a degradation parameter of a facial image using the low-quality facial image and the corresponding high-quality facial image, and apply the degradation parameter to a high-quality natural image in a high-quality natural image set to obtain a corresponding low-quality natural image; The training module is used to train the super-resolution restoration network model using the low-quality natural images and the corresponding high-quality natural images to obtain a trained super-resolution restoration network model.

9. A video restoration device, characterized in that: include The video frame processing module is used to obtain video works and use the trained super-resolution restoration network model to process the video frames in the video works frame by frame; A video frame synthesis module is used to synthesize the processed video frames into a new video file in sequence to obtain a restored video work; wherein the super-resolution restoration network model is trained by the method described in any one of claims 1-6.

10. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute a computer program stored in the memory, and when the computer program is executed, implement the method described in any one of claims 1 to 7.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method described in any one of claims 1 to 7 is implemented.

Images