Image quality restoration model training method and device, equipment and storage medium

Abstract

This invention discloses a method, apparatus, device, and storage medium for training an image quality restoration model. The method includes: determining a target high-definition video segment associated with a target restoration task; degrading the target high-definition video segment to obtain a corresponding target low-definition video segment; inputting the target low-definition video segment into a pre-constructed first network model to obtain model output; and training the first network model based on the model output and the corresponding target high-definition video segment until a preset first model training termination condition is met, thereby obtaining a target image quality restoration model. This invention improves the restoration effect and quality of video image quality.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a method, apparatus, device, and storage medium for training an image quality restoration model. Background Technology

[0002] Animation is a widely popular art form that plays a vital role in film, television, and games. However, due to technological limitations and cost considerations, many classic animated works were confined to low resolution during production. These older animations appear blurry and distorted when played on modern high-definition televisions and screens, exhibiting numerous image quality issues. Therefore, processing and restoring animation to achieve high-definition quality is of significant historical importance to the animation industry.

[0003] Current technologies for video quality restoration typically employ deep learning-based real-world video super-resolution techniques. However, these methods often result in unclean animation restorations, exhibiting issues such as hollow line artifacts and unwanted noise, leading to poor video restoration quality and overall restoration effectiveness. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and storage medium for training an image quality restoration model, in order to improve the restoration effect and quality of video image quality.

[0005] According to one aspect of the present invention, a method for training an image quality restoration model is provided, the method comprising:

[0006] Identify the target high-resolution video clips associated with the target repair task;

[0007] The target high-definition video clip is degraded to obtain a target low-definition video clip corresponding to the target high-definition video clip.

[0008] The target low-quality video clip is input into a pre-built first network model to obtain the model output. The first network model is then trained based on the model output and the corresponding target high-definition video clip until the preset first model training termination condition is met, thus obtaining the target image quality restoration model.

[0009] According to another aspect of the present invention, an image quality restoration model training apparatus is provided, the apparatus comprising:

[0010] The high-definition segment identification module is used to identify target high-definition video segments associated with the target repair task;

[0011] The low-quality segment determination module is used to degrade the target high-definition video segment to obtain the target low-quality video segment corresponding to the target high-definition video segment.

[0012] The target restoration model determination module is used to input the target low-quality video clip into a pre-built first network model to obtain the model output, and to train the first network model according to the model output and the corresponding target high-definition video clip until the preset first model training termination condition is met, thereby obtaining the target image quality restoration model.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] At least one processor; and

[0015] A memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the image quality restoration model training method according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the image quality restoration model training method according to any embodiment of the present invention.

[0018] The technical solution of this invention identifies a target high-definition video segment associated with a target restoration task, degrades the target high-definition video segment to obtain a target low-definition video segment corresponding to the target high-definition video segment, inputs the target low-definition video segment into a pre-constructed first network model to obtain model output, and trains the first network model based on the model output and the corresponding target high-definition video segment until a preset first model training termination condition is met, thus obtaining a target image quality restoration model. This achieves accurate training of the first network model, resulting in higher model accuracy for the trained target image quality restoration model, thereby improving the restoration effect and quality of video image quality restoration using the target image quality restoration model.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1A This is a flowchart of a method for training an image quality restoration model according to Embodiment 1 of the present invention;

[0022] Figure 1B This is a structural diagram of a multi-scale recurrent network of a first network model provided according to Embodiment 1 of the present invention;

[0023] Figure 2 This is a flowchart of a method for training an image quality restoration model according to Embodiment 2 of the present invention;

[0024] Figure 3A This is a flowchart of a method for training an image quality restoration model according to Embodiment 3 of the present invention;

[0025] Figure 3B This is a model structure diagram of a second network model provided according to Embodiment 3 of the present invention;

[0026] Figure 4 This is a schematic diagram of the structure of an image quality restoration model training device according to Embodiment 4 of the present invention;

[0027] Figure 5 This is a schematic diagram of the structure of an electronic device that implements the image quality restoration model training method of the present invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] Example 1

[0031] Figure 1A This is a flowchart of a method for training an image quality restoration model according to Embodiment 1 of the present invention. This embodiment is applicable to the situation of restoring the image quality of low-quality videos. The method can be executed by an image quality restoration model training device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1A As shown, the method includes:

[0032] S110. Identify the target high-definition video segment associated with the target repair task.

[0033] The target restoration task can be a video quality restoration task. Different target restoration tasks involve different video types, and the image frames of different video types have different image characteristics. Therefore, based on the image characteristics of the video to be restored in the target restoration task, a target high-definition image segment matching those characteristics can be identified.

[0034] The target high-definition image clip can be a video clip obtained by combining high-definition image frames. The image features of the target high-definition image clip match the image features of the video to be repaired in the target repair task. Specifically, at least one high-definition video clip can be extracted from the high-definition video that matches the image features of the video to be repaired, and the extracted high-definition video clip can be used as the target high-definition video clip.

[0035] For example, the image features of the image frames of the video to be repaired in the target repair task are determined; a high-definition video with the same video type or similar texture features as the video to be repaired is obtained; at least one video segment is extracted from the high-definition video as the target high-definition video segment. For example, if the video type of the video to be repaired is an animation video, and the video image texture features of the video to be repaired are relatively rich, then a high-definition video with the same video type and rich image texture features can be selected from the high-definition video database.

[0036] S120. Degrade the target high-definition video clip to obtain the target low-definition video clip corresponding to the target high-definition video clip.

[0037] Degradation processing can involve reducing the image quality of high-definition video. There can be multiple target high-definition video clips, each corresponding to its own target low-definition video clip.

[0038] For example, a preset degradation processing method can be used to degrade the target high-definition video segment to obtain the target high-definition video segment. The degradation processing method can be preset by relevant technical personnel; for example, the degradation processing method can be to reduce the resolution. Specifically, different low resolutions can be preset to obtain video segments of the target high-definition video segment at different low resolutions, which can then be used as the target low-definition video segment.

[0039] It should be noted that, in order to achieve precise degradation of the target high-definition video clips and to degrade the target high-definition video clips to different degrees, so that the target high-definition video clips can contain various degradation features, thereby making the obtained target low-quality video clips more realistic, the following methods can also be used to degrade the target high-definition video clips.

[0040] In one optional embodiment, degrading a target high-definition video segment to obtain a target low-definition video segment corresponding to the target high-definition video segment includes: acquiring at least one degradation processing method; combining the degradation processing methods to obtain at least one combined degradation processing method; and degrading the target high-definition video segment according to the combined degradation processing method to obtain the target low-definition video segment corresponding to the target high-definition video segment; wherein the degradation processing method includes at least one of video compression processing, image compression processing, downsampling processing, anti-aliasing processing, noise addition processing, and blurring processing.

[0041] The video compression process can be based on FFmpeg (Fast Forward MPEG, MPEG video coding standard) for encoding and decoding. Image compression can be based on the JPEG (Joint Photographic Experts Group, image compression standard) compression algorithm. Downsampling can be random downsampling, i.e., downsampling at different ratios, such as 2x or 4x downsampling. Noise processing can include Gaussian color noise processing and Gaussian gray noise processing. Blur processing can generate different blurred data based on different kernels. For example, it can include blurred data generated by Gaussian blur kernels, bivariate generalized Gaussian kernels, and multi-directionally flat anisotropic kernels. It should be noted that degradation processing can also include other methods, not limited to those described above in this embodiment.

[0042] The combined degradation processing method can be obtained by combining various degradation processing methods according to a pre-set degradation order. For example, the combined degradation processing method can be to perform image compression processing, downsampling processing, blur processing, video compression processing, noise addition processing, and anti-aliasing processing in sequence.

[0043] In one optional embodiment, the target high-definition video is degraded according to a combined degradation processing method to obtain a target low-definition video segment corresponding to the target high-definition video segment. This can be achieved by: performing video compression processing on the target high-definition video segment to obtain a first degraded video segment; performing image compression processing on the first degraded video segment to obtain a second degraded video segment; performing downsampling processing on the second degraded video segment to obtain a third degraded video segment; performing anti-aliasing processing on the third degraded video segment to obtain a fourth degraded video segment; performing noise addition processing on the fourth degraded video segment to obtain a fifth degraded video segment; and performing blurring processing on the fifth degraded video segment based on analog processing to obtain the target low-definition video segment.

[0044] S130. Input the target low-quality video clip into the pre-built first network model to obtain the model output, and train the first network model according to the model output and the corresponding target high-definition video clip until the preset first model training termination condition is met to obtain the target image quality restoration model.

[0045] The first network model can be a network model used for image restoration, which can be pre-built by relevant technicians or an existing image restoration model. For example, the first network model can be an RLSP (Recurrent Latent Space Propagation) network model.

[0046] For example, a target low-quality video clip can be input into a pre-built first network model to obtain the model output. The model output can be a predicted high-definition video clip. The first network model is trained based on the predicted high-definition video clip and the target high-definition video clip until a preset first model training termination condition is met, resulting in a target image quality restoration model. The first model training termination condition can be preset by relevant technical personnel. For example, the first model training termination condition can be reaching a set threshold for the number of model iterations. For example, the model iteration threshold could be 100 iterations.

[0047] It should be noted that the target high-definition video clip can be the ground truth data of the target low-quality video clip. During model training, the target high-definition video clip serves as the label data for the target low-quality video clip and participates in model training to obtain the predicted high-definition video clip in the model output. To further improve the accuracy of the trained target image quality restoration model, model training can be terminated only when the model's output results are sufficiently accurate.

[0048] In one optional embodiment, the first network model is trained based on the model output and the corresponding target high-definition video segment until a preset first model training termination condition is met to obtain a target image quality restoration model. This includes: determining a first loss value and a second loss value based on the model output of the first network model and the corresponding target high-definition video segment; determining whether the first network model meets the preset first model training termination condition based on the first loss value and the second loss value; if so, the first network model after the model training is completed is used as the target image quality restoration model.

[0049] The first loss value can be the LI loss value, and the second loss value can be the perceptual loss value. The first model training termination condition can be that the first and second loss values ​​tend to stabilize or reach a preset loss threshold, in which case model training can be terminated.

[0050] For example, based on the predicted high-definition video clip and the target high-definition video clip output by the first network model, a first loss value and a second loss value are determined based on a preset loss function. If both the first loss value and the second loss value tend to stabilize or reach a preset loss threshold, the model training is terminated, and the target image quality restoration model is obtained; if the first loss value or the second loss value does not tend to stabilize, or neither reaches the preset loss threshold, the first network model continues to be trained until the first model training termination condition is met, and the target image quality restoration model is obtained.

[0051] It should be noted that, in order to fully extract the image features of video images during model training, the internal structure of the RLSP cell module in the RLSP model can be optimized and improved based on the RLSP model. A multi-scale design can be adopted to make full use of feature fusion at different scales, thereby improving the image quality restoration effect of video images.

[0052] In an optional embodiment, the first network model includes a multi-scale recurrent network; the multi-scale recurrent network includes multiple recurrent branches, each with a different branch scale; and each recurrent branch is equipped with at least one residual block for image information feature extraction.

[0053] The multi-scale recurrent network can be a network with partial structural improvements to the RLSP cell module in the RLSP model. The multiple recurrent branches correspond to different scales; for example, it can include four branches, each assigned one of four feature extraction scales, such as ×1, ×0.5, ×0.25, and ×0.125.

[0054] like Figure 1B The diagram shows the structure of a multi-scale recurrent network (RCN) for a first network model. The RCN comprises 15 residual blocks (ResBlk) assigned four scales: ×1, ×0.5, ×0.25, and ×0.125. Upon receiving the output of the feature fusion from the fusion layer, the ResBlk blocks are downsampled at each of the four scales. The extracted features are then upsampled back to their initial size based on the corresponding scale, thus achieving feature fusion at different scales and resulting in more diverse extracted features. A unidirectional recurrent structure is used, where each time step, the recurrent block can only access the hidden state of the previous time step and the output of the previous frame. This simple and efficient structure offers greater practicality, lower computational overhead, and is more suitable for super-resolution restoration tasks in animated videos.

[0055] The detailed information in animated videos is typically related to elements such as lines, outlines, and flat color blocks, which differ significantly from real videos. Appropriate resolution adjustment does not affect the preservation of detail and can reduce image artifacts, helping to maintain the naturalness and improve the quality of animated videos during super-resolution restoration. Multi-scale design can fuse features from different scales, thereby better restoring the details and structure of animated videos. Recurrent network structures can effectively utilize temporal information and multi-scale feature fusion to improve super-resolution restoration performance.

[0056] The technical solution of this invention identifies a target high-definition video segment associated with a target restoration task, degrades the target high-definition video segment to obtain a target low-definition video segment corresponding to the target high-definition video segment, inputs the target low-definition video segment into a pre-constructed first network model to obtain model output, and trains the first network model based on the model output and the corresponding target high-definition video segment until a preset first model training termination condition is met, thus obtaining a target image quality restoration model. This achieves accurate training of the first network model, resulting in higher model accuracy for the trained target image quality restoration model, thereby improving the restoration effect and quality of video image quality restoration using the target image quality restoration model.

[0057] Example 2

[0058] Figure 2 This is a flowchart of a method for training an image quality restoration model according to Embodiment 2 of the present invention. This embodiment is an optimization and improvement based on the above technical solutions.

[0059] Furthermore, the step "determining the target high-definition video segment associated with the target repair task" is refined to "acquiring high-definition animation video data of the same video type as the video to be repaired in the target repair task; extracting image frames from the high-definition animation video data according to a preset number of image frames to obtain candidate high-definition video segments; and selecting the target high-definition video segment from the candidate high-definition video segments according to the target repair task." This improves the method for determining the target high-definition video segment. It should be noted that parts not described in detail in this embodiment of the invention can be found in the descriptions of other embodiments.

[0060] like Figure 2 As shown, the method includes the following specific steps:

[0061] S210. Obtain high-definition video data of the same video type as the video to be repaired in the target repair task.

[0062] For example, high-definition animated video data of the same video type as the video to be repaired in the target repair task can be obtained. For instance, if the video to be repaired in the target repair task is an animated video, then high-definition animated video data can be obtained.

[0063] S220. Based on the preset number of image frames, extract image frames from the high-definition video data to obtain candidate high-definition video segments.

[0064] The preset number of image frames can be set by relevant technical personnel according to actual needs. For example, the preset number of image frames can be 100 frames.

[0065] For example, image frames can be extracted from high-definition video data according to a preset number of image frames to obtain candidate high-definition video segments. It should be noted that the process of image frame extraction involves extracting consecutive image frames from the high-definition video data, and at least one candidate high-definition video segment can be extracted from the same high-definition video data.

[0066] S230. Based on the target repair task, select the target high-definition video clip from the candidate high-definition video clips.

[0067] It should be noted that the selection requirements for target high-definition video clips differ depending on the target restoration task. For example, for target restoration tasks with rich image content, the requirements for transition frames and image textures are higher, while for target restoration tasks with simple image content, the requirements for transition frames and image textures are lower. Therefore, the selection method for target high-definition video clips differs for different restoration tasks.

[0068] In one optional embodiment, selecting a target high-definition video segment from candidate high-definition video segments according to the target restoration task includes: performing transition frame detection on the candidate high-definition video segments to obtain transition frame detection results; and determining the target gradient value of the candidate high-definition video segments based on the image gradient values ​​corresponding to the image frames of the candidate high-definition video segments; and selecting the target high-definition video segment from the candidate high-definition video segments based on the transition frame detection results and the target gradient value, and based on the task type of the target restoration task.

[0069] For example, for tasks requiring transition frame detection, to avoid video clips with transition frames appearing in the sample training set, transition frame detection can be performed on candidate high-definition video clips to obtain transition frame detection results. These results can include whether or not a video clip contains transition frames.

[0070] Image gradient values ​​are used to characterize the richness of image texture. The richer the image texture, the higher the gradient value; the simpler the image texture, the lower the gradient value.

[0071] For example, the image gradient value corresponding to each image frame in the candidate high-definition video band can be determined, and the average value of the image gradient values ​​of each image frame can be used as the target gradient value of the candidate high-definition video segment. The larger the target gradient value, the richer the texture features of the candidate high-definition video segment; the smaller the target gradient value, the simpler the texture features of the candidate high-definition video segment.

[0072] The task type can be determined based on the video to be repaired in the target repair task. If the video to be repaired has rich image texture, the task type needs to include the determination of the target gradient value; if the video to be repaired has simple image texture, the task type does not need to include the determination of the target gradient value. The task type may also include whether transition frames exist.

[0073] For example, if it is determined that a high-definition video segment with rich texture and no transition frames is needed based on the task type of the target repair task, then the candidate high-definition video segment with no transition frames and a large target gradient value is selected from the candidate high-definition video segments as the target high-definition video segment.

[0074] Optionally, for candidate high-definition video clips with transition frames, if there is a need for transition frames, the transition frames in the candidate high-definition video clips can be removed to obtain the target high-definition video clip without transition frames.

[0075] This optional embodiment selects the target high-definition video segment from the candidate high-definition video segments based on the transition frame detection results and target gradient values ​​of the candidate high-definition video segments and the task type of the target restoration task. This achieves targeted selection of the target high-definition video segment, fully considers the task requirements of the target restoration task during the selection process, and combines the transition frame results and target gradient values ​​to achieve accurate selection of the target high-definition video segment.

[0076] Optionally, when using low-quality video clips and high-quality video clips of the target as the training set, transition frames and target gradient values ​​can be used as the sample label data in the sample set.

[0077] S240. Degrade the target high-definition video clip to obtain the target low-definition video clip corresponding to the target high-definition video clip.

[0078] S250. Input the target low-quality video clip into the pre-built first network model to obtain the model output, and train the first network model according to the model output and the corresponding target high-definition video clip until the preset first model training termination condition is met to obtain the target image quality restoration model.

[0079] The technical solution of this embodiment acquires high-definition animated video data of the same video type as the video to be repaired in the target repair task. Based on a preset number of image frames, image frames are extracted from the high-definition animated video data to obtain candidate high-definition video segments. According to the target repair task, the target high-definition video segment is selected from the candidate high-definition video segments, thereby achieving targeted selection of target high-definition video segments. This improves the accuracy of constructing the sample training set and thus improves the training accuracy of the target image quality repair model.

[0080] Example 3

[0081] Figure 3A This is a flowchart of a method for training an image quality restoration model according to Embodiment 3 of the present invention. This embodiment is an optimization and improvement based on the above technical solutions.

[0082] Furthermore, the step "downgrading the target high-definition video segment to obtain the target low-definition video segment corresponding to the target high-definition video segment" is refined to "inputting the target high-definition video segment into a pre-trained target downgrading model to obtain a target reference downgraded video segment; based on the target reference downgraded video segment and a preset downgrading method, obtaining the target low-definition video segment corresponding to the target high-definition video segment." This improves the generation method of the target low-definition video segment. It should be noted that parts not described in detail in this embodiment of the invention can be found in the descriptions of other embodiments.

[0083] like Figure 3A As shown, the method includes the following specific steps:

[0084] S310. Identify the target high-definition video segment associated with the target repair task.

[0085] S320. Input the target high-definition video clip into the pre-trained target degradation model to obtain the target reference degradation video clip.

[0086] The target degradation model can be pre-trained by relevant technical personnel, or it can be an existing video degradation model, or it can be an improved version of an existing video degradation model, which is then used to train the first network model.

[0087] For example, the target high-definition video clip is input into a pre-trained target degradation model to obtain the model output of the target degradation model, and the model output is used as the target reference degradation video clip.

[0088] It should be noted that, in order to further improve the quality of the target reference degraded video clips, the target degrade model can also be obtained in the following way, thereby improving the accuracy of the model output results by improving the model's precision.

[0089] In one optional embodiment, the target degradation model is trained as follows: acquiring reference low-quality video data and extracting image frames from the reference low-quality video data to obtain reference low-quality video segments; inputting the reference low-quality video segments into the target image quality restoration model to obtain reference high-quality video segments corresponding to the reference low-quality video segments; inputting the reference high-quality video segments into a pre-built second network model to obtain model output, and training the first network model based on the model output and the corresponding reference low-quality video segments until the preset second model training termination condition is met to obtain the target degradation model.

[0090] The acquired low-quality reference video data must possess video diversity, specifically including different degradation features such as blurring, noise, and compression, in order to capture different types of degradation features.

[0091] For example, image frames can be extracted from reference low-quality video data based on a preset number of image frames to obtain reference low-quality video clips. These reference low-quality video clips are then input into the target image quality restoration model to obtain corresponding reference high-definition video clips. It is understood that, to obtain different degrees of degradation effects, the reference low-quality video clips can be adjusted at different resolutions, such as 2x, 4x, and 8x. From the reference high-definition video clips output by the model after these different resolution adjustments, the output result that most closely matches the required video high-definition level for the target restoration task is selected as the reference high-definition video clip.

[0092] Specifically, based on the low-quality video clips output by the model and the reference low-quality video clips, the LI loss value, the perceptual loss value, and the GAN (Generative Adversarial Network) loss value can be obtained.

[0093] The first model training termination condition can be that the L1 loss value, perceptual loss value, and GAN loss value tend to stabilize or reach a pre-set loss threshold, in which case the model training can be terminated.

[0094] For example, based on the low-quality video clips output by the second network model and reference low-quality video clips, and using a preset loss function, the L1 loss, perceptual loss, and GAN loss values ​​are determined. If the L1 loss, perceptual loss, and GAN loss values ​​all tend to stabilize or reach a preset loss threshold, model training is terminated, and the target degraded model is obtained. If the L1 loss, perceptual loss, or GAN loss values ​​do not tend to stabilize, or do not reach the preset loss threshold, model training of the second network model continues until the second model training termination condition is met, and the target degraded model is obtained.

[0095] The second network model can be an existing network model or a network model pre-constructed by relevant technical personnel. For example... Figure 3B The diagram shows the model structure of a second network model. This second network model consists of a pixel unshuffle layer and four 3×3 convolutional layers, with activation functions used between the convolutional layers. The convolutional layers are used for feature extraction; the pixel unshuffle layer is used to convert high-resolution images into low-resolution images.

[0096] S330. Based on the target reference degraded video clip, and using a preset degrade processing method, obtain the target low-quality video clip corresponding to the target high-definition video clip.

[0097] The degradation processing method may include at least one of the following: video compression processing, image compression processing, downsampling processing, anti-aliasing processing, noise addition processing, and blurring processing.

[0098] For example, a secondary degradation process can be performed on the target reference degraded video clip to obtain the target low-quality video clip.

[0099] In one optional embodiment, the target reference degraded video segment can be downsampled to obtain a first reference degraded video segment; the downsampled reference degraded video segment can be input again into the target degraded model to obtain a second reference degraded video segment; the second reference degraded video segment can be downsampled again to obtain a third reference degraded video segment; the third reference degraded video segment can be processed to add jagged edges to obtain a fourth reference degraded video segment; the fourth reference degraded video segment can be processed to add noise to obtain a fifth reference degraded video segment; the fifth reference degraded video segment can be blurred to obtain a sixth reference degraded video segment; and the sixth reference degraded video segment can be compressed to obtain the target low-quality video segment.

[0100] S340. Input the target low-quality video clip into the pre-built first network model to obtain the model output, and train the first network model according to the model output and the corresponding target high-definition video clip until the preset first model training termination condition is met to obtain the target image quality restoration model.

[0101] This embodiment's technical solution inputs a target high-definition video clip into a pre-trained target degradation model to obtain a target reference degradation video clip. Based on the target reference degradation video clip and a preset degradation processing method, a target low-quality video clip corresponding to the target high-definition video clip is obtained. By introducing reference high-definition video data, it is equivalent to introducing pseudo-high-quality sample data. During the training process, the network is guided to generate low-quality video data that better matches the actual degradation. This can better simulate the degradation of actual low-quality videos, achieving a more realistic effect where the sample training set is closer to real low-quality video data. This improves the target image quality restoration model's ability to handle various quality impairments.

[0102] Example 4

[0103] Figure 4 This is a schematic diagram of a picture quality restoration model training device provided in Embodiment 4 of the present invention. The picture quality restoration model training device provided in this embodiment of the present invention is applicable to the situation of restoring the picture quality of low-quality videos. This picture quality restoration model training device can be implemented in hardware and / or software, such as... Figure 4 As shown, the device specifically includes: a high-definition segment determination module 401, a low-quality segment determination module 402, and a target repair model determination module 403.

[0104] in,

[0105] The high-definition segment determination module 401 is used to determine the target high-definition video segment associated with the target repair task;

[0106] The low-quality segment determination module 402 is used to perform degradation processing on the target high-definition video segment to obtain the target low-quality video segment corresponding to the target high-definition video segment.

[0107] The target restoration model determination module 403 is used to input the target low-quality video clip into a pre-constructed first network model to obtain the model output, and to train the first network model according to the model output and the corresponding target high-definition video clip until the preset first model training termination condition is met, so as to obtain the target image quality restoration model.

[0108] The technical solution of this invention identifies a target high-definition video segment associated with a target restoration task, degrades the target high-definition video segment to obtain a target low-definition video segment corresponding to the target high-definition video segment, inputs the target low-definition video segment into a pre-constructed first network model to obtain model output, and trains the first network model based on the model output and the corresponding target high-definition video segment until a preset first model training termination condition is met, thus obtaining a target image quality restoration model. This achieves accurate training of the first network model, resulting in higher model accuracy for the trained target image quality restoration model, thereby improving the restoration effect and quality of video image quality restoration using the target image quality restoration model.

[0109] Optionally, the high-definition segment determination module 401 includes:

[0110] The high-definition video data acquisition unit is used to acquire high-definition video data of the same type as the video to be repaired in the target repair task.

[0111] The candidate high-definition segment extraction unit is used to extract image frames from the high-definition video data according to a preset number of image frames to obtain candidate high-definition video segments;

[0112] The high-definition segment determination unit is used to select a target high-definition video segment from the candidate high-definition video segments according to the target repair task.

[0113] Optionally, the high-definition segment determination unit includes:

[0114] The detection result determination subunit is used to perform transition frame detection on the candidate high-definition video segments to obtain transition frame detection results; and,

[0115] The gradient value determination subunit is used to determine the target gradient value of the candidate high-definition video segment based on the image gradient value corresponding to the image frame of the candidate high-definition video segment.

[0116] The high-definition segment determination subunit is used to select the target high-definition video segment from the candidate high-definition video segments based on the transition frame detection results and target gradient values ​​of the candidate high-definition video segments and the task type of the target restoration task.

[0117] Optionally, the low-quality fragment determination module 402 includes:

[0118] A degradation treatment method acquisition unit is used to acquire at least one degradation treatment method;

[0119] A combination method determination unit is used to combine the various degradation treatment methods to obtain at least one combined degradation treatment method.

[0120] The first low-quality segment determination unit is used to perform degradation processing on the target high-definition video according to the combined degradation processing method to obtain the target low-quality video segment corresponding to the target high-definition video segment.

[0121] The degradation processing method includes at least one of video compression processing, image compression processing, downsampling processing, anti-aliasing processing, noise addition processing, and blurring processing.

[0122] Optionally, the low-quality fragment determination module 402 includes:

[0123] The reference degraded segment determination unit is used to input the target high-definition video segment into a pre-trained target degraded model to obtain a target reference degraded video segment;

[0124] The second low-quality segment determination unit is used to obtain the target low-quality video segment corresponding to the target high-definition video segment based on the target reference degraded video segment and a preset degrade processing method.

[0125] Optionally, the target degradation model is trained in the following manner:

[0126] Obtain reference low-quality video data, and extract image frames from the reference low-quality video data to obtain reference low-quality video clips;

[0127] The reference low-quality video clip is input into the target image quality restoration model to obtain the reference high-quality video clip corresponding to the reference low-quality video clip;

[0128] The reference high-definition video clip is input into a pre-built second network model to obtain the model output. The first network model is then trained based on the model output and the corresponding reference low-quality video clip until the preset second model training termination condition is met, thus obtaining the target degraded model.

[0129] Optionally, the first network model includes a multi-scale recurrent network; the multi-scale recurrent network includes multiple recurrent branches, each recurrent branch having a different branch scale; and each recurrent branch is equipped with at least one residual block for extracting image information features.

[0130] Optionally, the target repair model determination module 403 includes:

[0131] The loss value determination unit is used to determine a first loss value and a second loss value based on the model output of the first network model and the corresponding target high-definition video clip;

[0132] The termination condition determination unit is used to determine whether the first network model meets the preset first model training termination condition based on the first loss value and the second loss value.

[0133] The target restoration model determination unit is used to determine the first network model after model training is completed as the target image quality restoration model if the preset first model training termination condition is met.

[0134] The image quality restoration model training device provided in this embodiment of the invention can execute the image quality restoration model training method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0135] Example 5

[0136] Figure 5 A schematic diagram of an electronic device 50 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0137] like Figure 5 As shown, the electronic device 50 includes at least one processor 51 and a memory, such as a read-only memory (ROM) 52 and a random access memory (RAM) 53, communicatively connected to the at least one processor 51. The memory stores computer programs executable by the at least one processor. The processor 51 can perform various appropriate actions and processes based on the computer program stored in the ROM 52 or loaded into the RAM 53 from storage unit 58. The RAM 53 can also store various programs and data required for the operation of the electronic device 50. The processor 51, ROM 52, and RAM 53 are interconnected via a bus 54. An input / output (I / O) interface 55 is also connected to the bus 54.

[0138] Multiple components in electronic device 50 are connected to I / O interface 55, including: input unit 56, such as keyboard, mouse, etc.; output unit 57, such as various types of monitors, speakers, etc.; storage unit 58, such as disk, optical disk, etc.; and communication unit 59, such as network card, modem, wireless transceiver, etc. Communication unit 59 allows electronic device 50 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0139] Processor 51 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 51 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 51 performs the various methods and processes described above, such as image restoration model training methods.

[0140] In some embodiments, the image restoration model training method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 58. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 50 via ROM 52 and / or communication unit 59. When the computer program is loaded into RAM 53 and executed by processor 51, one or more steps of the image restoration model training method described above may be performed. Alternatively, in other embodiments, processor 51 may be configured to perform the image restoration model training method by any other suitable means (e.g., by means of firmware).

[0141] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0142] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0143] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0144] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0145] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0146] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0147] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0148] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for training an image quality restoration model, characterized in that, include: Acquire high-definition video data of the same video type as the video to be repaired in the target repair task; The video type of the video to be repaired is an animated video; Based on a preset number of image frames, image frames are extracted from the high-definition video data to obtain candidate high-definition video segments; Transition frame detection is performed on the candidate high-definition video segments to obtain the transition frame detection results; as well as, The target gradient value of the candidate high-definition video segment is determined based on the image gradient value corresponding to the image frame of the candidate high-definition video segment. Based on the transition frame detection results and target gradient values ​​of the candidate high-definition video segments, and according to the task type of the target restoration task, the target high-definition video segment is selected from the candidate high-definition video segments. Different target restoration tasks have different requirements for the selection of high-definition video clips of the target; The target high-definition video clip is degraded to obtain a target low-definition video clip corresponding to the target high-definition video clip. The target low-quality video clip is input into a pre-built first network model to obtain the model output. The first network model is then trained based on the model output and the corresponding target high-definition video clip until a preset first model training termination condition is met, thus obtaining a target image quality restoration model. The first network model includes a multi-scale recurrent network. The multi-scale recurrent network includes multiple recurrent branches, each with a different branch scale. Each of the loop branches is equipped with at least one residual block for image information feature extraction.

2. The method according to claim 1, characterized in that, The process of degrading the target high-definition video segment to obtain a corresponding target low-definition video segment includes: Obtain at least one method of deterioration treatment; By combining the aforementioned degradation treatment methods, at least one combined degradation treatment method can be obtained; According to the combined degradation processing method, the target high-definition video is degraded to obtain the target low-definition video segment corresponding to the target high-definition video segment; The degradation processing method includes at least one of video compression processing, image compression processing, downsampling processing, anti-aliasing processing, noise addition processing, and blurring processing.

3. The method according to claim 1, characterized in that, The process of degrading the target high-definition video segment to obtain a corresponding target low-definition video segment includes: The target high-definition video clip is input into the pre-trained target degradation model to obtain the target reference degradation video clip; Based on the target reference degraded video clip, and using a preset degrade processing method, a target low-quality video clip corresponding to the target high-definition video clip is obtained.

4. The method according to claim 3, characterized in that, The target degradation model is trained in the following manner: Obtain reference low-quality video data, and extract image frames from the reference low-quality video data to obtain reference low-quality video clips; The reference low-quality video clip is input into the target image quality restoration model to obtain the reference high-quality video clip corresponding to the reference low-quality video clip; The reference high-definition video clip is input into a pre-built second network model to obtain the model output. The model is then trained based on the model output and the corresponding reference low-quality video clip until the preset second model training termination condition is met, thus obtaining the target degraded model.

5. The method according to any one of claims 1-4, characterized in that, The step of training the first network model based on the model output and the corresponding target high-definition video clip until a preset first model training termination condition is met, to obtain the target image quality restoration model, includes: Based on the model output of the first network model and the corresponding target high-definition video clip, determine the first loss value and the second loss value; Based on the first loss value and the second loss value, determine whether the first network model meets the preset first model training termination condition; If so, the first network model after model training is completed will be used as the target image quality restoration model.

6. A training device for an image quality restoration model, characterized in that, include: The high-definition segment identification module is used to identify target high-definition video segments associated with the target repair task; The low-quality segment determination module is used to degrade the target high-definition video segment to obtain the target low-quality video segment corresponding to the target high-definition video segment. The target restoration model determination module is used to input the target low-quality video clip into a pre-constructed first network model to obtain the model output, and to train the first network model according to the model output and the corresponding target high-definition video clip until a preset first model training termination condition is met to obtain a target image quality restoration model; the first network model includes a multi-scale recurrent network; the multi-scale recurrent network includes multiple recurrent branches, and the branch scale of each recurrent branch is different; Each of the loop branches is equipped with at least one residual block for image information feature extraction; The high-definition segment determination module includes: The high-definition video data acquisition unit is used to acquire high-definition video data of the same video type as the video to be repaired in the target repair task; the video type of the video to be repaired is an animation video. The candidate high-definition segment extraction unit is used to extract image frames from the high-definition video data according to a preset number of image frames to obtain candidate high-definition video segments; The high-definition video segment determination unit is used to select a target high-definition video segment from the candidate high-definition video segments according to the target restoration task; different target restoration tasks have different requirements for the selection of the target high-definition video segment; The high-definition segment determination unit includes: The detection result determination subunit is used to perform transition frame detection on the candidate high-definition video segments to obtain transition frame detection results; and, The gradient value determination subunit is used to determine the target gradient value of the candidate high-definition video segment based on the image gradient value corresponding to the image frame of the candidate high-definition video segment. The high-definition segment determination subunit is used to select the target high-definition video segment from the candidate high-definition video segments based on the transition frame detection results and target gradient values ​​of the candidate high-definition video segments and the task type of the target restoration task.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the image quality restoration model training method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the image quality restoration model training method according to any one of claims 1-5.

Images