Training methods and devices for video restoration models, electronic devices, storage media and program products

Abstract

This disclosure relates to a training method, apparatus, electronic device, storage medium, and program product for a video restoration model. The training method for the video restoration model includes: training an image restoration model based on a first training sample set to obtain an initial video restoration model, wherein the first training sample set includes multiple training videos; and training the initial video restoration model based on a second training sample set to obtain a final video restoration model, wherein the second training sample set includes multiple training images and multiple training videos.

Description

Technical Field

[0001] This disclosure relates to the field of video processing, and more particularly to a training method and apparatus for a video restoration model, an electronic device, a storage medium, and a program product. Background Technology

[0002] With the iterative updates of deep learning models, large models have gradually emerged due to their superior performance. Examples include large language models, image processing models, and generative models, all of which have found applications in daily life. Generative image inpainting models excel in inpainting tasks, offering significantly more detail than traditional models. However, due to their powerful texture generation capabilities, when applied to video inpainting tasks, the restored video lacks inter-frame coherence. This manifests as inconsistent texture orientations for corresponding elements in each frame, resulting in flickering.

[0003] Therefore, for a video restoration model extended from an image restoration model, it is necessary to strengthen the model's ability to control temporal information. However, current methods only consider temporal continuity, which often leads to the model "lazy" and even to simply deblur frames to forge a temporal continuity. As a result, the extended video restoration model cannot restore videos that meet user needs. Summary of the Invention

[0004] This disclosure provides a training method and apparatus for a video restoration model, an electronic device, a storage medium, and a program product, to at least solve the problem that video restoration models derived from image restoration models in related technologies cannot restore videos that meet user needs.

[0005] According to a first aspect of the present disclosure, a method for training a video restoration model is provided, comprising: training an image restoration model based on a first training sample set to obtain an initial video restoration model, wherein the first training sample set includes multiple training videos; and training the initial video restoration model based on a second training sample set to obtain a final video restoration model, wherein the second training sample set includes multiple training images and multiple training videos.

[0006] Optionally, the multiple training videos and multiple training images in the second training sample set cover a variety of content and a variety of image qualities. Furthermore, the number of training videos and training images corresponding to each type of content is uniform in the second training sample set, and each type of image quality satisfies the preset high image quality conditions.

[0007] Optionally, the training videos in the second training sample set may also include predetermined videos, wherein the optical flow amplitude of the predetermined videos meets preset high motion conditions and the image quality meets high image quality conditions.

[0008] Optionally, the predetermined video is obtained as follows: videos that meet the high-quality conditions are obtained from the second training sample set; frames are extracted from each video in the video at random frame extraction intervals; for each video in the video, the predetermined video corresponding to the current video is obtained based on the frame extraction results of the current video.

[0009] Optionally, the training videos in the first training sample set cover a variety of content, image quality, and optical flow amplitude.

[0010] Optionally, the image quality of multiple training videos and multiple training images in the second training sample set is higher than that of multiple training videos in the first training sample set.

[0011] Optionally, the initial video restoration model is trained based on the second training sample set to obtain the final video restoration model, including: for each training sample in the second training sample set, the following processing is performed: compressing the current training sample to obtain a first sample; decompressing the first sample to obtain a second sample; performing degradation processing on the second sample to obtain a degradation sample corresponding to the current training sample; and training the initial video restoration model based on each training sample and the corresponding degradation sample in the second training sample set to obtain the final video restoration model.

[0012] Optionally, based on each training sample in the second training sample set and the corresponding degraded sample, the initial video restoration model is trained to obtain the final video restoration model, including: for each training sample in the second training sample set, the following processing is performed: inputting the degraded sample corresponding to the current training sample into the initial video restoration model to obtain the restoration sample of the degraded sample; in response to the current training sample being a training image, determining a first loss between the training image and the corresponding restoration image; in response to the current training sample being a training video, determining a second loss between the training video and the corresponding restoration video; and adjusting the parameters of the initial video restoration model based on the first loss and the second loss to obtain the final video restoration model.

[0013] According to a second aspect of the present disclosure, a training apparatus for a video restoration model is provided, comprising: a first training unit configured to train an image restoration model based on a first training sample set to obtain an initial video restoration model, wherein the first training sample set includes multiple training videos; and a second training unit configured to train the initial video restoration model based on a second training sample set to obtain a final video restoration model, wherein the second training sample set includes multiple training images and multiple training videos.

[0014] Optionally, the multiple training videos and multiple training images in the second training sample set cover a variety of content and a variety of image qualities. Furthermore, the number of training videos and training images corresponding to each type of content is uniform in the second training sample set, and each type of image quality satisfies the preset high image quality conditions.

[0015] Optionally, the training videos in the second training sample set may also include predetermined videos, wherein the optical flow amplitude of the predetermined videos meets preset high motion conditions and the image quality meets high image quality conditions.

[0016] Optionally, the aforementioned predetermined video is obtained as follows: videos whose image quality meets the high-quality condition are obtained from the second training sample set; frames are extracted from each video in the video at random frame extraction intervals; for each video in the video, the predetermined video corresponding to the current video is obtained based on the frame extraction result of the current video.

[0017] Optionally, the training videos in the first training sample set cover a variety of content, image quality, and optical flow amplitude.

[0018] Optionally, the image quality of multiple training videos and multiple training images in the second training sample set is higher than that of multiple training videos in the first training sample set.

[0019] Optionally, the second training unit is further configured to perform the following processing for each training sample in the second training sample set: compress the current training sample to obtain a first sample; decompress the first sample to obtain a second sample; perform degradation processing on the second sample to obtain a degradation sample corresponding to the current training sample; and train the initial video restoration model based on each training sample and the corresponding degradation sample in the second training sample set to obtain the final video restoration model.

[0020] Optionally, the second training unit is further configured to perform the following processing for each training sample in the second training sample set: inputting the degraded sample corresponding to the current training sample into the initial video restoration model to obtain the restoration sample of the degraded sample; determining a first loss between the training image and the corresponding restoration image in response to the current training sample being a training image; determining a second loss between the training video and the corresponding restoration video in response to the current training sample being a training video; and adjusting the parameters of the initial video restoration model based on the first loss and the second loss to obtain the final video restoration model.

[0021] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to perform a training method for any of the video restoration models described above.

[0022] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by at least one processor, causes at least one processor to perform a training method for any of the video restoration models described above.

[0023] According to a fifth aspect of the present disclosure, a computer program product is provided, including computer instructions that, when executed by a processor, implement a training method for any of the video restoration models described above.

[0024] The technical solutions provided by the embodiments of this disclosure bring at least the following beneficial effects: According to the training method, apparatus, electronic device, storage medium, and program product of the video restoration model disclosed herein, an initial video restoration model derived from an image restoration model is fine-tuned. Specifically, the initial video restoration model is trained using both training videos and training images to obtain a final video restoration model. This approach utilizes training images to enhance the spatial restoration and generation capabilities of the video restoration model, and training videos to enhance its temporal consistency capabilities. In other words, through this hybrid training method, both temporal and spatial capabilities are simultaneously improved, resulting in a better video restoration effect for the final model. Therefore, this disclosure solves the problem that video restoration models derived from image restoration models in related technologies cannot restore videos that meet user needs.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0027] Figure 1 This is a flowchart illustrating a training method for a video restoration model according to exemplary embodiments of the present disclosure; Figure 2 This is a schematic diagram comparing degradation paths according to exemplary embodiments of the present disclosure; Figure 3 This is a schematic diagram illustrating a training method for a generative video restoration model according to exemplary embodiments of the present disclosure; Figure 4 This is a block diagram illustrating a training apparatus for a video restoration model according to exemplary embodiments of the present disclosure; Figure 5This is a diagram illustrating a computing environment coupled to a user interface according to exemplary embodiments of the present disclosure. Detailed Implementation

[0028] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0029] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0030] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. Another example is "performing at least one of step one and step two", which means the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0031] This disclosure provides a training method for generative video restoration models, primarily aimed at enhancing the temporal consistency of the video restoration model while maintaining its spatial generation capabilities. Specifically, it utilizes both training videos and training images to perform hybrid training on an initial video restoration model derived from an image restoration model. This involves using training images to enhance the spatial restoration and generation capabilities of the video restoration model, and using training videos to strengthen its temporal consistency. This dual approach of enhancing temporal and spatial capabilities results in a better final video restoration performance. Furthermore, the multiple training videos used in this hybrid training method cover various content types and image qualities. The number of training videos for each content type is evenly distributed, and each image quality type meets preset high-quality conditions. Such training videos further improve the video restoration effect. Additionally, the image quality of the training data used to train the initial video restoration model can be higher than that used to train the image restoration model. This ensures low cost and high richness of the initial training data, allowing for secondary fine-tuning using high-quality training data to guarantee restoration capabilities. Furthermore, the training videos used in the hybrid training of this disclosure include high-quality and high-motion videos. These high-quality and high-motion videos can be obtained by extracting frames from the high-quality training videos, thereby enhancing the model's ability to control high-motion temporal information and achieving better restoration effects for videos with different motion characteristics. In addition, this disclosure adds a video encoding and decoding process to the original degradation method, thereby obtaining low-quality video data that is closer to real life, resulting in better restoration effects from the acquired model.

[0032] The following will describe in detail, with reference to the accompanying drawings, a training method and apparatus for a video restoration model according to exemplary embodiments of the present disclosure, an electronic device, a storage medium, and a program product.

[0033] Figure 1 This is a flowchart illustrating a training method for a video restoration model according to exemplary embodiments of the present disclosure, such as... Figure 1 As shown, the training method for the video restoration model includes the following steps: In step S101, the image restoration model is trained based on the first training sample set to obtain the initial video restoration model, wherein the first training sample set includes multiple training videos.

[0034] In step S102, the initial video restoration model is trained based on the second training sample set to obtain the final video restoration model. The second training sample set includes multiple training images and multiple training videos.

[0035] As an example, the initial video restoration model in step S101 can be extended from an image restoration model. Since the image restoration model is trained from a single frame image, obtaining the initial video restoration model from this image restoration model requires adapting the image restoration model to training data with temporal correlations, obtaining the temporal correlation patterns of multi-frame data, and thus possessing temporal consistency capabilities. Specifically, during the training process in step S101, for each training video, multiple consecutive frames can be randomly extracted from the training video and degraded to form data pairs of clean and degraded videos. Then, the clean video is used as the training target, and the degraded video is used as input data to be fed into the image restoration model to obtain the restored video. The parameters of the image restoration model are adjusted based on the loss between the restored video and the clean video.

[0036] For example, the initial video restoration model in step S101 above can be obtained by, but is not limited to, the following method: For each training video in the first training sample set, perform the following iterative processing: extract a predetermined number of time-continuous frames from the current training video to obtain a clean video corresponding to the current training video; perform degradation processing on the clean video to obtain a degraded video corresponding to the current training video; input the degraded video into the image restoration model to obtain a restored video of the degraded video; in response to the number of iterations reaching a first preset threshold, adjust the parameters of the image restoration model based on the loss between the clean video and the restored video to obtain the initial video restoration model.

[0037] According to an exemplary embodiment of this disclosure, the multiple training videos in the first training sample set in step S101 above cover various content, image quality, and optical flow amplitude. Through the training samples of this embodiment, the initial video restoration model can be better adapted to training data with temporal correlations, obtaining the temporal correlation patterns of multi-frame data, thereby possessing better temporal consistency and stability.

[0038] As an example, in the training phase of step S101, millions of training videos from diverse sources can be prepared. These diverse training videos can include, but are not limited to, content such as portraits, animals, plants, landscapes, and objects, and can also cover various image qualities and optical flow amplitudes.

[0039] As an example, the image quality of the aforementioned training videos can be characterized by resolution, Multi-scale Image Quality Transformer (MUSIQ), and Contrastive Learning-based Image Quality Assessment (CLITIQA), etc., and this disclosure does not limit the scope of the expression. Similarly, the aforementioned optical flow amplitude can be characterized by optical flow values, and this disclosure does not limit the scope of the expression.

[0040] It should be noted that the training phase in step S101 typically requires a large number of training steps and epochs to fully cycle through the entire first training sample set, resulting in a more stable temporal performance for the initial video inpainting model. Therefore, a generative initial video inpainting model trained on a large scale with diverse training videos usually exhibits good temporal consistency and stability.

[0041] However, since the initial video restoration model obtained in step S101 has generally poor overall video quality after restoration, it is necessary to train the initial video restoration model based on the second training sample set to obtain the final video restoration model. In the second training phase, the weights of the initial video restoration model obtained in the first training phase can be inherited, and fewer training steps can be performed on this basis.

[0042] For example, the final video restoration model in step S102 above can be obtained by, but is not limited to, the following method: For the second training sample set, perform the following iterative processing: extract a training sample from the second training sample set, wherein the training sample is a training video or a training image; perform degradation processing on the training sample to obtain the degradation sample corresponding to the training sample; input the degradation sample into the initial video restoration model to obtain the restored sample of the degradation sample; in response to the number of iterations reaching the second preset threshold, adjust the parameters of the initial video restoration model based on the loss between the training sample and the restored sample to obtain the final video restoration model.

[0043] According to an exemplary embodiment of this disclosure, the multiple training videos and multiple training images in the second training sample set in step S102 can cover multiple contents and multiple image qualities. Furthermore, the number of training videos and training images corresponding to each type of content is uniform in the second training sample set, and each image quality meets a preset high-quality condition. Through this embodiment, training the model using high-quality videos covering multiple contents can improve the video restoration effect.

[0044] As an example, the aforementioned training videos and training images may include, but are not limited to, content such as portraits, animals, plants, landscapes, and objects. Furthermore, the number of training data (training videos and training images) under various content categories may be uniform. For instance, assuming the second training sample set contains 100 training samples and includes the five content categories of portraits, animals, plants, landscapes, and objects, then the number of training samples for portrait content is 20, the number of training samples for animal content is 20, the number of training samples for plant content is 20, the number of training samples for landscape content is 20, and the number of training samples for object content is 20.

[0045] As an example, the aforementioned training videos and training images can also cover various high-quality aspects. The high-quality conditions can be set according to the representation method of image quality. For example, when image quality is represented by resolution, the aforementioned high-quality conditions can be that the resolution is higher than a preset value. This disclosure does not limit this. Moreover, the preset value can also be set as needed, and this disclosure does not limit this either.

[0046] It should be noted that the aforementioned multiple training images can be multiple images acquired individually, or they can be obtained by extracting frames from multiple training videos, such as extracting one frame from each training video. This disclosure does not limit this.

[0047] According to an exemplary embodiment of this disclosure, the image quality of multiple training videos and multiple training images in the second training sample set is higher than that of multiple training videos in the first training sample set. Through this embodiment, by fine-tuning the initial video restoration model using training samples of higher image quality, the restoration capability of the final video restoration model can be improved, thereby obtaining videos with higher restored image quality.

[0048] As an example, since high-quality training data is more difficult to obtain than low-quality training data, and the richness of high-quality training data is generally lower than that of low-quality training data, the image restoration model can be trained with abundant low-quality training data in the first training stage to obtain an initial video restoration model with strong generalization ability. Compared with the first training stage, the training data in the second training stage can use data with higher image quality and more uniform content distribution, thereby improving the restoration ability of the initial video restoration model, so that the trained video restoration model can restore videos or images with higher image quality.

[0049] According to an exemplary embodiment of this disclosure, the multiple training videos in the second training sample set in step S102 may further include predetermined videos, wherein the optical flow amplitude of the predetermined videos meets preset high motion conditions and the image quality meets high image quality conditions. Through this embodiment, by adding high-motion and high-quality videos to the second training sample set, the model's ability to control high-motion temporal information can be enhanced, and it has a better restoration effect on videos with different motion characteristics.

[0050] Specifically, since the second training phase aims to ensure high image quality, there are fewer high-motion videos. To increase the number of high-motion and high-quality videos, the final video restoration model can further enhance its ability to control high-motion temporal information, resulting in better restoration effects for videos with different motion characteristics.

[0051] As an example, the aforementioned high motion condition may be that the optical flow amplitude exceeds a certain threshold, which can be set as needed, and this disclosure does not limit it.

[0052] According to an exemplary embodiment of this disclosure, the predetermined video can be obtained, but is not limited to, the following methods: obtaining videos whose image quality meets the high-quality condition from a second training sample set; performing frame extraction on each video in the video at random frame extraction intervals; and for each video in the video, obtaining the predetermined video corresponding to the current video based on the frame extraction result of the current video. Through this embodiment, some high-motion and high-quality video data with inter-frame frames can be synthesized conveniently and quickly.

[0053] Specifically, since it is difficult to collect video data that is both high-motion and high-quality in reality, frames can be randomly extracted from high-quality training videos during the training process. For each training video, high-motion inter-frame video data is synthesized from the extracted frames, resulting in multiple predetermined videos. Introducing high-quality, high-motion video data into the second training sample set further enhances the model's ability to control high-motion temporal information, demonstrating good restoration effects for videos with different motion characteristics.

[0054] As an example, here we use random frame sampling intervals to sample frames from each video in the video. That is, in the second training sample set, one training video is sampled at a 1-frame interval, another training video at a 3-frame interval, and yet another training video at a 2-frame interval. In other words, the frame sampling interval for each training video is not fixed and is random.

[0055] According to an exemplary embodiment of this disclosure, step S102 above, which trains the initial video restoration model based on the second training sample set to obtain the final video restoration model, may include: for each training sample in the second training sample set, performing the following processing: compressing the current training sample to obtain a first sample; decompressing the first sample to obtain a second sample; performing degradation processing on the second sample to obtain a degradation sample corresponding to the current training sample; and training the initial video restoration model based on each training sample and the corresponding degradation sample in the second training sample set to obtain the final video restoration model. Through this embodiment, video encoding and result processes are added to the original degradation method, thereby obtaining low-quality video data that is closer to real life, resulting in better restoration performance of the obtained video restoration model.

[0056] Specifically, to enable restoration models to possess restoration capabilities, high-quality clean images or videos are typically degraded to obtain degraded images or videos. The restoration model then processes this degraded data, and its output is constrained to be consistent with the original high-quality clean data. In the training of video restoration models, the degradation path used in image restoration models is still followed, i.e., randomly performing blurring degradation, noise reduction degradation, scaling degradation, JPEG compression, etc., on single-frame data. However, this degradation path lacks degradation of the entire video or temporal correlation. Therefore, this embodiment improves upon the above degradation path by adding video encoding and decoding steps, namely… Figure 2 The added FFMPEG video compression degradation module, through methods such as... Figure 2 The degradation process shown in this embodiment of the present disclosure can obtain low-quality video data that is closer to real life, resulting in better model repair performance.

[0057] It should be noted that the FFMPEG video compression and degradation module is used for compressing and decompressing video. FFMPEG stands for "Fast Forward Moving Picture Experts Group," and it is a free, open-source tool that can be used to process audio and video files, supporting various operations such as encoding / decoding, transcoding, editing, and streaming media transmission.

[0058] According to an exemplary embodiment of this disclosure, training the initial video restoration model based on each training sample and corresponding degraded sample in the second training sample set to obtain the final video restoration model may include: for each training sample in the second training sample set, performing the following processing: inputting the degraded sample corresponding to the current training sample into the initial video restoration model to obtain a restoration sample for the degraded sample; determining a first loss between the training image and the corresponding restoration image in response to the current training sample being a training image; determining a second loss between the training video and the corresponding restoration video in response to the current training sample being a training video; and adjusting the parameters of the initial video restoration model based on the first loss and the second loss to obtain the final video restoration model. Through this embodiment, the losses of the image and the video are determined separately, and the parameters of the initial video restoration model are adjusted based on the total loss of both.

[0059] For ease of understanding of this disclosure, the following will be combined with... Figure 3 Provide a systematic explanation.

[0060] Figure 3 A training method for generative video inpainting models is demonstrated, such as... Figure 3 As shown, this training method mainly has two training phases: a pre-training phase (i.e., the first training phase in the above embodiment) and a high-quality fine-tuning phase (i.e., the second training phase in the above embodiment). During the pre-training phase, for the first training sample set (i.e. Figure 3 For each training video in a medium-to-large-scale video dataset, multiple consecutive frames can be randomly extracted from the training video and degraded to form a data pair of clean and degraded videos. Then, the clean video is used as the training target and the degraded video is used as the input data to the image inpainting model to obtain the restored video. The parameters of the image inpainting model are adjusted based on the loss between the restored video and the clean video to obtain the initial video inpainting model.

[0061] In the high-quality fine-tuning stage, to simultaneously control temporal consistency and spatial generation capabilities—that is, to train temporal consistency while also ensuring intra-frame inpainting (spatial generation) capabilities—this embodiment employs a hybrid training approach. Specifically, during training, samples are randomly selected from the second training sample set (i.e., Figure 3Training images or videos are extracted from medium-to-high quality video data and high-quality image data. When training images are extracted, they are degraded, and the model is trained using the degraded images and the training images, which enhances the model's spatial inpainting and generative capabilities. When training videos are extracted, they are degraded, and the model is trained using the degraded videos and the training videos, which enhances the model's temporal consistency capabilities. Through this hybrid training method, both temporal and spatial capabilities are addressed simultaneously, resulting in better video inpainting performance.

[0062] It should be noted that in this embodiment, the high-quality image data can be obtained by extracting frames from high-quality video data. For example, frames can be extracted from each video data in sequence with randomly increasing frame extraction intervals to obtain multiple high-quality image data. This disclosure does not limit this.

[0063] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0064] Figure 4 This is a block diagram illustrating a training apparatus for a video restoration model according to exemplary embodiments of the present disclosure. (Refer to...) Figure 4 The device includes a first training unit 40 and a second training unit 42.

[0065] The first training unit 40 is configured to train the image restoration model based on the first training sample set to obtain an initial video restoration model, wherein the first training sample set includes multiple training videos; the second training unit 42 is configured to train the initial video restoration model based on the second training sample set to obtain a final video restoration model, wherein the second training sample set includes multiple training images and multiple training videos.

[0066] According to an exemplary embodiment of this disclosure, the multiple training videos and multiple training images in the second training sample set cover multiple contents and multiple image qualities, and the number of training videos and training images corresponding to each content in the multiple contents is uniform in the second training sample set, and each image quality in the multiple image qualities meets the preset high image quality conditions.

[0067] According to an exemplary embodiment of this disclosure, the plurality of training videos in the second training sample set further include a predetermined video, wherein the optical flow amplitude of the predetermined video satisfies a preset high motion condition and the image quality satisfies a high image quality condition.

[0068] According to an exemplary embodiment of this disclosure, the predetermined video is obtained in the following manner: a video whose image quality meets the high-quality condition is obtained from the second training sample set; frames are extracted from each video in the video at random frame extraction intervals; for each video in the video, the predetermined video corresponding to the current video is obtained based on the frame extraction result of the current video.

[0069] According to an exemplary embodiment of this disclosure, the multiple training videos in the first training sample set cover a variety of content, image quality, and optical flow amplitude.

[0070] According to an exemplary embodiment of this disclosure, the image quality of multiple training videos and multiple training images in the second training sample set is higher than the image quality of multiple training videos in the first training sample set. According to an exemplary embodiment of this disclosure, the second training unit 42 is further configured to perform the following processing for each training sample in the second training sample set: compressing the current training sample to obtain a first sample; decompressing the first sample to obtain a second sample; performing degradation processing on the second sample to obtain a degraded sample corresponding to the current training sample; and training an initial video restoration model based on each training sample and the corresponding degraded sample in the second training sample set to obtain a final video restoration model.

[0071] According to an exemplary embodiment of this disclosure, the second training unit 42 is further configured to perform the following processing for each training sample in the second training sample set: inputting the degraded sample corresponding to the current training sample into the initial video restoration model to obtain the restoration sample of the degraded sample; determining a first loss between the training image and the corresponding restoration image in response to the current training sample being a training image; determining a second loss between the training video and the corresponding restoration video in response to the current training sample being a training video; and adjusting the parameters of the initial video restoration model based on the first loss and the second loss to obtain the final video restoration model.

[0072] Figure 5 A computing environment 510 coupled to a user interface 550 is shown. The computing environment 510 may be part of a data processing server. The computing environment 510 includes a processor 520, memory 530, and input / output (I / O) interface 540.

[0073] Processor 520 typically controls the overall operation of computing environment 510, such as operations associated with display, data acquisition, data communication, and image processing. Processor 520 may include one or more processors for executing instructions to perform all or some of the steps in the methods described above. Furthermore, processor 520 may include one or more modules that facilitate interaction between processor 520 and other components. The processor may be a central processing unit (CPU), microprocessor, microcontroller, graphics processing unit (GPU), etc.

[0074] Memory 530 is configured to store various types of data to support the operation of computing environment 510. Memory 530 may include predefined software 532. Examples of such data include instructions for any application or method operating on computing environment 510, video datasets, image data, etc. Memory 530 can be implemented using any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0075] I / O interface 540 provides an interface between processor 520 and peripheral interface modules (such as keyboard, click wheel, buttons, etc.). Buttons may include, but are not limited to, a home button, a start scan button, and a stop scan button. I / O interface 540 can be coupled to encoders and decoders.

[0076] In an embodiment, the computing environment 510 may be implemented by one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), FPGAs, GPUs, controllers, microcontrollers, microprocessors, or other electronic components for performing the methods described above.

[0077] According to embodiments of the present disclosure, an electronic device may be provided, the electronic device including at least one memory and at least one processor, the at least one memory storing a set of computer-executable instructions, wherein when the set of computer-executable instructions is executed by the at least one processor, a training method for a video restoration model according to embodiments of the present disclosure is executed.

[0078] As an example, the electronic device can be a PC, tablet, personal digital assistant, smartphone, or other device capable of executing the aforementioned set of instructions. Here, the electronic device 1000 is not necessarily a single electronic device, but can be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. The electronic device can also be part of an integrated control system or system manager, or can be configured to interconnect with a portable electronic device locally or remotely (e.g., via wireless transmission) through an interface.

[0079] In addition, electronic devices may include video displays (such as liquid crystal displays) and user interaction interfaces (such as keyboards, mice, touch input devices, etc.). All components of the electronic device may be interconnected via buses and / or networks.

[0080] According to embodiments of this disclosure, a computer-readable storage medium may also be provided, wherein when instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor causes to perform a training method for a video restoration model according to embodiments of this disclosure. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0081] According to embodiments of this disclosure, a computer program product is also provided, comprising, for example, a plurality of programs stored in a memory 530, which can be executed by a processor 520 in a computing environment 510 to perform the training method for the video restoration model described above. For example, the computer program product may include a non-transitory computer-readable storage medium.

[0082] Unless otherwise specifically stated, the order of steps in the method according to this disclosure is intended to be illustrative only, and the steps of the method according to this disclosure are not limited to the specific order described above, but may be changed according to actual circumstances. Furthermore, at least one step in the method according to this disclosure may be adjusted, combined, or omitted as needed.

[0083] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

[0084] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A training method for a video restoration model, characterized in that, include: Based on the first training sample set, the image restoration model is trained to obtain the initial video restoration model, wherein the first training sample set includes multiple training videos. The initial video restoration model is trained based on the second training sample set to obtain the final video restoration model. The second training sample set includes multiple training images and multiple training videos.

2. The training method as described in claim 1, characterized in that, The training videos and images in the second training sample set cover a variety of content and image quality. The number of training videos and images corresponding to each type of content is uniform in the second training sample set, and each type of image quality meets the preset high image quality conditions.

3. The training method as described in claim 2, characterized in that, The second training sample set also includes a predetermined video, wherein the optical flow amplitude of the predetermined video satisfies a preset high motion condition and the image quality satisfies the high image quality condition.

4. The training method as described in claim 3, characterized in that, The reserved video is obtained in the following way: Obtain videos whose image quality meets the high-quality conditions from the second training sample set; Each video in the video is sampled at random frame intervals. For each video in the video, a predetermined video corresponding to the current video is obtained based on the frame extraction result of the current video.

5. The training method as described in claim 1, characterized in that, The training videos in the first training sample set cover a variety of content, image quality, and optical flow amplitude.

6. The training method as described in claim 1, characterized in that, The image quality of multiple training videos and multiple training images in the second training sample set is higher than that of multiple training videos in the first training sample set.

7. The training method as described in claim 1, characterized in that, The step of training the initial video restoration model based on the second training sample set to obtain the final video restoration model includes: For each training sample in the second training sample set, the following processing is performed: Compress the current training samples to obtain the first sample; The first sample is decompressed to obtain the second sample; The second sample is degraded to obtain the degraded sample corresponding to the current training sample; The initial video restoration model is trained based on each training sample and the corresponding degradation sample in the second training sample set to obtain the final video restoration model.

8. The training method as described in claim 1, characterized in that, The step of training the initial video restoration model based on each training sample and the corresponding degraded sample in the second training sample set to obtain the final video restoration model includes: For each training sample in the second training sample set, the following processing is performed: Input the degraded sample corresponding to the current training sample into the initial video restoration model to obtain the restored sample of the degraded sample; In response to the current training sample being a training image, a first loss is determined between the training image and the corresponding repaired image; In response to the current training sample being a training video, a second loss is determined between the training video and the corresponding repaired video; Based on the first loss and the second loss, the parameters of the initial video restoration model are adjusted to obtain the final video restoration model.

9. A training device for a video restoration model, characterized in that, include: The first training unit is configured to train the image restoration model based on the first training sample set to obtain an initial video restoration model, wherein the first training sample set includes multiple training videos. The second training unit is configured to train the initial video restoration model based on the second training sample set to obtain the final video restoration model, wherein the second training sample set includes multiple training images and multiple training videos.

10. An electronic device, characterized in that, include: At least one processor; At least one memory that stores computer-executable instructions. The computer-executable instructions, when executed by the at least one processor, cause the at least one processor to perform the training method of the video restoration model as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor causes the at least one processor to perform the training method of the video restoration model as described in any one of claims 1 to 8.

12. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the training method of the video restoration model as described in any one of claims 1 to 8.

Images