Video super-resolution model training method and device

Abstract

The present disclosure provides a training method and device for a video super-resolution model. The video super-resolution model includes a reconstructor, a generator and a discriminator, and the training method includes: obtaining a video frame training sample including a plurality of video frames of first quality; performing denoising processing on the video frames of second quality by the reconstructor to obtain a first output image, wherein the video frames of second quality are obtained by encoding and decoding the video frames of first quality; calculating a first loss function; performing super-resolution processing on the first output image by the generator to obtain a second output image; calculating a second loss function; calculating a third loss function; and adjusting the model parameters of the reconstructor, the generator and the discriminator according to the first loss function, the second loss function and the third loss function respectively to train the video super-resolution model.

Description

Technical Field

[0001] The present disclosure relates to the field of video processing, and in particular to a training method and device for a video super-resolution model. Background Art

[0002] In the related technologies, the video super-resolution (SR) algorithm plays an important role in the video field. It can process low-quality, low-resolution videos into high-quality, high-resolution videos, improve video resolution, enhance video texture details, and improve the subjective quality of the video. At present, the super-resolution algorithm based on deep learning has become the mainstream super-resolution algorithm. Compared with the traditional super-resolution algorithm based on filtering and dictionary learning, the deep learning super-resolution algorithm can be driven by large-scale data and rely on the feature calculation of a large number of convolutional layers to improve the super-resolution ability of the algorithm.

[0003] Although some existing video super-resolution schemes can achieve super-resolution capabilities, when implementing super-resolution for low-resolution videos, existing schemes have problems such as excessive smoothing of the video after super-resolution processing, enhanced noise and erroneous texture, etc., resulting in a decrease in the quality of the super-resolution video. Summary of the invention

[0004] The present disclosure provides a video super-resolution model training method and device, as well as a video super-resolution method and device, to at least solve the problem of super-resolution results being overly smooth, having enhanced noise and erroneous textures in the above-mentioned related technologies, or may not solve any of the above-mentioned problems. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a training method for a video super-resolution model is provided, wherein the video super-resolution model includes a reconstructor, a generator and a discriminator, wherein the training method includes: obtaining video frame training samples for training the video super-resolution model, wherein the video frame training samples include multiple video frames of first quality; performing denoising processing on the video frames of second quality by the reconstructor to obtain a first output image, wherein the video frames of second quality are obtained by encoding and decoding the video frames of first quality; calculating a first loss function according to the video frames of first quality and the first output image; performing super-resolution processing on the first output image by the generator to obtain a second output image; calculating a second loss function according to the video frames of first quality, the second output image, and the result of the discriminator distinguishing the authenticity of the second output image and the video frames of first quality; calculating a third loss function according to the result of the discriminator distinguishing the authenticity of the second output image and the video frames of first quality; and adjusting the model parameters of the reconstructor, the generator and the discriminator according to the first loss function, the second loss function and the third loss function respectively to train the video super-resolution model.

[0006] Optionally, the training method may also include: processing the video frames of the second quality by the video super-resolution model to obtain a third output image; calculating a fourth loss function based on the result of the discriminator's authenticity judgment of the third output image and the video frames of the first quality; and further adjusting the model parameters of the reconstructor, generator and discriminator according to the fourth loss function to fine-tune the video super-resolution model.

[0007] Optionally, the step of calculating the second loss function may include: calculating an L1 loss function based on the second output image and the video frame of the first quality; calculating an adversarial loss function based on the result of the discriminator's true or false discrimination of the second output image and the video frame of the first quality; calculating a perceptual loss function based on the second output image and the video frame of the first quality; and calculating a weighted sum of the L1 loss function, the adversarial loss function and the perceptual loss function as the second loss function.

[0008] Optionally, the step of calculating the fourth loss function may include: calculating the L1 loss function based on the third output image and the video frame of the first quality; calculating the adversarial loss function based on the result of the discriminator's true or false discrimination of the third output image and the video frame of the first quality; calculating the perceptual loss function based on the third output image and the video frame of the first quality; and calculating the weighted sum of the L1 loss function, the adversarial loss function and the perceptual loss function as the fourth loss function.

[0009] Optionally, the step of obtaining a first output image may include: extracting depth features from a video frame of second quality by a reconstructor; processing the extracted depth features by multiple convolution modules in the reconstructor to obtain processed depth features; and converting the processed depth features to the same color space as the video frame of second quality by the reconstructor to obtain the first output image.

[0010] Optionally, the step of obtaining a second output image may include: extracting depth features from the first output image by a generator; processing the extracted depth features by multiple convolution modules in the generator to obtain processed depth features; and converting the processed depth features to the same color space as the first output image by the generator and performing upsampling to obtain the second output image.

[0011] Optionally, the step of obtaining a third output image may include: extracting depth features from the video frames of the second quality by the video super-resolution model; processing the extracted depth features by multiple convolution modules in the video super-resolution model to obtain processed depth features; and converting the processed depth features to the same color space as the video frames of the second quality by the video super-resolution model and performing upsampling processing to obtain the third output image.

[0012] Optionally, the multiple convolution modules included in the video super-resolution model can be connected to each other in series, wherein the step of processing the extracted depth features by the multiple convolution modules to obtain processed depth features may include: each convolution module performs the following processing: transforming the input depth features, and adding the transformed results to the input depth features to obtain output depth features, wherein the input depth features of the first convolution module are the extracted depth features, and the output depth features of the first convolution module are the input depth features of the next convolution module, and the input depth features of the last convolution module are the output depth features of the previous convolution module, and the output depth features of the last convolution module are the processed depth features.

[0013] Optionally, the step of processing the extracted depth features by the multiple convolution modules to obtain processed depth features may also include: at least one convolution module among the multiple convolution modules except the first convolution module also performs the following processing: adding the input depth features of at least one convolution module located before the current convolution module to the output depth features of the current convolution module to obtain the updated output depth features.

[0014] Optionally, the first loss function may be an L1 loss function.

[0015] Optionally, the third loss function may be an adversarial loss function.

[0016] Optionally, the denoising process may include: denoising and decompression artifacts, and the super-resolution process may include: supplementing texture details.

[0017] According to a second aspect of an embodiment of the present disclosure, a video super-resolution method is provided, comprising: a reconstructor in a video super-resolution model trained using the training method described above performs denoising processing on a reconstructed video frame to obtain a first output image; a generator in the video super-resolution model performs super-resolution processing on the first output image to obtain a second output image.

[0018] Optionally, the step of obtaining a first output image may include: extracting depth features from the video frame by a reconstructor; processing the extracted depth features by multiple convolution modules in the reconstructor to obtain processed depth features; and converting the processed depth features to the same color space as the video frame by the reconstructor to obtain the first output image.

[0019] Optionally, the step of obtaining a second output image may include: extracting depth features from the first output image by a generator; processing the extracted depth features by multiple convolution modules in the generator to obtain processed depth features; and converting the processed depth features to the same color space as the first output image by the generator and performing upsampling to obtain the second output image.

[0020] Optionally, the multiple convolution modules included in the video super-resolution model can be connected to each other in series, wherein the step of processing the extracted depth features by the multiple convolution modules to obtain processed depth features may include: each convolution module performs the following processing: transforming the input depth features, and adding the transformed results to the input depth features to obtain output depth features, wherein the input depth features of the first convolution module are the extracted depth features, and the output depth features of the first convolution module are the input depth features of the next convolution module, and the input depth features of the last convolution module are the output depth features of the previous convolution module, and the output depth features of the last convolution module are the processed depth features.

[0021] Optionally, the step of processing the extracted depth features by the multiple convolution modules to obtain processed depth features may also include: at least one convolution module among the multiple convolution modules except the first convolution module also performs the following processing: adding the input depth features of at least one convolution module located before the current convolution module to the output depth features of the current convolution module to obtain the updated output depth features.

[0022] Optionally, the denoising process may include: denoising and decompression artifacts, and the super-resolution process may include: supplementing texture details.

[0023] According to a third aspect of an embodiment of the present disclosure, a training device for a video super-resolution model is provided, wherein the video super-resolution model includes a reconstructor, a generator, and a discriminator, wherein the training device includes: a first acquisition module, configured to: obtain a first output image by performing denoising processing on a video frame of a second quality through the reconstructor, wherein the video frame of the second quality is obtained by encoding and decoding the video frame of the first quality, and the video frame training samples used for the training device include multiple video frames of the first quality; a second acquisition module, configured to: obtain a first output image by performing super-resolution processing on the first output image through the generator a second output image is obtained; a calculation module is configured to calculate a first loss function according to the video frame of the first quality and the first output image, calculate a second loss function according to the video frame of the first quality, the second output image, and the result of the discriminator distinguishing the second output image from the video frame of the first quality, and calculate a third loss function according to the result of the discriminator distinguishing the authenticity of the second output image from the video frame of the first quality; and a parameter adjustment module is configured to respectively adjust the model parameters of the reconstructor, the generator and the discriminator according to the first loss function, the second loss function and the third loss function to train the video super-resolution model.

[0024] Optionally, the training device may also include: a third acquisition module, configured to: process the video frames of the second quality through the video super-resolution model to obtain a third output image; wherein the calculation module is also configured to: calculate a fourth loss function based on the result of the discriminator's authenticity judgment of the third output image and the video frames of the first quality; the parameter adjustment module is also configured to: further adjust the model parameters of the reconstructor, generator and discriminator according to the fourth loss function to fine-tune the video super-resolution model.

[0025] Optionally, the calculation module can be configured to calculate the second loss function by the following operations: calculating the L1 loss function based on the second output image and the video frame of the first quality; calculating the adversarial loss function based on the result of the discriminator's true and false discrimination of the second output image and the video frame of the first quality; calculating the perceptual loss function based on the second output image and the video frame of the first quality; and calculating the weighted sum of the L1 loss function, the adversarial loss function and the perceptual loss function as the second loss function.

[0026] Optionally, the calculation module can be configured to calculate the fourth loss function by the following operations: calculating the L1 loss function based on the third output image and the video frame of the first quality; calculating the adversarial loss function based on the result of the discriminator's true and false discrimination of the third output image and the video frame of the first quality; calculating the perceptual loss function based on the third output image and the video frame of the first quality; and calculating the weighted sum of the L1 loss function, the adversarial loss function and the perceptual loss function as the fourth loss function.

[0027] Optionally, the first acquisition module can be configured to obtain the first output image through the following operations: extracting depth features from the video frame of the second quality through a reconstructor; processing the extracted depth features through multiple convolution modules in the reconstructor to obtain processed depth features; converting the processed depth features to the same color space as the video frame of the second quality through the reconstructor to obtain the first output image.

[0028] Optionally, the second acquisition module can be configured to obtain the second output image through the following operations: extracting depth features from the first output image through a generator; processing the extracted depth features through multiple convolution modules in the generator to obtain processed depth features; converting the processed depth features to the same color space as the first output image through the generator and performing upsampling to obtain the second output image.

[0029] Optionally, the third acquisition module can be configured to obtain a third output image through the following operations: extracting depth features from the video frames of the second quality through the video super-resolution model; processing the extracted depth features through multiple convolution modules in the video super-resolution model to obtain processed depth features; converting the processed depth features to the same color space as the video frames of the second quality through the video super-resolution model and performing upsampling to obtain the third output image.

[0030] Optionally, the multiple convolution modules included in the video super-resolution model can be connected to each other in series, wherein the multiple convolution modules can perform the following operations to obtain processed depth features: each convolution module performs the following processing: transforms the input depth features, and adds the transformed results to the input depth features to obtain output depth features, wherein the input depth features of the first convolution module are the extracted depth features, and the output depth features of the first convolution module are the input depth features of the next convolution module, and the input depth features of the last convolution module are the output depth features of the previous convolution module, and the output depth features of the last convolution module are the processed depth features.

[0031] Optionally, the processed depth features can also be obtained by performing the following operations through the multiple convolution modules: at least one convolution module among the multiple convolution modules except the first convolution module also performs the following processing: adding the input depth features of at least one convolution module located before the current convolution module to the output depth features of the current convolution module to obtain the updated output depth features.

[0032] Optionally, the first loss function may be an L1 loss function.

[0033] Optionally, the denoising process may include: denoising and decompression artifacts, and the super-resolution process may include: supplementing texture details.

[0034] According to a fourth aspect of an embodiment of the present disclosure, a video super-resolution device is provided, comprising: a first acquisition module, configured to: obtain a first output image by denoising the reconstructed video frame using a reconstructor in a video super-resolution model trained using the training method described above; and a second acquisition module, configured to: obtain a second output image by performing super-resolution processing on the first output image using a generator in the video super-resolution model.

[0035] Optionally, the first acquisition module can be configured to obtain a first output image through the following operations: extracting depth features from the video frame through a reconstructor; processing the extracted depth features through multiple convolution modules in the reconstructor to obtain processed depth features; and converting the processed depth features to the same color space as the video frame through the reconstructor to obtain the first output image.

[0036] Optionally, the second acquisition module can be configured to obtain the second output image through the following operations: extracting depth features from the first output image through a generator; processing the extracted depth features through multiple convolution modules in the generator to obtain processed depth features; converting the processed depth features to the same color space as the first output image through the generator and performing upsampling to obtain the second output image.

[0037] Optionally, the multiple convolution modules included in the video super-resolution model can be connected to each other in series, wherein the multiple convolution modules can perform the following operations to obtain processed depth features: each convolution module performs the following processing: transforms the input depth features, and adds the transformed results to the input depth features to obtain output depth features, wherein the input depth features of the first convolution module are the extracted depth features, and the output depth features of the first convolution module are the input depth features of the next convolution module, and the input depth features of the last convolution module are the output depth features of the previous convolution module, and the output depth features of the last convolution module are the processed depth features.

[0038] Optionally, the processed depth features can also be obtained by performing the following operations through the multiple convolution modules: at least one convolution module among the multiple convolution modules except the first convolution module also performs the following processing: adding the input depth features of at least one convolution module located before the current convolution module to the output depth features of the current convolution module to obtain the updated output depth features.

[0039] Optionally, the denoising process may include: denoising and decompression artifacts, and the super-resolution process may include: supplementing texture details.

[0040] According to a fifth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor, wherein the processor is configured to execute the instructions to implement the training method of the video super-resolution model as described above or the video super-resolution method as described above.

[0041] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device / server, the electronic device / server is enabled to execute the training method of the video super-resolution model as described above or the video super-resolution method as described above.

[0042] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, which includes a computer program / instructions, and when the computer program / instructions are executed by a processor, the training method of the video super-resolution model as described above or the video super-resolution method as described above is implemented.

[0043] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: through the video super-resolution model as described above, the excessive smoothing of the super-resolution results is reduced, enhanced noise and erroneous textures are avoided, and the stability of the super-resolution results is increased and the visual effect is better.

[0044] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0046] Figure 1 is an exemplary system architecture diagram in which exemplary embodiments of the present disclosure may be applied;

[0047] Figure 2 is a flowchart illustrating a method for training a video super-resolution model according to an exemplary embodiment of the present disclosure;

[0048] Figure 3 is a schematic diagram showing the training of a video super-resolution model according to an exemplary embodiment of the present disclosure;

[0049] Figure 4is a schematic diagram showing the structure of a reconstructor according to an exemplary embodiment of the present disclosure;

[0050] Figure 5 is a schematic diagram showing the structure of a generator according to an exemplary embodiment of the present disclosure;

[0051] Figure 6 is a schematic diagram showing the structure of a discriminator according to an exemplary embodiment of the present disclosure;

[0052] Figure 7 is a flowchart illustrating a video super-resolution method according to an exemplary embodiment of the present disclosure;

[0053] Figure 8 is a block diagram illustrating a training apparatus for a video super-resolution model according to an exemplary embodiment of the present disclosure.

[0054] Fig. 9 is a block diagram illustrating a video super-resolution apparatus according to an exemplary embodiment of the present disclosure.

[0055] Fig.10 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0056] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0057] It should be understood that the data used in this way can be interchanged where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0058] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0059] As mentioned in the background of the present disclosure, in the related art, the SR algorithm based on deep learning has become the mainstream super-resolution algorithm. One SR model is a model trained based on L1 or MSE as a loss function. This type of model has the problem of over-smoothing the output image. Another SR model is the SR type based on GAN (Generative Adversarial Network).

[0060] Specifically, the GAN model consists of a generative network model and a discriminative network model. The generative network is used to generate samples, and the discriminative network is used to discriminate the authenticity of the generated samples, the constructed data samples, and the GT (Ground Truth) samples. In order to make the generated samples closer to the GT samples, the generator needs to continuously improve its "faking" ability, so that the generated samples are closer to the GT samples, thereby confusing the discriminator; in order not to be deceived by the generator, the discriminator continuously improves its identification ability, so that it can distinguish between true and false samples. The above process is repeated continuously, and the two network models continue to compete with each other, so that the performance of both network models is improved until the discriminative network model can no longer distinguish the authenticity, and the generative network model converges.

[0061] Although the above-mentioned GAN-based SR model can improve subjective clarity, in practical applications (especially in video super-resolution tasks in UGC (i.e., user-generated content) scenarios), due to the wide range of sources of low-resolution videos, the non-uniform video degradation methods, and the presence of a large amount of interference content such as noise and compression artifacts, a simple GAN-based SR model will regard these interference contents as textures and details existing in the video itself, enhance and fill in these interferences, resulting in more serious noise and erroneous textures in the super-resolution video, resulting in a decrease in the quality of the generated video.

[0062] In order to solve the problems of over-smoothing, enhanced noise and erroneous texture in the prior art, the present disclosure proposes a GAN algorithm with a two-stage SR strategy, which adopts a two-stage training model, and the two-stage network model is composed in a cascade (e.g., series connection). Specifically, the first-stage network model has the functions of removing noise and compression artifacts, removing the interference content (e.g., noise and compression artifacts) of the input image, and ensuring that there is no additional interference content (e.g., noise) to interfere with the effect of the generated image after super-resolution; the two-stage network model has a GAN-based network structure, which can supplement texture and detail information, thereby ensuring the generation of stable texture details and avoiding the problem of over-smoothing of the generated image.

[0063] Below, we will refer to Figures 1 to 10The present invention specifically describes a video super-resolution model training method and training device and a video super-resolution method and device according to exemplary embodiments of the present invention.

[0064] Figure 1 is an exemplary system architecture diagram 100 in which exemplary embodiments of the present disclosure may be applied.

[0065] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. Users may use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages (such as video data upload requests, video data acquisition requests), etc. Various communication client applications may be installed on the terminal devices 101, 102, 103, such as video recording applications, video compression applications, video and audio editing applications, instant messaging tools, email clients, social platform software, etc. The terminal devices 101, 102, 103 may be hardware or software. When the terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with display screens that can play, record, and edit videos, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, personal digital assistants (PDAs), augmented reality (AR) / virtual reality (VR) devices, wearable devices, and any other electronic products that can interact with users through one or more methods such as keyboards, touch pads, touch screens, remote controls, voice interactions, or handwriting devices. When the terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above, which can be implemented as multiple software or software modules (for example, to provide distributed services), or as a single software or software module. No specific limitation is made here.

[0066] The terminal devices 101, 102, and 103 may be installed with an image acquisition device (e.g., a camera) to collect video data. In addition, the terminal devices 101, 102, and 103 may also be installed with a component for converting electrical signals into sound (e.g., a speaker) to play sound, and may also be installed with a device for converting analog audio signals into digital audio signals (e.g., a microphone) to collect sound.

[0067] The terminal devices 101, 102, and 103 can use the image acquisition device installed thereon to acquire video data, and use the audio acquisition device installed thereon to acquire audio data, and the terminal devices 101, 102, and 103 can encode, store, and transmit the acquired video data and audio data, and can decode and play the encoded video and audio received from another terminal device or from the server 105.

[0068] Server 105 can be a server that provides various services, such as a background server that provides support for video recording applications, video compression applications, video editing applications, etc. installed on terminal devices 101, 102, 103, or it can be a storage server that stores encoded video and audio uploaded by terminal devices 101, 102, 103, and can send the stored encoded video and audio to terminal devices 101, 102, 103 in response to requests from terminal devices 101, 102, 103.

[0069] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0070] It should be noted that the video super-resolution model training method and the video super-resolution method provided in the embodiments of the present application are generally executed by terminal devices 101, 102, and 103. Accordingly, the video super-resolution model training device and the video super-resolution device are generally arranged in terminal devices 101, 102, and 103.

[0071] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers, and the present disclosure has no limitation on this.

[0072] Figure 2 is a flowchart illustrating a training method 200 of a video super-resolution model according to an exemplary embodiment of the present disclosure. Figure 3 FIG. 3 is a schematic diagram showing the training of a video super-resolution model 300 according to an exemplary embodiment of the present disclosure. Figure 3 As shown in , the video super-resolution model 300 includes a reconstructor 310 , a generator 320 , and a discriminator 330 .

[0073] like Figure 2As shown in, according to an exemplary embodiment of the present disclosure, in step S201, a video frame training sample for training a video super-resolution model is obtained. Optionally, the video frame training sample includes a plurality of video frames of a first quality. According to an exemplary embodiment of the present disclosure, the video frame of the first quality is a video frame of a GT video, and the plurality of video frames of the GT video are used as GT sample data of the present disclosure for a subsequent series of operations (to be described in detail later).

[0074] Optionally, the acquisition process of the video frame training samples can be performed before the data is input into the model, or a module with the above acquisition function can be added at the very beginning of the model to process the video to obtain the video frame training samples.

[0075] According to an exemplary embodiment of the present disclosure, in step S202, the reconstructor 310 performs denoising processing on the video frame of the second quality to obtain a first output image. According to an exemplary embodiment of the present disclosure, the video frame of the second quality is obtained by encoding and decoding the video frame of the first quality. Specifically, the video frame of the second quality is a video frame obtained by encoding the video frame of the first quality according to a certain video codec and then decoding it. The video frame is a low-quality and / or low-resolution video frame relative to the video frame of the first quality, and the video frame of the second quality can be a video frame that has never been subjected to any encoding and decoding processing, or can also be a video frame that has been subjected to encoding and decoding processing by a certain video codec. In other words, the video frame of the first quality and the video frame of the second quality are the same video frame with different video qualities and / or different resolutions.

[0076] According to an exemplary embodiment of the present disclosure, the method of generating a second quality video frame by a first quality video frame is mainly based on the need to process low quality and / or low resolution video (hereinafter referred to as LR video) into high quality and / or high resolution video (hereinafter referred to as HR video) in actual application scenarios (for example, short video applications). In order to be closer to the LR video in the actual application scenario, a method for degradation processing of GT video is proposed to obtain LR video. Specifically, the process of obtaining LR video includes: performing degradation processing on GT video by using at least one combination of A) multiple downsampling and multiple video encoding, B) single downsampling and at least one video encoding, and C) multiple downsampling and multiple video encoding, and obtaining LR video after processing. Optionally, although the single downsampling degradation method is also applicable to the training method of the present disclosure, the effect of implementing the training method of the present disclosure by using the combination method as described above is better. For example, by using the degradation processing of the combination method, the degradation range of LR video can be broadened and the diversity of LR video can be constructed, so that the robustness of the entire video super-resolution model can be improved by training such data.

[0077] The following will be combined Figure 4 Describe this step in detail. Figure 4 is a schematic diagram showing the structure of a reconstructor 310 according to an exemplary embodiment of the present disclosure.

[0078] According to an exemplary embodiment of the present disclosure, the step of performing denoising processing on the video frame of the second quality by the reconstructor 310 to obtain the first output image may include: extracting depth features from the video frame of the second quality by the reconstructor 310; processing the extracted depth features by multiple convolution modules in the reconstructor 310 to obtain processed depth features; and converting the processed depth features to the same color space as the video frame of the second quality by the reconstructor 310 to obtain the first output image.

[0079] Optionally, the color space involved in the present disclosure may be any one of the RGB space, YUV space, etc. in the related art, which will not be described in detail below. Optionally, the denoising process involved in the present disclosure may include: denoising and decompression artifact removal, etc., which will not be described in detail below.

[0080] Specific reference Figure 4 For example, the first convolution module 410 extracts depth features from the second quality video frame and inputs the depth features to the second convolution module 420, wherein the second convolution module 420 includes a plurality of convolution modules 420-1 to 420-n. The second convolution module 420 processes the extracted depth features to obtain processed depth features; the third convolution module 430 converts the processed depth features to the same color space as the second quality video frame to obtain a first output image.

[0081] According to an exemplary embodiment of the present disclosure, multiple convolution modules 420-1 to 420-n (n is an integer greater than or equal to 1) can be connected to each other in series, wherein the step of processing the extracted depth features by multiple convolution modules 420-1 to 420-n to obtain processed depth features may include: each convolution module performs the following processing: transforming the input depth features, and adding the transformed results to the input depth features to obtain output depth features, wherein the input depth features of the first convolution module 420-1 are the depth features extracted by the first convolution module 410, and the output depth features of the first convolution module 420-1 are the input depth features of the next convolution module, and the input depth features of the last convolution module 420-n are the output depth features of the previous convolution module, and the output depth features of the last convolution module 420-1 are the processed depth features.

[0082] According to an exemplary embodiment of the present disclosure, the step of processing the extracted depth features by the plurality of convolution modules 420-1 to 420-n to obtain the processed depth features may further include: processing the extracted depth features by at least one convolution module (e.g., Figure 4 420-i, 420-j, 420-n (i and j are integers greater than or equal to 1 and less than or equal to n, respectively) shown in the also perform the following processing: add the input depth features of at least one convolution module located before the current convolution module to the output depth features of the current convolution module to obtain the updated output depth features.

[0083] Specifically, as an example, refer to Figure 4 , when i is greater than 1, j is greater than i and less than n, the depth feature of the output of the i-th convolution module 420-i is obtained as follows: the depth feature of the input of the i-th convolution module 420-i is transformed, and the transformed result is added to the input depth feature to obtain the depth feature of the output before updating, the depth feature of the input of the first convolution module 420-1 located before the i-th convolution module is added to the depth feature of the output of the i-th convolution module 420-i before updating to obtain the depth feature of the updated output, and the depth feature of the updated output is used as the depth feature of the output of the i-th convolution module 420-i, that is, the depth feature of the input of the i+1-th convolution module.

[0084] As an example, see Figure 4 , when i is greater than 1, j is greater than i and less than n, the depth features of the output of the nth convolution module 420-n are obtained as follows: the depth features of the input of the nth convolution module 420-n are transformed, and the transformed result is added to the input depth features to obtain the depth features of the output before updating, the depth features of the input of the first convolution module 420-1 located before the nth convolution module and the depth features of the input of the jth convolution module 420-j are added to the depth features of the output of the nth convolution module 420-n before updating to obtain the depth features of the updated output, and the updated output depth features are used as the depth features of the output of the nth convolution module 420-n, that is, the depth features of the input of the third convolution module 430.

[0085] like Figure 4 As described in , the short skip connections within each convolutional module (e.g., Figure 4 The short arrow corresponding to the i-th convolution module 420-i in FIG) adds the input depth feature of the current convolution module to the output depth feature, while the long jump connection of the reconstructor (for example, Figure 4The arrows between the first convolution module 420-1 and the i-th convolution module 420-i and the n-th convolution module 420-n, and the arrows between the j-th convolution module 420-j and the n-th convolution module 420-n) add the input depth features and output depth features of multiple other convolution modules as the updated output depth features, which can better enable the reconstructor to perform gradient backpropagation and accelerate the convergence of the model.

[0086] In addition, as another example, the step of converting the processed depth features into the same color space as the second quality video frame by the third convolution module 430 to obtain a first output image can also be implemented in the following manner: the third convolution module 430 converts the processed depth features into the same color space as the second quality video frame to obtain a first residual image, and the third convolution module 430 adds the first residual image to the second quality video frame to obtain the first output image.

[0087] According to an exemplary embodiment of the present disclosure, in addition to being able to implement the denoising function, the reconstructor 310 can also implement the function of data normalization. Specifically, taking a video of a complex application scene as input data, after being processed by the reconstructor 310, an intermediate result with a relatively consistent distribution can be obtained. Using such an intermediate result with a relatively consistent distribution as the input of the generator 320 can alleviate the problem of poor model robustness caused by the diversity of input data.

[0088] Reference Figure 2 , according to an exemplary embodiment of the present disclosure, in step S203, a first loss function is calculated according to the video frame of the first quality and the first output image.

[0089] According to an exemplary embodiment of the present disclosure, the step of calculating the first loss function may include: calculating a second L1 loss function as the first loss function based on the video frame of the first quality and the first output image. Here, the calculation method of the L1 loss function may adopt any corresponding calculation method involved in the relevant technology, which is not specifically limited in the present disclosure and will not be repeated below.

[0090] According to an exemplary embodiment of the present disclosure, in step S204, the generator 320 performs super-resolution processing on the first output image to obtain the second output image. Figure 4 Describe this step in detail. Figure 5 is a schematic diagram showing the structure of the generator 320 according to an exemplary embodiment of the present disclosure.

[0091] According to an exemplary embodiment of the present disclosure, the step of performing super-resolution processing on the first output image by the generator 320 to obtain the second output image may include: extracting depth features from the first output image by the generator 320; processing the extracted depth features by multiple convolution modules in the generator 320 to obtain processed depth features; and converting the processed depth features to the same color space as the first output image by the generator 320 to obtain the second output image.

[0092] Optionally, the super-resolution processing may include: processing of supplementing texture details, which will not be described in detail below.

[0093] Specific reference Figure 5 For example, the first convolution module 510 extracts depth features from the first output image and inputs the depth features to the second convolution module 520, wherein the second convolution module 520 includes a plurality of convolution modules 520-1 to 520-n. The second convolution module 520 processes the extracted depth features to obtain processed depth features; the third convolution module 530 converts the processed depth features to the same color space as the second quality video frame and performs upsampling processing by the upsampling module 540 to obtain the second output image.

[0094] Optionally, the upsampling module and upsampling processing involved in the present disclosure can be implemented using any known upsampling method in the relevant technology (for example, upsampling based on transposed convolution, etc.), which is not specifically limited in the present disclosure and will not be described in detail below.

[0095] According to an exemplary embodiment of the present disclosure, multiple convolution modules 520-1 to 520-n (n is an integer greater than or equal to 1) can be connected to each other in series, wherein the step of processing the extracted depth features by multiple convolution modules 420-1 to 420-n to obtain processed depth features may include: each convolution module performs the following processing: transforming the input depth features, and adding the transformed results to the input depth features to obtain output depth features, wherein the input depth features of the first convolution module 520-1 are the depth features extracted by the first convolution module 510, and the output depth features of the first convolution module 520-1 are the input depth features of the next convolution module, and the input depth features of the last convolution module 520-n are the output depth features of the previous convolution module, and the output depth features of the last convolution module 520-n are the processed depth features.

[0096] According to an exemplary embodiment of the present disclosure, the step of processing the extracted depth features by the plurality of convolution modules 520-1 to 520-n to obtain the processed depth features may further include: processing the extracted depth features by at least one convolution module (e.g., Figure 4 The 520-i, 520-j, and 520-n (i and j are integers greater than or equal to 1 and less than or equal to n, respectively) shown in the figure also perform the following processing: adding the input depth features of at least one convolution module located before the current convolution module to the output depth features of the current convolution module to obtain the updated output depth features.

[0097] Specifically, as an example, refer to Figure 5 , when i is greater than 1, j is greater than i and less than n, the depth features of the output of the i-th convolution module 520-i are obtained as follows: the depth features of the input of the i-th convolution module 520-i are transformed, and the transformed result is added to the input depth features to obtain the depth features of the output before updating, the depth features of the input of the first convolution module 520-1 located before the i-th convolution module are added to the depth features of the output of the i-th convolution module 520-i before updating to obtain the depth features of the updated output, and the depth features of the updated output are used as the depth features of the output of the i-th convolution module 520-i, that is, the depth features of the input of the i+1-th convolution module.

[0098] As an example, see Figure 5 , when i is greater than 1, j is greater than i and less than n, the depth features of the output of the nth convolution module 520-n are obtained as follows: the depth features of the input of the nth convolution module 520-n are transformed, and the transformed result is added to the input depth features to obtain the depth features of the output before updating, the input depth features of the first convolution module 520-1 located before the nth convolution module and the input depth features of the jth convolution module 520-j are added to the depth features of the output of the nth convolution module 520-n before updating to obtain the depth features of the updated output, and the updated output depth features are used as the output depth features of the nth convolution module 520-n, that is, the input depth features of the third convolution module 530.

[0099] like Figure 5 As described in , the short skip connections within each convolutional module (e.g., Figure 5 The short arrow corresponding to the i-th convolution module 520-i in FIG) adds the input depth feature of the current convolution module to the output depth feature, while the long jump connection of the reconstructor (for example, Figure 5The arrows between the first convolution module 520-1 and the i-th convolution module 520-i and the n-th convolution module 520-n, and the arrows between the j-th convolution module 520-j and the n-th convolution module 520-n) add the input depth features and output depth features of multiple other convolution modules as the updated output depth features, which can better enable the reconstructor to perform gradient backpropagation and accelerate the convergence of the model.

[0100] In addition, as another example, the step of converting the processed depth features to the same color space as the second quality video frame by the third convolution module 530 and performing upsampling processing by the upsampling module 540 to obtain the second output image can also be implemented in the following manner: the third convolution module 530 converts the processed depth features to the same color space as the second quality video frame to obtain a second residual image, the third convolution module 530 adds the second residual image to the first output image to obtain the added image, and the upsampling module 540 performs upsampling processing on the added image to obtain the second output image.

[0101] Reference Figure 2 According to an exemplary embodiment of the present disclosure, in step S205, a second loss function is calculated based on the video frame of the first quality, the second output image, and the result of the discriminator 330 distinguishing the second output image from the video frame of the first quality.

[0102] According to an exemplary embodiment of the present disclosure, the step of calculating the second loss function may include: calculating the first L1 loss function according to the second output image and the video frame of the first quality; calculating the first adversarial loss function according to the result of the true and false discrimination of the second output image and the video frame of the first quality by the discriminator 330; calculating the first perceptual loss function according to the second output image and the video frame of the first quality; and calculating the weighted sum of the first L1 loss function, the first adversarial loss function and the first perceptual loss function as the second loss function. Here, the calculation method of the adversarial loss function and the perceptual loss function can adopt any corresponding calculation method involved in the relevant technology, which is not specifically limited by the present disclosure and will not be repeated below.

[0103] The following will be combined Figure 6 The relevant operations of the discriminator 330 are described in detail. Figure 6 is a schematic diagram showing the structure of the discriminator 330 according to an exemplary embodiment of the present disclosure.

[0104] According to an exemplary embodiment of the present disclosure, the discriminator 330 may specifically perform the discriminator 330 discriminator process as follows: input the discriminator input image to at least one convolution layer 610 to perform multiple convolution processes to obtain an intermediate output result, and input the intermediate output result of the convolution layer 610 to a linear layer (i.e., Figure 6The linear layer in the image processing unit 620 is used to perform a linear transformation to obtain a discrimination result, which can be used to determine the authenticity of the input image of the discriminator. For example, the discrimination result can be used to determine the authenticity of the input image of the discriminator relative to the corresponding real image used as a comparison reference. Optionally, the discrimination result can be used as follows: Figure 6 It is represented by the Label in , which can be a floating point number.

[0105] According to an exemplary embodiment of the present disclosure, the above-mentioned discrimination process is applied to the process of discriminating the authenticity of the second output image as follows: the second output image is input to the convolution layer 610 to perform multiple convolution processes to obtain an intermediate output result, and the intermediate output result of the convolution layer 610 is input to the linear layer 620 to perform a linear transformation to obtain a discrimination result. The discrimination result can be used to determine the authenticity of the second output image relative to the video frame of the first quality. In addition, the discrimination result can be reflected in the adversarial loss function.

[0106] Reference Figure 2 According to an exemplary embodiment of the present disclosure, in step S206, a third loss function is calculated according to the result of the discriminator 330 distinguishing the second output image and the video frame of the first quality.

[0107] According to an exemplary embodiment of the present disclosure, the step of calculating the third loss function may include: calculating the first adversarial loss function as the third loss function according to a result of the discriminator 330 discriminating the second output image and the video frame of the first quality.

[0108] It should be understood that the above reference Figure 6 The structure and the discrimination process of the discriminator 330 have been described, and therefore, the process of calculating the third loss function will not be described in detail herein.

[0109] According to an exemplary embodiment of the present disclosure, in step S207, the model parameters of the reconstructor 310, the generator 320 and the discriminator 330 are respectively adjusted according to the first loss function, the second loss function and the third loss function to train the video super-resolution model.

[0110] Optionally, according to the exemplary embodiment of the present disclosure, the training method 100 of the video super-resolution model may also include: the video super-resolution model 300 fine-tunes the video frame of the second quality to obtain a third output image; calculates a fourth loss function based on the result of the discriminator 330 distinguishing the third output image from the video frame of the first quality; and further adjusts the model parameters of the reconstructor 310, the generator 320 and the discriminator 330 according to the fourth loss function to fine-tune the video super-resolution model 300.

[0111] Optionally, according to an exemplary embodiment of the present disclosure, the step of fine-tuning the second quality video frame by the video super-resolution model 300 to obtain the third output image may include: extracting depth features from the second quality video frame by the video super-resolution model 300; processing the extracted depth features by multiple convolution modules in the video super-resolution model 300 to obtain processed depth features; and converting the processed depth features to the same color space as the second quality video frame by the video super-resolution model to obtain the third output image.

[0112] Optionally, according to an exemplary embodiment of the present disclosure, multiple convolution modules in the video super-resolution model can be connected to each other in series. The step of processing the extracted depth features by multiple convolution modules in the video super-resolution model to obtain processed depth features may include: each convolution module performs the following processing: transforming the input depth features, and adding the transformed results to the input depth features to obtain output depth features, wherein the input depth features of the first convolution module are the extracted depth features, and the output depth features of the first convolution module are the input depth features of the next convolution module, and the input depth features of the last convolution module are the output depth features of the previous convolution module, and the output depth features of the last convolution module are the processed depth features.

[0113] Optionally, according to an exemplary embodiment of the present disclosure, the step of processing the extracted depth features by multiple convolution modules in the video super-resolution model to obtain processed depth features may also include: by,. At least one convolution module other than the first convolution module in the multiple convolution modules further performs the following processing: adding the input depth features of at least one convolution module located before the current convolution module to the output depth features of the current convolution module to obtain the updated output depth features.

[0114] Optionally, according to an exemplary embodiment of the present disclosure, the step of calculating the fourth loss function based on the result of the discriminator 330 distinguishing the third output image and the video frame of the first quality may include: calculating the second L1 loss function based on the third output image and the video frame of the first quality; calculating the second adversarial loss function based on the result of the discriminator distinguishing the third output image and the video frame of the first quality; calculating the second perceptual loss function based on the third output image and the video frame of the first quality; and calculating the weighted sum of the second L1 loss function, the second adversarial loss function and the second perceptual loss function as the fourth loss function.

[0115] Optionally, according to an exemplary embodiment of the present disclosure, as an example, the step of converting the processed depth features by the video super-resolution model 300 to the same color space as the video frame of the second quality to obtain a third output image can also be implemented in the following manner: the video super-resolution model 300 converts the processed depth features to the same color space as the video frame of the second quality to obtain a third residual image, the video super-resolution model 300 adds the third residual image to the video frame of the second quality to obtain an added image, and the video super-resolution model 300 upsamples the added image to obtain the third output image.

[0116] Optionally, according to an exemplary embodiment of the present disclosure, as another example, the step of converting the processed depth features into the same color space as the video frame of the second quality by the video super-resolution model 300 to obtain a third output image can also be implemented in the following manner: the video super-resolution model 300 converts the processed depth features into the same color space as the video frame of the second quality to obtain a third residual image, and the video super-resolution model 300 adds the third residual image to the video frame of the second quality to obtain a third output image.

[0117] It should be understood that the above examples can be used individually or in combination as examples of the training method for implementing the video super-resolution model of the present disclosure.

[0118] Figure 7 7 is a flowchart illustrating a video super-resolution method 700 according to an exemplary embodiment of the present disclosure.

[0119] like Figure 7 As shown in FIG. 1 , in step S701, the reconstructor 310 in the video super-resolution model 300 trained by the training method 100 described above performs denoising on the reconstructed video frame to obtain a first output image. In step S702, the generator 320 in the video super-resolution model 300 performs super-resolution processing on the first output image to obtain a second output image.

[0120] According to an exemplary embodiment of the present disclosure, the step of obtaining a first output image may include: extracting depth features from a video frame by a reconstructor 310; processing the extracted depth features by multiple convolution modules in the reconstructor 310 to obtain processed depth features; and converting the processed depth features to the same color space as the video frame by the reconstructor 310 to obtain a first output image.

[0121] According to an exemplary embodiment of the present disclosure, the step of obtaining a second output image may include: extracting depth features from the first output image by the generator 320; processing the extracted depth features by multiple convolution modules in the generator 320 to obtain processed depth features; and converting the processed depth features to the same color space as the first output image by the generator 320 and performing upsampling processing to obtain the second output image.

[0122] According to an exemplary embodiment of the present disclosure, multiple convolution modules in the reconstructor 310 can be connected to each other in series, wherein the step of processing the extracted depth features by the multiple convolution modules in the reconstructor 310 to obtain the processed depth features may include: each convolution module performs the following processing: transforming the input depth features, and adding the transformed results to the input depth features to obtain the output depth features, wherein the input depth features of the first convolution module are the extracted depth features, and the output depth features of the first convolution module are the input depth features of the next convolution module, and the input depth features of the last convolution module are the output depth features of the previous convolution module, and the output depth features of the last convolution module are the processed depth features.

[0123] According to an exemplary embodiment of the present disclosure, the step of processing the extracted depth features by multiple convolution modules in the reconstructor 310 to obtain processed depth features may also include: at least one convolution module other than the first convolution module among the multiple convolution modules also performs the following processing: adding the input depth features of at least one convolution module located before the current convolution module to the output depth features of the current convolution module to obtain the updated output depth features.

[0124] According to an exemplary embodiment of the present disclosure, multiple convolution modules in the generator 320 can be connected to each other in series, wherein the step of processing the extracted depth features by multiple convolution modules in the generator to obtain processed depth features may include: each convolution module performs the following processing: transforming the input depth features, and adding the transformed results to the input depth features to obtain output depth features, wherein the input depth features of the first convolution module are the extracted depth features, and the output depth features of the first convolution module are the input depth features of the next convolution module, and the input depth features of the last convolution module are the output depth features of the previous convolution module, and the output depth features of the last convolution module are the processed depth features.

[0125] According to an exemplary embodiment of the present disclosure, the step of processing the extracted depth features by multiple convolution modules in the generator to obtain processed depth features may also include: at least one convolution module other than the first convolution module among the multiple convolution modules also performs the following processing: adding the input depth features of at least one convolution module located before the current convolution module to the output depth features of the current convolution module to obtain the updated output depth features.

[0126] At this point, the final output image can be obtained through the above process, which reduces over-smoothing, avoids enhanced noise and erroneous textures, and has increased stability and better visual effects.

[0127] Figure 8 8 is a block diagram illustrating a training apparatus 800 for a video super-resolution model according to an exemplary embodiment of the present disclosure.

[0128] like Figure 8 As shown in , the training device 800 of the video super-resolution model according to an exemplary embodiment of the present disclosure includes a first acquisition module 810, a second acquisition module 820, a calculation module 830 and a parameter adjustment module 840.

[0129] The first acquisition module 810 is configured to obtain a first output image by performing denoising processing on a video frame of second quality through a reconstructor 310, wherein the video frame of second quality is obtained by encoding and decoding a video frame of first quality, and the video frame training samples used for the training device include multiple video frames of first quality.

[0130] The second acquisition module 820 is configured to obtain a second output image by performing super-resolution processing on the first output image through the generator 320 .

[0131] The calculation module 830 is configured to: calculate a first loss function based on the video frame of the first quality and the first output image, calculate a second loss function based on the video frame of the first quality, the second output image, and the result of the discriminator 330 discriminating the second output image and the video frame of the first quality, and calculate a third loss function based on the result of the discriminator discriminating the authenticity of the second output image and the video frame of the first quality.

[0132] The parameter adjustment module 840 is configured to adjust the model parameters of the reconstructor 310, the generator 320 and the discriminator 330 according to the first loss function, the second loss function and the third loss function respectively to train the video super-resolution model 300.

[0133] Optionally, the video super-resolution model training device 800 according to the exemplary embodiment of the present disclosure may further include a third acquisition module. The third acquisition module is configured to: obtain a third output image by fine-tuning the video frame of the second quality through the video super-resolution model. The calculation module 830 is also configured to: calculate a fourth loss function according to the result of the authenticity discrimination of the third output image and the video frame of the first quality by the discriminator 330. The parameter adjustment module 840 is also configured to further adjust the model parameters of the reconstructor, the generator and the discriminator according to the fourth loss function to fine-tune the video super-resolution model.

[0134] According to an exemplary embodiment of the present disclosure, the calculation module 830 may be configured to calculate the second loss function by the following operations: calculating the first L1 loss function based on the second output image and the video frame of the first quality; calculating the first adversarial loss function based on the result of the discriminator's true or false discrimination of the second output image and the video frame of the first quality; calculating the first perceptual loss function based on the second output image and the video frame of the first quality; and calculating the weighted sum of the first L1 loss function, the first adversarial loss function and the first perceptual loss function as the second loss function.

[0135] According to an exemplary embodiment of the present disclosure, the calculation module 830 may be configured to calculate the fourth loss function by the following operations: calculating the second L1 loss function based on the third output image and the video frame of the first quality; calculating the second adversarial loss function based on the result of the discriminator 330 distinguishing the third output image and the video frame of the first quality; calculating the second perceptual loss function based on the third output image and the video frame of the first quality; and calculating the weighted sum of the second L1 loss function, the second adversarial loss function and the second perceptual loss function as the fourth loss function.

[0136] According to an exemplary embodiment of the present disclosure, the first acquisition module 810 may be configured to obtain a first output image through the following operations: extracting depth features from a video frame of second quality through a reconstructor 310; processing the extracted depth features through a plurality of convolution modules in the reconstructor 310 to obtain processed depth features; and converting the processed depth features into the same color space as the video frame of second quality through the reconstructor 310 to obtain a first output image.

[0137] According to an exemplary embodiment of the present disclosure, the second acquisition module 820 may be configured to obtain a second output image through the following operations: extracting depth features from the first output image through the generator 320; processing the extracted depth features through multiple convolution modules in the generator 320 to obtain processed depth features; converting the processed depth features to the same color space as the first output image through the generator 320 and performing upsampling processing to obtain the second output image.

[0138] According to an exemplary embodiment of the present disclosure, the third acquisition module may be configured to obtain a third output image through the following operations: extracting depth features from the video frame of the second quality through the video super-resolution model 300; processing the extracted depth features through multiple convolution modules in the video super-resolution model 300 to obtain processed depth features; converting the processed depth features to the same color space as the video frame of the second quality through the video super-resolution model 300 and performing upsampling processing to obtain the third output image.

[0139] According to an exemplary embodiment of the present disclosure, multiple convolution modules in the video super-resolution model 300 can be connected to each other in series, wherein the third acquisition module can be configured to obtain the processed depth feature by the following operations: performing the following processing through each convolution module: transforming the input depth feature, and adding the transformed result to the input depth feature to obtain the output depth feature, wherein the input depth feature of the first convolution module is the extracted depth feature, and the output depth feature of the first convolution module is the input depth feature of the next convolution module, and the input depth feature of the last convolution module is the output depth feature of the previous convolution module, and the output depth feature of the last convolution module is the processed depth feature.

[0140] According to an exemplary embodiment of the present disclosure, the third acquisition module can also be configured to obtain the processed depth features through the following operations: at least one convolution module among multiple convolution modules except the first convolution module also performs the following processing: adding the input depth features of at least one convolution module located before the current convolution module to the output depth features of the current convolution module to obtain the updated output depth features.

[0141] According to an exemplary embodiment of the present disclosure, multiple convolution modules in the reconstructor 310 can be connected to each other in series, wherein the first acquisition module 810 can be configured to obtain the processed depth feature by the following operations: performing the following processing through each convolution module: transforming the input depth feature, and adding the transformed result to the input depth feature to obtain the output depth feature, wherein the input depth feature of the first convolution module is the extracted depth feature, and the output depth feature of the first convolution module is the input depth feature of the next convolution module, and the input depth feature of the last convolution module is the output depth feature of the previous convolution module, and the output depth feature of the last convolution module is the processed depth feature.

[0142] According to an exemplary embodiment of the present disclosure, the first acquisition module 810 can also be configured to obtain the processed depth features through the following operations: at least one convolution module other than the first convolution module among multiple convolution modules also performs the following processing: adding the input depth features of at least one convolution module located before the current convolution module to the output depth features of the current convolution module to obtain the updated output depth features.

[0143] According to an exemplary embodiment of the present disclosure, multiple convolution modules in the generator 320 can be connected to each other in series, wherein the second acquisition module 820 can be configured to obtain the processed depth feature by the following operations: performing the following processing through each convolution module: transforming the input depth feature, and adding the transformed result to the input depth feature to obtain the output depth feature, wherein the input depth feature of the first convolution module is the extracted depth feature, and the output depth feature of the first convolution module is the input depth feature of the next convolution module, and the input depth feature of the last convolution module is the output depth feature of the previous convolution module, and the output depth feature of the last convolution module is the processed depth feature.

[0144] According to an exemplary embodiment of the present disclosure, the second acquisition module 820 can also be configured to obtain the processed depth features through the following operations: at least one convolution module among multiple convolution modules except the first convolution module also performs the following processing: adding the input depth features of at least one convolution module located before the current convolution module to the output depth features of the current convolution module to obtain the updated output depth features.

[0145] According to an exemplary embodiment of the present disclosure, the calculation module 830 may be configured to calculate the first loss function by calculating a second L1 loss function as the first loss function according to the video frame of the first quality and the first output image.

[0146] As an example, according to an exemplary embodiment of the present disclosure, the reconstructor 310 can be used as a one-stage denoising network model, the generator 320 and the discriminator 330 can be used as a two-stage super-resolution network model, and the two-stage network models can be trained several times respectively, and the trained two-stage network models are connected in series to obtain a video super-resolution model 300, and then the video super-resolution model 300 including the two-stage network models is fine-tuned as a whole, so as to obtain a trained final video super-resolution module 300. In the present disclosure, the process of fine-tuning training can make the correlation between the two-stage network models closer, that is, the model parameters of the entire model can be optimized.

[0147] because Figure 2The training method shown can be Figure 8 The training device 800 shown is used to perform, therefore, Figure 8 Any details about the operations performed by the modules in the Figure 2 The corresponding description will not be repeated here.

[0148] Fig. 9 is a block diagram illustrating a video super-resolution apparatus 900 according to an exemplary embodiment of the present disclosure.

[0149] like Figure 8 As shown in , a video super-resolution apparatus 900 according to an exemplary embodiment of the present disclosure includes a first acquisition module 910 and a second acquisition module 920 .

[0150] The first acquisition module 910 is configured to obtain a first output image by performing denoising processing on the reconstructed video frame using the reconstructor in the video super-resolution model trained by the training method described above.

[0151] The second acquisition module 920 is configured to obtain a second output image by performing super-resolution processing on the first output image through a generator in a video super-resolution model.

[0152] According to an exemplary embodiment of the present disclosure, the first acquisition module 910 may be configured to obtain a first output image through the following operations: extracting depth features from a video frame through a reconstructor 310; processing the extracted depth features through a plurality of convolution modules in the reconstructor 310 to obtain processed depth features; and converting the processed depth features into the same color space as the video frame through the reconstructor 310 to obtain a first output image.

[0153] According to an exemplary embodiment of the present disclosure, the second acquisition module 920 may be configured to obtain a second output image through the following operations: extracting depth features from the first output image through the generator 320; processing the extracted depth features through multiple convolution modules in the generator 320 to obtain processed depth features; converting the processed depth features to the same color space as the first output image through the generator 320 and performing upsampling processing to obtain the second output image.

[0154] According to an exemplary embodiment of the present disclosure, multiple convolution modules in the reconstructor 310 can be connected to each other in series, wherein the first acquisition module 910 can be configured to obtain the processed depth feature by the following operations: performing the following processing through each convolution module: transforming the input depth feature, and adding the transformed result to the input depth feature to obtain the output depth feature, wherein the input depth feature of the first convolution module is the extracted depth feature, and the output depth feature of the first convolution module is the input depth feature of the next convolution module, and the input depth feature of the last convolution module is the output depth feature of the previous convolution module, and the output depth feature of the last convolution module is the processed depth feature.

[0155] According to an exemplary embodiment of the present disclosure, the first acquisition module 910 can also be configured to obtain the processed depth features through the following operations: at least one convolution module other than the first convolution module among multiple convolution modules also performs the following processing: adding the input depth features of at least one convolution module located before the current convolution module to the output depth features of the current convolution module to obtain the updated output depth features.

[0156] According to an exemplary embodiment of the present disclosure, multiple convolution modules in the generator 320 can be connected to each other in series, wherein the second acquisition module 920 can be configured to obtain the processed depth feature by the following operations: performing the following processing through each convolution module: transforming the input depth feature, and adding the transformed result to the input depth feature to obtain the output depth feature, wherein the input depth feature of the first convolution module is the extracted depth feature, and the output depth feature of the first convolution module is the input depth feature of the next convolution module, and the input depth feature of the last convolution module is the output depth feature of the previous convolution module, and the output depth feature of the last convolution module is the processed depth feature.

[0157] According to an exemplary embodiment of the present disclosure, the second acquisition module 920 can also be configured to obtain the processed depth features through the following operations: at least one convolution module among multiple convolution modules except the first convolution module also performs the following processing: adding the input depth features of at least one convolution module located before the current convolution module to the output depth features of the current convolution module to obtain the updated output depth features.

[0158] because Figure 3 The video super-resolution method shown can be obtained by Fig. 9 The video super-resolution device 900 shown in FIG. Fig. 9 Any details about the operations performed by the modules in the Figure 3 The corresponding description will not be repeated here.

[0159] Fig.10 1 is a block diagram showing an electronic device 1000 according to an embodiment of the present disclosure. The electronic device 1000 may include a memory 1010 and a processor 1020. A set of computer executable instructions is stored in the memory 1010. When the computer executable instruction set is executed by the processor, a training method for a video super-resolution model or a video super-resolution method according to an embodiment of the present disclosure is executed.

[0160] As an example, the electronic device may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instruction set. Here, the electronic device is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above instruction (or instruction set) individually or in combination. The electronic device may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0161] In electronic devices, the processor may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0162] The processor can execute instructions or codes stored in the memory, wherein the memory can also store data. Instructions and data can also be sent and received through the network via the network interface device, wherein the network interface device can adopt any known transmission protocol.

[0163] The memory may be integrated with the processor, for example, RAM or flash memory is arranged within an integrated circuit microprocessor or the like. In addition, the memory may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory and the processor may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor can read files stored in the memory.

[0164] In addition, the electronic device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device may be connected to each other via a bus and / or a network.

[0165] According to an embodiment of the present disclosure, a computer-readable storage medium may also be provided, for example, a memory 1020 including instructions, and the instructions may be executed by the processor 1010 of the electronic device 1000 to complete the above-mentioned video super-resolution model training method or video super-resolution method. Optionally, the computer-readable storage medium may be: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0166] In an exemplary embodiment of the present disclosure, a computer program product may also be provided, including a computer program / instruction, which, when executed by a processor, implements a training method for a video super-resolution model or a video super-resolution method according to an exemplary embodiment of the present disclosure.

[0167] According to the video super-resolution model training method and device and the video super-resolution method and device of the embodiments of the present disclosure, by adopting a two-stage super-resolution model with denoising function and super-resolution function respectively, the excessive smoothing of the super-resolution results is reduced, enhanced noise and erroneous texture are avoided, and the stability of the super-resolution results is increased and the visual effect is better.

[0168] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0169] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video super-resolution model training method, characterized in that: The video super-resolution model includes a reconstructor, a generator and a discriminator, wherein the training method includes: Acquire video frame training samples for training the video super-resolution model, wherein the video frame training samples include a plurality of video frames of a first quality; The reconstructor performs denoising processing on the video frame of the second quality to obtain a first output image, wherein the video frame of the second quality is obtained by encoding and decoding the video frame of the first quality; Calculating a first loss function based on the video frame of the first quality and the first output image; The generator performs super-resolution processing on the first output image to obtain a second output image; Calculating a second loss function according to the video frame of the first quality, the second output image, and a result of the discriminator distinguishing the second output image from the video frame of the first quality; Calculating a third loss function according to a result of the discriminator distinguishing the second output image from the video frame of the first quality; According to the first loss function, the second loss function and the third loss function, the model parameters of the reconstructor, the generator and the discriminator are adjusted respectively to train the video super-resolution model; Wherein, the training method further comprises: Processing the video frame of the second quality by the video super-resolution model to obtain a third output image; Calculating a fourth loss function according to a result of the discriminator distinguishing the authenticity of the third output image and the video frame of the first quality; The model parameters of the reconstructor, the generator, and the discriminator are further adjusted according to the fourth loss function to fine-tune the video super-resolution model.

2. The training method according to claim 1, characterized in that: The steps of calculating the second loss function include: Calculating an L1 loss function based on the second output image and the video frame of the first quality; Calculating an adversarial loss function according to a result of the discriminator distinguishing the authenticity of the second output image and the video frame of the first quality; calculating a perceptual loss function based on the second output image and the video frame of the first quality; and A weighted sum of the L1 loss function, the adversarial loss function, and the perceptual loss function is calculated as a second loss function.

3. The training method according to claim 1, characterized in that: The steps of calculating the fourth loss function include: calculating an L1 loss function based on the third output image and the video frame of the first quality; Calculating an adversarial loss function according to a result of the discriminator distinguishing the authenticity of the third output image and the video frame of the first quality; calculating a perceptual loss function based on the third output image and the video frame of the first quality; and A weighted sum of the L1 loss function, the adversarial loss function, and the perceptual loss function is calculated as a fourth loss function.

4. The training method according to claim 1, characterized in that: The step of obtaining a first output image comprises: extracting, by a reconstructor, deep features from the video frames of the second quality; The extracted deep features are processed by multiple convolution modules in the reconstructor to obtain processed deep features; The reconstructor converts the processed depth features into the same color space as the video frame of the second quality to obtain a first output image.

5. The training method according to claim 1, characterized in that: The step of obtaining a second output image comprises: extracting, by the generator, deep features from the first output image; The extracted deep features are processed by multiple convolution modules in the generator to obtain processed deep features; The generator converts the processed deep features into the same color space as the first output image and performs upsampling to obtain the second output image.

6. The training method according to claim 1, characterized in that: The step of obtaining the third output image comprises: Extracting depth features from the video frames of the second quality by the video super-resolution model; The extracted depth features are processed by a plurality of convolution modules in the video super-resolution model to obtain processed depth features; The video super-resolution model converts the processed depth features into the same color space as the video frame of the second quality and performs upsampling processing to obtain a third output image.

7. The training method according to any one of claims 4 to 6, characterized in that: The multiple convolution modules included in the video super-resolution model are connected to each other in series. The step of processing the extracted depth features by the multiple convolution modules to obtain the processed depth features includes: each convolution module performs the following processing: transforming the input depth features, and adding the transformed results to the input depth features to obtain the output depth features, Among them, the depth feature of the input of the first convolution module is the extracted depth feature, and the depth feature of the output of the first convolution module is the depth feature of the input of the next convolution module, and the depth feature of the input of the last convolution module is the depth feature of the output of the previous convolution module, and the depth feature of the output of the last convolution module is the processed depth feature.

8. The training method according to claim 7, characterized in that: The step of processing the extracted depth features by the multiple convolution modules to obtain processed depth features also includes: at least one convolution module among the multiple convolution modules except the first convolution module also performs the following processing: adding the input depth features of at least one convolution module located before the current convolution module to the output depth features of the current convolution module to obtain the updated output depth features.

9. The training method according to claim 1, characterized in that: The first loss function is the L1 loss function.

10. The training method according to claim 1, characterized in that: The third loss function is the adversarial loss function.

11. The training method according to claim 1, characterized in that: The denoising process includes: removing noise and removing compression artifacts, and the super-resolution process includes: supplementing texture details.

12. A video super-resolution method, characterized in that: include: A reconstructor in a video super-resolution model trained by the training method according to any one of claims 1 to 11 performs denoising on the reconstructed video frame to obtain a first output image; The generator in the video super-resolution model performs super-resolution processing on the first output image to obtain a second output image.

13. The video super-resolution method according to claim 12, characterized in that: The step of obtaining a first output image comprises: extracting, by a reconstructor, deep features from the video frame; The extracted deep features are processed by multiple convolution modules in the reconstructor to obtain processed deep features; The processed depth features are converted into the same color space as the video frame by a reconstructor to obtain a first output image.

14. The video super-resolution method according to claim 12, characterized in that: The step of obtaining a second output image comprises: extracting, by the generator, deep features from the first output image; The extracted deep features are processed by multiple convolution modules in the generator to obtain processed deep features; The generator converts the processed deep features into the same color space as the first output image and performs upsampling to obtain the second output image.

15. The video super-resolution method according to claim 13 or 14, characterized in that: The multiple convolution modules included in the video super-resolution model are connected to each other in series. The step of processing the extracted depth features by the multiple convolution modules to obtain the processed depth features includes: each convolution module performs the following processing: transforming the input depth features, and adding the transformed results to the input depth features to obtain the output depth features, Among them, the depth feature of the input of the first convolution module is the extracted depth feature, and the depth feature of the output of the first convolution module is the depth feature of the input of the next convolution module, and the depth feature of the input of the last convolution module is the depth feature of the output of the previous convolution module, and the depth feature of the output of the last convolution module is the processed depth feature.

16. The video super-resolution method according to claim 15, characterized in that: The step of processing the extracted depth features by the multiple convolution modules to obtain processed depth features also includes: at least one convolution module among the multiple convolution modules except the first convolution module also performs the following processing: adding the input depth features of at least one convolution module located before the current convolution module to the output depth features of the current convolution module to obtain the updated output depth features.

17. The video super-resolution method according to claim 12, characterized in that: The denoising process includes: removing noise and removing compression artifacts, and the super-resolution process includes: supplementing texture details.

18. A video super-resolution model training device, characterized in that: The video super-resolution model includes a reconstructor, a generator and a discriminator, wherein the training device includes: A first acquisition module is configured to: perform denoising processing on a video frame of a second quality through a reconstructor to obtain a first output image, wherein the video frame of the second quality is obtained by encoding and decoding the video frame of the first quality, and the video frame training sample used for the training device includes a plurality of video frames of the first quality; A second acquisition module is configured to: obtain a second output image by performing super-resolution processing on the first output image through the generator; a calculation module configured to: calculate a first loss function according to the video frame of the first quality and the first output image, calculate a second loss function according to the video frame of the first quality, the second output image, and a result of the discriminator distinguishing the second output image from the video frame of the first quality, and calculate a third loss function according to a result of the discriminator distinguishing the authenticity of the second output image from the video frame of the first quality; and A parameter adjustment module is configured to: adjust the model parameters of the reconstructor, the generator and the discriminator respectively according to the first loss function, the second loss function and the third loss function to train the video super-resolution model; Wherein, the training device further comprises: A third acquisition module is configured to: process the video frame of the second quality through the video super-resolution model to obtain a third output image; The calculation module is further configured to: calculate a fourth loss function according to the result of the discriminator distinguishing the authenticity of the third output image and the video frame of the first quality; The parameter adjustment module is also configured to further adjust the model parameters of the reconstructor, generator and discriminator according to the fourth loss function to fine-tune the video super-resolution model.

19. The training device according to claim 18, characterized in that The calculation module is configured to calculate the second loss function by the following operations: Calculating an L1 loss function based on the second output image and the video frame of the first quality; Calculating an adversarial loss function according to a result of the discriminator distinguishing the authenticity of the second output image and the video frame of the first quality; calculating a perceptual loss function based on the second output image and the video frame of the first quality; and A weighted sum of the L1 loss function, the adversarial loss function, and the perceptual loss function is calculated as a second loss function.

20. The training device according to claim 18, characterized in that The calculation module is configured to calculate the fourth loss function by the following operations: calculating an L1 loss function based on the third output image and the video frame of the first quality; Calculating an adversarial loss function according to a result of the discriminator distinguishing the authenticity of the third output image and the video frame of the first quality; calculating a perceptual loss function based on the third output image and the video frame of the first quality; and A weighted sum of the L1 loss function, the adversarial loss function, and the perceptual loss function is calculated as a fourth loss function.

21. The training device according to claim 18, characterized in that The first acquisition module is configured to obtain a first output image by performing the following operations: extracting deep features from the video frames of the second quality by a reconstructor; The extracted deep features are processed by multiple convolution modules in the reconstructor to obtain processed deep features; The processed depth features are converted into the same color space as the video frame of the second quality by a reconstructor to obtain a first output image.

22. The training device according to claim 18, characterized in that The second acquisition module is configured to obtain a second output image by performing the following operations: extracting deep features from the first output image by a generator; The extracted deep features are processed by multiple convolution modules in the generator to obtain processed deep features; The processed depth features are converted into the same color space as the first output image by the generator and upsampled to obtain the second output image.

23. The training device according to claim 18, characterized in that The third acquisition module is configured to obtain a third output image by performing the following operations: Extracting depth features from the video frames of the second quality by the video super-resolution model; Processing the extracted depth features through multiple convolution modules in the video super-resolution model to obtain processed depth features; The processed depth features are converted into the same color space as the video frame of the second quality through the video super-resolution model and up-sampled to obtain a third output image.

24. A training device according to any one of claims 21 to 23, characterized in that The multiple convolution modules included in the video super-resolution model are connected to each other in series. The processed depth features are obtained by performing the following operations through the multiple convolution modules: each convolution module performs the following processing: transforming the input depth features, and adding the transformed results to the input depth features to obtain the output depth features, Among them, the depth feature of the input of the first convolution module is the extracted depth feature, and the depth feature of the output of the first convolution module is the depth feature of the input of the next convolution module, and the depth feature of the input of the last convolution module is the depth feature of the output of the previous convolution module, and the depth feature of the output of the last convolution module is the processed depth feature.

25. The training device according to claim 24, characterized in that The processed depth features are also obtained by performing the following operations through the multiple convolution modules: at least one convolution module other than the first convolution module among the multiple convolution modules also performs the following processing: the input depth features of at least one convolution module located before the current convolution module are added to the output depth features of the current convolution module to obtain the updated output depth features.

26. The training device according to claim 18, characterized in that The first loss function is the L1 loss function.

27. The training device according to claim 18, characterized in that The denoising process includes: removing noise and removing compression artifacts, and the super-resolution process includes: supplementing texture details.

28. A video super-resolution device, characterized in that: include: A first acquisition module is configured to: obtain a first output image by performing denoising on the reconstructed video frame using a reconstructor in a video super-resolution model trained by the training method according to any one of claims 1 to 11; The second acquisition module is configured to: perform super-resolution processing on the first output image through the generator in the video super-resolution model to obtain a second output image.

29. The video super-resolution device according to claim 28, characterized in that: The first acquisition module is configured to obtain a first output image by performing the following operations: extracting deep features from the video frames by a reconstructor; The extracted deep features are processed by multiple convolution modules in the reconstructor to obtain processed deep features; The processed depth features are converted into the same color space as the video frame by a reconstructor to obtain a first output image.

30. The video super-resolution device according to claim 28, characterized in that: The second acquisition module is configured to obtain a second output image by performing the following operations: extracting deep features from the first output image by a generator; The extracted deep features are processed by multiple convolution modules in the generator to obtain processed deep features; The processed depth features are converted into the same color space as the first output image by the generator and upsampled to obtain the second output image.

31. The video super-resolution device according to claim 28 or 29, characterized in that: The multiple convolution modules included in the video super-resolution model are connected to each other in series. The processed depth features are obtained by performing the following operations through the multiple convolution modules: each convolution module performs the following processing: transforming the input depth features, and adding the transformed results to the input depth features to obtain the output depth features, Among them, the depth feature of the input of the first convolution module is the extracted depth feature, and the depth feature of the output of the first convolution module is the depth feature of the input of the next convolution module, and the depth feature of the input of the last convolution module is the depth feature of the output of the previous convolution module, and the depth feature of the output of the last convolution module is the processed depth feature.

32. The video super-resolution device according to claim 31, characterized in that: The processed depth features are also obtained by performing the following operations through the multiple convolution modules: at least one convolution module other than the first convolution module among the multiple convolution modules also performs the following processing: the input depth features of at least one convolution module located before the current convolution module are added to the output depth features of the current convolution module to obtain the updated output depth features.

33. The video super-resolution device according to claim 28, characterized in that: The denoising process includes: removing noise and removing compression artifacts, and the super-resolution process includes: supplementing texture details.

34. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor, The processor is configured to execute the instructions to implement the video super-resolution model training method as described in any one of claims 1 to 11 or the video super-resolution method as described in any one of claims 12 to 17.

35. A computer-readable storage medium, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device / server, the electronic device / server is enabled to execute the training method of the video super-resolution model as described in any one of claims 1 to 11 or the video super-resolution method as described in any one of claims 12 to 17.

36. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the training method of the video super-resolution model as described in any one of claims 1 to 11 or the video super-resolution method as described in any one of claims 12 to 17 is implemented.

Images