Video quality improvement model training method and device
Through the video quality improvement model, using multi-stage generators and discriminators to process video frames, the problems of insufficient universality and insufficient diversity compression distortion processing capabilities in the existing technology are solved, and the video quality is significantly improved and user experience optimization is achieved.
Patent Information
- Application Number
- CN202110357426.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-04-01
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-04-01
AI Technical Summary
The existing video encoding standard devideo compression distortion algorithm is not very versatile and cannot effectively deal with the diverse video compression distortion problem.
Using a video quality improvement model, including a first-stage generator, a second-stage generator and a discriminator, by processing the first-quality video frame, dynamically adjusting the generator output image using random variables, and calculating multiple loss functions to Adjust model parameters and optimize video quality.
Effectively removes compression distortion, weakens the impact on smooth areas, and increases the details of texture-rich areas, optimizing the user experience.
Smart Images

Figure CN113012073B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of video processing, and in particular to a method and device for training a video quality improvement model, and a method and device for improving video quality. Background Art
[0002] In related technologies, lossy compression algorithms are usually used in the video transmission process, that is, the client video will have certain compression distortion compared to the original video, such as block effect, edge blur, detail loss, etc. To this end, a series of algorithms have been proposed to improve the distortion problem caused by video compression. Among them, some robust and efficient traditional algorithms have been adopted into the video coding framework and are widely used. However, the rapidly developing algorithms based on deep learning have achieved higher quality results.
[0003] Although the video compression distortion removal algorithms adopted by existing video coding standards (such as deblocking filters and sample adaptive offset filters) can improve compression distortion to some extent, these traditional algorithms have common shortcomings, such as low versatility and inability to handle diverse video compression distortions.
[0004] At present, with the rapid development of deep convolutional neural networks, more and more works use deep learning methods to deal with compression distortion problems. Summary of the invention
[0005] The present disclosure provides a video quality improvement model training method and device, as well as a video quality improvement method and device, to at least solve the problem of low versatility and inability to handle diverse video compression distortion in the above-mentioned related technologies, or may not solve any of the above-mentioned problems. The technical solution of the present disclosure is as follows:
[0006] According to a first aspect of an embodiment of the present disclosure, a training method for a video quality improvement model is provided, the video quality improvement model comprising a first-stage generator, a second-stage generator and a discriminator, wherein the training method comprises: the first-stage generator processes a video frame of a first quality to obtain a first output image; the second-stage generator obtains a second output image based on the first output image and a random variable; a first loss function is calculated based on the video frame of the second quality and the first output image; a loss function is calculated based on the video frame of the second quality, the second output image, and a result of a discriminator discriminating between the second output image and the video frame of the second quality, wherein the video frame of the first quality is obtained by encoding and decoding the video frame of the second quality, and the first quality is lower than the second quality; a third loss function is calculated based on a result of a discriminator discriminating between the second output image and the video frame of the second quality; and the model parameters of the first-stage generator, the second-stage generator and the discriminator are adjusted according to the first loss function, the second loss function and the third loss function to train the video quality improvement model.
[0007] Optionally, the step of calculating the second loss function may include: calculating the L1 loss function based on the second output image and the video frame of the second quality; calculating the adversarial loss function based on the discrimination result of the discriminator for the second output image; calculating the perceptual loss function based on the second output image and the video frame of the second quality; and calculating the weighted sum of the L1 loss function, the adversarial loss function and the perceptual loss function as the second loss function.
[0008] Optionally, the random variable corresponds to each pixel in the first output image, and changes dynamically with the texture of a region where each pixel is located.
[0009] Optionally, the step of obtaining a first output image may include: extracting depth features from the video frame of the first quality by the first stage generator; processing the extracted depth features by multiple convolution modules in the first stage generator to obtain processed depth features; converting the processed depth features to the same color space as the video frame of the first quality by the first stage generator to obtain a first residual image; and adding the first residual image to the video frame of the first quality by the first stage generator to obtain a first output image.
[0010] Optionally, the step of calculating the first loss function may include: calculating an L1 loss function as the first loss function according to the video frame of the second quality and the first output image.
[0011] Optionally, the step of obtaining a second output image may include: extracting depth features from the first output image by a second-stage generator; processing the random variables and the extracted depth features by multiple convolution modules in the second-stage generator to obtain processed depth features; converting the processed depth features to the same color space as the first output image by the second-stage generator to obtain a second residual image; and adding the second residual image to the first output image by the second-stage generator to obtain the second output image.
[0012] Optionally, the multiple convolution modules are connected to each other in series, wherein the step of processing the random variables and the extracted depth features by the multiple convolution modules in the second stage generator to obtain the processed depth features may include: each convolution module performs the following processing: splicing the input depth features with the random variables, transforming the splicing result, and adding the transformed result to the input depth features to obtain the output depth features, wherein the input depth features of the first convolution module are the extracted depth features, and the output depth features of the first convolution module are the input depth features of the next convolution module, and the input depth features of the last convolution module are the output depth features of the previous convolution module, and the output depth features of the last convolution module are the processed depth features.
[0013] Optionally, the step of calculating the third loss function may include: calculating an adversarial loss function as the third loss function according to a result of the discriminator discriminating the second output image and the video frame of the second quality.
[0014] Optionally, according to the second aspect of an embodiment of the present disclosure, a video quality improvement method is provided, comprising: a first-stage generator in a video quality improvement model processes a reconstructed video frame to reduce compression distortion in the reconstructed video frame to obtain a first output image; and a second-stage generator in the video quality improvement model processes the first output image and a random variable to obtain a second output image.
[0015] Optionally, the random variable corresponds to each pixel in the first output image, and changes dynamically with the texture of a region where each pixel is located.
[0016] Optionally, the step of obtaining a first output image may include: extracting depth features from the video frame by a first-stage generator; processing the extracted depth features by multiple convolution modules in the first-stage generator to obtain processed depth features; converting the processed depth features to the same color space as the video frame by the first-stage generator to obtain a first residual image; and adding the first residual image to the video frame by the first-stage generator to obtain a first output image.
[0017] Optionally, the step of obtaining a second output image may include: extracting depth features from the first output image by a second-stage generator; processing the random variables and the extracted depth features by multiple convolution modules in the second-stage generator to obtain processed depth features; converting the processed depth features to the same color space as the first output image by the second-stage generator to obtain a second residual image; and adding the second residual image to the first output image by the second-stage generator to obtain the second output image.
[0018] Optionally, the multiple convolution modules are connected to each other in series, wherein the step of processing the random variables and the extracted depth features by the multiple convolution modules in the second stage generator to obtain the processed depth features may include: each convolution module performs the following processing: splicing the input depth features with the random variables, transforming the splicing result, and adding the transformed result to the input depth features to obtain the output depth features, wherein the input depth features of the first convolution module are the extracted depth features, and the output depth features of the first convolution module are the input depth features of the next convolution module, and the input depth features of the last convolution module are the output depth features of the previous convolution module, and the output depth features of the last convolution module are the processed depth features.
[0019] Optionally, the video quality improvement model is trained using the training method described above.
[0020] According to a third aspect of an embodiment of the present disclosure, a training device for a video quality improvement model is provided, the video quality improvement model may include a first-stage generator, a second-stage generator and a discriminator, wherein the training device may include: a first obtaining unit, configured to: obtain a first output image by processing a video frame of a first quality through the first-stage generator; a second obtaining unit, configured to: obtain a second output image according to the first output image and a random variable through the second-stage generator; a calculating unit, configured to: calculate a first loss function according to the video frame of the second quality and the first output image; calculate a second loss function according to the video frame of the second quality, the second output image, and a result of the discriminator discriminating the second output image and the video frame of the second quality, and calculate a third loss function according to a result of the discriminator discriminating the second output image and the video frame of the second quality, wherein the video frame of the first quality is obtained by encoding and decoding the video frame of the second quality, and the first quality is lower than the second quality; and a parameter adjustment unit, configured to: adjust model parameters of the first-stage generator, the second-stage generator and the discriminator according to the first loss function, the second loss function and the third loss function to train the video quality improvement model.
[0021] Optionally, the computing unit may be configured to calculate the second loss function by: calculating the L1 loss function based on the second output image and the video frame of the second quality; calculating the adversarial loss function based on the discrimination result of the discriminator for the second output image; calculating the perceptual loss function based on the second output image and the video frame of the second quality; and calculating the weighted sum of the L1 loss function, the adversarial loss function and the perceptual loss function as the second loss function.
[0022] Optionally, the random variable corresponds to each pixel in the first output image, and changes dynamically with the texture of a region where each pixel is located.
[0023] Optionally, the first obtaining unit may be configured to obtain a first output image by: extracting depth features from the video frame of the first quality through a first stage generator; processing the extracted depth features through multiple convolution modules in the first stage generator to obtain processed depth features; converting the processed depth features to the same color space as the video frame of the first quality through the first stage generator to obtain a first residual image; and adding the first residual image to the video frame of the first quality through the first stage generator to obtain a first output image.
[0024] Optionally, the calculation unit may be configured to calculate an L1 loss function as the first loss function according to the video frame of the second quality and the first output image.
[0025] Optionally, the second obtaining unit may be configured to obtain a second output image by the following operations: extracting depth features from the first output image through a second-stage generator; processing the random variables and the extracted depth features through multiple convolution modules in the second-stage generator to obtain processed depth features; converting the processed depth features to the same color space as the first output image through the second-stage generator to obtain a second residual image; and adding the second residual image to the first output image through the second-stage generator to obtain the second output image.
[0026] Optionally, the multiple convolution modules are connected to each other in series, wherein the second obtaining unit may be configured to obtain the processed depth feature through the following operations, including: performing the following processing through each convolution module: splicing the input depth feature with the random variable, transforming the splicing result, and adding the transformed result to the input depth feature to obtain the output depth feature, wherein the input depth feature of the first convolution module is the extracted depth feature, and the output depth feature of the first convolution module is the input depth feature of the next convolution module, and the input depth feature of the last convolution module is the output depth feature of the previous convolution module, and the output depth feature of the last convolution module is the processed depth feature.
[0027] Optionally, the calculation unit may be configured to calculate an adversarial loss function as a third loss function according to a result of the discriminator discriminating the second output image and the video frame of the second quality.
[0028] According to a fourth aspect of an embodiment of the present disclosure, a video quality improvement device is provided, comprising: a first obtaining unit, configured to: obtain a first output image by processing a reconstructed video frame through a first stage generator in a video quality improvement model to reduce compression distortion in the reconstructed video frame; a second obtaining unit, configured to: obtain a second output image by processing the first output image and a random variable through a second stage generator in the video quality improvement model.
[0029] Optionally, the random variable corresponds to each pixel in the first output image, and changes dynamically with the texture of a region where each pixel is located.
[0030] Optionally, the first obtaining unit may be configured to obtain a first output image by: extracting depth features from the video frame through a first-stage generator; processing the extracted depth features through multiple convolution modules in the first-stage generator to obtain processed depth features; converting the processed depth features to the same color space as the video frame through the first-stage generator to obtain a first residual image; and adding the first residual image to the video frame through the first-stage generator to obtain a first output image.
[0031] Optionally, the second obtaining unit may be configured to obtain a second output image by the following operations: extracting depth features from the first output image through a second-stage generator; processing the random variables and the extracted depth features through multiple convolution modules in the second-stage generator to obtain processed depth features; converting the processed depth features to the same color space as the first output image through the second-stage generator to obtain a second residual image; and adding the second residual image to the first output image through the second-stage generator to obtain the second output image.
[0032] Optionally, the multiple convolution modules are connected to each other in series, wherein the second obtaining unit may be configured to obtain the processed depth feature by the following operation: performing the following processing through each convolution module: concatenating the input depth feature with the random variable, transforming the concatenated result, and adding the transformed result to the input depth feature to obtain the output depth feature, wherein the input depth feature of the first convolution module is the extracted depth feature, and the output depth feature of the first convolution module is the input depth feature of the next convolution module, and the input depth feature of the last convolution module is the output depth feature of the previous convolution module, and the output depth feature of the last convolution module is the processed depth feature.
[0033] Optionally, the video quality improvement model is trained using the training method described above.
[0034] According to a fifth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor, wherein the processor is configured to execute the instructions to implement the video quality improvement model training method as described above or the video quality improvement method as described above.
[0035] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device / server, the electronic device / server is enabled to execute the training method of the video quality improvement model as described above or the video quality improvement method as described above.
[0036] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: it removes compression distortion, avoids GAN from extracting texture features with compression distortion, weakens the influence of GAN on smooth areas, and increases detail information in texture-rich areas, thereby optimizing the user experience.
[0037] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0039] Figure 1 is an exemplary system architecture diagram in which exemplary embodiments of the present disclosure may be applied;
[0040] Figure 2 is a flow chart illustrating a method for training a video quality improvement model according to an exemplary embodiment of the present disclosure;
[0041] Figure 3 is a schematic diagram showing the training of a video quality improvement model according to an exemplary embodiment of the present disclosure;
[0042] Figure 4 is a structural schematic diagram showing a video quality improvement model according to an exemplary embodiment of the present disclosure;
[0043] Figure 5 is a schematic diagram showing the structure of a first-stage generator and a second-stage generator according to an exemplary embodiment of the present disclosure;
[0044] Figure 6 is a schematic diagram showing the structure of a discriminator according to an exemplary embodiment of the present disclosure;
[0045] Figure 7 is a flow chart showing a method for improving video quality according to an exemplary embodiment of the present disclosure;
[0046] Figure 8 is a block diagram illustrating a training apparatus for a video quality improvement model according to an exemplary embodiment of the present disclosure.
[0047] Fig. 9 is a block diagram illustrating a video quality improvement apparatus according to an exemplary embodiment of the present disclosure.
[0048] Fig.10 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0049] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.
[0050] It should be understood that the data used in this way can be interchanged where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0051] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.
[0052] As mentioned in the background technology of the present disclosure, in the related art, videos often use lossy compression algorithms for transmission and storage, which will cause the client's video to have certain compression distortion relative to the original video, and the existing traditional algorithms have the problem of low versatility and inability to handle diverse video compression distortion. Therefore, more and more works use deep learning algorithms to deal with the compression distortion problem, but the existing deep learning algorithms may retain or amplify the compression distortion when dealing with the compression distortion problem. For example, the GAN model scheme will misjudge the compression distortion as the texture of the image itself while increasing the texture details, thereby retaining or amplifying these defects, and the subjective quality is poor. In addition, the GAN model scheme uses the same weighted adversarial loss function for the entire image space during training, adding the same degree of texture to the global, so that the originally smooth area adds defects, thereby producing a negative effect. In view of this, the present disclosure proposes a two-stage model, first removing the compression distortion in the first stage, thereby avoiding GAN from extracting the texture features of the compression distortion, and then using a dynamic random method to reference random variables in the second stage to reduce the influence of GAN on the smooth area, while increasing the detail information of the texture-rich area.
[0053] Figure 1 is an exemplary system architecture diagram in which exemplary embodiments of the present disclosure may be applied.
[0054] like Figure 1As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. Users may use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages (such as video data upload requests, video data acquisition requests), etc. Various communication client applications may be installed on the terminal devices 101, 102, 103, such as video recording applications, video compression applications, video and audio editing applications, instant messaging tools, email clients, social platform software, etc. The terminal devices 101, 102, 103 may be hardware or software. When the terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with display screens that can play, record, and edit videos, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers. When the terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above, and can be implemented as multiple software or software modules (for example, to provide distributed services), or as a single software or software module. No specific limitation is made here.
[0055] The terminal devices 101, 102, and 103 may be installed with an image acquisition device (such as a camera) to collect video data. In addition, the terminal devices 101, 102, and 103 may also be installed with a component for converting electrical signals into sound (such as a speaker) to play sound, and may also be installed with a device for converting analog audio signals into digital audio signals (such as a microphone) to collect sound.
[0056] The terminal devices 101, 102, and 103 can use the image acquisition device installed thereon to acquire video data, and use the audio acquisition device installed thereon to acquire audio data, and the terminal devices 101, 102, and 103 can encode, store, and transmit the acquired video data and audio data, and can decode and play the encoded video and audio received from another terminal device or from the server 105.
[0057] Server 105 can be a server that provides various services, such as a background server that provides support for video recording applications, video compression applications, video editing applications, etc. installed on terminal devices 101, 102, 103, or it can be a storage server that stores encoded video and audio uploaded by terminal devices 101, 102, 103, and can send the stored encoded video and audio to terminal devices 101, 102, 103 in response to requests from terminal devices 101, 102, 103.
[0058] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.
[0059] It should be noted that the video quality improvement model training method and video quality improvement method provided in the embodiments of the present application are generally executed by terminal devices 101, 102, and 103. Accordingly, the video quality improvement model training device and video quality improvement device are generally arranged in terminal devices 101, 102, and 103.
[0060] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers, and the present disclosure has no limitation on this.
[0061] Figure 2 Detailed description is a flowchart illustrating a method for training a video quality improvement model according to an exemplary embodiment of the present disclosure. Figure 3 is a schematic diagram showing the training of a video quality improvement model 10 according to an exemplary embodiment of the present disclosure, wherein the video quality improvement model 10 includes a first-stage generator 100 , a second-stage generator 200 , and a discriminator 300 .
[0062] like Figure 2 As shown in, in step S210, the first stage generator 100 processes the video frame of the first quality to obtain the first output image. The video frame of the first quality represents a video frame obtained by encoding the video frame of the second quality according to a certain video codec and then decoding it. The video frame is a low-quality video frame relative to the video frame of the second quality, and the video frame of the second quality can be a video frame that has never been subjected to any encoding and decoding processing, or a video frame that has been encoded and decoded by a certain video codec. In other words, the video frame of the first quality and the video frame of the second quality are the same video frame with different video qualities. Figure 4 This is described in detail.
[0063] Specifically, the step of processing the first quality video frame by the first stage generator 100 to obtain the first output image may include: extracting depth features from the first quality video frame by the first stage generator 100, such as Figure 4 As shown in , the first stage generator 100 uses the first convolution layer 110 to perform a convolution operation on the video frame of the first quality to extract deep features.
[0064] The step of processing the first quality video frame by the first stage generator 100 to obtain the first output image may also include: processing the extracted depth features by a plurality of convolution modules 100-1 to 100-n in the first stage generator 100 to obtain processed depth features, wherein n is an integer greater than or equal to 1. Figure 4 As shown in , the multiple convolution modules 100-1 to 100-n are connected to each other in series, and each convolution module has the same internal structure, that is, each convolution module adopts the structure of "convolution-linear correction-convolution-residual connection". When the multiple convolution modules 100-1 to 100-n process the extracted depth features to obtain the processed depth features, each convolution module performs the same processing, that is: transforming the input value and adding the transformed result to the input value to obtain the output value. Specifically, each convolution module first performs a convolution operation on the input value, then performs a linear correction on the convolution result by a corrected linear unit, then performs a convolution operation on the linear correction result, and finally adds the convolution result to the input value to obtain the output value. Since the multiple convolution modules 100-1 to 100-n are connected in series, as shown in Figure 4 As shown in , the input value of the first convolution module is the depth feature extracted by the first convolution layer, and the output value of the first convolution module is the input value of the next convolution module, and so on. The input value of the last convolution module is the output value of the previous convolution module, and the output value of the last convolution module is the processed depth feature.
[0065] The step of processing the first quality video frame by the first stage generator 100 to obtain the first output image may also include: converting the processed deep features to the same color space as the first quality video frame by the first stage generator 100 to obtain a first residual image and adding the first residual image to the first quality video frame by the first stage generator to obtain the first output image, wherein the color space may be any one of RGB space, YUV space, etc. Specifically, if Figure 4As shown in, the first stage generator 100 can use the second convolution layer 120 to perform a convolution operation on the processed depth features output by the last convolution module to convert the processed depth features into the same color space as the first quality video frame to obtain a first residual image, and then the first stage generator 100 can add the first residual image to the first quality video frame through a long jump connection to obtain an intermediate result image, i.e., a first output image.
[0066] like Figure 4 As described in , the short jump connection within each convolution module adds the input value and output value of the current convolution module, while the long jump connection of the entire first-stage generator adds the video frame and the output values of all convolution modules. This allows the first-stage generator to better perform gradient backpropagation and accelerate model convergence.
[0067] Return to reference Figure 2 In step S220, the second stage generator 200 obtains the second output image according to the first output image and the random variable. Figure 5 This is described in detail.
[0068] Specifically, the step of obtaining the second output image by the second stage generator 200 according to the first output image and the random variable may include: extracting the depth feature from the first output image by the second stage generator 200. Figure 5 As shown in , the second stage generator 200 utilizes the third convolutional layer 210 to perform a convolution operation on the first output image to extract deep features.
[0069] The step of obtaining the second output image based on the first output image and the random variable by the second stage generator 200 may also include: multiple convolution modules 200-1 to 200-m in the second stage generator 200 process the random variables and the extracted depth features to obtain processed depth features, where m is an integer greater than or equal to 1.
[0070] like Figure 5 As shown in , the multiple convolution modules 200-1 to 200-m are connected to each other in series, and each convolution module has the same internal structure. When the multiple convolution modules 200-1 to 200-m in the second-stage generator 200 process the random variables and the extracted depth features to obtain the processed depth features, each convolution module performs the same processing, that is, concatenating the input depth features with the random variables, transforming the concatenated results, and adding the transformed results to the input depth features to obtain the output depth features. Specifically, as Figure 5As shown in , each convolution module first concatenates the input deep features with random variables, then performs a convolution operation on the concatenated result, and then uses a rectified linear unit to linearly correct the convolution result, and then performs a convolution operation on the linear corrected result, and finally adds the convolution result to the input deep features. In other words, after each convolution module concatenates the input deep features with random variables, it obtains the output deep features through the concatenation result through the structure of "convolution-linear correction-convolution-residual connection". Since multiple convolution modules 200-1 to 200-m are connected in series, as shown in FIG. Figure 5 As shown in , the depth feature of the input of the first convolution module is the depth feature extracted by the third convolution layer 210, and the depth feature of the output of the first convolution module is the depth feature of the input of the next convolution module, and so on. The depth feature of the input of the last convolution module is the depth feature of the output of the previous convolution module, and the depth feature of the output of the last convolution module is the processed depth feature.
[0071] In addition, the random variables used in the above process correspond to each pixel in the first output image and change dynamically with the texture of the area where each pixel is located. Specifically, the random variables used in the present disclosure correspond to each pixel in the first output image and obey a one-dimensional Gaussian distribution, wherein the random variables are calculated by the following equation:
[0072]
[0073] Among them, Z i,j represents the random variable corresponding to the pixel with coordinates (i, j) in the first output image, with mean 0, is the variance of the neighboring pixels adjacent to the pixel with coordinates (i, j), such as the variance of RGB values, the variance of YUV values, etc. In addition, Figure 5 As shown in , the variance map of the first output image can be calculated, where the variance value at the coordinate (i, j) in the variance map represents the variance of the neighborhood pixels adjacent to the pixel with the coordinate (i, j), and then the variance map is multiplied by the random variable N(0, 1) to obtain the random variable corresponding to the first output image.
[0074] In the above process, if Figure 5As shown in , the random variable is first spliced to the input deep features of the current convolution module in each convolution module, and then the current convolution module performs a series of operations on the splicing result, which enables the second-stage generator to make shallow features to deep features all affected by random perturbations, which can enhance the randomness of the entire network and make the result more detailed. In addition, since the variance of the random variable is not globally consistent, but dynamically changes according to the texture of the area where the pixel is located, the more complex the texture of the area where the pixel is located, the larger the variance of the neighboring pixels of the pixel will generally be, and the larger the variance of the one-dimensional Gaussian distribution of the corresponding random variable will be. Therefore, the randomness of the deep features at the corresponding position is also stronger. In relatively smooth areas (such as the sky, walls, etc.), the variance of the random variable is almost 0, that is, the random variable corresponding to the current pixel will change dynamically with the texture of the area where the current pixel is located, so too many artificial defects can be avoided.
[0075] The step of obtaining the second output image according to the first output image and the random variable by the second stage generator 200 may further include: converting the processed depth features to the same color space as the first output image by the second stage generator 200 to obtain a second residual image, and adding the second residual image to the first output image by the second stage generator 200 to obtain the second output image. Specifically, Figure 5 As shown in, the second stage generator 200 can use the fourth convolution layer 220 to perform a convolution operation on the processed depth features output by the last convolution module to convert the processed depth features into the same color space as the first output image (e.g., RGB space, YUV space, etc.) to obtain a second residual image, and then the second stage generator 200 can add the second residual image to the first output image through a long jump connection to obtain a second output image.
[0076] Return to reference Figure 2 In step S230, a first loss function is calculated according to the video frame of the second quality and the first output image. Specifically, an L1 loss function may be calculated according to the video frame of the second quality and the first output image as the first loss function. For example, the L1 loss function, i.e., the first loss function, may be calculated according to the following equation 1:
[0077]
[0078] in, represents the L1 loss function (ie, the first loss function), Output1 represents the first output image, and GT represents the video frame of the second quality.
[0079] In step S240, a second loss function is calculated based on the video frame of the second quality, the second output image, and the result of the discriminator discriminating the second output image and the video frame of the second quality. Step S240 will be described in detail below.
[0080] First, refer to Figure 6 As shown in , the discriminator 300 performs the following operations when discriminating the second quality video frame / second output image: first, a convolution operation is performed on the second quality video frame / second output image using a convolution layer, and then a linear correction is performed on the convolution result using a corrected linear unit, and then a series of "convolution-instance normalization-linear correction" operations are performed on the linear corrected result and a convolution operation is performed, so as to obtain a discrimination result of the second quality video frame / second output image.
[0081] The step of calculating the second loss function may include: calculating the L1 loss function according to the second output image and the video frame of the second quality; calculating the adversarial loss function according to the discrimination result of the discriminator 300 for the second output image; calculating the perceptual loss function according to the second output image and the video frame of the second quality; calculating the weighted sum of the L1 loss function, the adversarial loss function and the perceptual loss function as the second loss function. For example, the second loss function may be calculated according to the following equations 2 to 5:
[0082]
[0083]
[0084]
[0085]
[0086] in, represents the second loss function, represents an L1 loss function calculated based on the second output image and the video frame of the second quality, represents the adversarial loss function calculated according to the discriminator's discrimination result for the second output image, represents the perceptual loss function calculated based on the second output image and the video frame of the second quality, α 1 and α 2 is a preset value, Output2 represents the second output image, GT represents the video frame of the second quality, D() represents the discriminator's discriminant result, represents the normalization factor of the feature network φ, represents the Euclidean distance. In addition, the adversarial loss function In addition to using the above equation (4) to calculate, other loss functions that improve the traditional GAN loss function to make the GAN training stable can also be used, for example, the WGAN loss function, the WGAN-GP loss function, the DRAGAN loss function, etc. In addition, if the loss function in equation (4) is replaced by another adversarial loss function, the weight α corresponding to the adversarial loss function in equation (2) needs to be adjusted accordingly. 1 .
[0087] In step S250, a third loss function is calculated according to the result of the discriminator discriminating the second output image and the video frame of the second quality. For example, the third loss function may be calculated according to the following equation (6):
[0088]
[0089] in, Represents the third loss function.
[0090] In step S260, the model parameters of the first-stage generator, the second-stage generator, and the discriminator are adjusted respectively according to the first loss function, the second loss function, and the third loss function to train the video quality improvement model.
[0091] Specifically, the step of training the video quality improvement model may include: adjusting the model parameters of the first stage generator according to the first loss function to train the first stage generator. Specifically, using the first loss function calculated as above can effectively remove block distortion in the compressed video.
[0092] In addition, the step of training the video quality improvement model may further include: adjusting the model parameters of the second stage generator 200 according to the second loss function to train the second stage generator 200. Specifically, using the loss function calculated according to equation (3), the pixel level accuracy can be guaranteed, and using the loss function calculated according to equation (4), the shortcomings of gradient diffusion and training instability can be overcome, thereby obtaining certain improvements in effect and stability, and using the loss function calculated according to equation (5), the consistency of semantic features can be guaranteed, and the above second loss function adopts the weighted sum form of these three loss functions, so that the above effects can be obtained at the same time.
[0093] In addition, the step of training the video quality improvement model may further include training the discriminator 300 by adjusting the model parameters of the discriminator according to the third loss function.
[0094] Figure 7 is a flowchart illustrating a video quality improvement method according to an exemplary embodiment of the present disclosure.
[0095] Reference Figure 7 In step S710, the first stage generator 100 in the video quality improvement model 10 processes the reconstructed video frame to reduce the compression distortion in the reconstructed video frame to obtain a first output image. The reconstructed video frame is a video frame obtained by encoding the original video according to a certain video codec and then decoding it.
[0096] Specifically, the step of obtaining the first output image may include: extracting depth features from the video frame by the first stage generator 100. Figure 4 As shown in , the first stage generator 100 uses the first convolutional layer 110 to perform a convolution operation on the video frame to extract deep features.
[0097] The step of obtaining the first output image may further include: processing the extracted depth features by a plurality of convolution modules 100-1 to 100-n in the first stage generator 100 to obtain processed depth features, where n is an integer greater than or equal to 1. Figure 4 As shown in , the multiple convolution modules 100-1 to 100-n are connected to each other in series, and each convolution module has the same internal structure, that is, each convolution module adopts the structure of "convolution-rectified linear unit-convolution-residual connection". When the multiple convolution modules 100-1 to 100-n process the extracted depth features to obtain the processed depth features, each convolution module performs the same processing, that is: transforming the input value and adding the transformed result to the input value to obtain the output value. Specifically, each convolution module first performs a convolution operation on the input value, and then the convolution result is linearly corrected by the corrected linear unit, and then a convolution operation is performed on the linear corrected result, and finally the convolution result is added to the input value to obtain the output value. Since the multiple convolution modules 100-1 to 100-n are connected in series, as shown in Figure 4 As shown in , the input value of the first convolution module is the depth feature extracted by the first convolution layer, and the output value of the first convolution module is the input value of the next convolution module, and so on. The input value of the last convolution module is the output value of the previous convolution module, and the output value of the last convolution module is the processed depth feature.
[0098] The step of obtaining the first output image may further include: the first stage generator 100 converts the processed depth features into the same color space as the video frame to obtain a first residual image, wherein the color space may be any one of RGB space, YUV space, etc. Specifically, Figure 4As shown in, the first stage generator 100 can use the second convolution layer 120 to perform a convolution operation on the processed depth features output by the last convolution module to convert the processed depth features into the same color space as the video frame (e.g., RGB space) to obtain a first residual image, and then the first stage generator 100 can add the first residual image to the video frame through a long jump connection to obtain an intermediate result image, i.e., a first output image.
[0099] The step of obtaining the first output image may further include: the first stage generator 100 adds the first residual image to the video frame to obtain the first output image.
[0100] In step S720, the second stage generator 200 in the video quality improvement model 10 processes the first output image and the random variable to obtain a second output image.
[0101] Specifically, the step of obtaining the second output image may include: extracting depth features from the first output image by the second stage generator 200. Figure 5 As shown in , the second stage generator 200 utilizes the third convolutional layer 210 to perform a convolution operation on the first output image to extract deep features.
[0102] The step of obtaining the second output image may also include: processing the random variables and the extracted depth features by multiple convolution modules 200-1 to 200-m in the second stage generator 200 to obtain processed depth features, where m is an integer greater than or equal to 1.
[0103] like Figure 5 As shown in , the multiple convolution modules 200-1 to 200-m are connected to each other in series, and each convolution module has the same internal structure. When the multiple convolution modules 200-1 to 200-m in the second-stage generator 200 process the random variables and the extracted depth features to obtain the processed depth features, each convolution module performs the same processing, that is, concatenating the input depth features with the random variables, transforming the concatenated results, and adding the transformed results to the input depth features to obtain the output depth features. Specifically, as Figure 5As shown in , each convolution module first concatenates the input deep features with random variables, then performs a convolution operation on the concatenated result, then uses a rectified linear unit to linearly correct the convolution result, then performs a convolution operation on the linear corrected result, and finally adds the convolution result to the input deep features. In other words, after each convolution module concatenates the input deep features with random variables, it obtains the output deep features through the concatenation result through the structure of "convolution-rectified linear unit-convolution-residual connection". Since multiple convolution modules 200-1 to 200-m are connected in series, as shown in FIG. Figure 5 As shown in , the depth feature of the input of the first convolution module is the depth feature extracted by the third convolution layer 210, and the depth feature of the output of the first convolution module is the depth feature of the input of the next convolution module, and so on. The depth feature of the input of the last convolution module is the depth feature of the output of the previous convolution module, and the depth feature of the output of the last convolution module is the processed depth feature.
[0104] In addition, the random variables used in the above process correspond to each pixel in the first output image and change dynamically with the texture of the area where each pixel is located. Specifically, the random variables used in the present disclosure correspond to each pixel in the first output image and obey a one-dimensional Gaussian distribution, wherein the random variables are calculated by the following equation:
[0105]
[0106] Among them, Z i,j represents the random variable corresponding to the pixel with coordinates (i, j) in the first output image, with mean 0, is the variance of the neighboring pixels adjacent to the pixel with coordinates (i, j), such as the variance of RGB values, the variance of YUV values, etc. Figure 5 This has been described in detail and will not be repeated here.
[0107] The step of obtaining the second output image may further include: converting the processed depth features to the same color space as the first output image by the second stage generator 200 to obtain a second residual image, and adding the second residual image to the first output image by the second stage generator 200 to obtain the second output image. Specifically, Figure 5As shown in, the second stage generator 200 can use the fourth convolution layer 220 to perform a convolution operation on the processed depth features output by the last convolution module to convert the processed depth features into the same color space as the first output image (e.g., RGB space, YUV space, etc.) to obtain a second residual image, and then the second stage generator 200 can add the second residual image to the first output image through a long jump connection to obtain a second output image.
[0108] At this point, the final output image can be obtained through the above process. The output image not only removes the compression distortion, avoids GAN from extracting the texture features of compression distortion, but also weakens the influence of GAN on the smooth area, and increases the detail information of the texture-rich area.
[0109] Figure 8 is a block diagram illustrating a training device 800 for a video quality improvement model according to an exemplary embodiment of the present disclosure, wherein the video quality improvement model includes a first-stage generator 100 , a second-stage generator 200 , and a discriminator 300 .
[0110] like Figure 8 As shown in , the training device 800 may include a first obtaining unit 810, a second obtaining unit 820, a calculating unit 830 and a parameter adjusting unit 840.
[0111] The first obtaining unit 810 may be configured to obtain a first output image by processing the video frame of the first quality through the first stage generator 100 .
[0112] Specifically, the first obtaining unit 810 is configured to obtain a first output image through the following operations: extracting depth features from the video frame of the first quality through the first stage generator 100; processing the extracted depth features through multiple convolution modules 100-1 to 100-n in the first stage generator 100 to obtain processed depth features, where n is an integer greater than or equal to 1; converting the processed depth features to the same color space as the video frame of the first quality through the first stage generator 100 to obtain a first residual image; adding the first residual image to the video frame of the first quality through the first stage generator 100 to obtain a first output image.
[0113] The multiple convolution modules 100-1 to 100-n are connected to each other in series, wherein the first obtaining unit 810 is configured to obtain the processed depth feature through the following operations, including: performing the following processing through each convolution module: transforming the input value and adding the transformed result to the input value to obtain the output value, wherein the input value of the first convolution module is the extracted depth feature, and the output value of the first convolution module is the input value of the next convolution module, the input value of the last convolution module is the output value of the previous convolution module, and the output value of the last convolution module is the processed depth feature.
[0114] The second obtaining unit 820 may be configured to obtain a second output image according to the first output image and the random variable through a second stage generator.
[0115] Specifically, the second obtaining unit 820 can be configured to obtain a second output image through the following operations: extracting depth features from the first output image through the second stage generator 200; processing the random variables and the extracted depth features through multiple convolution modules 200-1 to 200-m in the second stage generator 200 to obtain processed depth features, where m is an integer greater than or equal to 1; converting the processed depth features to the same color space as the first output image through the second stage generator 200 to obtain a second residual image; and adding the second residual image to the first output image through the second stage generator 200 to obtain the second output image.
[0116] The multiple convolution modules are connected to each other in series, wherein the second obtaining unit 820 is configured to obtain the processed depth feature through the following operations, including: performing the following processing through each convolution module: splicing the input depth feature with the random variable, transforming the splicing result, and adding the transformed result to the input depth feature to obtain the output depth feature, wherein the input depth feature of the first convolution module is the extracted depth feature, and the output depth feature of the first convolution module is the input depth feature of the next convolution module, and the input depth feature of the last convolution module is the output depth feature of the previous convolution module, and the output depth feature of the last convolution module is the processed depth feature.
[0117] The random variable used in the above process corresponds to each pixel in the first output image and changes dynamically with the texture of the area where each pixel is located. Specifically, the random variable corresponds to each pixel in the first output image and obeys a one-dimensional Gaussian distribution, wherein the random variable is calculated by the following equation:
[0118]
[0119] Among them, Z i,j represents the random variable corresponding to the pixel with coordinates (i, j) in the first output image, is the variance of the neighboring pixels adjacent to the pixel with coordinates (i, j).
[0120] The calculation unit 830 can be configured to: calculate a first loss function based on the video frame of the second quality and the first output image, calculate a second loss function based on the video frame of the second quality, the second output image, and the result of the discriminator 300 discriminating the second output image and the video frame of the second quality, and calculate a third loss function based on the result of the discriminator discriminating the second output image and the video frame of the second quality, wherein the video frame of the first quality is obtained by encoding and decoding the video frame of the second quality, and the first quality is lower than the second quality.
[0121] Specifically, the calculation unit 830 may be configured to calculate the L1 loss function as the first loss function according to the second quality video frame and the first output image, that is, the first loss function may be calculated according to the above equation (1). In addition, the calculation unit 830 may be configured to calculate the second loss function by the following operations: calculating the L1 loss function according to the second output image and the second quality video frame; calculating the adversarial loss function according to the discrimination result of the discriminator 300 for the second output image; calculating the perceptual loss function according to the second output image and the second quality video frame; and using the weighted sum of the L1 loss function, the GAN loss function and the perceptual loss function as the second loss function, that is, the second loss function may be calculated according to the above equations (2) to (5). In addition, the calculation unit 830 may be configured to calculate the adversarial loss function as the third loss function according to the discrimination result of the discriminator 300 for the second output image and the second quality video frame, that is, the third loss function may be calculated according to the above equation (6).
[0122] The parameter adjustment unit 840 can be configured to adjust the model parameters of the first-stage generator, the second-stage generator and the discriminator according to the first loss function, the second loss function and the third loss function respectively to train the video quality improvement model.
[0123] Specifically, the parameter adjustment unit 840 is configured to train the video quality improvement model through the following operations: adjusting the model parameters of the first-stage generator 100 according to the first loss function to train the first-stage generator 100; adjusting the model parameters of the second-stage generator 200 according to the second loss function to train the second-stage generator 200; and adjusting the model parameters of the discriminator according to the third loss function to train the discriminator 300.
[0124] because Figure 2 The training method shown can be Figure 8 The training device 800 shown is used to perform, therefore, Figure 8 Any details on the operations performed by the units in Figure 2 The corresponding description will not be repeated here.
[0125] Fig. 9 is a block diagram illustrating a video quality improving apparatus 900 according to an exemplary embodiment of the present disclosure.
[0126] Reference Fig. 9 , the video quality improvement device 900 may include a first obtaining unit 910 and a second obtaining unit 920.
[0127] The first obtaining unit 910 may be configured to obtain a first output image by processing the reconstructed video frame through the first stage generator in the video quality improvement model to reduce the compression distortion in the reconstructed video frame.
[0128] Specifically, the first obtaining unit 910 can be configured to obtain a first output image through the following operations: extracting depth features from the video frame through the first stage generator 100; processing the extracted depth features through multiple convolution modules 100-1 to 100-n in the first stage generator to obtain processed depth features, where n is an integer greater than or equal to 1; converting the processed depth features to the same color space as the video frame through the first stage generator 100 to obtain a first residual image; and adding the first residual image to the video frame through the first stage generator 100 to obtain a first output image.
[0129] The multiple convolution modules 100-1 to 100-n are connected to each other in series, wherein the first obtaining unit 910 is configured to obtain the processed depth feature by the following operation: the following processing is performed through each convolution module: the input value is transformed and the transformed result is added to the input value to obtain the output value, wherein the input value of the first convolution module is the extracted depth feature, and the output value of the first convolution module is the input value of the next convolution module, the input value of the last convolution module is the output value of the previous convolution module, and the output value of the last convolution module is the processed depth feature.
[0130] The second obtaining unit 920 may be configured to obtain a second output image by processing the first output image and the random variable through a second stage generator in the video quality improvement model.
[0131] Specifically, the second obtaining unit 920 can be configured to obtain a second output image through the following operations: extracting depth features from the first output image through the second stage generator; processing the random variables and the extracted depth features through multiple convolution modules 200-1 to 200-m in the second stage generator to obtain processed depth features, where m is an integer greater than or equal to 1; converting the processed depth features to the same color space as the first output image through the second stage generator to obtain a second residual image; and adding the second residual image to the first output image through the second stage generator to obtain the second output image.
[0132] The multiple convolution modules 200-1 to 200-m are connected to each other in series, wherein the second obtaining unit 920 is configured to obtain the processed depth feature by the following operation: the following processing is performed through each convolution module: the input depth feature is spliced with the random variable, the spliced result is transformed, and the transformed result is added to the input depth feature to obtain the output depth feature, wherein the input depth feature of the first convolution module is the extracted depth feature, and the output depth feature of the first convolution module is the input depth feature of the next convolution module, and the input depth feature of the last convolution module is the output depth feature of the previous convolution module, and the output depth feature of the last convolution module is the processed depth feature.
[0133] The random variable used in the above process corresponds to each pixel in the first output image and changes dynamically with the texture of the area where each pixel is located. Specifically, the random variable corresponds to each pixel in the first output image and obeys a one-dimensional Gaussian distribution, wherein the random variable is calculated by the following equation:
[0134]
[0135] where Zi,j represents the random variable corresponding to the pixel with coordinates (i,j) in the first output image, is the variance of the neighboring pixels adjacent to the pixel with coordinates (i, j).
[0136] because Figure 7 The video quality improvement method shown can be performed by Fig. 9 The video quality improvement device 900 shown in FIG. Fig. 9 Any details on the operations performed by the units in Figure 7 The corresponding description will not be repeated here.
[0137] Fig.101 is a block diagram showing an electronic device 1000 according to an embodiment of the present disclosure. The electronic device 1000 may include a memory 1010 and a processor 1020. A set of computer executable instructions is stored in the memory 1010. When the computer executable instruction set is executed by the processor, a training method for a video quality improvement model or a video quality improvement method according to an embodiment of the present disclosure is executed.
[0138] As an example, the electronic device may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instruction set. Here, the electronic device is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above instruction (or instruction set) individually or in combination. The electronic device may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.
[0139] In electronic devices, the processor may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.
[0140] The processor can execute instructions or codes stored in the memory, wherein the memory can also store data. Instructions and data can also be sent and received through the network via the network interface device, wherein the network interface device can adopt any known transmission protocol.
[0141] The memory may be integrated with the processor, for example, RAM or flash memory is arranged within an integrated circuit microprocessor or the like. In addition, the memory may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory and the processor may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor can read files stored in the memory.
[0142] In addition, the electronic device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device may be connected to each other via a bus and / or a network.
[0143] According to an embodiment of the present disclosure, a computer-readable storage medium may also be provided, for example, a memory 1020 including instructions, and the instructions may be executed by the processor 1010 of the electronic device 1000 to complete the above-mentioned video quality improvement model training method or video quality improvement method. Optionally, the computer-readable storage medium may be: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.
[0144] In an exemplary embodiment of the present disclosure, a computer program product may also be provided, including a computer program / instruction, which, when executed by a processor, implements a video quality improvement model training method or a video quality improvement method according to an exemplary embodiment of the present disclosure.
[0145] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0146] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A training method for a video quality improvement model, characterized in that: The video quality improvement model includes a first-stage generator, a second-stage generator and a discriminator, wherein the training method includes: The first stage generator processes the video frame of the first quality to obtain a first output image; The second stage generator obtains a second output image according to the first output image and the random variable; Calculating a first loss function according to a video frame of a second quality and a first output image, wherein the video frame of the first quality is obtained by encoding and decoding the video frame of the second quality; Calculate a second loss function according to the video frame of the second quality, the second output image, and a result of the discriminator discriminating the second output image and the video frame of the second quality; Calculating a third loss function according to a result of the discriminator discriminating the second output image and the video frame of the second quality; The video quality improvement model is trained by adjusting the model parameters of the first-stage generator, the second-stage generator, and the discriminator according to the first loss function, the second loss function, and the third loss function, respectively. The step of obtaining the first output image comprises: extracting, by a first stage generator, deep features from the video frames of the first quality; The extracted deep features are processed by multiple convolution modules in the first-stage generator to obtain processed deep features; The processed deep features are converted by the first stage generator into a color space that is the same as the video frame of the first quality to obtain a first residual image; The first stage generator adds the first residual image to the video frame of the first quality to obtain a first output image.
2. The training method according to claim 1, characterized in that: The steps of calculating the second loss function include: Calculating an L1 loss function based on the second output image and the video frame of the second quality; Calculating the adversarial loss function according to the discrimination result of the discriminator for the second output image; calculating a perceptual loss function based on the second output image and the video frame of the second quality; and The weighted sum of the L1 loss function, the adversarial loss function, and the perceptual loss function is calculated as the second loss function.
3. The training method according to claim 1, characterized in that: The random variable corresponds to each pixel in the first output image and changes dynamically with the texture of the region where each pixel is located.
4. The training method according to claim 1, characterized in that: The step of calculating the first loss function includes: calculating an L1 loss function as the first loss function according to the video frame of the second quality and the first output image.
5. The training method according to claim 1, characterized in that: The step of obtaining a second output image comprises: extracting deep features from the first output image by the second stage generator; The random variables and the extracted deep features are processed by a plurality of convolution modules in the second-stage generator to obtain processed deep features; The processed deep features are converted by the second stage generator into the same color space as the first output image to obtain a second residual image; The second residual image is added to the first output image by the second stage generator to obtain the second output image.
6. The training method according to claim 5, characterized in that: The multiple convolution modules are connected to each other in series. The step of processing the random variable and the extracted deep features by multiple convolution modules in the second stage generator to obtain the processed deep features includes: each convolution module performs the following processing: splicing the input deep features with the random variable, transforming the splicing result, and adding the transformed result to the input deep features to obtain the output deep features, Among them, the depth feature of the input of the first convolution module is the extracted depth feature, and the depth feature of the output of the first convolution module is the depth feature of the input of the next convolution module, and the depth feature of the input of the last convolution module is the depth feature of the output of the previous convolution module, and the depth feature of the output of the last convolution module is the processed depth feature.
7. The training method according to claim 1, characterized in that: The steps for calculating the third loss function include: An adversarial loss function is calculated as a third loss function according to a result of the discriminator discriminating the second output image and the video frame of the second quality.
8. A method for improving video quality, characterized in that: include: The first stage generator in the video quality improvement model processes the reconstructed video frame to reduce the compression distortion in the reconstructed video frame to obtain a first output image; The second stage generator in the video quality improvement model processes the first output image and the random variable to obtain a second output image, The video quality improvement model is trained using the training method described in any one of claims 1 to 7.
9. The video quality improvement method according to claim 8, characterized in that: The random variable corresponds to each pixel in the first output image and changes dynamically with the texture of the region where each pixel is located.
10. The video quality improvement method according to claim 8, characterized in that: The step of obtaining a first output image comprises: Extracting deep features from the video frame by a first-stage generator; The extracted deep features are processed by multiple convolution modules in the first-stage generator to obtain processed deep features; The processed deep features are converted by the first stage generator into a color space identical to the video frame to obtain a first residual image; The first stage generator adds the first residual image to the video frame to obtain a first output image.
11. The video quality improvement method according to claim 8, characterized in that: The step of obtaining a second output image comprises: extracting deep features from the first output image by the second stage generator; The random variables and the extracted deep features are processed by a plurality of convolution modules in the second-stage generator to obtain processed deep features; The processed deep features are converted by the second stage generator into the same color space as the first output image to obtain a second residual image; The second residual image is added to the first output image by the second stage generator to obtain the second output image.
12. The video quality improvement method according to claim 11, characterized in that: The multiple convolution modules are connected to each other in series. The step of processing the random variable and the extracted deep features by multiple convolution modules in the second stage generator to obtain the processed deep features includes: each convolution module performs the following processing: splicing the input deep features with the random variable, transforming the splicing result, and adding the transformed result to the input deep features to obtain the output deep features, Among them, the depth feature of the input of the first convolution module is the extracted depth feature, and the depth feature of the output of the first convolution module is the depth feature of the input of the next convolution module, and the depth feature of the input of the last convolution module is the depth feature of the output of the previous convolution module, and the depth feature of the output of the last convolution module is the processed depth feature.
13. A training device for a video quality improvement model, characterized in that: The video quality improvement model includes a first-stage generator, a second-stage generator and a discriminator, wherein the training device includes: A first obtaining unit is configured to: obtain a first output image by processing a video frame of a first quality through a first stage generator; A second obtaining unit is configured to: obtain a second output image according to the first output image and the random variable through a second stage generator; a calculation unit configured to: calculate a first loss function according to a video frame of a second quality and a first output image, calculate a second loss function according to the video frame of the second quality, the second output image, and a result of a discriminator discriminating the second output image and the video frame of the second quality, and calculate a third loss function according to a result of a discriminator discriminating the second output image and the video frame of the second quality, wherein the video frame of the first quality is obtained by encoding and decoding the video frame of the second quality; and A parameter adjustment unit is configured to: adjust the model parameters of the first-stage generator, the second-stage generator, and the discriminator respectively according to the first loss function, the second loss function, and the third loss function to train the video quality improvement model, The first obtaining unit is configured to obtain the first output image by performing the following operations: extracting deep features from the video frames of the first quality by a first stage generator; The extracted deep features are processed by multiple convolution modules in the first-stage generator to obtain processed deep features; Converting the processed deep features to the same color space as the first quality video frame by a first stage generator to obtain a first residual image; A first output image is obtained by adding a first residual image to the video frame of the first quality through a first stage generator.
14. The training device according to claim 13, characterized in that The computing unit is configured to calculate the second loss function by the following operations: Calculating an L1 loss function based on the second output image and the video frame of the second quality; Calculating the adversarial loss function according to the discrimination result of the discriminator for the second output image; calculating a perceptual loss function based on the second output image and the video frame of the second quality; and The weighted sum of the L1 loss function, the adversarial loss function, and the perceptual loss function is calculated as the second loss function.
15. The training device according to claim 13, characterized in that The random variable corresponds to each pixel in the first output image and changes dynamically with the texture of the region where each pixel is located.
16. The training device according to claim 13, characterized in that The calculation unit is configured to calculate an L1 loss function as a first loss function according to the video frame of the second quality and the first output image.
17. The training device according to claim 13, characterized in that The second obtaining unit is configured to obtain a second output image by: extracting deep features from the first output image via a second stage generator; Processing the random variables and the extracted deep features through a plurality of convolution modules in a second-stage generator to obtain processed deep features; Converting the processed deep features to the same color space as the first output image by a second stage generator to obtain a second residual image; The second residual image is added to the first output image by the second stage generator to obtain the second output image.
18. The training device according to claim 17, characterized in that The multiple convolution modules are connected to each other in series. The second obtaining unit is configured to obtain the processed depth feature by the following operation, comprising: performing the following processing through each convolution module: splicing the input depth feature with the random variable, transforming the splicing result, and adding the transformed result to the input depth feature to obtain the output depth feature, Among them, the depth feature of the input of the first convolution module is the extracted depth feature, and the depth feature of the output of the first convolution module is the depth feature of the input of the next convolution module, and the depth feature of the input of the last convolution module is the depth feature of the output of the previous convolution module, and the depth feature of the output of the last convolution module is the processed depth feature.
19. The training device according to claim 13, characterized in that The calculation unit is configured to calculate an adversarial loss function as a third loss function according to a result of the discriminator discriminating the second output image and the video frame of the second quality.
20. A video quality improvement device, characterized in that: include: A first obtaining unit is configured to: obtain a first output image by processing the reconstructed video frame through the first stage generator in the video quality improvement model to reduce the compression distortion in the reconstructed video frame; The second obtaining unit is configured to: obtain a second output image by processing the first output image and the random variable through the second stage generator in the video quality improvement model, The video quality improvement model is trained using the training method described in any one of claims 1 to 7.
21. The video quality improvement device according to claim 20, characterized in that: The random variable corresponds to each pixel in the first output image and changes dynamically with the texture of the region where each pixel is located.
22. The video quality improvement device according to claim 20, characterized in that: The first obtaining unit is configured to obtain a first output image by performing the following operations: Extracting deep features from the video frames via a first stage generator; The extracted deep features are processed by multiple convolution modules in the first-stage generator to obtain processed deep features; Converting the processed deep features to the same color space as the video frame by a first stage generator to obtain a first residual image; A first residual image is added to the video frame by a first stage generator to obtain a first output image.
23. The video quality improvement device according to claim 20, characterized in that: The second obtaining unit is configured to obtain a second output image by: extracting deep features from the first output image via a second stage generator; Processing the random variables and the extracted deep features through a plurality of convolution modules in a second-stage generator to obtain processed deep features; Converting the processed deep features to the same color space as the first output image by a second stage generator to obtain a second residual image; The second residual image is added to the first output image by the second stage generator to obtain the second output image.
24. The video quality improvement device according to claim 23, characterized in that: The multiple convolution modules are connected to each other in series. The second obtaining unit is configured to obtain the processed depth feature by the following operations: performing the following processing through each convolution module: concatenating the input depth feature with the random variable, transforming the concatenated result, and adding the transformed result to the input depth feature to obtain the output depth feature, Among them, the depth feature of the input of the first convolution module is the extracted depth feature, and the depth feature of the output of the first convolution module is the depth feature of the input of the next convolution module, and the depth feature of the input of the last convolution module is the depth feature of the output of the previous convolution module, and the depth feature of the output of the last convolution module is the processed depth feature.
25. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor, The processor is configured to execute the instructions to implement the video quality improvement model training method as described in any one of claims 1 to 7 or the video quality improvement method as described in any one of claims 8 to 12.
26. A computer-readable storage medium, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device / server, the electronic device / server is enabled to execute the video quality improvement model training method as described in any one of claims 1 to 7 or the video quality improvement method as described in any one of claims 8 to 12.
27. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method for training a video quality improvement model according to any one of claims 1 to 7 or the method for improving video quality according to any one of claims 8 to 12 is implemented.
