Video image quality enhancement method and device, equipment and storage medium

Abstract

The invention provides a video image quality enhancement method and device, equipment and a storage medium. The method comprises the following steps: acquiring a low-quality initial video; acquiring a key frame of the initial video; compressing the key frame into a low-quality implicit code through a diffusion model, obtaining a noise implicit code of each time step according to the low-quality implicit code, and generating a high-quality implicit code according to the noise implicit code of each time step; enhancing the high-quality implicit code and the low-quality implicit code to obtain an enhanced high-quality implicit code and an enhanced low-quality implicit code; and generating a high-quality video according to the enhanced high-quality implicit code and the enhanced low-quality implicit code. According to the method, after the low-quality initial video is obtained, the low-quality implicit code is obtained through the key frame of the initial video, then the high-quality implicit code is generated based on the noise implicit code of each time step obtained by the low-quality implicit code, and the high-quality video is generated after the high-quality implicit code and the low-quality implicit code are enhanced. The high-quality video not only contains fine-grained details, but also needs to maintain the continuity of inter-frame details.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, apparatus, device, and storage medium for enhancing video quality. Background Art

[0002] Video quality enhancement technology aims to improve video quality by analyzing and processing video pixel information.

[0003] Existing video quality enhancement technologies mainly use sliding window methods or recursive methods to aggregate information of low-resolution video frames using convolutional neural networks or Transformers to predict high-resolution frames.

[0004] While such methods perform well in upscaling original visual content, they have limitations in generating novel fine-grained details. Summary of the Invention

[0005] In order to solve one of the above technical defects, the present application provides a video quality enhancement method, device, equipment, and storage medium.

[0006] In a first aspect, the present application provides a method for enhancing video quality, the method comprising:

[0007] Get low-quality initial video;

[0008] Get the key frames of the initial video;

[0009] The key frame is compressed into a low-quality hidden code through the diffusion model, and the noise hidden code of each time step is obtained based on the low-quality hidden code, and the high-quality hidden code is generated based on the noise hidden code of each time step;

[0010] The high-quality hidden code and the low-quality hidden code are enhanced to obtain an enhanced high-quality hidden code and an enhanced low-quality hidden code;

[0011] A high-quality video is generated based on the enhanced high-quality codec and the enhanced low-quality codec.

[0012] Optionally, the noise hidden code at any time step t

[0013] Among them, α t and σ t is a hyperparameter, ∈ is Gaussian noise.

[0014] Optionally, a high-quality hidden code is generated based on the noisy hidden code at each time step, including:

[0015] Generate high-quality latent codes through iterative denoising at multiple time steps;

[0016] Among them, through the formula Perform denoising at any time step t;

[0017] z t To calculate the hidden code of the image after denoising at time step t through the diffusion model, is the noise hidden code at time step t, γ t is the denoising weight.

[0018] Optionally,

[0019] Where T is the maximum time step.

[0020] Optionally, enhancing the high-quality hidden code and the low-quality hidden code to obtain enhanced high-quality hidden code and enhanced low-quality hidden code includes:

[0021] By performing forward diffusion through the diffusion model, multi-step Gaussian noise is added to the high-quality hidden code and the low-quality hidden code respectively, thus obtaining the noise-enhanced high-quality hidden code and the enhanced low-quality hidden code;

[0022] Based on the enhanced low-quality hidden code, the high-quality hidden code after noise enhancement is warped through the optical flow model to generate a high-quality hidden code after warping that is aligned with the initial video motion.

[0023] The high-quality hidden code after the distortion transformation is masked and enhanced to generate an enhanced high-quality hidden code.

[0024] Optionally, generating a high-quality video according to the enhanced high-quality codec and the enhanced low-quality codec includes:

[0025] Extract feature blocks from the enhanced high-quality latent code and the enhanced low-quality latent code as query features, key features, and value features, respectively;

[0026] According to the query feature, key feature and value feature, the texture alignment feature embedding vector is generated as Conv(AV)+Q; where A is the attention weight, Q is the query feature, K is the key feature, v is the value feature, B is the local attention bias matrix, Norm(·) is the feature normalization function, is the scaling factor, T is the transposition operator, and softmax(·) is the normalization function;

[0027] Generate high-quality videos based on feature embedding vectors.

[0028] Optionally,

[0029] in, b ij is the element of the local attention bias matrix, i is the row identifier, j is the column identifier, and d is the length of the local neighborhood corresponding to the local attention.

[0030] In a second aspect of the present application, a video quality enhancement device is provided, the device comprising:

[0031] A first acquisition module is used to acquire a low-quality initial video;

[0032] The second acquisition module is used to obtain the key frames of the initial video;

[0033] A first generation module is configured to compress the key frame into a low-quality hidden code using a diffusion model, obtain a noise hidden code at each time step based on the low-quality hidden code, and generate a high-quality hidden code based on the noise hidden code at each time step;

[0034] an enhancement module, configured to enhance the high-quality hidden code and the low-quality hidden code to obtain an enhanced high-quality hidden code and an enhanced low-quality hidden code;

[0035] The second generating module is used to generate a high-quality video according to the enhanced high-quality hidden code and the enhanced low-quality hidden code.

[0036] In a third aspect of the present application, an electronic device is provided, comprising:

[0037] Memory;

[0038] processor; and

[0039] computer programs;

[0040] The computer program is stored in the memory and is configured to be executed by the processor to implement the method as described in the first aspect above.

[0041] In a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored; the computer program is executed by a processor to implement the method described in the first aspect above.

[0042] The present application provides a method, apparatus, device, and storage medium for enhancing video quality. The method includes: obtaining a low-quality initial video; obtaining key frames of the initial video; compressing the key frames into low-quality hidden codes through a diffusion model, obtaining noise hidden codes at each time step based on the low-quality hidden codes, and generating high-quality hidden codes based on the noise hidden codes at each time step; enhancing the high-quality hidden codes and the low-quality hidden codes to obtain enhanced high-quality hidden codes and enhanced low-quality hidden codes; and generating a high-quality video based on the enhanced high-quality hidden codes and the enhanced low-quality hidden codes. After obtaining a low-quality initial video, the method of the present application obtains a low-quality hidden code through the key frames of the initial video, and then generates a high-quality hidden code based on the noise hidden codes at each time step obtained from the low-quality hidden code. After enhancing the high-quality hidden codes and the low-quality hidden codes, a high-quality video is generated. The high-quality video not only contains fine-grained details, but also needs to maintain the coherence of details between frames. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0044] Figure 1 A flowchart of a method for enhancing video quality provided in an embodiment of the present application;

[0045] Figure 2 A schematic diagram of a video quality enhancement model provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of a local perception cross attention module provided in an embodiment of the present application;

[0047] Figure 4 A schematic structural diagram of a video quality enhancement device provided in an embodiment of the present application;

[0048] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the technical solutions and advantages of the embodiments of the present application more clearly understood, the exemplary embodiments of the present application are further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, and are not an exhaustive list of all the embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other unless they conflict.

[0050] During the development of this application, the inventors discovered that video quality enhancement technology aims to improve video quality by analyzing and processing video pixel information. Existing video quality enhancement technologies primarily employ sliding window or recursive methods, using convolutional neural networks or Transformers to aggregate information from low-resolution video frames to predict high-resolution frames. While these methods excel at amplifying original visual content, they are limited in generating novel, fine-grained details.

[0051] In response to the above problems, an embodiment of the present application provides a method, apparatus, device, and storage medium for enhancing video quality. The method includes: obtaining a low-quality initial video; obtaining key frames of the initial video; compressing the key frames into low-quality hidden codes through a diffusion model, obtaining noise hidden codes at each time step based on the low-quality hidden codes, and generating high-quality hidden codes based on the noise hidden codes at each time step; enhancing the high-quality hidden codes and the low-quality hidden codes to obtain enhanced high-quality hidden codes and enhanced low-quality hidden codes; and generating a high-quality video based on the enhanced high-quality hidden codes and the enhanced low-quality hidden codes. After obtaining a low-quality initial video, the method of the present application obtains a low-quality hidden code through the key frames of the initial video, and then generates a high-quality hidden code based on the noise hidden codes at each time step obtained from the low-quality hidden codes. After enhancing the high-quality hidden codes and the low-quality hidden codes, a high-quality video is generated. The high-quality video not only contains fine-grained details, but also needs to maintain the coherence of details between frames.

[0052] See also Figure 1 This embodiment provides a method for enhancing video quality. Figure 2 The video quality enhancement model shown is implemented as follows:

[0053] 101, obtain low-quality initial video.

[0054] In step 101, a low-quality initial video is obtained, such as Figure 2 The LQ Video (Low-quality Video) in it is the low-quality initial video.

[0055] 102, obtain the key frame of the initial video.

[0056] In step 102, the Figure 2 The key frames of the initial video LQ Video.

[0057] For example, the initial video LQ Video is upsampled (e.g. Figure 2 In the Upsampling step, the first frame after each upsampling is determined as a key frame, and then in step 102, the first frame after each upsampling of the LQ Video is obtained.

[0058] 103, compressing the key frame into a low-quality hidden code through a diffusion model, obtaining a noise hidden code of each time step according to the low-quality hidden code, and generating a high-quality hidden code according to the noise hidden code of each time step.

[0059] In step 103, the key frame is compressed into a low-quality hidden code z through a diffusion model (such as a variational autoencoder of a diffusion model), and the noise hidden code of each time step is obtained based on the low-quality hidden code z. According to the noise hidden code of each time step Generate high-quality hidden code y.

[0060] The high-quality hidden code y is a semantically aligned high-quality reference image hidden code y.

[0061] The process of compressing the key frame into a low-quality hidden code can be implemented by a variational autoencoder of a diffusion model. For example, the video key frame is compressed into a low-quality hidden code z by a variational autoencoder of a diffusion model.

[0062] According to the low-quality hidden code z, the noise hidden code of each time step is obtained According to the noise hidden code of each time step The process of generating high-quality hidden code y can be achieved through the reverse diffusion of the diffusion model.

[0063] For example, at each time step t (t = T, T-1, ..., 1, T is the preset maximum time step) during denoising, the image hidden code z after denoising at time step t is calculated by the reverse diffusion of the diffusion model. t , hide the image z t With the noisy hidden code with forward noise time step t Perform weighted summation to exploit the hidden code from noise The semantic components of the image are used to adjust the image latent code z t Through iterative denoising of multiple time steps (i.e., T time steps), a high-quality latent code y is generated.

[0064] Therefore, in the specific implementation, the noise hidden code of each time step is obtained according to the low-quality hidden code z When the noise hidden code at any time step t

[0065] Among them, α t and σ t is a hyperparameter, such as the constant hyperparameter of the noise scheduling function in DDPM (Denoising Diffusion Probabilistic Models). ∈ is Gaussian noise, such as standard Gaussian noise.

[0066] When generating a high-quality hidden code based on the noisy hidden code at each time step, the high-quality hidden code y can be generated by iterative denoising of multiple time steps.

[0067] Among them, through the formula Perform denoising at any time step t.

[0068] z t To calculate the hidden code of the image after denoising at time step t through the diffusion model, is the noise hidden code at time step t.

[0069] γ t is the denoising weight, γ t It can be defined based on the cosine function of t, such as

[0070] Wherein, T is the maximum time step, which may be the maximum time step used when pre-training the diffusion model.

[0071] Since the maximum value of the denoising time step t is the maximum time step T of the pre-trained diffusion model (such as T = 1000), according to γ t As t decreases from its maximum value T to 0, it decreases monotonically from 0.5 to 0, ensuring that more semantic information of the original frame is introduced in the early stages of the denoising phase, while adding as much underlying texture detail as possible in the later stages of the denoising phase. Finally, through iterative denoising, a semantically aligned and texture-rich high-quality reference image latent code y is generated.

[0072] Step 103 can be achieved by Figure 2 For example, the Semantics-Aligning Image Diffuser first compresses the video key frame into a hidden code (i.e., a low-quality hidden code z) through the variational self-encoder of the diffusion model in step 103. Figure 2 Z in, where i is the key frame identifier, z i is the low-quality hidden code corresponding to the i-th key frame, Z is the set of low-quality hidden codes corresponding to all key frames, i = 1, 2, ..., I, I is the total number of key frames), and Gaussian noise is added to the hidden code (i.e., the low-quality hidden code z); then the image diffusion model is used for reverse diffusion, and the noise hidden code (i.e., the noise hidden code z) is added to the hidden code by weighted summation. ) and denoising code (i.e. z t ) to generate high-quality hidden code y (i.e. Figure 2 Image Referencey).

[0073] It should be noted that the hidden code mentioned in this embodiment and the subsequent real-time examples is the feature obtained by encoding the image through the trained variational autoencoder, and the image can be decoded by the variational autoencoder to obtain the image again. Figure 2The video and image of the video quality enhancement model shown are processed into latent form by the variational autoencoder.

[0074] 104 , the high-quality hidden code and the low-quality hidden code are enhanced to obtain an enhanced high-quality hidden code and an enhanced low-quality hidden code.

[0075] In step 104, the high-quality hidden code y and the low-quality hidden code Z are enhanced to obtain enhanced high-quality hidden code Y and enhanced low-quality hidden code Z.

[0076] In step 104, noise enhancement, temporal motion enhancement, and mask enhancement are performed. The implementation process of step 104 is as follows:

[0077] 1. Perform forward diffusion through the diffusion model, add multi-step Gaussian noise to the high-quality hidden code and the low-quality hidden code respectively, and obtain the noise-enhanced high-quality hidden code and the enhanced low-quality hidden code.

[0078] For example, by performing forward diffusion through the diffusion model, n-step Gaussian noise is added to the high-quality hidden code y and the low-quality hidden code z, respectively, to obtain the enhanced high-quality hidden code y″ n and the enhanced low-quality hidden code Z n The enhanced low-quality hidden code Z n That is, the enhanced low-quality hidden code Z.

[0079] This part is noise enhancement, which can be performed on high-quality hidden code y and low-quality hidden code z to simulate video enhancement scenarios under different noise levels.

[0080] For example, by performing forward diffusion through the diffusion model, n-step Gaussian noise is added to the high-quality hidden code y and the low-quality hidden code z respectively to generate the enhanced high-quality hidden code y″ n and the enhanced low-quality hidden code Z n .

[0081] In specific implementations, visual fidelity and quality can be balanced by adjusting the noise level. For example, the noise level can vary randomly.

[0082] 2. Based on the enhanced low-quality hidden code, the high-quality hidden code after noise enhancement is warped through the optical flow model to generate a high-quality hidden code after warping that is aligned with the initial video motion.

[0083] For example, based on the enhanced low-quality hidden code (ie, the enhanced low-quality hidden code Z n ), the enhanced high-quality hidden code y″ is obtained through an optical flow model (such as the RAFT model) n Perform a warping transformation to generate a high-quality hidden code y′ after the warping transformation that is aligned with the initial video motion n .

[0084] This part is temporal motion enhancement, which can improve the temporal coherence of the video. It uses optical flow estimation technology to compensate for the motion of low-quality videos and incorporates motion information into the diffusion process of high-quality videos.

[0085] For example, the RAFT optical flow model is used to estimate the optical flow of low-quality videos, and the enhanced high-quality hidden code y″ is used to calculate the optical flow. n Perform distortion transformation based on the enhanced low-quality hidden code (i.e., the enhanced low-quality hidden code Z n ) Generates a high-quality hidden code y′ after warping that is aligned with the low-quality video motion n .

[0086] It's important to note that the RAFT model is a pre-trained optical flow extraction model; simply inputting a video into the RAFT model extracts optical flow. Warp transformations align content between video frames, so a high-quality reference image latent code can be transformed using optical flow extracted between different frames to create a sequence of high-quality reference image latent codes that are motion-aligned with the low-quality video.

[0087] 3. Perform mask enhancement on the high-quality hidden code after the distortion transformation to generate an enhanced high-quality hidden code.

[0088] For example, the high-quality hidden code y′ after distortion transformation n Perform mask enhancement to generate the mask-enhanced hidden code Y n , the hidden code Y after the mask enhancement n That is, the enhanced high-quality hidden code Y.

[0089] This part is mask enhancement, which can enhance the model training effect through mask operation, avoiding the video quality enhancement method and video quality enhancement model provided in this embodiment from over-relying on a single reference image.

[0090] For example, the high-quality hidden code y′ after distortion transformation n A random masking operation is used to set some pixel blocks to zero (e.g., randomly select a certain proportion (e.g., 0.5) of pixel blocks and set them to 0), generating a masked high-quality reference image hidden code (i.e., enhanced high-quality hidden code Y). The enhanced high-quality hidden code Y is a sequence that retains the complete high-quality reference image hidden code, ensuring that the model can utilize the complete reference image information.

[0091] Step 104 can be achieved by Figure 2 The Augmentor implementation in , obtains the enhanced high-quality hidden code Y (i.e. Figure 2 Y in n , Figure 2 middle is the enhanced high-quality hidden code corresponding to the i-th key frame, Y nThe enhanced high-quality hidden code corresponding to all key frames) and the enhanced low-quality hidden code Z (i.e. Figure 2 Z in n , Figure 2 middle is the enhanced low-quality hidden code corresponding to the i-th key frame, Z n is the enhanced low-quality hidden code set corresponding to all key frames).

[0092] 105, a high-quality video is generated according to the enhanced high-quality hidden code and the enhanced low-quality hidden code.

[0093] In step 105 , a high-quality video is generated according to the enhanced high-quality hidden code Y and the enhanced low-quality hidden code Z.

[0094] The implementation process of step 105 is:

[0095] 1. Extract feature blocks from the enhanced high-quality latent code and the enhanced low-quality latent code as query features, key features, and value features, respectively.

[0096] Feature blocks can be extracted from the enhanced high-quality latent code Y and the enhanced low-quality latent code Z as query features Q, key features K, and value features V, respectively. For example, three features are obtained using three independent mapping networks, namely Q, K, and V, that is, Q, K, and V have the same source but are mapped to different spaces.

[0097] 2. Generate the texture-aligned feature embedding vector e as Conv(AV)+Q based on the query feature, key feature, and value feature.

[0098] Among them, A is the attention weight, Q is the query feature, K is the key feature, V is the value feature, B is the local attention bias matrix, Norm(·) is the feature normalization function, is the scaling factor, T is the transpose operator, and softmax(·) is the normalization function.

[0099] in,

[0100] b ij is the element of the local attention bias matrix, i is the row identifier, j is the column identifier, and d is the length of the local neighborhood corresponding to the local attention.

[0101] For the above process of generating the texture aligned feature embedding vector e, we can Figure 3The local-aware crisscross attention module is shown in the figure. This module enables texture alignment between a high-quality reference image and a low-quality video. Specifically, feature patches are extracted from the enhanced high-quality latent code Y and the enhanced low-quality latent code Z, serving as query features Q, key features K, and value features V, respectively. When calculating attention, a local attention bias matrix B is added to the attention map, limiting the focus to a local neighborhood d and enhancing the alignment of local texture features.

[0102] 3. Generate high-quality video based on feature embedding vectors.

[0103] For example, high-quality videos are generated based on the feature embedding vector e.

[0104] In the specific implementation, after obtaining the feature embedding vector e, we can use Figure 2 The Texture-AligningVideo Controller in

[15] adds it to the output of the subsequent encoder block for multi-scale vision-guided learning.

[0105] Specifically, the Texture-Aligning Video Controller uses the texture-aligned feature embedding vector e output by the local perception cross-attention module, and obtains the visual guidance feature through feature propagation of the cascaded encoder block. The visual guidance feature is then used to adjust the denoising process of the video diffusion model to generate high-quality video.

[0106] In step 105, a texture-aligned feature embedding vector, e, is generated through a crisscross attention calculation and added to the output of the subsequent encoder block for multi-scale visually guided learning. Finally, the Texture-Aligning Video Controller uses the texture-aligned feature embedding vector output by the local-aware crisscross attention module to learn visually guided features through cascaded encoder blocks to adjust the denoising process of the video diffusion model, ultimately generating high-quality video.

[0107] The video quality enhancement method provided in this embodiment can greatly improve the ability to generate fine-grained details in high-quality videos. By introducing a high-quality reference image (i.e., the enhanced high-quality latent code Y) and using a local-aware cross-attention module for texture alignment, it can generate richer fine-grained details, such as textures and edges. This makes the generated video more visually realistic and detailed, addressing the shortcomings of existing technologies in detail generation.

[0108] The video quality enhancement method provided in this embodiment is highly robust. Through noise enhancement, temporal motion enhancement, and mask enhancement, it can simulate a variety of complex input scenarios, enhancing the robustness of the model. This enables the video quality enhancement method and model provided in this embodiment to more stably output high-quality video results under varying input conditions and degraded scenarios, resolving the lack of robustness in existing technologies.

[0109] The video quality enhancement method provided in this embodiment enables flexible control of generated quality and fidelity. By adjusting the noise level parameter, flexible control can be achieved between video quality and original visual fidelity. This allows video enhancement technology to better meet different needs.

[0110] This embodiment provides a method for enhancing video quality, which includes obtaining a low-quality initial video; obtaining key frames of the initial video; compressing the key frames into low-quality hidden codes using a diffusion model, obtaining noise hidden codes at each time step based on the low-quality hidden codes, and generating high-quality hidden codes based on the noise hidden codes at each time step; enhancing the high-quality hidden codes and the low-quality hidden codes to obtain enhanced high-quality hidden codes and enhanced low-quality hidden codes; and generating a high-quality video based on the enhanced high-quality hidden codes and the enhanced low-quality hidden codes. After obtaining a low-quality initial video, the method of this embodiment obtains a low-quality hidden code using the key frames of the initial video, and then generates a high-quality hidden code based on the noise hidden codes at each time step obtained from the low-quality hidden codes. After enhancing the high-quality hidden codes and the low-quality hidden codes, a high-quality video is generated. The high-quality video not only contains fine-grained details but also maintains the coherence of details between frames.

[0111] Based on the same inventive concept of the video quality enhancement method, this embodiment provides a video quality enhancement device, such as Figure 4 As shown, the device includes:

[0112] The first acquisition module 401 is configured to acquire a low-quality initial video.

[0113] The second acquisition module 402 is used to acquire key frames of the initial video.

[0114] The first generating module 403 is configured to compress the key frame into a low-quality hidden code through a diffusion model, obtain a noise hidden code at each time step based on the low-quality hidden code, and generate a high-quality hidden code based on the noise hidden code at each time step.

[0115] The enhancement module 404 is configured to enhance the high-quality hidden code and the low-quality hidden code to obtain an enhanced high-quality hidden code and an enhanced low-quality hidden code.

[0116] The second generating module 405 is configured to generate a high-quality video according to the enhanced high-quality hidden code and the enhanced low-quality hidden code.

[0117] Among them, the noise hidden code at any time step t is

[0118] Among them, α t and σ t is a hyperparameter, ∈ is Gaussian noise.

[0119] Among them, generating high-quality hidden codes based on the noise hidden codes at each time step includes:

[0120] High-quality latent codes are generated through iterative denoising over multiple time steps.

[0121] Among them, through the formula Perform denoising at any time step t.

[0122] z t To calculate the hidden code of the image after denoising at time step t through the diffusion model, is the noise hidden code at time step t, γ t is the denoising weight.

[0123] in,

[0124] Where T is the maximum time step.

[0125] The high-quality hidden code and the low-quality hidden code are enhanced to obtain the enhanced high-quality hidden code and the enhanced low-quality hidden code, including:

[0126] The forward diffusion is performed through the diffusion model, and multi-step Gaussian noise is added to the high-quality hidden code and the low-quality hidden code respectively, so as to obtain the noise-enhanced high-quality hidden code and the enhanced low-quality hidden code.

[0127] Based on the enhanced low-quality hidden code, the noise-enhanced high-quality hidden code is warped through an optical flow model to generate a warped high-quality hidden code that is aligned with the initial video motion.

[0128] The high-quality hidden code after the distortion transformation is masked and enhanced to generate an enhanced high-quality hidden code.

[0129] The high-quality video is generated according to the enhanced high-quality hidden code and the enhanced low-quality hidden code, including:

[0130] Feature blocks are extracted from the enhanced high-quality latent code and the enhanced low-quality latent code as query features, key features, and value features, respectively.

[0131] According to the query feature, key feature and value feature, the texture alignment feature embedding vector is generated as Conv(AV)+Q. Among them, A is the attention weight, Q is the query feature, K is the key feature, V is the value feature, B is the local attention bias matrix, Norm(·) is the feature normalization function, is the scaling factor, T is the transpose operator, and softmax(·) is the normalization function.

[0132] Generate high-quality videos based on feature embedding vectors.

[0133] in,

[0134] in, b ij is the element of the local attention bias matrix, i is the row identifier, j is the column identifier, and d is the length of the local neighborhood corresponding to the local attention.

[0135] After obtaining a low-quality initial video, the apparatus provided in this embodiment obtains a low-quality hidden code from the key frames of the initial video. High-quality hidden codes are then generated based on the noise hidden codes at each time step obtained from the low-quality hidden code. After enhancing the high-quality hidden code and the low-quality hidden code, a high-quality video is generated. This high-quality video not only contains fine-grained details but also maintains the coherence of details between frames.

[0136] Based on the same inventive concept of the video quality enhancement method, this embodiment provides an electronic device, such as Figure 5 As shown, it includes: a memory 501, a processor 502, and a computer program.

[0137] The computer program is stored in the memory 501 and is configured to be executed by the processor 502 to implement the above-mentioned video quality enhancement method.

[0138] Specifically,

[0139] Get a low-quality initial video.

[0140] Get the keyframe of the initial video.

[0141] The key frame is compressed into a low-quality hidden code through a diffusion model, and the noise hidden code of each time step is obtained according to the low-quality hidden code, and the high-quality hidden code is generated according to the noise hidden code of each time step.

[0142] The high-quality hidden code and the low-quality hidden code are enhanced to obtain an enhanced high-quality hidden code and an enhanced low-quality hidden code.

[0143] A high-quality video is generated based on the enhanced high-quality codec and the enhanced low-quality codec.

[0144] Among them, the noise hidden code at any time step t is

[0145] Among them, αt and σ t is a hyperparameter, ∈ is Gaussian noise.

[0146] Among them, generating high-quality hidden codes based on the noise hidden codes at each time step includes:

[0147] High-quality latent codes are generated through iterative denoising over multiple time steps.

[0148] Among them, through the formula Perform denoising at any time step t.

[0149] z t To calculate the hidden code of the image after denoising at time step t through the diffusion model, is the noise hidden code at time step t, γ t is the denoising weight.

[0150] in,

[0151] Where T is the maximum time step.

[0152] The high-quality hidden code and the low-quality hidden code are enhanced to obtain the enhanced high-quality hidden code and the enhanced low-quality hidden code, including:

[0153] The forward diffusion is performed through the diffusion model, and multi-step Gaussian noise is added to the high-quality hidden code and the low-quality hidden code respectively, so as to obtain the noise-enhanced high-quality hidden code and the enhanced low-quality hidden code.

[0154] Based on the enhanced low-quality hidden code, the noise-enhanced high-quality hidden code is warped through an optical flow model to generate a warped high-quality hidden code that is aligned with the initial video motion.

[0155] The high-quality hidden code after the distortion transformation is masked and enhanced to generate an enhanced high-quality hidden code.

[0156] The high-quality video is generated according to the enhanced high-quality hidden code and the enhanced low-quality hidden code, including:

[0157] Feature blocks are extracted from the enhanced high-quality latent code and the enhanced low-quality latent code as query features, key features, and value features, respectively.

[0158] According to the query feature, key feature and value feature, the texture alignment feature embedding vector is generated as Conv(AV)+Q. Among them, A is the attention weight, Q is the query feature, K is the key feature, V is the value feature, B is the local attention bias matrix, Norm(·) is the feature normalization function, is the scaling factor, T is the transpose operator, and softmax(·) is the normalization function.

[0159] Generate high-quality videos based on feature embedding vectors.

[0160] in,

[0161] in, b ij is the element of the local attention bias matrix, i is the row identifier, j is the column identifier, and d is the length of the local neighborhood corresponding to the local attention.

[0162] In the electronic device provided by this embodiment, a computer program is executed by a processor to obtain a low-quality initial video. The low-quality hidden code is then obtained from key frames of the initial video. A high-quality hidden code is then generated based on the noise hidden code at each time step obtained from the low-quality hidden code. After enhancing the high-quality hidden code and the low-quality hidden code, a high-quality video is generated. This high-quality video not only contains fine-grained details but also maintains the coherence of details between frames.

[0163] Based on the same inventive concept of the video quality enhancement method, this embodiment provides a computer-readable storage medium having a computer program stored thereon. The computer program is executed by a processor to implement the above-mentioned video quality enhancement method.

[0164] Specifically,

[0165] Get a low-quality initial video.

[0166] Get the keyframe of the initial video.

[0167] The key frame is compressed into a low-quality hidden code through a diffusion model, and the noise hidden code of each time step is obtained according to the low-quality hidden code, and the high-quality hidden code is generated according to the noise hidden code of each time step.

[0168] The high-quality hidden code and the low-quality hidden code are enhanced to obtain an enhanced high-quality hidden code and an enhanced low-quality hidden code.

[0169] A high-quality video is generated based on the enhanced high-quality codec and the enhanced low-quality codec.

[0170] Among them, the noise hidden code at any time step t is

[0171] Among them, α t and σ t is a hyperparameter, ∈ is Gaussian noise.

[0172] Among them, generating high-quality hidden codes based on the noise hidden codes at each time step includes:

[0173] High-quality latent codes are generated through iterative denoising over multiple time steps.

[0174] Among them, through the formula Perform denoising at any time step t.

[0175] z t To calculate the hidden code of the image after denoising at time step t through the diffusion model, is the noise hidden code at time step t, γ t is the denoising weight.

[0176] in,

[0177] Where T is the maximum time step.

[0178] The high-quality hidden code and the low-quality hidden code are enhanced to obtain the enhanced high-quality hidden code and the enhanced low-quality hidden code, including:

[0179] The forward diffusion is performed through the diffusion model, and multi-step Gaussian noise is added to the high-quality hidden code and the low-quality hidden code respectively, so as to obtain the noise-enhanced high-quality hidden code and the enhanced low-quality hidden code.

[0180] Based on the enhanced low-quality hidden code, the noise-enhanced high-quality hidden code is warped through an optical flow model to generate a warped high-quality hidden code that is aligned with the initial video motion.

[0181] The high-quality hidden code after the distortion transformation is masked and enhanced to generate an enhanced high-quality hidden code.

[0182] The high-quality video is generated according to the enhanced high-quality hidden code and the enhanced low-quality hidden code, including:

[0183] Feature blocks are extracted from the enhanced high-quality latent code and the enhanced low-quality latent code as query features, key features, and value features, respectively.

[0184] According to the query feature, key feature and value feature, the texture alignment feature embedding vector is generated as Conv(AV)+Q. Among them, A is the attention weight, Q is the query feature, K is the key feature, V is the value feature, B is the local attention bias matrix, Norm(·) is the feature normalization function, is the scaling factor, T is the transpose operator, and softmax(·) is the normalization function.

[0185] Generate high-quality videos based on feature embedding vectors.

[0186] in,

[0187] in, b ij is the element of the local attention bias matrix, i is the row identifier, j is the column identifier, and d is the length of the local neighborhood corresponding to the local attention.

[0188] The computer program on the computer-readable storage medium provided in this embodiment is executed by a processor to obtain a low-quality initial video, then obtain a low-quality hidden code from the key frames of the initial video. A high-quality hidden code is then generated based on the noise hidden code at each time step obtained from the low-quality hidden code. After enhancing the high-quality hidden code and the low-quality hidden code, a high-quality video is generated. This high-quality video not only contains fine-grained details but also maintains the coherence of details between frames.

[0189] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiment of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal translation scripting language JavaScript, etc.

[0190] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0191] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0192] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0193] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0194] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A method for enhancing video quality, characterized in that: The method comprises: Get low-quality initial video; Obtaining key frames of the initial video; compressing the key frame into a low-quality hidden code through a diffusion model, obtaining a noise hidden code at each time step based on the low-quality hidden code, and generating a high-quality hidden code based on the noise hidden code at each time step; enhancing the high-quality hidden code and the low-quality hidden code to obtain an enhanced high-quality hidden code and an enhanced low-quality hidden code; A high-quality video is generated based on the enhanced high-quality codec and the enhanced low-quality codec.

2. The method according to claim 1, characterized in that The noise hidden code at any time step t Among them, α t and σ t is a hyperparameter, ∈ is Gaussian noise.

3. The method according to claim 1, characterized in that Generating a high-quality hidden code according to the noise hidden code at each time step includes: Generate high-quality latent codes through iterative denoising at multiple time steps; Among them, through the formula Perform denoising at any time step t; z t To calculate the hidden code of the image after denoising at time step t through the diffusion model, is the noise hidden code at time step t, γ t is the denoising weight.

4. The method according to claim 3, characterized in that Where T is the maximum time step.

5. The method according to claim 1, characterized in that The step of enhancing the high-quality hidden code and the low-quality hidden code to obtain an enhanced high-quality hidden code and an enhanced low-quality hidden code includes: Performing forward diffusion through a diffusion model to add multi-step Gaussian noise to the high-quality hidden code and the low-quality hidden code, respectively, to obtain a noise-enhanced high-quality hidden code and an enhanced low-quality hidden code; Based on the enhanced low-quality hidden code, the high-quality hidden code after noise enhancement is warped through the optical flow model to generate a high-quality hidden code after warping that is aligned with the initial video motion. The high-quality hidden code after the distortion transformation is masked and enhanced to generate an enhanced high-quality hidden code.

6. The method according to claim 1, characterized in that The method of generating a high-quality video according to the enhanced high-quality hidden code and the enhanced low-quality hidden code comprises: Extract feature blocks from the enhanced high-quality latent code and the enhanced low-quality latent code as query features, key features, and value features, respectively; According to the query feature, key feature and value feature, a texture aligned feature embedding vector is generated as Conv(AV)+Q; wherein A is the attention weight, Q is the query feature, K is the key feature, V is the value feature, V is the local attention bias matrix, Norm(·) is the feature normalization function, is the scaling factor, T is the transposition operator, and softmax(·) is the normalization function; A high-quality video is generated based on the feature embedding vector.

7. The method according to claim 6, characterized in that in, b ij is the element of the local attention bias matrix, i is the row identifier, j is the column identifier, and d is the length of the local neighborhood corresponding to the local attention.

8. A video quality enhancement device, characterized in that: The device comprises: A first acquisition module is used to acquire a low-quality initial video; A second acquisition module is used to acquire key frames of the initial video; a first generating module, configured to compress the key frame into a low-quality hidden code using a diffusion model, obtain a noise hidden code at each time step based on the low-quality hidden code, and generate a high-quality hidden code based on the noise hidden code at each time step; an enhancement module, configured to enhance the high-quality hidden code and the low-quality hidden code to obtain an enhanced high-quality hidden code and an enhanced low-quality hidden code; The second generating module is used to generate a high-quality video according to the enhanced high-quality hidden code and the enhanced low-quality hidden code.

9. An electronic device, characterized in that: include: Memory; processor; as well as computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that A computer program is stored thereon; the computer program is executed by a processor to implement the method according to any one of claims 1 to 7.

Images