Video encoding, decoding method and non-transitory computer readable storage medium

Abstract

The application provides a video encoding and decoding method and a non-transitory computer readable storage medium, wherein the video decoding method comprises the following steps: after receiving a bit stream of a video and decoding to obtain a reconstructed video, calculating distribution parameters of each color channel of a key reference frame in the reconstructed video, wherein the distribution parameters are used to represent color characteristics of the corresponding color channel; for the same color channel, using the distribution parameters of the key reference frame to calibrate the color distribution of a subsequent frame in the reconstructed video to obtain a calibrated subsequent frame; and generating a target video based on the key reference frame and the calibrated subsequent frame. Through the application, the technical problem of color offset of the reconstructed video in the related art is solved, and the technical effect of improving the quality of the reconstructed video is achieved.

Description

Technical Field

[0001] This application relates to the field of video technology, and in particular to video encoding and decoding methods and non-transitory computer-readable storage media. Background Technology

[0002] Generative Video Compression (GVC) frameworks generally include an encoding-decoding process. At the encoder, key reference frames are compressed using traditional encoding techniques, while subsequent frames are represented using compact transmission symbols and encoded into the output bitstream. At the decoder, the decoded key reference frames and compacted facial information are input into a synthesis model to reconstruct the video. This approach enables ultra-low bitrate and high-quality video communication. However, current GVC frameworks commonly suffer from the technical problem of color shift during video reconstruction. No effective solution has yet been proposed to address this issue. Summary of the Invention

[0003] This application provides video encoding and decoding methods, as well as non-transitory computer-readable storage media, to solve one or more of the aforementioned technical problems.

[0004] In a first aspect, embodiments of this application provide a video decoding method applied to a generative video compression framework, comprising: receiving a bitstream of video and decoding it to obtain a reconstructed video; calculating the distribution parameters of each color channel in a key reference frame in the reconstructed video, wherein the distribution parameters are used to characterize the color features of the corresponding color channel; calibrating the color distribution of subsequent frames in the reconstructed video using the distribution parameters of the key reference frame for the same color channel, thereby obtaining calibrated subsequent frames; and generating a target video based on the key reference frame and the calibrated subsequent frames.

[0005] Secondly, embodiments of this application provide a video encoding method applied to a generative video compression framework, comprising: encoding key reference frames in an original video to obtain a first encoded bitstream; representing subsequent frames in the original video using compact transmission symbols and encoding the representation results to obtain a second encoded bitstream; determining the first encoded bitstream and the second encoded bitstream as the bitstream of the video corresponding to the original video, and sending the bitstream of the video, so that the decoder, after receiving the bitstream of the video, uses the above-described video decoding method to perform video decoding to obtain the target video.

[0006] Thirdly, embodiments of this application provide a video decoding apparatus applied to a generative video compression framework, comprising: a calculation module, configured to receive a bitstream of video and decode it to obtain a reconstructed video, and then calculate the distribution parameters of each color channel of a key reference frame in the reconstructed video, wherein the distribution parameters are used to characterize the color features of the corresponding color channel; a first calibration module, configured to calibrate the color distribution of subsequent frames in the reconstructed video for the same color channel using the distribution parameters of the key reference frame, to obtain calibrated subsequent frames; and a first generation module, configured to generate a target video based on the key reference frame and the calibrated subsequent frames.

[0007] Fourthly, embodiments of this application provide a video encoding apparatus applied to a generative video compression framework, comprising: a first encoding module for encoding key reference frames in an original video to obtain a first encoded bitstream; a second encoding module for representing subsequent frames in the original video using compact transmission symbols and encoding the representation results to obtain a second encoded bitstream; and a first transmission module for determining the first encoded bitstream and the second encoded bitstream as the bitstream of the video corresponding to the original video and transmitting the bitstream of the video, so that the decoder, after receiving the bitstream of the video, performs video decoding using the aforementioned video decoding method to obtain the target video.

[0008] Fifthly, embodiments of this application provide a non-transitory computer-readable storage medium storing a bitstream of video, the non-transitory computer-readable storage medium being part of a computing device configured to execute a set of instructions to decode the bitstream of video according to operations including: after decoding the bitstream of video to obtain a reconstructed video, calculating the distribution parameters of each color channel of a key reference frame in the reconstructed video, wherein the distribution parameters are used to characterize the color features of the corresponding color channel; calibrating the color distribution of subsequent frames in the reconstructed video using the distribution parameters of the key reference frame for the same color channel, obtaining calibrated subsequent frames; and generating a target video based on the key reference frame and the calibrated subsequent frames.

[0009] In this embodiment, after receiving and decoding the bitstream of a video to obtain a reconstructed video, the distribution parameters of each color channel in the key reference frame of the reconstructed video are calculated. These distribution parameters characterize the color features of the corresponding color channel. For the same color channel, the distribution parameters of the key reference frame are used to calibrate the color distribution of subsequent frames in the reconstructed video, resulting in calibrated subsequent frames. A target video is then generated based on the key reference frame and the calibrated subsequent frames. In other words, this embodiment uses a channel-by-channel calibration method, employing the distribution parameters of each color channel in the key reference frame to calibrate the color distribution of subsequent frames corresponding to that color channel. This solves the technical problem of color shift in reconstructed videos in related technologies and achieves the technical effect of improving the quality of the reconstructed video.

[0010] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0011] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0012] Figure 1 This illustration shows a video compression framework in the related technologies provided in the embodiments of this application that follows a prediction-transform architecture.

[0013] Figure 2 This illustration shows a basic framework diagram of the end-to-end video compression depth model in the related technologies provided in the embodiments of this application;

[0014] Figure 3 This illustration shows a basic framework diagram of a deep learning video generation and compression scheme based on a first-order motion model in the related technologies provided in the embodiments of this application;

[0015] Figure 4 A schematic diagram of the basic framework of a deep learning-based video generation and compression scheme based on compact feature representation in the related technologies provided in the embodiments of this application is shown.

[0016] Figures 5a-5b The diagrams show the data distribution of channel pixel values ​​in the original frame and the reconstructed frame at different resolutions.

[0017] Figures 6a-6c This diagram illustrates the color deviation between the original frame and the reconstructed frame with different pixels.

[0018] Figure 7 This illustration shows a schematic flowchart of a video decoding method provided in an embodiment of this application;

[0019] Figure 8 This illustration shows a schematic diagram of the encoding-decoding process of the GVC algorithm provided in an embodiment of this application;

[0020] Figure 9 This illustration shows a color plane and corresponding pixel values ​​provided in an embodiment of this application;

[0021] Figure 10 This paper illustrates a schematic flowchart of a video color calibration method provided in an embodiment of this application.

[0022] Figure 11 This paper shows a structural block diagram of a video decoding device provided in an embodiment of this application;

[0023] Figure 12 This paper illustrates a flowchart of another video decoding method provided in an embodiment of this application.

[0024] Figure 13 This paper shows a structural block diagram of another video decoding device provided in an embodiment of this application;

[0025] Figure 14 This illustration shows a flowchart of a video encoding method provided in an embodiment of this application;

[0026] Figure 15 This paper shows a structural block diagram of a video encoding device provided in an embodiment of this application;

[0027] Figure 16 This paper illustrates a flowchart of another video encoding method provided in an embodiment of this application.

[0028] Figure 17 A structural block diagram of another video encoding apparatus provided in an embodiment of this application is shown. Detailed Implementation

[0029] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0030] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0031] The upcoming era of AI-Generated Content (AIGC) is witnessing the rapid development of video coding technology in smarter, more immersive, and interactive applications. Generative Video Coding (GVC) is one of the key technologies. GVC utilizes the powerful inference capabilities of deep generative models for visual data compression, achieving superior rate-distortion (RD) performance compared to traditional hybrid encoders such as High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC).

[0032] In particular, most existing generative video encoders have evolved from deep image animation methods, which can characterize high-dimensional visual input signals into compact representations and leverage powerful deep generative models to achieve high-quality signal reconstruction / animation. For example, the Deep Animation Codec utilizes 2D keypoint representations to achieve ultra-low bitrate video conferencing. Similarly, in free-viewpoint controlled face conversation video coding, 3D keypoints are used, and the feature matrix can more compactly represent the temporal trajectory of the face.

[0033] However, the capabilities of generative coding are limited by its feature design and generation scheme. On the one hand, these generative video encoders primarily use explicit feature representations with actual physical characteristics, leading to unnecessary compression redundancy. Simultaneously, this representation lacks sufficient expressiveness and generality, failing to handle more complex scenarios, such as moving human figures. In some techniques, explicit features (including landmarks, keypoints, and segmentation maps) are used for low-bandwidth video chat compression, and it has been observed that different feature representations result in significantly different bandwidth requirements.

[0034] A related technology prior to this application is the traditional video compression standards, such as Advanced Video Coding (AVC), HEVC, and VVC, which have been meticulously developed to achieve excellent compression performance. These standards utilize a block-based hybrid video coding framework to take advantage of spatial redundancy, temporal redundancy, and information entropy redundancy in the video. In this framework, the video compression encoder generates a bitstream based on the input current frame, while the decoder reconstructs video frames based on the received bitstream. Figure 1 The classic video compression framework shown follows a prediction-transform architecture. Specifically, the input frame x t It is divided into a series of blocks of the same size, i.e., square regions (e.g., 8×8 pixel blocks). The encoding process at the encoder end of a traditional video compression algorithm mainly includes the following steps: Step 1, Motion estimation: Estimate the current frame xt Compared with the previous reconstructed frame The movement between blocks is used to obtain the motion vector v corresponding to each block. t Step 2, Motion Compensation: This is achieved by applying the motion vector v defined in Step 1. t The corresponding pixels from the previous reconstructed frame are copied to the current frame to obtain the predicted frame. Then calculate the original frame x. t With the predicted frame The residual r between t ,Right now Step 3, Transformation and Quantization: Transform the residual r obtained in Step 2 into a quantized form. t Quantified as Before quantization, a linear transformation (e.g., Discrete Cosine Transform (DCT)) is used to obtain better compression performance. Step 4, Inverse Transform: Utilizing the quantization results from Step 3. Perform an inverse transformation to obtain the reconstructed residuals. Step 5, Entropy Coding: Use the entropy coding method to encode the motion vector v from Step 1. t And the quantification results in step 3 Encode into a bitstream and send to the decoder. Step 6, Frame Reconstruction: Reconstruct the frame from step 2 by... and the reconstruction residual in step 4 Adding them together yields the reconstructed frame. Right now The reconstructed frame will be used for motion estimation of the (t+1)th frame in step 1. For the decoder, based on the bitstream provided by the encoder in step 5, motion compensation in step 2 and inverse quantization in step 4 are performed, followed by frame reconstruction in step 6 to obtain the reconstructed frame.

[0035] Another related technology prior to this application is end-to-end deep learning-based video compression. With the rapid development of deep learning, many deep learning-based algorithms have been introduced to replace or enhance video coding tools, including intra / inter-frame prediction, entropy coding, and intra-loop filtering. End-to-end image / video compression algorithms have been proposed that jointly optimize the entire image / video compression framework, rather than designing individual modules. For example, the end-to-end video coding scheme Deep Video Compression (DVC) jointly optimizes all components of video compression. Furthermore, content adaptation and error propagation awareness are considered, and an online encoder update scheme is developed to improve video compression performance. Moreover, Feature-based Video Compression (FVC) is implemented by developing all the main modules of the end-to-end compression framework within the feature space. Based on recurrent probabilistic models and weighted recurrent quality enhancement networks, Recurrent Learning Video Compression (RLVC) and Hierarchical Learned Video Compression (HLVC) are proposed to fully utilize the temporal correlation between video frames. Furthermore, four effective modules were introduced in Multiple Frames Prediction for Learned Video Compression (M-LVC). However, these traditional or learning-based methods target general natural scenes without specifically considering human content such as faces, bodies, or other parts. Figure 2 This paper presents the basic framework of the first end-to-end deep learning model for video compression, which jointly optimizes all components of video compression, such as motion estimation, motion compression, and residual compression. Specifically, it utilizes learning-based optical flow estimation to acquire motion information and reconstruct the current frame, and then uses two autoencoder-style neural networks to compress the corresponding motion and residual information. All modules learn together through a single loss function, and they collaborate to strike a tradeoff between reducing the number of compressed bits and improving the quality of the decoded video. Figure 2 The new end-to-end deep framework shown in the article and Figure 1 There is a one-to-one correspondence between the traditional video compression frameworks shown in the example. These relationships and their differences are briefly summarized as follows: Step 1, Motion Estimation and Compression: A Convolutional Neural Network (CNN) model is used to estimate optical flow, which is considered as motion information v. tInstead of directly encoding the raw optical flow values, a motion vector (MV) encoder-decoder network is used to compress and decode the optical flow values, where the quantized motion representation is denoted as... Then, the corresponding reconstructed motion information can be decoded using the MV decoder network. Step 2, Motion Compensation: A motion compensation network was designed to obtain the predicted frame based on the optical flow obtained in Step 1. Step 3, Transformation, Quantization, and Inverse Transformation: A highly nonlinear residual encoder-decoder network is used instead of a linear transform, and the residual... Nonlinear mapping to representation y t , then y t Quantified as Quantization was used to construct the end-to-end training scheme. The quantized representation... It is fed into the residual decoder network to obtain the reconstructed residual. Step 4, Entropy Encoding: During the testing phase, the quantized motion representation from Step 1 is... and the residual representation in step 3 The bits are encoded and sent to the decoder. During the training phase, a CNN is used to estimate the cost per bit. and The probability distribution of each symbol in the image. Step 5, frame reconstruction: This is the same as the traditional method and will not be described in detail here.

[0036] Another related technology prior to this application is generative video coding. With the emergence of deep generative models such as Variational Auto-Encoding (VAE) and Generative Adversarial Networks (GANs), facial video compression has achieved exciting performance improvements. X2Face has been designed to control face generation using images, audio, and pose codes. Furthermore, a realistic neural conversational head model implemented through few-shot adversarial learning has been proposed. Face-vidtovid has been proposed for video-to-video synthesis tasks. A novel scheme has also been proposed to drive the generative model to render target frames by utilizing compact 3D keypoint representations. Additionally, a mobile device video chat system based on a First Order Motion Model (FOMM) has been designed. VSBNet has been proposed to reconstruct original frames from landmarks using adversarial learning. Furthermore, an end-to-end conversational head video compression framework (CFTE) based on compact feature learning has been proposed, cleverly designed for efficient conversational face video compression, suitable for ultra-low bandwidth scenarios. The CFTE scheme utilizes compact feature representations to compensate for temporal evolution and reconstructs target face video frames in an end-to-end manner. Furthermore, it can be incorporated into video coding frameworks under rate-distortion target supervision. Although these algorithms have successfully achieved frame reconstruction using a small number of facial parameters thanks to the powerful rendering capabilities of deep generative models, some head pose movements and facial expression changes still fail to be accurately rendered compared to the original moving video. Figure 3This paper presents the basic framework of a deep learning-based video generation and compression scheme, which is based on a First Order Motion Model (FOMM). FOMM follows the motion of the driving video by deforming a reference source frame. Although this method is applicable to various types of videos (such as Tai Chi and cartoons), some research teams have focused on applying it to facial animation. FOMM follows an encoder-decoder architecture that includes motion transfer components: 1. A keypoint extractor is learned using an isovariant loss, without explicit labels. This keypoint extractor computes two sets of ten learned keypoints from the source and driving frames. The learned keypoints are derived from a feature map of size 64×64 channels using a Gaussian map function, so each corresponding keypoint can represent feature information from different channels. It is worth mentioning that each keypoint is a (x, y) coordinate, which can represent the most important information in the feature map. 2. A dense motion network generates a dense motion field and an occlusion map using landmarks and the source frame. 3. The encoder encodes the source frame using traditional image / video compression methods such as High Efficiency Video Coding (HEVC) / VVC or Joint Photographic Experts Group (JPEG) / BPG. Here, VVC is used to compress the source frame. 4. The generated feature map is warped using a dense motion field (through differentiable grid sampling operations) and then multiplied with the occlusion map. 5. The decoder generates an image from the warped map.

[0037] Another related technology prior to this application is... Figure 4The presented framework is a basic structure for another deep learning-based video generation compression scheme based on compact feature representation, following an encoder-decoder architecture. At the encoding end, the compression framework consists of three modules: an encoder for compressing keyframes, a feature extractor for extracting compact human features from subsequent frames, and a feature encoding module for compressing inter-frame prediction residuals of these compact human features. First, keyframes representing human textures are compressed using the VVC encoder. Each subsequent frame is represented by a compact feature matrix of size 1×4×4 using the compact feature extractor. It's important to note that the size of the compact feature matrix is ​​not fixed; the number of feature parameters can be increased or decreased depending on the specific bitrate requirements. Subsequently, these extracted features undergo inter-frame prediction and quantization, and finally, the residuals are entropy-encoded to generate the final bitstream. At the decoding end, the compression framework also includes three main modules: a decoding module for reconstructing keyframes, a module for reconstructing compact features through entropy decoding and compensation, and a module for generating the final video using the reconstructed features and decoded keyframes. More specifically, during the generation of the final video, the keyframes decoded from the VVC bitstream can be further represented in feature form through compact feature extraction. Subsequently, based on the features of keyframes and subsequent frames, the relevant sparse motion field is calculated, thereby generating pixel-level dense motion maps and occlusion maps. Finally, based on a deep generative model, using the decoded keyframes, pixel-level dense motion maps, and occlusion maps represented by implicit motion fields, a final video with accurate appearance, pose, and expression is generated.

[0038] While generative video compression schemes can achieve promising rate-distortion (RD) performance, they still face several drawbacks and challenges that limit further performance improvements and practical applications. The data distribution modeling in the generated reconstruction may be inaccurate compared to the original distribution due to the generative training process. This inaccurate data distribution modeling can lead to unpleasant subjective distortions such as color bias in the final reconstructed video. For example, we observed the CFTE model's performance in the first subsequent frame reconstruction at 256×256 and 512×512 resolutions. As... Figures 5a-5b As shown, the solid and dashed lines represent the data distribution of the channel pixel values ​​in the original and reconstructed frames, respectively. It can be observed that the distribution of the reconstructed frame deviates significantly in the R, G, and B channels.

[0039] Specifically, for Figure 6a In the original frame shown, for the 256×256 pixel reconstruction, all channel values ​​are biased towards smaller values. However, for the 512×512 pixel reconstruction, the red channel is biased towards larger values, while the blue channel is biased towards smaller values. Therefore, the 256×256 pixel reconstruction appears slightly darker, as... Figure 6b As shown, the 512×512 pixel reconstruction exhibits a noticeable warm tone, as... Figure 6c As shown.

[0040] In view of this, embodiments of this application provide a video decoding method to solve all or part of the above-mentioned technical problems. This method can be applied to generative video compression frameworks, and is also applicable to all scenarios involving color shifts caused by the use of generative models. In other words, the video decoding method provided in embodiments of this application can be applied to any video compression encoding-decoding related technologies.

[0041] like Figure 7 As shown, the above video decoding method may include:

[0042] S702, after receiving the bitstream of the video and decoding it to obtain the reconstructed video, calculate the distribution parameters of each color channel of the key reference frame in the reconstructed video, wherein the distribution parameters are used to characterize the color features of the corresponding color channel.

[0043] It should be noted that the reconstructed video includes a key reference frame and subsequent frames. The key reference frame typically contains complete image information, while subsequent frames generally store differences from the reference frame. The reconstructed video is obtained as follows: Figure 8 As shown. Figure 8 This paper demonstrates the general encoding-decoding process of the GVC algorithm, using generative face video coding as an example. At the encoder, key reference frames of the video are compressed using traditional coding techniques, while subsequent frames are represented using compact transmission symbols and encoded into the output bitstream. At the decoder, the decoded key reference frames and the compact face information are input into the synthesis model to reconstruct the video, thus obtaining the reconstructed video.

[0044] It should be noted that the color channels mentioned above, in addition to RGB (red, green, blue) channels, can also include channels from other color spaces or additional channels for specific application scenarios. For example, YCbCr (Y: luminance channel, representing the image's brightness information; Cb: blue chrominance channel, representing the difference between blue and luminance; Cr: red chrominance channel, representing the difference between red and luminance) and YUV (Y: luminance channel, representing the image's brightness information; U: chrominance U channel, representing blue chrominance information; V: chrominance V channel, representing red chrominance information). Depending on the specific application scenario, color spaces such as YPbPr (Y: luminance channel, representing the image's brightness information; Pb: blue chrominance channel; Pr: red chrominance channel), HSL / HSV (H: hue, representing color; S: saturation, representing color purity; L / V: brightness / value, representing color lightness / darkness), and Lab (L: luminance; a: representing color information from green to magenta; b: representing color information from blue to yellow) can also be used.

[0045] Additionally, it should be noted that the types of videos mentioned above include, but are not limited to: facial videos, motion videos, conference and speech videos, educational videos, entertainment and media videos, game videos, user-generated content, surveillance videos, real-time communication videos, virtual reality and augmented reality videos, etc., which can be expanded according to specific application scenarios and needs. Each type of video has its own specific visual and dynamic characteristics.

[0046] Optionally, the above distribution parameters include, but are not limited to, the mean and standard deviation, and may also be other characteristics of the color value distribution, such as kurtosis, skewness, frequency domain characteristics, etc., which are not limited here.

[0047] S704, for the same color channel, use the distribution parameters of the key reference frame to calibrate the color distribution of subsequent frames in the reconstructed video to obtain calibrated subsequent frames.

[0048] Optionally, in this embodiment, a channel-by-channel approach is used to calibrate the color distribution of subsequent frames in the reconstructed video using the distribution parameters of the key reference frame. For example, assuming the color channels are RGB channels, for the R channel, the distribution parameters of the R channel of the key reference frame are used to calibrate the color distribution of the R channel in subsequent frames of the reconstructed video. For the B channel, the distribution parameters of the B channel of the key reference frame are used to calibrate the color distribution of the B channel in subsequent frames of the reconstructed video.

[0049] S706, Generate the target video based on the key reference frame and the calibrated subsequent frames.

[0050] Optionally, in this embodiment, the calibration step is a post-processing step of GVC decoding, which can improve the subjective quality of the generated video without increasing the bitrate.

[0051] Through the above steps S702-S706, after receiving and decoding the bitstream of the video to obtain the reconstructed video, the distribution parameters of each color channel in the key reference frame of the reconstructed video are calculated, wherein the distribution parameters are used to characterize the color features of the corresponding color channel; for the same color channel, the color distribution of subsequent frames in the reconstructed video is calibrated using the distribution parameters of each color channel of the key reference frame to obtain calibrated subsequent frames; and a target video is generated based on the key reference frame and the calibrated subsequent frames. In other words, this embodiment of the application uses a channel-by-channel calibration method, using the distribution parameters of each color channel of the key reference frame to calibrate the color distribution of subsequent frames corresponding to the color channel, thereby solving the technical problem of color shift in reconstructed videos in related technologies and achieving the technical effect of improving the quality of reconstructed videos.

[0052] Since the combination of multiple color channels can represent various colors—for example, the RGB channel, through the combination of red, green, and blue channels, can represent various colors, becoming the most basic way of presenting images—in one possible implementation of this application, for the same color channel, the color distribution of subsequent frames in the video is calibrated and reconstructed using the distribution parameters of a key reference frame to obtain calibrated subsequent frames, including:

[0053] S11, obtain the first mean and first standard deviation of the first color channel of the key reference frame.

[0054] It should be noted that when the color channel is an RGB channel, the first color channel mentioned above can be an R channel, a G channel, or a B channel.

[0055] S12, using the first mean and the first standard deviation, calibrate the color values ​​of each pixel in the second color channel of the subsequent frame to obtain the calibrated color distribution of the second color channel, wherein the first color channel and the second color channel are the same color channel.

[0056] That is, after obtaining the first mean and the first standard deviation, the color values ​​of each pixel in the same color channel of subsequent frames are calibrated using the first mean and the first standard deviation. For example, taking the R channel and 4×4 pixels as an example, after obtaining the mean μ and standard deviation σ of the key reference frame, the color values ​​of 16 pixels are calibrated using the mean μ and the standard deviation σ respectively to obtain the adjusted color values ​​of the 16 pixels.

[0057] Since the final color of an image frame is determined by merging multiple color channels, this application embodiment also proposes:

[0058] S13, combine the color distributions of multiple color channels after calibration to obtain the calibrated subsequent frame.

[0059] By performing channel-by-channel calibration in steps S11 to S13, it can be ensured that the compressed video maintains the same color performance as the original video.

[0060] Optionally, the color values ​​of each pixel in the second color channel of the subsequent frame are calibrated using the first mean and the first standard deviation to obtain the calibrated color distribution of the second color channel, including:

[0061] S121, obtain the initial color value of the target pixel from the second color channel of the subsequent frame, wherein the initial color value is used to characterize the color intensity of the target pixel in the second color channel.

[0062] The target pixel mentioned above can be any pixel among all pixels in the second color channel of subsequent frames, or it can be any pixel among only a portion of the pixels. That is, for a frame of image, only the color distribution of the region of interest to the user is calibrated. For example, in an image that includes a background and a face, only the color distribution of the face is calibrated, while the color distribution of the background is not calibrated, further realizing the flexibility and interactivity of the video color calibration process.

[0063] Optionally, the color value of a pixel can be determined based on the number of bits in the color channel. For example, for an 8-bit color channel, the color value of a pixel can be from 0 to 255; for a 16-bit color channel, the color value of a pixel can be from 0 to 65535. The value range for a 32-bit floating-point color channel can be from 0.0 to 1.0.

[0064] S122, determine the product of the initial color value and the first standard deviation to obtain the first result.

[0065] S123, sum the first result and the first mean to obtain the second result.

[0066] S124, set the second result as the color value of the target pixel after calibration.

[0067] Through the above steps S121 to S124, the standard deviation of the key reference frame is multiplied by the initial color value of the target pixel to be calibrated, and then the product is summed with the mean of the key reference frame to calibrate the initial color of the target pixel. This can further ensure that the compressed video is consistent with the original video in terms of color performance.

[0068] To improve algorithm efficiency and accuracy, enhance data comparability, and reduce the impact of outliers, this application embodiment also proposes that, before determining the product of the initial color value and the first standard deviation to obtain the first result, the above method further includes:

[0069] S125, obtain the second mean and second standard deviation of the second color channel of the subsequent frame;

[0070] S126, using the second mean and the second standard deviation, the color values ​​of each pixel in the second color channel of the subsequent frame are standardized.

[0071] Optionally, standardizing the color values ​​of each pixel in the second color channel of subsequent frames using the second mean and the second standard deviation can specifically be done by: taking the difference between the color value of the pixel in the second color channel and the second mean, and then comparing the difference with the second standard deviation to obtain the standardized result of the color value of each pixel.

[0072] Optionally, in this embodiment of the application, the mean and standard deviation corresponding to the color channels are also proposed to be determined in the following way:

[0073] S21, Determine the average value based on the color value of each pixel in the color channel and the total number of pixels;

[0074] S22, Determine the standard deviation based on the color value of each pixel in the color channel and the mean value.

[0075] For example, taking the color channels as RGB channels, assuming the RGB channels are arranged in order, the mean and standard deviation of each RGB channel in the key reference frame are calculated as shown in the following formulas (1) and (2):

[0076]

[0077] in, The key reference frame for reconstruction is represented by H and W, which represent the height and width of the frame, respectively. The index [c, i, j] represents the value at the c-th channel, the i-th position in the height direction, and the j-th position in the width direction.

[0078] For example, for a 4×4 frame, its color plane and corresponding pixel values ​​are as follows: Figure 9 As shown. The mean of the red channel can be calculated as follows:

[0079]

[0080] Similarly, each subsequent frame of the reconstruction The channel-wise distribution is calculated as μ i,r / g / b and σ i,r / g / b Then, for each Channel calibration calculation is shown in Formula 3:

[0081]

[0082] In this process, each channel of a subsequent frame is first standardized using its own distribution parameters, and then adjusted based on the distribution parameters provided by the key reference frame for reconstruction.

[0083] In summary, the video decoding method in this application provides a channel-by-channel video color calibration method as a post-processing step for GVC decoding, such as... Figure 10As shown, after decoding the input video, an output video is obtained. This output video includes a key reference frame and reconstructed subsequent frames. The standard deviation and mean of the reconstructed key reference frame are calculated. Then, the mean and standard deviation of the reconstructed key reference frame are used to provide the distribution parameters required for subsequent frame calibration. The specific calibration method is described in Formula 3 above. Finally, the calibrated subsequent frames are output. Through the above method of the embodiments of this application, the subjective quality of the generated video can be improved without increasing the additional bitrate cost.

[0084] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0085] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0086] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide a video decoding device applied to a generative video compression framework, such as... Figure 11 The diagram shown is a structural block diagram of a video decoding apparatus according to an embodiment of this application, which may include:

[0087] The calculation module 1102 is used to receive the bit stream of the video and decode it to obtain the reconstructed video, and then calculate the distribution parameters of each color channel of the key reference frame in the reconstructed video, wherein the distribution parameters are used to characterize the color features of the corresponding color channel.

[0088] It should be noted that the reconstructed video includes a key reference frame and subsequent frames. The key reference frame typically contains complete image information, while subsequent frames generally store differences from the reference frame. The reconstructed video is obtained as follows: Figure 8 As shown. Figure 8 This paper demonstrates the general encoding-decoding process of the GVC algorithm, using generative face video coding as an example. At the encoder, key reference frames of the video are compressed using traditional coding techniques, while subsequent frames are represented using compact transmission symbols and encoded into the output bitstream. At the decoder, the decoded key reference frames and the compact face information are input into the synthesis model to reconstruct the video, thus obtaining the reconstructed video.

[0089] It should be noted that the color channels mentioned above, in addition to RGB (red, green, blue) channels, can also include channels from other color spaces or additional channels for specific application scenarios. For example, YCbCr (Y: luminance channel, representing the image's brightness information; Cb: blue chrominance channel, representing the difference between blue and luminance; Cr: red chrominance channel, representing the difference between red and luminance) and YUV (Y: luminance channel, representing the image's brightness information; U: chrominance U channel, representing blue chrominance information; V: chrominance V channel, representing red chrominance information). Depending on the specific application scenario, color spaces such as YPbPr (Y: luminance channel, representing the image's brightness information; Pb: blue chrominance channel; Pr: red chrominance channel), HSL / HSV (H: hue, representing color; S: saturation, representing color purity; L / V: brightness / value, representing color lightness / darkness), and Lab (L: luminance; a: representing color information from green to magenta; b: representing color information from blue to yellow) can also be used.

[0090] Additionally, it should be noted that the types of videos mentioned above include, but are not limited to: facial videos, motion videos, conference and speech videos, educational videos, entertainment and media videos, game videos, user-generated content, surveillance videos, real-time communication videos, virtual reality and augmented reality videos, etc., which can be expanded according to specific application scenarios and needs. Each type of video has its own specific visual and dynamic characteristics.

[0091] Optionally, the above distribution parameters include, but are not limited to, the mean and standard deviation, and may also be other characteristics of the color value distribution, such as kurtosis, skewness, frequency domain characteristics, etc., which are not limited here.

[0092] The first calibration module 1104 is used to calibrate the color distribution of subsequent frames in the reconstructed video using the distribution parameters of the key reference frame for the same color channel, so as to obtain the calibrated subsequent frames.

[0093] Optionally, in this embodiment, a channel-by-channel approach is used to calibrate the color distribution of subsequent frames in the reconstructed video using the distribution parameters of the key reference frame. For example, assuming the color channels are RGB channels, for the R channel, the distribution parameters of the R channel of the key reference frame are used to calibrate the color distribution of the R channel in subsequent frames of the reconstructed video. For the B channel, the distribution parameters of the B channel of the key reference frame are used to calibrate the color distribution of the B channel in subsequent frames of the reconstructed video.

[0094] The first generation module 1106 is used to generate the target video based on the key reference frame and the calibrated subsequent frames.

[0095] Optionally, in this embodiment, the calibration step is a post-processing step of GVC decoding, which can improve the subjective quality of the generated video without increasing the bitrate.

[0096] pass Figure 11 The apparatus shown receives the bitstream of a video and decodes it to obtain a reconstructed video. Then, it calculates the distribution parameters of each color channel in a key reference frame of the reconstructed video. These distribution parameters characterize the color features of the corresponding color channel. For the same color channel, the distribution parameters of the key reference frame are used to calibrate the color distribution of subsequent frames in the reconstructed video, resulting in calibrated subsequent frames. Finally, a target video is generated based on the key reference frame and the calibrated subsequent frames. In other words, this embodiment uses a channel-by-channel calibration method, employing the distribution parameters of each color channel in a key reference frame to calibrate the color distribution of subsequent frames corresponding to that color channel. This solves the technical problem of color shift in reconstructed videos in related technologies and achieves the technical effect of improving the quality of the reconstructed video.

[0097] The aforementioned first calibration module 1104 includes: an acquisition unit, configured to acquire the first mean and first standard deviation of the first color channel of the key reference frame; a calibration unit, configured to use the first mean and the first standard deviation to calibrate the color values ​​of each pixel of the second color channel of the subsequent frame to obtain the calibrated color distribution of the second color channel, wherein the first color channel and the second color channel are the same color channel; and a combination unit, configured to combine the calibrated color distributions of multiple color channels to obtain the calibrated subsequent frame.

[0098] The calibration unit includes: a first acquisition subunit, configured to acquire an initial color value of a target pixel from the second color channel of the subsequent frame, wherein the initial color value is used to characterize the color intensity of the target pixel in the second color channel; a second acquisition subunit, configured to determine the product of the initial color value and the first standard deviation to obtain a first result; a third acquisition subunit, configured to sum the first result and the first mean to obtain a second result; and a setting subunit, configured to set the second result as the calibrated color value of the target pixel.

[0099] Optionally, the first calibration module 1104 further includes: a first processing unit, configured to obtain a second mean and a second standard deviation of the second color channel of the subsequent frame before determining the product of the initial color value and the first standard deviation to obtain a first result; and a second processing unit, configured to use the second mean and the second standard deviation to standardize the color value of each pixel of the second color channel of the subsequent frame.

[0100] The aforementioned device is also used to determine the mean and standard deviation corresponding to the color channels in the following ways:

[0101] The mean is determined based on the color values ​​of each pixel in the color channel and the total number of pixels; the standard deviation is determined based on the color values ​​of each pixel in the color channel and the mean.

[0102] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0103] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide another video decoding method. For example... Figure 12 The image shown is a video decoding method according to an embodiment of this application, applied to a generative video compression framework, including:

[0104] S1202, receive the bit stream of video and indication information, wherein the indication information includes: first indication information, second indication information and third indication information, the first indication information is used to indicate the key reference frame, the second indication information is used to indicate the subsequent frame to be calibrated, and the third indication information is used to indicate the distribution parameters of each color channel of the key reference frame in the original video.

[0105] Optionally, in the embodiments of this application, the above-mentioned indication information may also include at least one of the above-mentioned indication information. For example, for the reconstructed video, there may be some distortion, so the distribution parameters of the color channels of the key reference frames in the reconstructed video may have some error. In this case, the distribution parameters of each color channel of the key reference frame corresponding to the original video can be received, that is, the above-mentioned third indication information. Then, based on the determined key reference frame of the reconstructed video and the subsequent frame to be calibrated, the color distribution calibration of the subsequent frame to be calibrated is performed.

[0106] Since the key reference frame may be in the first or last frame of the video, to further improve video decoding efficiency, the position of the key reference frame can be indicated by the indication information sent by the encoder (i.e., the first indication information mentioned above). Then, based on the distribution parameters of each color channel of the key reference frame in the reconstructed video and the subsequent frames to be calibrated, color distribution calibration is performed on the subsequent frames to be calibrated. Not every subsequent frame needs color distribution calibration; in this case, the subsequent frames to be calibrated can be indicated by the indication information sent by the encoder (i.e., the second indication information mentioned above). That is, in this embodiment, by combining the above-mentioned indication information sent by the encoder, the decoding efficiency and decoding accuracy of the decoder can be further improved.

[0107] Optionally, in the embodiments of this application, the above-mentioned indication information may be encoded in the bitstream of the video and sent, or it may be sent separately. Here, there is no specific limitation on the method of sending the above-mentioned indication information.

[0108] S1204 After decoding the bitstream of the video to obtain the reconstructed video, the key reference frame and the subsequent frame to be calibrated in the reconstructed video are determined based on the first indication information and the second indication information.

[0109] It should be noted that the reconstructed video includes a key reference frame and subsequent frames. The key reference frame typically contains complete image information, while subsequent frames generally store differences from the reference frame. The reconstructed video is obtained as follows: Figure 8 As shown. Figure 8 This paper demonstrates the general encoding-decoding process of the GVC algorithm, using generative face video coding as an example. At the encoder, key reference frames of the video are compressed using traditional coding techniques, while subsequent frames are represented using compact transmission symbols and encoded into the output bitstream. At the decoder, the decoded key reference frames and the compact face information are input into the synthesis model to reconstruct the video, thus obtaining the reconstructed video.

[0110] It should be noted that the color channels mentioned above, in addition to RGB (red, green, blue) channels, can also include channels from other color spaces or additional channels for specific application scenarios. For example, YCbCr (Y: luminance channel, representing the image's brightness information; Cb: blue chrominance channel, representing the difference between blue and luminance; Cr: red chrominance channel, representing the difference between red and luminance) and YUV (Y: luminance channel, representing the image's brightness information; U: chrominance U channel, representing blue chrominance information; V: chrominance V channel, representing red chrominance information). Depending on the specific application scenario, color spaces such as YPbPr (Y: luminance channel, representing the image's brightness information; Pb: blue chrominance channel; Pr: red chrominance channel), HSL / HSV (H: hue, representing color; S: saturation, representing color purity; L / V: brightness / value, representing color lightness / darkness), and Lab (L: luminance; a: representing color information from green to magenta; b: representing color information from blue to yellow) can also be used.

[0111] Additionally, it should be noted that the types of videos mentioned above include, but are not limited to: facial videos, motion videos, conference and speech videos, educational videos, entertainment and media videos, game videos, user-generated content, surveillance videos, real-time communication videos, virtual reality and augmented reality videos, etc., which can be expanded according to specific application scenarios and needs. Each type of video has its own specific visual and dynamic characteristics.

[0112] Optionally, the above distribution parameters include, but are not limited to, the mean and standard deviation, and may also be other characteristics of the color value distribution, such as kurtosis, skewness, frequency domain characteristics, etc., which are not limited here.

[0113] S1206, For the same color channel, use the third indication information to calibrate the color distribution of the subsequent frames to be calibrated in the reconstructed video, and obtain the calibrated subsequent frames.

[0114] Optionally, in this embodiment, a channel-by-channel approach is used to calibrate the color distribution of subsequent frames in the reconstructed video using the distribution parameters of the original video key reference frame. For example, assuming the color channels are RGB channels, for the R channel, the distribution parameters of the R channel of the original video key reference frame are used to calibrate the color distribution of the R channel in subsequent frames of the reconstructed video. For the B channel, the distribution parameters of the B channel of the original video key reference frame are used to calibrate the color distribution of the B channel in subsequent frames of the reconstructed video. For specific calibration methods, please refer to the calibration method in the first video decoding method described above, which will not be described in detail here.

[0115] S1208, generate the target video based on the key reference frame in the reconstructed video and the calibrated subsequent frame.

[0116] Optionally, in this embodiment, the calibration step is a post-processing step of GVC decoding, which can improve the subjective quality of the generated video without increasing the bitrate.

[0117] Through the aforementioned steps S1202-S1208, the video bitstream and indication information are received. The indication information includes: first indication information indicating a key reference frame, second indication information indicating a subsequent frame to be calibrated, and third indication information indicating the distribution parameters of each color channel in the key reference frame of the original video. After decoding the video bitstream to obtain the reconstructed video, the key reference frame and the subsequent frame to be calibrated in the reconstructed video are determined based on the first and second indication information. For the same color channel, the third indication information is used to calibrate the color distribution of the subsequent frame to be calibrated in the reconstructed video, resulting in a calibrated subsequent frame. A target video is generated based on the key reference frame in the reconstructed video and the calibrated subsequent frame. In other words, in the video decoding process of this application embodiment, based on the video bitstream and indication information sent by the encoder, a channel-by-channel calibration method is used to calibrate the color distribution of the corresponding color channel in the subsequent frame through the distribution parameters of each color channel in the indication information. This solves the technical problem of color shift in reconstructed video in related technologies and achieves the technical effect of improving the quality of the reconstructed video.

[0118] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide another video decoding device. For example... Figure 13 The diagram shown is a structural block diagram of a video decoding apparatus according to an embodiment of this application. The apparatus may include:

[0119] The receiving module 1302 is used to receive the bit stream of video and indication information, wherein the indication information includes: first indication information, second indication information and third indication information, the first indication information is used to indicate the key reference frame, the second indication information is used to indicate the subsequent frame to be calibrated, and the third indication information is used to indicate the distribution parameters of each color channel of the key reference frame in the original video.

[0120] The first determining module 1304 is used to determine the key reference frame and the subsequent frame to be calibrated in the reconstructed video based on the first indication information and the second indication information after decoding the bit stream of the video to obtain the reconstructed video.

[0121] The second calibration module 1306 is used to calibrate the color distribution of subsequent frames to be calibrated in the reconstructed video for the same color channel using the third indication information, so as to obtain the calibrated subsequent frames.

[0122] The second generation module 1308 generates a target video based on the key reference frame in the reconstructed video and the calibrated subsequent frames.

[0123] pass Figure 13 The apparatus shown receives a video bitstream and indication information, wherein the indication information includes: first indication information indicating a key reference frame, second indication information indicating a subsequent frame to be calibrated, and third indication information indicating the distribution parameters of each color channel of the key reference frame in the original video. After decoding the video bitstream to obtain a reconstructed video, the key reference frame and the subsequent frame to be calibrated in the reconstructed video are determined based on the first and second indication information. For the same color channel, the color distribution of the subsequent frame to be calibrated in the reconstructed video is calibrated using the third indication information to obtain calibrated subsequent frames. A target video is generated based on the key reference frame in the reconstructed video and the calibrated subsequent frames. In other words, in the video decoding process, this embodiment uses the video bitstream and indication information sent by the encoder to calibrate the color distribution of the corresponding color channel in the subsequent frames by means of the distribution parameters of each color channel in the indication information, thereby solving the technical problem of color offset in reconstructed video in related technologies and achieving the technical effect of improving the quality of reconstructed video.

[0124] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0125] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide a video coding method applied to a generative video compression framework, such as... Figure 14 The diagram shown is a flowchart of a video encoding method according to an embodiment of this application. The method may include:

[0126] S1402, the key reference frames in the original video are encoded to obtain the first encoded bitstream.

[0127] S1404, the subsequent frames in the original video are represented using compact transmission symbols, and the representation results are encoded to obtain the second coded bitstream.

[0128] It should be noted that the specific methods for determining the first and second encoded bitstreams mentioned above can be found in [reference needed]. Figure 8 This will not be discussed in detail here.

[0129] S1406, the first encoded bitstream and the second encoded bitstream are determined as the bitstream of the video corresponding to the original video, and the bitstream of the video is sent so that the decoder, after receiving the bitstream of the video, uses one of the video decoding methods described above to perform video decoding and obtain the target video.

[0130] Through steps S1402 to S1406, the key reference frames in the original video are encoded to obtain a first encoded bitstream; subsequent frames in the original video are represented using compact transmission symbols, and the representation result is encoded to obtain a second encoded bitstream; the first encoded bitstream and the second encoded bitstream are determined as the bitstream of the video corresponding to the original video, and the bitstream of the video is sent so that the decoder, after receiving the bitstream of the video, uses one of the video decoding methods described above to decode the video and obtain the target video. That is, by sending the bitstream of the encoded video from the original video to the decoder, the decoder can calibrate the color distribution of the corresponding color channels of subsequent frames by using the distribution parameters of each color channel of the key reference frames in a channel-by-channel calibration manner. This solves the technical problem of color shift in reconstructed video in related technologies and achieves the technical effect of improving the quality of the reconstructed video.

[0131] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide a video encoding apparatus. For example... Figure 15 The diagram shown is a structural block diagram of a video encoding apparatus according to an embodiment of this application. The apparatus may include:

[0132] The first encoding module 1502 is used to encode the key reference frames in the original video to obtain the first encoded bitstream;

[0133] The second encoding module 1504 is used to represent subsequent frames in the original video using compact transmission symbols and to encode the representation results to obtain a second encoded bitstream.

[0134] The first sending module 1506 is used to determine the first encoded bit stream and the second encoded bit stream as the bit stream of the video corresponding to the original video, and send the bit stream of the video so that the decoder, after receiving the bit stream of the video, uses the video decoding method described above to perform video decoding to obtain the target video.

[0135] pass Figure 15 The apparatus shown encodes key reference frames in the original video to obtain a first encoded bitstream; it then represents subsequent frames in the original video using compact transmission symbols and encodes the representation results to obtain a second encoded bitstream; the first and second encoded bitstreams are used to determine the bitstream of the video corresponding to the original video, and this bitstream is sent so that the decoder, upon receiving the bitstream, can use one of the video decoding methods described above to decode the video and obtain the target video. In other words, by sending the bitstream of the encoded video to the decoder, the decoder can calibrate the color distribution of subsequent frames corresponding to the same color channel using the distribution parameters of each color channel of the key reference frame in a channel-by-channel calibration manner. This solves the technical problem of color shift in reconstructed video in related technologies and achieves the technical effect of improving the quality of the reconstructed video.

[0136] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0137] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide another video coding method, applied to a generative video compression framework, such as... Figure 16 The diagram shown is a flowchart of a video encoding method according to another embodiment of this application. The method may include:

[0138] S1602, the key reference frames in the original video are encoded to obtain the first encoded bitstream.

[0139] S1604, the subsequent frames in the original video are represented using compact transmission symbols, and the representation results are encoded to obtain the second coded bitstream.

[0140] It should be noted that the specific methods for determining the first and second encoded bitstreams mentioned above can be found in [reference needed]. Figure 8 This will not be discussed in detail here.

[0141] S1606, the first encoded bitstream and the second encoded bitstream are determined as the bitstreams of the video corresponding to the original video.

[0142] S1608, determine the indication information and send the indication information and the bit stream of the video, so that after receiving the bit stream of the video and the indication information, the decoder uses the other video decoding method described above to decode the video and obtain the target video. The indication information includes: first indication information, second indication information and third indication information. The first indication information is used to indicate the key reference frame, the second indication information is used to indicate the subsequent frame to be calibrated, and the third indication information is used to indicate the distribution parameters of each color channel of the key reference frame in the original video.

[0143] Through steps S1602 to S1608, the key reference frames in the original video are encoded to obtain a first coded bitstream; subsequent frames in the original video are represented using compact transmission symbols, and the representation result is encoded to obtain a second coded bitstream; the first coded bitstream and the second coded bitstream are determined as the bitstream of the video corresponding to the original video; indication information is determined, and the indication information and the video bitstream are sent, so that the decoder, after receiving the video bitstream and the indication information, uses the other video decoding method described above to decode the video and obtain the target video. That is, by sending the bitstream of the encoded video and the indication information to the decoder, the decoder can combine the indication information and use a channel-by-channel calibration method to calibrate the color distribution of the corresponding color channels in subsequent frames by using the distribution parameters of each color channel of the key reference frames in the original video. This solves the technical problem of color shift in reconstructed video in related technologies and achieves the technical effect of improving the quality of reconstructed video.

[0144] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide another video encoding apparatus. For example... Figure 17 The diagram shown is a structural block diagram of a video encoding apparatus according to an embodiment of this application. The apparatus may include:

[0145] The third encoding module 1702 is used to encode the key reference frames in the original video to obtain the first encoded bitstream;

[0146] The fourth encoding module 1704 is used to represent subsequent frames in the original video using compact transmission symbols and to encode the representation results to obtain the second encoded bitstream;

[0147] The second determining module 1704 is used to determine the first encoded bitstream and the second encoded bitstream as the bitstream of the video corresponding to the original video;

[0148] The second sending module 1706 is used to determine the indication information and send the indication information and the bit stream of the video, so that after receiving the bit stream of the video and the indication information, the decoder uses the other video decoding method described above to decode the video and obtain the target video. The indication information includes: first indication information, second indication information and third indication information. The first indication information is used to indicate the key reference frame, the second indication information is used to indicate the subsequent frame to be calibrated, and the third indication information is used to indicate the distribution parameters of each color channel of the key reference frame in the original video.

[0149] pass Figure 17 The apparatus encodes key reference frames in the original video to obtain a first coded bitstream; it represents subsequent frames in the original video using compact transmission symbols and encodes the representation results to obtain a second coded bitstream; it determines the first and second coded bitstreams as the bitstreams of the video corresponding to the original video; it determines indication information and sends the indication information and the video bitstream, so that the decoder, upon receiving the video bitstream and the indication information, uses the aforementioned alternative video decoding method to decode the video and obtain the target video. That is, by sending the bitstream of the encoded original video and the indication information to the decoder, the decoder can combine the indication information with a channel-by-channel calibration method, using the distribution parameters of each color channel of the key reference frame in the original video to calibrate the color distribution of the corresponding color channel in subsequent frames, thereby solving the technical problem of color shift in reconstructed video in related technologies and achieving the technical effect of improving the quality of reconstructed video.

[0150] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0151] This application provides a non-transitory computer-readable storage medium storing a video bitstream, the non-transitory computer-readable storage medium being part of a computing device configured to execute a set of instructions to decode the bitstream according to the operations provided in this application embodiment, or the computing device being configured to execute a set of instructions to encode the video bitstream according to the operations provided in this application embodiment.

[0152] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0153] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0154] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0155] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0156] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0157] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0158] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0159] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0160] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0161] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0162] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0163] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A video decoding method, applied to a generative video compression framework, comprising: After receiving the bitstream of the video and decoding it to obtain the reconstructed video, the distribution parameters of each color channel in the key reference frame of the reconstructed video are calculated, wherein the distribution parameters are used to characterize the color features of the corresponding color channel; For the same color channel, the color distribution of subsequent frames in the reconstructed video is calibrated using the distribution parameters of the key reference frame to obtain calibrated subsequent frames; The target video is generated based on the key reference frame and the calibrated subsequent frames.

2. The method according to claim 1, wherein, The distribution parameters include at least the mean and standard deviation of each color channel.

3. The method according to claim 2, wherein, For the same color channel, the color distribution of subsequent frames in the reconstructed video is calibrated using the distribution parameters of the key reference frame to obtain calibrated subsequent frames, including: Obtain the first mean and first standard deviation of the first color channel of the key reference frame; The color values ​​of each pixel in the second color channel of the subsequent frame are calibrated using the first mean and the first standard deviation to obtain the calibrated color distribution of the second color channel, wherein the first color channel and the second color channel are the same color channel; The color distributions of multiple color channels after calibration are combined to obtain the calibrated subsequent frames.

4. The method according to claim 3, wherein, The color values ​​of each pixel in the second color channel of the subsequent frames are calibrated using the first mean and the first standard deviation to obtain the calibrated color distribution of the second color channel, including: The initial color value of the target pixel is obtained from the second color channel of the subsequent frame, wherein the initial color value is used to characterize the color intensity of the target pixel in the second color channel; The product of the initial color value and the first standard deviation is determined to obtain the first result; Summing the first result and the first mean yields the second result; Set the second result as the color value of the target pixel after calibration.

5. The method according to claim 4, wherein, Before determining the product of the initial color value and the first standard deviation to obtain the first result, the method further includes: Obtain the second mean and second standard deviation of the second color channel in the subsequent frames; The color values ​​of each pixel in the second color channel of the subsequent frame are standardized using the second mean and the second standard deviation.

6. The method according to claim 2, wherein, The mean and standard deviation of each color channel are determined as follows: The mean value is determined based on the color value of each pixel in the color channel and the total number of pixels. The standard deviation is determined based on the color value of each pixel in the color channel and the mean value.

7. The method according to any one of claims 1 to 6, wherein, The color channels include at least the red, green, and blue (RGB) channels.

8. A video coding method applied to a generative video compression framework, comprising: Encode the key reference frames in the original video to obtain the first encoded bitstream; Subsequent frames in the original video are represented using compact transmission symbols, and the representation results are encoded to obtain a second coded bitstream; The first encoded bitstream and the second encoded bitstream are determined as the bitstream of the video corresponding to the original video, and the bitstream of the video is sent so that the decoder, after receiving the bitstream of the video, performs video decoding using the method described in any one of claims 1 to 7 to obtain the target video.

9. A video decoding apparatus, applied to a generative video compression framework, comprising: The calculation module is used to receive the bit stream of the video and decode it to obtain the reconstructed video, and then calculate the distribution parameters of each color channel of the key reference frame in the reconstructed video, wherein the distribution parameters are used to characterize the color features of the corresponding color channel. The first calibration module is used to calibrate the color distribution of subsequent frames in the reconstructed video using the distribution parameters of the key reference frame for the same color channel, so as to obtain calibrated subsequent frames. The first generation module is used to generate a target video based on the key reference frame and the calibrated subsequent frames.

10. A video encoding apparatus, applied to a generative video compression framework, comprising: The first encoding module is used to encode the key reference frames in the original video to obtain the first encoded bitstream; The second encoding module is used to represent subsequent frames in the original video using compact transmission symbols and to encode the representation results to obtain a second encoded bitstream. The first sending module is configured to determine the first encoded bitstream and the second encoded bitstream as the bitstream of the video corresponding to the original video, and send the bitstream of the video, so that the decoder, after receiving the bitstream of the video, performs video decoding using the method of any one of claims 1 to 7 to obtain the target video.

11. A non-transitory computer-readable storage medium storing a bitstream of video, said non-transitory computer-readable storage medium being part of a computing device configured to execute a set of instructions to cause the computing device to decode the bitstream of video according to operations including: After decoding the bitstream of the video to obtain the reconstructed video, the distribution parameters of each color channel in the key reference frame of the reconstructed video are calculated, wherein, The distribution parameters are used to characterize the color features of the corresponding color channels; For the same color channel, the color distribution of subsequent frames in the reconstructed video is calibrated using the distribution parameters of the key reference frame to obtain calibrated subsequent frames; a target video is generated based on the key reference frame and the calibrated subsequent frames.

Images