Video decoding method, video encoding method and apparatus

Abstract

The application provides a video decoding method and device, a video encoding method and device, and relates to the technical field of video coding. The method comprises the following steps: decoding the encoding data of a current frame, obtaining the super-prior residual mean and super-prior residual variance of each pixel point of the current frame, and obtaining the residual information of the current frame; obtaining the residual skip threshold of the current frame according to a first quality adjustment parameter, wherein the first quality adjustment parameter is a parameter used for adjusting the encoding quality level of a video sequence to which the current frame belongs; obtaining the reconstructed residual feature of each pixel point of the current frame according to the residual skip threshold; and reconstructing the current frame according to the reconstructed residual feature of each pixel point of the current frame, so as to obtain the reconstructed frame of the current frame. The embodiment of the application is used for improving the encoding efficiency of a video.

Description

Technical Field

[0001] Some embodiments of this application relate to the field of video encoding and decoding technology. More specifically, they relate to a video decoding method, a video encoding method, and an apparatus. Background Technology

[0002] With the widespread use of terminal devices, people are not only pursuing high-quality videos, but also demanding higher video transmission speeds, which undoubtedly brings new challenges to video encoding and decoding technologies.

[0003] Current video encoding and decoding schemes rely on manually designed feature extractors, which struggle to effectively identify and extract high-dimensional redundant information from videos. This leads to excessive storage space consumption and wasted transmission resources. With the rapid development of deep learning, numerous learning-based encoding and decoding methods have emerged. Among them, neural network encoding and decoding based on Variational Autoencoders (VAEs) stand out. This network can perform complex nonlinear mappings on pixel-level features of images, transforming them into a latent feature space, thereby extracting redundant information from the video and obtaining key latent features. To further overcome the problem of the inability to completely eliminate spatial coupling relationships when processing latent features of image pixels by a single neural network, a super-prior prediction network is added to related technologies. The super-prior prediction network extracts super-prior features from the acquired latent features and decodes them into the mean and variance of the latent features. For the mean and variance predicted by the super-prior network, the skip model module determines the closeness between the mean of the super-prior prediction network and the true value based on the relationship between the variance and the skip threshold. If the variance is less than the skip threshold, the mean is used to directly replace the true value, thereby significantly reducing the bitrate required for video transmission and improving transmission efficiency while ensuring controllable video quality loss. However, different video sequences will be encoded using different coding quality levels based on video quality requirements, transmission stability requirements, network bandwidth, etc. The selected coding quality level has a significant impact on the overall video quality, and videos at different bitrates should use different skip thresholds. However, related technologies do not determine the skip threshold based on the coding quality level, so there is still room for improvement in coding efficiency. Summary of the Invention

[0004] Exemplary embodiments of this application provide a video decoding method, a video encoding method, and an apparatus for improving video encoding efficiency.

[0005] The technical solutions provided by some embodiments of this application are as follows:

[0006] In a first aspect, some embodiments of this application provide a video decoding method, including:

[0007] Decode the encoded data of the current frame, obtain the mean and variance of the prior residuals of each pixel in the current frame, as well as the residual information of the current frame;

[0008] The residual skip threshold of the current frame is obtained according to the first quality adjustment parameter, which is a parameter used to adjust the coding quality level of the video sequence to which the current frame belongs;

[0009] Obtain the reconstruction residual features of each pixel in the current frame; for each pixel in the current frame, if the variance of the prior residual of the pixel is less than or equal to the residual skip threshold of the current frame, then the mean of the prior residual of the pixel is determined as the reconstruction residual feature of the pixel; if the variance of the prior residual of the pixel is greater than the residual skip threshold of the current frame, then the reconstruction residual feature of the pixel is obtained according to the residual information.

[0010] The current frame is reconstructed based on the reconstruction residual features of each pixel to obtain the reconstructed frame.

[0011] Secondly, some embodiments of this application provide a video encoding method, including:

[0012] Obtain the potential residual features of each pixel in the current frame;

[0013] The potential residual features of each pixel in the current frame are quantized to obtain the quantized residual features of each pixel in the current frame.

[0014] Based on the latent residual features of each pixel in the current frame, obtain the mean and variance of the prior residuals of each pixel in the current frame.

[0015] The residual skip threshold of the current frame is obtained according to the first quality adjustment parameter, which is a parameter used to adjust the coding quality level of the video sequence to which the current frame belongs;

[0016] Obtain the residual information of the current frame, wherein the residual information includes the quantized residual features of each pixel whose super-prior residual variance is greater than the residual skip threshold;

[0017] Obtain the encoded data of the current frame based on the residual information of the current frame.

[0018] Thirdly, some embodiments of this application provide a video decoding apparatus, including:

[0019] The decoding unit is used to decode the encoded data of the current frame, obtain the mean and variance of the prior residuals of each pixel in the current frame, as well as the residual information of the current frame.

[0020] The residual threshold acquisition unit is used to acquire the residual skip threshold of the current frame according to the first quality adjustment parameter, wherein the first quality adjustment parameter is a parameter used to adjust the coding quality level of the video sequence to which the current frame belongs;

[0021] The residual reconstruction unit is used to obtain the reconstruction residual features of each pixel in the current frame. For each pixel in the current frame, if the variance of the prior residual of the pixel is less than or equal to the residual skip threshold of the current frame, the mean of the prior residual of the pixel is determined as the reconstruction residual feature of the pixel. If the variance of the prior residual of the pixel is greater than the residual skip threshold of the current frame, the reconstruction residual feature of the pixel is obtained according to the residual information.

[0022] The frame reconstruction unit is used to reconstruct the current frame based on the reconstruction residual features of each pixel in the current frame, so as to obtain the reconstructed frame of the current frame.

[0023] Fourthly, some embodiments of this application provide a video encoding apparatus, including:

[0024] The residual feature acquisition unit is used to acquire the potential residual features of each pixel in the current frame.

[0025] The residual quantization unit is used to quantize the potential residual features of each pixel in the current frame to obtain the quantized residual features of each pixel in the current frame.

[0026] The prior information acquisition unit is used to obtain the mean and variance of the prior residuals of each pixel in the current frame based on the potential residual features of each pixel in the current frame.

[0027] The residual threshold acquisition unit is used to acquire the residual skip threshold of the current frame according to the first quality adjustment parameter, wherein the first quality adjustment parameter is a parameter used to adjust the coding quality level of the video sequence to which the current frame belongs;

[0028] The residual information acquisition unit is used to acquire the residual information of the current frame, wherein the residual information includes the quantized residual features of each pixel whose super-prior residual variance is greater than the residual skip threshold;

[0029] The encoded data acquisition unit is used to acquire the encoded data of the current frame based on the residual information of the current frame.

[0030] Fifthly, some embodiments of this application provide a video decoding apparatus, including: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to implement the video decoding method described in the first aspect when executing the computer program.

[0031] In a sixth aspect, some embodiments of this application provide a video encoding apparatus, including: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to implement the video encoding method described in the second aspect when executing the computer program.

[0032] In a seventh aspect, some embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a computing device, causes the computing device to implement the video decoding method of the first aspect or the video encoding method of the second aspect.

[0033] Eighthly, some embodiments of this application provide a chip including a processor and a memory, the memory being used to store programs or instructions executable on the processor, and the processor being used to execute the programs or instructions to cause the video decoding method of the first aspect or the video encoding method of the second aspect to be executed.

[0034] Ninthly, some embodiments of this application provide a computer program product that, when run on a computer, causes the computer to implement the video decoding method described in the first aspect or the video encoding method described in the second aspect.

[0035] The video decoding method provided in this application first decodes the encoded data to obtain the mean and variance of the prior residuals of each pixel in the current frame, as well as the residual information of the current frame. Then, it determines the residual skip threshold of the current frame according to a first quality adjustment parameter used to adjust the encoding quality level of the video sequence to which the current frame belongs. For each pixel in the current frame, if the variance of the prior residuals of the pixel is less than or equal to the residual skip threshold of the current frame, the mean of the prior residuals of the pixel is determined as the reconstructed residual feature of the pixel. If the variance of the prior residuals of the pixel is greater than the residual skip threshold of the current frame, the reconstructed residual feature of the pixel is obtained according to the residual information, thereby obtaining the reconstructed residual features of each pixel in the current frame. Finally, the current frame is reconstructed according to the reconstructed residual features of each pixel in the current frame to obtain the reconstructed frame of the current frame. Since the video decoding method provided in this application embodiment can determine the residual skip threshold of the current frame based on the first quality adjustment parameter used to adjust the encoding quality level of the video sequence to which the current frame belongs, this application embodiment can determine whether to skip encoding the quantization residual features of the pixel based on the residual skip threshold adapted to the quality level, thus improving the encoding efficiency of video. Attached Figure Description

[0036] To more clearly illustrate the implementation methods in some embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0037] Figure 1 The diagram shows a schematic representation of the structure of a video decoder provided in some embodiments;

[0038] Figure 2 The diagram shows a schematic representation of the structure of a video encoder provided in some embodiments;

[0039] Figure 3 The diagram shows a schematic representation of the structure of a video decoding system provided in some embodiments;

[0040] Figure 4 The diagram shows a schematic representation of the skip module provided in some embodiments;

[0041] Figure 5 A flowchart illustrating the steps of a video decoding method provided in some embodiments is shown;

[0042] Figure 6 Flowcharts of the steps of a video decoding method provided in other embodiments are shown;

[0043] Figure 7 A flowchart of the steps of a video decoding method provided in some other embodiments is shown;

[0044] Figure 8 A flowchart illustrating the steps of a video encoding method provided in some embodiments is shown;

[0045] Figure 9 Flowcharts of the steps of a video encoding method provided in other embodiments are shown;

[0046] Figure 10 A flowchart of the steps of a video encoding method provided in some other embodiments is shown;

[0047] Figure 11 Schematic diagrams of the video decoding apparatus in some embodiments are shown;

[0048] Figure 12 Schematic diagrams of the video decoding apparatus in other embodiments are shown;

[0049] Figure 13 Schematic diagrams of the video encoding apparatus in some embodiments are shown;

[0050] Figure 14 Schematic diagrams of the video encoding apparatus in other embodiments are shown;

[0051] Figure 15 A schematic diagram of the hardware structure of an electronic device in some embodiments of this application is shown. Detailed Implementation

[0052] It should be noted that the brief descriptions of terms in this application are only for the purpose of facilitating the understanding of the embodiments described below. In order to make the purpose and implementation of this application clearer, the exemplary embodiments of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0053] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0054] The use of phrases such as "some implementations" or "some embodiments" in the specification indicates that the described implementations or embodiments may include specific features, structures, or characteristics, but not every embodiment may necessarily include that specific feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same implementation. Additionally, when describing a specific feature, structure, or characteristic in connection with an embodiment, it is considered that implementing that feature, structure, or characteristic in connection with other implementations (whether explicitly described herein or not) is within the knowledge of those skilled in the art.

[0055] Figure 1 This is a schematic diagram of the structure of a video decoder provided in some embodiments of this application, which can be used to perform the video decoding method provided in the embodiments of this application.

[0056] Reference Figure 1 As shown, in some embodiments, a video decoder that can be used to perform the video decoding method provided in the embodiments of this application includes: a decompression module 101. The decompression module 101 is used to decompress the encoded data B of the current frame. x Decode the data and obtain the mean value μ of the super-prior residuals for each pixel in the current frame. Y The variance of the super-prior residuals of each pixel in the previous frame The residual information Y of the current frame, and the hyper-prior motion mean μ of each pixel in the previous frame. M The hyperprior motion variance of each pixel in the previous frame And the motion information M of the current frame.

[0057] Wherein, μ is the mean of the prior residuals of each pixel in the current frame. Y and prior residual variance It is obtained by using a super-prior coding network to encode the potential residual features of each pixel in the current frame to obtain residual super-prior features, and then using a super-prior decoding network to decode the residual super-prior features. The mean of the super-prior residual is μ. Y This represents the most likely value of the quantization residual feature of the predicted pixel, and the variance of the prior residual. This represents the quantization residual feature of a pixel in the mean of the prior residual μ. Y The degree of dispersion or scattering in the vicinity. The residual information Y of the current frame includes the variance of the prior residuals. The quantized residual features of each pixel that exceeds the residual skip threshold. The latent residual features of a pixel refer to the residuals of the pixel obtained in the latent feature domain.

[0058] The hyperprior motion mean μ of each pixel in the current frame M and prior motion variance It is obtained by using a super-prior coding network to encode the motion vectors of each pixel in the current frame to obtain motion super-prior features, and then using a super-prior decoding network to decode the motion super-prior features. The mean of the super-prior residuals is μ. M This represents the most likely value of the quantized motion vector for each predicted pixel, and the variance of the prior residual. This represents the quantized motion vector of each pixel in the hyperprior motion mean μ. M The degree of dispersion or scattering in the vicinity. The motion information M of the current frame includes the quantized motion vectors of each pixel whose prior motion variance is greater than the motion skip threshold.

[0059] Reference Figure 1 As shown, in some embodiments, a video decoder that can be used to perform the video decoding method provided in the embodiments of this application includes: a residual threshold determination module 102. The residual threshold determination module 102 is used to obtain the residual skip threshold t of the current frame based on a first quality adjustment parameter and / or a second quality level adjustment parameter. Y The first quality adjustment parameter is used to adjust the coding quality level of the video sequence to which the current frame belongs, and the second quality adjustment parameter is used to adjust the coding quality of frames at different positions in the image group corresponding to the current frame.

[0060] Reference Figure 1 As shown, in some embodiments, a video decoder that can be used to perform the video decoding method provided in the embodiments of this application includes: a residual reconstruction module 103. The residual reconstruction module 103 is used to, for each pixel in the current frame, if the variance of the super-prior residual of the pixel... The residual skip threshold t is less than or equal to that of the current frame. YThen the mean of the prior residuals of the pixel points μ Y The reconstructed residual features of the pixel were determined. If the variance of the prior residual of the pixel is Residual greater than the current frame skip threshold t Y Then, the reconstructed residual features of the pixel are obtained based on the residual information Y.

[0061] Reference Figure 1 As shown, in some embodiments, a video decoder that can be used to perform the video decoding method provided in the embodiments of this application includes: a motion threshold determination module 104. The motion threshold determination module 104 is used to obtain the motion skip threshold t of the current frame according to a first quality adjustment parameter and / or a second quality level adjustment parameter. M The first quality adjustment parameter is used to adjust the coding quality level of the video sequence to which the current frame belongs, and the second quality adjustment parameter is used to adjust the coding quality of frames at different positions in the image group corresponding to the current frame.

[0062] It should be noted that the motion threshold determination module 104 obtains the motion skipping threshold t. M The implementation can be done by having the residual threshold determination module 102 obtain the residual skip threshold t. Y The implementation method is the same. Correspondingly, the motion skipping threshold t obtained by the motion threshold determination module 104 is... M The residual skip threshold t can be obtained from the residual threshold determination module 102. Y The motion threshold determination module 104 can obtain the motion skip threshold in a different way than the residual threshold determination module 102 can obtain the residual skip threshold. Correspondingly, the motion skip threshold t obtained by the motion threshold determination module 104... M It can also be used in conjunction with the residual skip threshold t obtained by the residual threshold determination module 102. Y They are not equal.

[0063] Reference Figure 1 As shown, in some embodiments, the video decoder that can be used to perform the video decoding method provided in the embodiments of this application includes: a motion reconstruction module 105. The motion reconstruction module 105 is used to, for each pixel of the current frame, if the hyperprior motion variance of the pixel... Motion skip threshold t less than or equal to the current frame M Then the hyper-prior motion mean μ of the pixel is... M The reconstructed motion vector of the pixel is determined. If the variance of the prior residual of the pixel is Motion skipping threshold t greater than the current frame MThen, the reconstructed motion vector of the pixel is obtained based on the motion information M.

[0064] Reference Figure 1 As shown, in some embodiments, the video decoder that can be used to perform the video decoding method provided in the embodiments of this application includes: a frame reconstruction module 106. The frame reconstruction module 106 is used to reconstruct the residual features of each pixel of the current frame based on the residual reconstruction module 103 output. and the reconstructed motion vectors of each pixel in the current frame output by the motion reconstruction module 105. Reconstruct the current frame to obtain the reconstructed frame of the current frame.

[0065] Figure 2 This is a schematic diagram of the structure of a video encoder provided for some embodiments of this application, which can be used to perform the video encoding method provided in the embodiments of this application.

[0066] Reference Figure 2 As shown, in some embodiments, a video encoder that can be used to perform the video encoding method provided in the embodiments of this application includes: an information extraction module 201. The information extraction module 201 is used to extract information from the current frame x in the latent feature domain. t Perform residual feature extraction to obtain the current frame x t The latent residual features Y of each pixel t and for the current frame x t Perform motion estimation (ME) to obtain the current video frame x. t The motion vector M of each pixel t Motion estimation is a technique that finds the most similar pixels to a given pixel in a reference frame and calculates the offset between them as a motion vector to describe the motion of objects in a video.

[0067] Reference Figure 2 As shown, in some embodiments, a video encoder that can be used to perform the video encoding method provided in the embodiments of this application includes: a residual quantization module 202. The residual quantization module 202 is used to perform residual quantization on the current frame x. t The latent residual features Y of each pixel t Quantization is performed to obtain the x-axis of the current video frame. t Quantization residual features y of each pixel tQuantization is a process that uses a specific algorithm to transform continuous sampled values ​​in a video signal into a finite number of discrete values. Its main purpose is to compress the amount of data to the greatest extent possible while ensuring the basic visual quality of the video. By reasonably setting the quantization parameter (QP), the originally wide range of values ​​can be mapped to a few specific values, thereby representing video data with fewer bits and achieving the goal of efficient encoding, storage and transmission.

[0068] Reference Figure 2 As shown, in some embodiments, a video encoder that can be used to perform the video coding method provided in the embodiments of this application includes: a residual prior prediction network 203. The residual prior prediction network 203 is used to predict the latent residual features Y of each pixel in the current frame. t The mean and variance of the Gaussian distribution are used to obtain the mean μ of the super-prior residuals for each pixel in the current frame. Y and prior residual variance

[0069] Reference Figure 2 As shown, in some embodiments, a video encoder that can be used to perform the video encoding method provided in the embodiments of this application includes: a residual threshold determination module 204. The residual threshold determination module 204 is used to obtain the residual skip threshold t of the current frame based on a first quality adjustment parameter and / or a second quality level adjustment parameter. Y The first quality adjustment parameter is used to adjust the coding quality level of the video sequence to which the current frame belongs, and the second quality adjustment parameter is used to adjust the coding quality of frames at different positions in the image group corresponding to the current frame.

[0070] Reference Figure 2 As shown, in some embodiments, a video encoder that can be used to perform the video encoding method provided in the embodiments of this application includes a residual skipping module 205. The residual skipping module 205 is used to calculate the variance of the super-prior residuals of each pixel in the current frame. Mean μ of the prior residuals of each pixel in the current frame Y The residual skip threshold t of the current frame Y and the quantization residual features y of each pixel in the current frame t Obtain the residual information Y of the current frame. The residual information Y of the current frame includes: the variance of the prior residual. Greater than the residual skip threshold t Y Quantization residual features y of each pixel t .

[0071] It should be noted that, unlike the residual information obtained from a video decoder which only includes the prior residual variance, this information is different. Greater than the residual skip threshold t Y The residual information Y obtained by the video encoder includes the residual information of each pixel in the current frame, and the variance of the residuals beyond the prior knowledge is also considered. Greater than the residual skip threshold t Y The residual information of each pixel is the quantized residual feature y of the pixel. t The variance of the prior residuals Less than or equal to the residual skip threshold t Y The residual information of each pixel is the mean of the prior residuals of the pixel, μ. Y The residual information obtained from the video decoder only includes the prior residual variance. Greater than the residual skip threshold t Y The residual information of each pixel, excluding the variance of the prior residual. Less than or equal to the residual skip threshold t Y The residual information of the pixels is because when performing arithmetic encoding on the residual information, it will not be the mean μ of the prior residual in the residual information. Y Allocate a specific bitrate.

[0072] Reference Figure 2 As shown, in some embodiments, a video encoder that can be used to perform the video encoding method provided in the embodiments of this application includes: a motion quantization module 206. The motion quantization module 206 is used to quantize the current frame x t The motion vector M of each pixel t Quantization is performed to obtain the x-axis of the current video frame. t The quantized motion vector m of each pixel t .

[0073] Reference Figure 2 As shown, in some embodiments, the video encoder that can be used to perform the video encoding method provided in the embodiments of this application includes: a motion prior prediction network 207. The motion prior prediction network 207 is used to predict the motion vector M of each pixel in the current frame. t The mean and variance of a Gaussian distribution are used to obtain the hyperprior motion mean μ for each pixel in the current frame. M and the variance of the preceding motion

[0074] Reference Figure 2 As shown, in some embodiments, a video encoder that can be used to perform the video encoding method provided in the embodiments of this application includes: a motion threshold determination module 208. The motion threshold determination module 208 is used to obtain the motion skip threshold t of the current frame based on a first quality adjustment parameter and / or a second quality level adjustment parameter. MThe first quality adjustment parameter is used to adjust the coding quality level of the video sequence to which the current frame belongs, and the second quality adjustment parameter is used to adjust the coding quality of frames at different positions in the image group corresponding to the current frame.

[0075] Reference Figure 2 As shown, in some embodiments, a video encoder that can be used to perform the video encoding method provided in the embodiments of this application includes a motion skipping module 209. The motion skipping module 209 is used to calculate the hyperprior motion variance of each pixel in the current frame. The hyper-prior motion mean μ of each pixel in the current frame M The motion skip threshold t for the current frame M and the quantized motion vector m of each pixel in the current frame t Obtain the motion information M of the current frame. The motion information M of the current frame includes: the prior motion variance. Greater than the motion skip threshold t M The quantized motion vector m of each pixel t .

[0076] Similarly, the motion information M obtained by the video encoder includes the motion information of every pixel in the current frame, and the prior motion variance is also included. Greater than the motion skip threshold t M The motion information of a pixel is the quantized motion vector m of that pixel. t The variance of the prior motion Less than or equal to the motion skip threshold t M The motion information of a pixel is the hyperprior motion mean μ of the pixel. M .

[0077] Reference Figure 2 As shown, in some embodiments, a video encoder that can be used to perform the video encoding method provided in the embodiments of this application includes an encoding module 210. The encoding module 210 is used to generate encoded data B for the current frame based on the residual information Y of the current frame and the motion information M of the current frame. x .

[0078] In some embodiments, the encoding module 210 is specifically used to encode the residual information Y of the current frame to generate residual encoded data of the current frame, encode the motion information M of the current frame to generate motion encoded data of the current frame, and combine the residual encoded data, motion encoded data, residual prior features, and motion prior features of the current frame to obtain the encoded data of the current frame. Specifically, the residual prior features and motion prior features of the current frame are used to obtain the encoded data of the current frame, respectively, by performing prior encoding on the residual and motion information of the current frame. Performing prior decoding on the residual prior features of the current frame yields the mean prior residual value μ of each pixel in the current frame. Y and prior residual variance By performing hyperprior decoding on the hyperprior motion features of the current frame, the hyperprior motion mean μ of each pixel in the current frame can be obtained. M and prior motion variance

[0079] In some embodiments, standard deviation σ can also be used instead of variance σ. 2 When using standard deviation σ instead of variance σ 2 At that time, the encoded data of the video frame no longer includes the super-prior residual variance of each pixel. and prior motion variance Instead, it includes the standard deviation σ of the prior residuals of each pixel. Y and the prior motion standard deviation σ M Furthermore, when using standard deviation σ instead of variance σ... 2 At that time, the residuals obtained at the decoding and encoding ends skip the threshold t. Y and the motion jump threshold t M The residual skip threshold t should be changed accordingly based on the above embodiments. Y arithmetic square root and the arithmetic square root of the motion jump threshold

[0080] Figure 3 This is a schematic diagram of the structure of a decoding system provided for some embodiments of this application, which can be used to execute the video encoding and video decoding methods provided in the embodiments of this application.

[0081] Reference Figure 3 As shown, in some embodiments, the video decoding system may include a feature extraction module 301. The feature extraction module 301 is used to extract features from the current frame x. t Perform latent feature extraction to obtain the latent features F of each pixel in the current frame. tThe feature extraction module 301 can be a feature extraction network model composed of layers such as convolutional layers, pooling layers, activation layers, normalization layers, fully connected layers, and attention mechanism layers. The latent features F of a pixel... t It refers to high-dimensional features that cannot be directly observed from the content of video frames, and can include features such as texture features, edge features, and color features.

[0082] Reference Figure 3 As shown, in some embodiments, the video decoding system may include a residual encoder 302. The residual encoder 302 is used to determine the optical flow information C of the current frame. t and the latent features F of each pixel in the current frame t Obtain the latent residual features Y of each pixel in the current frame. t Among them, the optical flow information C of the current frame t This includes optical flow information at multiple scales.

[0083] Reference Figure 3 As shown, in some embodiments, the video decoding system may include a residual quantization module 303. The residual quantization module 303 is used to process the latent residual features Y of each pixel in the current frame. t Quantization is performed to obtain the quantization residual features y of each pixel in the current frame. t .

[0084] Reference Figure 3 As shown, in some embodiments, the video decoding system may include a residual prior coding network 304. The residual prior coding network 304 is used to determine the optical flow information C of the current frame. t The residual information Y′ of the current frame and the reference frame are used to determine the potential residual features Y of each pixel in the current frame. t Perform super-prior encoding to obtain the residual super-prior features z of each pixel in the current frame. Y The reference frame for the current frame can be the previous video frame in the video sequence.

[0085] Reference Figure 3 As shown, in some embodiments, the video decoding system may include a residual super-prior decoding network 305. The residual super-prior decoding network 305 is used to process the residual super-prior features z of each pixel in the current frame. YDecode the data to obtain the mean μ of the super-prior residuals for each pixel in the current frame. Y and prior residual variance

[0086] Reference Figure 3 As shown, in some embodiments, the video decoding system may include a residual skipping module 306. The residual skipping module 306 is used to determine the residual skipping threshold t of the current frame. Y And in the prior residual variance of the pixel The residual skip threshold t is less than or equal to that of the current frame. Y At that time, the mean of the prior residuals μ of the pixel points Y The residual information is determined as the pixel-level residual variance. Residual greater than the current frame skip threshold t Y At that time, the quantization residual feature y of the pixel is... t The residual information of the pixel is determined to obtain the residual information Y of the current frame.

[0087] Reference Figure 3 As shown, in some embodiments, the video decoding system may include a motion estimation module 307. The motion estimation module 307 is used to estimate motion based on a reference frame. Motion estimation is performed on the current frame to obtain the motion estimation parameters v of each pixel in the current frame. t .

[0088] Reference Figure 3 As shown, in some embodiments, the video decoding system may include a motion encoder 308. The motion encoder 308 is used to estimate the motion parameters v of each pixel in the current video. t Reconstructed motion vectors of each pixel in the reference frame The latent features F of each pixel in the current frame t and the reconstructed latent features of each pixel in the reference frame. Obtain the motion vector M of each pixel in the current frame. t .

[0089] Reference Figure 3 As shown, in some embodiments, the video decoding system may include a motion quantization module 309. The motion quantization module 309 is used to quantize the motion vector M of each pixel in the current frame. t Quantization is performed to obtain the quantized motion vector m of each pixel in the current frame. t .

[0090] Reference Figure 3As shown, in some embodiments, the video decoding system may include a motion prior coding network 310. The motion prior coding network 310 is used to determine the encoded motion vector mv of each pixel in a reference frame. t-1 The motion vectors M of each pixel in the current frame and the motion information M′ of the reference frame. t Perform hyperprior encoding to obtain the motion hyperprior features z of each pixel in the current frame. M .

[0091] Reference Figure 3 As shown, in some embodiments, the video decoding system may include a motion hyperpriority decoding network 311. The motion hyperpriority decoding network 311 is used to process the motion hyperpriority features z of each pixel in the current frame. M Decode the data to obtain the hyperprior motion mean μ for each pixel in the current frame. M and prior motion variance

[0092] Reference Figure 3 As shown, in some embodiments, the video decoding system may include a motion skipping module 312. The motion skipping module 312 is used to determine the motion skipping threshold t of the current frame. M If the prior motion variance of a pixel is less than or equal to the motion skip threshold t of the current frame, M In this case, the hyperprior motion mean μ of the pixel is... M The motion information of the pixel is determined when the prior motion variance of the pixel is greater than the motion skip threshold t of the current frame. M In this case, the quantized motion vector m of the pixel is... t The motion information of the pixel is determined to obtain the motion information M of the current frame.

[0093] Reference Figure 3 As shown, in some embodiments, the video decoding system may include a compression module 313. The compression module 313 is used to encode the residual information Y of the current frame to obtain residual encoded data B. Y Arithmetic coding is performed on the motion information M of the current frame to obtain the motion-coded data B of the current frame. M And based on residual coding data B Y Motion coding data B M Residual prior features z Y and motion prior features z M Generate the encoded data for the current frame.

[0094] Reference Figure 3As shown, in some embodiments, the video decoding system may include a motion decoder 314. The motion decoder 314 is used to determine the motion mean μ of each pixel in the current frame. M and prior motion variance and the motion-coded data B of the current frame M Obtain the reconstructed motion vectors of each pixel in the current frame.

[0095] Reference Figure 3 As shown, in some embodiments, the video decoding system may include a context model 315. The context model 315 is used to reconstruct motion vectors based on the individual pixels of the current frame. Reconstructed latent features of each pixel in the reference frame Obtain optical flow information of the current frame c t .

[0096] Reference Figure 3 As shown, in some embodiments, the video decoding system may include a residual decoder 316. The residual decoder 316 is used to determine the mean of the super-prior residuals μ of each pixel in the current frame. Y and prior residual variance And the residual coded data B of the current frame Y Obtain the reconstruction residual features of each pixel in the current frame.

[0097] Reference Figure 3 As shown, in some embodiments, the video decoding system may include a motion compensation module 317. The motion compensation module 317 is used to calculate the reconstructed residual features of each pixel in the current frame. And the optical flow information C of the current frame t Obtain the reconstructed latent features of each pixel in the current frame.

[0098] Reference Figure 3 As shown, in some embodiments, the video decoding system may include a reconstruction module 318. The reconstruction module 318 is used to reconstruct latent features based on each pixel of the current frame. Reconstruct the current frame to obtain the current frame x. t Reconstructed frames

[0099] Reference Figure 4As shown, in some embodiments, the residual skipping module or motion skipping module in the above embodiments may include: a first coefficient determination unit 41. The first coefficient determination unit 41 is used to obtain a first coefficient f(λ) based on a first quality adjustment parameter λ and a first mapping function. The first mapping function in the residual skipping module and the motion skipping module may be different or the same. Accordingly, the first coefficient f(λ) obtained by the residual skipping module and the motion skipping module based on the first quality adjustment parameter and the first mapping function may be the same or different.

[0100] In some embodiments, the residual skipping module or motion skipping module in the above embodiments may include a second coefficient determination unit 42. The second coefficient determination unit 42 is used to obtain a second coefficient g(Δλ) based on a second quality adjustment parameter Δλ and a second mapping function. Similarly, the second mapping function in the residual skipping module and the motion skipping module may be different or the same, and the second coefficient g(Δλ) obtained by the residual skipping module and the motion skipping module based on the second quality adjustment parameter and the second mapping function may be the same or different.

[0101] In some embodiments, the residual skipping module or motion skipping module in the above embodiments may include: a skipping threshold determination unit 43. The skipping threshold determination unit 43 is used to obtain a skipping threshold (residual skipping threshold or motion skipping threshold) based on a first coefficient f(λ) and a second coefficient g(Δλ).

[0102] Reference Figure 4 As shown, in some embodiments, the residual skipping module or motion skipping module in the above embodiments may include a discrimination unit 44. The discrimination unit in the residual skipping module is used to determine the mean of the prior residuals of the pixel as the residual information of the pixel when the prior residual variance of the pixel is less than or equal to the residual feature skipping threshold of the current frame; and to determine the quantized residual feature of the pixel as the residual information of the pixel when the prior residual variance of the pixel is greater than the residual feature skipping threshold of the current frame. Similarly, the discrimination unit in the motion skipping module is used to determine the mean of the prior motion of the pixel as the motion information of the pixel when the prior motion variance of the pixel is less than or equal to the motion vector skipping threshold of the current frame; and to determine the quantized motion vector of the pixel as the motion information of the pixel when the prior motion variance of the pixel is greater than the motion vector skipping threshold of the current frame.

[0103] Some embodiments of this application provide a video decoding method, see below. Figure 5 As shown, the video decoding method includes the following steps:

[0104] S51. Decode the encoded data of the current frame, obtain the mean and variance of the prior residuals of each pixel in the current frame, and the residual information of the current frame.

[0105] In some embodiments, the encoded data of the current frame can be obtained by communicating with a media resource server.

[0106] In some embodiments, the encoded data of the current frame can be read from a specified storage space.

[0107] In some embodiments, the encoded data includes residual prior features obtained by performing prior encoding on the residual information of the current frame and residual encoded data obtained by performing arithmetic encoding on the residual information of the current frame. Decoding the encoded data of the current frame to obtain the mean and variance of the prior residuals for each pixel of the current frame, as well as the residual information of the current frame, includes: performing prior decoding on the residual prior features to obtain the mean and variance of the prior residuals for each pixel of the current frame, and performing arithmetic decoding on the residual encoded data to obtain the residual information of the current frame.

[0108] In this embodiment, the decoding end obtains the mean and variance of the super-prior residuals of each pixel in the current frame by decoding the encoded data of the current frame, while the encoding end obtains the mean and variance of the super-prior residuals of each pixel in the current frame by performing super-prior encoding on the potential residual features of each pixel in the current video frame, and then decoding the obtained residual super-prior features to obtain the mean and variance.

[0109] In some embodiments, the residual information of the current frame includes the quantized residual features of each pixel whose prior residual variance is greater than the residual skip threshold. That is, the residual information of the current frame only includes the quantized residual features of pixels whose prior residual variance is greater than the residual skip threshold, and does not include the residual information of pixels whose prior residual variance is less than or equal to the residual skip threshold.

[0110] S52. Obtain the residual skip threshold of the current frame based on the first quality adjustment parameter.

[0111] The first quality adjustment parameter is a parameter used to adjust the coding quality level of the video sequence to which the current frame belongs.

[0112] In some embodiments, the first mass adjustment parameter is a Lagrange multiplier.

[0113] Video coding needs to maintain good video quality while minimizing bitrate. However, bitrate and video quality are mutually restrictive factors. To balance bitrate and video quality, many optimization strategies have been introduced in the field of video coding, among which the Lagrange function plays a key role. The Lagrange function is a function constructed by combining bitrate and video quality distortion, providing support for finding a balance between video quality and bitrate in complex coding environments. The general form of the Lagrange function is shown in the following formula (1):

[0114] J=λD+R (1)

[0115] Where J is the value of the Lagrange function, λ is the Lagrange multiplier, D represents the distortion between the original video frame and the reconstructed video frame, and R represents the bit rate of the video frame.

[0116] As shown in equation (1) above: the selection of the Lagrange multiplier will affect the overall coding quality of the video sequence. Therefore, the first quality adjustment parameter in this embodiment can be the Lagrange multiplier.

[0117] Since the selection of the first quality adjustment parameter affects the overall coding quality of the video sequence, and the residual skip threshold also affects the coding quality of the video frame, obtaining the residual skip threshold of the current frame based on the first quality adjustment parameter can make the residual skip threshold of the current frame compatible with the overall coding quality of the video sequence, thereby improving the coding efficiency of the video.

[0118] S53. Obtain the reconstruction residual features of each pixel in the current frame.

[0119] Specifically, for each pixel in the current frame, if the variance of the prior residual of the pixel is less than or equal to the residual skip threshold of the current frame, then the mean of the prior residual of the pixel is determined as the reconstruction residual feature of the pixel; if the variance of the prior residual of the pixel is greater than the residual skip threshold of the current frame, then the reconstruction residual feature of the pixel is obtained based on the residual information.

[0120] That is, for any pixel, if the variance of the prior residual of the pixel is less than or equal to the residual skip threshold of the current frame, then it is not necessary to obtain the reconstruction residual feature of the pixel based on the residual information. Instead, the mean of the prior residual of the pixel is determined as the reconstruction residual feature of the pixel. If the variance of the prior residual of the pixel is greater than the residual skip threshold of the current frame, then the reconstruction residual feature of the pixel is obtained based on the residual information.

[0121] In some embodiments, obtaining the reconstructed residual features of the pixel based on the residual information includes: obtaining the quantized residual features of the pixel based on the residual information, and performing inverse quantization on the quantized residual features to obtain the reconstructed residual features of the pixel.

[0122] The quantization residual feature of a pixel is represented as y. t The reconstructed residual features of pixels are represented as The mean of the prior residuals of a pixel is represented as μ. Y The variance of the super-prior residuals of a pixel is expressed as The residual skip threshold is represented as sk i Then the reconstructed residual features of the pixels can be determined by the following formula (2):

[0123]

[0124] Here, IQ() represents the dequantization operation.

[0125] In some embodiments, obtaining the reconstructed residual features of each pixel in the current frame may include the following steps a and b:

[0126] Step a: Generate the mask feature tensor based on the residual skip threshold and the variance of the super-prior residuals of each pixel in the current frame.

[0127] The mask feature tensor includes the mask feature values ​​of each pixel in the current frame. For each pixel in the current frame, if the variance of the prior residual of the pixel is less than or equal to the residual skip threshold of the current frame, the mask feature value of the pixel is 1. If the variance of the prior residual of the pixel is greater than the residual skip threshold of the current frame, the mask feature value of the pixel is 0.

[0128] That is, the variance of the prior residuals of the pixels is expressed as The residual skip threshold is represented as sk i If the pixel mask feature value is represented as mask, then the pixel mask feature value can be determined by the following formula (3):

[0129]

[0130] Step b: Obtain the reconstructed residual features of each pixel in the current frame based on the mask feature values ​​of each pixel in the mask feature tensor, the residual information of the current frame, the mean of the super-prior residuals of each pixel in the current frame, and the following formula (4):

[0131]

[0132] Among them, yt These are the quantized residual features of pixels read from the residual information of the current frame. μ represents the residual features of pixel reconstruction. Y is the mean of the prior residuals of the pixel, and mask is the mask feature value of the pixel.

[0133] The above formula can also be used to determine the mean of the prior residuals of the pixel as the reconstruction residual feature of the pixel when the variance of the prior residuals of the pixel is less than or equal to the residual skip threshold of the current frame, and to obtain the reconstruction residual feature of the pixel based on the residual information (quantized residual feature) when the variance of the prior residuals of the pixel is greater than the residual skip threshold of the current frame.

[0134] S54. Reconstruct the current frame based on the reconstruction residual features of each pixel in the current frame to obtain the reconstructed frame of the current frame.

[0135] In some embodiments, reconstructing the current frame based on the reconstruction residual features of each pixel in the current frame may include: reconstructing the current frame based on the reconstruction residual features of each pixel in the current frame and the reconstructed motion vectors of each pixel in the current frame. The method for obtaining the reconstructed motion vectors of each pixel in the current frame can be as follows: the video encoding end obtains the motion vectors of each pixel in the current frame and encodes them using a conventional encoding scheme to obtain motion-coded data of the current frame; the video decoding end then decodes the motion-coded data of the current frame using a corresponding decoding scheme to obtain the reconstructed motion vectors of each pixel in the current frame. That is, in this embodiment, instead of using a skip mode to encode and reconstruct the motion vectors of each pixel in the current frame, conventional techniques can be used to encode and reconstruct the motion vectors of each pixel in the current frame.

[0136] The video decoding method provided in this application first decodes the encoded data of the current frame to obtain the mean of the prior residuals of each pixel in the current frame, the variance of the prior residuals of each pixel in the current frame, and the residual information of the current frame. Then, it determines the residual skip threshold of the current frame according to a first quality adjustment parameter used to adjust the encoded quality level of the video sequence to which the current frame belongs. For each pixel in the current frame, if the variance of the prior residuals of the pixel is less than or equal to the residual skip threshold of the current frame, the mean of the prior residuals of the pixel is determined as the reconstruction residual feature of the pixel. If the variance of the prior residuals of the pixel is greater than the residual skip threshold of the current frame, the reconstruction residual feature of the pixel is obtained according to the residual information, thereby obtaining the reconstruction residual features of each pixel in the current frame. Finally, the current frame is reconstructed according to the reconstruction residual features of each pixel in the current frame to obtain the reconstructed frame of the current frame. Since the video decoding method provided in this application embodiment can determine the residual skip threshold of the current frame based on the first quality adjustment parameter used to adjust the coding quality level of the video sequence to which the current frame belongs, this application embodiment can perform video coding based on the residual skip threshold adapted to the quality level, and specifically optimize the bitrate allocation under different coding quality levels. Therefore, this application embodiment can improve the coding efficiency of video.

[0137] This application also provides another video decoding method, see below. Figure 6 As shown, the video decoding method includes the following steps:

[0138] S601. Decode the encoded data of the current frame, obtain the mean of the prior residuals of each pixel in the current frame, the variance of the prior residuals of each pixel in the current frame, and the residual information of the current frame.

[0139] S602, Obtain the first quality adjustment parameter.

[0140] The first quality adjustment parameter is a parameter used to adjust the coding quality level of the video sequence to which the current frame belongs.

[0141] S603. Obtain the second quality adjustment parameter based on the position of the current frame in its group of pictures (GOP).

[0142] The second quality adjustment parameter is a parameter used to adjust the encoding quality of frames at different positions in the image group corresponding to the current frame.

[0143] When using a low-delay (LD) compression framework for video compression, only the first frame is transmitted using a high-quality image compression method, while the remaining frames are transmitted using motion estimation and motion compensation to achieve high-speed video transmission. While the LD framework can achieve high-speed video transmission, it also leads to the continuous accumulation of errors. Currently, the mainstream solution is to introduce Group of Pictures (GOPs) into the video to ensure the quality stability of later frames during long encoding periods. The core idea of ​​GOPs is to set the earlier frames within a group as high-quality frames and the later frames as low-quality frames, thereby preventing the cumulative propagation of errors. For example, the input video sequence can be grouped according to a fixed step size, with the first frame in each group set to a higher video quality level, while the remaining frames in the group are set to a lower quality level, thus achieving a balance between bitrate and quality. The encoding quality parameter corresponding to the current frame, used to adjust the frames at different positions in the GOP, is the second quality adjustment parameter in this embodiment. As mentioned above, the second quality adjustment parameter also affects the encoding quality of the current frame; therefore, obtaining the second quality adjustment parameter and setting the residual skip threshold based on it can further improve video encoding efficiency.

[0144] In some embodiments, obtaining the second quality adjustment parameter based on the position of the current frame in its respective image group includes: if the current frame is the first video frame in its respective image group, then determining the second quality adjustment parameter to be a first value; if the current frame is not the first video frame in its respective image group, then determining the second quality adjustment parameter to be a second value.

[0145] S604. Obtain the residual skip threshold of the current frame based on the first quality adjustment parameter and the second quality adjustment parameter.

[0146] In the coding compression model, two quality adjustment parameters are generally introduced. One quality adjustment parameter is λ, which controls the video sequence to be encoded at different coding quality levels (different bitrate points). The other quality adjustment parameter is Δλ, which controls the video frames at different positions within the GOP to be encoded at different coding quality levels after the overall coding quality level of the video sequence has been set. The influence of the quality adjustment parameter λ and the quality adjustment parameter Δλ on the video coding quality is shown in the following formula (5):

[0147]

[0148] Where J is the value of the Lagrange function. The distortion is calculated based on the original video frames and the reconstructed video frames. The bitrate is calculated based on the likelihood of latent features and prior parameters. The bitrate is calculated based on the likelihood of the motion vector and the hyperprior parameters. This indicates the bitrate of the video frame.

[0149] The above embodiments obtain the residual skip threshold of the current frame based on two quality adjustment parameters (the first quality adjustment parameter and the second quality adjustment parameter) that affect the video coding quality. Therefore, the above embodiments can comprehensively consider the relationship between video quality variation and residual skip threshold, thereby further improving the video coding efficiency.

[0150] In some embodiments, obtaining the residual skip threshold of the current frame based on the first quality adjustment parameter and the second quality adjustment parameter may include the following steps 1 to 4:

[0151] Step 1: Obtain the first coefficient based on the first quality adjustment parameter and the first mapping function.

[0152] In some embodiments, the first mapping function may be as shown in equation (6):

[0153]

[0154] Where f(λ) is the first coefficient, λ1, λ2, λ3, ... are different first mass adjustment parameters, and A, B, C, ... are the first coefficients corresponding to different first mass adjustment parameters.

[0155] Step 2: Obtain the second coefficient based on the second quality adjustment parameter and the second mapping function.

[0156] In some embodiments, obtaining the second quality adjustment parameter based on the position of the current frame in its respective image group includes: if the current frame is the first video frame in its respective image group, then determining the second quality adjustment parameter as a first value; if the current frame is not the first video frame in its respective image group, then determining the second quality adjustment parameter as a second value. Obtaining the second coefficient based on the second quality adjustment parameter and the second mapping function includes: if the current frame is the first video frame in its respective image group, then mapping the first value to a third value using the second mapping function to obtain the second coefficient; if the current frame is not the first video frame in its respective image group, then mapping the second value to a fourth value using the second mapping function to obtain the second coefficient. Wherein, the third value is less than the fourth value.

[0157] That is, the second mapping function can be represented by the following equation (7):

[0158]

[0159] Where g(Δλ) is the second coefficient, Δλ1 is the first value, Δλ2 is the second value, a is the third value, and b is the fourth value.

[0160] In other embodiments, obtaining the second quality adjustment parameter based on the position of the current frame in its respective image group includes: if the current frame is the first video frame in its respective image group, then determining the second quality adjustment parameter as a first value; if the current frame is not the first video frame in its respective image group, then determining the second quality adjustment parameter as a second value. Obtaining the second coefficient based on the second quality adjustment parameter and the second mapping function includes: if the current frame is the first video frame in its respective image group, then mapping the first value to a third value using the second mapping function to obtain the second coefficient; if the current frame is not the first video frame in its respective image group, then obtaining the second coefficient based on the second mapping function, the second value, and the sequence number of the current frame in its respective image group. Wherein, when the current frame is not the first video frame in its respective image group, the sequence number of the current frame in its respective image group is positively correlated with the second coefficient, and all obtained second coefficients are greater than the third value.

[0161] That is, the second mapping function can be represented by the following equation (8):

[0162]

[0163] Where Q2 is the second coefficient, Δλ1 is the first value, Δλ2 is the second value, a is the third value, n is the sequence number of the current frame in its respective image group, and a < g(Δλ2,n), when i is greater than j, g(Δλ2,i) > g(Δλ2,j).

[0164] Step c: Sum the first coefficient and the second coefficient to obtain the residual threshold multiplier.

[0165] Let the first coefficient be denoted as Q1, the second coefficient as Q2, and the residual threshold multiplier as k. Then, the calculation of the residual threshold multiplier can be shown in the following equation (9):

[0166] k=f(λ)+g(Δλ) (9)

[0167] Step d: Calculate the residual skip threshold for the current frame based on the residual threshold multiplier.

[0168] In some embodiments, calculating the residual skip threshold of the current frame based on the residual threshold multiplier includes: calculating the product of the residual threshold multiplier and the value of the residual skip threshold function to obtain the residual skip threshold of the current frame. The residual skip threshold function is a function with the frame index of the current frame as its independent variable, and the residual skip threshold function is monotonically increasing within the domain of the frame index of the video sequence to which the current frame belongs.

[0169] In some embodiments, the residual skip threshold function is expressed as shown in equation (10):

[0170]

[0171] Where i represents the timing of the current frame, n represents the total number of video frames in the video sequence to be encoded, and e is a constant.

[0172] Let sk represent the residual skip threshold of video frame i with time sequence i. i Then the formula for calculating the residual skip threshold can be shown in equation (11):

[0173]

[0174] S605. Obtain the reconstruction residual features of each pixel in the current frame.

[0175] Specifically, for each pixel in the current frame, if the variance of the prior residual of the pixel is less than or equal to the residual skip threshold of the current frame, then the mean of the prior residual of the pixel is determined as the reconstruction residual feature of the pixel; if the variance of the prior residual of the pixel is greater than the residual skip threshold of the current frame, then the reconstruction residual feature of the pixel is obtained based on the residual information.

[0176] S606. Reconstruct the current frame based on the reconstruction residual features of each pixel in the current frame to obtain the reconstructed frame of the current frame.

[0177] Similarly, reconstructing the current frame based on the reconstruction residual features of each pixel can include: reconstructing the current frame based on the reconstruction residual features of each pixel and the reconstruction motion vectors of each pixel. The method for obtaining the reconstruction motion vectors of each pixel in the current frame can also be the same as in the above embodiments, and will not be repeated here to avoid redundancy.

[0178] This application also provides another video decoding method, see below. Figure 7 As shown, the video decoding method includes the following steps:

[0179] S701. Decode the encoded data of the current frame and obtain the mean of the prior residuals of each pixel in the current frame, the variance of the prior residuals of each pixel in the current frame, the residual information of the current frame, the mean of the prior motion of each pixel in the current frame, the variance of the prior motion of each pixel in the current frame, and the motion information of the current frame.

[0180] S702. Based on the mean of the prior residuals of each pixel in the current frame, the variance of the prior residuals of each pixel in the current frame, and the residual information of the current frame, obtain the reconstruction residual features of each pixel in the current frame.

[0181] The method for obtaining the reconstruction residual features of each pixel in the current frame based on the mean and variance of the prior residuals of each pixel in the current frame and the residual information of the current frame can refer to steps S602 to S605 above. To avoid redundancy, it will not be repeated here.

[0182] S703. Obtain the motion skip threshold of the current frame based on the first quality adjustment parameter.

[0183] In some embodiments, obtaining the motion skip threshold of the current frame based on the first quality adjustment parameter includes: obtaining a second quality adjustment parameter based on the position of the current frame in its respective image group, and obtaining the motion skip threshold of the current frame based on the first quality adjustment parameter and the second quality adjustment parameter. The second quality adjustment parameter is a parameter corresponding to the current frame used to adjust the encoding quality of frames at different positions in the image group.

[0184] In some embodiments, obtaining the motion skip threshold of the current frame based on the first quality adjustment parameter and the second quality adjustment parameter may include the following steps ① to ④:

[0185] Step ①: Obtain the third coefficient based on the first quality adjustment parameter and the third mapping function.

[0186] In this embodiment, the third mapping function may be the same as or different from the first mapping function. When the third mapping function is the same as the first mapping function, the first coefficient and the third coefficient are equal; when the third mapping function is different from the first mapping function, the first coefficient and the third coefficient are not equal.

[0187] Step 2: Obtain the fourth coefficient based on the second quality adjustment parameter and the fourth mapping function.

[0188] In some embodiments, obtaining the second quality adjustment parameter based on the position of the current frame in its respective image group includes: if the current frame is the first video frame in its respective image group, then determining the second quality adjustment parameter as a first value; if the current frame is not the first video frame in its respective image group, then determining the second quality adjustment parameter as a second value; obtaining the fourth coefficient based on the second quality adjustment parameter and the fourth mapping function includes: if the current frame is the first video frame in its respective image group, then mapping the third value to a fifth value through the fourth mapping function to obtain the fourth coefficient; if the current frame is not the first video frame in its respective image group, then mapping the fifth value to a sixth value through the fourth mapping function to obtain the second coefficient. Wherein, the fifth value is less than the sixth value.

[0189] In some embodiments, the fourth mapping function is the same as the second mapping function. Accordingly, the fifth value is equal to the third value, and the sixth value is equal to the fourth value.

[0190] In some embodiments, obtaining the second quality adjustment parameter based on the position of the current frame in its respective image group includes: if the current frame is the first video frame in its respective image group, then determining the second quality adjustment parameter as a first value; if the current frame is not the first video frame in its respective image group, then determining the second quality adjustment parameter as a second value. Obtaining the fourth coefficient based on the second quality adjustment parameter and the fourth mapping function includes: if the current frame is the first video frame in its respective image group, then mapping the first value to a fifth value using the fourth mapping function to obtain the fourth coefficient; if the current frame is not the first video frame in its respective image group, then obtaining the fourth coefficient based on the fourth mapping function, the second value, and the sequence number of the current frame in its respective image group. Wherein, when the current frame is not the first video frame in its respective image group, the sequence number of the current frame in its respective image group is positively correlated with the second coefficient, and all obtained fourth coefficients are greater than the fifth value.

[0191] Step ③: Sum the third coefficient and the fourth coefficient to obtain the motion threshold multiplier.

[0192] Step 4: Calculate the motion skip threshold for the current frame based on the motion threshold multiplier.

[0193] In some embodiments, calculating the motion skip threshold of the current frame based on the motion threshold multiplier includes: calculating the product of the motion threshold multiplier and the value of the motion skip threshold function to obtain the motion skip threshold of the current frame;

[0194] The motion skip threshold function is a function with the frame index of the current frame as the independent variable, and the motion skip threshold function is monotonically increasing within the domain of the frame index of the video sequence to which the current frame belongs.

[0195] In some embodiments, the motion skipping threshold function is expressed as shown in equation (12):

[0196]

[0197] Where i represents the timing of the current frame, n represents the total number of video frames in the video sequence to be encoded, and e is a constant.

[0198] S704. Obtain the reconstructed motion vectors of each pixel in the current frame.

[0199] Specifically, for each pixel in the current frame, if the prior motion variance of the pixel is less than or equal to the motion skip threshold of the current frame, then the prior motion mean of the pixel is determined as the reconstructed motion vector of the pixel; if the prior residual variance of the pixel is greater than the motion skip threshold of the current frame, then the reconstructed motion vector of the pixel is obtained based on the motion information.

[0200] S705. Reconstruct the current frame based on the reconstruction residual features and reconstruction motion vectors of each pixel in the current frame.

[0201] In some embodiments, reconstructing the current frame based on the reconstruction residual features and reconstruction motion vectors of each pixel of the current frame includes the following steps: obtaining the predicted features of each pixel of the current frame based on the reconstruction motion vectors of each pixel of the current frame; obtaining the reconstruction potential features of each pixel of the current frame based on the reconstruction residual features and the predicted features of each pixel of the current frame; and reconstructing the current frame based on the reconstruction potential features of each pixel of the current frame.

[0202] Some embodiments of this application provide a video encoding method, see below. Figure 8 As shown, the video encoding method may include the following steps:

[0203] S81. Obtain the potential residual features of each pixel in the current frame.

[0204] In video coding technology, pixel residuals are a key indicator for measuring the accuracy of coding predictions. They are defined as the difference between the actual value of a pixel and the predicted value obtained using a specific prediction method. Taking frame-based video coding as an example, the value of pixels within the current frame is predicted using a reference frame; the difference between the actual value and the predicted value is the residual for that pixel. Latent residual features further deepen the concept of residuals. It first transforms the actual value of a pixel into a feature value in the latent domain. This feature value is an abstract expression containing the intrinsic characteristics and correlation information of the pixel. The difference between this feature value and the predicted feature is calculated; the information contained in this difference is the latent residual feature of the pixel.

[0205] For video compression tasks, the pixel domain has too low a dimension to represent sufficient local and global feature information. Directly performing motion estimation, motion compensation, and other operations on video frames in the pixel domain before compression results in poor coding performance. Therefore, in this application embodiment, the video frames are first converted into features of the latent domain, and then compression is performed based on the features of the latent domain to improve the video coding performance.

[0206] For example, the latent residual features of video frames can be extracted using a neural compressor based on a variational autoencoder (VAE) architecture, which can be used to extract the latent residual features of each pixel in the video frame.

[0207] S82. Quantize the potential residual features of each pixel in the current frame to obtain the quantized residual features of each pixel in the current frame.

[0208] Quantizing the latent residual features of each pixel in the current frame refers to the process of mapping continuous latent residual features to a finite number of discrete feature values. Quantizing the latent residual features of each pixel in the current frame can reduce the amount of data encoded for the latent residual features, but it will reduce the representation accuracy of the latent residual features.

[0209] S83. Obtain the mean and variance of the prior residuals of each pixel in the current frame based on the potential residual features of each pixel in the current frame.

[0210] In some embodiments, a super-prior coding method can be employed, using a pre-trained residual super-prior coding network to predict the super-prior information of the latent residual features of each pixel in the current frame. The super-prior information of the latent residual features, after quantization, can approximate the residual information actually transmitted to the decoding end. Therefore, directly decoding the super-prior information of the latent residual features of each pixel in the current frame yields the mean and variance of the latent residual feature distribution of each pixel in the current frame. The mean and variance of the latent residual feature distribution of each pixel in the current frame are the mean and variance of the super-prior residuals for each pixel. The mean of the super-prior residuals for each pixel represents the most likely value of the latent residual feature of each pixel, and the variance of the super-prior residuals for each pixel represents the degree of dispersion or dispersion of the latent residual feature of each pixel around the mean.

[0211] S84. Obtain the residual skip threshold of the current frame based on the first quality adjustment parameter.

[0212] The first quality adjustment parameter is a parameter used to adjust the coding quality level of the video sequence to which the current frame belongs.

[0213] The implementation method for obtaining the residual skip threshold of the current frame based on the first quality adjustment parameter is the same as the implementation method for obtaining the residual skip threshold of the current frame based on the first quality adjustment parameter in the video decoding method described above. To avoid redundancy, it will not be repeated here.

[0214] S85. Obtain the residual information of the current frame.

[0215] The residual information includes quantized residual features of each pixel whose prior residual variance is greater than the residual skip threshold.

[0216] To balance video quality and bitrate during transmission, and to achieve Rate-Distortion Optimization (RDO), this embodiment replaces the ground truth value of the quantization residual features for some pixels with the mean of the super-prior residuals. For each pixel, whether to trigger the replacement of the quantization residual features with the mean of the super-prior residuals is determined by comparing the variance of the super-prior residuals with the residual skip threshold of the current frame. If the variance of the super-prior residuals is less than or equal to the residual skip threshold, it is considered that the ground truth value of the quantization residual features will appear near the mean of the super-prior residuals. The mean of the super-prior residuals can approximate the ground truth value of the quantization residual features and can be used to replace them. However, if the variance of the super-prior residuals is greater than the residual skip threshold, it is considered that the ground truth value of the quantization residual features may appear far from the mean of the super-prior residuals, and the mean of the super-prior residuals cannot reasonably replace the ground truth value of the quantization residual features, therefore it is not used to replace them.

[0217] S86. Obtain the encoded data of the current frame based on the residual information of the current frame.

[0218] In some embodiments, obtaining the encoded data of the current frame based on the residual information of the current frame includes:

[0219] Encode the residual information of the current frame to obtain the residual encoded data of the current frame;

[0220] The residual encoded data of the current frame is obtained based on the residual encoded data and the residual super-prior features of the current frame. Specifically, the residual super-prior features and motion super-prior features of the current frame are obtained by super-prior encoding the residual and motion information of the current frame, respectively. Super-prior decoding of the residual super-prior features of the current frame yields the mean and variance of the super-prior residuals for each pixel in the current frame.

[0221] In some embodiments, encoding the residual information of the current frame to obtain the residual encoded data of the current frame may include the following steps:

[0222] Step A: Generate the mask feature tensor based on the residual skip threshold and the variance of the super-prior residuals of each pixel in the current frame.

[0223] The mask feature tensor includes the mask feature values ​​of each pixel in the current frame. For each pixel in the current frame, if the variance of the prior residual of the pixel is less than or equal to the residual skip threshold of the current frame, the mask feature value of the pixel is 1. If the variance of the prior residual of the pixel is greater than the residual skip threshold of the current frame, the mask feature value of the pixel is 0.

[0224] That is, the variance of the prior residuals of the pixels is expressed as The residual skip threshold is represented as sk i If the pixel mask feature value is represented as mask, then the pixel mask feature value can be determined by the following formula (13):

[0225]

[0226] Step B: Obtain the residual information and the likelihood of the residual information of each pixel in the current frame based on the mask feature value of each pixel in the current frame, the quantization residual feature of each pixel in the current frame, the mean of the super-prior residual of each pixel in the current frame, and the following formulas (14) and (15):

[0227] y t =y t *(1-mask)+mask*μ Y (14)

[0228] log y t =log y t *(1-mask)+mask (15)

[0229] Among them, y t μ represents the quantization residual feature of a pixel. Y Let be the mean of the prior residuals of the pixels, and let mask be the mask feature value of the pixels, log y t The likelihood of residual information is used to represent the number of bits of data occupied by the residual information of pixels in the encoded data.

[0230] Based on equations (14) and (15) above, for pixels whose prior residual variance is less than or equal to the residual skip threshold of the current frame, since mask = 1, therefore y t =μ Y , log y t =1, therefore the encoded data does not include residual information for pixels whose prior residual variance is less than or equal to the residual skip threshold of the current frame. For pixels whose prior residual variance is greater than the residual skip threshold of the current frame, since mask=0, y t =y t , log y t =log yt Therefore, the encoded data includes data obtained by encoding the quantized residual features of pixels whose super-prior residual variance is greater than the residual skip threshold of the current frame.

[0231] Step C: Perform arithmetic encoding on the residual information of each pixel in the current frame based on the likelihood of the residual information of each pixel in the current frame to obtain the residual encoded data of the current frame.

[0232] Since the residual coding data only includes the data obtained by arithmetic coding of the quantized residual features of each pixel whose prior residual variance is greater than the residual skip threshold, obtaining the coding data of the current frame based on the reconstruction residual features of each pixel of the current frame can reduce the amount of coding data of the video frame while ensuring the quality of the reconstructed video frame.

[0233] In some embodiments, obtaining the encoded data of the current frame based on the residual information of the current frame may include: obtaining the encoded data of the current frame based on the residual information and the motion information of the current frame. The motion information of the current frame includes the motion vectors of each pixel in the current frame. That is, in this embodiment, instead of using a skip mode to encode the motion vectors of each pixel in the current frame, conventional techniques may be used to encode the motion vectors of each pixel in the current frame.

[0234] The video coding method provided in this application first obtains the latent residual features of each pixel in the current frame, and quantizes the latent residual features of each pixel in the current frame to obtain the quantized residual features of each pixel in the current frame. Then, based on the latent residual features of each pixel in the current frame, it obtains the mean and variance of the prior residuals of each pixel in the current frame. Next, based on a first quality adjustment parameter used to adjust the coding quality level of the video sequence to which the current frame belongs, it obtains the residual skip threshold of the current frame. Based on the residual skip threshold, it obtains the residual information of the current frame, and based on the residual information of the current frame, it obtains the encoded data of the current frame. Since the video coding method provided in this application can determine the residual skip threshold of the current frame based on the first quality adjustment parameter used to adjust the coding quality level of the video sequence to which the current frame belongs, this application can determine whether to skip encoding the quantized residual features of pixels based on a residual skip threshold adapted to the quality level. Therefore, this application can improve the coding efficiency of video.

[0235] This application provides another video encoding method, see embodiments thereof. Figure 9 As shown, the video encoding method includes the following steps:

[0236] S901. Extract latent features from the current frame to obtain the latent features of each pixel in the current frame.

[0237] In some embodiments, latent feature extraction is performed on the current frame to obtain the latent features of each pixel in the current frame, including: inputting the current frame into a pre-trained latent feature extraction network and receiving the latent features of the current frame output by the latent feature extraction network. The latent features of the current frame include the latent features of each pixel in the current frame.

[0238] S902. Obtain the potential residual features of each pixel in the current frame based on the potential features of each pixel in the current frame and the optical flow information of the current frame.

[0239] In some embodiments, the latent residual features of each pixel in the current frame are obtained based on the latent features of each pixel in the current frame and the optical flow information of the current frame. This includes: inputting the latent features of each pixel in the current frame and the optical flow information of the current frame into a pre-trained residual encoder, and obtaining the latent residual features of each pixel in the current frame output by the residual encoder.

[0240] S903. Quantize the potential residual features of each pixel in the current frame to obtain the quantized residual features of each pixel in the current frame.

[0241] S904. Input the potential residual features of each pixel in the current frame, the optical flow information of the current frame, and the reconstructed residual features of each pixel in the reference frame into the residual super-prior coding network, and obtain the residual super-prior features output by the residual super-prior coding network.

[0242] S905. Decode the residual super-prior features based on the residual super-prior decoding network to obtain the mean super-prior residual and the variance of the super-prior residual for each pixel in the current frame.

[0243] S906, Obtain the first quality adjustment parameter.

[0244] The first quality adjustment parameter is a parameter used to adjust the coding quality level of the video sequence to which the current frame belongs.

[0245] S907. Obtain the second quality adjustment parameter based on the position of the current frame in its respective image group.

[0246] The second quality adjustment parameter is a parameter used to adjust the encoding quality of frames at different positions in the image group corresponding to the current frame.

[0247] S908. Obtain the residual skip threshold of the current frame based on the first quality adjustment parameter and the second quality adjustment parameter.

[0248] The method for obtaining the residual skip threshold of the current frame based on the first quality adjustment parameter and the second quality adjustment parameter is the same as the method for obtaining the residual skip threshold of the current frame based on the first quality adjustment parameter and the second quality adjustment parameter in the above video decoding method. To avoid redundancy, it will not be repeated here.

[0249] S909. Obtain the residual information of the current frame.

[0250] The residual information includes quantized residual features of each pixel whose prior residual variance is greater than the residual skip threshold.

[0251] S910. Obtain the encoded data of the current frame based on the residual information and residual prior features of the current frame.

[0252] Similarly, obtaining the encoded data of the current frame based on the residual information of the current frame can include: obtaining the encoded data of the current frame based on the residual information and the motion information of the current frame. The implementation method for obtaining the motion information of the current frame can also be the same as in the above embodiments, and will not be repeated here to avoid redundancy.

[0253] This application provides another video encoding method, see embodiments thereof. Figure 10 As shown, the video encoding method includes the following steps:

[0254] S101. Obtain the residual information and residual prior features of the current frame.

[0255] The method for obtaining the residual information of each pixel in the current frame can refer to any of the above embodiments. To avoid redundancy, it will not be repeated here.

[0256] S102. Obtain the motion vectors of each pixel in the current frame.

[0257] In some embodiments, obtaining the motion vectors of each pixel in the current frame may include the following steps 1 and 2:

[0258] Step 1: Perform motion estimation on the current frame based on the reference frame to obtain the motion estimation parameters of the current video.

[0259] In video sequences, adjacent video frames often exhibit strong correlations, but objects in the frame frequently shift positions. Motion estimation, through specific algorithms and strategies, identifies pixel-level motion relationships between the current frame and reference frames within a series of video frames. This allows the use of existing information from the reference frame to predict the content of the current frame, effectively reducing redundancy in the video data.

[0260] Step 2: Obtain the motion vector of each pixel in the current frame based on the motion estimation parameters of each pixel in the current frame, the reconstructed motion vector of the reference frame, the latent features of each pixel in the current frame, and the reconstructed latent features of the reference frame.

[0261] When the reference frame is the reconstructed frame of the previous frame of the current frame, the reconstructed motion vector of the reference frame encodes the coded motion vector of the previous frame of the current frame to obtain the coded data of the coded motion vector of the previous frame of the current frame. Then, the coded data of the coded motion vector of the previous frame of the current frame is decoded to obtain the motion vector. The reconstructed latent feature of the reference frame refers to the coded data of the reconstructed residual feature of the previous frame of the current frame obtained by encoding the reconstructed residual feature of the previous frame of the current frame, the residual feature obtained by decoding the coded data of the reconstructed residual feature of the previous frame of the current frame, and the latent feature obtained by compensating the residual feature with the optical flow information of the previous frame of the current frame.

[0262] In some embodiments, obtaining the motion vector of each pixel in the current frame based on the motion estimation parameters of each pixel in the current frame, the reconstructed motion vector of the reference frame, the latent features of each pixel in the current frame, and the reconstructed latent features of the reference frame includes: inputting the motion estimation parameters of each pixel in the current frame, the reconstructed motion vector of the reference frame, the latent features of each pixel in the current frame, and the reconstructed latent features of the reference frame into a motion encoder, and obtaining the motion vector of each pixel in the current frame output by the motion encoder.

[0263] S103. Quantize the motion vectors of each pixel in the current frame to obtain the quantized motion vectors of each pixel in the current frame.

[0264] S104. Perform hyperprior encoding on the motion vectors of each pixel in the current frame to obtain the hyperprior motion features of the current frame.

[0265] In some embodiments, obtaining the motion prior features of the current frame based on the motion vectors of each pixel in the current frame includes: inputting the motion vectors of each pixel in the current frame, the encoded motion vectors of each pixel in the reference frame, and the reconstructed residual features of each pixel in the reference frame into a motion prior coding network, and obtaining the motion prior features output by the motion prior coding network.

[0266] S105. Perform hyperprior decoding on the hyperprior features of the current frame to obtain the hyperprior motion mean and hyperprior motion variance of each pixel in the current frame.

[0267] S106. Obtain the motion skip threshold of the current frame based on the first quality adjustment parameter.

[0268] The first quality adjustment parameter is a parameter used to adjust the coding quality level of the video sequence to which the current frame belongs.

[0269] In some embodiments, obtaining the motion skip threshold of the current frame based on the first quality adjustment parameter includes: obtaining a second quality adjustment parameter based on the position of the current frame in its respective image group, and obtaining the motion skip threshold of the current frame based on the first quality adjustment parameter and the second quality adjustment parameter. The second quality adjustment parameter is a parameter corresponding to the current frame used to adjust the encoding quality of frames at different positions in the image group.

[0270] The implementation method for obtaining the motion skip threshold of the current frame based on the first quality adjustment parameter and the second quality adjustment parameter can be the same as the implementation method for obtaining the motion skip threshold of the current frame based on the first quality adjustment parameter and the second quality adjustment parameter in the above video decoding method. To avoid redundancy, it will not be repeated here.

[0271] S107. Obtain motion information for the current frame.

[0272] The motion information is encoded data obtained by encoding the quantized motion vectors of each pixel whose prior motion variance is greater than the motion skip threshold.

[0273] S108. Obtain the encoded data of the current frame based on the residual information, residual prior features, motion information, and motion prior features of the current frame.

[0274] Reference Figure 11 As shown, some embodiments of this application provide a video decoding device 1100, which may include:

[0275] Decoding unit 111 is used to decode the encoded data of the current frame, obtain the mean and variance of the prior residuals of each pixel in the current frame, and the residual information of the current frame;

[0276] The residual threshold acquisition unit 112 is used to acquire the residual skip threshold of the current frame according to the first quality adjustment parameter, wherein the first quality adjustment parameter is a parameter used to adjust the coding quality level of the video sequence to which the current frame belongs;

[0277] The residual reconstruction unit 113 is used to obtain the reconstruction residual features of each pixel in the current frame; for each pixel in the current frame, if the variance of the prior residual of the pixel is less than or equal to the residual skip threshold of the current frame, the mean of the prior residual of the pixel is determined as the reconstruction residual feature of the pixel; if the variance of the prior residual of the pixel is greater than the residual skip threshold of the current frame, the reconstruction residual feature of the pixel is obtained according to the residual information.

[0278] The frame reconstruction unit 114 is used to reconstruct the current frame based on the reconstruction residual features of each pixel of the current frame, so as to obtain the reconstructed frame of the current frame.

[0279] As an optional implementation method of this application,

[0280] The decoding unit 111 is also used to decode the encoded data, obtain the prior motion mean and prior motion variance of each pixel in the current frame, and the motion information of the current frame;

[0281] Reference Figure 12 As shown, the video decoding device further includes:

[0282] The motion threshold acquisition unit 115 is used to acquire the motion skip threshold of the current frame based on the first quality adjustment parameter;

[0283] The motion vector reconstruction unit 116 is used to obtain the reconstructed motion vectors of each pixel in the current frame. For each pixel in the current frame, if the prior motion variance of the pixel is less than or equal to the motion skip threshold of the current frame, the prior motion mean of the pixel is determined as the reconstructed motion vector of the pixel. If the prior residual variance of the pixel is greater than the motion skip threshold of the current frame, the reconstructed motion vector of the pixel is obtained according to the motion information.

[0284] The frame reconstruction unit 114 is specifically used to reconstruct the current frame based on the reconstruction residual features and reconstruction motion vectors of each pixel in the current frame.

[0285] Some embodiments of this application provide a video encoding apparatus, see reference Figure 13 As shown, the video encoding device 1300 may include:

[0286] The residual feature acquisition unit 131 is used to acquire the potential residual features of each pixel in the current frame;

[0287] The residual quantization unit 132 is used to quantize the potential residual features of each pixel in the current frame to obtain the quantized residual features of each pixel in the current frame.

[0288] The residual prior information acquisition unit 133 is used to acquire the mean and variance of the prior residuals of each pixel in the current frame based on the potential residual features of each pixel in the current frame.

[0289] The residual threshold acquisition unit 134 is used to acquire the residual skip threshold of the current frame according to the first quality adjustment parameter, wherein the first quality adjustment parameter is a parameter used to adjust the coding quality level of the video sequence to which the current frame belongs;

[0290] The residual information acquisition unit 135 is used to acquire the residual information of the current frame, wherein the residual information includes the quantized residual features of each pixel whose super-prior residual variance is greater than the residual skip threshold;

[0291] The encoded data acquisition unit 136 is used to acquire the encoded data of the current frame based on the residual information of the current frame.

[0292] As an optional implementation method of this application, refer to Figure 14 As shown, the video encoding device 1300 further includes:

[0293] The motion vector acquisition unit 137 is used to acquire the motion vectors of each pixel in the current frame;

[0294] The motion vector quantization unit 138 is used to quantize the motion vectors of each pixel in the current frame to obtain the quantized motion vectors of each pixel in the current frame.

[0295] The motion prior information acquisition unit 139 is used to acquire the prior motion mean and prior motion variance of each pixel in the current frame based on the motion vector of each pixel in the current frame.

[0296] The motion threshold acquisition unit 1310 is used to determine the motion skip threshold of the current frame. When the timing of the current frame meets the preset conditions, the motion skip threshold of the current frame is the product of the third coefficient and the value of the motion skip threshold function. When the timing of the current frame does not meet the preset conditions, the motion skip threshold of the current frame is the product of the fourth coefficient and the value of the motion skip threshold function. The third coefficient is less than the fourth coefficient.

[0297] The motion information acquisition unit 1311 is used to acquire the motion information of the current frame. The motion information is encoded data obtained by encoding the quantized motion vectors of each pixel whose prior motion variance is greater than the motion skip threshold.

[0298] The frame coding unit 136 specifically obtains the encoded data of the current frame based on the residual information and motion information of the current frame.

[0299] The video encoding apparatus provided in the above embodiments can execute the video encoding method provided in any of the above embodiments, and the implementation principle and the achieved technical effect are the same. To avoid redundancy, it will not be described in detail again.

[0300] Some embodiments of this application provide a video decoder, including: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to implement the video decoding method provided in any of the above embodiments when executing the computer program.

[0301] Some embodiments of this application provide a video encoder, which may include: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to implement the video encoding method provided in any of the above embodiments when executing the computer program.

[0302] Figure 15 Exemplary configuration block diagrams of video decoders or video encoders from some of the embodiments described above are shown. For example... Figure 15 As shown, the video decoder or video encoder 1500 may include at least one of the following: a tuner / demodulator 151, a communicator 152, a detector 153, an external device interface 154, a processor 155, a display 156, an audio output interface 157, a memory 158, a power supply 159, and a user interface.

[0303] In some embodiments, processor 155 includes at least one of: a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (random access memory), ROM (read-only memory), a first to an nth interface for input / output, a communication bus, etc.

[0304] The display 156 includes a display screen assembly for presenting images, a driving assembly for driving image display, a component for receiving image signals from the processor output, and a user interface for displaying video content, image content, menu control interface, and user control UI.

[0305] The display 156 can be a liquid crystal display, an organic light-emitting diode (OLED) display, or a projection display, etc.

[0306] The communicator 152 is a component used to communicate with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of the following: a Wi-Fi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or near-field communication protocol chips, and an infrared receiver. The video decoder or video encoder 1500 can use the communicator 152 to send and receive control signals and data signals with the control device or server.

[0307] The user interface can be used to receive control signals input by the user through a control device (such as an infrared remote control) or by touch or gesture.

[0308] Detector 153 can be used to acquire signals from the external environment or to interact with the external environment. For example, detector 153 may include a light receiver, a sensor for acquiring ambient light intensity; or, detector 153 may include an image acquisition device, such as a camera, for acquiring external environmental scenes, user attributes, or user interaction gestures; or, detector 153 may include a sound acquisition device, such as a microphone, for receiving external sounds.

[0309] External device interface 154 may include, but is not limited to, one or more of the following: High-Definition Multimedia Interface (HDMI), analog or high-definition component input interface (component), Composite Video Broadcast Signal (CVBS), Universal Serial Bus (USB), RGB (Red, Green, Blue) port, etc. It may also be a composite input / output interface formed by multiple of the above interfaces.

[0310] The tuner / demodulator 151 receives broadcast television signals via wired or wireless means, and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals. In some embodiments, the processor 155 and the tuner / demodulator 151 may be located in different separate devices, that is, the tuner / demodulator 151 may also be located in an external device of the main device where the processor 155 is located, such as an external set-top box.

[0311] Processor 155 controls the operation of electronic devices and responds to user operations through various software control programs stored in memory. Processor 155 controls the overall operation of video decoder or video encoder 1500. For example, in response to receiving a user command to select a UI object to be displayed on display 156, processor 155 can execute operations related to the object selected by the user command.

[0312] In some embodiments, processor 155 includes at least one of: a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (random access memory), ROM (read-only memory), a first to an nth interface for input / output, a communication bus, etc.

[0313] Some embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a computing device, causes the computing device to implement the video encoding method or the video reconstruction method described in any of the above embodiments.

[0314] In a seventh aspect, some embodiments of this application provide a chip including a processor and a memory, the memory being used to store programs or instructions executable on the processor, and the processor being used to execute the programs or instructions to cause the video encoding method or the video reconstruction method described in any of the above embodiments to be executed.

[0315] Eighthly, some embodiments of this application provide a computer program product that, when run on a computer, enables the computer to implement the video encoding method or the video reconstruction method described in any of the above embodiments.

[0316] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0317] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

Claims

1. A video decoding method, characterized in that, The method includes: Decode the encoded data of the current frame, obtain the mean and variance of the prior residuals of each pixel in the current frame, as well as the residual information of the current frame; The residual skip threshold of the current frame is obtained according to the first quality adjustment parameter, which is a parameter used to adjust the coding quality level of the video sequence to which the current frame belongs; Obtain the reconstruction residual features of each pixel in the current frame; for each pixel in the current frame, if the variance of the prior residual of the pixel is less than or equal to the residual skip threshold of the current frame, then the mean of the prior residual of the pixel is determined as the reconstruction residual feature of the pixel; if the variance of the prior residual of the pixel is greater than the residual skip threshold of the current frame, then the reconstruction residual feature of the pixel is obtained according to the residual information. The current frame is reconstructed based on the reconstruction residual features of each pixel to obtain the reconstructed frame.

2. The method according to claim 1, characterized in that, The step of obtaining the residual skip threshold for the current frame based on the first quality adjustment parameter includes: The second quality adjustment parameter is obtained based on the position of the current frame in its respective image group. The second quality adjustment parameter is a parameter used to adjust the encoding quality of frames at different positions in the image group corresponding to the current frame. The residual skip threshold for the current frame is obtained based on the first quality adjustment parameter and the second quality adjustment parameter.

3. The method according to claim 2, characterized in that, The step of obtaining the residual skip threshold of the current frame based on the first quality adjustment parameter and the second quality adjustment parameter includes: The first coefficient is obtained based on the first quality adjustment parameter and the first mapping function; The second coefficient is obtained based on the second quality adjustment parameter and the second mapping function; The first coefficient and the second coefficient are summed to obtain the residual threshold multiplier; The residual skip threshold for the current frame is calculated based on the residual threshold multiplier.

4. The method according to claim 3, characterized in that, The step of obtaining the second quality adjustment parameter based on the position of the current frame in the image group includes: if the current frame is the first video frame in the image group, then the second quality adjustment parameter is determined to be a first value; if the current frame is not the first video frame in the image group, then the second quality adjustment parameter is determined to be a second value. The step of obtaining the second coefficient according to the second quality adjustment parameter and the second mapping function includes: if the current frame is the first video frame in the image group, then the first value is mapped to a third value through the second mapping function to obtain the second coefficient; if the current frame is not the first video frame in the image group, then the second value is mapped to a fourth value through the second mapping function to obtain the second coefficient. The third value is less than the fourth value.

5. The method according to claim 3, characterized in that, The step of obtaining the second quality adjustment parameter based on the position of the current frame in the image group includes: if the current frame is the first video frame in the image group, then the second quality adjustment parameter is determined to be a first value; if the current frame is not the first video frame in the image group, then the second quality adjustment parameter is determined to be a second value. The step of obtaining the second coefficient based on the second quality adjustment parameter and the second mapping function includes: if the current frame is the first video frame in the image group, then the first value is mapped to a third value through the second mapping function to obtain the second coefficient; if the current frame is not the first video frame in the image group, then the second coefficient is obtained based on the second mapping function, the second value, and the sequence number of the current frame in the image group. Wherein, when the current frame is not the first video frame in its respective image group, the sequence number of the current frame in its respective image group is positively correlated with the second coefficient, and the obtained second coefficient is greater than the third value.

6. The method according to claim 3, characterized in that, The step of calculating the residual skip threshold for the current frame based on the residual threshold multiplier includes: Calculate the product of the residual threshold multiplier and the residual skip threshold function to obtain the residual skip threshold for the current frame; The residual skip threshold function is a function with the frame index of the current frame as the independent variable, and the residual skip threshold function is monotonically increasing within the domain of the frame index of the video sequence to which the current frame belongs.

7. The method according to claim 6, characterized in that, The residual skip threshold function is: Where i is the frame index of the current frame, n is the total number of video frames in the video sequence to which the current frame belongs, and e is a constant.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: Decode the encoded data to obtain the mean and variance of the prior motion of each pixel in the current frame, as well as the motion information of the current frame; The motion skip threshold of the current frame is obtained based on the first quality adjustment parameter; Obtain the reconstructed motion vector of each pixel in the current frame; for each pixel in the current frame, if the prior motion variance of the pixel is less than or equal to the motion skip threshold of the current frame, then the prior motion mean of the pixel is determined as the reconstructed motion vector of the pixel; if the prior residual variance of the pixel is greater than the motion skip threshold of the current frame, then the reconstructed motion vector of the pixel is obtained according to the motion information. The step of reconstructing the current frame based on the reconstruction residual features of each pixel in the current frame includes: The current frame is reconstructed based on the reconstruction residual features and reconstruction motion vectors of each pixel.

9. The method according to claim 8, characterized in that, The step of obtaining the motion skip threshold of the current frame based on the first quality adjustment parameter includes: The second quality adjustment parameter is obtained based on the position of the current frame in its respective image group. The second quality adjustment parameter is a parameter used to adjust the encoding quality of frames at different positions in the image group corresponding to the current frame. The motion skip threshold for the current frame is obtained based on the first quality adjustment parameter and the second quality adjustment parameter.

10. The method according to claim 8, characterized in that, The process of reconstructing the current frame based on the reconstruction residual features and reconstruction motion vectors of each pixel includes: Based on the reconstructed motion vectors of each pixel in the current frame, obtain the predicted features of each pixel in the current frame; Based on the reconstruction residual features of each pixel in the current frame and the prediction features of each pixel in the current frame, obtain the reconstruction potential features of each pixel in the current frame. The current frame is reconstructed based on the potential reconstruction features of each pixel.

11. A video encoding method, characterized in that, include: Obtain the potential residual features of each pixel in the current frame; The potential residual features of each pixel in the current frame are quantized to obtain the quantized residual features of each pixel in the current frame. Based on the latent residual features of each pixel in the current frame, obtain the mean and variance of the prior residuals of each pixel in the current frame. The residual skip threshold of the current frame is obtained according to the first quality adjustment parameter, which is a parameter used to adjust the coding quality level of the video sequence to which the current frame belongs; Obtain the residual information of the current frame, wherein the residual information includes the quantized residual features of each pixel whose super-prior residual variance is greater than the residual skip threshold; Obtain the encoded data of the current frame based on the residual information of the current frame.

12. The method according to claim 11, characterized in that, The step of obtaining the residual skip threshold of the current frame based on the first quality adjustment parameter of the current frame includes: The second quality adjustment parameter is obtained based on the position of the current frame in its respective image group. The second quality adjustment parameter is a parameter used to adjust the encoding quality of frames at different positions in the image group corresponding to the current frame. The residual skip threshold for the current frame is obtained based on the first quality adjustment parameter and the second quality adjustment parameter.

13. A video decoder, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to implement the video decoding method as described in any one of claims 1-10 when executing the computer program.

14. A video encoder, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to implement the video encoding method as described in claim 11 or 12 when executing the computer program.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a computing device, causes the computing device to perform the method according to any one of claims 1-12.

16. A chip, characterized in that, The chip includes a processor and a memory, the memory being used to store programs or instructions that can run on the processor, and the processor being used to execute the programs or instructions to cause the method as described in any one of claims 1-12 to be performed.

Images