Encoding method, decoding method, apparatus, and related device

Abstract

Provided are an encoding method, a decoding method, an apparatus, a device, and a storage medium. A method comprises: determining a temporal frame interpolation mode; and on the basis of the temporal frame interpolation mode, performing, on sampled frames in an input frame queue, frame interpolation by temporal interpolation and / or frame interpolation by temporal extrapolation.

Description

Encoding, decoding methods, devices, and related equipment

[0001] Cross-reference to related applications

[0002] This disclosure claims priority to Chinese patent applications filed on July 10, 2024, with application number 202410925591.7 entitled "Encoding, Decoding Method, Apparatus, Device and Storage Medium"; and on April 25, 2025, with application number 202510536682.6 entitled "Encoding, Decoding Method, Apparatus, Device and Storage Medium"; and on April 25, 2025, with application number 202510536651.0 entitled "Encoding, Decoding Method, Apparatus, Device and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to the field of image processing, and more particularly to an encoding and decoding method, apparatus, device, medium, and computer program product. Background Technology

[0004] With the development of the times, the speed of information transmission has accelerated. Currently, video information accounts for a large proportion of the transmitted information.

[0005] Frame extraction coding is a commonly used technique in signal coding technology during information transmission. Current frame extraction coding techniques typically extract frames from video at a fixed extraction ratio, and then encode the extracted video. However, with technological advancements, single-sampling-rate encoding and decoding methods are no longer sufficient to meet user needs. How to achieve variable sampling-rate encoding and decoding is a problem urgently needing to be solved in this field. Currently, existing image decoding schemes cannot achieve mixed interpolation frames between the temporal and extra-temporal domains.

[0006] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] This disclosure provides an encoding and decoding method, apparatus, device, medium, and computer program product that at least partially realizes variable-rate encoding and decoding.

[0008] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0009] According to one aspect of this disclosure, a decoding method is provided, applied at a decoding end, comprising: determining whether the current sampling frame is a first sampling frame; and if the current sampling frame is not the first sampling frame, determining the temporal sampling information corresponding to the first sampling frame.

[0010] In one embodiment of this disclosure, the temporal sampling information corresponding to the first sampling frame includes a temporal sampling rate hold bit; the method further includes: updating the global temporal frame extraction ratio when the temporal sampling rate hold bit is active.

[0011] In one embodiment of this disclosure, the method further includes: saving the first sampling frame.

[0012] In one embodiment of this disclosure, saving the first sampled frame includes: configuring the first sampled frame in an output queue; and saving the output queue.

[0013] In one embodiment of this disclosure, the method further includes: determining whether the global temporal frame sampling ratio is 1; if the global temporal frame sampling ratio is 1, determining the frame buffer information corresponding to the current sampled frame as the frame buffer information corresponding to the first sampled frame.

[0014] In one embodiment of this disclosure, the method further includes: determining the number of interpolated frames between the current sampling frame and the first sampling frame, including: determining the number of interpolated frames between the current sampling frame and the first sampling frame based on the temporal frame extraction ratio of the first sampling frame, wherein the temporal frame extraction ratio of the first sampling frame can be determined by the global temporal frame extraction ratio.

[0015] In one embodiment of this disclosure, the method further includes: performing frame interpolation based on the number of interpolated frames, the current sampled frame, the first sampled frame, and a preset decoding method.

[0016] In one embodiment of this disclosure, the method further includes: saving the first sampled frame, the current sampled frame, and the frame obtained by the supplementary frame in sequence.

[0017] In one embodiment of this disclosure, saving the first sampled frame, the current sampled frame, and the frame obtained by the supplementary frame in sequence order includes: configuring the first sampled frame, the current sampled frame, and the frame obtained by the supplementary frame in sequence order in the output queue; and saving the output queue.

[0018] In one embodiment of this disclosure, the method further includes: configuring frame buffer information for the current sampling frame when the current sampling frame is the first sampling frame.

[0019] In one embodiment of this disclosure, determining whether the current sampling frame is the first sampling frame includes: acquiring the current sampling frame; and, if the current sampling frame is acquired, determining whether the current sampling frame is the first sampling frame.

[0020] In one embodiment of this disclosure, the method further includes: saving the remaining frames if the current sampling frame is not acquired.

[0021] In one embodiment of this disclosure, the method further includes: upon receiving end information and obtaining the current sampling frame, saving the current sampling frame and the remaining frames.

[0022] According to another aspect of this disclosure, a decoding method is provided, applied at a decoding end, comprising: acquiring temporal information of a sampled image corresponding to a sampled frame; and determining whether a temporal sampling flag is set based on the temporal information of the sampled image.

[0023] In one embodiment of this disclosure, the method further includes: determining that the time domain recovery function of the current sampling frame is disabled when the time domain sampling flag bit is not set.

[0024] In one embodiment of this disclosure, the method further includes: obtaining the sampling rate corresponding to the sampling frame while keeping the time-domain sampling rate at a fixed position.

[0025] According to another aspect of this disclosure, a decoding method is provided, applied at a decoding end, comprising: performing frame interpolation on the current frame based on target frames preceding the current frame.

[0026] In one embodiment of this disclosure, the current frame is supplemented based on the previous frames, which includes: copying the target frame to obtain the current frame, wherein the target frame includes a reference frame.

[0027] In one embodiment of this disclosure, copying a target frame to obtain a current frame, wherein the target frame includes a reference frame, includes: copying the target frame to obtain a current frame, wherein the target frame includes a reference frame, if the number of target frames preceding the current frame does not meet a preset number.

[0028] In one embodiment of this disclosure, the process of interpolating the current frame based on the previous frames includes: copying the target frame to obtain the current frame, wherein the target frame includes the interpolated frame.

[0029] In one embodiment of this disclosure, the current frame is interpolated based on the frames preceding the current frame, which includes: interpolating the current frame based on multiple target frames to obtain the current frame, wherein the target frames include reference frames.

[0030] In one embodiment of this disclosure, the current frame is interpolated based on the frames preceding the current frame, which includes: interpolating the current frame based on multiple target frames to obtain the current frame, wherein the target frames include the reference frame and the interpolated frame.

[0031] In one embodiment of this disclosure, the current frame includes a reference frame.

[0032] In one embodiment of this disclosure, the reference frame, the current frame, and the frame obtained by the interpolation frame are determined as the interpolation frame segment.

[0033] According to another aspect of this disclosure, an encoding method is provided, applied at an encoding end, comprising:

[0034] The video is encoded to obtain bitstream data, so that the decoding end can implement any of the decoding methods in the above embodiments based on the bitstream data.

[0035] According to another aspect of this disclosure, a decoding apparatus is provided, applied at a decoding end, comprising: a first determining module configured to determine whether the current sampling frame is a first sampling frame; and a second determining module configured to determine the temporal sampling information corresponding to the first sampling frame when the current sampling frame is not the first sampling frame.

[0036] According to another aspect of this disclosure, a decoding apparatus is provided, applied at a decoding end, comprising: a first acquisition module configured to acquire temporal information of a sampled image corresponding to a sampled frame; and a third determination module configured to determine whether a temporal sampling flag is set based on the temporal information of the sampled image.

[0037] According to another aspect of this disclosure, a decoding apparatus is provided, applied at a decoding end, comprising: a frame interpolation module configured to interpolate the current frame based on a target frame preceding the current frame.

[0038] According to another aspect of this disclosure, an encoding apparatus is provided for use at an encoding end, comprising: an encoding module configured to encode video to obtain bitstream data, so that a decoding end can implement any decoding method based on the bitstream data.

[0039] According to another aspect of this disclosure, a decoding method is also provided, comprising: determining a temporal recovery frame interpolation mode; and performing temporal interpolation and / or temporal extrapolation on sampled frames in an input frame queue according to the temporal recovery frame interpolation mode.

[0040] In one embodiment of this disclosure, before performing temporal interpolation and / or temporal extrapolation on the sampled frames in the input frame queue according to the temporal recovery interpolation mode, the method further includes: performing a temporal recovery initialization operation; when no bitstream end information is received, performing temporal interpolation or temporal extrapolation processing on each sampled frame in the frame queue according to the temporal recovery interpolation mode; when bitstream end information is received, determining the tail prediction input frame queue, and performing temporal tail recovery processing on the input frame queue according to the tail prediction input frame queue.

[0041] In one embodiment of this disclosure, performing a time-domain recovery initialization operation includes at least one of the following: initializing the output frame queue to an empty frame queue; initializing the global time-domain recovery parameters to sequence-level time-domain recovery parameters; and initializing the reference frame queue to an empty frame queue.

[0042] In one embodiment of this disclosure, for each sampled frame in the frame queue, temporal interpolation or temporal extrapolation is performed according to the temporal recovery interpolation mode, including: if the temporal recovery interpolation mode of the current frame is temporal interpolation and the current frame is not the first frame, then temporal interpolation is performed; if the temporal recovery interpolation mode of the current frame is temporal extrapolation, then the current frame is added to the reference frame queue, and when the length of the reference frame sequence reaches the temporal extrapolation prediction number, temporal extrapolation is performed.

[0043] In one embodiment of this disclosure, the method further includes: if the temporal recovery interpolation mode of the current frame is temporal interpolation and the current frame is the first frame, then only the step of adding the current frame to the output frame queue is performed, and temporal interpolation processing is not performed.

[0044] In one embodiment of this disclosure, performing temporal interpolation frame processing includes: taking the previous frame, the current frame, and temporal sampling parameters as inputs to the temporal interpolation frame processing process, outputting interpolated frames, and adding them to the output frame queue.

[0045] In one embodiment of this disclosure, the method further includes: after performing the temporal interpolation frame processing, adding the current frame to the output frame queue.

[0046] In one embodiment of this disclosure, the time-domain sampling parameters include: time-domain resampling rate.

[0047] In one embodiment of this disclosure, performing time-domain interpolation frame processing further includes: if the time-domain sampling rate change flag / time-domain recovery update flag of the current frame is 1, updating the global time-domain recovery parameters to the time-domain recovery parameters of the current frame; if the time-domain sampling rate change flag / time-domain recovery update flag of the current frame is 0, keeping the global time-domain recovery parameters unchanged and the same as the time-domain recovery parameters of the previous frame.

[0048] In one embodiment of this disclosure, after updating the global temporal recovery parameters to the temporal recovery parameters of the current frame, the method further includes: if the temporal recovery interpolation mode of the current frame is temporal extrapolation, then adding the current frame to the reference frame queue.

[0049] In one embodiment of this disclosure, performing temporal extrapolation frame interpolation processing includes: taking a reference frame sequence and the number of temporal extrapolation frame predictions as inputs to the temporal extrapolation frame interpolation process, outputting interpolated frames, and adding them to the output frame sequence.

[0050] In one embodiment of this disclosure, the method further includes: after performing the temporal extrapolation frame processing, adding the current frame to the output frame queue.

[0051] In one embodiment of this disclosure, the method further includes: if the temporal sampling rate change flag / temporal recovery update flag of the current frame is 1, updating the global temporal recovery parameters to the temporal recovery parameters of the current frame; if the temporal sampling rate change flag / temporal recovery update flag of the current frame is 0, the global temporal recovery parameters remain unchanged and are the same as the temporal recovery parameters of the previous frame.

[0052] In one embodiment of this disclosure, after updating the global temporal recovery parameters to the temporal recovery parameters of the current frame, the method further includes: if the temporal recovery frame interpolation mode of the current frame is temporal interpolation, then setting the reference frame queue to an empty frame queue; if the temporal recovery frame interpolation mode of the current frame is temporal extrapolation, then adding the current frame to the reference frame queue; when the length of the reference frame sequence reaches the temporal extrapolation prediction number, using the reference frame sequence and the temporal extrapolation prediction number as input to the temporal extrapolation processing procedure, outputting the interpolated frame, adding it to the output frame sequence, and setting the reference frame queue to an empty frame queue.

[0053] In one embodiment of this disclosure, a tail prediction input frame queue is determined, and temporal tail recovery processing is performed on the input frame queue according to the tail prediction input frame queue, including: if the temporal recovery frame interpolation mode is temporal interpolation, then the tail prediction input frame queue is set as a queue containing the last frame in the input frame queue; if the temporal recovery frame interpolation mode is temporal extrapolation, then the tail prediction input frame queue is set as a queue containing the last N frames in the output frame queue, where N is the number of temporal extrapolation prediction frames; the tail prediction input frame queue, the remaining number of temporal recovery frames, and the temporal recovery frame interpolation mode are used as inputs to the temporal tail recovery processing, and a tail frame queue is output; if there are frames after the last sampled frame in the output frame queue, then the frames after the last sampled frame are deleted, and the tail frame queue is added after the last sampled frame.

[0054] According to another aspect of this disclosure, an encoding method is also provided, comprising: encoding a video to obtain bitstream data, so that a decoding end implements the decoding method as described in any of the preceding claims based on the bitstream data.

[0055] According to another aspect of this disclosure, a decoding apparatus is also provided, comprising: a frame interpolation mode determination module configured to determine a temporal recovery frame interpolation mode; and a temporal recovery frame interpolation module configured to perform temporal interpolation and / or temporal extrapolation on sampled frames in an input frame queue according to the temporal recovery frame interpolation mode.

[0056] According to another aspect of this disclosure, an encoding apparatus is also provided, comprising: a video encoding module configured to encode a video to obtain bitstream data, so that a decoding end implements the decoding method described in any of the preceding claims based on the bitstream data.

[0057] According to one aspect of this disclosure, a decoding method is provided, applied at a decoding end, comprising: determining whether the current sampling frame is a first sampling frame; and if the current sampling frame is not the first sampling frame, determining the temporal sampling information corresponding to the first sampling frame.

[0058] In one embodiment of this disclosure, the temporal sampling information corresponding to the first sampling frame includes a temporal sampling rate hold bit; the method further includes: updating the global temporal frame extraction ratio when the temporal sampling rate hold bit is active.

[0059] In one embodiment of this disclosure, the method further includes: saving the first sampling frame.

[0060] In one embodiment of this disclosure, saving the first sampled frame includes: configuring the first sampled frame in an output queue; and saving the output queue.

[0061] In one embodiment of this disclosure, the method further includes: determining whether the global temporal frame sampling ratio is 1; if the global temporal frame sampling ratio is 1, determining the frame buffer information corresponding to the current sampled frame as the frame buffer information corresponding to the first sampled frame.

[0062] In one embodiment of this disclosure, the method further includes: determining the number of interpolated frames between the current sampling frame and the first sampling frame, including: determining the number of interpolated frames between the current sampling frame and the first sampling frame based on the temporal frame extraction ratio of the first sampling frame, wherein the temporal frame extraction ratio of the first sampling frame can be determined by the global temporal frame extraction ratio.

[0063] In one embodiment of this disclosure, the method further includes: performing frame interpolation based on the number of interpolated frames, the current sampled frame, the first sampled frame, and a preset decoding method.

[0064] In one embodiment of this disclosure, the method further includes: saving the first sampled frame, the current sampled frame, and the frame obtained by the supplementary frame in sequence.

[0065] In one embodiment of this disclosure, saving the first sampled frame, the current sampled frame, and the frame obtained by the supplementary frame in sequence order includes: configuring the first sampled frame, the current sampled frame, and the frame obtained by the supplementary frame in sequence order in the output queue; and saving the output queue.

[0066] In one embodiment of this disclosure, the method further includes: configuring frame buffer information for the current sampling frame when the current sampling frame is the first sampling frame.

[0067] In one embodiment of this disclosure, determining whether the current sampling frame is the first sampling frame includes: acquiring the current sampling frame; and, if the current sampling frame is acquired, determining whether the current sampling frame is the first sampling frame.

[0068] In one embodiment of this disclosure, the method further includes: saving the remaining frames if the current sampling frame is not acquired.

[0069] In one embodiment of this disclosure, the method further includes: upon receiving end information and obtaining the current sampling frame, saving the current sampling frame and the remaining frames.

[0070] According to another aspect of this disclosure, a decoding method is provided, applied at a decoding end, comprising: acquiring temporal information of a sampled image corresponding to a sampled frame; and determining whether a temporal sampling flag is set based on the temporal information of the sampled image.

[0071] In one embodiment of this disclosure, the method further includes: determining that the time domain recovery function of the current sampling frame is disabled when the time domain sampling flag bit is not set.

[0072] In one embodiment of this disclosure, the method further includes: obtaining the sampling rate corresponding to the sampling frame while keeping the time-domain sampling rate at a fixed position.

[0073] According to another aspect of this disclosure, a decoding method is provided, applied at a decoding end, comprising: performing frame interpolation on the current frame based on target frames preceding the current frame.

[0074] In one embodiment of this disclosure, the current frame is supplemented based on the previous frames, which includes: copying the target frame to obtain the current frame, wherein the target frame includes a reference frame.

[0075] In one embodiment of this disclosure, copying a target frame to obtain a current frame, wherein the target frame includes a reference frame, includes: copying the target frame to obtain a current frame, wherein the target frame includes a reference frame, if the number of target frames preceding the current frame does not meet a preset number.

[0076] In one embodiment of this disclosure, the process of interpolating the current frame based on the previous frames includes: copying the target frame to obtain the current frame, wherein the target frame includes the interpolated frame.

[0077] In one embodiment of this disclosure, the current frame is interpolated based on the frames preceding the current frame, which includes: interpolating the current frame based on multiple target frames to obtain the current frame, wherein the target frames include reference frames.

[0078] In one embodiment of this disclosure, the current frame is interpolated based on the frames preceding the current frame, which includes: interpolating the current frame based on multiple target frames to obtain the current frame, wherein the target frames include the reference frame and the interpolated frame.

[0079] In one embodiment of this disclosure, the current frame includes a reference frame.

[0080] In one embodiment of this disclosure, the reference frame, the current frame, and the frame obtained by the interpolation frame are determined as the interpolation frame segment.

[0081] According to another aspect of this disclosure, an encoding method is provided, applied at an encoding end, comprising:

[0082] The video is encoded to obtain bitstream data, so that the decoding end can implement any of the decoding methods in the above embodiments based on the bitstream data.

[0083] According to another aspect of this disclosure, a decoding apparatus is provided, applied at a decoding end, comprising: a first determining module configured to determine whether the current sampling frame is a first sampling frame; and a second determining module configured to determine the temporal sampling information corresponding to the first sampling frame when the current sampling frame is not the first sampling frame.

[0084] According to another aspect of this disclosure, a decoding apparatus is provided, applied at a decoding end, comprising: a first acquisition module configured to acquire temporal information of a sampled image corresponding to a sampled frame; and a third determination module configured to determine whether a temporal sampling flag is set based on the temporal information of the sampled image.

[0085] According to another aspect of this disclosure, a decoding apparatus is provided, applied at a decoding end, comprising: a frame interpolation module configured to interpolate the current frame based on a target frame preceding the current frame.

[0086] According to another aspect of this disclosure, an encoding apparatus is provided for use at an encoding end, comprising: an encoding module configured to encode video to obtain bitstream data, so that a decoding end can implement any decoding method based on the bitstream data.

[0087] According to another aspect of this disclosure, a decoding method is also provided, comprising: performing a format conversion on an input frame queue of a first format to obtain a local frame queue of a second format; performing temporal interpolation on the frame queue of the second format to generate an output frame queue of the second format; and performing a format conversion on the output frame queue of the second format to obtain an output frame queue of the first format.

[0088] In one embodiment of this disclosure, before performing temporal frame interpolation on the second-format frame queue to generate the second-format output frame queue, the method further includes: performing at least one of the following preprocessing operations on each frame image in the local frame queue: padding, normalization, and brightness enhancement; before performing format conversion on the second-format output frame queue to obtain the first-format output frame queue, the method further includes: performing at least one of the following postprocessing operations on the second-format output frame queue: brightness restoration, denormalization, and depadding.

[0089] In one embodiment of this disclosure, the first format is YUV format; the second format is RGB format.

[0090] In one embodiment of this disclosure, the YUV format is YCbCr420 format; the RGB format is RGB24.

[0091] In one embodiment of this disclosure, the conversion process from YCbCr420 format to RGB24 format is implemented through the following steps:

[0092] Input: An image picIn[3,hIn,wIn] in YcbCr 4:2:0 chroma format and a variable bitDepthOut set to specify the output bit depth for all color planes;

[0093] Output: An RGB24 4:4:4 chroma image picOut[3,hIn,wIn];

[0094] The conversion process is as follows:

[0095] 1) Set variables hOut = Ceil(hIn / 32) * 32, wOut = Ceil(wIn / 32) * 32;

[0096] 2) Perform the following steps for i in the range of 0 to hIn–1:

[0097] Perform the following steps for j in the range of 0 to wIn–1:

[0098] y[i,j]=Clip3(0,255,picInY[i,j]>>(bitDepthLocal[0]–8)-16);

[0099] u[i,j]=picInU[i / 2,j / 2]>>(bitDepthLocal[1]–8)-128;

[0100] v[i,j]=picInV[i / 2,j / 2]>>(bitDepthLocal[2]–8)-128;

[0101] pictOut[0,i,j]=Clip3(0,255,(1220542*y[i,j]+1673527*v[i,j]+(1<<19)) / / (1<<20));

[0102] pictOut[1,i,j]=Clip3(0,255,(1220542*y[i,j]-852492*v[i,j]-409993*u[i,j]+(1<<19)) / / (1<<20));

[0103] pictOut[2,i,j]=Clip3(0,255,(1220542*y[i,j]+2116026*u[i,j]+(1<<19)) / / (1<<20));

[0104] Perform the following steps on j within the range wIn to wOut-1:

[0105] picOut[0,i,j]=picOut[0,i,wIn–1];

[0106] picOut[1,i,j]=picOut[1,i,wIn–1];

[0107] picOut[2,i,j]=picOut[2,i,wIn–1];

[0108] 3) Perform the following steps for i within the range hIn to hOut-1:

[0109] Perform the following steps for j in the range of 0 to wIn-1:

[0110] picOut[0,i,j]=picOut[0,hIn–1,j];

[0111] picOut[1,i,j]=picOut[1,hIn–1,j];

[0112] picOut[2,i,j]=picOut[2,hIn–1,j];

[0113] Perform the following steps on j within the range wIn to wOut-1:

[0114] picOut[0,i,j]=picOut[0,hIn–1,wIn–1];

[0115] picOut[1,i,j]=picOut[1,hIn–1,wIn–1];

[0116] picOut[2,i,j]=picOut[2,hIn–1,wIn–1];

[0117] When bitDepthOut is not specified, the variable bitDepthLocal[3] is set to BitDepth; otherwise, each value in bitDepthLocal[3] is assigned the value bitDepthOut.

[0118] In one embodiment of this disclosure, the conversion process from RGB24 format to YCbCr420 is implemented through the following steps:

[0119] Input: An RGB24 4:4:4 chroma format image picOut[3,hIn,wIn] and a variable bitDepthOut set to specify the output bit depth for all color planes;

[0120] Output: A YcbCr 4:2:0 chroma format image picIn[3,hIn,wIn];

[0121] The conversion process is as follows:

[0122] For k in the range 0k 2, i in the range 0i hIn–1, and j in the range 0j wIn–1, perform the following steps:

[0123] 1) Set p[k,i,j] = Clip3(0,255,(pictIn[k,i,j]*256));

[0124] 2) Set the variable scale[3] = 2bitDepthLocal[cIdx] – 8, where 0 = 2 for cIdx;

[0125] 3) For i in the range 0i hOut / 2–1, and j in the range 0j wOut / 2–1, perform the following steps:

[0126] picOut[0,2*i,2*j]=Clip3(0,bitDepthLocal[0],(269484*p[0,2*i,2*j]+52848 2*p[1,2*i,2*j]+102760*p[2,i,j]+(1<<19)+(16<<20)) / / (1<<20))*scale[0]);

[0127] picOut[0,2*i,2*j+1]=Clip3(0,bitDepthLocal[0],(269484*p[0,2*i,2*j+1]+52848 2*p[1,2*i,2*j+1]+102760*p[2,i,j+1]+(1<<19)+(16<<20)) / / (1<<20))*scale[0]);

[0128] picOut[0,2*i+1,2*j]=Clip3(0,bitDepthLocal[0],(269484*p[0,2*i+1,2*j]+52848 2*p[1,2*i+1,2*j]+102760*p[2,i+1,j]+(1<<19)+(16<<20)) / / (1<<20))*scale[0]);

[0129] picOut[0,2*i+1,2*j+1]=Clip3(0,bitDepthLocal[0],(269484*p[0,2*i+1,2*j+1]+52848 2*p[1,2*i+1,2*j+1]+102760*p[2,i+1,j+1]+(1<<19)+(16<<20)) / / (1<<20))*scale[0]);

[0130] picOut[1,i,j]=Clip3(0,bitDepthLocal[1],(–155188*p[0,i,j]–305135*p[1,i,j]+460324*p[2,i,j])*scale[1]);

[0131] picOut[2,i,j]=Clip3(0,bitDepthLocal[2],(460324*p[0,i,j]–385875*p[1,i,j]–74448*p[2,i,j])*scale[2]);

[0132] When bitDepthOut is not specified, the variable bitDepthLocal[3] is set to BitDepth; otherwise, each value in bitDepthLocal[3] is assigned the value bitDepthOut.

[0133] In one embodiment of this disclosure, the brightness enhancement process is implemented through the following steps:

[0134] Input: An RGB24 formatted image picOut[3,hIn,wIn];

[0135] Output: An RGB24 formatted image picOut[3,hIn,wIn] and the variable lumaEnhancedFlag;

[0136] The brightness enhancement process is as follows:

[0137] 1) Initialize lumaEnhancedFlag to 0;

[0138] 2) Perform the following format conversion on picIn to obtain the variable picTmp:

[0139] For i in the range 0i hIn–1 and j in the range 0j wIn–1, perform the following steps:

[0140] picTmp[0,i,j]=(picIn[0,i,j]*4899+picIn[1,i,j]*9617+picIn[2,i,j]*1868+(1<<(14-1)))>>14

[0141] picTmp[1,i,j]=Clip3(0,255,((picIn[2,i,j]-picTmp[0,i,j])*1868+128<<14+(1<<(14-1)))>>14)

[0142] picTmp[2,i,j]=Clip3(0,255,((picIn[0,i,j]-picTmp[0,i,j])*14369+128<<14+(1<<(14-1)))>>14)

[0143] picTmp[0,i,j]=Clip3(0,255,picTmp[0,i,j])

[0144] Wherein, picIn[0] represents the R channel, picIn[1] represents the G channel, picIn[2] represents the B channel; picTmp[0] represents the Y channel; picTmp[1] represents the U channel; picTmp[2] represents the V channel;

[0145] 3) When the mean of picTmp[0,hOut,wOut] is less than 50, picTmp[0] is shifted left by 1 bit, and lumaEnhancedFlag is set to 1;

[0146] 4) Perform the following format conversion on picTmp to obtain the output picOut:

[0147] For i in the range 0i hIn–1 and j in the range 0j wIn–1, perform the following steps:

[0148] picOut[0,i,j]=picTmp[0,i,j]+(((picTmp[1,i,j]-128)*18678+1<<(14-1))>>14);

[0149] picOut[1,i,j]=picTmp[0,i,j]+(((picTmp[1,i,j]-128)*(-6472)+(picTmp[2,i,j]-128)*(-9519)+1<<(14-1))>>14);

[0150] picOut[2,i,j]=picTmp[0,i,j]+(((picTmp[1,i,j]-128)*33292+1<<(14-1))>>14).

[0151] In one embodiment of this disclosure, the brightness restoration process is achieved through the following steps:

[0152] Input: An RGB24 formatted image picOut[3,hIn,wIn] and a variable lumaEnhancedFlag;

[0153] Output: An RGB24 formatted image picOut[3,hIn,wIn];

[0154] The brightness recovery process is as follows:

[0155] When lumaEnhancedFlag equals 1, perform the following steps:

[0156] 1) Perform the following format conversion on picIn to obtain the variable picTmp:

[0157] 2) For i in the range 0i hIn–1 and j in the range 0j wIn–1, perform the following steps:

[0158] picTmp[0,i,j]=(picIn[0,i,j]*4899+picIn[1,i,j]*9617+picIn[2,i,j]*1868+(1<<(14-1)))>>14;

[0159] picTmp[1,i,j]=Clip3(0,255,((picIn[2,i,j]-picTmp[0,i,j])*1868+128<<14+(1<<(14-1)))>>14);

[0160] picTmp[2,i,j]=Clip3(0,255,((picIn[0,i,j]-picTmp[0,i,j])*14369+128<<14+(1<<(14-1)))>>14);

[0161] picTmp[0,i,j]=Clip3(0,255,picTmp[0,i,j])>>1;

[0162] 3) Perform the following format conversion on picTmp to obtain the output picOut:

[0163] For i in the range 0i hIn–1 and j in the range 0j wIn–1, perform the following steps:

[0164] picOut[0,i,j]=picTmp[0,i,j]+(((picTmp[1,i,j]-128)*18678+1<<(14-1))>>14);

[0165] picOut[1,i,j]=picTmp[0,i,j]+(((picTmp[1,i,j]-128)*(-6472)+(picTmp[2,i,j]-128)*(-9519)+1<<(14-1))>>14);

[0166] picOut[2,i,j]=picTmp[0,i,j]+(((picTmp[1,i,j]-128)*33292+1<<(14-1))>>14).

[0167] In one embodiment of this disclosure, temporal frame interpolation is performed through the following steps:

[0168] Determine the temporal recovery frame interpolation mode; according to the temporal recovery frame interpolation mode, perform temporal interpolation and / or temporal extrapolation on the sampled frames in the input frame queue.

[0169] In one embodiment of this disclosure, before performing temporal interpolation and / or temporal extrapolation on the sampled frames in the input frame queue according to the temporal recovery interpolation mode, the method further includes: performing a temporal recovery initialization operation; when no bitstream end information is received, performing temporal interpolation or temporal extrapolation processing on each sampled frame in the frame queue according to the temporal recovery interpolation mode; when bitstream end information is received, determining the tail prediction input frame queue, and performing temporal tail recovery processing on the input frame queue according to the tail prediction input frame queue.

[0170] In one embodiment of this disclosure, performing a time-domain recovery initialization operation includes at least one of the following: initializing the output frame queue to an empty frame queue; initializing the global time-domain recovery parameters to sequence-level time-domain recovery parameters; and initializing the reference frame queue to an empty frame queue.

[0171] In one embodiment of this disclosure, for each sampled frame in the frame queue, temporal interpolation or temporal extrapolation is performed according to the temporal recovery interpolation mode, including: if the temporal recovery interpolation mode of the current frame is temporal interpolation and the current frame is not the first frame, then temporal interpolation is performed; if the temporal recovery interpolation mode of the current frame is temporal extrapolation, then the current frame is added to the reference frame queue, and when the length of the reference frame sequence reaches the temporal extrapolation prediction number, temporal extrapolation is performed.

[0172] In one embodiment of this disclosure, the method further includes: if the temporal recovery interpolation mode of the current frame is temporal interpolation and the current frame is the first frame, then only the step of adding the current frame to the output frame queue is performed, and temporal interpolation processing is not performed.

[0173] In one embodiment of this disclosure, performing temporal interpolation frame processing includes: taking the previous frame, the current frame, and temporal sampling parameters as inputs to the temporal interpolation frame processing process, outputting interpolated frames, and adding them to the output frame queue.

[0174] In one embodiment of this disclosure, the method further includes: after performing the temporal interpolation frame processing, adding the current frame to the output frame queue.

[0175] In one embodiment of this disclosure, the time-domain sampling parameters include: time-domain resampling rate.

[0176] In one embodiment of this disclosure, performing time-domain interpolation frame processing further includes: if the time-domain sampling rate change flag / time-domain recovery update flag of the current frame is 1, updating the global time-domain recovery parameters to the time-domain recovery parameters of the current frame; if the time-domain sampling rate change flag / time-domain recovery update flag of the current frame is 0, keeping the global time-domain recovery parameters unchanged and the same as the time-domain recovery parameters of the previous frame.

[0177] In one embodiment of this disclosure, after updating the global temporal recovery parameters to the temporal recovery parameters of the current frame, the method further includes: if the temporal recovery interpolation mode of the current frame is temporal extrapolation, then adding the current frame to the reference frame queue.

[0178] In one embodiment of this disclosure, performing temporal extrapolation frame interpolation processing includes: taking a reference frame sequence and the number of temporal extrapolation frame predictions as inputs to the temporal extrapolation frame interpolation process, outputting interpolated frames, and adding them to the output frame sequence.

[0179] In one embodiment of this disclosure, the method further includes: after performing the temporal extrapolation frame processing, adding the current frame to the output frame queue.

[0180] In one embodiment of this disclosure, the method further includes: if the temporal sampling rate change flag / temporal recovery update flag of the current frame is 1, updating the global temporal recovery parameters to the temporal recovery parameters of the current frame; if the temporal sampling rate change flag / temporal recovery update flag of the current frame is 0, the global temporal recovery parameters remain unchanged and are the same as the temporal recovery parameters of the previous frame.

[0181] In one embodiment of this disclosure, after updating the global temporal recovery parameters to the temporal recovery parameters of the current frame, the method further includes: if the temporal recovery frame interpolation mode of the current frame is temporal interpolation, then setting the reference frame queue to an empty frame queue; if the temporal recovery frame interpolation mode of the current frame is temporal extrapolation, then adding the current frame to the reference frame queue; when the length of the reference frame sequence reaches the temporal extrapolation prediction number, using the reference frame sequence and the temporal extrapolation prediction number as input to the temporal extrapolation processing procedure, outputting the interpolated frame, adding it to the output frame sequence, and setting the reference frame queue to an empty frame queue.

[0182] In one embodiment of this disclosure, a tail prediction input frame queue is determined, and temporal tail recovery processing is performed on the input frame queue according to the tail prediction input frame queue, including: if the temporal recovery frame interpolation mode is temporal interpolation, then the tail prediction input frame queue is set as a queue containing the last frame in the input frame queue; if the temporal recovery frame interpolation mode is temporal extrapolation, then the tail prediction input frame queue is set as a queue containing the last N frames in the output frame queue, where N is the number of temporal extrapolation prediction frames; the tail prediction input frame queue, the remaining number of temporal recovery frames, and the temporal recovery frame interpolation mode are used as inputs to the temporal tail recovery processing, and a tail frame queue is output; if there are frames after the last sampled frame in the output frame queue, then the frames after the last sampled frame are deleted, and the tail frame queue is added after the last sampled frame.

[0183] According to another aspect of this disclosure, an encoding method is also provided, comprising: encoding a video to obtain bitstream data, so that a decoding end implements the decoding method described in any of the preceding claims based on the bitstream data.

[0184] According to another aspect of this disclosure, a decoding apparatus is also provided, comprising: a first format conversion module configured to perform format conversion on an input frame queue of a first format to obtain a local frame queue of a second format; a temporal frame interpolation module configured to perform temporal frame interpolation on the frame queue of the second format to generate an output frame queue of the second format; and a second format conversion module configured to perform format conversion on the output frame queue of the second format to obtain an output frame queue of the first format.

[0185] According to another aspect of this disclosure, an encoding apparatus is also provided, comprising: an encoding module configured to encode video to obtain bitstream data, so that a decoding end implements the decoding method described in any of the preceding claims based on the bitstream data.

[0186] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory configured to store executable instructions of the processor; wherein the processor is configured to perform the above-described encoding or decoding method by executing the executable instructions.

[0187] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the above-described encoding or decoding method.

[0188] According to another aspect of this disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements the encoding method or decoding method of any of the above.

[0189] The encoding and decoding methods, apparatuses, devices, media, and computer program products provided in the embodiments of this disclosure can perform time-domain intra-interpolation and / or time-domain extra-interpolation on sampled frames in the input frame queue according to the time-domain recovery interpolation mode, thereby achieving mixed time-domain and extra-time interpolation and improving decoding performance.

[0190] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0191] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0192] Figure 1 shows a flowchart of a decoding method according to an embodiment of the present disclosure;

[0193] Figure 2 shows a schematic diagram of an example of a decoding method in an embodiment of this disclosure;

[0194] Figure 3 shows a schematic diagram of another decoding method example in the embodiments of this disclosure;

[0195] Figure 4 shows another decoding schematic diagram in an embodiment of this disclosure;

[0196] Figure 5 shows another decoding schematic diagram in an embodiment of this disclosure;

[0197] Figure 6 shows another decoding schematic diagram in an embodiment of this disclosure;

[0198] Figure 7 shows another decoding schematic diagram in an embodiment of this disclosure;

[0199] Figure 8 shows another decoding schematic diagram in an embodiment of this disclosure;

[0200] Figure 9 shows another decoding schematic diagram in an embodiment of this disclosure;

[0201] Figure 10 shows another decoding schematic diagram in an embodiment of this disclosure;

[0202] Figure 11 shows another decoding schematic diagram in an embodiment of this disclosure;

[0203] Figure 12 shows a schematic diagram of a decoding process according to an embodiment of the present disclosure;

[0204] Figure 13 illustrates another decoding process in an embodiment of this disclosure;

[0205] Figure 14 shows a flowchart of an encoding method according to an embodiment of the present disclosure;

[0206] Figure 15 shows a schematic diagram of a decoding device according to an embodiment of the present disclosure;

[0207] Figure 16 shows a schematic diagram of a decoding device according to an embodiment of the present disclosure;

[0208] Figure 17 shows a schematic diagram of a decoding device according to an embodiment of the present disclosure;

[0209] Figure 18 shows a schematic diagram of another encoding device in an embodiment of this disclosure;

[0210] Figure 19 shows a flowchart of another decoding method in an embodiment of this disclosure;

[0211] Figure 20 shows a flowchart of another encoding method in an embodiment of this disclosure;

[0212] Figure 21 shows a schematic diagram of a decoding device according to an embodiment of the present disclosure;

[0213] Figure 22 shows a schematic diagram of another encoding device in an embodiment of this disclosure;

[0214] Figure 23 shows a flowchart of another decoding method in an embodiment of this disclosure;

[0215] Figure 24 shows a flowchart of another encoding method in an embodiment of this disclosure;

[0216] Figure 25 shows a schematic diagram of another decoding device in an embodiment of this disclosure;

[0217] Figure 26 shows a schematic diagram of another encoding device in an embodiment of this disclosure;

[0218] Figure 27 shows a flowchart of another decoding method according to an embodiment of the present disclosure;

[0219] Figure 28 shows a flowchart of another encoding method in an embodiment of this disclosure;

[0220] Figure 29 shows a schematic diagram of another decoding device in an embodiment of this disclosure;

[0221] Figure 30 shows a structural block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0222] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0223] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0224] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0225] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0226] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0227] To address the aforementioned technical problems, this disclosure provides an encoding and decoding method, apparatus, device, and storage medium. To provide a detailed description of this disclosure, the identification method is described first.

[0228] Figure 1 shows a flowchart of a decoding method according to an embodiment of this disclosure. It should be noted that the identification method in this embodiment is applied at the decoding end.

[0229] As shown in Figure 1, the decoding method may include:

[0230] S110, Determine whether the current sampling frame is the first sampling frame;

[0231] S120, if the current sampling frame is not the first sampling frame, determine the time-domain sampling information corresponding to the first sampling frame.

[0232] In some embodiments, the first sampled frame may be the first frame in a supplementary frame segment, the first frame between two sampled frames, or any sampled frame.

[0233] In some embodiments, the current sampling frame can be the first sampling frame, a consecutive sampling frame following the current sampling frame, or any frame following the current sampling frame.

[0234] For example, the current sampling frame can be a frame.

[0235] In some embodiments, the time-domain sampling flag may be a time-domain sampling change flag, a time-domain sampling update flag, a time-domain sampling switch flag, etc.

[0236] In some embodiments, the frame buffer information can be a frameBuffer.

[0237] In some embodiments, the temporal sampling information corresponding to the first sampling frame includes a temporal sampling rate hold bit.

[0238] In some embodiments, the time-domain sampling flag can be enabledFlag.

[0239] In some embodiments, the global temporal frame extraction ratio is updated while the temporal sampling rate remains unchanged.

[0240] For example, the temporal sampling flag setting may include enabledFlag=1.

[0241] In some embodiments, the global temporal frame extraction ratio can also be the historical temporal frame extraction ratio.

[0242] In some embodiments, the global temporal frame extraction ratio can be TemporalRatio.

[0243] In some embodiments, TemporalRatio is a sequence-level parameter.

[0244] In some embodiments, the encoder may set the time-domain sampling flag to 1 at a specific location as needed, so that the magnification is maintained.

[0245] In some embodiments, the method further includes: saving the first sampled frame.

[0246] In some embodiments, the first sample frame can be saved separately directly after it has been determined.

[0247] In some embodiments, the first sampled frame can also be configured in an output queue and the output queue can be saved. The first sampled frame is saved in the form of an output queue.

[0248] In some embodiments, the method further includes:

[0249] Determine if the global temporal frame sampling ratio is 1;

[0250] When the global temporal frame sampling ratio is 1, the frame buffer information corresponding to the current sampled frame is determined as the frame buffer information corresponding to the first sampled frame.

[0251] In some embodiments, determining the number of interpolation frames between the current sampled frame and the first sampled frame includes:

[0252] The number of interpolated frames between the current sampling frame and the first sampling frame is determined based on the temporal frame extraction ratio of the first sampling frame, wherein the temporal frame extraction ratio of the first sampling frame can be determined by the global temporal frame extraction ratio.

[0253] Frame interpolation is performed based on the number of interpolated frames, the current sampled frame, the first sampled frame, and the preset decoding method.

[0254] For example, after determining the current sampling frame, the first sampling frame, and the number of interpolated frames, interpolation can be performed after the first sampling frame and before the current sampling frame based on the number of interpolated frames.

[0255] In some embodiments, the number of frames can be TemporalRatio-1.

[0256] For example, the above frames can be saved in the order of the first sampled frame, the supplementary frame, and the current sampled frame.

[0257] For example, the above frames can be saved in the order of the first sampled frame, the supplementary frame, and the current sampled frame.

[0258] For example, the above frames can also be set in the output queue and the output queue can be saved.

[0259] In some embodiments, the first sampled frame, the current sampled frame, and the frame obtained by the supplementary frame are saved in sequence.

[0260] In some embodiments, the first sampled frame, the current sampled frame, and the frame obtained by the supplementary frame are arranged in the output queue in sequence order;

[0261] Save the output queue.

[0262] In some embodiments, when the current sampled frame is the first sampled frame, the frame buffer information of the current sampled frame is configured.

[0263] In some embodiments, determining whether the current sampled frame is the first sampled frame includes:

[0264] Get the current sampling frame;

[0265] Given the current sampled frame, determine whether the current sampled frame is the first sampled frame.

[0266] In some embodiments, the remaining frames are saved if the current sampled frame is not obtained.

[0267] In some embodiments, the remaining frame includes remain.

[0268] In some embodiments, upon receiving end information and obtaining the current sampling frame, the current sampling frame and the remaining frames are saved.

[0269] To provide a detailed explanation of this disclosure, a schematic diagram illustrating an example of a decoding method is also provided. As shown in Figure 2, the method may include:

[0270] S201. Obtain an image frame. If the acquisition is successful, proceed to step S202; if the acquisition fails, proceed to step S206.

[0271] S202. Determine whether the previous sampled frame buffer is empty. If it is empty, it means that the current frame is the first sampled frame. Directly assign the frame to the frame buffer and proceed to step S201; if it is not empty, proceed to step S203.

[0272] S203. Obtain the temporal information of the previous sampled frame, determine whether enabledFlag is S201, if yes, update the global temporal frame sampling ratio TemporalRatio and save the image of the previous sampled frame; if no, directly save the image of the previous sampled frame.

[0273] S204. Determine if the global temporal frame extraction ratio TemporalRatio is equal to S201. If yes, proceed to step S205. If no, send the three parameters (frameBuffer, frame, TemporalRatio-1) to the decoding method and save the TemporalRatio-1 supplementary frame images returned by the decoding method.

[0274] S205. Assign frame to frameBuffer and proceed to step S201.

[0275] S206. If the remain parameter is not 0, if frame is empty, save (remain+1) frameBuffer images; if frame is not empty, save (remain+1) frame images.

[0276] To provide a detailed explanation of this disclosure, a schematic diagram illustrating an example of a decoding method is also provided. As shown in Figure 3, TemporalRatio is a sequence-level parameter, and its initial value can be specified in SPS.

[0277] The frame-level parameter (prdTemEnabledFlag or enabledFlag+temporalRatioIdx) ends with information (remain). The temporalRatioIdx and TemporalRatio range from 0 to 3, corresponding to temporal sampling ratios of 1, 2, 4, and 8, respectively.

[0278] Figure 4 shows a flowchart of another decoding method according to an embodiment of this disclosure. It should be noted that the identification method in this embodiment is applied at the decoding end.

[0279] S410, Obtain the temporal information of the sampled image corresponding to the sampled frame;

[0280] S420 determines whether the time-domain sampling flag is set based on the time-domain information of the sampled image.

[0281] In some embodiments, if the time-domain sampling flag is not set, it is determined that the time-domain recovery function of the current sampling frame is disabled.

[0282] In some embodiments, if the temporal sampling flag is not set, frame interpolation based on the historical temporal sampling rate is determined.

[0283] In some embodiments, the sampling rate corresponding to the sampling frame is obtained while the temporal sampling rate is kept in place.

[0284] In some embodiments, temporalRatio is a frame-level parameter, frame-level parameters (enabledFlag and temporalRatioIdx) and end information (remain).

[0285] For example, the temporal information of each sampled image includes enabledFlag and temporalRatioIdx. When enabledFlag is 1, temporalRatioIdx ranges from 0 to 3, corresponding to 1, 2, 4, and 8 respectively. When enabledFlag is 0, it means that the temporal recovery function is turned off.

[0286] For example, the temporal information of each sampled image includes enabledFlag and temporalRatioIdx. When enabledFlag is 1, temporalRatioIdx ranges from 0 to 3, corresponding to 1, 2, 4, and 8 respectively. When enabledFlag is 0, it indicates that a fixed sequence-level temporal sampling ratio TemporalRatio is used.

[0287] Figure 5 shows a flowchart of another decoding method according to an embodiment of this disclosure. It should be noted that the identification method in this embodiment is applied at the decoding end.

[0288] As shown in Figure 5, the method may include:

[0289] S510, performs frame interpolation on the current frame based on the target frames preceding the current frame.

[0290] In some embodiments, the current frame is padded with frames based on previous frames, including:

[0291] The target frame is copied to obtain the current frame. The target frame includes the reference frame.

[0292] In some embodiments, the target frame is copied to obtain the current frame, wherein the target frame includes a reference frame, including:

[0293] If the number of target frames preceding the current frame does not meet the preset number, the target frame is copied to obtain the current frame. The target frame includes the reference frame.

[0294] In some embodiments, the current frame is padded with frames based on previous frames, including:

[0295] The target frame is copied to obtain the current frame. The target frame includes the frame obtained by piecing together the original frame.

[0296] In some embodiments, the current frame is padded with frames based on previous frames, including:

[0297] The current frame is obtained by interpolating multiple target frames, where the target frames include the reference frame.

[0298] In some embodiments, the current frame is padded with frames based on previous frames, including:

[0299] The current frame is obtained by interpolating multiple target frames. The target frames include the reference frame and the interpolated frame.

[0300] In some embodiments, the current frame includes a reference frame.

[0301] It should be noted that this disclosure can be used to compose a frame interpolation segment based on the frame obtained by any of the above frame interpolation, the reference frame, and the current frame.

[0302] In some embodiments, when multiple target frames are interpolated, the multiple target frames can be discontinuous target frames.

[0303] In some embodiments, it is determined whether the number of supplementary frames after the last sampled frame is the same as the number of remaining frames.

[0304] In some embodiments, if the number of supplementary frames after the last sampled frame is different from the number of remaining frames, the number of supplementary frames after the last sampled frame is modified to be the same as the number of remaining frames.

[0305] In some embodiments, if the number of supplementary frames after the last sampled frame is greater than the number of remaining frames, the supplementary frames after the last sampled frame are deleted so that the number of supplementary frames after the last sampled frame is the same as the number of remaining frames.

[0306] In some embodiments, the direct copying frame extraction ratio is an integer greater than or equal to 2.

[0307] In some embodiments, a frame is added after each of the N adjacent decoded frames, and there may be interpolated frames between the N adjacent decoded frames. The decoded frame can be a reference frame.

[0308] In some embodiments, a new frame is added after each of the N adjacent decoded frames, and there may be no interpolation between the N adjacent decoded frames. The decoded frame may be a reference frame.

[0309] In some embodiments, adjacent N decoded frames are back-complemented by M frames, wherein the first complement operation uses the preceding adjacent N decoded frames as reference frames, and the remaining M-1 complement frames are copies of the first complement frame.

[0310] In some embodiments, adjacent N decoded frames are back-complemented by M frames, wherein the first complement operation uses the preceding adjacent N decoded frames as reference frames, the second complement operation uses the preceding adjacent N-1 decoded frames and the first complement frame as reference frames, and the remaining M-2 complement frames are used to copy the second complement frame.

[0311] In some embodiments, the forward adjacent N-decoded frames may not be adjacent to the supplementary frames.

[0312] In some embodiments, when the current frame does not meet the number of consecutive sampling frames, the subsequent backscrambled frames directly copy the current frame.

[0313] In some embodiments, when the number of backward frames of the current frame is greater than 1, it can be copied from the frames that have been padded after the current frame.

[0314] In some embodiments, the reference consecutive sampling frame number is greater than or equal to 2.

[0315] The ratio of additional frame extraction might be 3:2, etc.

[0316] In some embodiments, the sequence-level syntax parameter can be the newly added sequence-level syntax parameter temporal_restoration_mode in Table 1.

[0317] Table 1

[0318] temporal_restoration_data(){Descriptor

[0319] temporal_restoration_flag u(1)

[0320] if(temporal_restoration_flag){

[0321] temporal_resampling_ratio_idxu(2)

[0322] temporal_restoration_mode u(1)

[0323] }

[0324] byte_alignment()

[0325] }

[0326] A temporal_restoration_mode of 0 indicates forward and backward frame interpolation;

[0327] A temporal_restoration_mode of 1 indicates backward frame interpolation.

[0328] Figure 6 shows a decoding schematic diagram according to an embodiment of the present disclosure.

[0329] Figure 6 shows a decoded segment diagram obtained based on at least one of the above embodiments.

[0330] Figure 7 shows another decoding schematic diagram in an embodiment of this disclosure.

[0331] Figure 7 shows a decoded segment diagram obtained based on at least one of the above embodiments.

[0332] Figure 8 shows another decoding schematic diagram in an embodiment of the present disclosure.

[0333] Figure 8 shows a decoded segment diagram obtained based on at least one of the above embodiments.

[0334] Figure 9 shows another decoding schematic diagram in an embodiment of this disclosure.

[0335] Figure 9 shows a decoded segment diagram obtained based on at least one of the above embodiments.

[0336] Figure 10 shows another decoding schematic diagram in an embodiment of the present disclosure.

[0337] Figure 10 shows a decoded segment diagram obtained based on at least one of the above embodiments.

[0338] Figure 11 shows another decoding schematic diagram in an embodiment of the present disclosure.

[0339] Figure 11 shows a decoded segment diagram obtained based on at least one of the above embodiments.

[0340] Figure 12 illustrates another decoding process in an embodiment of this disclosure.

[0341] As shown in Figure 12, the method may include:

[0342] S1201, Obtain a frame of image;

[0343] S1202, in the event of acquisition failure, proceed to remaining frame processing.

[0344] S1203, if successful, acquire the temporal information of the current frame image.

[0345] S1204, the frame*tempoRatio frame is stored in the output queue.

[0346] Figure 13 illustrates another decoding process in an embodiment of this disclosure.

[0347] As shown in Figure 13, the method may include:

[0348] S1301, the last frame.

[0349] S1302, frame interpolation mode determination: in the case of forward interpolation, the frame*remain frame is stored in the output queue.

[0350] S1303, frame interpolation mode judgment: in the case of backward frame interpolation, determine whether Remain is less than (tempoRatio_last-1); otherwise, end.

[0351] S1304, if so, then pop the (tempoRatio_last-1-remain) frame from the output queue.

[0352] Figure 14 shows a flowchart of an encoding method according to an embodiment of this disclosure. It should be noted that the recognition method in this embodiment is applied at the decoding end.

[0353] As shown in Figure 14, the method may include:

[0354] S1410 encodes the video to obtain the bitstream data.

[0355] In some embodiments, the bitstream data is configured to implement any of the decoding methods described above.

[0356] Based on the same inventive concept, this disclosure also provides a decoding device, as shown in the following embodiments. Since the principle by which this device embodiment solves the problem is similar to that of the above-described method embodiments, the implementation of this device embodiment can refer to the implementation of the above-described method embodiments, and repeated details will not be elaborated further.

[0357] Figure 15 shows a schematic diagram of a decoding device according to an embodiment of the present disclosure.

[0358] As shown in Figure 15, the decoding device 1500 may include:

[0359] The first determining module 1501 is configured to determine whether the current sampling frame is the first sampling frame;

[0360] The second determining module 1502 is configured to determine the temporal sampling information corresponding to the first sampling frame when the current sampling frame is not the first sampling frame.

[0361] In some embodiments, the apparatus further includes:

[0362] The update module is configured to update the global temporal frame extraction ratio while maintaining the position of the temporal sampling rate.

[0363] In some embodiments, the apparatus further includes:

[0364] The first storage module is configured to store the first sampled frame.

[0365] In some embodiments, the apparatus further includes:

[0366] The first configuration module is configured to configure the first sampled frame in the output queue;

[0367] The second storage module is configured to store the output queue.

[0368] In some embodiments, the apparatus further includes:

[0369] The fourth module is set to determine whether the global temporal frame extraction ratio is 1.

[0370] The fifth determination module is configured to determine the frame buffer information corresponding to the current sampled frame as the frame buffer information corresponding to the first sampled frame when the global temporal frame sampling ratio is 1.

[0371] In some embodiments, the apparatus further includes:

[0372] The sixth determining module is configured to determine the number of interpolation frames between the current sampling frame and the first sampling frame.

[0373] In some embodiments, the sixth determining module includes:

[0374] The first determining submodule is configured to determine the number of interpolated frames between the current sampling frame and the first sampling frame based on the temporal frame extraction ratio of the first sampling frame, wherein the temporal frame extraction ratio of the first sampling frame can be determined by the global temporal frame extraction ratio.

[0375] In some embodiments, the apparatus further includes:

[0376] The second frame interpolation module is configured to perform frame interpolation based on the number of interpolated frames, the current sampled frame, the first sampled frame, and a preset decoding method.

[0377] In some embodiments, the apparatus further includes:

[0378] The third saving module is configured to save the first sampled frame, the current sampled frame, and the frames obtained by the supplementary frame in sequence.

[0379] In some embodiments, the third storage module further includes:

[0380] Configure the submodule to arrange the first sampled frame, the current sampled frame, and the frames obtained by the supplementary frame in the output queue in sequence order;

[0381] Save the submodule and set it to save the output queue.

[0382] In some embodiments, the apparatus further includes:

[0383] The second configuration module is configured to configure the frame buffer information of the current sampling frame when the current sampling frame is the first sampling frame.

[0384] In some embodiments, the first determining module includes:

[0385] The acquisition submodule is set to acquire the current sampled frame;

[0386] The sub-unit is defined to determine whether the current sampling frame is the first sampling frame when the current sampling frame is obtained.

[0387] In some embodiments, the apparatus further includes:

[0388] The third saving module is configured to save the remaining frames if the current sampled frame is not obtained.

[0389] In some embodiments, the apparatus further includes:

[0390] The fourth saving module is configured to save the current sampling frame and the remaining frames when the end information is received and the current sampling frame is obtained.

[0391] Figure 16 shows a schematic diagram of another decoding device in an embodiment of this disclosure.

[0392] As shown in Figure 16, the decoding device is applied at the decoding end, and the decoding device may include:

[0393] The first acquisition module 1601 is configured to acquire temporal information of the sampled image corresponding to the sampled frame;

[0394] The third determining module 1602 is configured to determine whether the time-domain sampling flag is set based on the time-domain information of the sampled image.

[0395] In some embodiments, the apparatus further includes:

[0396] The sixth determination module is configured to disable the time-domain recovery function of the current sampling frame if the time-domain sampling flag is not set.

[0397] In some embodiments, the apparatus further includes:

[0398] The seventh determination module is configured to determine frame interpolation based on historical time-domain sampling rate when the time-domain sampling flag is not set.

[0399] In some embodiments, the fourth acquisition module is configured to acquire the sampling rate corresponding to the sampling frame while the time-domain sampling rate holding position is set.

[0400] Figure 17 shows a schematic diagram of another decoding device according to an embodiment of the present disclosure.

[0401] As shown in Figure 17, the decoding device is applied at the decoding end, and the decoding device may include:

[0402] The interpolation module 1701 is configured to interpolate the current frame based on the target frame preceding the current frame.

[0403] In some embodiments, the frame interpolation module includes:

[0404] The first copy submodule is configured to copy the target frame to obtain the current frame, where the target frame includes the reference frame.

[0405] In some embodiments, the copy submodule includes:

[0406] The copy unit is configured to copy the target frame to obtain the current frame if the number of target frames preceding the current frame does not meet a preset number. The target frame includes the reference frame.

[0407] In some embodiments, the frame interpolation module includes:

[0408] The second copying submodule is configured to perform frame interpolation on the current frame based on multiple target frames to obtain the current frame, where the target frames include the reference frame.

[0409] In some embodiments, the frame interpolation module includes:

[0410] The first frame interpolation submodule is configured to interpolate the current frame based on multiple target frames to obtain the current frame. The target frames include the reference frame and the frame obtained by interpolation.

[0411] In some embodiments, the apparatus further includes:

[0412] The seventh determination module is set to determine whether the number of supplementary frames after the last sampled frame is the same as the number of remaining frames.

[0413] In some embodiments, the apparatus further includes:

[0414] Modify the module to make the number of supplementary frames after the last sampled frame the same as the number of remaining frames if the number of supplementary frames after the last sampled frame is different from the number of remaining frames.

[0415] Figure 18 shows a schematic diagram of an encoding device according to an embodiment of the present disclosure.

[0416] As shown in Figure 18, the encoding device is applied at the encoding end, and the encoding device may include:

[0417] Encoding module 1801 is configured to encode video to obtain bitstream data.

[0418] The decoding device and encoding device provided in this embodiment can be configured to execute the recognition methods provided in the above-described method embodiments. Their implementation principles and technical effects are similar, and for the sake of simplicity, they will not be described in detail here.

[0419] Figure 19 shows a flowchart of another decoding method according to an embodiment of the present disclosure. As shown in Figure 19, the decoding method provided in this embodiment of the present disclosure may include the following steps:

[0420] S1902, determine the time-domain recovery frame interpolation mode.

[0421] It should be noted that, in the embodiments of this disclosure, the temporal recovery frame interpolation mode is set to indicate the frame interpolation mode used in temporal frame interpolation. It can be the frame interpolation mode of mode 1 (i.e., temporal extrapolation frame interpolation mode, also known as temporal extrapolation frame interpolation mode) or the frame interpolation mode of mode 0 (i.e., temporal interpolation frame interpolation mode, also known as temporal forward and backward frame interpolation mode). The temporal recovery frame interpolation mode in the embodiments of this disclosure can be at the sequence level or at the frame level.

[0422] S1904, according to the time-domain recovery interpolation mode, perform time-domain interpolation and / or time-domain extrapolation on the sampled frames in the input frame queue.

[0423] In one embodiment of this disclosure, before performing temporal interpolation and / or temporal extrapolation on the sampled frames in the input frame queue according to the temporal recovery interpolation mode, the decoding method provided in this embodiment may further include the following steps: performing a temporal recovery initialization operation; when no end-of-stream information is received, performing temporal interpolation or temporal extrapolation processing on each sampled frame in the frame queue according to the temporal recovery interpolation mode; when end-of-stream information is received, determining the tail prediction input frame queue, and performing temporal tail recovery processing on the input frame queue according to the tail prediction input frame queue. In some embodiments, when the temporal recovery interpolation function is enabled, the temporal recovery initialization operation is performed. If the temporal recovery interpolation function is not enabled, the temporal recovery initialization operation is not performed.

[0424] In one embodiment of this disclosure, performing a time-domain recovery initialization operation includes at least one of the following: initializing the output frame queue to an empty frame queue; initializing the global time-domain recovery parameters to sequence-level time-domain recovery parameters; and initializing the reference frame queue to an empty frame queue.

[0425] In one embodiment of this disclosure, for each sampled frame in the frame queue, temporal interpolation or temporal extrapolation is performed according to the temporal recovery interpolation mode, including: if the temporal recovery interpolation mode of the current frame is temporal interpolation and the current frame is not the first frame, then temporal interpolation is performed; if the temporal recovery interpolation mode of the current frame is temporal extrapolation, then the current frame is added to the reference frame queue, and when the length of the reference frame sequence reaches the temporal extrapolation prediction number, temporal extrapolation is performed.

[0426] In one embodiment of this disclosure, the decoding method provided in this embodiment may further include the following steps: if the temporal recovery interpolation mode of the current frame is temporal interpolation and the current frame is the first frame, then only the step of adding the current frame to the output frame queue is performed, and temporal interpolation is not performed.

[0427] In one embodiment of this disclosure, performing temporal interpolation frame processing includes: taking the previous frame, the current frame, and temporal sampling parameters as inputs to the temporal interpolation frame processing process, outputting interpolated frames, and adding them to the output frame queue.

[0428] In one embodiment of this disclosure, the method further includes: after performing the temporal interpolation frame processing, adding the current frame to the output frame queue.

[0429] In one embodiment of this disclosure, the time-domain sampling parameters include: time-domain resampling factor.

[0430] In one embodiment of this disclosure, performing time-domain interpolation frame processing further includes: if the time-domain sampling rate change flag / time-domain recovery update flag of the current frame is 1, updating the global time-domain recovery parameters to the time-domain recovery parameters of the current frame; if the time-domain sampling rate change flag / time-domain recovery update flag of the current frame is 0, keeping the global time-domain recovery parameters unchanged and the same as the time-domain recovery parameters of the previous frame.

[0431] Typically, temporal restoration operations are performed based on global temporal restoration parameters, which include, but are not limited to, at least one of the following: ① Temporal Restoration Mode; ② Temporal Resampling Ratio; ③ Temporal Extrapolation Resample Num; ④ Temporal Extrapolation Predict Num.

[0432] When there are no temporal restoration parameters in the Parameter Restoration Descriptor (PRD), the global temporal restoration parameters are the same as the sequence-level temporal restoration parameters. Otherwise, the global temporal restoration parameters are the same as the image-level temporal restoration parameters. When the PRD temporal resampling ratio change flag prd_temporal_resampling_ratio_changed_flag is equal to 1, the global temporal restoration parameters are updated using the current temporal restoration parameters in the PRD; otherwise, when the extrapolation frame is equal to 0, the global temporal restoration parameters remain unchanged and are the same as the temporal restoration parameters of the previous frame.

[0433] In one embodiment of this disclosure, after updating the global temporal recovery parameters to the temporal recovery parameters of the current frame, the method further includes: if the temporal recovery interpolation mode of the current frame is temporal extrapolation, then adding the current frame to the reference frame queue.

[0434] In one embodiment of this disclosure, performing temporal extrapolation frame interpolation processing includes: taking a reference frame sequence and the number of temporal extrapolation frame predictions as inputs to the temporal extrapolation frame interpolation process, outputting interpolated frames, and adding them to the output frame sequence.

[0435] In one embodiment of this disclosure, the method further includes: after performing the temporal extrapolation frame processing, adding the current frame to the output frame queue.

[0436] In one embodiment of this disclosure, the decoding method provided in this embodiment may further include the following steps: if the temporal sampling rate change flag bit / temporal recovery update flag bit of the current frame is 1, the global temporal recovery parameter is updated to the temporal recovery parameter of the current frame; if the temporal sampling rate change flag bit / temporal recovery update flag bit of the current frame is 0, the global temporal recovery parameter remains unchanged and is the same as the temporal recovery parameter of the previous frame.

[0437] In one embodiment of this disclosure, after updating the global temporal recovery parameters to the temporal recovery parameters of the current frame, the decoding method provided in this embodiment may further include the following steps: if the temporal recovery and interpolation mode of the current frame is temporal interpolation, then the reference frame queue is set to an empty frame queue; if the temporal recovery and interpolation mode of the current frame is temporal extrapolation, then the current frame is added to the reference frame queue; when the length of the reference frame sequence reaches the temporal extrapolation prediction number, the reference frame sequence and the temporal extrapolation prediction number are used as the input of the temporal extrapolation processing procedure, the interpolated frame is output, added to the output frame sequence, and the reference frame queue is set to an empty frame queue.

[0438] In one embodiment of this disclosure, a tail-predicted input frame queue is determined, and temporal tail recovery processing is performed on the input frame queue based on the tail-predicted input frame queue, including:

[0439] If the temporal recovery frame interpolation mode is temporal interpolation frame interpolation, then the tail prediction input frame queue is set to a queue containing the last frame in the input frame queue.

[0440] If the temporal recovery frame interpolation mode is temporal extrapolation frame interpolation, then the tail prediction input frame queue is set to a queue containing the last N frames in the output frame queue, where N is the number of temporal extrapolation frame prediction frames;

[0441] The tail prediction input frame queue, the remaining number of frames in temporal recovery, and the temporal recovery frame interpolation mode are used as inputs to temporal tail recovery processing, and the output is the tail frame queue;

[0442] If there are frames after the last sampled frame in the output frame queue, delete the frames after the last sampled frame and add the tail frame queue after the last sampled frame.

[0443] In specific implementation, the decoding method provided in this disclosure embodiment may include the following temporal domain recovery processing procedure, the input of which is a list of one or more images picListIn, referred to as the input frame queue or input image list; the output of which is a list of one or more images picListOut, referred to as the output frame queue or output image list.

[0444] (I) Initialization Operation:

[0445] Initialize the output frame queue picListOut to an empty list of images;

[0446] Initialize the global time-domain recovery parameters to the sequence-level time-domain recovery parameters;

[0447] Initialize the reference frame queue (reference frame image list) RefList as an empty list of images.

[0448] (ii) When the ERD RBSP (time domain recovery end condition, which can be a flag indicating the end of time domain recovery processing) has not arrived, perform the following operations:

[0449] For the input frame queue picListIn, perform the following operations sequentially for i from 0 to (picListIn-1):

[0450] 1) When the time domain recovery mode is 0 (time domain interpolation mode), perform the following steps:

[0451] ① When i is not 0, the (i-1)th and i-th frame images in picListIn are assigned to the local frame queue (local image list) picLocalListIn. The temporal sampling parameters (such as temporal sampling rate) and the local frame queue picLocalListIn are used as inputs to the temporal forward and backward frame interpolation process. The output of this process is zero or more images, which are added to the output frame queue picListOut.

[0452] ② The i-th frame image in the input frame queue PicListIn is added to the output frame queue picListOut.

[0453] ③ When the prd_temporal_resampling_ratio_changed_flag (temporal sampling ratio change flag) / prd_temporal_restoration_update_flag (temporal restoration update flag) of the i-th frame image is 1, the global temporal restoration parameters are updated to the temporal restoration parameters of the i-th frame image; otherwise, the global temporal restoration parameters remain unchanged and are the same as the temporal restoration parameters of the previous frame image.

[0454] ④ When the time domain recovery mode is 1, the i-th frame image in PicListIn is added to the reference frame queue (reference frame image list) RefList.

[0455] 2) When the domain recovery mode is 1, perform the following operations:

[0456] ① The i-th frame image in the input frame queue PicListIn is added to the output frame queue picListOut.

[0457] ② When the prd_temporal_resampling_ratio_changed_flag (temporal sampling ratio change flag) / prd_temporal_restoration_update_flag (temporal restoration update flag) of the i-th frame image is 1, the global temporal restoration parameters are updated to the temporal restoration parameters of the i-th frame image; otherwise, the global temporal restoration parameters remain unchanged and are the same as the temporal restoration parameters of the previous frame image.

[0458] ③ When the time domain recovery mode is 0, the reference frame queue RefList is set to empty.

[0459] ④ When the time-domain recovery mode is 1, the i-th frame image in the input frame queue is added to the reference frame queue RefList. If the length of the reference frame queue RefList reaches the temporal extrapolation prediction number TemporalExtrapolationResampleNum, then the reference frame queue RefList and TemporalExtrapolationPredictNum are used as inputs to the temporal extrapolation process. The output of this process is zero or more images. This output is added to picListOut. The reference frame queue RefList is then set to empty.

[0460] (iii) When ERD_RBSP arrives, perform the following operations:

[0461] 1) Set the tail prediction input frame queue (tail prediction image list) tailPredictList as follows:

[0462] 2) When the time domain recovery mode is 0, set the tail prediction input frame queue tailPredictPicList to a list containing the last frame image of picListIn.

[0463] 3) When the time domain recovery mode is 1, set the tail prediction input frame queue tailPredictPicList to a list of images containing the last TemporalExtrapolationPredictNum values ​​in picListOut.

[0464] 4) The tail prediction input frame queue tailPredictPicList, the remaining number of temporal recovery frames erd_num_temporal_remain, and the temporal recovery frame interpolation mode TemporalRestorationMode are used as inputs to the temporal tail processing process to obtain the tail frame list tailPicList as output. tailPicList is then added to picList.

[0465] 5) First, remove the images after the last sampled frame from the output frame queue picListOut (if they exist). Then, add the tail frame queue (tail image list) tailPicList to the output frame queue picListOut after the last sampled frame.

[0466] Figure 20 shows a flowchart of another encoding method according to an embodiment of the present disclosure. As shown in Figure 20, the encoding method provided in this embodiment of the present disclosure may include the following steps:

[0467] S2002, the video is encoded to obtain bitstream data, so that the decoding end can implement the decoding method shown in Figure 19 based on the bitstream data.

[0468] Figure 21 shows a schematic diagram of a decoding device according to an embodiment of the present disclosure. As shown in Figure 21, the decoding device 2100 includes: a frame interpolation mode determination module 2101 and a temporal recovery frame interpolation module 2102.

[0469] The interpolation mode determination module 2101 is configured to determine the temporal recovery interpolation mode; the temporal recovery interpolation module 2102 is configured to perform temporal interpolation and / or temporal extrapolation on the sampled frames in the input frame queue according to the temporal recovery interpolation mode.

[0470] Figure 22 shows a schematic diagram of another encoding device in an embodiment of the present disclosure. As shown in Figure 22, the encoding device 2200 includes a video encoding module 2201, configured to encode video to obtain bitstream data, so that the decoding end can implement any of the above decoding methods based on the bitstream data.

[0471] Figure 23 shows a flowchart of another decoding method according to an embodiment of the present disclosure. As shown in Figure 23, the decoding method provided in this embodiment of the present disclosure may include the following steps:

[0472] S2302, determine the time-domain recovery frame interpolation mode.

[0473] It should be noted that, in the embodiments of this disclosure, the temporal recovery frame interpolation mode is set to indicate the frame interpolation mode used in temporal frame interpolation. It can be the frame interpolation mode of mode 1 (i.e., temporal extrapolation frame interpolation mode, also known as temporal extrapolation frame interpolation mode) or the frame interpolation mode of mode 0 (i.e., temporal interpolation frame interpolation mode, also known as temporal forward and backward frame interpolation mode). The temporal recovery frame interpolation mode in the embodiments of this disclosure can be at the sequence level or at the frame level.

[0474] S2304, according to the time-domain recovery interpolation mode, performs time-domain interpolation and / or time-domain extrapolation on the sampled frames in the input frame queue.

[0475] In one embodiment of this disclosure, before performing temporal interpolation and / or temporal extrapolation on the sampled frames in the input frame queue according to the temporal recovery interpolation mode, the decoding method provided in this embodiment may further include the following steps: performing a temporal recovery initialization operation; when no end-of-stream information is received, performing temporal interpolation or temporal extrapolation processing on each sampled frame in the frame queue according to the temporal recovery interpolation mode; when end-of-stream information is received, determining the tail prediction input frame queue, and performing temporal tail recovery processing on the input frame queue according to the tail prediction input frame queue. In some embodiments, when the temporal recovery interpolation function is enabled, the temporal recovery initialization operation is performed. If the temporal recovery interpolation function is not enabled, the temporal recovery initialization operation is not performed.

[0476] In one embodiment of this disclosure, performing a time-domain recovery initialization operation includes at least one of the following: initializing the output frame queue to an empty frame queue; initializing the global time-domain recovery parameters to sequence-level time-domain recovery parameters; and initializing the reference frame queue to an empty frame queue.

[0477] In one embodiment of this disclosure, for each sampled frame in the frame queue, temporal interpolation or temporal extrapolation is performed according to the temporal recovery interpolation mode, including: if the temporal recovery interpolation mode of the current frame is temporal interpolation and the current frame is not the first frame, then temporal interpolation is performed; if the temporal recovery interpolation mode of the current frame is temporal extrapolation, then the current frame is added to the reference frame queue, and when the length of the reference frame sequence reaches the temporal extrapolation prediction number, temporal extrapolation is performed.

[0478] In one embodiment of this disclosure, the decoding method provided in this embodiment may further include the following steps: if the temporal recovery interpolation mode of the current frame is temporal interpolation and the current frame is the first frame, then only the step of adding the current frame to the output frame queue is performed, and temporal interpolation is not performed.

[0479] In one embodiment of this disclosure, performing temporal interpolation frame processing includes: taking the previous frame, the current frame, and temporal sampling parameters as inputs to the temporal interpolation frame processing process, outputting interpolated frames, and adding them to the output frame queue.

[0480] In one embodiment of this disclosure, the method further includes: after performing the temporal interpolation frame processing, adding the current frame to the output frame queue.

[0481] In one embodiment of this disclosure, the time-domain sampling parameters include: time-domain resampling factor.

[0482] In one embodiment of this disclosure, performing time-domain interpolation frame processing further includes: if the time-domain sampling rate change flag / time-domain recovery update flag of the current frame is 1, updating the global time-domain recovery parameters to the time-domain recovery parameters of the current frame; if the time-domain sampling rate change flag / time-domain recovery update flag of the current frame is 0, keeping the global time-domain recovery parameters unchanged and the same as the time-domain recovery parameters of the previous frame.

[0483] Typically, temporal restoration operations are performed based on global temporal restoration parameters, which include, but are not limited to, at least one of the following: ① Temporal Restoration Mode; ② Temporal Resampling Ratio; ③ Temporal Extrapolation Resample Num; ④ Temporal Extrapolation Predict Num.

[0484] When there are no temporal restoration parameters in the Parameter Restoration Descriptor (PRD), the global temporal restoration parameters are the same as the sequence-level temporal restoration parameters. Otherwise, the global temporal restoration parameters are the same as the image-level temporal restoration parameters. When the PRD temporal resampling ratio change flag prd_temporal_resampling_ratio_changed_flag is equal to 1, the global temporal restoration parameters are updated using the current temporal restoration parameters in the PRD; otherwise, when the extrapolation frame is equal to 0, the global temporal restoration parameters remain unchanged and are the same as the temporal restoration parameters of the previous frame.

[0485] In one embodiment of this disclosure, after updating the global temporal recovery parameters to the temporal recovery parameters of the current frame, the method further includes: if the temporal recovery interpolation mode of the current frame is temporal extrapolation, then adding the current frame to the reference frame queue.

[0486] In one embodiment of this disclosure, performing temporal extrapolation frame interpolation processing includes: taking a reference frame sequence and the number of temporal extrapolation frame predictions as inputs to the temporal extrapolation frame interpolation process, outputting interpolated frames, and adding them to the output frame sequence.

[0487] In one embodiment of this disclosure, the method further includes: after performing the temporal extrapolation frame processing, adding the current frame to the output frame queue.

[0488] In one embodiment of this disclosure, the decoding method provided in this embodiment may further include the following steps: if the temporal sampling rate change flag bit / temporal recovery update flag bit of the current frame is 1, the global temporal recovery parameter is updated to the temporal recovery parameter of the current frame; if the temporal sampling rate change flag bit / temporal recovery update flag bit of the current frame is 0, the global temporal recovery parameter remains unchanged and is the same as the temporal recovery parameter of the previous frame.

[0489] In one embodiment of this disclosure, after updating the global temporal recovery parameters to the temporal recovery parameters of the current frame, the decoding method provided in this embodiment may further include the following steps: if the temporal recovery and interpolation mode of the current frame is temporal interpolation, then the reference frame queue is set to an empty frame queue; if the temporal recovery and interpolation mode of the current frame is temporal extrapolation, then the current frame is added to the reference frame queue; when the length of the reference frame sequence reaches the temporal extrapolation prediction number, the reference frame sequence and the temporal extrapolation prediction number are used as the input of the temporal extrapolation processing procedure, the interpolated frame is output, added to the output frame sequence, and the reference frame queue is set to an empty frame queue.

[0490] In one embodiment of this disclosure, a tail-predicted input frame queue is determined, and temporal tail recovery processing is performed on the input frame queue based on the tail-predicted input frame queue, including:

[0491] If the temporal recovery frame interpolation mode is temporal interpolation frame interpolation, then the tail prediction input frame queue is set to a queue containing the last frame in the input frame queue.

[0492] If the temporal recovery frame interpolation mode is temporal extrapolation frame interpolation, then the tail prediction input frame queue is set to a queue containing the last N frames in the output frame queue, where N is the number of temporal extrapolation frame prediction frames;

[0493] The tail prediction input frame queue, the remaining number of frames in temporal recovery, and the temporal recovery frame interpolation mode are used as inputs to temporal tail recovery processing, and the output is the tail frame queue;

[0494] If there are frames after the last sampled frame in the output frame queue, delete the frames after the last sampled frame and add the tail frame queue after the last sampled frame.

[0495] In specific implementation, the decoding method provided in this disclosure embodiment may include the following temporal domain recovery processing procedure, the input of which is a list of one or more images picListIn, referred to as the input frame queue or input image list; the output of which is a list of one or more images picListOut, referred to as the output frame queue or output image list.

[0496] (I) Initialization Operation:

[0497] Initialize the output frame queue picListOut to an empty list of images;

[0498] Initialize the global time-domain recovery parameters to the sequence-level time-domain recovery parameters;

[0499] Initialize the reference frame queue (reference frame image list) RefList as an empty list of images.

[0500] (ii) When the ERD RBSP (time domain recovery end condition, which can be a flag indicating the end of time domain recovery processing) has not arrived, perform the following operations:

[0501] For the input frame queue picListIn, perform the following operations sequentially for i from 0 to (picListIn-1):

[0502] 1) When the time domain recovery mode is 0 (time domain interpolation mode), perform the following steps:

[0503] ① When i is not 0, the (i-1)th and i-th frame images in picListIn are assigned to the local frame queue (local image list) picLocalListIn. The temporal sampling parameters (such as temporal sampling rate) and the local frame queue picLocalListIn are used as inputs to the temporal forward and backward frame interpolation process. The output of this process is zero or more images, which are added to the output frame queue picListOut.

[0504] ② The i-th frame image in the input frame queue PicListIn is added to the output frame queue picListOut.

[0505] ③ When the prd_temporal_resampling_ratio_changed_flag (temporal sampling ratio change flag) / prd_temporal_restoration_update_flag (temporal restoration update flag) of the i-th frame image is 1, the global temporal restoration parameters are updated to the temporal restoration parameters of the i-th frame image; otherwise, the global temporal restoration parameters remain unchanged and are the same as the temporal restoration parameters of the previous frame image.

[0506] ④ When the time domain recovery mode is 1, the i-th frame image in PicListIn is added to the reference frame queue (reference frame image list) RefList.

[0507] 2) When the domain recovery mode is 1, perform the following operations:

[0508] ① The i-th frame image in the input frame queue PicListIn is added to the output frame queue picListOut.

[0509] ② When the prd_temporal_resampling_ratio_changed_flag (temporal sampling ratio change flag) / prd_temporal_restoration_update_flag (temporal restoration update flag) of the i-th frame image is 1, the global temporal restoration parameters are updated to the temporal restoration parameters of the i-th frame image; otherwise, the global temporal restoration parameters remain unchanged and are the same as the temporal restoration parameters of the previous frame image.

[0510] ③ When the time domain recovery mode is 0, the reference frame queue RefList is set to empty.

[0511] ④ When the time-domain recovery mode is 1, the i-th frame image in the input frame queue is added to the reference frame queue RefList. If the length of the reference frame queue RefList reaches the temporal extrapolation prediction number TemporalExtrapolationResampleNum, then the reference frame queue RefList and TemporalExtrapolationPredictNum are used as inputs to the temporal extrapolation process. The output of this process is zero or more images. This output is added to picListOut. The reference frame queue RefList is then set to empty.

[0512] (iii) When ERD_RBSP arrives, perform the following operations:

[0513] 1) Set the tail prediction input frame queue (tail prediction image list) tailPredictList as follows:

[0514] 2) When the time domain recovery mode is 0, set the tail prediction input frame queue tailPredictPicList to a list containing the last frame image of picListIn.

[0515] 3) When the time domain recovery mode is 1, set the tail prediction input frame queue tailPredictPicList to a list of images containing the last TemporalExtrapolationPredictNum values ​​in picListOut.

[0516] 4) The tail prediction input frame queue tailPredictPicList, the remaining number of temporal recovery frames erd_num_temporal_remain, and the temporal recovery frame interpolation mode TemporalRestorationMode are used as inputs to the temporal tail processing process to obtain the tail frame list tailPicList as output. tailPicList is then added to picList.

[0517] 5) First, remove the images after the last sampled frame from the output frame queue picListOut (if they exist). Then, add the tail frame queue (tail image list) tailPicList to the output frame queue picListOut after the last sampled frame.

[0518] Figure 24 shows a flowchart of another encoding method in an embodiment of the present disclosure. As shown in Figure 24, the encoding method provided in this embodiment may include the following steps:

[0519] S2402, the video is encoded to obtain bitstream data, so that the decoding end can implement the decoding method shown in Figure 23 based on the bitstream data.

[0520] Figure 25 shows a schematic diagram of a decoding device according to an embodiment of the present disclosure. As shown in Figure 25, the decoding device 2500 includes: a frame interpolation mode determination module 2501 and a temporal recovery frame interpolation module 2502.

[0521] The interpolation mode determination module 2501 is configured to determine the temporal recovery interpolation mode; the temporal recovery interpolation module 2502 is configured to perform temporal interpolation and / or temporal extrapolation on the sampled frames in the input frame queue according to the temporal recovery interpolation mode.

[0522] Figure 26 shows a schematic diagram of another encoding device in an embodiment of the present disclosure. As shown in Figure 26, the encoding device 2600 includes a video encoding module 2601, configured to encode video to obtain bitstream data, so that the decoding end can implement the decoding method of any of the above-mentioned methods based on the bitstream data.

[0523] Figure 27 shows a flowchart of another decoding method according to an embodiment of the present disclosure. As shown in Figure 27, the decoding method includes the following steps:

[0524] S2702, Perform format conversion on the input frame queue of the first format to obtain the local frame queue of the second format;

[0525] S2704, Perform time-domain frame interpolation on the second-format frame queue to generate the second-format output frame queue;

[0526] S2706, perform format conversion on the second format output frame queue to obtain the first format output frame queue.

[0527] It should be noted that in the embodiments of this disclosure, the first format and the second format can be any image format. In some embodiments, they can be color formats, such as YUV format, RGB format, etc. In the embodiments of this disclosure, the first format and the second format are not limited.

[0528] In one embodiment of this disclosure, before performing temporal frame interpolation on the second-format frame queue to generate the second-format output frame queue, the method further includes: performing at least one of the following preprocessing on each frame image in the local frame queue: padding, normalization, and brightness enhancement; before performing format conversion on the second-format output frame queue to obtain the first-format output frame queue, the method further includes: performing at least one of the following postprocessing on the second-format output frame queue: brightness restoration, denormalization, and depadding.

[0529] In one embodiment of this disclosure, the first format is YUV format; the second format is RGB format. In some embodiments, the first and second formats can be interchanged.

[0530] In one embodiment of this disclosure, the YUV format is YCbCr420; the RGB format is RGB24.

[0531] In one embodiment of this disclosure, the conversion process from YCbCr420 format to RGB24 format is implemented through the following steps:

[0532] Input: An image picIn[3,hIn,wIn] in YcbCr 4:2:0 chroma format and a variable bitDepthOut set to specify the output bit depth for all color planes;

[0533] Output: An RGB24 4:4:4 chroma image picOut[3,hIn,wIn];

[0534] The conversion process is as follows:

[0535] 1) Set variables hOut = Ceil(hIn / 32) * 32, wOut = Ceil(wIn / 32) * 32;

[0536] 2) Perform the following steps for i in the range of 0 to hIn–1:

[0537] Perform the following steps for j in the range of 0 to wIn–1:

[0538] y[i,j]=Clip3(0,255,picInY[i,j]>>(bitDepthLocal[0]–8)-16);

[0539] u[i,j]=picInU[i / 2,j / 2]>>(bitDepthLocal[1]–8)-128;

[0540] v[i,j]=picInV[i / 2,j / 2]>>(bitDepthLocal[2]–8)-128;

[0541] pictOut[0,i,j]=Clip3(0,255,(1220542*y[i,j]+1673527*v[i,j]+(1<<19)) / / (1<<20));

[0542] pictOut[1,i,j]=Clip3(0,255,(1220542*y[i,j]-852492*v[i,j]-409993*u[i,j]+(1<<19)) / / (1<<20));

[0543] pictOut[2,i,j]=Clip3(0,255,(1220542*y[i,j]+2116026*u[i,j]+(1<<19)) / / (1<<20));

[0544] Perform the following steps on j within the range wIn to wOut-1:

[0545] picOut[0,i,j]=picOut[0,i,wIn–1];

[0546] picOut[1,i,j]=picOut[1,i,wIn–1];

[0547] picOut[2,i,j]=picOut[2,i,wIn–1];

[0548] 3) Perform the following steps for i within the range hIn to hOut-1:

[0549] Perform the following steps for j in the range of 0 to wIn-1:

[0550] picOut[0,i,j]=picOut[0,hIn–1,j];

[0551] picOut[1,i,j]=picOut[1,hIn–1,j];

[0552] picOut[2,i,j]=picOut[2,hIn–1,j];

[0553] Perform the following steps on j within the range wIn to wOut-1:

[0554] picOut[0,i,j]=picOut[0,hIn–1,wIn–1];

[0555] picOut[1,i,j]=picOut[1,hIn–1,wIn–1];

[0556] picOut[2,i,j]=picOut[2,hIn–1,wIn–1];

[0557] When bitDepthOut is not specified, the variable bitDepthLocal[3] is set to BitDepth; otherwise, each value in bitDepthLocal[3] is assigned the value bitDepthOut.

[0558] In one embodiment of this disclosure, the conversion process from RGB24 format to YCbCr420 is implemented through the following steps:

[0559] Input: An RGB24 4:4:4 chroma format image picOut[3,hIn,wIn] and a variable bitDepthOut set to specify the output bit depth for all color planes;

[0560] Output: A YcbCr 4:2:0 chroma format image picIn[3,hIn,wIn];

[0561] The conversion process is as follows:

[0562] For k in the range 0k 2, i in the range 0i hIn–1, and j in the range 0j wIn–1, perform the following steps:

[0563] 1) Set p[k,i,j] = Clip3(0,255,(pictIn[k,i,j]*256));

[0564] 2) Set the variable scale[3] = 2bitDepthLocal[cIdx] – 8, where 0 = 2 for cIdx;

[0565] 3) For i in the range 0i hOut / 2–1, and j in the range 0j wOut / 2–1, perform the following steps:

[0566] picOut[0,2*i,2*j]=Clip3(0,bitDepthLocal[0],(269484*p[0,2*i,2*j]+52848 2*p[1,2*i,2*j]+102760*p[2,i,j]+(1<<19)+(16<<20)) / / (1<<20))*scale[0]);

[0567] picOut[0,2*i,2*j+1]=Clip3(0,bitDepthLocal[0],(269484*p[0,2*i,2*j+1]+52848 2*p[1,2*i,2*j+1]+102760*p[2,i,j+1]+(1<<19)+(16<<20)) / / (1<<20))*scale[0]);

[0568] picOut[0,2*i+1,2*j]=Clip3(0,bitDepthLocal[0],(269484*p[0,2*i+1,2*j]+52848 2*p[1,2*i+1,2*j]+102760*p[2,i+1,j]+(1<<19)+(16<<20)) / / (1<<20))*scale[0]);

[0569] picOut[0,2*i+1,2*j+1]=Clip3(0,bitDepthLocal[0],(269484*p[0,2*i+1,2*j+1]+52848 2*p[1,2*i+1,2*j+1]+102760*p[2,i+1,j+1]+(1<<19)+(16<<20)) / / (1<<20))*scale[0]);

[0570] picOut[1,i,j]=Clip3(0,bitDepthLocal[1],(–155188*p[0,i,j]–305135*p[1,i,j]+460324*p[2,i,j])*scale[1]);

[0571] picOut[2,i,j]=Clip3(0,bitDepthLocal[2],(460324*p[0,i,j]–385875*p[1,i,j]–74448*p[2,i,j])*scale[2]);

[0572] When bitDepthOut is not specified, the variable bitDepthLocal[3] is set to BitDepth; otherwise, each value in bitDepthLocal[3] is assigned the value bitDepthOut.

[0573] In one embodiment of this disclosure, the brightness enhancement process is implemented through the following steps:

[0574] Input: An RGB24 formatted image picOut[3,hIn,wIn];

[0575] Output: An RGB24 formatted image picOut[3,hIn,wIn] and the variable lumaEnhancedFlag;

[0576] The brightness enhancement process is as follows:

[0577] 1) Initialize lumaEnhancedFlag to 0;

[0578] 2) Perform the following format conversion on picIn to obtain the variable picTmp:

[0579] For i in the range 0i hIn–1 and j in the range 0j wIn–1, perform the following steps:

[0580] picTmp[0,i,j]=(picIn[0,i,j]*4899+picIn[1,i,j]*9617+picIn[2,i,j]*1868+(1<<(14-1)))>>14

[0581] picTmp[1,i,j]=Clip3(0,255,((picIn[2,i,j]-picTmp[0,i,j])*1868+128<<14+(1<<(14-1)))>>14)

[0582] picTmp[2,i,j]=Clip3(0,255,((picIn[0,i,j]-picTmp[0,i,j])*14369+128<<14+(1<<(14-1)))>>14)

[0583] picTmp[0,i,j]=Clip3(0,255,picTmp[0,i,j])

[0584] Wherein, picIn[0] represents the R channel, picIn[1] represents the G channel, picIn[2] represents the B channel; picTmp[0] represents the Y channel; picTmp[1] represents the U channel; picTmp[2] represents the V channel;

[0585] 3) When the mean of picTmp[0,hOut,wOut] is less than 50, picTmp[0] is shifted left by 1 bit, and lumaEnhancedFlag is set to 1;

[0586] 4) Perform the following format conversion on picTmp to obtain the output picOut:

[0587] For i in the range 0i hIn–1 and j in the range 0j wIn–1, perform the following steps:

[0588] picOut[0,i,j]=picTmp[0,i,j]+(((picTmp[1,i,j]-128)*18678+1<<(14-1))>>14);

[0589] picOut[1,i,j]=picTmp[0,i,j]+(((picTmp[1,i,j]-128)*(-6472)+(picTmp[2,i,j]-128)*(-9519)+1<<(14-1))>>14);

[0590] picOut[2,i,j]=picTmp[0,i,j]+(((picTmp[1,i,j]-128)*33292+1<<(14-1))>>14).

[0591] In one embodiment of this disclosure, the brightness restoration process is achieved through the following steps:

[0592] Input: An RGB24 formatted image picOut[3,hIn,wIn] and a variable lumaEnhancedFlag;

[0593] Output: An RGB24 formatted image picOut[3,hIn,wIn];

[0594] The brightness recovery process is as follows:

[0595] When lumaEnhancedFlag equals 1, perform the following steps:

[0596] 1) Perform the following format conversion on picIn to obtain the variable picTmp:

[0597] 2) For i in the range 0i hIn–1 and j in the range 0j wIn–1, perform the following steps:

[0598] picTmp[0,i,j]=(picIn[0,i,j]*4899+picIn[1,i,j]*9617+picIn[2,i,j]*1868+(1<<(14-1)))>>14;

[0599] picTmp[1,i,j]=Clip3(0,255,((picIn[2,i,j]-picTmp[0,i,j])*1868+128<<14+(1<<(14-1)))>>14);

[0600] picTmp[2,i,j]=Clip3(0,255,((picIn[0,i,j]-picTmp[0,i,j])*14369+128<<14+(1<<(14-1)))>>14);

[0601] picTmp[0,i,j]=Clip3(0,255,picTmp[0,i,j])>>1;

[0602] 3) Perform the following format conversion on picTmp to obtain the output picOut:

[0603] For i in the range 0i hIn–1 and j in the range 0j wIn–1, perform the following steps:

[0604] picOut[0,i,j]=picTmp[0,i,j]+(((picTmp[1,i,j]-128)*18678+1<<(14-1))>>14);

[0605] picOut[1,i,j]=picTmp[0,i,j]+(((picTmp[1,i,j]-128)*(-6472)+(picTmp[2,i,j]-128)*(-9519)+1<<(14-1))>>14);

[0606] picOut[2,i,j]=picTmp[0,i,j]+(((picTmp[1,i,j]-128)*33292+1<<(14-1))>>14).

[0607] In one embodiment of this disclosure, temporal frame interpolation is performed through the following steps:

[0608] Determine the temporal recovery frame interpolation mode; based on the temporal recovery frame interpolation mode, perform temporal interpolation and / or temporal extrapolation on the sampled frames in the input frame queue.

[0609] In one embodiment of this disclosure, before performing temporal interpolation and / or temporal extrapolation on the sampled frames in the input frame queue according to the temporal recovery interpolation mode, the method further includes: performing a temporal recovery initialization operation; when no bitstream end information is received, performing temporal interpolation or temporal extrapolation processing on each sampled frame in the frame queue according to the temporal recovery interpolation mode; when bitstream end information is received, determining the tail prediction input frame queue, and performing temporal tail recovery processing on the input frame queue according to the tail prediction input frame queue.

[0610] In one embodiment of this disclosure, performing a time-domain recovery initialization operation includes at least one of the following: initializing the output frame queue to an empty frame queue; initializing the global time-domain recovery parameters to sequence-level time-domain recovery parameters; and initializing the reference frame queue to an empty frame queue.

[0611] In one embodiment of this disclosure, for each sampled frame in the frame queue, temporal interpolation or temporal extrapolation is performed according to the temporal recovery interpolation mode, including: if the temporal recovery interpolation mode of the current frame is temporal interpolation and the current frame is not the first frame, then temporal interpolation is performed; if the temporal recovery interpolation mode of the current frame is temporal extrapolation, then the current frame is added to the reference frame queue, and when the length of the reference frame sequence reaches the temporal extrapolation prediction number, temporal extrapolation is performed.

[0612] In one embodiment of this disclosure, the method further includes: if the temporal recovery frame interpolation mode of the current frame is temporal interpolation and the current frame is the first frame, then only the step of adding the current frame to the output frame queue is performed, and temporal interpolation is not performed.

[0613] In one embodiment of this disclosure, performing temporal interpolation frame processing includes: taking the previous frame, the current frame, and temporal sampling parameters as inputs to the temporal interpolation frame processing process, outputting interpolated frames, and adding them to the output frame queue.

[0614] In one embodiment of this disclosure, the method further includes: after performing temporal interpolation frame processing, adding the current frame to the output frame queue.

[0615] In one embodiment of this disclosure, the time-domain sampling parameters include: time-domain resampling factor.

[0616] In one embodiment of this disclosure, performing time-domain interpolation frame processing further includes: if the time-domain sampling rate change flag / time-domain recovery update flag of the current frame is 1, updating the global time-domain recovery parameters to the time-domain recovery parameters of the current frame; if the time-domain sampling rate change flag / time-domain recovery update flag of the current frame is 0, keeping the global time-domain recovery parameters unchanged and the same as the time-domain recovery parameters of the previous frame.

[0617] In one embodiment of this disclosure, after updating the global temporal recovery parameters to the temporal recovery parameters of the current frame, the method further includes: if the temporal recovery interpolation mode of the current frame is temporal extrapolation, then adding the current frame to the reference frame queue.

[0618] In one embodiment of this disclosure, performing temporal extrapolation frame interpolation processing includes: taking a reference frame sequence and the number of temporal extrapolation frame predictions as inputs to the temporal extrapolation frame interpolation process, outputting interpolated frames, and adding them to the output frame sequence.

[0619] In one embodiment of this disclosure, the method further includes: after performing the temporal extrapolation frame processing, adding the current frame to the output frame queue.

[0620] In one embodiment of this disclosure, the method further includes: if the temporal sampling rate change flag / temporal recovery update flag of the current frame is 1, updating the global temporal recovery parameters to the temporal recovery parameters of the current frame; if the temporal sampling rate change flag / temporal recovery update flag of the current frame is 0, the global temporal recovery parameters remain unchanged and are the same as the temporal recovery parameters of the previous frame.

[0621] In one embodiment of this disclosure, after updating the global temporal recovery parameters to the temporal recovery parameters of the current frame, the method further includes: if the temporal recovery frame interpolation mode of the current frame is temporal interpolation, then setting the reference frame queue to an empty frame queue; if the temporal recovery frame interpolation mode of the current frame is temporal extrapolation, then adding the current frame to the reference frame queue; when the length of the reference frame sequence reaches the temporal extrapolation prediction number, using the reference frame sequence and the temporal extrapolation prediction number as input to the temporal extrapolation processing procedure, outputting the interpolated frame, adding it to the output frame sequence, and setting the reference frame queue to an empty frame queue.

[0622] In one embodiment of this disclosure, a tail prediction input frame queue is determined, and temporal tail recovery processing is performed on the input frame queue according to the tail prediction input frame queue, including: if the temporal recovery frame interpolation mode is temporal interpolation, then the tail prediction input frame queue is set as a queue containing the last frame in the input frame queue; if the temporal recovery frame interpolation mode is temporal extrapolation, then the tail prediction input frame queue is set as a queue containing the last N frames in the output frame queue, where N is the number of temporal extrapolation prediction frames; the tail prediction input frame queue, the remaining number of temporal recovery frames, and the temporal recovery frame interpolation mode are used as inputs to the temporal tail recovery processing, and a tail frame queue is output; if there are frames after the last sampled frame in the output frame queue, then the frames after the last sampled frame are deleted, and the tail frame queue is added after the last sampled frame.

[0623] Figure 28 shows a flowchart of another encoding method according to an embodiment of the present disclosure. As shown in Figure 24, the encoding method includes:

[0624] S2802, the video is encoded to obtain bitstream data, so that the decoding end can implement the decoding method shown in Figure 27 based on the bitstream data.

[0625] Figure 29 shows a schematic diagram of another decoding device in an embodiment of the present disclosure. As shown in Figure 29, the decoding device 2900 includes: a first format conversion module 2901, configured to perform format conversion on an input frame queue of a first format to obtain a local frame queue of a second format; a time-domain frame interpolation module 2902, configured to perform time-domain frame interpolation on the frame queue of the second format to generate an output frame queue of the second format; and a second format conversion module 2903, configured to perform format conversion on the output frame queue of the second format to obtain an output frame queue of a first format.

[0626] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0627] The electronic device 3000 according to this embodiment of the present disclosure will now be described with reference to FIG30. The electronic device 3000 shown in FIG30 is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present disclosure.

[0628] As shown in Figure 30, the electronic device 3000 is presented in the form of a general-purpose computing device. The components of the electronic device 3000 may include, but are not limited to: at least one processing unit 3010, at least one storage unit 3030, and a bus 3030 connecting different system components (including storage unit 3030 and processing unit 3010).

[0629] The storage unit stores program code, which can be executed by the processing unit 3010, causing the processing unit 3010 to perform the steps described in the "Exemplary Methods" section above according to various exemplary embodiments of this disclosure.

[0630] In some embodiments, the processing unit 3010 may perform the following steps of the above method embodiment: determining whether the current sampling frame is the first sampling frame; and if the current sampling frame is not the first sampling frame, determining the temporal sampling information corresponding to the first sampling frame.

[0631] In some embodiments, the processing unit 3010 may perform the following steps of the above method embodiment: obtain the temporal information of the sampling image corresponding to the sampling frame; determine whether the temporal sampling flag is set based on the temporal information of the sampling image.

[0632] In some embodiments, the processing unit 3010 may perform the following steps of the above method embodiment: perform frame interpolation on the current frame based on the target frames preceding the current frame.

[0633] In some embodiments, the processing unit 3010 may perform the following steps of the above method embodiment: encoding the video to obtain bitstream data.

[0634] In some embodiments, the processing unit 3010 may perform the following steps of the above method embodiment: determining a temporal recovery frame interpolation mode; and performing temporal interpolation and / or temporal extrapolation on the sampled frames in the input frame queue according to the temporal recovery frame interpolation mode.

[0635] Storage unit 3030 may include readable media in the form of volatile storage units, such as random access memory (RAM) 30301 and / or cache memory 30302, and may further include read-only memory (ROM) 30303.

[0636] Storage unit 3030 may also include a program / utility 30304 having a set (at least one) of program modules 30305, such program modules 30305 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0637] Bus 3030 can represent one or more of several types of bus structures, including memory cell bus or memory cell controller, peripheral bus, graphics acceleration port, processing unit, or local bus using any of the multiple bus structures.

[0638] Electronic device 3000 can also perform image processing with one or more external devices 3040 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 3000, and / or any device that enables electronic device 3000 to perform image processing with one or more other computing devices (e.g., router, modem, etc.). This image processing can be performed via input / output (I / O) interface 3050. Furthermore, electronic device 3000 can also perform image processing with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 3060. As shown, network adapter 3060 performs image processing with other modules of electronic device 3000 via bus 3030. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 3000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0639] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0640] In particular, according to embodiments of this disclosure, the process described above with reference to the flowchart can be implemented as a computer program product, which includes a computer program that, when executed by a processor, implements the above-described decoding or encoding method.

[0641] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. The computer-readable storage medium stores a program product capable of implementing the decoding or encoding methods described above. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code, which, when run on a terminal device, is configured to cause the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0642] More specific examples of computer-readable storage media in this disclosure may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0643] In this disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program configured for use by or in conjunction with an instruction execution system, apparatus, or device.

[0644] In some embodiments, program code contained on a computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0645] In practical implementation, program code configured to perform the operations of this disclosure can be written using any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0646] It should be noted that although several modules or units of the device configured to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0647] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0648] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0649] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A decoding method. Applied to the decoding end, comprising: Determine whether the current sampled frame is the first sampled frame; If the current sampling frame is not the first sampling frame, determine the temporal sampling information corresponding to the first sampling frame.

2. The decoding method according to claim 1, wherein, The temporal sampling information corresponding to the first sampling frame includes a temporal sampling rate hold bit; The method further includes: While maintaining the position of the time-domain sampling rate, update the global time-domain frame extraction ratio.

3. The decoding method according to claim 1, wherein, The method further includes: Save the first sampled frame.

4. The decoding method according to claim 3, wherein, Saving the first sample frame includes: Configure the first sampled frame in the output queue; Save the output queue.

5. The decoding method according to claim 1, wherein, The method further includes: Determine if the global temporal frame sampling ratio is 1; When the global temporal frame sampling ratio is 1, the frame buffer information corresponding to the current sampled frame is determined to be the frame buffer information corresponding to the first sampled frame.

6. The decoding method according to claim 1, wherein, The method further includes: Determine the number of interpolation frames between the current sampling frame and the first sampling frame.

7. The decoding method according to claim 1, wherein, Determining the number of interpolation frames between the current sampling frame and the first sampling frame includes: The number of supplementary frames between the current sampling frame and the first sampling frame is determined based on the temporal frame extraction ratio of the first sampling frame, wherein the temporal frame extraction ratio of the first sampling frame can be determined by the global temporal frame extraction ratio.

8. The decoding method according to claim 7, wherein, The method further includes: Frames are interpolated based on the number of interpolated frames, the current sampled frame, the first sampled frame, and a preset decoding method.

9. The decoding method according to claim 8, wherein, The method further includes: The first sampled frame, the current sampled frame, and the frame obtained by the supplementary frame are saved in sequence.

10. The decoding method according to claim 9, wherein, The step of saving the first sampled frame, the current sampled frame, and the frame obtained by the supplementary frame in sequence includes: The first sampled frame, the current sampled frame, and the frame obtained by the supplementary frame are arranged in the output queue in sequence order; Save the output queue.

11. The decoding method according to claim 1, wherein, The method further includes: When the current sampled frame is the first sampled frame, configure the frame buffer information of the current sampled frame.

12. The decoding method according to claim 1, wherein, Determining whether the current sampling frame is the first sampling frame includes: Obtain the current sampling frame; If the current sampling frame is obtained, determine whether the current sampling frame is the first sampling frame.

13. The decoding method according to claim 1, wherein, The method further includes: If the current sampled frame is not obtained, the remaining frames are saved.

14. The decoding method according to claim 1, wherein, The method further includes: Upon receiving the end information and obtaining the current sampling frame, save the current sampling frame and the remaining frames.

15. A decoding method, applied at a decoding end, comprising: Obtain the temporal information of the sampled image corresponding to the sampled frame; Determine whether the time-domain sampling flag is set based on the time-domain information of the sampled image.

16. The decoding method according to claim 15, wherein, The method further includes: If the time-domain sampling flag is not set, it is determined that the time-domain recovery function of the current sampling frame is disabled.

17. The decoding method according to claim 15, wherein, The method further includes: If the time-domain sampling flag is not set, frame interpolation based on the historical time-domain sampling rate is determined.

18. The decoding method according to claim 15, wherein, The method further includes: While maintaining the time-domain sampling rate, obtain the sampling rate corresponding to the sampling frame.

19. A decoding method, applied at a decoding end, comprising: The current frame is padded with frames based on the target frames preceding it.

20. The decoding method according to claim 19, wherein, The process of interpolating the current frame based on previous frames includes: The target frame is copied to obtain the current frame, and the target frame includes the reference frame.

21. The decoding method according to claim 19, wherein, The step of copying the target frame to obtain the current frame, wherein the target frame includes a reference frame, includes: If the number of target frames preceding the current frame does not meet a preset number, the target frame is copied to obtain the current frame, wherein the target frame includes a reference frame.

22. The decoding method according to claim 19, wherein, The process of interpolating the current frame based on previous frames includes: The target frame is copied to obtain the current frame, and the target frame includes the frame obtained by piecing together the original frame.

23. The decoding method according to claim 19, wherein, The process of interpolating the current frame based on previous frames includes: The current frame is obtained by interpolating multiple target frames, where the target frames include reference frames.

24. The decoding method according to claim 19, wherein, The process of interpolating the current frame based on previous frames includes: The current frame is obtained by interpolating multiple target frames. The target frames include the reference frame and the interpolated frame.

25. The decoding method according to claim 19, wherein, The current frame includes the reference frame.

26. The decoding method according to claim 19, wherein, The method further includes: Determine whether the number of supplementary frames after the last sampled frame is the same as the number of remaining frames.

27. The decoding method according to claim 26, wherein, The method further includes: If the number of supplementary frames after the last sampled frame is different from the number of remaining frames, the number of supplementary frames after the last sampled frame will be modified to be the same as the number of remaining frames.

28. The decoding method according to any one of claims 19 to 27, wherein, The reference frame, the current frame, and the frame obtained by the interpolation are determined as the interpolation segment.

29. An encoding method, applied at an encoding end, comprising: The video is encoded to obtain bitstream data, so that the decoding end can implement the decoding method as described in any one of claims 1 to 28 based on the bitstream data.

30. A decoding device, applied at a decoding end, comprising: The first determining module is used to determine whether the current sampling frame is the first sampling frame; The second determining module is used to determine the temporal sampling information corresponding to the first sampling frame when the current sampling frame is not the first sampling frame.

31. A decoding device, applied at a decoding end, comprising: The first acquisition module is used to acquire the temporal information of the sampled image corresponding to the sampled frame; The third determining module is used to determine whether the time-domain sampling flag bit is set based on the time-domain information of the sampled image.

32. A decoding device, applied at a decoding end, comprising: The frame interpolation module is used to interpolate the current frame based on the target frames preceding the current frame.

33. An encoding device, applied at an encoding end, comprising: The encoding module is used to encode the video to obtain bitstream data, so that the decoding end can implement any decoding method based on the bitstream data.

34. A decoding method, comprising: Determine the temporal recovery frame interpolation mode; According to the time-domain recovery interpolation mode, the sampled frames in the input frame queue are interpolated in the time domain and / or interpolated in the time domain.

35. The decoding method according to claim 34, wherein, Before performing temporal interpolation and / or temporal extrapolation on the sampled frames in the input frame queue according to the temporal recovery interpolation mode, the method further includes: Perform time-domain recovery initialization operations; When no end-of-stream information is received, for each sampled frame in the frame queue, perform time-domain interpolation or time-domain extrapolation processing according to the time-domain recovery interpolation mode. When the end-of-stream information is received, the tail prediction input frame queue is determined, and the time-domain tail recovery processing is performed on the input frame queue based on the tail prediction input frame queue and the remaining number of time-domain recovered frames.

36. The decoding method according to claim 35, wherein, Perform time-domain recovery initialization operations, including at least one of the following: Initialize the output frame queue to an empty frame queue; Initialize the global time-domain recovery parameters to the sequence-level time-domain recovery parameters; Initialize the reference frame queue to an empty frame queue.

37. The decoding method according to claim 35, wherein, For each sampled frame in the frame queue, based on the temporal recovery interpolation mode, temporal interpolation or temporal extrapolation is performed, including: If the temporal recovery frame interpolation mode of the current frame is temporal interpolation and the current frame is not the first frame, then temporal interpolation frame processing is performed. If the temporal recovery interpolation mode of the current frame is temporal extrapolation, then the current frame is added to the reference frame queue. When the length of the reference frame sequence reaches the number of temporal extrapolation predictions, temporal extrapolation processing is performed.

38. The decoding method according to claim 37, wherein, The method further includes: If the temporal recovery frame interpolation mode of the current frame is temporal interpolation and the current frame is the first frame, then only the step of adding the current frame to the output frame queue is executed, and the temporal interpolation frame processing is not performed.

39. The decoding method according to claim 37, wherein, Perform temporal interpolation frame processing, including: The previous frame, the current frame, and the temporal sampling parameters are used as inputs to the temporal interpolation frame processing, and the interpolated frame is output and added to the output frame queue.

40. The decoding method according to claim 37, wherein, The method further includes: After adding the interpolated frames output from the temporal interpolation frame processing to the output queue, the current frame is added to the output frame queue.

41. The decoding method according to claim 39, wherein, The time-domain sampling parameters include: time-domain sampling rate.

42. The decoding method according to claim 39, wherein, Performing temporal interpolation frame processing also includes: If the temporal sampling rate change flag / temporal recovery update flag of the current frame is 1, the global temporal recovery parameters will be updated to the temporal recovery parameters of the current frame. If the temporal sampling rate change flag / temporal recovery update flag of the current frame is 0, the global temporal recovery parameters remain unchanged and are the same as the temporal recovery parameters of the previous frame.

43. The decoding method according to claim 42, wherein, After updating the global temporal recovery parameters to the temporal recovery parameters of the current frame, the method further includes: If the temporal recovery interpolation mode of the current frame is temporal extrapolation, then the current frame is added to the reference frame queue.

44. The decoding method according to claim 37, wherein, Perform temporal extrapolation frame interpolation, including: The reference frame sequence and the number of temporal extrapolation frame predictions are used as inputs to the temporal extrapolation frame processing procedure, and the output extrapolated frames are added to the output frame sequence.

45. The decoding method according to claim 37, wherein, The method further includes: Before adding the interpolated frames output from the temporal extrapolation frame processing to the output frame queue, add the current frame to the output frame queue.

46. ​​The decoding method according to claim 44, wherein, Performing temporal extrapolation frame interpolation also includes: If the temporal sampling rate change flag / temporal recovery update flag of the current frame is 1, the global temporal recovery parameters will be updated to the temporal recovery parameters of the current frame. If the temporal sampling rate change flag / temporal recovery update flag of the current frame is 0, the global temporal recovery parameters remain unchanged and are the same as the temporal recovery parameters of the previous frame.

47. The decoding method according to claim 46, wherein, After updating the global temporal recovery parameters to the temporal recovery parameters of the current frame, the method further includes: If the temporal recovery frame interpolation mode of the current frame is temporal interpolation frame interpolation, then the reference frame queue is set to an empty frame queue. If the temporal recovery frame interpolation mode of the current frame is temporal extrapolation frame interpolation, then the current frame is added to the reference frame queue. When the length of the reference frame sequence reaches the number of temporal extrapolation frame interpolation predictions, the reference frame sequence and the number of temporal extrapolation frame interpolation predictions are used as inputs to the temporal extrapolation frame interpolation process, the interpolated frame is output, added to the output frame sequence, and the reference frame queue is set to an empty frame queue.

48. The decoding method according to claim 35, wherein, Determine the tail prediction input frame queue, and perform temporal tail recovery processing on the input frame queue based on the tail prediction input frame queue, including: If the temporal recovery frame interpolation mode is temporal interpolation frame interpolation, then the tail prediction input frame queue is set to a queue containing the last frame in the input frame queue. If the temporal recovery frame interpolation mode is temporal extrapolation frame interpolation, then the tail prediction input frame queue is set to a queue containing the last N frames in the output frame queue, where N is the number of temporal extrapolation frame prediction frames; The tail prediction input frame queue, the remaining number of frames in temporal recovery, and the temporal recovery frame interpolation mode are used as inputs to temporal tail recovery processing, and the output is the tail frame queue; If there is a frame after the last sampled frame in the output frame queue, delete the frame after the last sampled frame and add the tail frame queue after the last sampled frame.

49. An encoding method, comprising: The video is encoded to obtain bitstream data, so that the decoding end can implement the decoding method as described in any one of claims 34 to 48 based on the bitstream data.

50. A decoding method, comprising: Perform format conversion on the input frame queue of the first format to obtain the local frame queue of the second format; Perform temporal frame interpolation on the second format frame queue to generate the second format output frame queue; Perform format conversion on the output frame queue of the second format to obtain the output frame queue of the first format.

51. The decoding method according to claim 50, wherein, Before performing temporal frame interpolation on the frame queue of the second format to generate the output frame queue of the second format, the method further includes performing at least one of the following preprocessing steps on each frame image in the local frame queue: padding, normalization, and brightness enhancement; Before performing format conversion on the output frame queue of the second format to obtain the output frame queue of the first format, the method further includes performing at least one of the following post-processing operations on the output frame queue of the second format: luminance restoration, denormalization, and defilling.

52. The decoding method according to claim 50 or 51, wherein, The first format is YUV format; the second format is RGB format.

53. The decoding method according to claim 52, wherein, The YUV format is YCbCr420; the RGB format is RGB24.

54. The decoding method according to claim 50, wherein, Temporal frame interpolation is achieved through the following steps: Determine the temporal recovery frame interpolation mode; According to the time-domain recovery interpolation mode, the sampled frames in the input frame queue are interpolated in the time domain and / or interpolated in the time domain.

55. The decoding method according to claim 54, wherein, Before performing temporal interpolation and / or temporal extrapolation on the sampled frames in the input frame queue according to the temporal recovery interpolation mode, the method further includes: Perform time-domain recovery initialization operations; When no end-of-stream information is received, for each sampled frame in the frame queue, perform time-domain interpolation or time-domain extrapolation processing according to the time-domain recovery interpolation mode. When the end-of-stream information is received, the tail prediction input frame queue is determined, and the time-domain tail recovery processing is performed on the input frame queue based on the tail prediction input frame queue and the remaining number of time-domain recovered frames.

56. The decoding method according to claim 55, wherein, Perform time-domain recovery initialization operations, including at least one of the following: Initialize the output frame queue to an empty frame queue; Initialize the global time-domain recovery parameters to the sequence-level time-domain recovery parameters; Initialize the reference frame queue to an empty frame queue.

57. The decoding method according to claim 55, wherein, For each sampled frame in the frame queue, based on the temporal recovery interpolation mode, temporal interpolation or temporal extrapolation is performed, including: If the temporal recovery frame interpolation mode of the current frame is temporal interpolation and the current frame is not the first frame, then temporal interpolation frame processing is performed. If the temporal recovery interpolation mode of the current frame is temporal extrapolation, then the current frame is added to the reference frame queue. When the length of the reference frame sequence reaches the number of temporal extrapolation predictions, temporal extrapolation processing is performed.

58. The decoding method according to claim 57, wherein, The method further includes: If the temporal recovery frame interpolation mode of the current frame is temporal interpolation and the current frame is the first frame, then only the step of adding the current frame to the output frame queue is executed, and the temporal interpolation frame processing is not performed.

59. The decoding method according to claim 57, wherein, Perform temporal interpolation frame processing, including: The previous frame, the current frame, and the temporal sampling parameters are used as inputs to the temporal interpolation frame processing, and the interpolated frame is output and added to the output frame queue.

60. The decoding method according to claim 54, wherein, The method further includes: After adding the interpolated frames output from the temporal interpolation frame processing to the output queue, the current frame is added to the output frame queue.

61. The decoding method according to claim 59, wherein, The time-domain sampling parameters include: time-domain sampling rate.

62. The decoding method according to claim 59, wherein, Performing temporal interpolation frame processing also includes: If the temporal sampling rate change flag / temporal recovery update flag of the current frame is 1, the global temporal recovery parameters will be updated to the temporal recovery parameters of the current frame. If the temporal sampling rate change flag / temporal recovery update flag of the current frame is 0, the global temporal recovery parameters remain unchanged and are the same as the temporal recovery parameters of the previous frame.

63. The decoding method according to claim 62, wherein, After updating the global temporal recovery parameters to the temporal recovery parameters of the current frame, the method further includes: If the temporal recovery interpolation mode of the current frame is temporal extrapolation, then the current frame is added to the reference frame queue.

64. The decoding method according to claim 57, wherein, Perform temporal extrapolation frame interpolation, including: The reference frame sequence and the number of temporal extrapolation frame predictions are used as inputs to the temporal extrapolation frame processing procedure, and the output extrapolated frames are added to the output frame sequence.

65. The decoding method according to claim 57, wherein, The method further includes: Before adding the interpolated frames output from the temporal extrapolation frame processing to the output frame queue, add the current frame to the output frame queue.

66. The decoding method according to claim 64, wherein, Performing temporal extrapolation frame interpolation also includes: If the temporal sampling rate change flag / temporal recovery update flag of the current frame is 1, the global temporal recovery parameters will be updated to the temporal recovery parameters of the current frame. If the temporal sampling rate change flag / temporal recovery update flag of the current frame is 0, the global temporal recovery parameters remain unchanged and are the same as the temporal recovery parameters of the previous frame.

67. The decoding method according to claim 66, wherein, After updating the global temporal recovery parameters to the temporal recovery parameters of the current frame, the method further includes: If the temporal recovery frame interpolation mode of the current frame is temporal interpolation frame interpolation, then the reference frame queue is set to an empty frame queue. If the temporal recovery frame interpolation mode of the current frame is temporal extrapolation frame interpolation, then the current frame is added to the reference frame queue. When the length of the reference frame sequence reaches the number of temporal extrapolation frame interpolation predictions, the reference frame sequence and the number of temporal extrapolation frame interpolation predictions are used as inputs to the temporal extrapolation frame interpolation process, the interpolated frame is output, added to the output frame sequence, and the reference frame queue is set to an empty frame queue.

68. The decoding method according to claim 55, wherein, Determine the tail prediction input frame queue, and perform temporal tail recovery processing on the input frame queue based on the tail prediction input frame queue, including: If the temporal recovery frame interpolation mode is temporal interpolation frame interpolation, then the tail prediction input frame queue is set to a queue containing the last frame in the input frame queue. If the temporal recovery frame interpolation mode is temporal extrapolation frame interpolation, then the tail prediction input frame queue is set to a queue containing the last N frames in the output frame queue, where N is the temporal extrapolation frame prediction value; The tail prediction input frame queue, the remaining number of frames in temporal recovery, and the temporal recovery frame interpolation mode are used as inputs to temporal tail recovery processing, and the output is the tail frame queue; If there is a frame after the last sampled frame in the output frame queue, delete the frame after the last sampled frame and add the tail frame queue after the last sampled frame.

69. An encoding method, comprising: The video is encoded to obtain bitstream data, so that the decoding end can implement the decoding method as described in any one of claims 50 to 68 based on the bitstream data.

70. A decoding device, comprising: The first format conversion module is used to perform format conversion on the input frame queue of the first format to obtain the local frame queue of the second format; The temporal frame interpolation module is used to perform temporal frame interpolation processing on the frame queue of the second format to generate an output frame queue of the second format; The second format conversion module is used to perform format conversion on the output frame queue of the second format to obtain the output frame queue of the first format.

71. An encoding device, comprising: An encoding module is used to encode the video to obtain bitstream data, so that the decoding end can implement the decoding method as described in any one of claims 50 to 68 based on the bitstream data.

72. A decoding device, comprising: The frame interpolation mode determination module is used to determine the temporal recovery frame interpolation mode; The temporal recovery and interpolation module is used to perform temporal interpolation and / or temporal extrapolation on the sampled frames in the input frame queue according to the temporal recovery and interpolation mode.

73. An encoding device, comprising: A video encoding module is used to encode video to obtain bitstream data, so that the decoding end can implement the decoding method as described in any one of claims 34 to 46 based on the bitstream data.

74. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by the processor, it implements the decoding method according to any one of claims 1 to 28, 34 to 48, or 50 to 68, or the encoding method according to claims 29, 49, or 69.

75. An electronic device comprising: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the decoding method of any one of claims 1 to 28, 34 to 48, or 50 to 68, or the encoding method of claims 29, 49, or 69, by executing the executable instructions.

76. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the decoding method according to any one of claims 1-28, 34-48, or 50-68, or the encoding method according to claims 29, 49, or 69.

Images