Encoding and decoding methods, encoder, decoder, and storage medium

Abstract

The present application discloses encoding and decoding methods, an encoder, a decoder, and a storage medium. The methods comprise: either on an encoding end or a decoding end, determining a target adjustment factor for a current frame; on the basis of the target adjustment factor, adjusting a first virtual reference frame of the current frame, and obtaining a second virtual reference frame; and on the basis of the second virtual reference frame, predicting the current frame. In this way, a virtual reference frame is adjusted by using a target adjustment factor, so as to improve the quality of the virtual reference frame, as well as improving prediction accuracy of the current frame, and improving encoding and decoding performance.

Description

A coding and decoding method, encoder, decoder and storage medium Technical Field

[0001] The embodiments of the present application relate to the field of video coding and decoding technology, and specifically to a coding and decoding method, an encoder, a decoder, and a storage medium. Background Art

[0002] In inter-frame coding, the selection of reference frames is crucial for coding efficiency and performance. Reference frames with less compression distortion or more relevant content can reduce the residual error of the current frame prediction. Therefore, if some methods can be used to generate higher-quality virtual reference frames, the coding performance of the current frame can be improved.

[0003] In related virtual reference frame generation technologies, the generated virtual reference frames still have certain distortion information, which affects the encoding and decoding performance.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide a coding and decoding method, an encoder, a decoder, and a storage medium, which can improve the quality of virtual reference frames, thereby improving the accuracy of current frame prediction and improving coding and decoding performance.

[0006] The technical solution of the embodiment of the present application can be implemented as follows:

[0007] In a first aspect, an embodiment of the present application provides a decoding method, applied to a decoder, the method comprising:

[0008] Decode the code stream and determine the value of the first syntax element of the current frame;

[0009] determining a target adjustment factor for the current frame when a virtual reference frame for adjusting the current frame is determined according to the value of the first syntax element;

[0010] adjusting the first virtual reference frame of the current frame according to the target adjustment factor to obtain a second virtual reference frame;

[0011] The current frame is predicted according to the second virtual reference frame.

[0012] In a second aspect, an embodiment of the present application provides an encoding method, applied to an encoder, the method comprising:

[0013] determining at least one candidate adjustment factor for the current frame when adjustment of the virtual reference frame of the current frame is allowed;

[0014] adjusting the first virtual reference frame of the current frame according to the at least one candidate adjustment factor to obtain at least one second virtual reference frame;

[0015] Performing cost calculation based on the original frame of the current frame and the first virtual reference frame to obtain a first generation value;

[0016] Performing cost calculation based on the original frame of the current frame and the at least one second virtual reference frame to obtain at least one second generation value;

[0017] Making a coding decision based on the first generation value and the at least one second generation value to determine a value of a first syntax element for indicating whether to adjust a virtual reference frame of a current frame;

[0018] The value of the first syntax element is encoded, and the obtained encoded bits are written into a bitstream.

[0019] In a third aspect, an embodiment of the present application provides an encoder, comprising a first determining unit, a second determining unit, and an encoding unit; wherein:

[0020] The first determining unit is configured to, when adjustment of the virtual reference frame of the current frame is allowed, determine at least one candidate adjustment factor of the current frame; and adjust the first virtual reference frame of the current frame according to the at least one candidate adjustment factor to obtain at least one second virtual reference frame;

[0021] The second determining unit is configured to perform a cost calculation based on the original frame of the current frame and the first virtual reference frame to obtain a first generation value; perform a cost calculation based on the original frame of the current frame and the at least one second virtual reference frame to obtain at least one second generation value; make a coding decision based on the first generation value and the at least one second generation value to determine a value of a first syntax element used to indicate whether to adjust the virtual reference frame of the current frame;

[0022] The encoding unit is configured to encode the value of the first syntax element and write the obtained coded bits into a bitstream.

[0023] In a fourth aspect, an embodiment of the present application provides an encoder, comprising a first memory and a first processor; wherein,

[0024] a first memory for storing a computer program capable of running on the first processor;

[0025] The first processor is configured to execute the method according to the first aspect when running a computer program.

[0026] In a fifth aspect, an embodiment of the present application provides a decoder, comprising a decoding unit, a third determining unit, and a prediction unit; wherein:

[0027] The decoding unit is configured to decode the code stream and determine the value of the first syntax element of the current frame;

[0028] The third determining unit is configured to, when determining the virtual reference frame for adjusting the current frame according to the value of the first syntax element, determine a target adjustment factor for the current frame; and adjust the first virtual reference frame of the current frame according to the target adjustment factor to obtain a second virtual reference frame;

[0029] The prediction unit is configured to predict the current frame according to the second virtual reference frame.

[0030] In a sixth aspect, an embodiment of the present application provides a decoder, the decoder comprising a second memory and a second processor; wherein,

[0031] a second memory for storing a computer program capable of running on the second processor;

[0032] The second processor is configured to execute the method according to the second aspect when running a computer program.

[0033] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a code stream generated by the encoding method as described.

[0034] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed, implements the method described in the first aspect or the method described in the second aspect.

[0035] The present invention provides a coding and decoding method, an encoder, a decoder, and a storage medium. Whether at the encoding or decoding end, a target adjustment factor for a current frame is determined; a first virtual reference frame of the current frame is adjusted based on the target adjustment factor to obtain a second virtual reference frame; and the current frame is predicted based on the second virtual reference frame. In this manner, the target adjustment factor is used to adjust the virtual reference frame, thereby improving the quality of the virtual reference frame, thereby improving the accuracy of current frame prediction and enhancing coding and decoding performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0037] FIG1 is a schematic diagram of a composition framework of an encoder provided in an embodiment of the present application;

[0038] FIG2 is a schematic diagram of a bidirectional motion compensation for inter-frame prediction provided by an embodiment of the present application;

[0039] FIG3 is a schematic diagram of a composition framework of an NNVRF-based encoder provided in an embodiment of the present application;

[0040] FIG4 is a schematic diagram of a network architecture of a codec provided in an embodiment of the present application;

[0041] FIG5 is a schematic diagram of a flow chart of a decoding method provided in an embodiment of the present application;

[0042] FIG6 is a first structural diagram of a first virtual reference frame generation module provided in an embodiment of the present application;

[0043] FIG7 is a second structural diagram of a first virtual reference frame generation module provided in an embodiment of the present application;

[0044] FIG8 is a first schematic diagram of an implementation framework of a virtual reference frame adjustment method provided in an embodiment of the present application;

[0045] FIG9 is a second schematic diagram of an implementation framework of the virtual reference frame adjustment method provided in an embodiment of the present application;

[0046] FIG10 is a schematic diagram of a composition framework of a decoder for adjusting a virtual reference frame provided by an embodiment of the present application;

[0047] FIG11 is a schematic diagram of a flow chart of an encoding method provided in an embodiment of the present application;

[0048] FIG12 is a schematic diagram of a composition framework of an encoder for adjusting a virtual reference frame provided by an embodiment of the present application;

[0049] FIG13 is a schematic diagram of the structure of an encoder provided in an embodiment of the present application;

[0050] FIG14 is a schematic diagram of a specific hardware structure of an encoder provided in an embodiment of the present application;

[0051] FIG15 is a schematic diagram of the structure of a decoder provided in an embodiment of the present application;

[0052] FIG16 is a schematic diagram of a specific hardware structure of a decoder provided in an embodiment of the present application;

[0053] FIG17 is a schematic diagram of the composition structure of a coding and decoding system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will describe the technical solutions in the embodiments of this application in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0056] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0057] It should also be pointed out that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0058] In video images, a coding block (CB) is generally represented by a first image component, a second image component, and a third image component. These three image components are a luminance component, a blue chrominance component, and a red chrominance component. Specifically, the luminance component is typically represented by the symbol Y, the blue chrominance component is typically represented by the symbols Cb or U, and the red chrominance component is typically represented by the symbols Cr or V. Thus, video images can be represented in either the YCbCr or YUV format.

[0059] Before further explaining the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained first. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations:

[0060] Moving Picture Experts Group (MPEG)

[0061] International Standardization Organization (ISO)

[0062] International Electrotechnical Commission (IEC)

[0063] Joint Video Experts Team (JVET)

[0064] Alliance for Open Media (AOM)

[0065] Next-generation video coding standard H.266 / Versatile Video Coding (VVC)

[0066] VVC Test Model (VTM)

[0067] Audio Video Standard (AVS)

[0068] AVS High-Performance Model (HPM)

[0069] Transform coefficients

[0070] Quantization Parameter (QP)

[0071] Neural Network based Video Coding (NNVC)

[0072] Rate Distortion Optimization (RDO)

[0073] Mean Squared Error (MSE)

[0074] Mean Absolute Error (MAE)

[0075] Neural Network based Loop Filter (NNLF)

[0076] Random Access (RA)

[0077] Picture Order Count (POC)

[0078] Temporal Layer Identity (TID)

[0079] Reference Picture List (RPL)

[0080] Decoded picture buffer (DPB)

[0081] Virtual Reference Frame (VRF)

[0082] Neural Network based Virtual Reference Frame Generation (NNVRF)

[0083] Virtual Frame Adjust (VFA)

[0084] Digital video compression technology is understood to primarily compress massive amounts of digital video data for easier transmission and storage. With the surge in Internet video usage and increasing demand for higher-quality video, while existing digital video compression standards can conserve significant video data, there remains a need for improved digital video compression technologies to reduce bandwidth and traffic pressures associated with digital video transmission.

[0085] During digital video encoding, the encoder reads unequal pixels from raw video sequences in different color formats, including luminance and chrominance components. This means the encoder reads a black-and-white or color image. The image is then divided into blocks, and the block data is passed to the encoder for encoding. Currently, encoders typically use a hybrid frame coding scheme. Figure 1 shows a schematic diagram of the encoder architecture, which generally includes intra-frame and inter-frame prediction, transform and quantization, inverse transform and inverse quantization, loop filtering, and entropy coding. Intra-frame prediction refers only to information from the same frame, predicting pixel information within the current block to eliminate spatial redundancy. Inter-frame prediction can reference image information from different frames and use motion estimation to search for the motion vector that best matches the current block, eliminating temporal redundancy. Transformation converts the predicted image block to the frequency domain, redistributing the energy. Combined with quantization, it removes information that is insensitive to the human eye, eliminating visual redundancy. Entropy coding eliminates character redundancy based on the current context model and the probabilistic information of the binary bitstream. Loop filtering primarily processes the inverse-transformed and inverse-quantized pixels to compensate for distortion and provide a better reference for subsequent pixel encoding.

[0086] 1. Reference Frame

[0087] For the inter-frame prediction module, the decoded picture buffer (DPB) stores several reconstructed images, which can be used as reference frames for the current frame to perform operations such as inter-frame motion estimation and motion compensation.

[0088] In the current video coding standard VVC, three types of frames are defined: I, P, and B. I frames represent intra-frame coded frames, while P and B frames represent inter-frame coded frames. P frames can only reference reference frames that are before the current coded frame, meaning they can only use forward prediction. B frames can reference not only reference frames that are before the current coded frame, but also reference frames that are after the current coded frame, meaning they can use forward prediction, backward prediction, and bidirectional prediction. As shown in Figure 2, during bidirectional prediction, information from two reference frames can be used simultaneously, and the weighted average of the two reference macroblocks can be used as the reference information for the current block.

[0089] VVC establishes two reference picture lists (RPLs) for inter-frame prediction, including list0 (L0) for forward prediction and list1 (L1) for backward prediction. A single image can appear in different positions in both lists at the same time, providing a high degree of flexibility in the selection of multiple reference images. For each block in a P frame, only the reference image in L0 can be used, while each block in a B frame can use the reference images in both L0 and L1 (in bidirectional prediction, one image is selected from L0 and one from L1 as a reference).

[0090] Specifically, taking the Random Access (RA) configuration in VVC as an example, the default setting for IntraPeriod = 32 means an I-frame is inserted every 32 frames. In this configuration, the corresponding L0 and L1 settings are shown in Table 1. For example, for POC = 8, the reference L0 is frame 0 and frame 16, and the reference L1 is frame 16 and frame 32.

[0091] Table 1 VVC RA configuration (IntraPeriod=32)

[0092] As can be seen from Table 1, for the current frame, traditional reference frames are all encoded reconstructed frames, which are somewhat temporally distant from the current frame. In recent years, with the development of deep learning technology, neural network-based virtual reference frame generation technology has also gradually been developed.

[0093] 2. Virtual Reference Frame

[0094] The performance of inter-frame prediction is highly dependent on the content and quality of the reference frame. Reference frames with less compression distortion or more relevant content can make the prediction residual smaller. In the current research work of NNVC, some experts have proposed a virtual reference frame generation technology (NNVRF) based on the idea of ​​frame interpolation. NNVRF is a reference frame generation method based on deep neural networks for the VVC hierarchical coding structure. A better quality reference frame is generated by NNVRF, also known as a virtual reference frame. Specifically, the reconstructed image is obtained from the reference image buffer, and the NNVRF module generates a virtual reference frame for inter-frame prediction coding of the current frame.

[0095] The specific location of the NNVRF module in the current coding frame is shown in Figure 3. As can be seen, the input of NNVRF is the reference frame taken from the DPB, and the output is a virtual reference frame used for inter-frame prediction coding of the current frame.

[0096] As shown in Figure 3, the reference frames extracted from the DPB undergo the complete encoding and decoding process, are filtered by the loop filter module, and are then stored in the decoder buffer. However, the virtual reference frames generated by the NNVRF module are generated after the loop filter module and are not processed by it. As a result, the distortion information in the virtual reference frames is not properly filtered, affecting their quality and, consequently, encoding and decoding performance.

[0097] Based on this, the embodiments of the present application provide a coding and decoding method, an encoder, a decoder, and a storage medium, which can improve the quality of virtual reference frames, thereby improving the accuracy of current frame prediction and improving coding and decoding performance.

[0098] It should be noted that the method of the embodiment of the present application is mainly applied to the inter-frame prediction part. In other words, the embodiment of the present application can be applied to both the encoder and the decoder, or even to both the encoder and the decoder at the same time, but the embodiment of the present application does not make any specific limitation.

[0099] Figure 4 is a schematic diagram of a network architecture for encoding and decoding provided in an embodiment of the present application. As shown in Figure 4, the network architecture includes one or more electronic devices 13 to 1N and a communication network 01, wherein the electronic devices 13 to 1N can perform video interaction through the communication network 01. During implementation, the electronic device can be various types of devices with encoding and decoding functions. For example, the electronic device can include a mobile phone, a tablet computer, a personal computer, a personal digital assistant, a navigator, a digital phone, a video phone, a television, a sensing device, a server, etc., which is not limited by the embodiment of the present application. Among them, the decoder or encoder in the embodiment of the present application can be the above-mentioned electronic device. Among them, the electronic device in the embodiment of the present application has encoding and decoding functions, generally including an encoder and a decoder.

[0100] To facilitate understanding of the technical solutions of the embodiments of the present application, the technical solutions of the present application are described in detail below through specific embodiments. The above related technologies can be combined arbitrarily with the technical solutions of the embodiments of the present application as optional solutions, and all of them fall within the scope of protection of the embodiments of the present application. The embodiments of the present application include at least part of the following contents.

[0101] In one embodiment of the present application, referring to FIG5 , a schematic diagram of a process flow of an encoding method provided by an embodiment of the present application is shown. As shown in FIG5 , the method may include:

[0102] S501: Decode the code stream and determine the value of the first syntax element of the current frame;

[0103] It should be noted that in the embodiment of the present application, the current frame refers to any image to be decoded in the current sequence, and the virtual reference frame can be a virtual reference image generated based on the decoded image (also called the reconstructed image) and used to predict the current frame.

[0104] The first syntax element is used to indicate whether to adjust the virtual reference frame of the current frame. In some embodiments, the method further includes: when the value of the first syntax element is a first value, determining not to adjust the virtual reference frame of the current frame; when the value of the first syntax element is a second value, determining to adjust the virtual reference frame of the current frame.

[0105] Exemplarily, the first syntax element may be represented as picture_vfa_enable_flag, where the value of picture_vfa_enable_flag is "0", indicating that the virtual reference frame of the current frame is not adjusted; and picture_vfa_enable_flag, where the value of picture_vfa_enable_flag is "1", indicating that the virtual reference of the current frame is adjusted.

[0106] In some embodiments, the first syntax element is used to indicate whether to adjust the virtual reference frame of the first image component of the current frame. The first image component may be any image component of the current frame, or a specific image component of the current frame.

[0107] Exemplarily, the first syntax element may be represented as picture_vfa_enable_flag[Idx], which indicates whether to adjust the virtual reference frame of the Idx-th image component of the current frame. Exemplarily, the values ​​of Idx may be 0, 1, or 2, representing a luma component Y, a blue chroma component Cb, and a red chroma component Cr, respectively. The values ​​of Idx may be 0 or 1, representing luma and chroma, respectively.

[0108] In some embodiments, the method further includes: determining, based on the value of the third syntax element, that adjustment of the virtual reference frame of the current frame is permitted, decoding the bitstream and determining the value of the first syntax element of the current frame.

[0109] The third syntax element is used to indicate whether the virtual reference frame of the current frame is allowed to be adjusted. In some embodiments, the method further includes: when the value of the third syntax element is a first value, determining that the virtual reference frame of the current frame is not allowed to be adjusted; when the value of the third syntax element is a second value, determining that the virtual reference frame of the current frame is allowed to be adjusted.

[0110] The third syntax element may include a sequence-level syntax element that indicates whether adjustment of the virtual reference frame of the current sequence is permitted, that is, whether adjustment of the virtual reference frame of all frames in the current sequence is permitted. Exemplarily, the third syntax element may be represented as vfa_enable_flag. If vfa_enable_flag is "1," virtual reference frame adjustment is attempted for the current sequence; if vfa_enable_flag is "0," virtual reference frame adjustment is not performed for the current sequence. The definition of vfa_enable_flag in the sequence header is shown in Table 2.

[0111] Table 2

[0112] S502: Determine a target adjustment factor for the current frame when a virtual reference frame for adjusting the current frame is determined according to a value of the first syntax element;

[0113] The target adjustment factor is used to adjust the first virtual reference frame of the current frame (i.e., the original virtual reference frame) to generate a second virtual reference frame of higher quality. That is, the second virtual reference frame has higher quality than the first virtual reference frame, and using the second virtual reference frame can improve the prediction accuracy of the current frame.

[0114] In some embodiments, the target adjustment factor may be a filtering factor of the first virtual reference frame, and the first virtual reference frame is filtered according to the target adjustment factor to obtain the second virtual reference frame.

[0115] In some embodiments, the target adjustment factor may be a weighting factor of the first virtual reference frame, and the first virtual reference frame and other reference frames are weightedly fused according to the target adjustment factor to obtain the second virtual reference frame.

[0116] In some embodiments, determining a target adjustment factor for the current frame includes: decoding a bitstream to determine a value of a second syntax element of the current frame; and determining the target adjustment factor based on the value of the second syntax element. In other words, the decoder can determine the target adjustment factor based on the value of the second syntax element obtained by decoding the bitstream.

[0117] In some embodiments, the value of the second syntax element is used to indicate the value of the target adjustment factor. Exemplarily, the value of the second syntax element can directly indicate the target adjustment factor itself.

[0118] Exemplarily, the index value of the target adjustment factor is determined based on the value of the second syntax element; the target adjustment factor is then determined from a candidate set of adjustment factors based on the index value of the target adjustment factor. In other words, the value of the second syntax element can also indirectly indicate the index value of the target adjustment factor. By configuring the candidate set of adjustment factors, the encoder traverses the candidate set of adjustment factors to determine the target adjustment factor and writes the index value of the target adjustment factor into the bitstream for retrieval by the decoder. This method reduces codeword overhead. Exemplarily, the adjustment factor calculated by derivation requires encoding using a certain number of bits and can only be used after parsing by the decoder. For example, for a value range of 16 to 320, 9 bits are required to be encoded into the bitstream, resulting in a certain amount of codeword overhead. The candidate set of adjustment factors can be SF_List[N]. After statistical experiments, N adjustment factors with a high probability of being used are determined and an SF_List is established for decision-making by the encoder. The sequence number of the optimal candidate adjustment factor is then encoded into the bitstream for retrieval by the decoder.

[0119] Exemplarily, a reference value of the target adjustment factor is determined based on the value of the second syntax element; and the target adjustment factor is determined based on the reference value of the target adjustment factor. That is, the value of the second syntax element can also indirectly indicate the reference value of the target adjustment factor. When the encoding reference value has a smaller codeword overhead than the encoding target adjustment factor, the encoder can perform a preset transform on the target adjustment factor to obtain a reference value, write the reference value into the bitstream, and allow the decoder to read the reference value and recover the target adjustment factor through a corresponding inverse transform.

[0120] In some embodiments, the value of the second syntax element is used to indicate a target adjustment factor of a first image component of the current frame. The first image component may be any image component of the current frame, or a specific image component of the current frame.

[0121] For example, the second syntax element may be expressed as picture_scaling_factor[Idx], which indicates the target scaling factor for the Idxth image component of the current frame. The first and second syntax elements may be picture-level syntax elements, and picture_vfa_enable_flag and picture_scaling_factor[Idx] are defined in the picture header as shown in Table 3.

[0122] Table 3

[0123] When vfa_enable_flag is 1, the following semantics are defined:

[0124] Parse picture_vfa_enable_flag[Idx], where Idx is the image component index and takes a value of 0-N-1. N is the number of image components that allow adjustment of the virtual reference frame. When picture_vfa_enable_flag[Idx] is 1, parse the image-level adjustment factor picture_scaling_factor[Idx] to determine the value of the adjustment factor.

[0125] In some embodiments, the value of the second syntax element may also be used to indicate a derivation strategy of the target adjustment factor, and the encoding end and the decoding end use the same derivation strategy to determine the value of the target adjustment factor.

[0126] In some embodiments, determining a target adjustment factor for the current frame includes determining a target adjustment factor for a second image component of the current frame based on the target adjustment factor for the first image component of the current frame. Exemplarily, the target adjustment factor for the second image component may be obtained by reference to the target adjustment factor for the first image component. For example, the target adjustment factor for the luma component may be directly used as the target adjustment factor for the chroma component.

[0127] S503: Adjust the first virtual reference frame of the current frame according to the target adjustment factor to obtain a second virtual reference frame;

[0128] The first virtual reference frame is determined based on at least two reference frames reconstructed from the current frame. In some embodiments, the method further includes: obtaining at least two reference frames of the current frame from a reference frame list of the current frame; the at least two reference frames are input to a first virtual reference frame generation module, which outputs the first virtual reference frame.

[0129] In some embodiments, obtaining at least two reference frames of the current frame from the reference frame list of the current frame includes: if the current frame type is a bidirectionally predicted frame (i.e., a B frame), obtaining the first reference frame from the first reference frame list, and obtaining the second reference frame from the second reference frame list. If the current frame type is a unidirectionally predicted frame (i.e., a P frame), obtaining the first reference frame and the second reference frame from the third reference frame list.

[0130] A reference frame list is a set of reference frames stored in a buffer. These reference frames are used for the current frame. The reference frame list can include short-term reference frames and long-term reference frames. Short-term reference frames are reconstructed images that are closer to the current image, while long-term reference frames are reconstructed images that are farther away from the current image. The decoder uses different reference frame lists for P-frames and B-frames. For P-frames, one reference frame list is used; for B-frames, two reference frame lists are used.

[0131] For example, for a B-frame, the first reference frame is the forward reference frame closest to the current frame in the first reference frame list, and the second reference frame is the backward reference frame closest to the current frame in the second reference frame list. For a P-frame, the first reference frame and the second reference frame are the two reference frames closest to the current frame in the time domain in the third reference frame list.

[0132] In some embodiments, the first virtual reference frame generation module is a neural network-based generation module configured to: perform optical flow estimation based on at least two reference frames to determine optical flow information for the current frame; perform multi-scale feature extraction based on the at least two reference frames and the optical flow information to obtain a multi-scale feature map; and obtain the first virtual reference frame based on the multi-scale feature map. Using a neural network to predict changes in temporal optical flow information can achieve good generation results.

[0133] Exemplarily, the first virtual reference frame generation module includes at least: an optical flow estimation unit, a multi-layer feature extraction unit and an output unit. The optical flow estimation unit is used to estimate optical flow information and provide the optical flow information as new feature information to each layer of feature extraction unit for multi-scale feature extraction. The output unit is used to fuse the multi-scale feature map and output the first virtual reference frame.

[0134] FIG6 is a schematic diagram of the composition structure of a first virtual reference frame generation module provided by an embodiment of the present application. As shown in FIG6 , a reference image list L0 and a reference image list L1 are obtained from the reference image buffer DPB, and a reference frame is selected from each of the reference image list L0 and the reference image list L1 to form a set of image pairs as the input of the neural network-based generation module, which is input into the three-level feature extraction unit to extract feature maps of different granularities. At the same time, the existing classic interpolation algorithm module is used to predict the optical flow information between the current frame and the two input reference frames. The predicted or estimated optical flow information is input into the channel a' of the bilinear module to extract the optical flow information features of the first scale to obtain the first scale optical flow features; after one downsampling, it enters the channel b' to extract the optical flow information features of the second scale to obtain the second scale optical flow features; after another downsampling, it enters the channel c' to extract the optical flow information features of the third scale to obtain the third scale optical flow features; thereby obtaining optical flow features of three scales. In addition, the image features of the two reconstructed reference images of the input are extracted respectively through the convolution (Convolution, Conv) module to obtain two initial feature information. The two initial feature information corresponding to the two reconstructed reference images are input into channel a of the bilinear model for first-scale image feature extraction, resulting in two first-scale image features. After a single downsampling, the information is fed into channel b for second-scale image feature extraction, resulting in two second-scale image features. After another downsampling, the information is fed into channel c for third-scale image feature extraction, resulting in two third-scale image features. The first-scale optical flow features and the first-scale image features are further processed by the first feature processing module based on optical flow to extract deeper feature information, resulting in first-scale features. The second-scale optical flow features and the second-scale image features are further processed by the second feature processing module based on optical flow to obtain second-scale features. The third-scale optical flow features and the third-scale image features are further processed by the third feature processing module based on optical flow to obtain third-scale features. The first-, second-, and third-scale features are aggregated by the feature aggregation and prediction module, and prediction is performed based on the aggregated features to obtain the virtual reference image.

[0135] The classic frame interpolation algorithm module may be an IFRNet network, specifically a relatively small and simple small IFRNet. The small IFRNet has advantages in computational efficiency and resource usage.

[0136] FIG7 is a second schematic diagram of the composition structure of a first virtual reference frame generation module provided in an embodiment of the present application. As shown in FIG7 , the neural network-based generation module can be an IFRNet-L network. The IFRNet-L network inputs two reference frames and outputs a virtual reference frame. The IFRNet-L network merges the optical flow estimation with the intermediate feature extraction and reconstruction process into an encoder-decoder network, achieving a more compact model structure and faster inference speed. IFRNet-L first uses a pyramid encoder network to extract pyramid features for the two input reference frames respectively, and then uses multiple Coarse-to-Fine decoder networks to jointly refine the bidirectional intermediate optical flow and an intermediate frame feature with strong representation capability until the original input resolution is reached. A more accurate intermediate optical flow can reversely deform to produce intermediate frame features that are better aligned with the target frame, thereby facilitating intermediate frame feature reconstruction. On the other hand, better intermediate frame features can provide better anchor information, thereby facilitating intermediate optical flow estimation.

[0137] IFRNet-L is a larger and more complex model with more parameters and a deeper network structure. Due to its larger size, IFRNet-L is able to capture richer contextual information and achieve higher performance when handling complex video interpolation tasks.

[0138] Exemplarily, when the target adjustment factor can be a weighting factor of the first virtual reference frame, adjusting the first virtual reference frame of the current frame according to the target adjustment factor to obtain the second virtual reference frame includes: determining the first weight of the first virtual reference frame and the second weight of the third reference frame of the current frame according to the target adjustment factor; and performing weighted fusion of the first virtual reference frame and the third reference frame according to the first weight and the second weight to obtain the second virtual reference frame.

[0139] In some embodiments, the target adjustment factor can be used as the first weight, and the total weight minus the target adjustment factor is used to obtain the second weight. The weighting formula is as follows: output2(x i ,y i )=SF*output1(x i ,y i )+(1-SF)*input3(x i ,y i )

[0140] Among them, output1 is the first virtual reference frame, SF is the adjustment factor, that is, the first weight, input3 is the third reference frame, 1-SF is the second weight, and output2 is the second virtual reference frame.

[0141] The third reference frame can be any reference frame of the current frame. In some embodiments, the method further includes: performing weighted fusion on at least two reference frames of the current frame to determine the third reference frame. Exemplarily, as shown in FIG8 , the at least two reference frames can be weighted equally, i.e., each reference frame has an equal weight of 1 / n, where n is the number of reference frames. The at least two reference frames can also be weighted according to their respective corresponding weights.

[0142] The third reference frame may also be another virtual reference frame different from the first virtual reference frame. In some embodiments, the method further includes: inputting at least two reference frames of the current frame into a second virtual reference frame generation module, outputting a third reference frame; wherein the second virtual reference frame generation module is different from the first virtual reference frame generation module. Exemplarily, the second virtual reference frame generation module may be generated based on a traditional interpolation algorithm, or may be a generation module based on a neural network, but the neural network structure is different from that of the first virtual reference frame generation module.

[0143] As shown in Figure 9, two reference frames are input into the first virtual reference frame generation module NNVRF1, which outputs the first virtual reference frame. The two reference frames are input into the second virtual reference frame generation module NNVRF2, which outputs the third reference frame (which can also be understood here as a virtual reference frame). The first virtual reference frame and the third reference frame are weightedly fused according to the adjustment factor to obtain a high-quality second virtual reference frame.

[0144] S504: Predict the current frame according to the second virtual reference frame.

[0145] In some embodiments, predicting the current frame based on the second virtual reference frame includes: inserting the second virtual reference frame into a reference frame list of the current frame; and predicting the current frame based on the reference frame list. Specifically, performing inter-frame prediction on a to-be-encoded image block in the current frame based on the reference frame list.

[0146] In this embodiment of the present application, an adjustment factor is introduced to adjust the output virtual reference frame of the first virtual reference frame generation module to further optimize the decoding performance of the virtual reference frame generation tool. The specific position of the NNVRF module in the current decoding frame is shown in Figure 10. A virtual reference frame adjustment module (Virtual Frame Adjust, VFA) is provided after NNVRF to adjust the output virtual reference frame of the first virtual reference frame generation module according to the adjustment factor.

[0147] The decoding method provided in the embodiment of the present application is specifically implemented at the decoding end as follows:

[0148] First, according to the requirements of the NNVRF module, two reference frames are extracted from the decoded image buffer DPB as input information of the NNVRF module, and the first virtual reference frame vrf_normal is generated by the NNVRF module. Then:

[0149] a) Determine whether virtual reference frame adjustment is allowed for the current sequence based on the sequence-level flag vfa_enable_flag. If vfa_enable_flag is "1," virtual reference frame adjustment is attempted for the current sequence, and the process goes to b); if vfa_enable_flag is "0," virtual reference frame adjustment is not performed for the current sequence, and the process goes to d).

[0150] b) For the current frame of the current sequence, parse the adjustment enable flag picture_vfa_enable_flag of the current frame. If picture_vfa_enable_flag is "1", jump to c); if picture_vfa_enable_flag is "0", jump to d);

[0151] c) For the current frame of the current sequence, parse the adjustment factor picture_scaling_factor and adjust the first virtual reference frame vrf_normal to obtain the second virtual reference frame vrf_adj. Jump to d);

[0152] d) Insert the second virtual reference frame vrf_adj or the first virtual reference frame vrf_normal into the reference frame lists L0 and L1, and perform inter-frame prediction on the current frame. If the current frame has been processed, load the next frame for processing and jump to step b).

[0153] In an embodiment of the present application, a target adjustment factor is used to adjust the virtual reference frame, and the output information of the neural network-based virtual reference frame generation module is adjusted relatively simply to improve the quality of the virtual reference frame, thereby improving the accuracy of the current frame prediction, with basically no increase in the complexity of encoding and decoding.

[0154] In yet another embodiment of the present application, see Figure 11, which shows a schematic flow chart of an encoding method provided by an embodiment of the present application. As shown in Figure 11, the method may include:

[0155] S1101: Determine at least one candidate adjustment factor of the current frame when adjustment of the virtual reference frame of the current frame is allowed;

[0156] It should be noted that in the embodiment of the present application, the current frame refers to any image to be encoded in the current sequence, and the virtual reference frame can be a virtual reference image generated based on the encoded image (also called the reconstructed image) and used to predict the current frame.

[0157] In some embodiments, the method further includes: determining, based on the value of a third syntax element, whether adjustment of the virtual reference frame of the current frame is permitted. The third syntax element is used to indicate whether adjustment of the virtual reference frame of the current frame is permitted. In some embodiments, the method further includes: determining, when the value of the third syntax element is a first value, that adjustment of the virtual reference frame of the current frame is not permitted; and determining, when the value of the third syntax element is a second value, that adjustment of the virtual reference frame of the current frame is permitted.

[0158] The third syntax element may include a sequence-level syntax element that indicates whether adjustment of the virtual reference frame of the current sequence is permitted, that is, whether adjustment of the virtual reference frame of all frames in the current sequence is permitted. Exemplarily, the third syntax element may be represented as vfa_enable_flag. If vfa_enable_flag is "1," virtual reference frame adjustment is attempted for the current sequence; if vfa_enable_flag is "0," virtual reference frame adjustment is not performed for the current sequence. The definition of vfa_enable_flag in the sequence header is shown in Table 2.

[0159] In some embodiments, determining at least one candidate adjustment factor for the current frame includes traversing a candidate adjustment factor set to obtain multiple candidate adjustment factors. The candidate adjustment factor set may be SF_List[N]. After statistical experiments, N adjustment factors with a high probability of being used are determined and created into SF_List for decision-making by the encoder. The sequence number of the optimal candidate adjustment factor is then encoded into the bitstream for reading and use by the decoder.

[0160] In some embodiments, determining at least one candidate adjustment factor for the current frame includes: determining the candidate adjustment factor based on an original frame of the current frame, a first virtual reference frame, and a third reference frame based on an adjustment factor derivation algorithm. The third reference frame may be any reference frame of the current frame, or may be a virtual reference frame different from the first virtual reference frame.

[0161] In some embodiments, the adjustment factor derivation algorithm comprises a least squares method.

[0162] Exemplarily, the following steps are implemented based on the least squares method: determine a first residual feature map based on the original samples of the original frame and the reference samples of the third reference frame; determine a second residual feature map based on the virtual reference samples of the first virtual reference frame and the reference samples of the third reference frame; determine a first residual sum and a residual square sum based on the first residual feature map; determine a second residual sum based on the second residual feature map; determine a cross residual sum based on the first residual feature map and the second residual feature map; determine a candidate adjustment factor based on the first residual sum, the residual square, the second residual sum, and the cross residual sum.

[0163] In some embodiments, candidate adjustment factors are determined based on the first residual sum, the residual square, the second residual sum, and the cross residual sum, including: calculating the product of the height, width, and cross residual sum of the current frame to obtain a first parameter; calculating the product of the first residual sum and the second residual sum to obtain a second parameter; calculating the product of the height, width, and residual square sum of the current frame to obtain a third parameter; calculating the square of the second residual sum to obtain a fourth parameter; and dividing the difference between the first parameter and the second parameter by the difference between the third parameter and the fourth parameter to obtain a candidate adjustment factor.

[0164] The derivation formula is as follows: Assume that the width of the current frame is W and the height is H, and P(x,y) represents the horizontal and vertical coordinates of the reference pixel point P. The third reference frame is input3(x,y), the first virtual reference frame is output1(x,y), the second virtual reference frame is output2(x,y), and the original frame is org(x,y).

[0165] First, calculate the pixel-level residual parameters of the two images, orgResi(x i ,y i )=[org(x i ,y i )-input3(x i ,y i )] (Formula 1) cnnResi(x i ,y i )=[output1(x i ,y i )-input3(x i ,y i )] (Formula 2)

[0166] Then, based on the pixel-level residual parameters, the residual parameters of the entire image level are obtained through various forms of weighted sum.

[0167] Finally, the one-dimensional adjustment factor SF at the image level is calculated statistically.

[0168] In some embodiments, the adjustment factor derivation algorithm further performs the following steps: clamping the candidate adjustment factors to within preset upper and lower adjustment factor limits.

[0169] In some embodiments, determining at least one candidate adjustment factor for the current frame includes determining a candidate adjustment factor for a second image component of the current frame based on a target adjustment factor for a first image component of the current frame. That is, at the encoder, the candidate adjustment factor for the second image component may be derived by reference to the target adjustment factor for the first image component, and further determining whether the target adjustment factor for the first image component is used to adjust the virtual reference frame for the second image component. For example, the target adjustment factor for the luma component may be directly used as a candidate adjustment factor for the chroma component.

[0170] S1102: Adjust a first virtual reference frame of a current frame according to at least one candidate adjustment factor to obtain at least one second virtual reference frame;

[0171] The first virtual reference frame is determined based on at least two reference frames reconstructed from the current frame. In some embodiments, the method further includes: obtaining at least two reference frames of the current frame from a reference frame list of the current frame; the at least two reference frames are input to a first virtual reference frame generation module, which outputs the first virtual reference frame.

[0172] In some embodiments, obtaining at least two reference frames of the current frame from the reference frame list of the current frame includes: if the current frame type is a bidirectionally predicted frame (i.e., a B frame), obtaining the first reference frame from the first reference frame list, and obtaining the second reference frame from the second reference frame list. If the current frame type is a unidirectionally predicted frame (i.e., a P frame), obtaining the first reference frame and the second reference frame from the third reference frame list.

[0173] A reference frame list is a set of reference frames stored in a buffer. These reference frames are used for the current frame. The reference frame list can include short-term reference frames and long-term reference frames. Short-term reference frames are reconstructed images that are closer to the current image, while long-term reference frames are reconstructed images that are farther away from the current image. The decoder uses different reference frame lists for P-frames and B-frames. For P-frames, one reference frame list is used; for B-frames, two reference frame lists are used.

[0174] For example, for a B-frame, the first reference frame is the forward reference frame closest to the current frame in the first reference frame list, and the second reference frame is the backward reference frame closest to the current frame in the second reference frame list. For a P-frame, the first reference frame and the second reference frame are the two reference frames closest to the current frame in the time domain in the third reference frame list.

[0175] In some embodiments, the first virtual reference frame generation module is a neural network-based generation module configured to: perform optical flow estimation based on at least two reference frames to determine optical flow information for the current frame; perform multi-scale feature extraction based on the at least two reference frames and the optical flow information to obtain a multi-scale feature map; and obtain the first virtual reference frame based on the multi-scale feature map. Using a neural network to predict changes in temporal optical flow information can achieve good generation results.

[0176] Exemplarily, the first virtual reference frame generation module includes at least: an optical flow estimation unit, a multi-layer feature extraction unit and an output unit. The optical flow estimation unit is used to estimate optical flow information and provide the optical flow information as new feature information to each layer of feature extraction unit for multi-scale feature extraction. The output unit is used to fuse the multi-scale feature map and output the first virtual reference frame.

[0177] In some embodiments, adjusting the first virtual reference frame of the current frame according to at least one candidate adjustment factor to obtain at least one second virtual reference frame includes: determining a first weight of the first virtual reference frame and a second weight of the third reference frame of the current frame according to the candidate adjustment factor; and performing weighted fusion on the first virtual reference frame and the third reference frame according to the first weight and the second weight to obtain the second virtual reference frame.

[0178] In some embodiments, the target adjustment factor can be used as the first weight, and the total weight minus the target adjustment factor is used to obtain the second weight. i ,y i ) and the third reference frame input3(x i ,y i ) is weighted and adjusted to obtain the final second virtual reference frame output2(x i ,y i ), as shown in Equation 9. output2(x i ,y i )=SF*output1(x i ,y i )+(1-SF)*input3(x i ,y i ) (Formula 9)

[0179] The third reference frame can be any reference frame of the current frame. In some embodiments, the method further includes: performing weighted fusion on at least two reference frames of the current frame to determine the third reference frame. Exemplarily, as shown in FIG8 , the at least two reference frames can be weighted equally, i.e., each reference frame has an equal weight of 1 / n, where n is the number of reference frames. The at least two reference frames can also be weighted according to their respective corresponding weights.

[0180] The third reference frame may also be another virtual reference frame different from the first virtual reference frame. In some embodiments, the method further includes: inputting at least two reference frames of the current frame into a second virtual reference frame generation module, outputting a third reference frame; wherein the second virtual reference frame generation module is different from the first virtual reference frame generation module. Exemplarily, the second virtual reference frame generation module may be generated based on a traditional interpolation algorithm, or may be a generation module based on a neural network, but the neural network structure is different from that of the first virtual reference frame generation module.

[0181] As shown in Figure 9, two reference frames are input into the first virtual reference frame generation module NNVRF1, which outputs the first virtual reference frame. The two reference frames are input into the second virtual reference frame generation module NNVRF2, which outputs the third reference frame (which can also be understood here as a virtual reference frame). The first virtual reference frame and the third reference frame are weightedly fused according to the adjustment factor to obtain a high-quality second virtual reference frame.

[0182] In some embodiments, the method further includes: inserting the second virtual reference frame into a reference frame list of the current frame; and predicting the current frame based on the reference frame list. Specifically, performing inter-frame prediction on the image blocks to be encoded in the current frame based on the reference frame list.

[0183] S1103: Calculate the cost based on the original frame of the current frame and the first virtual reference frame to obtain a first generation value;

[0184] S1104: Calculate the cost based on the original frame of the current frame and at least one second virtual reference frame to obtain at least one second generation value;

[0185] In some embodiments, the cost function used may be rate-distortion optimization (RDO), and based on the RDOs of different virtual reference frames, whether to adjust the virtual reference frame of the current frame is determined based on the virtual reference frame corresponding to the best RDO. Specifically, when the virtual reference frame corresponding to the best RDO is the first virtual reference frame, the virtual reference frame of the current frame is not adjusted; when the virtual reference frame corresponding to the best RDO is the second virtual reference frame, the virtual reference frame of the current frame is adjusted.

[0186] Exemplarily, a first virtual reference frame is inserted into a reference frame list of a current frame, the current frame is reconstructed according to the reference frame list to obtain a first reconstructed frame, the distortion of the original frame and the first reconstructed frame of the current frame is calculated, and an RDO cost value is further determined. A second virtual reference frame is inserted into a reference frame list of the current frame, the current frame is reconstructed according to the reference frame list to obtain a second reconstructed frame, the distortion of the original frame and the second reconstructed frame of the current frame is calculated, and an RDO cost value is further determined.

[0187] In other embodiments, the cost function used may also be the Sum of Absolute Errors (SAD), the Sum of Absolute Transform Differences (SATD), the Mean Squared Error (MSE), the Sum of Squared Errors (SSD), the Mean Absolute Difference (MAD), the Mean Squared Error (MSD), the absolute value of the transform coefficients (DCT), the Hadamard transform, etc., without specific limitation herein. For example, the SAD cost value is calculated for the first virtual reference frame and the original frame, and the SAD cost value is calculated for the second virtual reference frame and the original frame.

[0188] S1105: Making a coding decision based on the first generation value and at least one second generation value to determine a value of a first syntax element for indicating whether to adjust a virtual reference frame of a current frame;

[0189] In some embodiments, when the first cost value is the best cost value, the value of the first syntax element is determined to be a first value, which is used to indicate that the virtual reference frame of the current frame is not adjusted; when the second cost value is the best cost value, the value of the first syntax element is determined to be a second value, which is used to indicate that the virtual reference frame of the current frame is adjusted.

[0190] The first syntax element is used to indicate whether to adjust the virtual reference frame of the current frame. In some embodiments, the method further includes: when the value of the first syntax element is a first value, determining not to adjust the virtual reference frame of the current frame; when the value of the first syntax element is a second value, determining to adjust the virtual reference frame of the current frame.

[0191] Exemplarily, the first syntax element may be represented as picture_vfa_enable_flag, where the value of picture_vfa_enable_flag is "0", indicating that the virtual reference frame of the current frame is not adjusted; and picture_vfa_enable_flag, where the value of picture_vfa_enable_flag is "1", indicating that the virtual reference of the current frame is adjusted.

[0192] In some embodiments, the first syntax element is used to indicate whether to adjust the virtual reference frame of the first image component of the current frame. The first image component may be any image component of the current frame, or a specific image component of the current frame.

[0193] Exemplarily, the first syntax element may be represented as picture_vfa_enable_flag[Idx], which indicates whether to adjust the virtual reference frame of the Idx-th image component of the current frame. Exemplarily, the values ​​of Idx may be 0, 1, or 2, representing a luma component Y, a blue chroma component Cb, and a red chroma component Cr, respectively. The values ​​of Idx may be 0 or 1, representing luma and chroma, respectively.

[0194] In some embodiments, the method further includes: when the second cost value is the optimal cost value, encoding a target adjustment factor corresponding to the optimal cost value, and writing the obtained coded bits into the bitstream.

[0195] In some embodiments, encoding a target adjustment factor corresponding to the optimal cost value includes: determining a value of a second syntax element of the current frame according to the target adjustment factor; encoding the value of the second syntax element, and writing the obtained coded bits into a bitstream.

[0196] In some embodiments, determining the value of a second syntax element of the current frame based on a target adjustment factor includes: determining an index value of the target adjustment factor in a candidate set of adjustment factors; and determining the value of the second syntax element based on the index value of the target adjustment factor. In other words, the value of the second syntax element can also indirectly indicate the index value of the target adjustment factor. By configuring the candidate set of adjustment factors, the encoder traverses the candidate set of adjustment factors to determine the target adjustment factor, and writes the index value of the target adjustment factor into the bitstream for reading and use by the decoder. This method reduces codeword overhead.

[0197] In some embodiments, determining a value of a second syntax element of the current frame according to a target adjustment factor includes: determining a reference value of the target adjustment factor according to the target adjustment factor; and determining a value of the second syntax element according to the reference value of the target adjustment factor.

[0198] In some embodiments, the best second generation value may be determined first, and then compared with the first generation value to determine whether to adjust the virtual reference frame. The method for determining the best second generation value and its corresponding index value is as follows:

[0199] Among them, minD is the best second-generation value, and bestSF_index is the index value of the target adjustment factor.

[0200] S1106: Encode the value of the first syntax element, and write the obtained coded bits into the bitstream.

[0201] In an embodiment of the application, an adjustment factor is introduced to adjust the output virtual reference frame of the first virtual reference frame generation module to further optimize the encoding performance of the virtual reference frame generation tool. The specific position of the NNVRF module in the current encoding frame is shown in Figure 12. A virtual reference frame adjustment module (Virtual Frame Adjust, VFA) is provided after NNVRF to adjust the output virtual reference frame of the first virtual reference frame generation module according to the adjustment factor.

[0202] The encoding method provided in the embodiment of the present application is specifically implemented on the encoding end as follows:

[0203] First, according to the requirements of the NNVRF module, extract 2 reference frames from the decoded picture buffer (DPB) as the input information of the NNVRF module, and use the NNVRF module to generate the first virtual reference frame vrf_normal. Then:

[0204] a) Determine whether virtual reference frame adjustment is allowed in the current sequence according to the sequence-level flag vfa_enable_flag. If vfa_enable_flag is "1", attempt to adjust the virtual reference frame for the current sequence and jump to b); if vfa_enable_flag is "0", do not perform virtual reference frame adjustment for the current sequence and jump to d);

[0205] [[ID=,6]]b) For the current frame of the current sequence, attempt to perform adjustment using the adjustment method provided in the embodiments of the present application. Specifically, adjust the first virtual reference frame vrf_normal using an adjustment factor to obtain the adjusted second virtual reference frame vrf_adj, and jump to c);

[0206] c) Compare the second virtual reference frame vrf_adj and the first virtual reference frame vrf_normal with the original frame of the current frame respectively and calculate the distortion costs, denoted as D_adj and D_normal. Compare the two costs. If D_adj < D_normal, assign the adjustment flag picture_vfa_enable_flag of the current frame as "1", and use the second virtual reference frame vrf_adj as the virtual reference frame of the current frame, and encode the adjustment factor picture_scaling_factor into the code stream; otherwise, if D_adj ≥ D_normal, assign the adjustment flag picture_vfa_enable_flag of the current frame as "0", and still use the original first virtual reference frame vrf_normal as the virtual reference frame of the current frame. Jump to d);

[0207] d) Insert the second virtual reference frame vrf_adj or the first virtual reference frame vrf_normal into the reference frame lists L0 and L1, and perform the operation of inter-frame prediction on the current frame. If the current frame has been processed, load the next frame for processing and jump to step b).

[0208] In the embodiments of the present application, the virtual reference frame is adjusted using the target adjustment factor, and the output information of the virtual reference frame generation module based on the neural network is adjusted simply, improving the quality of the virtual reference frame, and thus improving the prediction accuracy of the current frame, and basically not increasing the encoding and decoding complexity.

[0209] The present embodiment further tests a method using a candidate set of adjustment factors, specifically SF_List[4] = {192, 256, 288, 320}, so only 2 bits need to be encoded into the bitstream. For example, an experiment was conducted on the chrominance component of a coded frame with temporal layer 5.

[0210] Under the general test condition Random Access configuration, the general sequence specified by JVET was tested, and the comparison anchor was NNVRF. The performance of some sequences was tested. The results are shown in Table 4, which can bring certain performance improvements on the chrominance component.

[0211] Table 4 Random Access performance of NNVRF

[0212] Analyzing the data in Table 4, we can see that by introducing a virtual reference frame adjustment method, we can improve the quality of virtual reference frames based on virtual reference frame generation technology, further enhancing encoding performance. Furthermore, adjusting virtual reference frames through a simple weighted calculation barely increases encoding and decoding complexity.

[0213] In yet another embodiment of the present application, based on the same inventive concept as the previous embodiment, see Figure 13, which shows a schematic diagram of the composition structure of an encoder provided in an embodiment of the present application. As shown in Figure 13, the encoder 130 may include a first determination unit 1301, a second determination unit 1302, and an encoding unit 1303; wherein:

[0214] The first determining unit 1301 is configured to determine at least one candidate adjustment factor of the current frame when adjustment of the virtual reference frame of the current frame is allowed; and adjust the first virtual reference frame of the current frame according to the at least one candidate adjustment factor to obtain at least one second virtual reference frame;

[0215] The second determining unit 1302 is configured to perform a cost calculation based on the original frame of the current frame and the first virtual reference frame to obtain a first generation value; perform a cost calculation based on the original frame of the current frame and at least one second virtual reference frame to obtain at least one second generation value; make a coding decision based on the first generation value and the at least one second generation value to determine a value of a first syntax element used to indicate whether to adjust the virtual reference frame of the current frame;

[0216] The encoding unit 1303 is configured to encode the value of the first syntax element and write the obtained coded bits into the bitstream.

[0217] It can be understood that each functional unit of the encoder also executes the encoding method of any one of the aforementioned embodiments.

[0218] It is understandable that in the embodiments of the present application, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and of course it can also be a module, or it can be non-modular. Moreover, the various components in this embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional modules.

[0219] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the portion that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0220] Therefore, an embodiment of the present application provides a computer-readable storage medium, which is applied to the encoder 130. The computer-readable storage medium stores a computer program, and when the computer program is executed by the first processor, it implements the method of any one of the aforementioned embodiments.

[0221] Based on the composition of the encoder 130 and the computer-readable storage medium, refer to Figure 14, which shows a specific hardware structure diagram of the encoder 140 provided in an embodiment of the present application. As shown in Figure 14, the encoder 130 may include: a first communication interface 1401, a first memory 1402 and a first processor 1403; each component is coupled together through a first bus system 1404. It can be understood that the first bus system 1404 is used to achieve connection and communication between these components. In addition to the data bus, the first bus system 1404 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are labeled as the first bus system 1404 in Figure 14. Among them,

[0222] The first communication interface 1401 is used to receive and send signals when sending and receiving information with other external network elements;

[0223] A first memory 1402 is used to store computer programs that can be run on the first processor 1403;

[0224] The first processor 1403 is configured to, when running the computer program, execute:

[0225] determining at least one candidate adjustment factor for the current frame when adjustment of the virtual reference frame of the current frame is allowed;

[0226] adjusting a first virtual reference frame of the current frame according to at least one candidate adjustment factor to obtain at least one second virtual reference frame;

[0227] The cost is calculated based on the original frame and the first virtual reference frame of the current frame to obtain the first generation value;

[0228] Performing cost calculation based on an original frame of the current frame and at least one second virtual reference frame to obtain at least one second generation value;

[0229] Making a coding decision based on the first generation value and at least one second generation value to determine a value of a first syntax element for indicating whether to adjust a virtual reference frame of a current frame;

[0230] The value of the first syntax element is encoded, and the obtained coded bits are written into the bitstream.

[0231] It is understood that the first memory 1402 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The first memory 1402 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0232] The first processor 1403 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in the first processor 1403. The above-mentioned first processor 1403 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the first memory 1402 , and the first processor 1403 reads the information in the first memory 1402 and completes the steps of the above method in combination with its hardware.

[0233] It is understood that the embodiments described herein can be implemented with hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSP devices, DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions of the present application or a combination thereof. For software implementation, the technology of the present application can be implemented by a module (such as a process, a function, etc.) that performs the functions of the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0234] Optionally, as another embodiment, the first processor 1403 is further configured to execute any one of the methods in the foregoing embodiments when running a computer program.

[0235] This embodiment provides an encoder in which a target adjustment factor is used to adjust a virtual reference frame to improve the quality of the virtual reference frame, thereby improving the current frame prediction accuracy and improving encoding performance.

[0236] In yet another embodiment of the present application, based on the same inventive concept as the aforementioned embodiment, FIG15 is a schematic diagram showing the structure of a decoder 150 provided in an embodiment of the present application. As shown in FIG15 , the decoder 150 may include: a decoding unit 1501, a third determination unit 1502, and a prediction unit 1503; wherein:

[0237] The decoding unit 1501 is configured to decode the code stream and determine the value of the first syntax element of the current frame;

[0238] The third determining unit 1502 is configured to determine a target adjustment factor for the current frame when the virtual reference frame for adjusting the current frame is determined according to the value of the first syntax element; and adjust the first virtual reference frame of the current frame according to the target adjustment factor to obtain a second virtual reference frame;

[0239] The prediction unit 1503 is configured to predict the current frame according to the second virtual reference frame.

[0240] It can be understood that each functional unit of the decoder also executes the decoding method of any one of the aforementioned embodiments.

[0241] Based on the composition of the decoder 150 and the computer-readable storage medium, refer to Figure 16, which shows a specific hardware structure diagram of the decoder 150 provided in an embodiment of the present application. As shown in Figure 16, the decoder 150 may include: a second communication interface 1601, a second memory 1602 and a second processor 1603; each component is coupled together through a second bus system 1604. It can be understood that the second bus system 1604 is used to achieve connection and communication between these components. In addition to the data bus, the second bus system 1604 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are labeled as the second bus system 1604 in Figure 16. Among them,

[0242] The second communication interface 1601 is used to receive and send signals when sending and receiving information with other external network elements;

[0243] The second memory 1602 is used to store computer programs that can be run on the second processor 1603;

[0244] The second processor 1603 is configured to, when running the computer program, execute:

[0245] Decode the code stream and determine the value of the first syntax element of the current frame;

[0246] determining a target adjustment factor for the current frame when a virtual reference frame for adjusting the current frame is determined according to a value of the first syntax element;

[0247] Adjust the first virtual reference frame of the current frame according to the target adjustment factor to obtain a second virtual reference frame;

[0248] The current frame is predicted according to the second virtual reference frame.

[0249] Optionally, as another embodiment, the second processor 1603 is further configured to execute any one of the methods in the foregoing embodiments when running a computer program.

[0250] It can be understood that the hardware functions of the second memory 1602 are similar to those of the first memory 1402, and the hardware functions of the second processor 1603 are similar to those of the first processor 1403; they will not be described in detail here.

[0251] This embodiment provides a decoder in which a target adjustment factor is used to adjust a virtual reference frame to improve the quality of the virtual reference frame, thereby improving the current frame prediction accuracy and decoding performance.

[0252] In yet another embodiment of the present application, referring to FIG17 , a schematic diagram of the structure of a coding and decoding system provided by an embodiment of the present application is shown. As shown in FIG17 , the coding and decoding system 170 may include an encoder 1701 and a decoder 1702 .

[0253] In the embodiment of the present application, the encoder 1701 may be the encoder of any one of the aforementioned embodiments, and the decoder 1702 may be the decoder of any one of the aforementioned embodiments.

[0254] It should be noted that, in this application, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0255] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0256] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined, if they do not conflict, to obtain new method embodiments. The features disclosed in the several product embodiments provided in this application can be arbitrarily combined, if they do not conflict, to obtain new product embodiments. The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined, if they do not conflict, to obtain new method embodiments or device embodiments.

[0257] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims. Industrial Applicability

[0258] The present invention provides a coding and decoding method, an encoder, a decoder, and a storage medium. At either the encoding or decoding end, a target adjustment factor for a current frame is determined; a first virtual reference frame of the current frame is adjusted based on the target adjustment factor to obtain a second virtual reference frame; and the current frame is predicted based on the second virtual reference frame. In this manner, the target adjustment factor is used to adjust the virtual reference frame to improve the quality of the virtual reference frame, thereby improving the accuracy of current frame prediction and encoding and decoding performance.

Claims

1. A decoding method, applied to a decoder, comprising: Decode the code stream and determine the value of the first syntax element of the current frame; determining a target adjustment factor for the current frame when a virtual reference frame for adjusting the current frame is determined according to the value of the first syntax element; adjusting the first virtual reference frame of the current frame according to the target adjustment factor to obtain a second virtual reference frame; The current frame is predicted according to the second virtual reference frame.

2. The method according to claim 1, wherein The method further comprises: When the value of the first syntax element is a first value, determining not to adjust the virtual reference frame of the current frame; When the value of the first syntax element is the second value, it is determined to adjust the virtual reference frame of the current frame.

3. The method according to claim 1, wherein Determining the target adjustment factor of the current frame includes: Decoding the code stream to determine the value of the second syntax element of the current frame; The target adjustment factor is determined according to the value of the second syntax element.

4. The method according to claim 3, wherein: The determining the target adjustment factor according to the value of the second syntax element includes: Determining an index value of the target adjustment factor according to a value of the second syntax element; The target adjustment factor is determined from an adjustment factor candidate set according to the index value of the target adjustment factor.

5. The method according to claim 3, wherein The determining the target adjustment factor according to the value of the second syntax element includes: determining a reference value of the target adjustment factor according to a value of the second syntax element; The target adjustment factor is determined according to a reference value of the target adjustment factor.

6. The method according to claim 1, wherein Determining the target adjustment factor of the current frame includes: A target adjustment factor of a second image component of the current frame is determined according to the target adjustment factor of the first image component of the current frame.

7. The method according to claim 1, wherein The method further comprises: Obtain at least two reference frames of the current frame from the reference frame list of the current frame; The at least two reference frames are input to a first virtual reference frame generation module, which outputs the first virtual reference frame.

8. The method according to claim 7, wherein: The acquiring at least two reference frames of the current frame from the reference frame list of the current frame includes: The current frame type is a bidirectional prediction frame, a first reference frame is obtained from the first reference frame list, and a second reference frame is obtained from the second reference frame list.

9. The method according to claim 8, wherein The first reference frame is a forward reference frame closest to the current frame in the first reference frame list, and the second reference frame is a backward reference frame closest to the current frame in the second reference frame list.

10. The method according to claim 7, wherein: The acquiring at least two reference frames of the current frame from the reference frame list of the current frame includes: The current frame type is a unidirectional prediction frame, and the first reference frame and the second reference frame are obtained from the third reference frame list.

11. The method according to claim 10, wherein: The first reference frame and the second reference frame are two reference frames in the third reference frame list that are closest to the current frame in the time domain.

12. The method according to any one of claims 7 to 11, wherein: The predicting the current frame according to the second virtual reference frame includes: inserting the second virtual reference frame into the reference frame list of the current frame; The current frame is predicted according to the reference frame list.

13. The method according to claim 7, wherein: The first virtual reference frame generation module is a generation module based on a neural network; The first virtual reference frame generation module is configured to perform the following steps: Performing optical flow estimation based on the at least two reference frames to determine optical flow information of a current frame; Performing multi-scale feature extraction based on the at least two reference frames and the optical flow information to obtain a multi-scale feature map; The first virtual reference frame is obtained according to the multi-scale feature map.

14. The method according to claim 7, wherein: The step of adjusting the first virtual reference frame of the current frame according to the target adjustment factor to obtain a second virtual reference frame includes: determining, according to the target adjustment factor, a first weight of the first virtual reference frame and a second weight of a third reference frame of the current frame; The first virtual reference frame and the third reference frame are weightedly fused according to the first weight and the second weight to obtain the second virtual reference frame.

15. The method according to claim 14, wherein The method further comprises: The at least two reference frames of the current frame are weightedly fused to determine the third reference frame.

16. The method according to claim 14, wherein The method further comprises: The at least two reference frames of the current frame are input to a second virtual reference frame generation module, which outputs the third reference frame; The second virtual reference frame generation module is different from the first virtual reference frame generation module.

17. The method according to claim 1, wherein The method further comprises: When it is determined, based on the value of the third syntax element, that the virtual reference frame of the current frame is allowed to be adjusted, the code stream is decoded to determine the value of the first syntax element of the current frame.

18. The method according to claim 17, wherein The third syntax element comprises a sequence level syntax element.

19. A coding method, applied to an encoder, comprising: determining at least one candidate adjustment factor for the current frame when adjustment of the virtual reference frame of the current frame is allowed; adjusting the first virtual reference frame of the current frame according to the at least one candidate adjustment factor to obtain at least one second virtual reference frame; Performing cost calculation based on the original frame of the current frame and the first virtual reference frame to obtain a first generation value; Performing cost calculation based on the original frame of the current frame and the at least one second virtual reference frame to obtain at least one second generation value; Making a coding decision based on the first generation value and the at least one second generation value to determine a value of a first syntax element for indicating whether to adjust a virtual reference frame of a current frame; The value of the first syntax element is encoded, and the obtained encoded bits are written into a bitstream.

20. The method according to claim 19, wherein The performing a coding decision based on the first generation value and the at least one second generation value to determine a value of a first syntax element for indicating whether to adjust the virtual reference frame of the current frame includes: When the first cost value is an optimal cost value, determining that the value of the first syntax element is a first value, which is used to indicate that the virtual reference frame of the current frame is not adjusted; When the second cost value is the optimal cost value, the value of the first syntax element is determined to be a second value, which is used to indicate the adjustment of the virtual reference frame of the current frame.

21. The method according to claim 19, wherein The method further comprises: When the second cost value is the optimal cost value, the target adjustment factor corresponding to the optimal cost value is encoded, and the obtained encoded bits are written into the bitstream.

22. The method according to claim 21, wherein The encoding of the target adjustment factor corresponding to the optimal cost value includes: determining a value of a second syntax element of the current frame according to the target adjustment factor; The value of the second syntax element is encoded, and the obtained encoded bits are written into a bitstream.

23. The method according to claim 22, wherein The determining of at least one candidate adjustment factor for the current frame includes: Traverse the adjustment factor candidate set to obtain multiple candidate adjustment factors; The determining, according to the target adjustment factor, a value of the second syntax element of the current frame includes: Determine an index value of the target adjustment factor in the adjustment factor candidate set; A value of the second syntax element is determined according to the index value of the target adjustment factor.

24. The method according to claim 22, wherein The determining, according to the target adjustment factor, a value of the second syntax element of the current frame includes: Determining a reference value of the target adjustment factor according to the target adjustment factor; A value of the second syntax element is determined according to a reference value of the target adjustment factor.

25. The method according to claim 19, wherein The method further comprises: Obtain at least two reference frames of the current frame from the reference frame list of the current frame; The at least two reference frames are input to a first virtual reference frame generation module, which outputs the first virtual reference frame.

26. The method according to claim 25, wherein The acquiring at least two reference frames of the current frame from the reference frame list of the current frame includes: The current frame type is a bidirectional prediction frame, a first reference frame is obtained from the first reference frame list, and a second reference frame is obtained from the second reference frame list.

27. The method according to claim 26, wherein The first reference frame is a forward reference frame closest to the current frame in the first reference frame list, and the second reference frame is a backward reference frame closest to the current frame in the second reference frame list.

28. The method according to claim 25, wherein The acquiring at least two reference frames of the current frame from the reference frame list of the current frame includes: The current frame type is a unidirectional prediction frame, and the first reference frame and the second reference frame are obtained from the third reference frame list.

29. The method according to claim 28, wherein The first reference frame and the second reference frame are two reference frames in the third reference frame list that are closest to the current frame in the time domain.

30. The method according to any one of claims 25 to 29, wherein The predicting the current frame according to the second virtual reference frame includes: inserting the second virtual reference frame into the reference frame list of the current frame; The current frame is predicted according to the reference frame list.

31. The method of claim 25, wherein: The first virtual reference frame generation module is a generation module based on a neural network; The first virtual reference frame generation module is configured to perform the following steps: Performing optical flow estimation based on the at least two reference frames to determine optical flow information of a current frame; Performing multi-scale feature extraction based on the at least two reference frames and the optical flow information to obtain a multi-scale feature map; The first virtual reference frame is obtained according to the multi-scale feature map.

32. The method of claim 25, wherein: The step of adjusting the first virtual reference frame of the current frame according to the at least one candidate adjustment factor to obtain at least one second virtual reference frame includes: determining, based on the candidate adjustment factors, a first weight of the first virtual reference frame and a second weight of a third reference frame of the current frame; The first virtual reference frame and the third reference frame are weightedly fused according to the first weight and the second weight to obtain the second virtual reference frame.

33. The method according to claim 32, wherein The method further comprises: The at least two reference frames of the current frame are weightedly fused to determine the third reference frame.

34. The method of claim 32, wherein: The method further comprises: The at least two reference frames of the current frame are input to a second virtual reference frame generation module, which outputs the third reference frame; The second virtual reference frame generation module is different from the first virtual reference frame generation module.

35. The method of claim 32, wherein: The determining of at least one candidate adjustment factor for the current frame includes: Based on an adjustment factor derivation algorithm, candidate adjustment factors are determined according to the original frame of the current frame, the first virtual reference frame, and the third reference frame.

36. The method according to claim 35, wherein The adjustment factor derivation algorithm includes a least squares method.

37. The method according to claim 36, wherein Implement the following steps based on the least squares method: Determining a first residual feature map according to the original samples of the original frame and the reference samples of the third reference frame; Determining a second residual feature map according to the virtual reference samples of the first virtual reference frame and the reference samples of the third reference frame; Determining a first residual sum and a residual sum of squares according to the first residual feature map; Determining a second residual sum according to the second residual feature map; Determining a cross residual sum according to the first residual feature map and the second residual feature map; The candidate adjustment factor is determined according to the first residual sum, the residual square sum, the second residual sum, and the cross residual sum.

38. The method according to claim 37, wherein The determining the candidate adjustment factor according to the first residual sum, the residual sum of squares, the second residual sum, and the cross residual sum includes: Calculate the product of the height and width of the current frame and the cross residual sum to obtain a first parameter; Calculating the product of the first residual sum and the second residual sum to obtain a second parameter; Calculate the product of the height and width of the current frame and the residual square sum to obtain a third parameter; Calculate the square of the second residual sum to obtain a fourth parameter; The candidate adjustment factor is obtained by dividing the difference between the first parameter and the second parameter by the difference between the third parameter and the fourth parameter.

39. The method of claim 35, wherein: The adjustment factor derivation algorithm further performs the following steps: The candidate adjustment factor is clamped to within a preset upper limit and a lower limit of the adjustment factor.

40. The method of claim 19, wherein The determining of at least one candidate adjustment factor for the current frame includes: According to the target adjustment factor of the first image component of the current frame, a candidate adjustment factor of the second image component of the current frame is determined.

41. The method of claim 19, wherein The method further comprises: Whether adjustment of the virtual reference frame of the current frame is allowed is determined according to the value of the third syntax element.

42. The method according to claim 41, wherein The third syntax element comprises a sequence level syntax element.

43. An encoder comprising a first determining unit, a second determining unit, and an encoding unit; wherein: The first determining unit is configured to, when adjustment of the virtual reference frame of the current frame is allowed, determine at least one candidate adjustment factor of the current frame; and adjust the first virtual reference frame of the current frame according to the at least one candidate adjustment factor to obtain at least one second virtual reference frame; The second determining unit is configured to perform a cost calculation based on the original frame of the current frame and the first virtual reference frame to obtain a first generation value; perform a cost calculation based on the original frame of the current frame and the at least one second virtual reference frame to obtain at least one second generation value; make a coding decision based on the first generation value and the at least one second generation value to determine a value of a first syntax element used to indicate whether to adjust the virtual reference frame of the current frame; The encoding unit is configured to encode the value of the first syntax element and write the obtained coded bits into a bitstream.

44. An encoder comprising a first memory and a first processor; wherein: The first memory is used to store a computer program that can be run on the first processor; The first processor is configured to execute the method according to any one of claims 19 to 42 when running the computer program.

45. A decoder comprising a decoding unit, a third determining unit, and a prediction unit; wherein: The decoding unit is configured to decode the code stream and determine the value of the first syntax element of the current frame; The third determining unit is configured to, when determining the virtual reference frame for adjusting the current frame according to the value of the first syntax element, determine a target adjustment factor for the current frame; and adjust the first virtual reference frame of the current frame according to the target adjustment factor to obtain a second virtual reference frame; The prediction unit is configured to predict the current frame according to the second virtual reference frame.

46. A decoder comprising a second memory and a second processor; wherein: The second memory is used to store a computer program that can be run on the second processor; The second processor is configured to execute the method according to any one of claims 1 to 18 when running the computer program.

47. A computer-readable storage medium, wherein: The computer-readable storage medium stores a code stream generated by the encoding method according to any one of claims 19 to 42.

48. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, which, when executed, implements the method according to any one of claims 1 to 18 or the method according to any one of claims 19 to 42.

Images