Neural video coding

Abstract

According to implementations of the subject matter described herein, a solution for neural video coding is provided. According to the solution, a quantization parameter for a target frame is obtained. A quantization scaling value for the target frame is determined based on the quantization parameter and at least a first upper limit value and a first lower limit value for quantization scaling. The quantization scaling value is between the first upper limit value and the first lower limit value, and the first upper limit value and the first lower limit value are determined from a training process of a video codec model. Quantization is performed, using the video codec model, on the target frame based on the quantization scaling value in at least one of a frame encoding process or a motion vector encoding process of the target frame, to obtain a quantized representation of the target frame.

Description

NEURAL VIDEO CODING BACKGROUND

[0001] Traditional standard codec, which relies on a hybrid residual coding-based framework, has been in development for several years and is still being refined. However, the improvement in compression ratio has diminished, while the increase in complexity has grown significantly. This makes further advancements within the traditional framework more and more challenging. Recently, neural video codec (NVC) has gained considerable attention, as it holds the potential to break this development bottleneck. SUMMARY

[0002] According to implementations of the subject matter described herein, a solution for neural video coding is proposed. In this solution, a quantization parameter for a target frame of a video is obtained. A quantization scaling value for the target frame is determined based on the quantization parameter and at least a first upper limit value and a first lower limit value for quantization scaling. The quantization scaling value is between the first upper limit value and the first lower limit value, and the first upper limit value and the first lower limit value are determined from a training process of a video codec model. Quantization is performed on the target frame using the video codec model based on the quantization scaling value in at least one of a frame encoding process or a motion vector encoding process of the target frame, to obtain a quantized representation of the target frame. According to implementations of the subject matter described herein, a uniform sampling mechanism for the quantization scaling value is proposed, and the range of quantization scaling values used for sampling is learned in model training. In this way, a single model may be trained to support more flexible selection of quantization scaling values in video coding, thereby improving the coordination between encoding and quantization, and achieving a wider range of video quality.

[0003] The Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. The Summary is neither intended to identify key features or essential features of the subject matter described herein, nor is it intended to be used to limit the scope of the subject matter described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] FIG. 1 illustrates a block diagram of an example environment in which various implementations of the subject matter described herein can be implemented;

[0005] FIG.2 illustrates a schematic block diagram of a partial structure of a video codec model in accordance with some implementations of the subject matter described herein;

[0006] FIG.3 illustrates a schematic block diagram of an example architecture of a frame codingmodel in accordance with some implementations of the subject matter described herein;

[0007] FIG. 4A illustrates bit rate control in high bit rate scenarios in accordance with some implementations of the subject matter described herein;

[0008] FIG. 4B illustrates bit rate control in low bit rate scenarios in accordance with some implementations of the subject matter described herein;

[0009] FIG. 5 illustrates a schematic block diagram of an example architecture of a context extraction model in accordance with some implementations of the subject matter described herein;

[0010] FIG. 6 illustrates a comparison between the quality ranges in accordance with some implementations of the subject matter described herein and the quality ranges of other codecs;

[0011] FIG.7 illustrates a flowchart of a process for video processing in accordance with some implementations of the subject matter described herein; and

[0012] FIG. 8 illustrates a schematic block diagram of an electronic device in which various implementations of the subject matter described herein can be implemented.

[0013] Throughout the drawings, the same or similar reference symbols refer to the same or similar elements. DETAILED DESCRIPTION OF EMBODIMENTS

[0014] The subject matter described herein will now be described with reference to some example implementations. It is to be understood that these implementations are described only for the purpose of illustration and help those skilled in the art to better understand and thus implement the subject matter described herein, without suggesting any limitations to the scope of the subject matter described herein.

[0015] As used herein, the term “includes” and its variants are to be read as open terms that mean “includes but is not limited to.” The term “based on” is to be read as “based at least in part on.” The terms “an implementation” and “one implementation” are to be read as “at least one implementation.” The term “another implementation” is to be read as “at least one other implementation.” The term “first,” “second,” and the like may refer to different or the same objects. Other definitions, either explicit or implicit, may be included below.

[0016] As used herein, the term “model” may learn an association between corresponding input and output from training data, and thus a corresponding output may be generated for a given input after the training. The generation of the model may be based on machine learning techniques. Deep learning (DL) is one of machine learning algorithms that processes the input and provides the corresponding output using a plurality of layers of processing units. A neural network model is an example of a deep learning-based model. As used herein, “model” may also be referred to as “machine learning model”, “learning model”, “machine learning network” or “learning network”, which are used interchangeably herein.

[0017] Generally, machine learning may roughly include three stages, i.e., a training stage, a test stage, and an application stage (also referred to as an interference stage). In the training stage, a given model may be trained using a large scale of training data, with parameter values being iteratively updated until the model can obtain, from the training data, consistent interference that meets an expected target. Through the training, the model may be considered as being capable of learning the association between the input and the output (also referred to as an input-to-output mapping) from the training data. The parameter values of the trained model are determined. In the test stage, test inputs are applied to the trained model to test whether the model can provide correct outputs, so as to determine the performance of the model. In the interference stage, the model may be utilized to process a practical input based on the parameter values obtained from the training and to determine the corresponding output. Example Environment

[0018] FIG. 1 illustrates a block diagram of an example environment 100 in which various implementations of the subject matter described herein can be implemented. In the environment of FIG.1, an electronic device 110 includes a video codec 112 configured to encode and / or decode a video. An electronic device 120 includes a video codec 122 configured to encode and / or decode a video. The video codec 112 or 122 may include encoders and / or decoders. In the encoding, an encoder may encode a video 130 into a bitstream 132. In the decoding, a decoder may decode the bitstream 132 into the video 130.

[0019] The electronic devices 110 and 120 can communicate with each other through any appropriate communication network. In some codec scenarios, the electronic device 110 and the electronic device 120 may perform video communication, and the video codec 112 and 122 may both implement the encoding and decoding of the video. For example, the electronic device 110 may provide a bitstream obtained after video encoding to the electronic device 120 for decoding, and the electronic device 120 may decode the received bitstream to obtain the corresponding video. In addition, the electronic device 120 may also provide a video encoding result to the electronic device 110 for decoding. In some codec scenarios, the video codec 112 in the electronic device 110 may include an encoder for encoding a video into a bitstream. The electronic device 120 may include a video playback tool, where the video codec 122 includes a decoder for decoding the bitstream generated by the video codec 112 to obtain the video for playback.

[0020] It would be appreciated that the devices and elements shown in FIG.1 are only examples. In practical applications, there may exist more electronic devices, and each electronic device may have video encoding and / or decoding functions.

[0021] The early neural video codec (NVC) model still follows traditional codec and uses the residual coding-based framework, where all sub-modules are replaced by the neural network toachieve end-to-end learning. Later, many works are also based on this paradigm and propose stronger sub-modules to improve the performance.

[0022] By contrast, in the NVC application, conditional coding is proposed, which shows a lower entropy bound than residual coding and has larger potential. The condition may be freely defined and learned rather than being limited to the predicted frame in pixel domain. Meanwhile, the condition may be flexibly used to help encoding, decoding, and entropy modelling. Currently, the NVC based on conditional coding has achieved a better compression ratio than traditional codec by mining diverse spatial and temporal contexts as the condition.

[0023] Despite the progress made by some NVC, for example, supporting multiple quality levels in a single model, it is still far from practical usage, because its quality range is quite limited which may not meet the needs of a practical product. Therefore, it is expected to support multiple quality levels within a wider quality range in NVC.

[0024] According to example implementations of the subject matter described herein, an improved solution for video coding is proposed. This solution is implemented based on NVC. In this solution, for a target frame to be coded, a quantization scaling value for the target frame is determined based on the quantization parameter and at least a first upper limit value and a first lower limit value for quantization scaling. The quantization scaling value is between the first upper limit value and the first lower limit value. The determined quantization scaling value is used to obtain a quantized representation of the target frame. In neural video coding, the first upper limit value and the first lower limit value determined from the training process of the video codec model may be used to achieve the selection of quantization scaling values for the target frame. In this way, a single model may be trained to support more flexible selection of quantization scaling values in video coding, thereby improving the coordination between encoding and quantization, and achieving a wider range of video quality. Architecture of Video Codec Model

[0025] Some example implementations of the subject matter described herein will be described in more detail below with reference to the accompanying drawings.

[0026] FIG.2 illustrates a schematic block diagram of a partial structure of a video codec model 200 in accordance with some implementations of the subject matter described herein. The respective components in the video codec model 200 may be implemented in hardware, software, firmware or any combination thereof. The video codec model 200 may be implemented in the video codec 112 and / or the video codec 122 of FIG. 1. It is to be noted that FIG. 2 only shows a partial structure of the entire video codec model, and the model may have more other components.

[0027] The video codec model 200 generally includes a motion vector coding (containing motion estimation) model 210, a context extraction model 220, and a frame coding model 230. Thosemodels may be implemented based on the machine learning techniques, for example, the neural network architecture. To obtain a higher compression ratio, the video codec model 200 is implemented based on condition coding, which is more flexible and can guide the coding of frames under the condition of the extracted context information.

[0028] The video codec model 200 can implement a conversion between each frame in a video and a bitstream of the video. The conversion includes an encoding process of the video, a decoding process of the video, or both. In the encoding process, the video codec model 200 receives a sequence of frames of the video, and can perform video coding on respective frames to obtain the bitstream of the video. In the decoding process, the video codec model 200 receives the bitstream of the video and decodes the sequence of frames of the video. In the following, except for the operations that are explicitly specified, other operations may be considered to be performed at both the encoding and decoding sides of the video. For ease of understanding, the basic working principle of the video codec model 200 will be briefly described in the following.

[0029] In this specification, a target frame xt refers to the current frame to be coded in the video. As shown in FIG. 2, to encode and decode the target frame xt with an index t, the motion vector coding model 210 is configured to determine motion information vt of the target frame xt based on a quantization parameter qt of the target frame, encode the motion information vt and then decode it as estimated motion information ^^^௧. The above operations are performed at the encoding side of the video. At the decoding side of the video, the encoded motion information vtmay be transmitted to the decoding side for decoding the estimated motion information ^^^௧. The processing of the motion vector coding model 210 is represented as fmotion.

[0030] The motion information indicates motion offsets of elements in the target frame xtrelative to a reference frame, including the offset sizes and directions. For example, the motion information may include a motion vector (MV). The reference frame may also be used as an input for the motion estimation of the target frame xt. In some implementations, the reference frame of the target frame xt may be one or more previous frames of the target frame xt. In the following implementations, only a single reference frame is taken as an example for illustration, although a plurality of reference frames are also feasible. As shown in FIG.2, a previous frame xt-1 before the target frame xtor its decoded reconstruction result is used as the reference frame. In some implementations, the motion vector coding model 210 may be configured to determine the estimated motion information ^^^௧based on an optical flow network. In addition to the optical flow network, the motion vector coding model 210 may also be implemented based on any other appropriate models that can determine the motion information of the frame.

[0031] The context extraction model 220 is configured to determine context information Ctfor the target frame xt. In the implementations of the subject matter described herein, the contextextraction model 220 is configured to determine the context information Ctfor the target frame xtbased on the estimated motion information ^^^௧of the target frame xtand the relevant information of the reference frame (the frame xt-1). The used relevant information of the reference frame includes reference feature information Ft-1and a reference reconstructed frame ^^^௧ି^for the reference frame xt-1. The reference feature information Ft-1 of the reference frame xt-1 may characterize feature parameter representations of the reference frame in the feature space of the reference frame. The reference reconstructed frame ^^^௧ି^is the reconstruction result for the reference frame xt-1. In the encoding and decoding, the respective frames will be reconstructed. The reference feature information Ft-1 and the reference reconstructed frame ^^^௧ି^can complement each other to provide richer and more relevant context information of the target frame xt. Moreover, on the basis of the estimated motion information ^^^௧of the target frame xt, the context extraction model 220 can extract motion-aligned context information Ct. The processing of the context extraction model 220 is represented as fTcontext.

[0032] The frame coding model 230 is configured to generate a target reconstructed frame ^^^௧of the target frame xtbased on at least the quantization parameter qtof the target frame and the context information Ct. In addition, the frame coding model 230 is further configured to generate target feature information Ftof the target frame based on at least the context information Ct. The target feature information Ft and the target reconstructed frame ^^^௧are buffered and transferred for coding of a next frame until the coding of the whole video is completed. At the decoding side, the target reconstructed frame ^^^௧may be output as the decoding target.

[0033] At the encoding side, conditioned on the context information Ct, the frame coding model 230 may encode the target frame xt into a quantized code representation ^^^௧. At the decoding side, the quantized code representation ^^^௧may be determined from the bitstream of the video. After entropy-coding on the quantized code representation ^^^௧, the target reconstructed frame ^^^௧and the target feature information Ft are reconstructed based on the entropy coding result. The processing of the frame coding model 230 is represented as fframe.

[0034] FIG. 2 illustrates the coding pipeline for respective frames xt-2, xt-1, xt and so on by the video codec model 200. For each frame, the motion vector coding model 210, the context extraction model 220, and the frame coding model 230 in the model 200 perform the similar operations.

[0035] It should be understood that FIG.2 illustrates that the video codec model 200 is a part of the codec and not a complete codec.

[0036] The overall workflow of the neural video codec system 200 is described above. Example implementations of the performing quantization and inverse quantization, and determining the quantization scaling value and the inverse quantization scaling value will be discussed in moredetail below. Example Implementations of Quantization and Inverse Quantization

[0037] In example implementations of the present disclosure, an improved solution for neural video coding is proposed. In the process of video coding, it is usually necessary to perform quantization and inverse quantization on video frames.

[0038] The basic processes of quantization and inverse quantization may be represented as follows: ^^^ ൌ ^^^^ ∙ ⌊^^ (1) ⌉ ^^^^ where ^^ represents step, ⌊∙⌉ represents arounding operation. of output ^^^.

[0039] In this solution, for a target frame xtto be coded, the quantization scaling value for the target frame xt is determined based on the quantization parameter qt and a first upper limit value and a first lower limit value for quantization scaling. The quantization scaling value is determined to be between the first upper limit value and the first lower limit value, and the first upper limit value and the first lower limit value are determined from the training process of the video codec model 200. The determined quantization scaling value is used to obtain a quantized representation of the target frame xt.

[0040] Similarly, after obtaining the quantized representation of the target frame xt, in the process of inverse quantization, the inverse quantization scaling value for the target frame xt may be determined based on the quantization parameter qtand a second upper limit value and a second lower limit value for quantization scaling. The inverse quantization scaling value is determined to be between the second upper limit value and the second lower limit value, and the second upper limit value and the second lower limit value are determined from the training process of the video codec model 200. The determined inverse quantization scaling value is used to perform inverse quantization on quantized code representation corresponding to the quantized representation.

[0041] In the neural video coding, the first upper limit value, the first lower limit value, the second upper limit value and the second lower limit value determined from the training process of the video codec model 200 are used to achieve the selection of quantization scaling values for the target frame. In this way, by sampling the quantization scaling value and the inverse quantization scaling value, the video codec model 200 may experience various tradeoffs between bit rate and distortion. This can enhance the harmonization between encoding and quantization processes, thereby supporting a wider quality range and bitrate range.

[0042] The quantization and inverse quantization mechanisms proposed by example implementations of the present disclosure may be applied in the frame coding model 230 and themotion vector coding model 210 of the video codec model 200. In an example, the example implementation of the quantization and inverse quantization mechanisms proposed by the present disclosure will be discussed in conjunction with the frame coding model 230 in detail below. It should be understood that the quantization and inverse quantization mechanisms proposed by the present disclosure will be similarly applied to the motion vector coding model 210.

[0043] FIG.3 illustrates a schematic block diagram of an example architecture of a frame coding model 230 in accordance with some implementations of the subject matter described herein. As illustrated in the example of FIG. 3, an encoder 310 may include Ehigh 312, a quantizer 313 and Elow314, where Ehigh312 indicates an encoder processing high resolution, Elow314 indicates an encoder processing low resolution, and the quantizer 313 is configured to perform quantization on the output result of Ehigh312 and provide the quantization result to Elow314 as an input. A decoder 320 may include Dhigh 322, an inverse quantizer 323 and Dlow 324, where Dhigh 322 indicates a decoder processing high resolution, Dlow324 indicates a decoder processing low resolution, and the inverse quantizer 323 is configured to perform inverse quantization on the output result of Dlow 324 and provide the inverse quantization result to Dhigh 322 as an input. At the decoding side of the video, the encoder 310 may be omitted. In some implementations, the encoder 310 may include any number of encoders and the decoder 320 may include any number of corresponding decoders.

[0044] As shown in FIG. 3, to perform quantization on the target frame xt, the quantization parameter qtfor the target frame xtin the video may be obtained. Here, the quantization parameter qt may be set by a user or set by default in the video codec model 200.

[0045] The selection of quantization scaling has an impact on video quality and compression efficiency. The first upper limit value ^^^^^^^௫ and the first lower limit value ^^^^^^^^ for quantization scaling may ensure that the videoand compression efficiency for the target frame xt are within an appropriate range. The quantization scaling value ^^௧^^^of the target frame xt may be determined based on the quantization parameter qt, the first upper limit value ^^^^^^^௫ and the first lower limit value ^^^^^^^^ . The quantization scaling value ^^௧^^^is betweenfirst upper limit value ^^^^^^^௫ and the first lower limit value ^^^^^^^^ , and the first upper limit value ^^^^^^^௫ and the first lower value ^^^^^^^^ are determinedtraining process of the video codec model 200. The quantizationvalue ^^௧^^^may be used to scale the latent feature.

[0046] After determining the quantization scaling value ^^௧^^^, using the frame coding model 230 in the video codec model 200, in the frame encoding process of the target frame xt, the quantization scaling value ^^௧^^^may be used to perform quantization on the output result of Ehigh 312, to obtain the quantized representation ^^௧of the target frame xt.

[0047] In some implementations, the quantization scaling value ^^௧^^^may be used to performquantization on the output result of Elow314, to obtain the quantized representation of the target frame xt. The quantization scaling value ^^௧^^^may be used to perform quantization on the target frame xt at any position of the encoder 310, and implementation of the present disclosure are not limited in this regard.

[0048] Similarly, the quantization operation performed on the target frame xt in the frame encoding process may be applied to the motion vector coding model 210 in the video codec model 200. Using the motion vector coding model 210, in the motion estimation and motion vector encoding process of the target frame xt, the quantization scaling value ^^௧^^^may be used to perform quantization on the target frame xt, to obtain the quantized representation ^^௧of the target frame xt.

[0049] Although the quantizer 313 is shown between Ehigh312 and Elow314 in FIG. 3, the quantizer 313 may also be located after Elow 314 depending on the specific model design. In addition, although only a single quantizer is shown in FIG.3, in actual model design, the encoder 310 may include a plurality of network processing layers, and similar quantizers may be deployed in the plurality of network processing layers. For quantizers deployed at different network processing layers, the upper and lower limits of quantization scaling may be configured to be a pair of identical values, or multiple pairs of different values can be learned during the training process of the video codec model 200.

[0050] In some implementations, as shown in FIG.3, after performing quantization on the target frame xt, a corresponding inverse quantization may be performed to restore the information of the original target frame xt to a certain extent. The selection of inverse quantization scaling has an impact on the recovery of original video information. The second upper limit value ^^^ௗ^^^௫ and the second lower limit value ^^^ௗ^^^^ for inverse quantization scaling may ensure that the information recovery for the target frame xt is within an appropriate range. The inverse quantization scaling value ^^௧ௗ^^of the target frame xt may be determined based on the quantization parameter qt, the second upper limit value ^^^ௗ^^^௫ and the second lower limit value ^^^ௗ^^^^ . The inverse quantization scaling value ^^௧ௗ^^is between the second upper limit value ^^^ௗ^^^௫ and the second lower limit value ^^^ௗ^^^^ ,second upper limit value ^^^ௗ^^^௫ and thelower limit value ^^^ௗ^^^^ are from the training process of the video codec model 200.

[0051] After determining the inverse quantization scaling value ^^௧ௗ^^, in the frame encoding process of the target frame xt, the inverse quantization scaling value ^^௧ௗ^^may be used to perform inverse quantization on the output result of Dlow 324, thereby performing inverse quantization on the quantized code representation ^^^௧corresponding to the quantized representation ^^௧.

[0052] In some implementations, the inverse quantization scaling value ^^௧ௗ^^may be used toperform inverse quantization on the input of Dlow324, thereby performing inverse quantization on quantized code representation ^^^௧corresponding to the quantized representation ^^௧. The inverse quantization scaling value ^^௧ௗ^^may be used to perform inverse quantization on the quantized code representation ^^^௧at any position of the decoder 320, and implementation of the present disclosure are not limited in this regard.

[0053] Similarly, the above inverse quantization operation performed on the quantized code representation ^^^௧in the frame encoding process may be applied to the motion vector coding model 210 in the video codec model 200. In the motion estimation and motion vector coding process of the target frame xt, inverse quantization may be performed on the quantized code representation ^^^௧based on the inverse quantization scaling value ^^௧ௗ^^.

[0054] Although the inverse quantizer 323 is shown between Dhigh322 and Dlow324 in FIG. 3, the inverse quantizer 323 may also be located before Dlow 324 depending on the specific model design. In addition, although only a single inverse quantizer is shown in FIG. 3, in actual model design, the decoder 320 may include a plurality of network processing layers, and similar inverse quantizers may be deployed in the plurality of network processing layers. For inverse quantizers deployed at different network processing layers, the upper and lower limits of inverse quantization scaling may be configured to be a pair of identical values, or multiple pairs of different values can be learned during the training process of the video codec model 200. Example Implementations Of Quantization Scaling Value and Inverse Quantization Scaling Value Determination

[0055] In some implementations, interpolation may be performed, using an interpolation function, between the first upper limit value ^^^^^^^௫ and the first lower limit value ^^^^^^^^ based on at least the quantization parameter qt,the quantization scaling value ^^௧^^^. The interpolation function may be represented as a function related to qt, ^^^^^^^௫ and ^^^^^^^^ as follows: ^^௧^^^=f(^^^^^^^^, ^^^^^^^௫, qt)(2)where f() represents the interpolation function, for example, f() may be an exponential function, logarithmic function, linear function, quadratic function, or any combination thereof, which has monotonicity between the first upper limit value ^^^^^^^௫ and the first lower limit value ^^^^^^^^ . In this way, the quantization scaling value ^^௧^^^sampled between the firstvalue ^^^^^^^௫ and the first lower limit value ^^^^^^^^ and thus the video codec model 200 may experience different quantization scaling valuesand various tradeoffs between bit rates and distortions, thereby enhancing the harmonization between encoding and quantization and supporting a wider quality range.

[0056] In some implementations, the quantization parameter qtmay be selected from a range of quantization parameters [0, q_num-1], where q_num represents the number of values to beselected for the quantization parameter qt. The q_num may be input by the user or preset in the video codec model 200, for example, the value of q_num may be set to 64. After selecting the quantization parameter qtfrom the range of quantization parameters, interpolation may be performed, using the interpolation function, between the first upper limit value ^^^^^^^௫ and the first lower limit value ^^^^^^^^ based on the quantization parameter qt and the upper limit value of the range of quantization parameters q_num-1, to obtain the quantizationvalue ^^௧^^^. An example of thefunction is as follows: ^^௧^^^=f(^^^^^^^^ , ^^^^^^^௫ , qt, q_num-1) (3)

[0057] In some implementations, the interpolation function may be configured to exponentially increase the quantization scaling value ^^௧^^^from the first lower limit value ^^^^^^^^ to the first upper limit value ^^^^^^^௫ as the quantization parameter qt linearly increases within range of thequantization parameters [0, q_num-1]. In some implementations, the function is configured in the form of an exponential or logarithmic function, where the exponent of the exponential function is determined based on the quantization parameter qt and the range of the quantization parameters [0, q_num-1] and the base of the exponential function is determined based on the first upper limit value ^^^^^^^௫ and the first lower limit value ^^^^^^^^ .

[0058] Specifically, the interpolation function f() may be the function, an exampleof the exponential function is as follows: ^^^^^ ^^^^^^^^௫ ^^(4A) ^^௧ ൌ ^^^^^ ∙ ^^^_^௨^ି^^^^^^^^^

[0059] In (4A) may be represented as alogarithmic function, an example of which is as follows: ^^^^^௧ ൌ ^^^^ ^^^^^^^ ^^ ^௧ ^^^^^^ (4B)^1 ∙ ^^^^^ ^^^^௫ െ ^^^^ ^^^^^ ^

[0060] as in Eq. (4B) may provide better numerical stability and support uniform sampling in the logarithmic domain.

[0061] The above provides some examples of interpolation functions, but it may be understood that other interpolation functions may be involved as needed in different applications. In some implementations, besides exponential or logarithmic functions, other functions such as linear interpolation functions may also be used. In an example, the linear interpolation function may be: ^^^^^ ൌ ^^^^^^^ ^௧ ∙ ^^^^^^ െ ^^^^^^^^(5)

[0062] Inupper limit value ^^^^^^^௫ and the first lower limit value ^^^^^^^^ need to be learned to limit the range of quantization scaling values in the quantizationIn some implementations, interpolation functions related to two or more parameters may also be designed, and these parameter values may be learned duringthe learning process of video codecs. For example, the interpolation of the scaled quantization value ^^௧^^^may be a line segment interpolation function, which includes two or more interpolation functions used to characterize falling into two or more interpolation scaling value intervals. Different interpolation functions may be used to determine the quantization parameter qt that falls into different value ranges.

[0063] Assuming that the line segment interpolation includes two line segment interpolation functions, the parameter values to be learned include ^^^^^ ^^^ ^^^^^^ , ^^^^ௗ , ^^^^௫ , where ^^^^^^^ௗ is an intermediate value for quantization scaling. For the quantization qt that falls within the range of quantization parameters [0, qmid], a first line segment function related ^^^^^^^^ and ^^^^^^^ௗ may be used to determine the quantization scaling value ^^௧^^^and thisbe similar to the form of Eq. (4A), Eq. (4B) or Eq. (5) above, except that௫is ^^^^^^^ௗ . For the quantization parameter qt that falls within the range of ers [qmid, q_num-1], a second line segment interpolation function related ^^^^^parametௗ^^^^^^^௫ may be used to determine the quantization scaling value ^^௧^^^and this interpolation function can be similar to the form of Eq. (4A), Eq. (4B) or Eq. (5) above, except that ^^^^^^^^ is replaced with ^^^^^^^ௗ . The types of the first interpolation function and the second interpolation function can be the same or different.

[0064] In some implementations, after determining the quantization scaling value ^^௧^^^by using the interpolation function, interpolation may be performed, using the same interpolation function, between the second upper limit value ^^^ௗ^^^௫ and the second lower limit value ^^^ௗ^^^^ based on the quantization parameter qt, to obtain the inverse quantization scaling valueexample, the interpolation function used to determine the quantization scaling value ^^௧^^^based on the quantization parameter qt is the interpolation function shown in Eq. (2), then interpolation may be performed, using Eq. (2), between the second upper limit value ^^^ௗ^^^௫ and the second lower limit value ^^^ௗ^^^^ based on the quantization parameter qt, to obtain the inverse quantization scaling value ^^௧ௗ^^.

[0065] In some implementations, after determining the quantization scaling value ^^௧^^^by using the interpolation function, interpolation may be performed, using the same interpolation function, between the second upper limit value ^^^ௗ^^^௫ and the second lower limit value ^^^ௗ^^^^ based on the quantization parameter qt and the upper limit value q_num-1 of theof quantization parameters, to obtain the inverse quantization scaling value ^^௧ௗ^^. For example, the interpolation function used to determine the quantization scaling value ^^௧^^^based on the quantization parameter qt and the upper limit value q_num-1 of the range of quantization parameters is the interpolation function shown in Eq. (4B), then interpolation may be performed,using Eq. (4B), between the second upper limit value ^^^ௗ^^^௫ and the second lower limit value ^^^ௗ^^^^ based on the quantization parameter qt and the upper limit value q_num-1 of the range of quantization parameters, to obtain the inversescaling value ^^௧ௗ^^. The interpolation function used may also satisfy that as the quantization parameter qt linearly increases within the range of quantization parameters [0, q_num-1], the inverse quantization scaling value ^^௧ௗ^^exponentially increases from the second lower limit value ^^^ௗ^^^^ to the second upper limit value ^^^ௗ^^^௫.

[0066] The quantization and inverse quantizationin the present disclosure are performed at the latent feature domain, and it is not required that the quantization scaling value ^^௧^^^used during encoding and the inverse scaling value ^^௧ௗ^^used during decoding be the same. The inverse scaling value ^^௧ௗ^^corresponding to the quantization scaling value ^^௧^^^is independently learned during decoding and thus the codec model 200 may have greater flexibility to smoothly adjust quality. In addition, during the encoding process, the quantization scaling value ^^௧^^^may be multiplied by the latent feature instead of being used as a divisor, making the training process of the video codec model 200 more stable by avoiding the potential issue of division by zero.

[0067] During the training process of the video codec model 200, a loss function may be used as an objective function for optimizing the video codec model 200. By minimizing the loss function, the parameters of the video codec model 200 may be updated to optimize the video codec model 200. In some implementations, the loss function used by the video codec model 200 is determined by using a weighting factor to weight between a bit rate loss and a distortion loss. The weighting factor may be determined based on the quantization parameter qt, a weighting upper limit value λmax and a weighting lower limit value λmin. The weighting factor is used to control the tradeoffs between a bit rate loss and a distortion loss.

[0068] Specifically, in the loss function, the weighting factor may be used to weight the distortion loss. An example of such loss function is as follows: LossRD = R + λ D (6) where R represents the bit rate loss, D represents the distortion loss and λ represents the weighting factor.

[0069] In some implementations, the weighting factor in the loss function may be used to weight the bit rate loss. An example of such loss function is as follows: LossRD =λ R +D (7)

[0070] The training of video codec model 200 involves continuously optimizing model parameters to continuously reduce the value of the loss function until the value of the loss function is minimized or reaches a predetermined goal.

[0071] In some implementations, interpolation may be performed, using a same interpolation function as used in determining the quantization scaling value ^^௧^^^, between the weighting upper limit value λmax and the lower limit value λmin based on the quantization parameter qt, to obtain the weighting factor λ. For example, the interpolation function used in determining the quantization scaling value ^^௧^^^is the interpolation function shown in Eq. (4B), then the interpolation function used to determine the weighting factor λ is as follows: ^^ ൌ ^^^^ ఒ^^^ ^^^௧ (8)^^_^^^^^^ െ 1 ∙ ^^^^^ ^^^^^^^^ െ ^^^^ ^^^^^^^^^model 200, the weightingupper limit value λmax and the weighting lower limit value λmin are hyperparameters. The weighting upper limit value λmax may be set to 768 and the weighting lower limit value λmin may be set to 1. For each training step, the quantization parameter qt is uniformly sampled from the range of quantization parameters [0, q_num-1], and then the weighting factor λ may be obtained via the interpolation function (8) and the value of the loss function may be calculated via the loss function (6) or (7). The range between the weighting upper limit value λmax and the weighting lower limit value λmin ([λmin, λmax]) will guide the learning of the range between the first upper limit value ^^^^^^^௫ and the first lower limit value ^^^^^^^^ ([^^^^^^^௫ , ^^^^^^^^ ]) via backpropagation of the value of function. Therefore,the range quantization scaling value ^^௧^^^may be adjusted by controlling the range of thevalue of the weighting factor λ, which causes the video codec model 200 to experience various tradeoffs between bit rates and distortion, thereby enhancing the harmonization between encoding and quantization and supporting a wider quality range.

[0073] For all spatial positions, the quantization scaling value ^^௧^^^is the same, which may ignore the spatial features of the video content. Therefore, it is also necessary to use a spatial- channel-wise quantization scaler ^^௧^^^which may be adjusted for different positions. As shown in FIG. 3, an entropy model 338 determines the spatial-channel-wise quantization scaler ^^௧^^^based on the quantized code representation ^^^௧ି^for the reference frame xt-1, hyperprior information ^̂^௧and / or context information Ctfor the reference frame xt-1. The spatial-channel- wise quantization scaler ^^௧^^^may be used to perform quantization on the quantized representation yt, and the quantization result may be provided to an arithmetic encoder (AE) 332. AE 332 generates bitstream 334 of the video. The bitstream 334 is decoded by an arithmetic decoder (AD) 336 to obtain the quantized code representation ^^^௧. On the decoding device side of the video, AE 332 may not be required, and the decoding device may start from the received bit stream 334. The quantized code representation ^^^௧may be decoded by the AD 336 and provided to the decoder 320. In this way, the spatial-channel-wise quantization scaler ^^௧^^^not only helpsto achieve precise adjustment at each spatial position, but also adapts to the video content of each frame, which can improve the final compression efficiency.

[0074] Based on the above implementations, video codec models implemented in the present disclosure may support quality adjustment over a wider range of quality, which is also a prerequisite for controlling bit rate. Controlling bit rate is the core function of the implemented codec. FIGS. 4A and 4B respectively illustrate bit rate control in high bit rate scenarios and low bit rate scenarios in accordance with some implementations of the present disclosure. As shown in FIG. 4A, the curve 410 shows the variation of the target bit rate with the frame index and the curve 412 shows the variation of the actual bit rate with the frame index. As shown in FIG. 4B, the curve 420 shows the variation of the target bit rate with the frame index and the curve 422 shows the variation of the actual bit rate with the frame index. Due to the fact that video codec models implemented in the present disclosure may adjust the quantization parameter qt for each frame, the fluctuating target bit rate may be supported, and the actual bit rate may approach the target bit rate. It may be seen that video codec models implemented in the present disclosure support bit rate control, which may control video quality, balance video quality with data transmission requirements, and improve user experience. Example Implementations of Context Extraction

[0075] In addition, in NVC, a long prediction chain may lead to the temporal error accumulation, causing a decrease in video quality. To alleviate this issue, one solution is to use a small intra- period setting to insert high-quality intra (I)-frame more frequently, but this will decrease the compression efficiency. Therefore, it is also expected to reduce error accumulation in NVC.

[0076] The temporal quality degradation is a fundamental problem for all video codecs, but is especially serious for NVC. Therefore, it is also expected to reduce temporal quality degradation caused by temporal error accumulation. FIG. 5 illustrates a schematic block diagram of an example architecture of a context extraction model 220 in accordance with some implementations of the subject matter described herein. As shown in FIG.5, the context extraction model 220 may include a feature extractor 510 for the reference reconstructed frame ^^^t-1, a feature extractor 520 for the reference feature information Ft-1 of the reference reconstructed frame, a refresh control 530, a motion alignment unit 540 and a context generation unit 550.

[0077] Usually, the context extraction model 220 extracts features from the reference feature information Ft-1 by the feature extractor 520, thereby generating the context information Ct of the target frame xt. The accumulated error may have an impact on the propagated features (such as the reference feature information Ft-1), or the propagated features may contain some irrelevant information, resulting in lower quality context information Ct.

[0078] Due to the fact that the reference reconstructed frame ^^^t-1of the target frame xtonlyinformation of three dimensions, it has less information compared to the reference feature information Ft-1 of reference reconstructed frame, feature information containing less temporal accumulated errors may be extracted from the reference reconstructed frame ^^^ t-1. In some implementations, the context extraction model 220 in the video codec model is used, and in a context extraction process of the target frame xt, it is necessary to determine whether to extract feature information for the target frame xtfrom the reference reconstructed frame ^^^t-1of the target frame xt, or to extract feature information for the target frame xt from the reference feature information Ft-1of reference reconstructed frame.

[0079] In an example, an update period may be set to determine whether to extract feature information from the reference reconstructed frame ^^^t-1 or extract feature information from the reference feature information Ft-1. For example, assuming that after 32 cycles, the reference feature information Ft-1 will contain a large amount of accumulated temporal errors, the update period may be set to 32. In a case where the update period has expired, feature information for the target frame xt is extracted, using the refresh control 530, from the reference reconstructed frame ^^^t-1via the feature extractor 510. In a case where the update period has not expired, feature information for the target frame xt is extracted, using the refresh control 530, from the reference feature information Ft-1via the feature extractor 520. In some implementations, the update period may be adaptive to the content of the video. For example, for videos with significant changes between each frame, a shorter update period may be set. By periodically refreshing temporal features, the quality of video coding may be improved.

[0080] In some implementations, the update period for context information may be determined based on the content type of the video. In an example, if the visual changes of a video are minimal and the content is relatively stable (for example, a landscape video containing natural scenery, a still life video mainly capturing stationary objects, a surveillance video used for security monitoring, etc.), then the context information of the video changes less, and a longer update period for context information may be set for the video. If the visual changes of a video are frequent and the content is relatively active (such as sports videos, special effects videos, music videos, etc.), then the context information of the video changes more, and a shorter update period for context information may be set for the video.

[0081] In an example, a neural network may be constructed to determine whether to extract feature information from the reference reconstructed frame ^^^t-1or extract feature information from the reference feature information Ft-1. By training the neural network, the neural network may be caused to determine whether to extract feature information from the reference reconstructed frame ^^^t-1or extract feature information from the reference feature information Ft-1 based on the input video.

[0082] After extracting the feature information for the target frame xt, context information Ctfor the target frame xt may be determined based on the feature information. As shown in FIG.5, after extracting the feature information for the target frame xtby using the refresh control 530, the feature information may be fed back to the motion alignment unit 540. The motion alignment unit 540 processes based on the feature information and estimated motion information ^^^௧of the target frame xtand provide the feature information after motion alignment to the context generation unit 550, so that the context information Ct for the target frame xt is generated. The context information Ctis used in the frame encoding process of the target frame xt. In this way, by setting the update period for context information and periodically refreshing the feature information for the target frame, the long-term problem of temporal error accumulation may be alleviated, and the quality of video encoding and decoding may be improved.

[0083] In some implementations, it is possible to determine whether to extract feature information from the reference reconstructed frame ^^^t-1 or from the reference feature information Ft-1by determining whether the difference between multiple reconstructed frames before the target frame xt exceeds a difference threshold. The index of the target frame xt is t, multiple reconstructed frames before the target frame xt, for example, may be the reconstructed frame xt-1with index t-1, the reconstructed frame xt-2 with index t-2, the reconstructed frame xt-3 with index t-3, etc. In an example, the difference between multiple reconstructed frames before the target frame xtmay be the difference between the reconstructed frame xt-1 and the reconstructed frame xt-2, or the difference between the reconstructed frame xt-2and the reconstructed frame xt-3, etc. In accordance with a determination that the difference exceeds the difference threshold, it may be determined that there is a significant change in context information, thereby extracting feature information for the target frame from a reference reconstructed frame ^^^௧ି^for the target frame. In accordance with a determination that the difference does not exceed the difference threshold, it may be determined that there is not a significant change in context information, thereby extracting the feature information for the target frame from reference feature information Ft-1of the reference reconstructed frame.

[0084] After extracting the feature information for the target frame xt, the context information Ctfor the target frame xt may be determined based on the feature information. The context information Ctis used in a frame encoding process of the target frame xt. In this way, by comparing the difference thresholds, the timing of refreshing the feature information of the target frame is determined, and thus the efficiency of refreshing the feature information for the target frame is improved, the long-term problem of temporal error accumulation may be alleviated, and the quality of video encoding and decoding may be improved.

[0085] In some implementations, both periodic refreshing the feature information for the targetframe and refreshing the feature information for the target frame by comparing the difference threshold may be used simultaneously. For example, if either of these two ways satisfies the conditions, the feature information for the target frame may be refreshed. In some implementations, these two ways may be used interchangeably to refresh the feature information for the target frame. Example Implementations of Model Training

[0086] In the scenario where a long prediction chain leads to the temporal error accumulation, in addition to refreshing the feature information for the target frame, a longer video may also be used to train the video codec model 200 to improve the quality of video coding. In some implementations, during the training process of the video codec model 200, training data of the video codec model 200 comprises at least one sample video segment with a number of frames exceeding a predetermined number threshold. In an example, the predetermined number threshold may be set to numbers such as 1000, 2000, 3000, etc. After a period of training, whether the performance of the video codec model 200 has improved is determined, and the predetermined number threshold is adaptively updated. In this way, the video codec model 200 may recognize patterns in longer videos, explore temporal correlations better, and thus improve the quality of video coding when using the video codec model 200 in practice.

[0087] To improve the versatility of the video codec model 200, the video codec model 200 may be trained to support a plurality of color spaces simultaneously without additional fine-tuning. In some implementations, during the training process of the video codec model 200, a loss function may be determined at least based on a distortion loss, and the distortion loss may comprise a distortion loss of a first sample video in a first color space and a distortion loss of a second sample video in a second color space. The first sample video and the second sample video may be the same video or different videos. The first color space, for example, may be the YUV color space, and the second color space, for example, may be the RGB color space. An example of the loss function based on the distortion loss is as follows: ^^^^^^^^^^^^ ൌ ^^ ^ ^^ ∙ ^^^ ∙ ^^YUV ^ ^1 െ ^^^ ∙ ^^RGB^ ∙ ^^YUV (9)where ^^YUV represents the distortion loss in the YUV color space (i.e., the distortion loss of the first sample video in the first color space), ^^RGB represents the distortion loss in the RGB color space (i.e., the distortion loss of the second sample video in the second color space), and ^^ represents the hyperparameter used for weighting, which may, for example, be set to 0.8. Of course, the hyperparameter ^^ may also be set to any other value within the range of 0 to 1, and the scope of the implementations of the present disclosure is not limited to this.

[0088] In this way, the loss function simultaneously covers the distortion loss in the YUV color space and the distortion loss in the RGB color space, which may support both color spaces in a single model and improve the versatility of the video codec model 200.

[0089] FIG. 6 illustrates a comparison between the quality ranges in accordance with some implementations of the subject matter described herein and the quality ranges of other codecs. As shown in FIG.6, the curve 610 shows the peak signal-to-noise ratio (PSNR) range in accordance with some implementations of the subject matter described herein, the curves 612-616 show the PSNR range of traditional video codec models, and curve 618 shows the PSNR range of previous neural video codec models. PSNR is an indicator of image quality. It may be seen that compared with other video codec models, the video codec models in accordance with some implementations of the subject matter described herein have a wider quality range, and PSNR increases with the increase of bit per pixel (BPP), maintaining a monotonic upward trend, which may meet practical requirements. Example Process

[0090] FIG. 7 illustrates a flowchart of a process 700 for video processing in accordance with some implementations of the subject matter described herein. The process 700 may be implemented at the video codec model 200 of FIG. 2, which may be applied, for example, by the electronic device 110 or 120 in FIG. 1.

[0091] At block 710, the video codec model 200 obtains a quantization parameter for a target frame of a video.

[0092] At block 720, the video codec model 200 determines a quantization scaling value for the target frame based on the quantization parameter and at least a first upper limit value and a first lower limit value for quantization scaling, the quantization scaling value being between the first upper limit value and the first lower limit value, and the first upper limit value and the first lower limit value being determined from a training process of a video codec model.

[0093] At block 730, the video codec model 200 performs, using the video codec model, quantization on the target frame based on the quantization scaling value in at least one of a frame encoding process or a motion vector encoding process of the target frame, to obtain a quantized representation of the target frame.

[0094] In some implementations, the process 700 further comprises: determining an inverse quantization scaling value for the target frame based on the quantization parameter and at least a second upper limit value and a second lower limit value for inverse quantization scaling, the inverse quantization scaling value being between the second upper limit value and the second lower limit value, and the second upper limit value and the second lower limit value being determined from the training process of the video codec model; and performing inverse quantization on a quantized code representation corresponding to the quantized representation based on the inverse quantization scaling value in at least one of the frame encoding process of the target frame or the motion vector encoding process of the target frame.

[0095] In some implementations, determining the quantization scaling value for the target frame comprises: performing, using an interpolation function, interpolation between the first upper limit value and the first lower limit value based on at least the quantization parameter, to obtain the quantization scaling value, wherein the interpolation function has monotonicity between the first upper limit value and the first lower limit value.

[0096] In some implementations, the quantization parameter is selected from a range of quantization parameters. In some implementations, performing, based on at least the quantization parameter, interpolation between the first upper limit value and the first lower limit value comprises: performing, using the interpolation function, interpolation between the first upper limit value and the first lower limit value based on at least the quantization parameter and an upper limit value of the range of quantization parameters, wherein the interpolation function is configured to exponentially increase the quantization scaling value from the first lower limit value to the first upper limit value as the quantization parameter linearly increases within the range of the quantization parameters.

[0097] In some implementations, the interpolation function is configured in the form of an exponential or logarithmic function, where the exponent of the exponential function is determined based on the quantization parameter and the range of the quantization parameters, and the base of the exponential function is determined based on the first upper limit value and the first lower limit value. In some implementations, determining the inverse quantization scaling value for the target frame comprises: performing, using a same interpolation function as used in determining the quantization scaling value, interpolation between the second upper limit value and the second lower limit value based on the quantization parameter, to obtain the inverse quantization scaling value.

[0098] In some implementations, a loss function in the training process of the video codec model is determined by using a weighting factor to weight between a bit rate loss and a distortion loss, and the weighting factor is determined based on the quantization parameter, a weighting upper limit value and a weighting lower limit value.

[0099] In some implementations, the weighting factor is determined by: performing, using a same interpolation function as used in determining the quantization scaling value, interpolation between the weighting upper limit value and the weighting lower limit value based on the quantization parameter, to obtain the weighting factor.

[0100] In some implementations, the process 700 further comprises: using the video codec model and in a context extraction process of the target frame, in accordance with a determination that an update period for context information has expired, extracting feature information for the target frame from a reference reconstructed frame for the target frame;in accordance with a determination that the update period for context information has not expired, extracting feature information for the target frame from reference feature information of the reference reconstructed frame; and determining, based on at least the feature information for the target frame, context information for the target frame, and wherein the context information for the target frame is used in a frame encoding process of the target frame.

[0101] In some implementations, the update period for context information is determined based on a content type of the video.

[0102] In some implementations, the process 700 further comprises: using the video codec model and in a context extraction process of the target frame, determining a difference between a plurality of reconstructed frames before the target frame, in accordance with a determination that the difference exceeds a difference threshold, extracting feature information for the target frame from a reference reconstructed frame for the target frame; in accordance with a determination that the difference does not exceed a difference threshold, extracting the feature information for the target frame from reference feature information of the reference reconstructed frame; and determining, based on at least the feature information for the target frame, context information for the target frame, and wherein the context information for the target frame is used in a frame encoding process of the target frame.

[0103] In some implementations, during the training process of the video codec model, training data of the video codec model comprises at least one sample video segment with a number of frames exceeding a predetermined number threshold.

[0104] In some implementations, a loss function in the training process of the video codec model is determined at least based on a distortion loss, the distortion loss comprising a distortion loss of a first sample video in a first color space and a distortion loss of a second sample video in a second color space. Example Device

[0105] FIG.8 illustrates a schematic block diagram of an electronic device in which various implementations of the subject matter described herein can be implemented. It would be appreciated that the electronic device 800 as shown in FIG. 8 is merely provided as an example, without suggesting any limitation to the functionalities and scope of implementations of the subject matter described herein. One or more electronic devices 800 may, for example, be used to implement the video codec model 200 of FIG. 2, the electronic device 110 or 120 in FIG. 1.

[0106] As shown in FIG. 8, the electronic device 800 is in form of a general-purpose computing device. Components of the electronic device 800 may include, but are not limitedto, one or more processors or processing devices 810, a memory 820, a storage device 830, one or more communication units 840, one or more input devices 850, and one or more output devices 860.

[0107] In some implementations, the electronic device 800 may be implemented as a device with computing capability, such as a computing device, a computing system, a server, a mainframe and the like.

[0108] The processing device 810 can be a physical or virtual processor and can execute various processing based on the programs stored in the memory 820. In a multi-processor system, a plurality of processing units execute computer-executable instructions in parallel so as to enhance parallel processing capability of the electronic device 800. The processing device 810 may include a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a controller, and / or a microcontroller.

[0109] The electronic device 800 usually includes various computer storage medium. Such medium may be any available medium accessible by the electronic device 800, including but not limited to, volatile and non-volatile medium, or detachable and non-detachable medium. The memory 820 may be a volatile memory (for example, a register, cache, Random Access Memory (RAM)), non-volatile memory (for example, a Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), a flash memory), or any combination thereof. The storage device 1330 may be any detachable or non-detachable medium and may include computer-readable medium such as a memory, a flash memory drive, a magnetic disk or any other medium that can be used for storing information and / or data and are accessible by the electronic device 1300.

[0110] The electronic device 800 may further include additional detachable / non-detachable, volatile / non-volatile memory medium. Although not shown in FIG.8, there may be provided a disk drive for reading from or writing into a detachable and non-volatile disk, and an optical disk drive for reading from and writing into a detachable non-volatile optical disc. In such cases, each drive may be connected to a bus (not shown) via one or more data medium interfaces.

[0111] The communication unit 840 implements communication with another computing device via the communication medium. In addition, the functionalities of components in the electronic device 800 may be implemented by a single computing cluster or a plurality of computing machines that can communicate with each other via communication connections. Thus, the electronic device 800 may operate in a networked environment using a logic connection with one or more other servers, network personal computers (PCs), or further general network nodes.

[0112] The input device 850 may include one or more of a variety of input devices, such as a mouse, keyboard, data import device and the like. The output device 860 may be one or more output devices, such as a display, data export device and the like. By means of the communication unit 840, the electronic device 800 may further communicate with one or more external devices (not shown) such as storage devices and display devices, one or more devices that enable the user to interact with the electronic device 800, or any devices (such as a network card, a modem and the like) that enable the electronic device 1300 to communicate with one or more other computing devices, if required. Such communication may be performed via input / output (I / O) interfaces (not shown).

[0113] In some implementations, as an alternative of being integrated on a single device, some or all components of the electronic device 800 may also be arranged in the form of cloud computing architecture. In the cloud computing architecture, the components may be provided remotely and work together to implement the functionalities described in the subject matter described herein. In some implementations, cloud computing provides computing, software, data access and storage service, which will not require end users to be aware of the physical locations or configurations of the systems or hardware provisioning these services. In various implementations, the cloud computing provides the services via a wide area network (such as Internet) using proper protocols. For example, a cloud computing provider provides applications over the wide area network, which may be accessed through a web browser or any other computing components. The software or components of the cloud computing architecture and corresponding data may be stored in a server at a remote position. The computing resources in the cloud computing environment may be aggregated or distributed at locations of remote data centers. Cloud computing infrastructure may provide the services through a shared data center, though they behave as a single access point for the users. Therefore, the cloud computing infrastructure may be utilized to provide the components and functionalities described herein from a service provider at remote locations. Alternatively, they may be provided from a conventional server or may be installed directly or otherwise on a client device.

[0114] The electronic device 800 may be used to implement resource management in accordance with various implementations of the subject matter described herein. The memory 820 may include one or more modules having one or more program instructions. These modules may be accessed and run by the processing unit 810 to perform functions of various implementations described herein. For example, the memory 820 may include a video coding module 822 for performing video coding using a neural video codec. As shown in FIG. 8, the electronic device 800 may obtain a video to be encoded or a bitstream to bedecoded through the input device 850 and provide the encoded bitstream or the decoded video through the output device 860. In some implementations, the electronic device 800 may further receive the input from other devices (not shown) via the communication unit 840.

[0115] Some example implementations of the subject matter described herein are listed below.

[0116] In an aspect, the subject matter described herein provides a computer-implemented method. The method comprises: obtaining a quantization parameter for a target frame of a video; determining a quantization scaling value for the target frame based on the quantization parameter and at least a first upper limit value and a first lower limit value for quantization scaling, the quantization scaling value being between the first upper limit value and the first lower limit value, and the first upper limit value and the first lower limit value being determined from a training process of a video codec model; and performing, using the video codec model, quantization on the target frame based on the quantization scaling value in at least one of a frame encoding process or a motion vector encoding process of the target frame, to obtain a quantized representation of the target frame.

[0117] In some implementations, the method further comprises: determining an inverse quantization scaling value for the target frame based on the quantization parameter and at least a second upper limit value and a second lower limit value for inverse quantization scaling, the inverse quantization scaling value being between the second upper limit value and the second lower limit value, and the second upper limit value and the second lower limit value being determined from the training process of the video codec model; and performing inverse quantization on a quantized code representation corresponding to the quantized representation based on the inverse quantization scaling value in at least one of the frame encoding process of the target frame or the motion vector encoding process of the target frame.

[0118] In some implementations, determining the quantization scaling value for the target frame comprises: performing, using an interpolation function, interpolation between the first upper limit value and the first lower limit value based on at least the quantization parameter, to obtain the quantization scaling value, wherein the interpolation function has monotonicity between the first upper limit value and the first lower limit value.

[0119] In some implementations, the quantization parameter is selected from a range of quantization parameters. In some implementations, performing, based on at least the quantization parameter, interpolation between the first upper limit value and the first lower limit value comprises: performing, using the interpolation function, interpolation between the first upper limit value and the first lower limit value based on at least the quantization parameter and an upper limit value of the range of quantization parameters, wherein theinterpolation function is configured to exponentially increase the quantization scaling value from the first lower limit value to the first upper limit value as the quantization parameter linearly increases within the range of the quantization parameters.

[0120] In some implementations, the interpolation function is configured in the form of an exponential or logarithmic function, where the exponent of the exponential function is determined based on the quantization parameter and the range of the quantization parameters, and the base of the exponential function is determined based on the first upper limit value and the first lower limit value.

[0121] In some implementations, determining the inverse quantization scaling value for the target frame comprises: performing, using a same interpolation function as used in determining the quantization scaling value, interpolation between the second upper limit value and the second lower limit value based on the quantization parameter, to obtain the inverse quantization scaling value.

[0122] In some implementations, a loss function in the training process of the video codec model is determined by using a weighting factor to weight between a bit rate loss and a distortion loss, and the weighting factor is determined based on the quantization parameter, a weighting upper limit value and a weighting lower limit value.

[0123] In some implementations, the weighting factor is determined by: performing, using a same interpolation function as used in determining the quantization scaling value, interpolation between the weighting upper limit value and the weighting lower limit value based on the quantization parameter, to obtain the weighting factor.

[0124] In some implementations, the method further comprises: using the video codec model and in a context extraction process of the target frame, in accordance with a determination that an update period for context information has expired, extracting feature information for the target frame from a reference reconstructed frame for the target frame; in accordance with a determination that the update period for context information has not expired, extracting feature information for the target frame from reference feature information of the reference reconstructed frame; and determining, based on at least the feature information for the target frame, context information for the target frame, and wherein the context information for the target frame is used in a frame encoding process of the target frame.

[0125] In some implementations, the update period for context information is determined based on a content type of the video.

[0126] In some implementations, the method further comprises: using the video codec model and in a context extraction process of the target frame, determining a difference between a plurality of reconstructed frames before the target frame, in accordance with a determinationthat the difference exceeds a difference threshold, extracting feature information for the target frame from a reference reconstructed frame for the target frame; in accordance with a determination the difference does not exceed a difference threshold, extracting the feature information for the target frame from reference feature information of the reference reconstructed frame; and determining, based on at least the feature information for the target frame, context information for the target frame, and wherein the context information for the target frame is used in a frame encoding process of the target frame.

[0127] In some implementations, during the training process of the video codec model, training data of the video codec model comprises at least one sample video segment with a number of frames exceeding a predetermined number threshold.

[0128] In some implementations, a loss function in the training process of the video codec model is determined at least based on a distortion loss, the distortion loss comprising a distortion loss of a first sample video in a first color space and a distortion loss of a second sample video in a second color space.

[0129] In another aspect, the subject matter described herein provides an electronic device. The electronic device comprises a processor; and a memory coupled to the processor and comprising instructions stored thereon which, when executed by the processor, cause the device to perform acts comprising: obtaining a quantization parameter for a target frame of a video; determining a quantization scaling value for the target frame based on the quantization parameter and at least a first upper limit value and a first lower limit value for quantization scaling, the quantization scaling value being between the first upper limit value and the first lower limit value, and the first upper limit value and the first lower limit value being determined from a training process of a video codec model; and performing, using the video codec model, quantization on the target frame based on the quantization scaling value in at least one of a frame encoding process or a motion vector encoding process of the target frame, to obtain a quantized representation of the target frame.

[0130] In some implementations, the acts further comprise: determining an inverse quantization scaling value for the target frame based on the quantization parameter and at least a second upper limit value and a second lower limit value for inverse quantization scaling, the inverse quantization scaling value being between the second upper limit value and the second lower limit value, and the second upper limit value and the second lower limit value being determined from the training process of the video codec model; and performing inverse quantization on a quantized code representation corresponding to the quantized representation based on the inverse quantization scaling value in at least one of the frame encoding process of the target frame or the motion vector encoding process of the target frame.

[0131] In some implementations, determining the quantization scaling value for the target frame comprises: performing, using an interpolation function, interpolation between the first upper limit value and the first lower limit value based on at least the quantization parameter, to obtain the quantization scaling value, wherein the interpolation function has monotonicity between the first upper limit value and the first lower limit value.

[0132] In some implementations, the quantization parameter is selected from a range of quantization parameters. In some implementations, performing, based on at least the quantization parameter, interpolation between the first upper limit value and the first lower limit value comprises: performing, using the interpolation function, interpolation between the first upper limit value and the first lower limit value based on at least the quantization parameter and an upper limit value of the range of quantization parameters, wherein the interpolation function is configured to exponentially increase the quantization scaling value from the first lower limit value to the first upper limit value as the quantization parameter linearly increases within the range of the quantization parameters.

[0133] In some implementations, the interpolation function is configured in the form of an exponential or logarithmic function, where the exponent of the exponential function is determined based on the quantization parameter and the range of the quantization parameters, and the base of the exponential function is determined based on the first upper limit value and the first lower limit value.

[0134] In some implementations, determining the inverse quantization scaling value for the target frame comprises: performing, using a same interpolation function as used in determining the quantization scaling value, interpolation between the second upper limit value and the second lower limit value based on the quantization parameter, to obtain the inverse quantization scaling value.

[0135] In some implementations, a loss function in the training process of the video codec model is determined by using a weighting factor to weight between a bit rate loss and a distortion loss, and the weighting factor is determined based on the quantization parameter, a weighting upper limit value and a weighting lower limit value.

[0136] In some implementations, the weighting factor is determined by: performing, using a same interpolation function as used in determining the quantization scaling value, interpolation between the weighting upper limit value and the weighting lower limit value based on the quantization parameter, to obtain the weighting factor.

[0137] In some implementations, the acts further comprise: using the video codec model and in a context extraction process of the target frame, in accordance with a determination that an update period for context information has expired, extracting feature information for thetarget frame from a reference reconstructed frame for the target frame; in accordance with a determination that the update period for context information has not expired, extracting feature information for the target frame from reference feature information of the reference reconstructed frame; and determining, based on at least the feature information for the target frame, context information for the target frame, and wherein the context information for the target frame is used in a frame encoding process of the target frame.

[0138] In some implementations, the update period for context information is determined based on a content type of the video.

[0139] In some implementations, the acts further comprise: using the video codec model and in a context extraction process of the target frame, determining a difference between a plurality of reconstructed frames before the target frame, in accordance with a determination that the difference exceeds a difference threshold, extracting feature information for the target frame from a reference reconstructed frame for the target frame; in accordance with a determination that the difference does not exceed a difference threshold, extracting the feature information for the target frame from reference feature information of the reference reconstructed frame; and determining, based on at least the feature information for the target frame, context information for the target frame, and wherein the context information for the target frame is used in a frame encoding process of the target frame.

[0140] In some implementations, during the training process of the video codec model, training data of the video codec model comprises at least one sample video segment with a number of frames exceeding a predetermined number threshold.

[0141] In some implementations, a loss function in the training process of the video codec model is determined at least based on a distortion loss, the distortion loss comprising a distortion loss of a first sample video in a first color space and a distortion loss of a second sample video in a second color space.

[0142] In yet another aspect, the subject matter described herein provides a computer program product that is tangibly stored in a computer storage medium and comprises computer executable instructions that, when executed by a device, cause the device to perform acts comprising: obtaining a quantization parameter for a target frame of a video; determining a quantization scaling value for the target frame based on the quantization parameter and at least a first upper limit value and a first lower limit value for quantization scaling, the quantization scaling value being between the first upper limit value and the first lower limit value, and the first upper limit value and the first lower limit value being determined from a training process of a video codec model; and performing, using the video codec model, quantization on the target frame based on the quantization scaling value in atleast one of a frame encoding process or a motion vector encoding process of the target frame, to obtain a quantized representation of the target frame.

[0143] In yet another aspect, the subject matter described herein provides a computer- readable medium having computer executable instructions stored thereon that, when executed by a device, cause the device to perform one or more example implementations of the method in the above aspect.

[0144] The functionalities described herein can be performed, at least in part, by one or more hardware logic components. As an example, and without limitation, illustrative types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), Application-specific Integrated Circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), and the like.

[0145] Program code for carrying out the methods of the subject matter described herein may be written in any combination of one or more programming languages. The program code may be provided to a processor or controller of a general-purpose computer, special purpose computer, or other programmable data processing flowchart such that the program code, when executed by the processor or controller, causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely or partly on a machine, executed as a stand-alone software package partly on the machine, partly on a remote machine, or entirely on the remote machine or server.

[0146] In the context of the subject matter described herein, a machine-readable medium may be any tangible medium that may contain or store a program for use by or in connection with an instruction execution system, flowchart, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, flowchart, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random-access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0147] Further, although the operations are depicted in a particular order, this should not be understood as requiring that such operations are performed in the particular order shown or in sequential order, or that all illustrated operations are performed to achieve the desiredresults. In certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are contained in the above discussions, these should not be construed as limitations on the scope of the subject matter described herein, but rather as descriptions of features that may be specific to particular implementations. Certain features that are described in the context of separate implementations may also be implemented in combination in a single implementation. Rather, various features described in a single implementation may also be implemented in various implementations separately or in any suitable sub-combination.

[0148] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter specified in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

CLAIMS 1. A computer-implemented method comprising: obtaining a quantization parameter for a target frame of a video; determining a quantization scaling value for the target frame based on the quantization parameter and at least a first upper limit value and a first lower limit value for quantization scaling, the quantization scaling value being between the first upper limit value and the first lower limit value, and the first upper limit value and the first lower limit value being determined from a training process of a video codec model; and performing, using the video codec model, quantization on the target frame based on the quantization scaling value in at least one of a frame encoding process or a motion vector encoding process of the target frame, to obtain a quantized representation of the target frame.

2. The method of claim 1, further comprising: determining an inverse quantization scaling value for the target frame based on the quantization parameter and at least a second upper limit value and a second lower limit value for inverse quantization scaling, the inverse quantization scaling value being between the second upper limit value and the second lower limit value, and the second upper limit value and the second lower limit value being determined from the training process of the video codec model; and performing inverse quantization on a quantized code representation corresponding to the quantized representation based on the inverse quantization scaling value in at least one of the frame encoding process of the target frame or the motion vector encoding process of the target frame.

3. The method of claim 1, wherein determining the quantization scaling value for the target frame comprises: performing, using an interpolation function, interpolation between the first upper limit value and the first lower limit value based on at least the quantization parameter, to obtain the quantization scaling value, wherein the interpolation function has monotonicity between the first upper limit value and the first lower limit value.

4. The method of claim 3, wherein the quantization parameter is selected from a range of quantization parameters, and wherein performing, based on at least the quantization parameter, interpolation between the first upper limit value and the first lower limit value comprises: performing, using the interpolation function, interpolation between the first upper limit value and the first lower limit value based on at least the quantization parameter and an upper limit value of the range of quantization parameters, wherein the interpolation function is configured to exponentially increase the quantizationscaling value from the first lower limit value to the first upper limit value as the quantization parameter linearly increases within the range of the quantization parameters.

5. The method of claim 2, wherein determining the inverse quantization scaling value for the target frame comprises: performing, using a same interpolation function as used in determining the quantization scaling value, interpolation between the second upper limit value and the second lower limit value based on the quantization parameter, to obtain the inverse quantization scaling value.

6. The method of claim 1, wherein a loss function in the training process of the video codec model is determined by using a weighting factor to weight between a bit rate loss and a distortion loss, and the weighting factor is determined based on the quantization parameter, a weighting upper limit value and a weighting lower limit value.

7. The method of claim 6, wherein the weighting factor is determined by: performing, using a same interpolation function as used in determining the quantization scaling value, interpolation between the weighting upper limit value and the weighting lower limit value based on the quantization parameter, to obtain the weighting factor.

8. The method of claim 1, further comprising: using the video codec model and in a context extraction process of the target frame, in accordance with a determination that an update period for context information has expired, extracting feature information for the target frame from a reference reconstructed frame for the target frame; in accordance with a determination that the update period for context information has not expired, extracting feature information for the target frame from reference feature information of the reference reconstructed frame; and determining, based on at least the feature information for the target frame, context information for the target frame, and wherein the context information for the target frame is used in a frame encoding process of the target frame.

9. The method of claim 8, wherein the update period for context information is determined based on a content type of the video.

10. The method of claim 1, further comprising: using the video codec model and in a context extraction process of the target frame, determining a difference between a plurality of reconstructed frames before the target frame, in accordance with a determination that the difference exceeds a difference threshold, extracting feature information for the target frame from a referencereconstructed frame for the target frame; in accordance with a determination that the difference does not exceed a difference threshold, extracting the feature information for the target frame from reference feature information of the reference reconstructed frame; and determining, based on at least the feature information for the target frame, context information for the target frame, and wherein the context information for the target frame is used in a frame encoding process of the target frame.

11. The method of claim 1, wherein during the training process of the video codec model, training data of the video codec model comprises at least one sample video segment with a number of frames exceeding a predetermined number threshold.

12. The method of claim 1, wherein a loss function in the training process of the video codec model is determined at least based on a distortion loss, the distortion loss comprising a distortion loss of a first sample video in a first color space and a distortion loss of a second sample video in a second color space.

13. An electronic device comprising: a processor; and a memory coupled to the processor and comprising instructions stored thereon which, when executed by the processor, cause the device to perform acts comprising: obtaining a quantization parameter for a target frame of a video; determining a quantization scaling value for the target frame based on the quantization parameter and at least a first upper limit value and a first lower limit value for quantization scaling, the quantization scaling value being between the first upper limit value and the first lower limit value, and the first upper limit value and the first lower limit value being determined from a training process of a video codec model; and performing, using the video codec model, quantization on the target frame based on the quantization scaling value in at least one of a frame encoding process or a motion vector encoding process of the target frame, to obtain a quantized representation of the target frame.

14. The device of claim 13, wherein the acts further comprise: determining an inverse quantization scaling value for the target frame based on the quantization parameter and at least a second upper limit value and a second lower limit value for inverse quantization scaling, the inverse quantization scaling value being between the second upper limit value and the second lower limit value, and the second upper limit value and the second lower limit value being determined from the training process of the video codec model; andperforming inverse quantization on quantized code corresponding to the quantized representation based on the inverse quantization scaling value in at least one of the frame encoding process of the target frame or the motion vector encoding process of the target frame.

15. A computer program product being tangibly stored in a computer storage medium and comprising computer executable instructions that, when executed by a device, cause the device to perform acts comprising: obtaining a quantization parameter for a target frame of a video; determining a quantization scaling value for the target frame based on the quantization parameter and at least a first upper limit value and a first lower limit value for quantization scaling, the quantization scaling value being between the first upper limit value and the first lower limit value, and the first upper limit value and the first lower limit value being determined from a training process of a video codec model; and performing, using the video codec model, quantization on the target frame based on the quantization scaling value in at least one of a frame encoding process or a motion vector encoding process of the target frame, to obtain a quantized representation of the target frame.

Images