Neural Video Coding

Abstract

According to an implementation of the subject matter described herein, a solution for neural video coding is proposed. According to this solution, estimated motion information of a target frame in a video, and reference feature information and a reference reconstructed frame of a reference frame related to the target frame are obtained. Using a context extraction model, context information related to the target frame is determined based on the estimated motion information, the reference reconstructed frame, and the reference feature information. During conversion between the target frame and a video bitstream, a frame coding model is used to generate a reconstructed target frame of the target frame based on at least the context information. In this way, richer context information can be extracted for coding, thereby improving coding efficiency.

Description

[Background technology]

[0001] background The main principle of video codecs is that when a current frame is coded, the codec finds relevant context information (e.g., various predictions as context information) from previously reconstructed frames to reduce spatio-temporal redundancy. The more relevant the context information, the greater the bitrate reduction achieved. Conventional video coding (e.g., H.261 to H.266) extracts and utilizes context information from various manual coding modes. Currently, neural video codecs (NVCs) have been proposed to extract and utilize context information in an automatic learning manner, which provides greater flexibility and improves coding efficiency. Summary of the Invention

[0002] overview According to an implementation of the subject matter described herein, a solution for neural video coding is proposed. In this solution, estimated motion information of a target frame in a video, and reference feature information and a reference reconstructed frame of a reference frame related to the target frame are obtained. Using a context extraction model, context information for the target frame is determined based on the estimated motion information, the reference reconstructed frame, and the reference feature information. In conversion between the target frame and a video bitstream, a frame coding model is used to generate a reconstructed target frame of the target frame based on at least the context information. In this way, richer context information can be extracted for coding, thereby improving coding efficiency.

[0003] This Summary section is provided to introduce some concepts in a simplified form that are further described below in the Detailed Description section. This Summary is not intended to identify key features or essential features of the subject matter described herein, nor is it intended to be used to limit the scope of the subject matter described herein. [Brief explanation of the drawings]

[0004] BRIEF DESCRIPTION OF THE DRAWINGS [Figure 1] 1 illustrates a block diagram of an example environment in which various implementations of the subject matter described herein may be implemented. [Figure 2] 1 shows a schematic block diagram of a neural video codec system according to some implementations of the subject matter described herein. [Figure 3] 1 shows a schematic block diagram of an example architecture of a context extraction model according to some implementations of the subject matter described herein. [Figure 4] 1 shows a schematic block diagram of an example architecture of an offset predictor within a context extraction model, according to some implementations of the subject matter described herein. [Figure 5] 1 shows a schematic block diagram of an example architecture of a cross-group aggregator in a context extraction model, according to some implementations of the subject matter described herein. [Figure 6] 1 shows a schematic block diagram of an example architecture of a frame coding model according to some implementations of the subject matter described herein. [Figure 7] 1 shows a schematic block diagram of an example model implementation of a frame coding model in accordance with some implementations of the subject matter described herein. [Figure 8] 1 shows a diagram illustrating an example flow of a group-based entropy coding operation according to some implementations of the subject matter described herein. [Figure 9]1 shows a schematic block diagram of an example model implementation of an entropy coding network in accordance with some implementations of the subject matter described herein. [Figure 10] 1 shows a schematic block diagram of an example model implementation of a deep convolution block in an entropy coding network, in accordance with some implementations of the subject matter described herein. [Figure 11] 1 shows a diagram illustrating a comparison of a hierarchical quality structure of a frame according to some implementations of the subject matter described herein with the hierarchical quality structure of a conventional codec. [Figure 12] 1 shows a flowchart of a process for video processing according to some implementations of the subject matter described herein. [Figure 13] 1 shows a schematic block diagram of an electronic device in which various implementations of the subject matter described herein may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0005] Throughout the drawings, the same or similar reference numbers represent the same or similar elements.

[0006] Detailed Description of the Embodiments The subject matter described herein will now be described with reference to several example implementations. It should be understood that these implementations are provided for illustrative purposes only, to help those skilled in the art better understand and realize the subject matter described herein, and do not imply any limitation on the scope of the subject matter described herein.

[0007] As used herein, the term "comprises" and its conjugations shall be interpreted as open terms meaning "including but not limited to." The term "based on" shall be interpreted as "based at least in part on." The terms "implementation" and "an implementation" shall be interpreted as "at least one implementation." The term "another implementation" shall be interpreted as "at least one other implementation." Terms such as "first" and "second" may refer to different entities or the same entity. Other definitions, either explicit or implicit, may be included below.

[0008] As used herein, the term "model" may learn associations between corresponding inputs and outputs from training data, so that after training, a corresponding output can be generated for a given input. The generation of a model may be based on machine learning techniques. Deep learning (DL) is a type of machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs. A neural network model is an example of a deep learning-based model. As used herein, a "model" may also be referred to as a "machine learning model," a "learning model," a "machine learning network," or a "learning network," which are used interchangeably herein.

[0009] In general, machine learning may roughly include three stages: a training stage, a testing stage, and an application stage (also called an interference stage). In the training stage, a given model may be trained using large-scale training data, and parameter values ​​are iteratively updated until the model can obtain consistent interference that meets the expected target from the training data. Through training, the model may be considered to be able to learn the association between input and output (also called input-output mapping) from the training data. Parameter values ​​of the trained model are determined. In the testing stage, a test input is applied to the trained model to test whether the model can provide a correct output and determine the performance of the model. In the interference stage, the model may be used to process actual inputs based on the parameter values ​​obtained from training and determine the corresponding outputs.

[0010] Example Environment 1 shows a block diagram of an example environment 100 in which various implementations of the subject matter described herein may be implemented. In the environment of FIG. 1, an electronic device 110 includes a video codec 112 configured to encode and / or decode video. An electronic device 120 includes a video codec 122 configured to encode and / or decode video. The video codec 112 or 122 may include an encoder and / or a decoder. During encoding, the encoder may encode video 130 into a bitstream 132. During decoding, the decoder may decode the bitstream 132 into video 130.

[0011] Electronic devices 110 and 120 can communicate with each other via any suitable communication network. In some codec scenarios, electronic device 110 and electronic device 120 may perform video communication, and video codecs 112 and 122 may implement both video encoding and decoding. For example, electronic device 110 may provide a bitstream obtained after video encoding to electronic device 120 for decoding, and electronic device 120 may decode the received bitstream to obtain corresponding video. Furthermore, electronic device 120 may also provide a result of video encoding to electronic device 110 for decoding. In some codec scenarios, video codec 112 of electronic device 110 may include an encoder for encoding video into a bitstream. Electronic device 120 may include a video playback tool, and video codec 122 includes a decoder for decoding the bitstream generated by video codec 112 to obtain video for playback.

[0012] It should be understood that the devices and elements shown in Figure 1 are merely examples. In a practical application, there may be more electronic devices, and each electronic device may have video encoding and / or decoding functions.

[0013] In conventional video coding (e.g., H.261 through H.266), coding gains are primarily derived from a continuously expanding range of coding modes, each of which uses a specially designed method to extract and utilize context information. For example, H.266 uses up to 65 intra-prediction directions. While the large number of modes can extract diverse context information and reduce redundancy, it also creates enormous complexity because rate-distortion optimization (RDO) is used to search for the best mode. For example, encoding a frame with a resolution of 1080p can take up to 30 minutes with conventional video codecs still under development.

[0014] In contrast, neural video codecs (NVC) have changed the method of context extraction and utilization from manual design to automatic learning. Neural video codecs can be classified into residual coding-based and conditional coding-based. Residual coding directly uses reconstructed frames as context, and the utilization of context is limited to its use in difference calculation for redundancy removal. Conditional coding explicitly learns feature domain context. High-dimensional context can convey richer information to facilitate encoding, decoding, and entropy modeling.

[0015] However, most NVCs still have limited methods for extracting and utilizing context, and only a single optical flow is used to explore temporal correlations. This makes NVC susceptible to parameter uncertainty or prone to local optima. One solution is to add a coding mode like a traditional codec to NVC, but this brings about significant computational complexity. Therefore, NVC is expected to learn and utilize context information better while keeping computational cost low.

[0016] Video Encoding System Architecture In an exemplary implementation of the subject matter described herein, an improved solution for neural video coding is proposed, in which, for a target frame to be coded, context information for the target frame is determined based on estimated motion information of the target frame and reference reconstructed frames and reference feature information for the reference frame. The extracted context information is used to generate a target reconstructed frame for the target frame, and the target reconstructed frame is used for conversion between the target frame and a video bitstream.

[0017] During neural video coding, both feature information in feature regions and reference reconstructed frames in non-feature regions are used to extract the context of the target frame. In this way, the diversity of context information can be increased, which can further improve coding and compression efficiency.

[0018] Certain exemplary implementations of the subject matter described herein are described in more detail below with reference to the accompanying drawings.

[0019] 2 shows a schematic block diagram of a neural video codec system 200 according to some implementations of the subject matter described herein. Each component in the neural video codec system 200 may be implemented in hardware, software, firmware, or any combination thereof. The neural video codec system 200 may be implemented in the video codecs 112 and / or 122 of FIG. 1.

[0020] The neural video codec system 200 generally includes a motion estimation model 210, a context extraction model 220, and a frame encoding model 230. These models may be implemented based on machine learning techniques, such as neural network architectures. To achieve higher compression ratios, the neural video codec system 200 is implemented based on conditional encoding, which is more flexible and can guide the encoding of frames under the conditions of extracted context information.

[0021] The neural video codec system 200 can implement a conversion between each frame in a video and a video bitstream. The conversion includes a video encoding process, a video decoding process, or both. During the encoding process, the neural video codec system 200 can receive a sequence of video frames and perform video coding on each frame to obtain a video bitstream. During the decoding process, the neural video codec system 200 receives a video bitstream and decodes the sequence of video frames therefrom. In the following, except for operations explicitly specified, other operations can be considered to be performed on both the video encoding side and the video decoding side.

[0022] Herein, the target frame x t represents the current frame to be coded in the video. As shown in Figure 2, the target frame x with index t t To encode and decode the target frame x t Motion information v t and motion information v t and then encode it as the estimated motion information

number

number

[0023] The motion information includes the offset size and direction of the target frame x relative to the reference frame. t For example, the motion information may include motion vectors (MVs). The reference frame is the target frame x t In some implementations, the target frame x t The reference frame of the target frame x t In the following implementation, only a single reference frame is adopted as an example for illustration, but multiple reference frames are also feasible. As shown in FIG. 2, the target frame x t Previous frame before x t-1 is used as a reference frame. In some implementations, the motion estimation model 210 estimates the motion information based on an optical flow network.

number

[0024] The context extraction model 220 extracts the target frame x t Contextual information about C t In an implementation of the subject matter described herein, the context extraction model 220 is configured to determine the target frame x t Estimated motion information

number

number

number

number

number

[0025] The frame coding model 230 includes at least the context information C tBased on the target frame x t Target reconstruction frame of

number

number

number

[0026] On the encoding side, the context information C t , the frame coding model 230 is t quantized code representation

number

number

number

number

[0027] FIG. 2 shows the neural video codec system 20 generating each frame x t-2 , x t-1 , x t 2 shows the coding pipeline for each frame. For each frame, the motion estimation model 210, the context extraction model 220, and the frame coding model 230 in the system 200 perform similar operations.

[0028] The overall workflow of the neural video codec system 200 is described above. Exemplary implementations of context extraction and frame encoding are discussed in more detail below.

[0029] Exemplary Implementation of Context Extraction In some implementations, offset diversity is applied to enhance the extraction of context information in the context extraction process. Multiple motion offsets can reduce motion errors for complex or large movements. Diverse offsets can complement each other and provide better temporal references to reduce redundancy and provide opportunities for improving coding efficiency. In some implementations, multiple motion offsets may be divided into groups, and inter-group fusion can be performed to provide benefits for mining temporal context information.

[0030] 3 shows a schematic block diagram of an example architecture of the context extraction model 220 according to some implementations of the subject matter described herein. As shown in FIG. 3, the context extraction model 220 includes a concatenator 310, a motion alignment unit 320, an offset predictor 330, an adder 335, and an inter-group aggregator 340.

[0031] Due to the different motions between frames, the raw reference feature information F t-1 Therefore, the motion estimation model 210 extracts motion information in neural video coding and uses the motion-aligned context information C t The input of the context extraction model 220 is the estimated motion information of the target frame.

number

number

number

[0032] Most existing NVCs perform motion alignment operations directly on a single MV in the estimated motion information. Such single motion-based alignment is not robust to complex motion or occlusions. In some implementations of the subject matter described herein, the offset predictor 330 performs a motion alignment operation on a single MV in the estimated motion information.

number

number

number

number

number

number

number

number

number

number

[0033] In some implementations, the reference frame x t-1 to target frame x t When determining the offsets, the reference feature information F t-1 can be divided into multiple groups. t-1 may be expressed in the form of a feature map including multiple reference feature maps along the channel dimension. For example, the reference feature information F t-1 The dimension of F can be C×W×H, where C represents the number of channels in the channel dimension, and W and H represent the two spatial dimensions, i.e., the width and height of the feature map. t-1 The plurality of reference feature maps in may be divided into a first number (denoted as G) of feature map groups 312, where G is equal to or greater than 1. For example, the plurality of reference feature maps may be divided into G feature map groups along the channel dimension. For each feature map group, estimated motion information

number

number

number

[0034] The values ​​of G and N can be set according to the actual requirements of the application. In some implementations, the reference feature information F along the channel dimension is t-1 In addition to grouping, reference feature information F t-1 may be grouped along one or two spatial dimensions, and multiple motion offsets are determined for each group to increase offset diversity. The multiple determined motion offsets may be complementary. The diversity of motion offsets may help the codec cope with complex object motion and occlusion.

[0035] G×N motion offset o t 332 is the target frame x t Context information C t In some implementations, the residual offset d t In addition, the offset predictor 330 also uses weights m t 334 may also be generated, with weight m t 334 is considered to reflect the reliability of each motion offset. t 334 and motion offset t 332 is context information C t are used together to determine

[0036] 4 shows a schematic block diagram of an example architecture of the offset predictor 330 in the context extraction model 220 according to some implementations of the subject matter described herein. As shown in FIG. 4, the offset predictor 330 may be based on a convolutional neural network including multiple convolutional layers 410-1, 410-2, and 410-3. An activation layer, for example, including activation layers 420-1 and 420-2, may be disposed between the two convolutional layers. The convolution kernel size, the number of input channels, the number of output channels, and the step size of each convolutional layer may be configured according to actual requirements. The activation layer may be based on, for example, a Leaky ReLU function or any other suitable activation function.

[0037] In some implementations, to speed up computation, the resolution of the input may be reduced by a factor of 2 or more in the first layer of the offset predictor 330. In such implementations, the offset predictor 330 may also include a bilinear upsampling layer 430 to upsample the output to a resolution corresponding to the input.

[0038] The input of the offset predictor 330 is the motion-aligned

number

number

number

[0039] 4 illustrates an exemplary model architecture implementation of the offset predictor 330. Other model architectures may also be applied to implement the offset predictor 330 according to the needs of a practical application.

[0040] G×N motion offset o t 332 is the reference feature information F t-1 Perform motion alignment on the reference feature information F t-1 can be used to align the motion-aligned reference feature information F t-1 is the target frame x t Contextual information about C t In some implementations, for each feature map group in the G feature map groups 312, the N motion offsets o determined for that feature map group are used to determine t Then, N motion alignments are performed on the feature map groups using the context information C. t The G×N aligned feature map groups determined for the G feature map groups may be aggregated to determine

[0041] 5 shows a schematic block diagram of an example architecture of the inter-group aggregator 340 in the context extraction model 220, according to some implementations of the subject matter described herein. As shown in FIG. 5, the inter-group aggregator 340 includes a motion alignment unit 510, a weighting component 520, a reordering unit 530, and an aggregator 540.

[0042] For each feature map group among the G feature map groups 312, the motion alignment unit 510 aligns the N motion offsets o determined for that feature map group. t332 to perform motion alignment on the feature map groups to obtain G×N aligned feature map groups 512. In some implementations, the weighting component 520 is configured to assign weights m for each motion offset. t 334 is configured to weight the G×N aligned feature map groups 512. The weighted G×N aligned feature map groups are provided to the sorting unit 530.

[0043] In some implementations, during the aggregation of aligned feature map groups, merging between feature map groups is proposed, where each merged feature map group is merged from a different aligned feature map group. Thus, G×N aligned feature map groups 512 can be merged to obtain G merged feature map groups. Specifically, after motion alignment is performed on each feature map group using multiple motion offsets and weighted with corresponding weights, the aligned feature map groups can be sorted by the sorting unit 530 before aggregation.

[0044] Aligned feature map group

number

number

number

[0045] Therefore, during this process, group reordering allows for increased interaction between groups without increasing computational complexity. Such aggregation can also enjoy benefits similar to weighted prediction from different reference frames in traditional codecs. Inter-group fusion can introduce more diverse combinations when extracting temporal context from different feature map groups, further improving the effectiveness of offset diversity.

[0046] In some implementations, the G merged feature map groups 542 are combined using context information C t In some implementations, the context information C t Other transformations may be applied to the G merged feature map groups 542 to obtain

[0047] Compared to traditional codec solutions in which more coding modes are introduced to implement context information extraction, the context information extraction proposed in implementations of the subject matter described herein can extract and utilize high-quality temporal context information without introducing additional model reference costs.

[0048] Exemplary Implementation of Frame Encoding In some implementations, in addition to increasing temporal context diversity in the context extraction model, a method is also proposed to increase spatial context diversity when encoding a frame into a quantized code representation (also called a latent representation) during frame encoding. Group-based entropy coding improves the prediction of statistical information (such as distribution information). Compared with traditional entropy coding methods, group-based entropy coding can achieve more diverse correlation modeling, and therefore the model has a better opportunity to find more relevant spatial context information.

[0049] 6 shows a schematic block diagram of an example architecture of a frame coding model 230 according to some implementations of the subject matter described herein. As shown in FIG. 6, the frame coding model 230 may include an encoder 610, a concatenator 620, a decoder 630, and a frame generator 640. On the video decoding side, the encoder 610 may be omitted.

[0050] The encoder 610 converts the above information C t conditional on the target frame x t quantized code representation

number

number

[0051] The decoder 630 converts the information C t Subject to the condition, the quantization code representation

number

number

[0052] Figure 7 shows a schematic block diagram of an example model implementation of the frame coding model 230 according to some implementations of the subject matter described herein. In the example of Figure 7, the encoder 610 and the decoder 630 in the frame coding model 230 may be implemented based on convolutional layers. As shown in Figure 7, the encoder 610 includes one or more convolutional and residual layers, including a convolutional layer 710, a residual block 712, a convolutional layer 714, a residual block 716, and convolutional layers 718 and 720, connected in series. Similarly, the decoder 630 includes convolutional layers 730 and 732, a residual block 734, a convolutional layer 736, a residual block 738, and a convolutional layer 740, connected in series. In some implementations, the encoder 610 and / or the decoder 630 may be configured to apply unequal channel numbers to speed up processing and reduce computational costs. In particular, features with different resolutions are assigned different numbers of channels, where features with higher resolutions are assigned fewer channels.

[0053] As shown in FIG. 7, in the encoder 610, a convolutional layer 710 and a residual block 712 are used to generate a target frame x t and context information C t Processing high-resolution context information C tAs an input, the convolutional layer 710 may be configured to have fewer channels. On the decoding side, the high-resolution context information C t as input, and high-resolution target feature information F t The frame generator 640 having as its output the quantized code representation σ ...

number

[0054] In some implementations, some quantization operations may be moved to higher resolutions for more accurate bitrate adjustment. As shown in Figure 7, the quantization parameter qp can be used to control the bitrate via user input. Depending on the quantization parameter qp, a learnable quantization parameter-quantization step table, i.e.

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0055] In some implementations, separate learnable quantization parameter-quantization step tables are applied by the encoder and decoder in the frame coding model 230, which may provide enhanced flexibility.

[0056] In the frame coding model 230, the quantized code representation after quantization and rounding may be provided to an arithmetic encoder (AE) 724. The AE 724 generates a video bitstream 726. The bitstream 726 is decoded by an arithmetic decoder (AD) 728 to generate the quantized code representation

number

number

[0057] We have discussed above some specific implementation forms of the frame coding model 230. It should be understood that Figure 7 only illustrates an example model structure of the frame coding model 230. In practical applications, other model structures may be configured for the frame coding model 230 as needed.

[0058] As mentioned above, in the frame encoding process, the quantized code representation

number

number

number

number

number

[0059] Each channel element group is further divided into multiple spatial element groups corresponding to multiple spatial positions, and the multiple spatial element groups overlap each other in the frame space. For example, each channel element group can be divided into 2x2 spatial regions, each spatial region having a corresponding spatial index (e.g., 0, 1, 2, 3), thereby obtaining the corresponding divided spatial element group. By dividing the quantization code representation, the entropy coding operation can be divided into multiple entropy coding operations, and entropy coding results 613, 615, 617, and 619 corresponding to each channel element group are obtained. After the entropy coding is completed, the multiple channel element groups are concatenated by a concatenator 620, and channel merging is completed to obtain the quantization code representation.

number

[0060] In each entropy coding operation, a plurality of spatial element groups corresponding to a plurality of spatial positions within the plurality of channel element groups are entropy coded, and the combinations of the plurality of spatial element groups entropy coded in the plurality of entropy coding operations are different.

[0061] 8 illustrates an example flow of a group-based entropy coding operation according to some implementations of the subject matter described herein. For illustrative purposes, in the example of FIG.

number

[0062]

number

number

number

number

number

number

number

number

number

[0063] In some implementations, the entropy encoding operation is performed on a reference frame x t-1 Quantized code representation for

number

number

[0064] 9 shows a schematic block diagram of an example model implementation of an entropy coding network 900 according to some implementations of the subject matter described herein. In the example of FIG. 9, the entropy coding operation of each step may be implemented based on one or more deep convolutional blocks. At step 0, the reference frame x t-1 Quantized code representation for

number

number

number

[0065] Group-based entropy coding can be implemented with highly parallelized computing, which can reduce coding time costs. Specifically, in some implementations, multiple spatial element groups in each entropy coding operation can be processed in parallel, reducing computational costs. In some implementations, to further reduce computational overhead and improve computational efficiency, depthwise separable convolutions can be further adopted in the model design, and unequal channel numbers can be set for features with different resolutions. Group-based entropy coding allows more neighboring information to be utilized in entropy coding. For example, in the above example, information from 0, 4, 4, and 8 neighboring positions is used in the four steps, respectively. Furthermore, such entropy coding is highly efficient because all spatial positions can be coded in parallel at each step. Furthermore, group-based entropy coding can also realize correlation between groups. For example, in step 3, for one particular spatial location of a channel group, other channels at the same location have already been coded from other channel groups in the previous step, and the coding results can be used as context information to perform entropy coding in the current step. This helps further reduce redundancy in video coding. Overall, group-based entropy coding benefits from diverse context information with finer granularity, which fully mines correlations from both the spatial and channel dimensions of a video frame.

[0066] In some implementations, to further reduce computational overhead, the deep convolution block in the entropy coding network 900 may include depthwise separable convolutions. Figure 10 shows a schematic block diagram of an example model implementation of a deep convolution block 1000 in the entropy coding network 900 in accordance with some implementations of the subject matter described herein. The deep convolution block 1000 may be used to implement one or more deep convolution blocks within the entropy coding network 900. As shown in Figure 10, the deep convolution block 1000 includes a separable convolution portion, i.e., a first convolution portion including a convolution layer 1010, an activation layer 1012 (e.g., based on a Leaky ReLU function or other suitable activation function), a depthwise convolution layer 1014, and a convolution layer 1016. The output of the first convolutional section and its input are summed and provided to a second convolutional section having a structure similar to that of the first convolutional section. For example, the second convolutional section includes a convolutional layer 1020, an activation layer 1022 (e.g., based on a Leaky ReLU function or other suitable activation function), a depthwise convolutional layer 1024, and a convolutional layer 1026. Although FIG. 10 illustrates only two convolutional sections, more or fewer convolutional sections can be configured as needed.

[0067] It should be understood that Figures 9 and 10 only show exemplary model structures of the entropy coding network and its components, and in practical applications, other model structures can also be configured for the entropy coding network and its components as needed.

[0068] Example implementation of model training Conventional codecs employ a hierarchical quality structure for frames, with multiple frames assigned to different layers and using different quantization parameters (QPs). It has been demonstrated that hierarchical quality structures in conventional codecs can realize scalable video coding and improve coding performance. On the one hand, hierarchical quality structures periodically improve the quality of coded frames, which can reduce error propagation. During inter-frame prediction, high-quality reference frames enable the codec to find more accurate motion information during motion estimation. On the other hand, motion-compensated prediction also has high quality, resulting in a reduction in prediction error. On the other hand, multiple reference frames can be selected and weighted in the coding process, resulting in a more diverse combination of predictions from neighboring reference frames and long-term, high-quality reference frames. Inspired by the benefits brought by conventional codecs, it is expected that a hierarchical quality structure for frames can also be introduced into NVC. One simple solution is to directly assign hierarchical QPs in NVC according to the rules of conventional codecs. However, unlike traditional codecs that use well-defined rules to perform motion estimation and motion compensation (MEMC), NVC uses a model, and MEMC is implicitly implemented in the feature domain. While the practical advantage of NVC is that it can be automatically learned to achieve better performance, it suffers from poor robustness to out-of-distribution quality patterns and poor generalization ability. Therefore, if a hierarchical QP is directly provided to NVC, such a hierarchical quality pattern may not be well adapted to all videos, and the performance achieved by MEMC may not be optimal.

[0069] Therefore, in some implementations, during training of the neural video codec system 200, each model in the system 200 can be further guided to learn hierarchical quality patterns across multiple frames. Based on the guidance during training, the model can implicitly learn long-term, high-quality context information that is very useful for reconstructing subsequent frames during feature transfer. This can further help exploit long-term temporal correlations in video and effectively mitigate the problem of quality degradation in NVC.

[0070] Specifically, for multiple sample frames in a sample video used to train the neural video codec system 200, each sample frame is assigned a weight w t At least one sample frame has a higher weight than the other sample frames. In particular, in some implementations, weights w for multiple sample frames are assigned. t The values ​​of follow a hierarchical structure. In the sample video, weights may be periodically assigned with a hierarchical structure. For example, one set of sample frames in the sample video may follow a hierarchical structure, the next set of sample frames may further follow a hierarchical structure, and so on.

[0071] In training the neural video codec system 200, the training objective is typically based on the error, e.g., inter-frame rate distortion (RD), between the sample frames and the reconstructed sample frames output by the neural video codec system 200. Based on the hierarchical weighting, for each sample frame, a weight w is assigned to weight the error corresponding to that sample frame. t may be used. The training objective is determined based on the weighted error. The training objective may be expressed as a loss function, an example of which is as follows:

number

number

[0072] By weighting with hierarchical weights, the errors for training purposes also have a hierarchical structure, which can guide the neural video codec system 200 to learn to control each frame in the video in a hierarchical manner. In this way, the neural video codec system 200 can produce high-quality reconstructed frames.

number

[0073] 11 shows a comparison of a hierarchical quality structure of frames according to some implementations of the subject matter described herein with the hierarchical quality structure of a conventional codec. As shown in FIG. 11, curve 1110 shows the bit change of each frame encoded by a conventional codec, and curve 1120 shows the bit change of each frame encoded by a neural codec according to some implementations of the subject matter described herein. Curve 1112 shows the signal-to-noise ratio (PSNR) of each frame encoded by a conventional codec, and curve 1122 shows the PSNR of each frame encoded by a neural codec according to some implementations of the subject matter described herein. It can be seen that, compared to a conventional codec, video coding according to implementations of the subject matter described herein can achieve better average quality at a lower bit cost.

[0074] Some examples of model training processes are described above. In other implementations, the model in the neural video codec may be trained in any other suitable manner depending on the actual application. Implementations of the subject matter described herein are not limited in this respect.

[0075] Example Process 12 shows a flowchart of a process 1200 for video processing according to some implementations of the subject matter described herein. The process 1200 may be implemented in the neural video codec system 200 of FIG.

[0076] In block 1210, the neural video codec system 200 obtains estimated motion information of a target frame in a video, and reference feature information and a reference reconstructed frame of a reference frame relative to the target frame.

[0077] In block 1220, the neural video codec system 200 uses the context extraction model to determine context information for the target frame based on the estimated motion information, the reference reconstructed frame, and the reference feature information.

[0078] At block 1230, the neural video codec system 200 generates a target reconstructed frame of the target frame based at least on the context information using the frame coding model in converting between the target frame and the video bitstream.

[0079] In some implementations, the reference feature information includes a plurality of reference feature maps. In some implementations, determining the context information includes dividing the plurality of reference feature maps into a first number of feature map groups; for each feature map group among the first number of feature map groups, determining a second number of motion offsets from the reference frame to the target frame based on the estimated motion information, the reference reconstruction frame, and the feature map group, where the second number is greater than 1, performing respective motion alignments on the feature map groups based on the second number of motion offsets to obtain the second number of aligned feature map groups; and determining the context information based on the aligned feature map groups determined for the first number of feature map groups.

[0080] In some implementations, determining the context information based on the aligned feature map groups determined for the first number of feature map groups includes: sorting the aligned feature map groups determined for the first number of feature map groups; merging the sorted aligned feature map groups to obtain a first number of merged feature map groups, each merged feature map group being merged from a second number of aligned feature maps selected from the sorted aligned feature map groups; and determining the context information based on the first number of merged feature map groups.

[0081] In some implementations, dividing the plurality of reference feature maps into a first number of feature map groups includes dividing the plurality of reference feature maps into the first number of feature map groups along a channel dimension.

[0082] In some implementations, generating a target reconstructed frame includes determining a quantization code representation for the target frame, the quantization code representation including elements organized along a channel dimension and a spatial dimension; dividing the elements of the quantization code representation into a plurality of channel element groups along the channel dimension; dividing each channel element group of the plurality of channel element groups into a plurality of spatial element groups corresponding to a plurality of spatial locations; and performing a plurality of entropy encoding operations, wherein each entropy encoding operation entropy encodes a plurality of spatial element groups corresponding to a plurality of spatial locations within the plurality of channel element groups, and the combinations of the plurality of spatial element groups entropy encoded in the plurality of entropy encoding operations are different.

[0083] In some implementations, performing the plurality of entropy encoding operations includes, for a given entropy encoding operation of the plurality of entropy encoding operations, performing the given entropy encoding operation based on an entropy encoding result of at least one of the plurality of spatial element groups from an entropy encoding operation preceding the given entropy encoding operation.

[0084] In some implementations, the context extraction model and the frame coding model are trained using a plurality of sample frames in the sample video, and a plurality of weights are assigned to the plurality of sample frames, at least one of the plurality of weights being higher than the other weights. In some implementations, a training objective for training the context extraction model and the frame coding model is determined by determining a plurality of errors between the plurality of sample frames and a plurality of reconstructed sample frames output using the context extraction model and the frame coding model; and weighting the plurality of errors with the plurality of assigned weights.

[0085] In some implementations, the weight values ​​follow a hierarchical structure.

[0086] In some implementations, the conversion includes encoding the target frame as at least part of a bitstream or decoding the target frame from the bitstream.

[0087] In some implementations, the process 1200 further includes generating target feature information for the target frame based at least on the context information using a frame coding model in converting between the target frame and the video bitstream.

[0088] Exemplary Devices 13 shows a schematic block diagram of an electronic device in which various implementations of the subject matter described herein may be implemented. It should be understood that the electronic device 1300 shown in FIG. 13 is provided merely as an example and does not imply any limitation on the scope of functionality and implementations of the subject matter described herein. For example, one or more electronic devices 1300 may be used to implement the neural video codec system 200 of FIG. 2.

[0089] 13, the electronic device 1300 is a form of general-purpose computing device. Components of the electronic device 1300 may include, but are not limited to, one or more processors or processing devices 1310, memory 1320, storage devices 1330, one or more communication units 1340, one or more input devices 1350, and one or more output devices 1360.

[0090] In some implementations, the electronic device 1300 may be implemented as a device with computing capabilities, such as a computing device, computing system, server, mainframe, or the like.

[0091] The processing device 1310 may be a physical or virtual processor and may perform various operations based on programs stored in the memory 1320. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel, enhancing the parallel processing capabilities of the electronic device 1300. The processing device 1310 may include a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a controller, and / or a microcontroller.

[0092] The electronic device 1300 typically includes a variety of computer storage media. Such media may be any available media that can be accessed by the electronic device 1300, including, but not limited to, volatile and nonvolatile media, or removable and non-removable media. The memory 1320 may be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or any combination thereof. The storage device 1330 may be any removable or non-removable media and may include a computer-readable medium, such as a memory, a flash memory drive, a magnetic disk, or any other medium that can be used to store information and / or data and that can be accessed by the electronic device 1300.

[0093] The electronic device 1300 may further include additional removable / non-removable, volatile / non-volatile memory media. Although not shown in Figure 13, a disk drive for reading from or writing to a removable, non-volatile disk and an optical disk drive for reading from and writing to a removable, non-volatile optical disk may also be provided. In such cases, each drive may be connected to a bus (not shown) via one or more data medium interfaces.

[0094] The communication unit 1340 facilitates communication with another computing device over a communication medium. Furthermore, the functionality of the components within the electronic device 1300 may be implemented by a single computing cluster or by multiple computers that can communicate with each other over a communication connection. Thus, the electronic device 1300 may operate in a networked environment using logical connections with one or more other servers, network personal computers (PCs), or further general network nodes.

[0095] The input device(s) 1350 may include one or more of various input devices such as a mouse, a keyboard, a data import device, etc. The output device(s) 1360 may be one or more output devices such as a display and a data export device. The communication unit 1340 may enable the electronic device 1300 to further communicate, as needed, with one or more external devices (not shown), such as a storage device or a display device, one or more devices that allow a user to interact with the electronic device 1300, or any devices (such as a network card and modem) that allow the electronic device 1300 to communicate with one or more other computing devices. Such communication may be performed via an input / output (I / O) interface (not shown).

[0096] In some implementations, instead of being integrated into a single device, some or all of the components of electronic device 1300 may be arranged in the form of a cloud computing architecture. In a cloud computing architecture, components can be provided remotely and work together to implement the functionality described in the subject matter described herein. In some implementations, cloud computing provides computing, software, data access, and storage services, without end users needing to be aware of the physical location or configuration of the systems or hardware providing these services. In various implementations, cloud computing provides services over a wide area network (such as the Internet) using appropriate protocols. For example, a cloud computing provider provides applications over a wide area network that can be accessed by a web browser or any other computing component. Software or components of a cloud computing architecture and corresponding data may be stored on servers in remote locations. Computational resources in a cloud computing environment may be aggregated or distributed at remote data center locations. While servers act as a single access point for users, the cloud computing infrastructure may provide services through a shared data center. Thus, a cloud computing infrastructure may be utilized to provide the components and functionality described herein from a remotely located service provider. Alternatively, it may be provided from a conventional server, or may be installed directly or otherwise on the client device.

[0097] The electronic device 1300 may be used to implement resource management according to various implementations of the subject matter described herein. The memory 1320 may include one or more modules having one or more program instructions. These modules may be accessed and executed by the processing unit 1310 to perform functions of various implementations described herein. For example, the memory 1320 may include a video encoding module 1322 for performing video encoding using a neural video codec. As shown in FIG. 13 , the electronic device 1300 may obtain a video to be encoded or a bitstream to be decoded via the input device 1350 and provide the encoded bitstream or the decoded video via the output device 1360. In some implementations, the electronic device 1300 may further receive input from other devices (not shown) via the communication unit 1340.

[0098] Some example implementations of the subject matter described herein are listed below.

[0099] In one aspect, the subject matter described herein provides a computer-implemented method that includes obtaining estimated motion information of a target frame in a video and reference feature information and a reference reconstructed frame of a reference frame related to the target frame; determining context information for the target frame based on the estimated motion information, the reference reconstructed frame, and the reference feature information using a context extraction model; and generating a target reconstructed frame of the target frame based on at least the context information using a frame coding model in converting between the target frame and a video bitstream.

[0100] In some implementations, the reference feature information includes a plurality of reference feature maps. In some implementations, determining the context information includes dividing the plurality of reference feature maps into a first number of feature map groups; for each feature map group among the first number of feature map groups, determining a second number of motion offsets from the reference frame to the target frame based on the estimated motion information, the reference reconstruction frame, and the feature map group, where the second number is greater than 1, performing respective motion alignments on the feature map groups based on the second number of motion offsets to obtain the second number of aligned feature map groups; and determining the context information based on the determined aligned feature map groups for the first number of feature map groups.

[0101] In some implementations, determining the context information based on the aligned feature map groups determined for the first number of feature map groups includes: sorting the aligned feature map groups determined for the first number of feature map groups; merging the sorted aligned feature map groups to obtain a first number of merged feature map groups, each merged feature map group being merged from a second number of aligned feature maps selected from the sorted aligned feature map groups; and determining the context information based on the first number of merged feature map groups.

[0102] In some implementations, dividing the plurality of reference feature maps into a first number of feature map groups includes dividing the plurality of reference feature maps into the first number of feature map groups along a channel dimension.

[0103] In some implementations, generating a target reconstructed frame includes determining a quantization code representation for the target frame, the quantization code representation including elements organized along a channel dimension and a spatial dimension; dividing the elements of the quantization code representation into a plurality of channel element groups along the channel dimension; dividing each channel element group of the plurality of channel element groups into a plurality of spatial element groups corresponding to a plurality of spatial locations; and performing a plurality of entropy encoding operations, wherein each entropy encoding operation entropy encodes a plurality of spatial element groups corresponding to a plurality of spatial locations within the plurality of channel element groups, and the combinations of the plurality of spatial element groups entropy encoded in the plurality of entropy encoding operations are different.

[0104] In some implementations, performing the plurality of entropy encoding operations includes, for a given entropy encoding operation of the plurality of entropy encoding operations, performing the given entropy encoding operation based on an entropy encoding result of at least one of the plurality of spatial element groups from an entropy encoding operation preceding the given entropy encoding operation.

[0105] In some implementations, the context extraction model and the frame coding model are trained using a plurality of sample frames in the sample video, and a plurality of weights are assigned to the plurality of sample frames, at least one of the plurality of weights being higher than the other weights. In some implementations, a training objective for training the context extraction model and the frame coding model is determined by determining a plurality of errors between the plurality of sample frames and a plurality of reconstructed sample frames output using the context extraction model and the frame coding model; and weighting the plurality of errors with the plurality of assigned weights.

[0106] In some implementations, the weight values ​​follow a hierarchical structure.

[0107] In some implementations, the conversion includes encoding the target frame as at least part of a bitstream or decoding the target frame from the bitstream.

[0108] In some implementations, the method further includes generating target feature information for the target frame based at least on the context information using a frame coding model in converting between the target frame and the video bitstream.

[0109] In another aspect, the subject matter described herein provides an electronic device comprising: a processor; and a memory, coupled to the processor, having instructions stored thereon, that, when executed by the processor, cause the device to perform operations of obtaining estimated motion information of a target frame in a video and reference feature information and a reference reconstructed frame of a reference frame for the target frame; determining, using a context extraction model, context information for the target frame based on the estimated motion information, the reference reconstructed frame, and the reference feature information; and generating, in a conversion between the target frame and a video bitstream, a target reconstructed frame of the target frame based on at least the context information, using a frame coding model.

[0110] In some implementations, the reference feature information includes a plurality of reference feature maps. In some implementations, determining the context information includes dividing the plurality of reference feature maps into a first number of feature map groups; for each feature map group among the first number of feature map groups, determining a second number of motion offsets from the reference frame to the target frame based on the estimated motion information, the reference reconstruction frame, and the feature map group, where the second number is greater than 1, performing respective motion alignments on the feature map groups based on the second number of motion offsets to obtain the second number of aligned feature map groups; and determining the context information based on the determined aligned feature map groups for the first number of feature map groups.

[0111] In some implementations, determining the context information based on the aligned feature map groups determined for the first number of feature map groups includes: sorting the aligned feature map groups determined for the first number of feature map groups; merging the sorted aligned feature map groups to obtain a first number of merged feature map groups, each merged feature map group being merged from a second number of aligned feature maps selected from the sorted aligned feature map groups; and determining the context information based on the first number of merged feature map groups.

[0112] In some implementations, dividing the plurality of reference feature maps into a first number of feature map groups includes dividing the plurality of reference feature maps into the first number of feature map groups along a channel dimension.

[0113] In some implementations, generating a target reconstructed frame includes determining a quantization code representation for the target frame, the quantization code representation including elements organized along a channel dimension and a spatial dimension; dividing the elements of the quantization code representation into a plurality of channel element groups along the channel dimension; dividing each channel element group of the plurality of channel element groups into a plurality of spatial element groups corresponding to a plurality of spatial locations; and performing a plurality of entropy encoding operations, wherein each entropy encoding operation entropy encodes a plurality of spatial element groups corresponding to a plurality of spatial locations within the plurality of channel element groups, and the combinations of the plurality of spatial element groups entropy encoded in the plurality of entropy encoding operations are different.

[0114] In some implementations, performing the plurality of entropy encoding operations includes, for a given entropy encoding operation of the plurality of entropy encoding operations, performing the given entropy encoding operation based on an entropy encoding result of at least one of the plurality of spatial element groups from an entropy encoding operation preceding the given entropy encoding operation.

[0115] In some implementations, the context extraction model and the frame coding model are trained using a plurality of sample frames in the sample video, and a plurality of weights are assigned to the plurality of sample frames, at least one of the plurality of weights being higher than the other weights. In some implementations, a training objective for training the context extraction model and the frame coding model is determined by determining a plurality of errors between the plurality of sample frames and a plurality of reconstructed sample frames output using the context extraction model and the frame coding model; and weighting the plurality of errors with the plurality of assigned weights.

[0116] In some implementations, the weight values ​​follow a hierarchical structure.

[0117] In some implementations, the conversion includes encoding the target frame as at least part of a bitstream or decoding the target frame from the bitstream.

[0118] In some implementations, the operations further include generating target feature information for the target frame based at least on the context information using a frame coding model in converting between the target frame and the video bitstream.

[0119] In yet another aspect, the subject matter described herein provides a computer program product tangibly stored on a computer storage medium and including computer-executable instructions that, when executed by a device, cause the device to perform operations of obtaining estimated motion information of a target frame in a video and reference feature information and a reference reconstructed frame of a reference frame related to the target frame; determining context information for the target frame based on the estimated motion information, the reference reconstructed frame, and the reference feature information using a context extraction model; and generating a target reconstructed frame of the target frame based on at least the context information using a frame coding model in converting between the target frame and a video bitstream.

[0120] In yet another aspect, the subject matter described herein provides a computer-readable medium having stored thereon computer-executable instructions that, when executed by a device, cause the device to perform one or more example implementations of the methods in the above aspects.

[0121] The functionality described herein may be performed, at least in part, by one or more hardware logic components. By way of example, and not limitation, exemplary types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0122] Program code for performing the subject methods described herein may be written in any combination of one or more programming languages. The program code may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing flowchart, and when executed by the processor or controller, the program code causes the processor or controller to perform the functions / operations specified in the flowchart and / or block diagram. The program code may be executed in whole or in part on a machine, partly on a machine as a stand-alone software package, partly on a remote machine, or entirely on a remote machine or server.

[0123] In the context of the subject matter described herein, a machine-readable medium may be any tangible medium that can contain or store a program used by or in connection with an instruction execution system, flowchart, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium includes, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, flowchart, device, or any suitable combination of the above. More specific examples of machine-readable storage media include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0124] Furthermore, while operations are shown in a particular order, this should not be understood as requiring such operations to be performed in the particular order or sequential order shown, or that all illustrated operations be performed to achieve desirable results. In some situations, multitasking and parallel processing may be advantageous. Similarly, while the above discussion includes details of several specific implementations, these should not be construed as limitations on the scope of the subject matter described herein, but rather as descriptions of functionality that may be specific to particular implementations. Also, certain features that are described in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features that are described in a single implementation may also be implemented in various implementations separately or in any suitable subcombination.

[0125] Although the subject matter has been described in language specific to structural features and / or method acts, it is to be understood that the subject matter specified in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. Obtaining estimated motion information of a target frame in a video, and reference feature information and a reference reconstructed frame of a reference frame relative to the target frame; determining context information for the target frame based on the estimated motion information, the reference reconstructed frame, and the reference feature information using a context extraction model; generating a target reconstructed frame of the target frame based on at least the context information using a frame coding model in converting between the target frame and the video bitstream; 10. A computer-implemented method comprising:

2. The reference feature information includes a plurality of reference feature maps, and determining the context information comprises: Dividing the plurality of reference feature maps into a first number of feature map groups; For each feature map group in the first number of feature map groups: determining a second number of motion offsets from the reference frame to the target frame based on the estimated motion information, the reference reconstructed frame, and the feature map group, wherein the second number is greater than one; performing respective motion alignments on the feature map groups based on the second number of motion offsets to obtain the second number of aligned feature map groups; determining the context information based on the aligned feature map groups determined for the first number of feature map groups; The method of claim 1 , comprising:

3. determining the context information based on the aligned feature map groups determined for the first number of feature map groups; Sorting the aligned feature map groups determined with respect to the first number of feature map groups; merging the sorted aligned feature map groups to obtain the first number of merged feature map groups, each merged feature map group being merged from the second number of aligned feature maps selected from the sorted aligned feature map groups; determining the context information based on the first number of merged feature map groups; and The method of claim 2 , comprising:

4. Dividing the plurality of reference feature maps into the first number of feature map groups, Dividing the plurality of reference feature maps into the first number of feature map groups along a channel dimension. The method of claim 2 , comprising:

5. generating the target reconstruction frame; determining a quantization code representation for the target frame, the quantization code representation including elements organized along a channel dimension and a spatial dimension; dividing the elements of the quantized code representation into a plurality of channel element groups along the channel dimension; Dividing each channel element group of the plurality of channel element groups into a plurality of spatial element groups corresponding to a plurality of spatial locations; performing a plurality of entropy coding operations, wherein in each entropy coding operation, a plurality of spatial element groups corresponding to the plurality of spatial locations within the plurality of channel element groups are entropy coded, and a combination of the plurality of spatial element groups that is entropy coded in the plurality of entropy coding operations is different. The method of claim 1.

6. performing the plurality of entropy coding operations includes, for a given entropy coding operation of the plurality of entropy coding operations: performing the given entropy coding operation based on entropy coding results of at least one of a plurality of spatial element groups from an entropy coding operation preceding the given entropy coding operation; The method of claim 5 , comprising:

7. the context extraction model and the frame coding model are trained using a plurality of sample frames in a sample video, and a plurality of weights are assigned to the plurality of sample frames, at least one of the weights being higher than the other weights; a training objective for training the context extraction model and the frame coding model, determining a plurality of errors between the plurality of sample frames and a plurality of reconstructed sample frames output using the context extraction model and the frame coding model; weighting the plurality of errors with the plurality of assigned weights; is determined by The method of claim 1.

8. The method of claim 7 , wherein the values ​​of the plurality of weights follow a hierarchical structure.

9. The conversion is encoding the target frame as at least part of the bitstream; or decoding the target frame from the bitstream; The method of claim 1 , comprising:

10. generating target feature information for the target frame based on at least the context information using the frame coding model in the conversion between the target frame and the bitstream of the video; The method of claim 1 further comprising:

11. a processor; a memory coupled to the processor and having instructions stored thereon, the instructions, when executed by the processor, causing the device to: obtaining estimated motion information of a target frame in a video, and reference feature information and a reference reconstructed frame of a reference frame relative to the target frame; determining context information for the target frame based on the estimated motion information, the reference reconstructed frame, and the reference feature information using a context extraction model; generating a target reconstructed frame of the target frame based on at least the context information using a frame coding model in converting between the target frame and the video bitstream; An electronic device that runs

12. The reference feature information includes a plurality of reference feature maps, and determining the context information comprises: Dividing the plurality of reference feature maps into a first number of feature map groups; For each feature map group in the first number of feature map groups: determining a second number of motion offsets from the reference frame to the target frame based on the estimated motion information, the reference reconstructed frame, and the feature map group, wherein the second number is greater than one; performing respective motion alignments on the feature map groups based on the second number of motion offsets to obtain the second number of aligned feature map groups; determining the context information based on the aligned feature map groups determined for the first number of feature map groups; The device of claim 11 , comprising:

13. determining the context information based on the aligned feature map groups determined for the first number of feature map groups; Sorting the aligned feature map groups determined with respect to the first number of feature map groups; merging the sorted aligned feature map groups to obtain the first number of merged feature map groups, each merged feature map group being merged from the second number of aligned feature maps selected from the sorted aligned feature map groups; determining the context information based on the first number of merged feature map groups; and The device of claim 12, comprising:

14. generating the target reconstruction frame; determining a quantization code representation for the target frame, the quantization code representation including elements organized along a channel dimension and a spatial dimension; dividing the elements of the quantized code representation into a plurality of channel element groups along the channel dimension; Dividing each channel element group of the plurality of channel element groups into a plurality of spatial element groups corresponding to a plurality of spatial locations; performing a plurality of entropy coding operations, wherein in each entropy coding operation, a plurality of spatial element groups corresponding to the plurality of spatial locations within the plurality of channel element groups are entropy coded, and a combination of the plurality of spatial element groups that is entropy coded in the plurality of entropy coding operations is different. The device of claim 11.

15. A computer program product tangibly stored on a computer storage medium and comprising computer-executable instructions, the instructions, when executed by a device, causing the device to: obtaining estimated motion information of a target frame in a video, and reference feature information and a reference reconstructed frame of a reference frame relative to the target frame; determining context information for the target frame based on the estimated motion information, the reference reconstructed frame, and the reference feature information using a context extraction model; generating a target reconstructed frame of the target frame based on at least the context information using a frame coding model in converting between the target frame and the video bitstream; A computer program product that causes the

Images