Improved tile averaging and frame averaging for frame context initialization
By using a weighted average of multiple reference frames to initialize the probability model of the current frame in video coding, the problem of inaccurate probability model initialization in existing technologies is solved, coding efficiency and accuracy are improved, and signaling overhead is reduced.
Patent Information
- Application Number
- CN202580026813.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2025-05-07
- Filing Date
- 2025-05-09
- Publication Date
- 2026-11-10
AI Technical Summary
Existing video coding technologies suffer from inaccurate initialization of probability models during entropy coding, resulting in high signaling overhead and insufficient coding precision.
The probability model of the current frame is initialized by a weighted average from multiple reference frames. The primary and secondary reference frames are selected, and tile averaging or frame averaging is selectively applied based on the number of tiles to improve coding accuracy and reduce signaling overhead.
By initializing the context of the current frame using a weighted average, the efficiency and accuracy of entropy coding are improved, while signaling overhead is reduced.
Smart Images

Figure CN123077084A_ABST
Abstract
Description
[0001] Related applications This application is a continuation-to-file of U.S. Patent Application No. 19 / 201,835, filed May 7, 2025, which claims priority to U.S. Provisional Patent Application No. 63 / 720,082, filed November 13, 2024, entitled "Improved Tile Averaging and Frame Averaging for Frame Context Initialization," the entire contents of which are incorporated herein by reference. Technical Field
[0002] The disclosed embodiments generally relate to video encoding and decoding, and more particularly to, but not limited to, systems and methods for entropy coding and probabilistic models. Background Technology
[0003] Digital video is supported by a variety of electronic devices, such as digital televisions, laptops or desktop computers, tablets, digital cameras, digital recording devices, digital media players, video game consoles, smartphones, video conferencing equipment, and video streaming devices. These electronic devices send and receive, or otherwise transmit, digital video data via communication networks, and / or store digital video data on storage devices. Due to the limited bandwidth capacity of communication networks and the limited storage resources of storage devices, video encoding can be employed to compress video data according to at least one video encoding standard before transmission or storage. Video encoding can be performed by hardware and / or software on electronic devices / client devices or on servers providing cloud services.
[0004] Video coding typically employs predictive methods (e.g., inter-frame prediction, intra-frame prediction, etc.) that leverage the inherent redundancy in video data. Video coding aims to compress video data into a form using a lower bitrate while avoiding or minimizing video quality degradation. Several video codec standards have been developed. For example, High Efficiency Video Coding (HEVC / H.265) is a video compression standard designed as part of the MPEG-H project. The ITU-T and ISO / IEC released the HEVC / H.265 standard in 2013 (Revision 1), 2014 (Revision 2), 2015 (Revision 3), and 2016 (Revision 4). Multifunctional Video Coding (VVC / H.266) is a video compression standard designed to succeed HEVC. The ITU-T and ISO / IEC released the VVC / H.266 standard in 2020 (Revision 1) and 2022 (Revision 2). AOMedia Video 1 (AV1) is an open video coding format designed as an alternative to HEVC. On January 8, 2019, specification version 1.0.0, along with errata 1, was released. Summary of the Invention
[0005] This invention describes a set of methods for video (image) compression, including methods for switching probabilistic models during entropy coding and decoding of video bitstream data. The probabilistic model can be characterized as a cumulative density function (CDF) for arithmetic coding. The probabilistic model can be initialized based on encoded information, making the probabilities of entropy coding more accurate than probabilistic models using default initialization values. In one example, a weighted average of the corresponding probabilistic models from multiple reference frames can be used as the initialization value for at least one context of the current frame. Initializing the current frame context using a weighted average of the context from reference frames can reduce signaling overhead (e.g., due to the increased efficiency of entropy coding) and improve coding accuracy (e.g., a more accurate context can provide better predictions). In one example, when multiple reference frames exist in the reference frame buffer, and the primary reference frame is not equal to the derived reference frame, the primary reference frame and the derived primary reference frame can be selected to generate a weighted average of the frame probabilistic models. Using a weighted average of multiple reference frames or selectively initializing the current frame with a single reference can improve coding accuracy and reduce signaling overhead. In addition, selectively initializing the current frame using a weighted average of multiple reference frames or a weighted average of multiple tiles in a single reference frame can improve coding accuracy and efficiency compared to always using multiple reference frames and / or always using tile-based averaging.
[0006] According to some embodiments, a video decoding method includes: (i) receiving a video bitstream comprising multiple frames (e.g., an encoded video sequence); (ii) selecting a primary reference frame for the current frame among multiple frames from a set of reference frames; (iii) when the number of tiles of the primary reference frame is less than a threshold: (a) selecting a secondary reference frame from the set of reference frames; (b) obtaining the corresponding contexts of the primary and secondary reference frames; and (c) initializing the current frame context of the current frame by performing a weighted average on the corresponding contexts; (iv) when the number of tiles of the primary reference frame is greater than the threshold, initializing the current frame context of the current frame without using the secondary reference frames; (v) using the current frame context to entropy decode at least one syntax element of the current frame; and (vi) reconstructing the current frame using information from at least one syntax element.
[0007] According to some embodiments, a video decoding method includes: (i) receiving a video bitstream comprising multiple frames; (ii) selecting a primary reference frame for a current frame from a set of reference frames; (iii) for each tile of a plurality of tiles of the current frame in the plurality of frames: (a) when the number of tiles of the primary reference frame is greater than a threshold, determining whether to apply tile averaging or frame averaging to initialize the context of the tile; (b) when using frame averaging to initialize the context of the tile: (1) selecting a secondary reference frame from a set of reference frames; (2) obtaining the corresponding contexts of the primary reference frame and the secondary reference frame; and (3) (c) When using tile averaging to initialize the context of a tile: (i) select two or more tiles from the main reference frame; (ii) obtain the context of the two or more tiles; and (iii) initialize the context of the current frame by performing a weighted average on the context of the two or more tiles; (iv) use the context to entropy decode at least one syntax element of the tile; and (v) reconstruct the tile using the information from at least one syntax element.
[0008] According to some embodiments, a video coding method includes: (i) receiving video data (e.g., a source video sequence) comprising multiple frames; (ii) selecting a primary reference frame for the current frame among the multiple frames from a set of reference frames; (iii) when the number of tiles of the primary reference frame is less than a threshold: (a) selecting a secondary reference frame from the set of reference frames; (b) obtaining the corresponding contexts of the primary and secondary reference frames; and (c) initializing the current frame context of the current frame by performing a weighted average on the corresponding contexts; (iv) when the number of tiles of the primary reference frame is greater than the threshold, initializing the current frame context of the current frame without using the secondary reference frames; and (v) using the current frame context to entropy encode at least one parameter of the current frame.
[0009] According to some embodiments, a method for processing visual media data includes: (i) obtaining a source video sequence comprising multiple frames; and (ii) performing a conversion between the source video sequence and a video stream of visual media data according to format rules, wherein the video stream comprises multiple encoded frames; and wherein the format rules specify: (a) selecting a primary reference frame from a set of reference frames for the current frame among the multiple frames; (b) when the number of tiles of the primary reference frame is less than a threshold: (1) selecting a secondary reference frame from a set of reference frames; (2) obtaining the corresponding contexts of the primary and secondary reference frames; and (3) initializing the current frame context of the current frame by performing a weighted average on the corresponding contexts; (c) when the number of tiles of the primary reference frame is greater than a threshold, initializing the current frame context of the current frame without using the secondary reference frames; (d) using the current frame context to entropy decode at least one syntax element of the current frame; and (e) using information from at least one syntax element to reconstruct the current frame.
[0010] According to some embodiments, a computing system, such as a streaming media system, a server system, a personal computer system, or other electronic devices, is provided. The computing system includes control circuitry and a memory storing at least one instruction set. The at least one instruction set includes instructions for performing any of the methods described in this application. In some embodiments, the computing system includes an encoder component and a decoder component (e.g., a transcoder).
[0011] According to some embodiments, a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium stores at least one set of instructions executable by a computing system. The at least one set of instructions includes instructions for performing any of the methods described in this application.
[0012] Therefore, apparatus and systems utilizing video encoding and decoding methods are disclosed. Such methods, apparatus, and systems can complement or replace conventional methods, apparatus, and systems used for video encoding / decoding. The features and advantages described in this specification are not necessarily comprehensive; in particular, some additional features and advantages will become more apparent to those skilled in the art from the accompanying drawings, description, and claims provided with this invention. Furthermore, it should be noted that the wording used in this specification has been chosen primarily for readability and guidance purposes and is not necessarily intended to define or limit the subject matter described in this application. Attached Figure Description
[0013] To gain a more detailed understanding of the invention, reference can be made to the features of various embodiments, some of which are illustrated in the accompanying drawings. However, the drawings only illustrate relevant features of the invention and are therefore not necessarily intended to be limiting, as other effective features may be covered in this specification, as will be understood by those skilled in the art upon reading this invention.
[0014] Figure 1 This is a block diagram illustrating an exemplary communication system according to some embodiments.
[0015] Figure 2A This is a block diagram illustrating exemplary elements of an encoder assembly according to some embodiments.
[0016] Figure 2B This is a block diagram illustrating exemplary elements of a decoder component according to some embodiments.
[0017] Figure 3 This is a block diagram illustrating an exemplary server system according to some embodiments.
[0018] Figures 4A-4C Exemplary prediction blocks, residual blocks, and reconstruction blocks are shown according to some embodiments.
[0019] Figure 5A An exemplary master reference frame selection process according to some embodiments is shown.
[0020] Figure 5B An exemplary secondary reference frame selection process according to some embodiments is shown.
[0021] Figure 6A An exemplary video decoding process according to some embodiments is shown.
[0022] Figure 6B An exemplary video encoding process according to some embodiments is shown.
[0023] By convention, the various features shown in the accompanying drawings are not necessarily drawn to scale, and the same reference numerals may be used to denote the same features throughout the specification and the accompanying drawings. Detailed Implementation
[0024] This invention describes a set of methods for video (image) compression, including methods for initializing a probabilistic model. Initialization of the probabilistic model can occur midway through the encoding and decoding process. For example, initialization can occur at the beginning of a sequence, at the beginning of a group of pictures (GOP), at the beginning of a frame, for each coding block, for each partition, and / or for each transform block. As an example, the initialization of the context of the current frame can be achieved by: (1) performing a weighted average of the context from multiple reference frames when the number of tiles in the main reference frame is less than a threshold; and (2) performing a weighted average of the context from multiple tiles in the main reference frame when the number of tiles in the main reference frame is greater than a threshold. Selectively applying tile-based averaging or frame-based averaging based on the number of tiles can improve coding accuracy and efficiency compared to always applying tile-based averaging and / or always applying frame-based averaging.
[0025] Exemplary systems and devices Figure 1 This is a block diagram illustrating a communication system 100 according to some embodiments. The communication system 100 includes a source device 102 and a plurality of electronic devices 120 (e.g., electronic devices 120-1 to 120-m) that are communicatively coupled to each other via at least one network. In some embodiments, the communication system 100 is a streaming media system, for example, for use with video-enabled applications such as video conferencing applications, digital TV applications, and media storage and / or distribution applications.
[0026] Source device 102 includes a video source 104 (e.g., a camera assembly or media storage device) and an encoder assembly 106. In some embodiments, the video source 104 is a digital camera (e.g., configured to produce an uncompressed video sample stream). The encoder assembly 106 generates at least one encoded video stream from the video stream. The video stream from the video source 104 may have a larger data volume compared to the encoded video stream 108 generated by the encoder assembly 106. Because the encoded video stream 108 has a smaller data volume (less data) compared to the video stream from the video source, less bandwidth is required to transmit the encoded video stream 108, and less storage space is required to store the encoded video stream 108 compared to the video stream from the video source 104. In some embodiments, source device 102 does not include the encoder assembly 106 (e.g., configured to transmit uncompressed video to one or more networks 110).
[0027] At least one network 110 represents any number of networks that transmit information between source device 102, server system 112, and / or electronic device 120, including, for example, wired (connected) and / or wireless communication networks. At least one network 110 can exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and / or the Internet.
[0028] At least one network 110 includes a server system 112 (e.g., a distributed / cloud computing system). In some embodiments, the server system 112 is or includes a streaming media server (e.g., configured to store and / or distribute video content, such as an encoded video stream from source device 102). The server system 112 includes a codec component 114 (e.g., configured to encode and / or decode video data). In some embodiments, the codec component 114 includes an encoder component and / or a decoder component. In various embodiments, the codec component 114 is instantiated as hardware, software, or a combination thereof. In some embodiments, the codec component 114 is configured to decode the encoded video stream 108 and re-encode the video data using different encoding standards and / or methods to generate encoded video data 116. In some embodiments, the server system 112 is configured to generate multiple video formats and / or encodings based on the encoded video stream 108. In some embodiments, the server system 112 serves as a Media-Aware Network Element (MANE). For example, server system 112 can be configured to prune the encoded video stream 108 to customize a potentially different stream for at least one electronic device 120. In some embodiments, MANE is set up separately from server system 112.
[0029] Electronic device 120-1 includes a decoder component 122 and a display 124. In some embodiments, the decoder component 122 is configured to decode encoded video data 116 to generate an output video stream that can be displayed on a display or other type of display device. In some embodiments, at least one electronic device 120 does not include a display component (e.g., it is communicatively coupled to an external display device and / or includes a media storage device). In some embodiments, electronic device 120 is a streaming media client. In some embodiments, electronic device 120 is configured to access server system 112 to obtain encoded video data 116. In some embodiments, source device 102 and / or at least one electronic device 120 are examples of: server systems, personal computers, portable devices (e.g., smartphones, tablets, or laptops), wearable devices, video conferencing equipment, and / or other types of electronic devices.
[0030] In an exemplary operation of the communication system 100, a source device 102 transmits an encoded video stream 108 to a server system 112. For example, the source device 102 may encode an image stream acquired by the source device. The server system 112 receives the encoded video stream 108 and may decode and / or encode the encoded video stream 108 using a codec component 114. For example, the server system 112 may apply encoding more suitable for network transmission and / or storage to the video data. The server system 112 may transmit encoded video data 116 (e.g., at least one encoded video stream) to at least one electronic device 120. Each electronic device 120 may decode the encoded video data 116 and optionally display the video image.
[0031] Figure 2A This is a block diagram illustrating exemplary elements of an encoder assembly 106 according to some embodiments. The encoder assembly 106 receives video data (e.g., a source video sequence) from a video source 104. In some embodiments, the encoder assembly includes a receiver (e.g., transceiver) assembly configured to receive the source video sequence. In some embodiments, the encoder assembly 106 receives the video sequence from a remote video source (e.g., a component of another device different from the encoder assembly 106). The video source 104 can provide a source video sequence in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8-bit, 10-bit, or 12-bit), any color space (e.g., BT.601 YCrCB or RGB), and any suitable sampling structure (e.g., YCrCb 4:2:0 or YCrCb 4:4:4). In some embodiments, the video source 104 is a storage device storing previously acquired / prepared video. In some embodiments, the video source 104 is a camera that acquires local image information as a video sequence. The video data can be provided as multiple independent images that, when viewed sequentially, present motion effects. An image itself can be organized as a spatial array of pixels, where each pixel may include at least one sample, depending on the sampling structure, color space, etc., used. Those skilled in the art can readily understand the relationship between pixels and samples.
[0032] Encoder component 106 is configured to encode and / or compress images of a source video sequence into an encoded video sequence 216 in real time or under other time constraints, as required by the application. In some embodiments, encoder component 106 is configured to perform a conversion between the source video sequence and a visual media data stream (e.g., a video stream). Enforcing an appropriate encoding rate is one of the functions of controller 204. In some embodiments, controller 204 controls and is functionally coupled to other functional units as described below. Parameters set by controller 204 may include rate control related parameters (e.g., image skipping, quantizer, and / or lambda value of rate-distortion optimization techniques), image size, GOP layout, maximum motion vector search range, etc. Other functions of controller 204 will be readily identified by those skilled in the art, as these other functions may involve encoder component 106 being optimized for a specific system design.
[0033] In some embodiments, encoder component 106 is configured to operate in an encoding loop. In a simplified example, the encoding loop includes a source encoder 202 (e.g., responsible for generating symbols, such as a symbol stream, based on the input image to be encoded and one or more reference images) and a (local) decoder 210. Decoder 210 reconstructs the symbols in a manner similar to that of the (remote) decoder to produce sample data (when the compression between the symbols and the encoded video stream is lossless). The reconstructed sample stream (sample data) is input to reference image memory 208. Since decoding the symbol stream produces a bit-by-bit accurate result regardless of the decoder's location (local or remote), the contents of reference image memory 208 are also bit-by-bit accurate between the local encoder and the remote encoder. In this way, the sample values interpreted by the encoder's prediction portion as reference image samples are the same as the sample values interpreted by the decoder when using the prediction during decoding.
[0034] The operation of decoder 210 can be the same as that of a remote decoder, such as decoder component 122, as described below. Figure 2B It will be described in detail. However, a brief reference is provided. Figure 2B When symbols are available and the entropy encoder 214 and the parser 254 are able to encode / decode the symbols into an encoded / decoded video sequence without loss, the entropy decoding portion of the decoder component 122, including the buffer memory 252 and the parser 254, may not be fully implemented in the local decoder 210.
[0035] Aside from parsing / entropy decoding, the decoder techniques described in this application can exist in the corresponding encoders with substantially the same functional form. Therefore, the disclosed subject matter focuses on decoder operation. Furthermore, the description of the encoder techniques can be brief, as these encoder techniques may be the inverse process of the decoder techniques.
[0036] As part of the operation of the source encoder 202, the source encoder 202 may perform motion-compensated predictive coding, referencing at least one previously encoded frame in the video sequence designated as a reference frame, to predictively encode the input frame. In this manner, the encoding engine 212 encodes the differences between pixel blocks of an input frame and pixel blocks of one or more reference frames, wherein the one or more reference frames may be selected as one or more predictive references for the input frame. The controller 204 may manage the encoding operations of the source encoder 202, including, for example, setting parameters and subgroup parameters for encoding video data.
[0037] Decoder 210 decodes encoded video data of frames that can be designated as reference frames based on symbols generated by source encoder 202. The operation of encoding engine 212 can advantageously be a lossy process. When in video decoder ( Figure 2A When decoding encoded video data at a location (not shown), the reconstructed video sequence may be a copy of the source video sequence with some errors. Decoder 210 replicates the decoding process, which can be performed on the reference frame by a remote video decoder, and the reconstructed reference frame can be stored in reference image memory 208. In this way, encoder component 106 locally stores a copy of the reconstructed reference frame that shares common content with the reconstructed reference frame obtained by the remote video decoder (in the absence of transmission errors).
[0038] Predictor 206 can perform a prediction search against encoding engine 212. That is, for a new frame to be encoded, predictor 206 can search the reference image memory 208 for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., which can be used as an appropriate prediction reference for the new image. Predictor 206 can operate pixel-by-pixel based on the sample blocks to find an appropriate prediction reference. Based on the search results obtained by predictor 206, the input image can have prediction references taken from multiple reference images stored in reference image memory 208.
[0039] The outputs of all the aforementioned functional units can be entropy encoded in the entropy encoder 214. The entropy encoder 214 converts these symbols into an encoded video sequence by lossless compression of the symbols generated by the various functional units according to techniques known to those skilled in the art (e.g., Huffman coding, variable-length coding, and / or arithmetic coding).
[0040] In some embodiments, the output of entropy encoder 214 is coupled to a transmitter. The transmitter can be configured to buffer one or more encoded video sequences generated by entropy encoder 214 in preparation for transmission via communication channel 218, which may be a hardware / software link to a storage device for storing the encoded video data. The transmitter can be configured to combine encoded video data from source encoder 202 with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (data sources not shown). In some embodiments, the transmitter may transmit additional data along with the encoded video. Source encoder 202 may include such data as part of the encoded video sequence. Additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant images and slices, Supplementary Enhancement Information (SEI) messages, Visual Usability Information (VUI) parameter set fragments, etc.
[0041] Controller 204 can manage the operation of encoder component 106. During encoding, controller 204 can assign a specific type of encoded image to each encoded image, which may affect the encoding technique applied to the corresponding image. For example, an image can be assigned as an intra-frame image (I-image), a predictive image (P-image), or a bidirectional predictive image (B-image). Intra-frame images can be encoded and decoded without using any other frames in the sequence as prediction sources. Some video codecs support different types of intra-frame images, including, for example, Independent Decoder Refresh (IDR) images. These variations of I-images and their corresponding applications and characteristics are well known to those skilled in the art and will not be repeated here. Predictive images can be encoded and decoded by predicting sample values for each block via intra-frame prediction or via inter-frame prediction using at most one motion vector and reference index. Bidirectional predictive images can be encoded and decoded by predicting sample values for each block via intra-frame prediction or via inter-frame prediction using at most two motion vectors and reference indices. Similarly, multiple predictive images can reconstruct a single block using more than two reference images and associated metadata.
[0042] Source images can typically be spatially divided into multiple sample blocks (e.g., each block contains 4×4, 8×8, 4×8, or 16×16 samples) and encoded block by block. Blocks can be predictively coded with reference to other (already coded) blocks, which are determined by the coding assignments of the corresponding images applied to those blocks. For example, blocks of an I-image can be unpredictably coded, or these blocks can be predictively coded (spatial prediction or intra-frame prediction) with reference to already coded blocks of the same image. Pixel blocks of a P-image can be unpredictably coded, coded via spatial prediction, or coded via temporal prediction with reference to a previously coded reference image. Blocks of a B-image can be unpredictably coded with reference to one or two previously coded reference images, via spatial prediction or via temporal prediction.
[0043] Video can be captured as multiple source images (video images) in a time-series manner. Intra-frame image prediction (often simply called intra-prediction) utilizes spatial correlations within a given image, while inter-frame image prediction utilizes (temporal or other) correlations between images. In one example, a specific image in encoding / decoding (called the current image) is divided into multiple blocks. When a block in the current image is similar to a reference block in a previously encoded reference image in the video that is still being buffered, that block in the current image can be encoded using a vector called a motion vector. The motion vector points to the reference block in the reference image and, in the case of using multiple reference images, can have a third dimension that identifies that reference image.
[0044] Encoder component 106 can perform encoding operations according to a predetermined video coding technique or standard, which is any technique or standard such as those described in this application. In operation, encoder component 106 can perform various compression operations, including predictive coding operations utilizing temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.
[0045] Figure 2B This is a block diagram illustrating exemplary elements of a decoder component 122 according to some embodiments. Figure 2B The decoder component 122 is coupled to the channel 218 and the display 124. In some embodiments, the decoder component 122 includes a transmitter coupled to the loop filter 256 and configured to transmit data to the display 124 (e.g., via a wired or wireless connection).
[0046] In some embodiments, decoder component 122 includes a receiver coupled to channel 218 and configured to receive data from channel 218 (e.g., via a wired or wireless connection). The receiver may be configured to receive at least one encoded video sequence to be decoded by decoder component 122. In some embodiments, decoding of each encoded video sequence is independent of other encoded video sequences. Each encoded video sequence may be received from channel 218, which may be a hardware / software link to a storage device storing the encoded video data. The receiver may receive the encoded video data along with other data, such as encoded audio data and / or auxiliary data streams, which may be forwarded to their respective user entities (not depicted). The receiver may separate the encoded video sequences from other data. In some embodiments, the receiver receives additional (redundant) data along with the encoded video. The additional data may be included as part of one or more encoded video sequences. The additional data may be used by decoder component 122 to decode the data and / or more accurately reconstruct the original video data. Additional data can take the form of, for example, temporal, spatial, or SNR enhancement layers, redundant slices, redundant images, forward error correction codes, etc.
[0047] According to some embodiments, decoder component 122 includes buffer memory 252, parser 254 (sometimes also called entropy decoder), scaler / inverse transform unit 258, intra-frame image prediction unit 262, motion compensation prediction unit 260, aggregator 268, loop filter unit 256, reference image memory 266, and current image memory 264. In some embodiments, decoder component 122 is implemented as an integrated circuit, a series of integrated circuits, and / or other electronic circuits. Decoder component 122 may be implemented at least partially in software.
[0048] Buffer memory 252 is coupled between channel 218 and parser 254 (e.g., for combating network jitter). In some embodiments, buffer memory 252 is separate from decoder component 122. In some embodiments, a separate buffer memory is provided between the output of channel 218 and decoder component 122. In some embodiments, in addition to buffer memory 252 inside decoder component 122 (e.g., buffer memory configured to handle playback timing), a separate buffer memory is also provided outside decoder component 122 (e.g., for combating network jitter). Buffer memory 252 may not be necessary or may be small when receiving data from a store / forward device with sufficient bandwidth and controllability or from an isosynchronous network. Buffer memory 252 may be required for use on best-effort packet networks such as the Internet. Buffer memory 252 may be relatively large and / or have an adaptive size and may be implemented at least partially in the operating system or in a similar element outside decoder component 122.
[0049] Parser 254 is configured to reconstruct symbols 270 from the encoded video sequence. These symbols may include, for example, information for managing the operation of decoder component 122, and / or information for controlling display devices such as display 124. Control information for one or more display devices may be in the form of, for example, SEI messages or VUI parameter set fragments (not depicted). Parser 254 parses (entropy decodes) the encoded video sequence. Encoding of the encoded video sequence may be performed according to video coding techniques or standards and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. Parser 254 may extract a subgroup parameter set from the encoded video sequence for at least one pixel subgroup in the video decoder based on at least one parameter corresponding to a group. Subgroups may include GOPs, images, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. Parser 254 may also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the encoded video sequence.
[0050] Depending on the type of the encoded video image or its portions (such as inter-frame and intra-frame images, inter-frame blocks and intra-frame blocks) and other factors, the reconstruction of symbol 270 may involve multiple different units. Which units are involved and how these units are involved can be controlled by subgroup control information parsed from the encoded video sequence by parser 254. For clarity, the flow of such subgroup control information between parser 254 and the multiple units described below is not depicted.
[0051] The decoder component 122 can conceptually be divided into multiple functional units, and in some embodiments, these units interact closely with each other and can be at least partially integrated with each other. However, for clarity, the conceptual division of functional units is maintained in this application.
[0052] The scaler / inverse transform unit 258 receives from the parser 254 quantized transform coefficients as one or more symbols 270, along with control information (such as which transform to use, block size, quantization factor, and / or quantization scaling matrix). The scaler / inverse transform unit 258 can output blocks comprising sample values that can be input to the aggregator 268. In some cases, the output samples of the scaler / inverse transform unit 258 relate to intra-coded blocks; that is, the blocks do not use predictive information from previously reconstructed images, but instead can use predictive information from previously reconstructed portions of the current image. Such predictive information can be provided by the intra-image prediction unit 262. The intra-image prediction unit 262 can use surrounding reconstructed information obtained from the current (partially reconstructed) image in the current image memory 264 to generate blocks of the same size and shape as the blocks in the reconstruction. The aggregator 268 can add the predictive information generated by the intra-image prediction unit 262 to the output sample information provided by the scaler / inverse transform unit 258 on a per-sample basis.
[0053] In other cases, the output samples of the scaler / inverse transform unit 258 may relate to inter-frame coded and potentially motion-compensated blocks. In such cases, the motion compensation prediction unit 260 can access the reference image memory 266 to obtain samples for prediction. After motion compensation of the obtained samples according to the block-related symbols 270, these samples can be added by the aggregator 268 to the output of the scaler / inverse transform unit 258 (referred to in this case as residual samples or residual signals) to generate output sample information. The address within the reference image memory 266 for the motion compensation prediction unit 260 to obtain predicted samples can be controlled by motion vectors. Motion vectors can be used by the motion compensation prediction unit 260 in the form of symbols 270, which may have, for example, X, Y, and reference image components. Motion compensation may also include, for example, interpolation of sample values obtained from the reference image memory 266 when using subsampled precise motion vectors, and motion vector prediction mechanisms.
[0054] The output samples of aggregator 268 can be employed by various loop filtering techniques in loop filter unit 256. Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video bitstream and provided to loop filter unit 256 from parser 254 in the form of symbol 270, but may also respond to metadata obtained during decoding of previous (in decoding order) portions of the encoded image or encoded video sequence, and to previously reconstructed and loop-filtered sample values. The output of loop filter unit 256 may be a sample stream, which can be output to a display device such as display 124 and stored in reference image memory 266 for future inter-frame image prediction.
[0055] Once a certain encoded image is reconstructed, it can be used as a reference image for future predictions. Once an encoded image is reconstructed and has been identified as a reference image (e.g., by the parser 254), the current reference image can become part of the reference image memory 266, and a new current image memory can be reallocated before reconstructing subsequent encoded images begins.
[0056] Decoder component 122 can perform decoding operations according to predetermined video compression techniques that may be described in standards such as any standards described in this application. If the encoded video sequence follows the syntax of a video compression technique or standard, such as the syntax specified in a video compression technique document or standard, particularly in its configuration file document, then the encoded video sequence can conform to the syntax specified by the video compression technique or standard used. Furthermore, in order to conform to some video compression techniques or standards, the complexity of the encoded video sequence can be within the range defined by the hierarchy of the video compression technique or standard. The hierarchy can define the maximum image size, maximum frame rate, maximum reconstruction sampling rate (measured, for example, in megasamples per second), maximum reference image size, etc. The limitations set by the hierarchy can be further defined by the Hypothetical Reference Decoder (HRD) specification and the HRD buffer management metadata indicated by signaling in the encoded video sequence.
[0057] Figure 3This is a block diagram illustrating a server system 112 according to some embodiments. The server system 112 includes control circuitry 302, at least one network interface 304, memory 314, a user interface 306, and at least one communication bus 312 for interconnecting these components. In some embodiments, the control circuitry 302 includes at least one processor (e.g., CPU, GPU, and / or DPU). In some embodiments, the control circuitry includes one or more field-programmable gate arrays, multiple hardware accelerators, and / or one or more integrated circuits (e.g., application-specific integrated circuits).
[0058] One or more network interfaces 304 can be configured to connect to at least one communication network (e.g., a wireless network, a wired network, and / or an optical network). The communication network can be a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a vehicular and industrial network, a real-time network, a latency-tolerant network, etc. Examples of communication networks include: LANs such as Ethernet; wireless LANs; cellular networks including GSM, 3G, 4G, 5G, LTE, etc.; wired or wireless wide area digital television networks, including cable TV, satellite TV, and terrestrial broadcast TV; vehicular and industrial networks, including CANBus, etc. Such communication can be one-way receiving (e.g., broadcast TV), one-way transmitting (e.g., CANBus to certain CANBus devices), or bidirectional (e.g., connecting to other computer systems using a LAN or WAN digital network). Such communication can include communication with at least one cloud computing network.
[0059] User interface 306 includes at least one output device 308 and / or at least one input device 310. One or more input devices 310 may include at least one of the following: keyboard, mouse, touchpad, touchscreen, data glove, joystick, microphone, scanner, camera, etc. One or more output devices 308 may include at least one of the following: audio output device (e.g., speaker), visual output device (e.g., display or monitor), etc.
[0060] Memory 314 may include high-speed random access memory (such as DRAM, SRAM, DDR RAM, and / or other random access solid-state storage devices) and / or non-volatile memory (such as at least one disk storage device, optical disk storage device, flash memory device, and / or other non-volatile solid-state storage devices). Memory 314 may optionally include at least one storage device located remotely from control circuitry 302. Memory 314, or alternatively, one or more non-volatile solid-state storage devices within memory 314, include non-transitory computer-readable storage media. In some embodiments, memory 314 or the non-transitory computer-readable storage media of memory 314 stores programs, modules, instructions, and data structures, or subsets or supersets thereof: ● Operating system 316, which includes programs for handling various basic system services and for performing hardware-dependent tasks; ● Network communication module 318, which is used to connect server system 112 to other computing devices via at least one network interface 304 (e.g., via wired and / or wireless connection); ● An encoding / decoding module 320 is used to perform various functions related to encoding and / or decoding data such as video data. In some embodiments, the encoding / decoding module 320 is an instance of the codec component 114. The encoding / decoding module 320 includes, but is not limited to, at least one of the following: ○ Decoding module 322, which performs various functions related to decoding encoded data, such as those previously described for decoder component 122; and ○ Encoding module 340, which performs various functions related to encoding data, such as those previously described for encoder component 106; and ● Image memory 352, which is used to store images and image data, for example, for use by the encoding / decoding module 320. In some embodiments, image memory 352 includes at least one of the following: reference image memory 208, buffer memory 252, current image memory 264, and reference image memory 266.
[0061] In some embodiments, the decoding module 322 includes a parsing module 324 (e.g., configured to perform various functions previously described for the parser 254), a transformation module 326 (e.g., configured to perform various functions previously described for the scaler / inverse transformation unit 258), a prediction module 328 (e.g., configured to perform various functions previously described for the motion compensation prediction unit 260 and / or the intra-frame image prediction unit 262), and a filter module 330 (e.g., configured to perform various functions previously described for the loop filter 256).
[0062] In some embodiments, the encoding module 340 includes a code module 342 (e.g., configured to perform various functions previously described for the source encoder 202 and / or encoding engine 212) and a prediction module 344 (e.g., configured to perform various functions previously described for the predictor 206). In some embodiments, the decoding module 322 and / or the encoding module 340 includes Figure 3 A subset of the modules shown. For example, the shared prediction module is shared by both the decoding module 322 and the encoding module 340.
[0063] The modules identified above and stored in memory 314 correspond to instruction sets for performing the functions described herein. The modules identified above (e.g., instruction sets) do not need to be implemented as separate software programs, processes, or modules; therefore, in various embodiments, subsets of these modules may be combined or otherwise rearranged. For example, the codec module 320 may optionally not include separate decoding and encoding modules, but instead use the same set of modules to perform both sets of functions. In some embodiments, memory 314 stores a subset of the modules and data structures identified above. In some embodiments, memory 314 stores additional modules and data structures not described above.
[0064] although Figure 3 A server system 112 according to some embodiments is shown, but... Figure 3 This is intended more as a functional description of various features that may exist in at least one server system, rather than a structural schematic diagram of the embodiments described in this application. In practice, items shown individually may be combined, and some items may be separated. For example, Figure 3 Some of the items shown separately can be implemented on a single server, while a single item can be implemented by at least one server. The actual number of servers used to implement server system 112 and how features are distributed among these servers will vary depending on the specific implementation and, optionally, in part, on the amount of data traffic processed by the server system during peak and average usage periods.
[0065] Exemplary codec techniques The encoding and decoding processes and techniques described below can be performed by the aforementioned devices and systems (e.g., source device 102, server system 112, and / or electronic device 120).
[0066] As previously stated, the present invention relates to entropy coding and probabilistic model initialization. Entropy coding in video codecs uses context-based (e.g., conditional) probabilities, enabling the encoding of symbols using at least one probabilistic model that optimally captures statistical data governed by conditional or higher-order conditions.
[0067] As used in this application, the term "probabilistic model" (sometimes referred to as "symbol probabilistic model") refers to any aspect relating to the derivation of the probability of symbols during entropy coding, which is used to determine the output of a context-based arithmetic coding engine. Examples of design aspects of the probabilistic model may include context initialization probability / state, probability update window size / speed, context derivation process, number of contexts, number of windows, or codewords.
[0068] The term "block" can refer to a coding tree block, a maximum coding block, a predefined fixed block size, a coding block, a prediction block, a residual block, or a transform block.
[0069] In some embodiments, a default pattern is encoded without searching for a better pattern decision (e.g., a more efficient encoding decision). The default pattern can be used when the encoder lacks sufficient computational power to perform a search and / or determine a better pattern. The probabilistic model options used to encode this default pattern can be designed to optimally encode it.
[0070] In some embodiments, the initialization or adaptation of the probabilistic model can occur at any point during the encoding and / or decoding process, for example, at the beginning of the sequence, at the beginning of the GOP, at the beginning of the frame, for each coded block, for each partition, and / or for each transform block. The group of symbolic probabilistic models associated with the entire frame may be referred to as the “frame context” (or “tile context”).
[0071] Now let's turn to an example encoding and decoding using predictive blocks and residual blocks. Figure 4A The calculation of a prediction block is illustrated according to some embodiments. Figure 4A In the example, intra-frame prediction is performed on the current block 402 to generate prediction block 404. In some embodiments, inter-frame prediction is performed to generate prediction blocks. The current block 402 includes a set of samples (e.g., pixel blocks), and prediction block 404 includes a set of predictions corresponding to that set of samples. Figure 4B The calculation of residual blocks according to some embodiments is shown. Figure 4B As shown, prediction block 404 is subtracted from current block 402 to generate residual block 406, which includes a set of residual values. For example, the corresponding difference between each sample and its corresponding prediction is calculated. Figure 4C The calculation of the reconstructed block is shown according to some embodiments. For example... Figure 4C As shown, residual block 406 undergoes at least one transform and quantization to generate a set of residual coefficients. This set of residual coefficients can be transmitted from the encoder component to the decoder component as part of the video bitstream. This set of residual coefficients undergoes inverse quantization and inverse transform to generate a reconstructed residual block 408. The reconstructed residual block 408 is combined with the prediction block 404 (e.g., the reconstructed residual value of the reconstructed residual block 408 is added to the prediction of the prediction block 404) to generate a reconstructed block 410 corresponding to the current block 402.
[0072] These coefficients and other encoding parameters are indicated by signaling in the video bitstream. This signaling indication may involve bypass coding, arithmetic coding, and / or other types of coding (e.g., to reduce signaling overhead). Some systems use tree-based Boolean non-adaptive binary arithmetic encoders to encode the syntax elements of the video bitstream.
[0073] Entropy coding in modern codecs has widely adopted context-based (e.g., conditional) probabilities, enabling the encoding of symbols using probabilistic models that optimally capture statistics governed by conditional or higher-order conditions. Some systems employ adaptive multi-symbol (M-ary) arithmetic encoders (e.g., symbol-to-symbol adaptive encoders). For example, each grammatical element can be a member of a specific alphabet comprising N elements, and the corresponding context consists of a set of N probabilities and a count, for example, to facilitate fast early adaptation. Higher precision (compared to binary arithmetic encoders) allows for accurate tracking of the probabilities of less common elements in the alphabet. The probabilities can be adaptively adjusted via simple recursive scaling, with the update factor based on the alphabet size.
[0074] Some systems use many Symbolic arithmetic coding (CDF) is used to compress syntax elements. In some embodiments, the probability model used for arithmetic coding is updated for each coded symbol. CDF can be used to characterize the probability that a random variable might take a value less than or equal to a specific threshold. For example, some codecs store data in CDF format. many Symbol probability in a number system. Arithmetic coding can directly use CDF to compress symbols. For example, for many A random variable in base n has a probability mass function (PMF) at time n that can be defined as shown in Equation 1 below.
[0075]
[0076] Formula 1 – Exemplary PMF The corresponding CDF can be defined as shown in Formula 2 below.
[0077]
[0078] Formula 2 – Exemplary CDF in The elements in Formula 2 can be scaled (e.g., by...). To achieve integer precision.
[0079] In this example, a new result was observed when the symbols were encoded. Furthermore, the probability model can be updated as shown in Equation 3 below.
[0080]
[0081] Formula 3 – Exemplary Probabilistic Model Update in It is an indicator vector whose k-th element is 1 and all other elements are 0, and This is the update rate of the probabilistic model. The CDF can be updated as shown in Equation 4 below.
[0082]
[0083] Formula 4 – Exemplary CDF Update many Number system symbol arithmetic encoding can be based on All floating-point data is scaled and represented by 15-bit unsigned integers. Some implementations employ a dual-model approach to adapt the multiplication operations involved to 16 bits. For example, the probabilistic model CDF is updated and maintained with 15-bit precision, but when used for entropy coding, only the most significant 9 bits are input into the arithmetic encoder.
[0084] As mentioned earlier, the initialization of the probabilistic model can occur at various stages of the encoding and decoding process. Initialization can use a reference context to generate probabilities that are more accurate than those provided by the default initialization values. In some embodiments, the reference context is obtained from a reference frame of the current frame.
[0085] Figure 5A An exemplary master reference frame selection process according to some embodiments is shown. Figure 5A Method 500 can be executed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having control circuitry and a memory storing instructions for execution by the control circuitry. In some embodiments, method 500 is executed by executing instructions stored in the computing system's memory (e.g., memory 314).
[0086] The system (e.g., the system's encoding or decoding component) defines a list of reference frames for the current frame (502). One or more contexts for each reference frame may be stored along with the frame data.
[0087] The system derives (504) a master reference frame for the current frame. This master reference frame is sometimes referred to as the “derived master reference frame”. The encoding component and the corresponding decoding component derive the same master reference frame using available encoded data (e.g., previously decoded data) and at least one predefined rule.
[0088] The system selects (506) a master reference frame. For example, the encoding component may perform additional tests on each reference frame to confirm whether the derived master reference frame is the optimal reference frame (e.g., whether it has the lowest associated cost). In some embodiments, the encoding component signals via the video bitstream whether the derived master reference frame is the optimal reference frame. In some embodiments, the encoding component signals with the index of the optimal reference frame in the reference frame list. For example, if the derived master reference frame is the same as the optimal master reference frame, it signals with 0 (or 1), otherwise it signals with 1 (or 0). The context of the master reference frame can be used to initialize the context of the current frame. In some embodiments, frame context initialization uses a weighted average of the contexts of two reference frames. For example, one reference frame is the selected master reference frame, and the other reference frame is selected as the second reference frame. In some embodiments, the encoder determines and signals with the master reference frame in the frame header, which is used to initiate the frame context of that frame. To signal with the master reference frame, a flag can be signaled to indicate whether it is the same as the derived master reference frame. If so, the master reference frame is set as the derived master reference frame. Otherwise, signaling can be used to indicate the optimal reference frame.
[0089] Figure 5B An exemplary secondary reference frame selection process according to some embodiments is shown. Figure 5B Method 550 can be executed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having control circuitry and memory storing instructions for execution by the control circuitry. In some embodiments, method 550 is executed by executing instructions stored in the computing system's memory (e.g., memory 314). In some embodiments, method 550 is executed by the same system as method 500 described above.
[0090] The system (e.g., the system's encoding or decoding component) defines a list of reference frames for the current frame (552). One or more contexts for each reference frame may be stored along with the frame data. The same list of reference frames can be used to identify primary and secondary reference frames.
[0091] The system derives (554) secondary reference frames for the current frame. These secondary reference frames are sometimes referred to as "derived secondary reference frames". The encoding component and the corresponding decoding component derive the same secondary reference frames using available encoded data (e.g., previously decoded data) and at least one predefined rule.
[0092] In some embodiments, the reference frame with the lowest index in the reference frame buffer and located on the same side as the main reference frame is used as the second reference frame. Table 1 below illustrates the improvements in signal-to-noise ratio and coding time achieved by performing simulations on various video data (e.g., characterizing AOM test conditions) using existing designs (e.g., AVM research-v8).
[0093]
[0094] Table 1 – Simulation Results In some embodiments, the selected secondary reference frame is used as a second reference frame (e.g., the secondary reference frame is indicated by signaling in the video stream). Table 2 below shows the improvement in signal-to-noise ratio achieved by performing simulations on various video data (e.g., characterizing AOM test conditions) using an existing design (e.g., AVM research-v8).
[0095]
[0096] Table 2 – Simulation Results The system selects (556) secondary reference frames. For example, the encoding component may perform additional tests on each reference frame to confirm whether the derived secondary reference frame is the optimal reference frame (e.g., whether it has the lowest association cost). In some embodiments, the encoding component uses signaling via the video bitstream to indicate whether the derived secondary reference frame is the optimal secondary reference frame. In some embodiments, the encoding component uses signaling to indicate the index of the optimal secondary reference frame in the reference frame list.
[0097] In some embodiments, when encoding a frame using multiple tiles, the CDFs of the multiple tiles are averaged (instead of using only the CDF of a single tile) to obtain the final CDF of the frame. Using the averaged CDF can improve coding accuracy, thereby improving coding efficiency. In some embodiments, a simple averaging algorithm is used to calculate the average CDF of each tile (e.g., to simplify hardware implementation). In some embodiments, a weighted averaging algorithm is used. In some embodiments, only shift-sum operations are used to calculate the average CDF.
[0098] Figure 6A This is a flowchart illustrating a method 600 for decoding video according to some embodiments. Method 600 can be executed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having control circuitry and a memory storing instructions for execution by the control circuitry. In some embodiments, method 600 is executed by executing instructions stored in the computing system's memory (e.g., memory 314). In some embodiments, method 600 is executed by the same system as methods 500 and 550 described above.
[0099] The system receives (602) a video stream (e.g., an encoded video sequence) comprising multiple frames. The system selects (604) a primary reference frame for the current frame from a set of reference frames. For example, the system may use a primary reference frame derived at the decoding component, or it may use a primary reference frame indicated by signaling. When the number of tiles in the primary reference frame is less than a threshold (606): the system selects (608) secondary reference frames from the set of reference frames and initializes (610) the current frame context of the current frame by performing a weighted average of the corresponding contexts from the primary and secondary reference frames. When the number of tiles in the primary reference frame is greater than the threshold, the system initializes (612) the current frame context of the current frame without using secondary reference frames. In some embodiments, when the number of tiles in the primary reference frame is greater than the threshold, the system initializes the current frame context using a weighted average of the corresponding contexts from the multiple tiles of the primary reference frame. In some embodiments, the system uses the current frame context to perform entropy decoding on at least one syntax element of the current frame. In some embodiments, the system uses information from at least one syntax element to reconstruct the current frame. In this way, when the number of tiles in a frame is less than or equal to N (N is a positive integer), the frame context of the current frame is initialized using a weighted average of multiple frame CDFs from the reference frame buffer.
[0100] In some embodiments, the weighting factors used to perform a context-weighted average of multiple reference frames can be predefined, implicitly derived, or explicitly indicated by signaling. In some embodiments, only one predefined weighting factor is used. In some embodiments, multiple predefined weighting factors are supported, and the selection of a predefined weighting factor can depend on the distance between the current frame and the reference frames, and / or the QP value of the reference frames, and / or the difference in QP values between the current frame and the reference frames. In some embodiments, the weighting factor is proportional to the distance between the current frame and the reference frames. As an example, if the distance between the first reference frame and the current reference frame is 3, and the distance between the second reference frame and the current reference frame is 1, then a weighting factor of ¼ is assigned to the first reference frame, and a weighting factor of ¾ is assigned to the second reference frame.
[0101] In some embodiments, a larger weight is assigned to a reference frame when it is closer to the current frame. In some embodiments, the weighting factor used for averaging the context of multiple reference frames may also depend on the syntax elements. In some embodiments, equal weighting factors are assigned to one set of syntax elements, and unequal weighting factors are assigned to another set of syntax elements. In some embodiments, unequal weighting factors are assigned to motion vector-related syntax. In some embodiments, equal weighting factors are always applied to one set of syntax elements, and weighting factors indicated by signaling or derived are applied only to the second set of syntax elements. As an example, motion vector-related syntax is included in the second set of syntax elements.
[0102] In some embodiments, previously encoded syntax elements can be used to determine the weighting factor of the current syntax element. In some embodiments, multiple predefined weighting factors are supported, and the selection of predefined weighting factors can be explicitly indicated by signaling at the sequence, frame, slice, and / or tile level.
[0103] In some embodiments, the supported weighting factors may be [1 / 2, 1 / 2], [1 / 4, 3 / 4], or [1 / 8, 7 / 8], and the index of the supported weighting factors is indicated by signaling in the bitstream. In one embodiment, when multiple reference frames exist in the reference frame buffer, the probability model of which two (or more) reference frames to use for weighted averaging can be determined implicitly or explicitly indicated by signaling.
[0104] In some embodiments, when multiple reference frames exist in the reference frame buffer, at least one reference frame is implicitly determined based on encoded information. In some embodiments, when the initial value of the probability model of the current frame is obtained using a weighted averaging method, one of the selected reference frames is set as the primary reference frame or the derived primary reference frame. In some embodiments, when the initial value of the context of the current frame is obtained using a weighted averaging method, one of the selected reference frames is set as the primary reference frame or the derived primary reference frame, and a second reference frame is set as the reference frame with the lowest index in the reference buffer, and the second reference frame is not equal to the first reference frame. In some embodiments, the selection of the second reference frame is indicated by signaling in the bitstream.
[0105] In some embodiments, when multiple reference frames exist in the reference frame buffer, the selection of two (or more) reference frames is explicitly indicated by signaling in the bitstream. In some embodiments, when multiple reference frames exist in the reference frame buffer, only a subset of combinations of two (or more) reference frames is allowed, and this subset is indicated by signaling in the bitstream. In some embodiments, when multiple reference frames exist in the reference frame buffer, the index of each reference frame in the reference frame buffer must be equal to or less than a threshold T1. As an example, T1 can be set to 2. In some embodiments, when multiple reference frames exist in the reference frame buffer, the distance between a reference frame and the current frame must be equal to or less than a threshold T2.
[0106] In some embodiments, the decision to use a weighted average of the probability models of multiple reference frames as the initial value for the context of the current frame is implicitly determined based on encoded information. In some embodiments, the weighted average may be applied only to a portion of the symbol probability model in the frame probability model. In some embodiments, the encoded information used to determine whether to use the weighted average of the context of multiple reference frames may include, but is not limited to, at least one bypass coded symbol in the high-level syntax (HLS). In some embodiments, the weighted average may be applied only to a portion of the symbol probability model in the frame probability model.
[0107] In some embodiments, when the primary reference frame is equal to the derived primary reference frame, a context-weighted average of multiple reference frames is applied. Otherwise, only the frame probability model of the primary reference frame is used for the current frame.
[0108] In some embodiments, when the primary reference frame is not equal to the derived primary reference frame, a context-weighted average of multiple reference frames is applied. Otherwise, only the frame probability model of the primary reference frame is used for the current frame.
[0109] In some embodiments, when more than one reference frame exists in the reference frame buffer, a context-weighted average of the multiple reference frames is applied. Otherwise, only the frame probability model of the primary reference frame is used for the current frame.
[0110] In some embodiments, when multiple reference frames exist in the reference frame buffer, and the distance difference between the selected two reference frames and the current frame is less than a threshold T, a weighted average of the contexts of the multiple reference frames is applied. Otherwise, only the frame probability model of the primary reference frame is used for the current frame.
[0111] In some embodiments, when multiple reference frames exist in the reference frame buffer and all reference frames originate from the same side of the current frame, a context-weighted average of the multiple reference frames is applied. Otherwise, only the frame probability model of the primary reference frame is used for the current frame.
[0112] In some embodiments, when multiple reference frames exist in the reference frame buffer, and at least two reference frames are from the same side of the current frame, a context-weighted average of the multiple reference frames is applied. Otherwise, only the frame probability model of the primary reference frame is used for the current frame.
[0113] In some embodiments, the decision of whether to use a weighted average of the contexts of multiple reference frames as the initial value for the context of the current frame is explicitly indicated by signaling in the bitstream or conditionally indicated by signaling. In some embodiments, a flag is indicated by signaling at the frame / tile / slice level to indicate whether to use a weighted average of the contexts of multiple reference frames or a weighted average of the contexts of the main reference frame as the initial value for the context of the current frame. In some embodiments, the decision of whether to use a weighted average of the contexts of multiple reference frames as the initial value for the context of the current frame is conditionally indicated by signaling in the bitstream based on encoded information.
[0114] In some embodiments, when the primary reference frame is equal to (or not equal to) a predefined reference frame, a flag is signaled at the frame / tile / slice level to indicate whether a weighted average of the contexts of multiple reference frames is used, or a weighted average of the contexts of the primary reference frame is used as the initial value for the context of the current frame. In some embodiments, this predefined reference frame is a derived primary reference frame.
[0115] In some embodiments, when the reference frames in the reference frame buffer are not from the same side of the current frame, a flag is indicated at the frame / tile / slice level using signaling to indicate whether a weighted average of the contexts of multiple reference frames is used, or a weighted average of the contexts of the main reference frame is used as the initial value of the context of the current frame.
[0116] In some embodiments, when the difference between the number of frames on both sides is greater than / less than a threshold, a flag is indicated by signaling at the frame / tile / slice level to indicate whether a weighted average of the contexts of multiple reference frames or a weighted average of the contexts of the main reference frame is used as the initial value of the context of the current frame.
[0117] The selection of multiple reference frames can be derived based on predefined rules. In some embodiments, when multiple reference frames exist in the reference frame buffer, and one of the reference frames is a key frame (or intra-frame), this key (or intra-frame) frame can be selected as one of the reference frames for the weighted average of the probability model for generating the frame.
[0118] In some embodiments, when a key (or intra-frame) frame is selected as the derived master reference frame, a weighted average of the probability models of multiple reference frames cannot be applied. In some embodiments, when a key (or intra-frame) frame is selected as the derived master reference frame, a weighted average of the probability models of multiple reference frames can be applied. In some embodiments, when a key (or intra-frame) frame exists in the reference frame buffer, a weighted average of the probability models of multiple reference frames can be applied. In some embodiments, when multiple reference frames exist in the reference frame buffer, one of the reference frames selected for weighted averaging is assigned as the master reference frame or the derived master reference frame.
[0119] In some embodiments, when multiple reference frames exist in the reference frame buffer, one of the reference frames selected for weighted averaging is assigned as the derived master reference frame (or primary reference frame), and another reference frame is assigned as a reference frame from the same direction as the derived master reference frame. If no other reference frame from the same direction as the derived master reference frame exists, weighted averaging cannot be applied to multiple reference frames.
[0120] In some embodiments, if multiple reference frames exist in the reference frame buffer that are in the same direction as the derived main reference frame (or main reference frame), the reference frame with the lowest sort index in the reference frame buffer (or reordered reference frame buffer) is selected as the second reference frame. In some embodiments, if multiple reference frames exist in the reference frame buffer that are in the same direction as the derived main reference frame (or main reference frame), the reference frame closest to the derived main reference frame (or main reference frame) is selected as the second reference frame. In some embodiments, when multiple reference frames exist in the reference frame buffer, one of the reference frames selected for weighted averaging is assigned as the derived main reference frame or main reference frame, and another reference frame is assigned as the reference frame closest to the main reference frame, the derived main reference frame, or the current frame. In some embodiments, when multiple reference frames exist in the reference frame buffer, one of the reference frames selected for weighted averaging is assigned as the derived main reference frame or main reference frame, and another reference frame is assigned as the reference frame with the time level ID closest to the current frame or the main reference frame.
[0121] In some embodiments, when multiple reference frames exist in the reference frame buffer and the primary reference frame is not equal to the derived reference frame, a weighted average can be applied to the probability models of the two reference frames. In some embodiments, when multiple reference frames exist in the reference frame buffer and the primary reference frame is not equal to the derived reference frame, the primary reference frame and the derived primary reference frame are selected to generate a weighted average of the frame probability models. In some embodiments, when multiple reference frames exist in the reference frame buffer and the primary reference frame is equal to the derived reference frame, a weighted average can be applied to the probability models of the two reference frames. In some embodiments, when multiple reference frames exist in the reference frame buffer and the primary reference frame is equal to the derived reference frame, one of the selected reference frames is set as the derived primary reference frame, and the other reference frame is set as a reference frame located in the same direction as the derived primary reference frame.
[0122] In some embodiments, when multiple reference frames exist in the reference frame buffer, and the primary reference frame is equal to the derived reference frame, one of the selected reference frames is set as the derived primary reference frame, and the other reference frame is set as the reference frame with the lowest sort index in the reference frame buffer (or the reordered reference frame buffer) and located in the same direction as the derived primary reference frame. In some embodiments, when multiple reference frames exist in the reference frame buffer, a set of predefined rules can be applied sequentially to select two reference frames to generate a weighted average of the probability model of that frame.
[0123] In some embodiments, a derived master reference frame or master reference frame is selected as the first selected frame for generating a weighted average. Two rules can be applied to find the second reference frame. First, check if there is a reference frame in the reference buffer with the same orientation as the first selected frame. If this condition is met, this reference frame is selected as the second reference frame. Otherwise, check the distances between reference frames in the reference frame buffer (or the reordered reference frame buffer) and select the reference frame closest to the first reference frame.
[0124] When the master reference frame has multiple tiles, the final CDF of the largest tile (which may also be the last tile) can be used to initialize the CDF model of the current frame or tile. The final CDF, along with the reconstructed master reference frame, can be stored in the decoded picture buffer (DPB).
[0125] In some embodiments, when the number of tiles in a frame is less than or equal to N (N is a positive integer), the frame context of the current frame is initialized using a weighted average of the CDFs of multiple frames from the reference frame buffer. In some embodiments, N is set to 1 or 2. In some embodiments, when the number of tiles in a frame is less than or equal to N, HLS is signaled to indicate that a weighted average of the contexts of multiple frames is used to initialize the context of the current frame. In some embodiments, when the number of tiles in a frame is less than or equal to N, the HLS flag is used to switch between: disabling CDF initialization using the reference frame; using only the primary reference frame; using only the secondary reference frame; or combining both the primary and secondary reference frames for CDF initialization. In some embodiments, when the number of tiles in a frame is greater than N, the default value of the context of the current frame / tile may also be averaged together with the updated context of that tile to generate the final CDF.
[0126] In some embodiments, when the number of tiles in a frame is greater than N, it is proposed to initialize the CDF of the current frame using a weighted average of the final CDFs of all or some tiles in the primary reference frame. Otherwise, when the number of tiles in a frame is less than or equal to N, the frame context of the current frame can be initialized using a weighted average of the CDFs of multiple frames from the reference frame buffer.
[0127] In some embodiments, when multiple tiles exist in a frame, a weighted average is applied to the multiple tiles in the current frame to generate the stored frame context of the current frame. The weighting factors may depend on the coding size of each tile. In some embodiments, there are some predefined weighting factors, and the selection of multiple predefined weighting factors may depend on the ratio of the coding sizes of different tiles. As an example, the predefined weighting factors may be [1 / 2, 1 / 2], [1 / 4, 3 / 4], [1 / 8, 7 / 8], [3 / 8, 5 / 8].
[0128] In some embodiments, if multiple tiles exist in a frame, each tile can decide whether to use the frame context stored in the reference frame buffer or to use the default CDF value as the starting point of the context.
[0129] In some embodiments, if there are multiple tiles in a frame, each tile can decide whether to use the frame context stored in the reference frame buffer or to use a weighted average of multiple reference frames as the starting point of the frame context.
[0130] In some embodiments, the CDF of the current frame, tile, or slice is initialized using a weighted average of the final CDFs of all or some tiles in the master reference frame. In one example, if the master reference frame is encoded using K tiles, a weighted average can be obtained using the CDFs from M tiles to generate the final stored CDF, where M <= K. Here, K <= 16. The final stored CDF can be used as a reference for subsequent frames for context initialization. In some embodiments, the maximum value of M can be a predefined value or a signaling indication value in the HLS, such as a sequence, frame, tile, and / or slice header. In some embodiments, only one tile from each tile row or tile column is used to perform a weighted average on the final CDF.
[0131] In some embodiments, the default value of the current frame / tile's context can also be averaged together with the tile's updated context to generate the final CDF. In some embodiments, the weights used for averaging can depend on the qindex (or the quantization parameter for each tile).
[0132] In some embodiments, equal weights are used to average the CDF from M tiles. In some embodiments, the weights may depend on the resolution of the tile (i.e., the width and height of the tile).
[0133] In some embodiments, the weights may depend on the relative positions of these tiles within the frame. In some embodiments, the weights vary for different syntaxes. In some embodiments, the HLS flag can be used in the sequence header to enable or disable weighted averaging.
[0134] In some embodiments, weighted averaging can be implicitly disabled or enabled based on the frame's time layer ID. In some embodiments, the HLS flag can be used in the frame header to signal the weights to be used for each tile. In one example, the index of the lookup table can be signaled in the frame header. This index will point to the actual weights used.
[0135] In some embodiments, the CDF of the current frame or tiles is initialized using a weighted average of the final CDFs of all or some tiles in the secondary reference frame. In some embodiments, the aspects described above for the primary reference frame also apply to the secondary reference frame.
[0136] In some embodiments, the CDF of the current frame or tile is initialized using a combination of the aspects described above for the primary and secondary reference frames. In some embodiments, this combination can be implemented by using a weighted average of the final CDFs obtained from the primary and secondary reference frames.
[0137] In some embodiments, the HLS flag is used to switch between: disabling CDF initialization using a reference frame; using only the primary reference frame; using only the secondary reference frame; or combining both the primary and secondary reference frames for CDF initialization. In some embodiments, the HLS flag is signaled to switch between a weighted average of the context of multiple tiles within a frame and a weighted average of the frame context of multiple frames. In some embodiments, this combination may be a weighted average based on the distances of the primary and secondary reference frames to the current frame.
[0138] In some embodiments, the context of the current frame is initialized using a weighted average of the contexts of two reference frames in the reference frame buffer. For example, a frame may be the primary reference frame in the reference frame buffer. In some embodiments, the reference frame with the lowest index in the reference frame buffer and located on the same side as the primary reference frame is used as the second reference frame.
[0139] In some embodiments, if a primary reference frame is indicated by signaling, the context is initialized using a weighted average of the context of the optimal primary reference frame and the context of the derived primary reference frame (if available). Otherwise, a weighted average of the contexts of the derived primary and secondary reference frames is used.
[0140] Figure 6B This is a flowchart illustrating a video encoding method 650 according to some embodiments. Method 650 can be executed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having control circuitry and a memory storing instructions for execution by the control circuitry. In some embodiments, method 650 is executed by executing instructions stored in the computing system's memory (e.g., memory 314). In some embodiments, method 650 is executed by the same system as method 600 described above.
[0141] The system receives (652) video data comprising multiple frames (e.g., a source video sequence). The system selects (654) a primary reference frame for the current frame from a set of reference frames. When the number of tiles in the primary reference frame is less than a threshold (656), the system selects (658) secondary reference frames from the set of reference frames and initializes (660) the current frame context of the current frame by performing a weighted average of the corresponding contexts from the primary and secondary reference frames. When the number of tiles in the primary reference frame is greater than the threshold, the system initializes (660) the current frame context of the current frame without using the secondary reference frames. In some embodiments, the system uses the current frame context to entropy-encode at least one parameter of the current frame.
[0142] Exemplary main reference frame selection In some embodiments, for the current encoded frame A, a list of reference frames is defined, and the context of each reference frame is stored along with its frame data. In some embodiments, in both the encoder and decoder, the main reference frame is derived for A based on the frame-level qindex (or QP parameter value), and the frame distances to the reference frames can be sorted based on the most recently encoded frame (as a second tie-break condition). This process may involve the following (1) through (3): (1) From the reference frames whose qindex is less than or equal to the qindex of A, select the reference frame R1 whose qindex is closest to the qindex of A. If more than one reference frame satisfies the above condition, select the reference frame closest to A. If more than one reference frame satisfies the above condition, select the reference frame with the later encoding order, which may have the most recently updated frame context. (2) Similarly, from the reference frames whose qindex is greater than or equal to the qindex of A, select the reference frame R2 whose qindex is closest to the qindex of A. If more than one reference frame satisfies the above condition, select the reference frame closest to A. If more than one reference frame meets the above conditions, the reference frame with the later encoding order is selected, which may have the most recently updated frame context. (3) If R1 exists, R1 is selected; otherwise, R2 is selected. The selected reference frame is the main reference frame derived from A.
[0143] In some embodiments, the encoder further performs an optimal selection of the primary reference frame. This process may involve the following steps (1) and (2): (1) After frame encoding, the reference frames are traversed for a search. For each reference frame, its frame context is used as the starting frame context of A, and entropy encoding of A is run. The resulting encoded frame size is recorded. (2) The reference frame with the smallest encoded frame size is selected as the optimal primary reference frame of A.
[0144] In some embodiments, a primary reference frame is indicated by signaling. This process may involve the following (1) and (2): (1) If the derived primary reference frame of A is the same as the optimal primary reference frame of A, then a signaling "0" is used in the bitstream; otherwise, a signaling "1" is used. Here, "0" indicates the use of the derived primary reference frame, and "1" indicates the use of the optimal primary reference frame. (2) If a signaling "1" is used above, then a signaling "1" is used in the uncompressed frame header to indicate the optimal primary reference frame.
[0145] Exemplary primary and secondary reference frame selection In some embodiments, a secondary reference frame is selected and signaled such that a weighted average of the probability models of the primary and secondary reference frames can be used as the initial value for the context of the current frame. The probability model for each reference frame may include, but is not limited to, context probability values, context states, and / or context update rates. The selection of the primary and secondary reference frames can be derived based on predefined rules.
[0146] In some embodiments, the selection and signaling indication of primary and secondary reference frames involve the following (I) to (IV). (I) For the current coded frame A, a list of reference frames is defined. The context of each reference frame is stored together with its frame data.
[0147] (II) In both the encoder and decoder, the primary and secondary reference frames are derived for A based on the frame-level qindex (or QP parameter value) and frame distance. This process may involve the following steps (1) through (3): (1) From the reference frames with qindex <= A, select the pair of reference frames (R1, R2) whose qindex is closest to A, if available. If more than two reference frames have the same qindex, select the one closest to A. If more than two reference frames have the same qindex and are closest to A, select the one with the later encoding order, which may have the most recently updated frame context. (2) Similarly, find the reference frames (R3, R4) with the higher and closest qindex. (3) If (R1, R2) exists, select (R1, R2); otherwise, select (R3, R4). The selected pair of reference frames are the primary and secondary reference frames derived from A.
[0148] (III) In the encoder, further optimization of the primary and secondary reference frames is performed. This process may involve the following (1) to (2). (1) After frame encoding, the reference frames are searched. For each reference frame, its frame context is used as the starting frame context of A, and entropy encoding of A is run. The resulting encoded frame size is recorded. (2) The two best reference frames with the smallest encoded frame size are selected, if available, as the optimal primary and secondary reference frames of A.
[0149] (IV) The encoder uses signaling to indicate the primary reference frame and the secondary reference frame. This process may involve the following (1)-(4). (1) If the derived primary reference frame of A is the same as the optimal primary reference frame of A, then signaling “0” is used in the bitstream; otherwise, signaling “1” is used. Here, “0” indicates the use of the derived primary reference frame, and “1” indicates the use of the optimal primary reference frame. (2) If signaling “1” is used above, then signaling the optimal primary reference frame is used in the uncompressed frame header using 3 bits. (3) If the derived secondary reference frame of A is the same as the optimal secondary reference frame of A, then signaling “0” is used in the bitstream; otherwise, signaling “1” is used. Here, “0” indicates the use of the derived secondary reference frame, and “1” indicates the use of the optimal secondary reference frame. (4) If signaling “1” is used above, then signaling the optimal secondary reference frame is used in the uncompressed frame header using 3 bits.
[0150] In some embodiments, when only two reference frames are available to the encoder or decoder, the steps previously described in the "Exemplary Primary Reference Frame Selection" section are used to select and signal the primary reference frame. In some embodiments of these embodiments, the selection and signaling of secondary reference frames are skipped.
[0151] In one embodiment, an HLS flag (e.g., an uncompressed sequence-level or frame-level flag) can be used to control the selection and signaling indication of secondary reference frames. In one example, if the HLS flag is "0", the selection and signaling indication of secondary reference frames can be skipped. In this case, only the steps described in Section 1 are used to select and signal the primary reference frame, and initialization is not performed using a weighted average of the context probability models (e.g., initialization is performed using only the primary reference frame context). In another example, if the HLS flag is "1", initialization is performed using a weighted average of the derived secondary reference frame and the context probability models of the primary reference frame, which was selected and signaled using the steps described in Section 1. In another example, if the HLS flag is "2", initialization is performed using a weighted average of the context probabilities of the selected and signaled primary reference frame and the secondary reference frame models.
[0152] In some embodiments, step (II) is modified to introduce temporal scalability. In some embodiments, in both the encoder and decoder, the primary and secondary reference frames are derived for A based on the frame-level qindex (or QP parameter value), frame distance, and temporal layer id of the current frame. This process may involve the following (1) through (3): (1) From the reference frames with qindex <= A and temporal_layer_id <= A, select a pair of reference frames (R1, R2) whose qindex is closest to A, if available. If more than two reference frames have the same qindex, select the (R1, R2) closest to A. If more than two reference frames have the same qindex and are closest to A, select the (R1, R2) whose temporal_layer_id is closest to A. If more than two reference frames have the same qindex, are closest to A, and have temporal_layer_id closest to A, select the (R1, R2) whose encoding order is later (which may have the most recently updated frame context). (2) Similarly, find the reference frame (R3, R4) with the higher and closest qindex. (3) If (R1, R2) exists, select (R1, R2); otherwise, select (R3, R4). The selected pair of reference frames are the primary and secondary reference frames derived from A.
[0153] In some embodiments, the weighted average of the context probability model is applied only to a subset of syntax elements. In one example, the weighted average of the context probability model is applied only to coefficient-encoded related syntax elements. In another example, the weighted average of the context probability model is not applied to coefficient-encoded related syntax elements. In yet another example, the weighted average of the context probability model is applied to a subset of coefficient-encoded related syntax elements. In yet another example, the subset of coefficient-encoded related syntax elements may include, but is not limited to, coefficient-level related syntax, coefficient EOB related syntax, and coefficient sign related syntax. In yet another example, the weighted average of the context probability model is applied only to motion vector related syntax elements. In yet another example, the weighted average of the frame context is applied only to inter-frame prediction modes and motion vector related syntax elements.
[0154] In some embodiments, the selection of the primary reference frame and the secondary reference frame may involve the following (I) to (IV). (I) For the current coded frame A, a list of reference frames is defined. The context of each reference frame is stored together with its frame data.
[0155] (II) For the derived primary and secondary reference frames, the primary and secondary reference frames are derived for A based on the frame-level qindex (or QP parameter value) and frame distance, as well as other possible parameters, in both the encoder and decoder. This process may involve the following (1) to (3): (1) From the reference frames with qindex less than A, select the pair of reference frames (R1, R2) whose qindex is closest to A, if available. If more than two reference frames have the same qindex, select the (R1, R2) closest to A. If more than two reference frames have the same qindex and are closest to A, select the (R1, R2) with the later encoding order, which may have the most recently updated frame context. (2) From the reference frames with qindex >= A, select the pair of reference frames (R3, R4) whose qindex is closest to A, if available. If more than two reference frames have the same qindex, select the (R3, R4) closest to A. If two or more reference frames have the same qindex and are closest to A, then the later-encoded frame (R3, R4) is selected, which may have the most recently updated frame context. (3) If (R1, R2) exists, then (R1, R2) is selected; otherwise, (R3, R4) is selected. The selected pair of reference frames is the primary reference frame and the secondary reference frame derived from A.
[0156] (III) In the encoder, the optimal selection of the main reference frame is further performed. This process may involve the following (1) to (2). After frame encoding, the reference frames are traversed for searching. For each reference frame, its frame context is used as the starting frame context of A, and entropy encoding of A is run. The resulting encoded frame size is recorded. (2) The reference frame with the smallest encoded frame size is selected as the optimal main reference frame of A.
[0157] (IV) Signaling the primary reference frame. In some embodiments, signaling indicates a binary flag. If the derived primary reference frame of A is the same as the optimal primary reference frame of A, then signaling indicates "0" in the bitstream; otherwise, signaling indicates "1". Here, "0" indicates the use of the derived primary reference frame, and "1" indicates the use of the reference frame to be signaled later. If signaling indicates "1" above, then three bits are used in the uncompressed frame header to signal the optimal primary reference frame.
[0158] In some embodiments, when the optimal primary reference frame is indicated by signaling in the bitstream, the optimal primary reference frame is not equal to the derived primary reference frame, and the current frame probability model is initialized using a weighted average of the probability models of the optimal primary reference frame and the derived primary reference frame.
[0159] In some embodiments, the weighted average is applied only if the optimal master reference frame and the derived master reference frame have the same orientation. Otherwise, the probability model for the current frame is initialized using the probability model from the optimal master reference frame.
[0160] In some embodiments, the weighted average is applied only if the relative distance between the optimal master reference frame and the derived master reference frame is less than a threshold (e.g., T2 <= {1, 2, 3, …, sub-GOP size}). Otherwise, the probability model for the current frame is initialized using the probability model from the optimal master reference frame.
[0161] In some embodiments, the weighted average is applied only if the difference in QP (e.g., qindex, quantization parameters, etc.) between the optimal master reference frame and the derived master reference frame is less than a threshold. Otherwise, the probability model for the current frame is initialized using the probability model from the optimal master reference frame.
[0162] In some embodiments, when the optimal primary reference frame is equal to the derived primary reference frame, the weighted average of the derived primary reference frame and the derived secondary reference frame is used to initialize the current frame probability model.
[0163] In some embodiments, the weighted average is applied only if the derived primary reference frame and the derived secondary reference frame have the same orientation. Otherwise, the probability model for the current frame is initialized using the probability model from the derived primary reference frame.
[0164] In some embodiments, the weighted average is applied only if the relative distance between the derived master reference frames is less than a threshold (e.g., T2 <= {1, 2, 3, … sub-GOP size}). Otherwise, the probability model for the current frame is initialized using the probability model from the derived master reference frames.
[0165] In some embodiments, the weighted average is applied only if the difference in QP (e.g., qindex, quantization parameters, etc.) between the derived master reference frame and the derived master reference frame is less than a threshold. Otherwise, the probability model of the current frame is initialized using the probability model from the derived master reference frame.
[0166] In some embodiments, when multiple tiles / slices exist in a frame, a weighted average of the contexts of the multiple tiles / slices is stored as the updated context of the current frame, which can be used as the context for subsequent frames. When averaging the contexts of multiple tiles / slices, a syntax can be indicated in the bitstream using signaling to indicate which tile / slice's context is used for the weighted average.
[0167] In some embodiments, at the encoding end, since the master reference frame has not yet been determined during the block-level RDO process, the frame context is initialized, and rate estimation is performed by setting a first reference frame as the derived master reference frame. A second reference frame is set as the reference frame with the lowest index in the reference frame buffer, which is located on the same side as the derived reference frame.
[0168] As mentioned earlier, the encoding process can be very similar to the decoding process described in this application (e.g., selecting the context and initializing the probability model). For the sake of brevity, these details will not be repeated here.
[0169] although Figure 6A and 6B Multiple logical stages are shown in a specific order, but these stages are independent of the order and can be reordered, and other stages can be combined or split. Some reorderings or other groupings not specifically mentioned will be obvious to those skilled in the art, and therefore the orderings and groupings presented in this application are not exhaustive. Furthermore, it should be understood that the stages can be implemented in hardware, firmware, software, or any combination thereof.
[0170] (A1) In one aspect, some embodiments include a video decoding method (e.g., method 600). In some embodiments, the method is executed at a computing system (e.g., server system 112) having memory and control circuitry. In some embodiments, the method is executed at an encoding / decoding module (e.g., encoding / decoding module 320). In some embodiments, the method is executed at a source encoding component (e.g., source encoder 202), an encoding engine (e.g., encoding engine 212), and / or an entropy encoder (e.g., entropy encoder 214). The method includes: (i) receiving a video stream comprising multiple frames; (ii) selecting a primary reference frame for the current frame among multiple frames from a set of reference frames; (iii) when the number of tiles in the primary reference frame is less than a threshold: (a) selecting a secondary reference frame from the set of reference frames; (b) obtaining the corresponding contexts of the primary and secondary reference frames; and (c) initializing the current frame context of the current frame by performing a weighted average on the corresponding contexts; (iv) when the number of tiles in the primary reference frame is greater than the threshold, initializing the current frame context of the current frame without using the secondary reference frames; (v) using the current frame context to entropy decode at least one syntax element of the current frame; and (vi) reconstructing the current frame using information from at least one syntax element. For example, when the number of tiles in a frame is less than or equal to N (N is a positive integer), the frame context of the current frame is initialized using a weighted average of multiple frame contexts from the reference frame buffer.
[0171] (A2) In some embodiments of A1, the threshold for the number of tiles is 2. For example, N can be set to 1 or 2.
[0172] (A3) In some embodiments of A1 or A2, the method further includes: parsing syntax elements from the HLS to determine whether a weighted average of multiple reference frames is enabled for the current frame, wherein when the syntax element indicates that a weighted average of multiple reference frames is enabled for the current frame, the current frame context of the current frame is initialized by performing a weighted average on the corresponding context. For example, when the number of tiles in a frame is less than or equal to N, the HLS is signaled (e.g., at the sequence level) to instruct the use of a weighted average of the contexts of multiple frames to initialize the context of the current frame.
[0173] (A4) In some embodiments of A3, the syntax element indicates whether to use zero reference frames, one reference frame, or multiple reference frames to initialize the current frame context of the current frame. For example, when the number of tiles in a frame is less than or equal to N, the HLS flag is used to switch between: disabling CDF initialization using reference frames; using only the primary reference frame; using only the secondary reference frame; or combining both primary and secondary reference frames for CDF initialization.
[0174] (A5) In some embodiments of any of A1 to A4, initializing the current frame context of the current frame without using a secondary reference frame includes: initializing the current frame context by performing a weighted average of the default value and the corresponding context of the tiles of the primary reference frame. For example, when the number of tiles in a frame is greater than N, the default value of the current frame / tile context may also be averaged together with the updated context of that tile to generate the final CDF.
[0175] (A6) In some embodiments of any of A1 to A4, initializing the current frame context of the current frame without using a secondary reference frame includes: initializing the current frame context by performing a weighted average of the respective contexts of multiple tiles from the primary reference frame. For example, when the number of tiles in a frame is greater than N, the weighted average of the final CDFs of all or some of the tiles in the primary reference frame can be used to initialize the CDF of the current frame. Otherwise, when the number of tiles in a frame is less than or equal to N, the weighted average of the CDFs of multiple frames from the reference frame buffer can be used to initialize the frame context of the current frame.
[0176] (A7) In some embodiments of A6, the weights of the weighted average of the corresponding contexts of multiple tiles from the master reference frame are based on the coding size of the multiple tiles. For example, when multiple tiles exist in a frame, a weighted average is applied to the multiple tiles within the current frame to generate the stored frame context of the current frame. The weighting factors may depend on the coding size of each tile.
[0177] (A8) In some embodiments of A7, weights are selected from a set of predefined weights for a weighted average of the corresponding contexts of multiple tiles from the master reference frame. For example, multiple predefined weight factors may exist, and the selection of multiple predefined weight factors may depend on the ratio of the coding sizes of the different tiles.
[0178] (A9) In some embodiments of A8, the weights of the corresponding contexts of multiple tiles from the main reference frame are equal. For example, the predefined weight factors can be [1 / 2, 1 / 2], [1 / 4, 3 / 4], [1 / 8, 7 / 8], [3 / 8, 5 / 8], or other values.
[0179] (A10) In some embodiments of any of A1 to A9, at least one first tile of the current frame has parameters for entropy coding using the current frame context, and at least one second tile of the current frame has parameters for coding without using the current frame context. For example, if there are multiple tiles in a frame, each tile of the current frame can use (e.g., stored in a reference frame buffer) the frame context, or can use a default CDF value as the starting point of the context.
[0180] (B1) In another aspect, some embodiments include a video coding method (e.g., method 650). In some embodiments, the method is performed at a computing system having memory and at least one processor. The method includes: (i) receiving video data (e.g., a source video sequence) comprising a plurality of frames; (ii) selecting a primary reference frame for a current frame among the plurality of frames from a set of reference frames; (iii) when the number of tiles of the primary reference frame is less than a threshold: (a) selecting a secondary reference frame from the set of reference frames; (b) obtaining the corresponding contexts of the primary and secondary reference frames; and (c) initializing a current frame context of the current frame by performing a weighted average on the corresponding contexts; (iv) when the number of tiles of the primary reference frame is greater than the threshold, initializing the current frame context of the current frame without using the secondary reference frames; and (v) using the current frame context to entropy-encode at least one parameter of the current frame. In some embodiments, the method includes signaling entropy-encoding parameters and the encoded current frame in the video bitstream.
[0181] (B2) In some embodiments of B1, the threshold for the number of tiles is 2.
[0182] (B3) In some embodiments of B1 or B2, the method further includes signaling a syntax element in the HLS to indicate whether a weighted average of multiple reference frames is enabled for the current frame.
[0183] (B4) In some embodiments of B3, the syntax element indicates whether to use zero reference frames, one reference frame, or multiple reference frames to initialize the current frame context of the current frame.
[0184] (B5) In some embodiments of any of B1 to B4, initializing the current frame context of the current frame without using a secondary reference frame includes: initializing the current frame context by performing a weighted average of the corresponding contexts of the default values and the tiles of the primary reference frame.
[0185] (B6) In some embodiments of any of B1 to B4, initializing the current frame context of the current frame without using a secondary reference frame includes: initializing the current frame context by performing a weighted average of the respective contexts of multiple tiles from the primary reference frame.
[0186] (B7) In some embodiments of B6, the weights of the weighted average of the corresponding contexts of multiple tiles from the main reference frame are equal.
[0187] (B8) In some embodiments of any of B1 to B7, the method further includes an encoding end similar to any of the features described above for A1 to A10.
[0188] (C1) In another aspect, some embodiments include a visual media data processing method. In some embodiments, the method is performed on a computing system having memory and at least one processor. The method includes: (i) obtaining a source video sequence comprising a plurality of images; and (ii) performing a conversion between the source video sequence and a bitstream of visual media data, wherein the bitstream comprises a plurality of encoded frames; and wherein a format rule specifies: (a) to select a primary reference frame from a set of reference frames for the current frame among a plurality of frames; (b) when the number of tiles of the primary reference frame is less than a threshold: (1) to select a secondary reference frame from a set of reference frames; (2) to obtain the corresponding contexts of the primary and secondary reference frames; and (3) to initialize the current frame context of the current frame by performing a weighted average on the corresponding contexts; (c) when the number of tiles of the primary reference frame is greater than a threshold, to initialize the current frame context of the current frame without using the secondary reference frames; (d) to use the current frame context to entropy decode at least one syntax element of the current frame; and (e) to reconstruct the current frame using information from at least one syntax element.
[0189] (D1) In another aspect, some embodiments include a video decoding method. In some embodiments, the method is performed at a computing system having memory and at least one processor. The method includes: (i) receiving a video bitstream comprising a plurality of frames; (ii) selecting a primary reference frame for a current frame from a set of reference frames; (iii) for each of a plurality of tiles of the current frame in the plurality of frames: (a) when the number of tiles of the primary reference frame is greater than a threshold, determining whether to apply tile averaging or frame averaging to initialize the context of the tile; (b) when frame averaging is used to initialize the context of the tile: (1) selecting a secondary reference frame from a set of reference frames; (2) obtaining the corresponding contexts of the primary reference frame and the secondary reference frame; and (3) by analyzing the primary reference frame... (c) When using tile averaging to initialize the context of a tile: (i) select two or more tiles from the primary reference frame; (ii) obtain the corresponding contexts of the two or more tiles; and (iii) initialize the context of the current frame by performing a weighted average on the corresponding contexts of the two or more tiles; (d) use the context to entropy decode at least one syntax element of the tile; and (e) reconstruct the tile using the information from at least one syntax element. For example, if multiple tiles exist in a frame, each tile of the current frame can use the frame context stored in the reference frame buffer, or a weighted average of multiple reference frames, as the starting point of the frame context.
[0190] In another aspect, some embodiments include a computing system (e.g., server system 112) including control circuitry (e.g., control circuitry 302) and a memory (e.g., memory 314) coupled to the control circuitry, the memory storing at least one set of instructions configured to be executed by the control circuitry, the at least one set of instructions including instructions for performing any of the methods described in this application (e.g., methods 500, 550, 600, and 650, and A1 to A10, B1 to B8, C1, and D1 above). In another aspect, some embodiments include a non-transitory computer-readable storage medium storing at least one set of instructions executable by the control circuitry of the computing system, the at least one set of instructions including instructions for performing any of the methods described in this application (e.g., methods 500, 550, 600, and 650, and A1 to A10, B1 to B8, C1, and D1 above).
[0191] Unless otherwise stated, any syntax element described in this application can be an HLS. As used in this application, HLS is indicated by signaling at a level higher than the block level. For example, HLS can correspond to the sequence level, frame level, slice level, or tile level. As another example, HLS elements can be indicated by signaling in a video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), adaptation parameter set (APS), slice header, picture header, tile header, and / or CTU header.
[0192] It should be understood that although the terms “first,” “second,” etc., may be used to describe various elements in this application, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to limit the claims. As used in the description of the embodiments and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and covers any and all possible combinations of at least one associated listed item. It should also be further understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of at least one other feature, integer, step, operation, element, component, and / or combination thereof.
[0193] As used in this application, the term "when" may be interpreted, depending on the context, as meaning "if" the stated prerequisite is true, or "when" the stated prerequisite is true, or "in response to determination" the stated prerequisite is true, or "according to determination" the stated prerequisite is true, or "in response to detection" the stated prerequisite is true. Similarly, the phrases "if it is determined [the stated prerequisite is true]" or "if [the stated prerequisite is true]" or "when [the stated prerequisite is true]" may be interpreted, depending on the context, as meaning "when" the stated prerequisite is true, or "in response to determination" the stated prerequisite is true, or "according to determination" the stated prerequisite is true, or "when" the stated prerequisite is true, or "in response to detection" the stated prerequisite is true.
[0194] For illustrative purposes, the foregoing description has been illustrated with reference to specific embodiments. However, the exemplary arguments presented above are not intended to be exhaustive, nor are they intended to limit the claims to the precise forms disclosed. Many modifications and variations are possible in light of the foregoing teachings. These embodiments were chosen and described to best explain the principles of operation and practical application, thereby enabling those skilled in the art to implement them.
Claims
1. A video decoding method executed at a computing system, the computing system having a memory and at least one processor, characterized in that, The method includes: Receives a video stream consisting of multiple frames; Select a primary reference frame for the current frame from a set of reference frames; When the number of tiles in the main reference frame is less than a threshold: Select a secondary reference frame from the set of reference frames; Obtain the corresponding contexts of the primary reference frame and the secondary reference frame; and The current frame context of the current frame is initialized by performing a weighted average on the corresponding context; When the number of tiles of the primary reference frame is greater than the threshold, the current frame context of the current frame is initialized without using the secondary reference frame; Using the current frame context, entropy decoding is performed on at least one syntax element of the current frame; and The current frame is reconstructed using information from at least one of the syntax elements.
2. The method according to claim 1, characterized in that, The threshold for the number of tiles is 2.
3. The method according to claim 1, characterized in that, Further includes: Parse the syntax element from HLS to determine whether to enable weighted averaging of multiple reference frames for the current frame, wherein when the syntax element indicates that weighted averaging of multiple reference frames is enabled for the current frame, the current frame context of the current frame is initialized by performing the weighted averaging on the corresponding context.
4. The method according to claim 3, characterized in that, The syntax element indicates whether the current frame context of the current frame is initialized using zero reference frames, one reference frame, or multiple reference frames.
5. The method according to claim 1, characterized in that, Initializing the current frame context of the current frame without using the secondary reference frame includes: initializing the current frame context by performing a weighted average of the default value and the corresponding context of the tiles of the primary reference frame.
6. The method according to claim 1, characterized in that, Initializing the current frame context of the current frame without using the secondary reference frame includes: initializing the current frame context by performing a weighted average of the corresponding contexts of multiple tiles from the primary reference frame.
7. The method according to claim 6, characterized in that, The weights of the weighted average of the corresponding contexts of the multiple tiles from the main reference frame are based on the encoding size of the multiple tiles.
8. The method according to claim 6, characterized in that, The weights of the weighted average of the corresponding contexts of the multiple tiles from the main reference frame are selected from a set of predefined weights.
9. The method according to claim 8, characterized in that, The weights of the weighted average of the corresponding contexts of the multiple tiles from the main reference frame are equal.
10. The method according to claim 1, characterized in that, At least one first tile of the current frame has parameters for entropy encoding using the current frame context, and at least one second tile of the current frame has parameters for encoding without using the current frame context.
11. A video encoding method executed at a computing system having a memory and at least one processor, characterized in that, The method includes: Receive video data including multiple frames; Select a primary reference frame for the current frame from a set of reference frames; When the number of tiles in the main reference frame is less than a threshold: Select a secondary reference frame from the set of reference frames; Obtain the corresponding contexts of the primary reference frame and the secondary reference frame; and The current frame context of the current frame is initialized by performing a weighted average on the corresponding context; When the number of tiles in the primary reference frame exceeds the threshold, the current frame context of the current frame is initialized without using the secondary reference frame; and Using the current frame context, at least one parameter of the current frame is entropy encoded.
12. The method according to claim 11, characterized in that, The threshold for the number of tiles is 2.
13. The method according to claim 11, characterized in that, It further includes using signaling to indicate syntax elements in the HLS to indicate whether a weighted average of multiple reference frames is enabled for the current frame.
14. The method according to claim 13, characterized in that, The syntax element indicates whether the current frame context of the current frame is initialized using zero reference frames, one reference frame, or multiple reference frames.
15. The method according to claim 11, characterized in that, Initializing the current frame context of the current frame without using the secondary reference frame includes: initializing the current frame context by performing a weighted average of the default value and the corresponding context of the tiles of the primary reference frame.
16. The method according to claim 11, characterized in that, Initializing the current frame context of the current frame without using the secondary reference frame includes: initializing the current frame context by performing a weighted average of the corresponding contexts of multiple tiles from the primary reference frame.
17. The method according to claim 16, characterized in that, The weights of the weighted average of the corresponding contexts of the multiple tiles from the main reference frame are equal.
18. A non-transitory computer-readable storage medium for storing a video stream, the video stream being generated by a video coding method, characterized in that, The video encoding method includes: Receive video data including multiple frames; Select a primary reference frame for the current frame from a set of reference frames; When the number of tiles in the main reference frame is less than a threshold: Select a secondary reference frame from the set of reference frames; Obtain the corresponding contexts of the primary reference frame and the secondary reference frame; and The current frame context of the current frame is initialized by performing a weighted average on the corresponding context; When the number of tiles in the primary reference frame exceeds the threshold, the current frame context of the current frame is initialized without using the secondary reference frame; and Using the current frame context, at least one parameter of the current frame is entropy encoded.
19. The computer-readable storage medium according to claim 18, characterized in that, The video stream includes at least one entropy coding parameter and coding information of the current frame.
20. The computer-readable storage medium according to claim 18, characterized in that, The video stream includes syntax elements in HLS that indicate whether a weighted average of multiple reference frames is enabled for the current frame.
