Block-based weighting factors for joint motion vector difference coding modes
Through the combined motion vector differential coding mode and composite weighted prediction mode, the scaling factor and weighted factor list are used to solve the problem of low motion vector coding efficiency in the prior art, and efficient video encoding and decoding is achieved, suitable for multi-reference frame scenarios .
Patent Information
- Application Number
- CN202380014465.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-06-28
- Filing Date
- 2023-06-29
- Publication Date
- 2024-11-15
AI Technical Summary
When handling motion vector differences, existing video encoding technologies are difficult to effectively utilize block-level weighting factors, resulting in low encoding efficiency. Especially in the case of multiple reference frames, it is difficult to optimize the encoding and decoding of motion vectors.
The joint motion vector difference (JMVD) encoding mode and the composite weighted prediction mode are used to determine the motion information by scaling the list of factors and weighting factors, and efficient encoding and decoding of the motion vector is achieved, which is suitable for video encoding scenarios of multiple reference frames.
Improves the efficiency and quality of video encoding, especially in the case of multiple reference frames, reduces the amount of data and improves the encoding accuracy, and reduces signaling costs.
Smart Images

Figure CN118975228A_ABST
Abstract
Description
Incorporated by Reference
[0001] This application claims the benefit of priority to U.S. Patent Application No. 18 / 215,704, filed on June 28, 2023, "BLOCK BASED WEIGHTING FACTOR FOR JOINT MOTION VECTOR DIFFERENCE CODING MODE," which claims the benefit of priority to U.S. Provisional Application No. 63 / 451,807, filed on March 13, 2023, "BlockBased Weighting Factor for Joint Motion Vector Difference Coding Mode." The disclosure of the prior application is incorporated herein by reference in its entirety. Technical Field
[0002] This disclosure generally describes implementations related to video encoding. Background Art
[0003] The background description provided herein is for the purpose of generally presenting the context of the present disclosure. To the extent the work is described in this background section, the work of the presently named inventors and aspects of the description that may not otherwise be qualified as prior art at the time of filing are neither explicitly nor implicitly admitted to be prior art against the present disclosure.
[0004] Image / video compression can help to transmit image / video files across different devices, storages, and networks with minimal quality degradation. In some examples, video codec technology can compress video based on spatial redundancy and temporal redundancy. In an example, a video codec can use a technique called intra-frame prediction, which can compress images based on spatial redundancy. For example, intra-frame prediction can use reference data from the current picture being reconstructed for sample prediction. In another example, a video codec can use a technique called inter-frame prediction, which can compress images based on temporal redundancy. For example, inter-frame prediction can use motion compensation to predict samples in the current picture based on previously reconstructed pictures. Motion compensation is usually indicated by a motion vector (MV). Summary of the invention
[0005] Aspects of the present disclosure provide methods and apparatus for video and / or picture encoding / decoding. The apparatus includes a processing circuit system configured to receive a bitstream including a frame. Coding information for a block in the frame indicates that the block is encoded using a joint motion vector difference (JMVD) coding mode and a composite weighted prediction mode. The coding information also includes scaling factor information for the JMVD coding mode. In response to the scaling factor information indicating that each of the scaling factors of the components of at least one MVD associated with at least one corresponding reference frame of the block is a predefined scaling factor, the processing circuit system determines the weighting factor of the composite weighted prediction mode based on a list of weighting factors. In response to the scaling factor information indicating that at least one of the scaling factors is different from the predefined scaling factor, the processing circuit system determines the weighting factor of the composite weighted prediction mode based on a subset of the list of weighting factors. The processing circuit system can also use the JMVD coding mode to determine motion information associated with the corresponding reference frame of the block based on the scaling factor of the components of at least one MVD associated with at least one corresponding reference frame of the block. The reference frame of the block includes at least one corresponding reference frame. The processing circuitry reconstructs the block using a composite weighted prediction mode based on motion information associated with a corresponding reference frame of the block and the determined weighting factors. In an example, the list is signaled in the bitstream. In an example, only a subset of the list is signaled in the bitstream.
[0006] In an embodiment, the processing circuit system is configured to receive a bitstream including a frame. The encoding information of a block in the frame indicates that the block is encoded using a joint MVD encoding mode and a composite weighted prediction mode. The encoding information also indicates weighting factor information of the composite weighted prediction mode. In response to the weighting factor information indicating that an equal weighting factor is applied to each of the reference frames of the block, the processing circuit system determines a scaling factor of a component of at least one MVD associated with at least one corresponding reference frame in the reference frames based on a list of scaling factors. In response to the weighting factor information indicating that non-equal weighting factors are applied to the reference frames of the block, the processing circuit system determines a scaling factor of a component of at least one MVD based on a subset of the list of scaling factors. The processing circuit system also uses the joint MVD encoding mode to determine motion information associated with the corresponding reference frame of the block based on the determined scaling factor, and uses the composite weighted prediction mode to reconstruct the block based on the motion information and weighting factor information associated with the corresponding reference frame of the block.
[0007] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions, which, when executed by a computer for video encoding / decoding, cause the computer to perform a method for video encoding / decoding. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] Additional features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:
[0009] Figure 1 is a schematic illustration of an exemplary block diagram of a communication system (100).
[0010] Figure 2 is a schematic illustration of an exemplary block diagram of a decoder.
[0011] Figure 3 is a schematic illustration of an exemplary block diagram of an encoder.
[0012] Figure 4 The relationship between the category of the non-zero component of the motion vector difference (MVD) and the amplitude range of the non-zero component of the MVD according to an embodiment of the present disclosure is shown.
[0013] Figure 5 The relationship between the category of the non-zero component of the MVD and the magnitude of the non-zero component of the MVD according to an embodiment of the present disclosure is shown.
[0014] Figure 6 Supported motion vector (MV) precisions in two precision sets are shown according to an embodiment of the present disclosure.
[0015] Figure 7 Exemplary scaling factors for the JOINT_AMVDNEWMV mode are shown in accordance with an embodiment of the present disclosure.
[0016] Figure 8 Exemplary scaling factors for the JOINT_NEWMV mode are shown in accordance with an embodiment of the present disclosure.
[0017] Fig. 9 An exemplary lookup table indicating the relationship between scaling factor pairs and corresponding weighting factors according to an embodiment of the present disclosure is shown.
[0018] Fig.10 A flowchart outlining a decoding process according to an embodiment of the present disclosure is shown.
[0019] Fig.11 A flowchart outlining an encoding process according to an embodiment of the present disclosure is shown.
[0020] Fig.12 A flowchart outlining a decoding process according to an embodiment of the present disclosure is shown.
[0021] Fig.13 A flowchart outlining an encoding process according to an embodiment of the present disclosure is shown.
[0022] Fig.14 is a schematic illustration of a computer system according to an embodiment. DETAILED DESCRIPTION
[0023] Figure 1 A block diagram of a video processing system (100) in some examples is shown. The video processing system (100) is an example of an application of the disclosed subject matter, a video encoder and a video decoder in a streaming environment. The disclosed subject matter can be equally applied to other video-enabled applications including, for example, video conferencing, digital TV, streaming services, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.
[0024] The video processing system (100) includes a capture subsystem (113) that may include a video source (101), such as a digital camera, that creates, for example, an uncompressed video picture stream (102). In an example, the video picture stream (102) includes samples captured by the digital camera. The video picture stream (102) is depicted as a thick line to emphasize the high amount of data when compared to the encoded video data (104) (or encoded video bitstream), which may be processed by an electronic device (120) coupled to the video source (101) including a video encoder (103). The video encoder (103) may include hardware, software, or a combination thereof to implement or embody aspects of the disclosed subject matter as described in more detail below. The encoded video data (104) (or encoded video bitstream) is depicted as a thin line to emphasize the lower amount of data when compared to the video picture stream (102), which can be stored on the streaming server (105) for future use. One or more streaming client subsystems such as Figure 1The client subsystems (106) and (108) in the streaming server (105) can access the streaming server (105) to retrieve copies (107) and (109) of the encoded video data (104). The client subsystem (106) can include, for example, a video decoder (110) in an electronic device (130). The video decoder (110) decodes the incoming copy (107) of the encoded video data and creates an output video picture stream (111) that can be presented on a display (112) (e.g., a display screen) or another presentation device (not depicted). In some streaming systems, the encoded video data (104), (107) and (109) (e.g., a video bitstream) can be encoded according to certain video encoding / compression standards. Examples of these standards include ITU-T Recommendation H.265. In the example, the video coding standard under development is informally referred to as Versatile Video Coding (VVC). The disclosed subject matter can be used in the context of VVC.
[0025] Note that the electronic devices (120) and (130) may include other components (not shown). For example, the electronic device (120) may include a video decoder (not shown), and the electronic device (130) may also include a video encoder (not shown).
[0026] Figure 2 An exemplary block diagram of a video decoder (210) is shown. The video decoder (210) may be included in an electronic device (230). The electronic device (230) may include a receiver (231) (e.g., receiving circuitry). The video decoder (210) may be used to replace Figure 1 A video decoder (110) is shown in an example.
[0027] The receiver (231) may receive one or more encoded video sequences to be decoded by the video decoder (210). In an embodiment, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of the decoding of other encoded video sequences. The encoded video sequence may be received from a channel (201), which may be a hardware / software link to a storage device storing the encoded video data. The receiver (231) may receive the encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams, which may be forwarded to their respective consuming entities (not depicted). The receiver (231) may separate the encoded video sequence from the other data. To prevent network jitter, a buffer memory (215) may be coupled between the receiver (231) and an entropy decoder / parser (220) (hereinafter referred to as "parser (220)"). In some applications, the buffer memory (215) is part of the video decoder (210). In other applications, the buffer memory (215) may be external to the video decoder (210) (not depicted). In still other applications, a buffer memory (not depicted) may be present outside the video decoder (210) to, for example, prevent network jitter, and another buffer memory (215) may be present inside the video decoder (210) to, for example, handle playout timing. When the receiver (231) receives data from a store / forward device with sufficient bandwidth and controllability or from an isochronous network, the buffer memory (215) may not be required, or the buffer memory (215) may be smaller. In order to make the most of a packet network such as the Internet, a buffer memory (215) may be required, which may be relatively large and may advantageously have an adaptive size, and may be implemented at least in part in an operating system or similar element (not depicted) external to the video decoder (210).
[0028] The video decoder (210) may include a parser (220) to reconstruct symbols (221) from the encoded video sequence. The categories of these symbols include information for managing the operation of the video decoder (210) and potential information for controlling a presentation device such as a presentation device (212) (e.g., a display screen) that is not part of the electronic device (230) but can be coupled to the electronic device (230), such as Figure 2As shown. The control information for the rendering device may be in the form of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not depicted). The parser (220) may parse / entropy decode the received coded video sequence. The encoding of the coded video sequence may be performed according to a video coding technique or standard, and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (220) may extract a subgroup parameter set of at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a Group of Pictures (GOP), a picture, a tile, a slice, a macroblock, a Coding Unit (CU), a block, a Transform Unit (TU), a Prediction Unit (PU), etc. The parser (220) may also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the encoded video sequence.
[0029] The parser (220) may perform entropy decoding / parsing operations on a video sequence received from the buffer memory (215) to create symbols (221).
[0030] Depending on the type of the coded video picture or part of the coded video picture (e.g., inter- and intra-pictures, inter- and intra-blocks) and other factors, the reconstruction of the symbol (221) may involve multiple different units. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the coded video sequence by the parser (220). For the sake of brevity, such subgroup control information flow between the parser (220) and the following multiple units is not depicted.
[0031] In addition to the functional blocks already mentioned, the video decoder (210) can be conceptually subdivided into a number of functional units as described below. In a practical implementation operating under commercial constraints, many of these units interact closely with each other and may be at least partially integrated with each other. However, for the purposes of describing the disclosed subject matter, the conceptual subdivision into the following functional units is appropriate.
[0032] The first unit is a sealer / inverse transform unit (251). The sealer / inverse transform unit (251) receives quantized transform coefficients as symbols (221) from the parser (220) and control information including which transform to use, block size, quantization factor, quantization scaling matrix, etc. The sealer / inverse transform unit (251) can output a block including sample values, which can be input into an aggregator (255).
[0033] In some cases, the output samples of the scaler / inverse transform unit (251) may belong to an intra-coded block. An intra-coded block is a block that does not use predictive information from a previously reconstructed picture, but may use predictive information from a previously reconstructed portion of a current picture. Such predictive information may be provided by an intra-picture prediction unit (252). In some cases, the intra-picture prediction unit (252) uses surrounding already reconstructed information obtained from a current picture buffer (258) to generate a block of the same size and shape as the block being reconstructed. For example, the current picture buffer (258) buffers a partially reconstructed current picture and / or a fully reconstructed current picture. In some cases, the aggregator (255) adds the prediction information already generated by the intra-prediction unit (252) to the output sample information as provided by the scaler / inverse transform unit (251) on a per-sample basis.
[0034] In other cases, the output samples of the scaler / inverse transform unit (251) may belong to an inter-frame coded block and potentially a motion compensated block. In this case, the motion compensated prediction unit (253) may access the reference picture memory (257) to obtain samples for prediction. After motion compensation of the obtained samples according to the symbols (221) belonging to the block, these samples may be added to the output of the scaler / inverse transform unit (251) (in this case referred to as residual samples or residual signals) by the aggregator (255) to generate output sample information. The address within the reference picture memory (257) from which the motion compensated prediction unit (253) obtains the predicted samples may be controlled by a motion vector, which may be obtained by the motion compensated prediction unit (253) in the form of a symbol (221), which may have, for example, an X component, a Y component and a reference picture component. Motion compensation may also include interpolation of sample values obtained from the reference picture memory (257) when using sub-sample accurate motion vectors, motion vector prediction mechanisms, etc.
[0035] The output samples of the aggregator (255) may be subjected to various loop filtering techniques in a loop filter unit (256). The video compression techniques may include in-loop filter techniques controlled by parameters included in the coded video sequence (also referred to as the coded video bitstream) and available to the loop filter unit (256) as symbols (221) from the parser (220). The video compression may also be responsive to meta-information obtained during decoding of a coded picture or a previous (in decoding order) portion of the coded video sequence, and to previously reconstructed and loop filtered sample values.
[0036] The output of the loop filter unit (256) may be a sample stream that may be output to a rendering device (212) and stored in a reference picture memory (257) for use in future inter-picture prediction.
[0037] Once fully reconstructed, certain coded pictures may be used as reference pictures for future prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture has been identified as a reference picture (e.g., by the parser (220)), the current picture buffer (258) may become part of the reference picture memory (257) and a new current picture buffer may be reallocated before starting to reconstruct a subsequent coded picture.
[0038] The video decoder (210) may perform decoding operations according to a standard or predetermined video compression technology such as ITU-T H.265 Recommendation. The encoded video sequence may conform to the syntax specified by the video compression technology or standard used in the sense that the encoded video sequence follows both the syntax of the video compression technology or standard and the profile recorded in the video compression technology or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technology or standard as tools available only under the profile. For compliance, the complexity of the encoded video sequence is also required to be within a range as defined by the hierarchy of the video compression technology or standard. In some cases, the hierarchy limits the maximum picture size, the maximum frame rate, the maximum reconstruction sampling rate (measured in, for example, mega samples per second), the maximum reference picture size, etc. In some cases, the limits set by the hierarchy may be further defined by the Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the encoded video sequence.
[0039] In an embodiment, the receiver (231) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The video decoder (210) may use the additional data to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.
[0040] Figure 3 An exemplary block diagram of a video encoder (303) is shown. The video encoder (303) is included in an electronic device (320). The electronic device (320) includes a transmitter (340) (e.g., a transmission circuit system). The video encoder (303) can be used to replace Figure 1 A video encoder (103) in an example.
[0041] The video encoder (303) can be obtained from a video source (301) (not Figure 3 In another example, the video source (301) is a part of the electronic device (320).
[0042] The video source (301) may provide a source video sequence in the form of a digital video sample stream to be encoded by a video encoder (303), the digital video sample stream may have any suitable bit depth (e.g., 8 bits, 10 bits, 12 bits, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., YCrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the video source (301) may be a storage device storing previously prepared videos. In a video conferencing system, the video source (301) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of separate pictures that are given motion when viewed sequentially. The pictures themselves may be organized as a spatial pixel array, wherein each pixel may include one or more samples, depending on the sampling structure, color space, etc. used. The relationship between pixels and samples may be easily understood by those skilled in the art. The following description focuses on samples.
[0043] According to an embodiment, the video encoder (303) can encode and compress the pictures of the source video sequence into an encoded video sequence (343) in real time or under any other time constraints required. Implementing the appropriate encoding speed is a function of the controller (350). In some embodiments, the controller (350) controls other functional units as described below and is functionally coupled to the other functional units. For the sake of brevity, the coupling is not depicted. The parameters set by the controller (350) may include rate control related parameters (picture skipping, quantizer, lambda value of rate distortion optimization technology, ...), picture size, picture group (Group of Picture, GOP) layout, maximum motion vector search range, etc. The controller (350) can be configured to have other suitable functions belonging to the video encoder (303) optimized for a specific system design.
[0044] In some embodiments, the video encoder (303) is configured to operate in a decoding loop. As a simplified description, in an example, the decoding loop may include a source encoder (330) (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be encoded and a reference picture) and a (local) decoder (333) embedded in the video encoder (303). The decoder (333) reconstructs the symbols and creates sample data in a manner similar to that which the (remote) decoder will also create. The reconstructed sample stream (sample data) is input to a reference picture memory (334). Since the decoding of the symbol stream produces a bit-accurate result that is independent of the decoder location (local or remote), the contents of the reference picture memory (334) are also bit-accurate between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction part of the encoder are exactly the same as the sample values that the decoder will "see" when using prediction during decoding. The basic principles of reference picture synchronization (and drift caused when synchronization cannot be maintained, such as due to channel errors) are also used in some related technologies.
[0045] The operation of the "local" decoder (333) can be combined with the "remote" decoder such as has been described above. Figure 2 The operation of the video decoder (210) described in detail is the same. However, reference is also briefly made to Figure 2 , since the symbols are available and the encoding of the symbols into a coded video sequence by the entropy encoder (345) and the decoding of the symbols by the parser (220) can be lossless, the entropy decoding portion of the video decoder (210) including the buffer memory (215) and the parser (220) may not be fully implemented in the local decoder (333).
[0046] In an embodiment, decoder techniques other than parsing / entropy decoding present in a decoder are present in a corresponding encoder in the same or substantially the same functional form. Therefore, the disclosed subject matter focuses on decoder operation. The description of encoder techniques can be simplified because encoder techniques are contrary to the decoder techniques described comprehensively. A more detailed description is provided below in some parts.
[0047] In some examples, during operation, the source encoder (330) may perform motion compensated predictive coding that predictively encodes an input picture with reference to one or more previously encoded pictures from a video sequence designated as "reference pictures." In this manner, the encoding engine (332) encodes the differences between pixel blocks of the input picture and pixel blocks of a reference picture that may be selected as a prediction reference for the input picture.
[0048] The local video decoder (333) can decode the encoded video data of the picture that can be designated as the reference picture based on the symbol created by the source encoder (330). The operation of the encoding engine (332) can advantageously be a lossy process. When the encoded video data can be decoded at the video decoder ( Figure 3 When the video encoder (303) is decoded at a remote location (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (333) replicates the decoding process that may be performed by the video decoder on the reference picture and may cause the reconstructed reference picture to be stored in the reference picture memory (334). In this way, the video encoder (303) may locally store a copy of the reconstructed reference picture that has common content (absent transmission errors) with the reconstructed reference picture to be obtained by the remote video decoder.
[0049] The predictor (335) may perform a prediction search for the encoding engine (332). That is, for a new picture to be encoded, the predictor (335) may search the reference picture memory (334) for sample data (as candidate reference pixel blocks) or specific metadata such as reference picture motion vectors, block shapes, etc. that may be used as appropriate prediction references for the new picture. The predictor (335) may operate on a pixel block by pixel block basis to find an appropriate prediction reference. In some cases, as determined by the search results obtained by the predictor (335), the input picture may have prediction references taken from multiple reference pictures stored in the reference picture memory (334).
[0050] The controller (350) can manage encoding operations of the source encoder (330), including, for example, setting parameters and subgroup parameters for encoding video data.
[0051] The outputs of all the above mentioned functional units may be subjected to entropy coding in an entropy encoder (345). The entropy encoder (345) converts the symbols generated by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.
[0052] The transmitter (340) can buffer the encoded video sequence created by the entropy encoder (345) in preparation for transmission via a communication channel (360), which can be a hardware / software link to a storage device where the encoded video data will be stored. The transmitter (340) can combine the encoded video data from the video encoder (303) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).
[0053] The controller (350) may manage the operation of the video encoder (303). During encoding, the controller (350) may assign a certain encoding picture type to each encoded picture, which may affect the encoding techniques that may be applied to the corresponding picture. For example, a picture may generally be assigned one of the following picture types:
[0054] An intra picture (I picture) may be a picture that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of those variations of I pictures and their corresponding applications and features.
[0055] A predictive picture (P picture) may be a picture that can be encoded and decoded using intra prediction or inter prediction to predict sample values of each block using at most one motion vector and a reference index.
[0056] Bi-directional predictive pictures (B pictures), which can be pictures that can be encoded and decoded using intra-prediction or inter-prediction that uses up to two motion vectors and reference indices to predict sample values for each block. Similarly, multi-predictive pictures can use more than two reference pictures and associated metadata for reconstruction of a single block.
[0057] The source picture may typically be spatially subdivided into a number of blocks of samples (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples, respectively), and encoded on a block-by-block basis. The blocks may be predictively encoded with reference to other (already encoded) blocks, which are determined by the coding allocation applied to the corresponding picture of the block. For example, blocks of an I picture may be non-predictively encoded, or may be predictively encoded (spatial prediction or intra-prediction) with reference to already encoded blocks of the same picture. Blocks of pixels of a P picture may be predictively encoded via spatial prediction or via temporal prediction with reference to one previously encoded reference picture. Blocks of a B picture may be predictively encoded via spatial prediction or via temporal prediction with reference to one or two previously encoded reference pictures.
[0058] The video encoder (303) may perform encoding operations according to a standard or predetermined video encoding technique such as ITU-T H.265 Recommendation. In the operation of the video encoder (303), the video encoder (303) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in an input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard used.
[0059] In an embodiment, the transmitter (340) may transmit additional data along with the encoded video. The source encoder (330) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.
[0060] Video can be captured as multiple source pictures (video pictures) in a temporal sequence. Intra-picture prediction (often referred to simply as intra-picture prediction) exploits spatial correlation in a given picture, while inter-picture prediction exploits (temporal or other) correlations between pictures. In an example, a particular picture in encoding / decoding, which is referred to as the current picture, is divided into blocks. In the case where a block in the current picture is similar to a reference block in a reference picture that has been previously encoded in the video and is still buffered, the block in the current picture can be encoded by a vector called a motion vector. The motion vector points to a reference block in a reference picture, and in the case of using multiple reference pictures, the motion vector may have a third dimension that identifies the reference picture.
[0061] In some embodiments, a bidirectional prediction technique may be used in inter-picture prediction. According to the bidirectional prediction technique, two reference pictures are used, for example, a first reference picture and a second reference picture that are both before the current picture in the video in decoding order (but may be in the past and in the future in display order, respectively). A block in the current picture may be encoded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. A block may be predicted by a combination of the first reference block and the second reference block.
[0062] In addition, merge mode techniques can be used in inter-picture prediction to improve coding efficiency.
[0063] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, according to the HEVC standard, a picture in a video picture sequence is divided into coding tree units (CodingTree Unit, CTU) for compression, and the CTUs in the picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Typically, a CTU includes three coding tree blocks (Coding Tree Block, CTB), each of which is a luminance CTB and two chrominance CTBs. Each CTU can be recursively divided into one or more coding units (Coding Unit, CU) in a quadtree. For example, a 64×64 pixel CTU can be divided into a 64×64 pixel CU, or 4 32×32 pixel CUs, or 16 16×16 pixel CUs. In the example, each CU is analyzed to determine the prediction type for the CU, such as an inter-frame prediction type or an intra-frame prediction type. The CU is divided into one or more prediction units (Prediction Unit, PU) according to temporal and / or spatial predictability. Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In an embodiment, the prediction operation in decoding (encoding / decoding) is performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values (e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.
[0064] Note that the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using any suitable technology. In an embodiment, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more processors executing software instructions.
[0065] In an embodiment, inter-mode coding such as used in Codec Working Group (CWG)-B018 is described below. In an example, such as in AOMedia Video 1 (AV1), for each coding block in an inter frame (or inter picture), if the mode of the current block is not skip mode and is an inter coding mode, a flag indicating whether a single reference mode or a composite reference mode is used to encode the current block is signaled. A prediction block can be generated by a motion vector (MV) of a block (e.g., a current block) in a single reference mode. A prediction block can be generated by averaging (e.g., weighted averaging) multiple prediction blocks (or multiple reference blocks), such as two prediction blocks derived from two MVs in a composite reference mode.
[0066] For a single reference case using a single reference mode, the following modes may be signaled: including but not limited to NEARMV mode, NEWMV mode, GLOBALMV mode, etc.
[0067] In NEARMV mode, a motion vector predictor (MVP) of an MVP in a list may be used. The MVP may be indicated by an index such as a dynamic reference list (DRL) index. In an example, the MVP is used as the MV of a block.
[0068] In NEWMV mode, an MVP in the list of MVPs may be used as a reference. The MVP may be signaled by an index such as a DRL index. An increment (e.g., an increment MV) may be applied to a reference (e.g., an MVP indicated by a DRL index). In an example, the MV of a block is determined based on the increment and the MVP. For example, the increment may indicate an MV difference (MVD) between the MVP and the MV of the block.
[0069] In GLOBALMV mode, MVs based on frame-level global motion parameters may be used, for example, to predict blocks.
[0070] For the composite reference mode, the following modes may be signaled, including but not limited to NEAR_NEARMV mode, NEAR_NEWMV mode, NEW_NEARMV mode, NEW_NEWMV mode, GLOBAL_GLOBALMV mode, etc.
[0071] In NEAR_NEARMV mode, an MVP in the MVP in the list may be used. The MVP may be signaled by an index such as a DRL index. In an example, an MVP in the MVP in the list may be used to predict a block. For example, the MVP may indicate a first MV and a second MV of a block, and the first MV and the second MV of the block may be used to generate a first reference block and a second reference block, respectively, to predict the block.
[0072] In NEAR_NEWMV mode, an MVP in the list of MVPs may be used as a reference. The MVP may be signaled by an index such as a DRL index. An increment (e.g., an increment MV) may be applied to a reference (e.g., an MVP indicated by a DRL index) to determine a second MV of the block. In an example, the reference is used directly as the first MV of the block, for example.
[0073] In NEW_NEARMV mode, an MVP in the list of MVPs may be used as a reference. The MVP may be signaled by an index such as a DRL index. An increment (e.g., an increment MV) may be applied to a reference (e.g., an MVP indicated by a DRL index) to determine a first MV of a block. In an example, the reference is used directly as a second MV of a block, for example.
[0074] In NEW_NEWMV mode, an MVP in the list of MVPs may be used as a reference. The MVP may be signaled by an index such as a DRL index. Two increments (e.g., two increment MVs) may be applied to a reference (e.g., an MVP indicated by a DRL index) to determine a first MV of a block and a second MV of a block. In an example, two increments are signaled.
[0075] In GLOBAL_GLOBALMV mode, the MV from each reference based on the corresponding frame-level global motion parameters can be used, for example, to predict a block.
[0076] In the present disclosure, the term first component and second component of a 2D vector (e.g., MV, MVD, delta MV, etc.) may refer to the values of the 2D vector along the first axis and the second axis, respectively. The first axis and the second axis may be two predefined directions perpendicular to each other. In an example, the first axis and the second axis are the x-axis and the y-axis, and the first component and the second component are the horizontal component and the vertical component of the 2D vector. In an example, the first axis and the second axis are the 45-degree axis and the 135-degree axis, and the first component and the second component are the components of the 2D vector along the 45-degree axis and the 135-degree axis, respectively.
[0077] The embodiments in the present disclosure may be applied to the first component and the second component of a 2D vector (e.g., MV, MVD, delta MV, etc.). In an example, the x-axis and the y-axis may be replaced by two other axes along two predefined directions perpendicular to each other, and the embodiments in the disclosure regarding the horizontal component and the vertical component of a 2D parameter may also be applied to the two other axes. For example, the x-axis and the y-axis may be replaced by a 45 degree axis and a 135 degree axis.
[0078] Various methods may be used to encode the MVD. An exemplary method such as that used in AV1 is described below. In an embodiment, for example in AV1, 1 / 8 pixel motion vector precision (or accuracy) may be used, and the following syntax (e.g., including but not limited to one or more of mv_joint, mv_sign, mv_class, mv_bit, mv_fr, mv_hp, etc.) may be used to signal or indicate the MVD in reference frame list 0 or reference frame list 1.
[0079] The syntax mv_joint can specify which components of the MVD (e.g., the first component and the second component) are non-zero. For example, the syntax mv_joint is a first value (e.g., 0) indicating that there is no non-zero MVD along the first axis (e.g., horizontal direction) or the second axis (e.g., vertical direction); the syntax mv_joint is a second value (e.g., 1) indicating that there is only a non-zero MVD along the first axis (e.g., horizontal direction); the syntax mv_joint is a third value (e.g., 2) indicating that there is only a non-zero MVD along the second axis (e.g., vertical direction); and the syntax mv_joint is a fourth value (e.g., 3) indicating that there is a non-zero MVD along both the first axis and the second axis (e.g., horizontal direction and vertical direction).
[0080] The syntax mv_sign may specify whether the MVD is positive or negative. For example, the syntax mv_sign may be associated with each non-zero component of the MVD and may indicate the sign of the corresponding non-zero component of the MVD.
[0081] In an embodiment, the syntax mv_class, mv_bit, mv_fr, and mv_hp are used to indicate the magnitude of the non-zero components of the MVD. The integer part of the magnitude of the non-zero components of the MVD can be indicated by the syntax mv_class and mv_bit. The fractional part of the magnitude of the non-zero components of the MVD can be indicated by the syntax
[0082] The syntax mv_class (MV class) may specify the class (eg, amplitude class) of the MVD. For example, the syntax mv_class may be associated with each non-zero component of the MVD and may indicate the class of the corresponding non-zero component of the MVD, such as Figure 4 shown.
[0083] Figure 4 The relationship between the category of the non-zero component of the MVD (or MV category) and the amplitude range of the non-zero component of the MVD is shown. In an embodiment, a higher category indicates that the MVD (eg, the non-zero component of the MVD) has a larger amplitude. Figure 4 The first column in shows various MV classes, for example, from MV class 0 (MV_CLASS_0) to MV class 10 (MV_CLASS_10). MV class 10 is higher than MV class 0 to MV class 9. Figure 4 The second column in shows the corresponding amplitude range indicated by (m1, m2]. The value m1 is called the starting amplitude of the MV class corresponding to the amplitude range (m1, m2]. Figure 4 In the example shown, the starting amplitudes of the MV classes (e.g., MV class 0 (MV_CLASS_0) to MV class 10 (MV_CLASS_10)) are 0, 2, 4, 8, 16, 32, 64, 128, 256, 512, and 1024, respectively. For the amplitude range (m1, m2], the amplitude of the non-zero component of the MVD is greater than m1 and less than or equal to m2. For example, MV_CLASS_0 corresponds to an amplitude range (0, 2] greater than 0 and less than or equal to 2 (e.g., m1=0, m2=2). The value m1 (e.g., 0) is the starting amplitude of MV_CLASS_0. If the syntax mv_class of the non-zero component of the MVD is MV_CLASS_0, the magnitude of the non-zero component of the MVD is greater than 0 and less than or equal to 2.
[0084] The syntax mv_bit may specify the integer part of the offset between the MVD and the starting amplitude of each MV category. For example, the syntax mv_bit may be associated with each non-zero component of the MVD and may indicate the integer part of the difference (or offset) between the corresponding non-zero component of the MVD and the starting amplitude of the corresponding MV category. For example, the non-zero component of the MVD has an MV category of MV_CLASS_10, and the syntax mv_bit associated with the non-zero component of the MVD indicates the integer part of the difference between the non-zero component of the MVD and 1024.
[0085] The syntax mv_fr may specify the first two decimal places of the MVD. For example, the syntax mv_fr may be associated with each non-zero component of the MVD and may indicate the first two decimal places of the corresponding non-zero component of the MVD.
[0086] The syntax mv_hp may specify the third decimal place of the MVD. For example, the syntax mv_hp may be associated with each non-zero component of the MVD and may indicate the third decimal place of the corresponding non-zero component of the MVD.
[0087] Adaptive MVD resolution (AMVD) may be used, for example, in CWG-B092. In an embodiment, for certain inter prediction modes such as NEW_NEARMV mode and NEAR_NEWMV mode, the precision of the MVD may depend on the associated class (e.g., MV class) and / or the magnitude of the MVD. For example, the precision of the non-zero components of the MVD may depend on the associated class (e.g., MV class) and / or the magnitude of the non-zero components of the MVD. In an example, fractional MVD is allowed only when the magnitude of the MVD is equal to or less than one pixel. For example, fractional parts of non-zero components of the MVD are allowed only when the magnitude of the non-zero components of the MVD is equal to or less than one pixel.
[0088] In the example of AMVD, when the value of the associated MV class is equal to or greater than MV_CLASS_1, only one MVD value is allowed. For MV class 1 (MV_CLASS_1), MV class 2 (MV_CLASS_2), MV class 3 (MV_CLASS_3), MV class 4 (MV_CLASS_4), or MV class 5 (MV_CLASS_5), the MVD value in each MV class can be derived as 4, 8, 16, 32, or 64. For example, for MV class 1 (MV_CLASS_1), MV class 2 (MV_CLASS_2), MV class 3 (MV_CLASS_3), MV class 4 (MV_CLASS_4), or MV class 5 (MV_CLASS_5), the non-zero component of the MVD in each MV class can be derived as 4, 8, 16, 32, or 64.
[0089] Figure 5 The adaptive MVD in each MV amplitude category according to an embodiment of the present disclosure is shown. For example, Figure 5 Exemplary allowed MVD values in each MV class are shown. For example, if the MV class is MV_CLASS_0, the amplitude of the non-zero component of the MVD can be between 0 and 1 (excluding 0), 1 or 2. If the MV class is one of the MV classes including MV_CLASS_1 to MV_CLASS_10, the amplitude of the non-zero component of the MVD can be a single value, such as 4, 8, 16, 32, 64, 128, 256, 512, 1024 or 2048.
[0090] In an example, if the current block is encoded in NEW_NEARMV mode or NEAR_NEWMV mode, one context (e.g., for context adaptive binary arithmetic coding (CABAC)) is used to signal syntax such as syntax mv_joint or syntax mv_class. Otherwise, another context (e.g., for CABAC) may be used to signal syntax such as syntax mv_joint or syntax mv_class.
[0091] The joint MVD coding (JMVD) mode may be applied in some embodiments, such as in CWG-B092.
[0092] A new inter-coding mode referred to as JOINT_NEWMV mode may be applied to indicate whether the MVDs of multiple reference lists (e.g., two reference lists) are jointly signaled. If the inter-prediction mode is equal to JOINT_NEWMV mode, the MVDs of multiple reference lists (e.g., reference list 0 and reference list 1) may be jointly signaled. Therefore, in the example of JOINT_NEWMV mode, only one MVD (also referred to as joint MVD or joint_mvd) is signaled and sent to the decoder, and the delta MVs (e.g., MVDs) of the corresponding multiple reference lists (e.g., reference list 0 and reference list 1) are derived from the one MVD (e.g., joint_mvd). In the example, joint MVD information indicating the joint MVD (e.g., joint_mvd) is signaled, and the delta MVs for the multiple reference lists are derived from the joint MVD information.
[0093] In an embodiment, the JOINT_NEWMV mode is one of the composite reference modes. For example, the JOINT_NEWMV mode is signaled together with other composite reference modes such as NEAR_NEARMV mode, NEAR_NEWMV mode, NEW_NEARMV mode, NEW_NEWMV mode, GLOBAL_GLOBALMV mode, etc. In an embodiment, no additional context is added. In an example, the JOINT_NEWMV mode is signaled as one of the composite reference modes.
[0094] When JOINT_NEWMV mode is signaled and the picture order count (POC) distances between multiple reference frames (e.g., two reference frames) and the current frame are different, the MVD of reference frame list 0 or reference frame list 1 may be scaled based on the POC distance. In an embodiment, the POC distance between the first reference frame in reference frame list 0 and the current frame is represented by parameter td0, and the POC distance between the second reference frame in reference frame list 1 and the current frame is represented by parameter td1. If td0 is equal to or greater than td1, joint_mvd may be used directly for the first reference frame in reference list 0 as the first MVD (represented by MVP0). The second MVD of the second reference frame in reference frame list 1 (represented by MVP1 or derived_mvd1) may be derived from joint_mvd, for example, based on equation (1).
[0095] Otherwise, if td1 is equal to or greater than td0, joint_mvd can be used directly for the second reference frame in reference list 1 as the second MVD (e.g., MVP1). The first MVD of the first reference frame in reference frame list 0 (e.g., represented by MVP0 or derived_mvd0) can be derived from joint_mvd based on equation (2).
[0096] In an embodiment, such as in CWG-C011, the adaptive MVD resolution (AMVD) may be modified or improved.
[0097] A new inter-frame coding mode called AMVDMV mode may be added to the single reference mode. When AMVDMV mode is selected, it is indicated to apply AMVD to signal MVD.
[0098] In an embodiment, a flag called amvd_flag may be added to the JOINT_NEWMV mode to indicate whether AMVD is applied to the joint MVD coding mode. When adaptive MVD resolution (AMVD) is applied to the joint MVD coding mode, the joint MVD coding mode may be referred to as the joint AMVD coding mode. In the joint AMVD coding mode, the joint MVD of multiple reference frames (e.g., two reference frames) may be jointly signaled, and the accuracy of the joint MVD may be implicitly determined by the MVD amplitude. Otherwise, for example, when AMVD is not applied to the joint MVD coding mode, the joint MVD of multiple reference frames (e.g., two reference frames) may be jointly signaled, and an MVD encoding different from AMVD (e.g., reference frames) may be applied. Figure 4 describe).
[0099] Alternatively, a new inter prediction mode in composite reference mode, called JOINT_AMVDNEWMV mode, is added to indicate that AMVD applies to joint MVD coding mode. In an embodiment, in JOINT_AMVDNEWMV mode, the joint MVD of multiple reference frames (e.g., two reference frames) may be jointly signaled, and the accuracy of the joint MVD may be implicitly determined by the MVD amplitude.
[0100] In an embodiment, such as in CWG-C012 and CWG-C020, adaptive motion vector resolution (AMVR) may be applied.
[0101] In an example, such as in CWG-C012, AMVR is used to support a total of 7 MV precisions (e.g., 8 pixels or 8 pixels, 4 pixels, 2 pixels, 1 pixel, 1 / 2 pixel, 1 / 4 pixel, and 1 / 8 pixel). For each prediction block, an encoder (e.g., an AOM video model (AVM) encoder) can search for supported MV precisions, such as all supported precision values, and can signal the selected precision (e.g., the best precision) to the decoder.
[0102] To reduce encoder runtime, two precision sets can be supported, such as Figure 6 shown. Figure 6 The supported MV precisions in two precision sets are shown. The number of MV precisions in each of the two precision sets (e.g., 4) is less than the number of MV precisions in the supported precision values (e.g., 7). In the example, each precision set includes 4 predefined precisions. The precision set may be adaptively selected at the frame level based on the value of the maximum precision of the frame. In the example, such as used in AV1, the maximum precision may be signaled in the frame header. Figure 6The supported precision values based on the maximum precision of the frame level according to the implementation of the present disclosure are shown. If the maximum precision of the frame level is 1 / 8 pixel, the precision set includes 1 / 8 pixel, 1 / 2 pixel, 1 pixel and 4 pixels. If the maximum precision of the frame level is 1 / 4 pixel, the precision set includes 1 / 4 pixel, 1 pixel, 4 pixels and 8 pixels.
[0103] In an example, for example, in current AVM software similar to AV1, a flag such as a frame level flag (e.g., a cur_frame_force_integer_mv flag) is used to indicate whether the MV of a frame may include sub-pixel precision, such as 1 / 2 pixel, 1 / 4 pixel, and 1 / 8 pixel. In an example, AMVR is enabled only when the value of the frame level flag (e.g., the cur_frame_force_integer_mv flag) is 0. In an example, in AMVR, if the precision of a block is lower than the maximum precision, the motion model and the interpolation filter are not signaled. If the precision of a block is lower than the maximum precision, the motion mode may be inferred as translational motion, and the interpolation filter may be inferred as a regular interpolation filter. Similarly, in an example, if the precision of a block is 4 pixels or 8 pixels, the inter-frame intra-frame mode is not signaled and may be inferred to be 0.
[0104] In an embodiment, for example in CWG-C053, the joint MVD coding mode may be modified or improved.The joint MVD coding mode may refer to a JOINT_NEWMV mode or a JOINT_AMVDNEWMV mode.
[0105] If a block is encoded in a joint MVD coding mode such as JOINT_NEWMV mode or JOINT_AMVDNEWMV mode, a syntax such as mvd_scaling_factor_idx (e.g., scaling factor index) may be signaled into the bitstream to explicitly indicate, for example, the scaling factor between the MVDs associated with reference frame 0 and reference frame 1 (e.g., denoted by jmvd_scale).
[0106] like Figures 7 and 8 As shown, two lookup tables (eg, two predefined lookup tables) may be used to store supported or allowed scaling factors for the JOINT_AMVDNEWMV mode and the JOINT_NEWMV mode, respectively. The corresponding lookup tables (eg, Figure 7 or Figure 8 The relevant entry index of the selected scaling factor in the lookup table in .
[0107] Figure 7Exemplary scaling factors for JOINT_AMVDNEWMV mode are shown.For JOINT_AMVDNEWMV mode, the same scaling factor may be applied to the first and second components (eg, vertical and horizontal components) of the MVD of reference frame list 0 and / or reference frame list 1.
[0108] Figure 8 Exemplary scaling factors for the JOINT_NEWMV mode are shown. For the JOINT_NEWMV mode, the scaling factor of one component of the MVD (e.g., the vertical component or the horizontal component) can be limited to a first value (e.g., 1), and the scaling factor of another component of the MVD can be another value (e.g., 2, 1 / 2, etc.) different from the first value (e.g., 1).
[0109] The MVD of reference frame list 0 or reference frame list 1 can be calculated based on equations (3) to (4), respectively. mvd_ref0=joint_mvd (3)
[0110] In an example, in JOINT_AMVDNEWMV mode, for a first component and a second component (e.g., a vertical component and a horizontal component) of an MVD (e.g., mvd_ref1), a scaling factor (e.g., jmvd_scale) is the same, and equations (3) to (4) may be applied to the MVD or each component of the MVD. For example, a first MVD (e.g., MVD0) of reference frame list 0 is indicated by mvd_ref0 and is obtained directly from joint_mvd (e.g., being signaled), and a second MVD of reference frame list 1 is indicated by mvd_ref1 and is obtained by modifying joint_mvd by a ratio of td1 / td0, such as described in equation (1), and a scaling factor (e.g., jmvd_scale). Equations (3) to (4) may be modified to determine a first MVD based on a scaling factor and a ratio of td0 / td1, such as described in equation (2), and to determine a second MVD directly from joint_mvd.
[0111] In an example, in JOINT_NEWMV mode, the scaling factors (eg, jmvd_scale) of the first and second components (eg, vertical and horizontal components) of the MVD are different. Equations (3) to (4) may be applied to each component of the MVD (eg, MVD0 or MVD1).
[0112] In an embodiment, a bi-prediction (BCW) mode with CU-level weights (also referred to as composite weighted prediction) may be applied to a block. In an example, such as in HEVC, a bi-prediction signal is generated by averaging two prediction signals obtained from two different reference pictures and / or using two different MVs. In an example, such as in VVC, the bi-prediction mode may be extended beyond simple averaging to allow a weighted average of two prediction signals, such as shown in equation (5). P bi-pred =((8-w)×P0+w×P1+4)>>3 (5)
[0113] In the example, the bi-prediction signal P bi-pred is a dual prediction sample, and the two prediction signals P0 and P1 are two prediction samples.
[0114] Referring to equation (5), the bi-prediction signal P bi-pred It may be a weighted average of two prediction signals P0 and P1 having weights (8-w) and w, respectively. In the example, the sum of the weights (8-w) and w is a constant (eg, 8).
[0115] In an embodiment, a list of weighting factors (e.g., including five weights) is allowed in weighted average bi-prediction or in BCW mode. In an example, the list of weighting factors includes five weights -2, 3, 4, 5, and 10, such as w∈{-2, 3, 4, 5, 10}. When w is equal to 4, an equal weighting factor (e.g., 4) can be used in the weighted average of the two prediction samples. For example, the weighting factors of the two prediction signals P0 and P1 are equal (e.g., 4), and equation (5) becomes P bi-pred =(4×P0+4×P1+4)>>3.
[0116] For each bi-predicted CU, the weight w can be determined in one of two ways: 1) for non-merged CUs, the weight index can be signaled after MVD; 2) for merged CUs, the weight index can be inferred from neighboring blocks based on the merge candidate index.
[0117] In an example, the BCW mode is applied only to CUs with 256 or more luma samples (e.g., CU width × CU height ≥ 256). In an example, for low-latency pictures, a list of weighting factors is used (e.g., including all 5 weights). For non-low-latency pictures, a subset of the list of weighting factors is used. In an example, the subset of the list of weighting factors includes only 3 weights in the list of weighting factors (e.g., w∈{3, 4, 5}). In an example, a low-latency picture may refer to a picture in which the encoding order of the pictures is the same as the display order of the pictures, e.g., the pictures are encoded sequentially.
[0118] For example, equation (5) can be adjusted so that the bi-prediction signal P bi-pred It can be a weighted average of two prediction signals P0 and P1 with weights (16-w) and w, respectively, such as shown in equation (6). In the example, the sum of weights (16-w) and w is a constant (eg, 16). P bi-pred =((16-w)×P0+w×P1+8)>>4 (6)
[0119] In an embodiment, for the joint MVD coding mode (JMVD), a scaling factor (e.g., jmvd_scale) between the MVDs associated with reference frame list 0 and reference frame list 1 is used, such as described above in Figures 7 and 8 and described in equation (4). In the example, when the BCW mode and the JMVD mode are applied to the block, for example, when the BCW mode is applied to the JMVD mode, 5 weighting factors are used for each supported scaling factor, for example, w∈{-2, 3, 4, 5, 10}, and therefore a large number of bits are to be signaled, and the signaling cost is relatively high.
[0120] This disclosure describes a set of advanced image and video coding techniques, including embodiments related to block-based weighting factors for a joint MVD coding mode.
[0121] In the present disclosure, the direction of the reference frame may be determined based on whether the reference frame is before the current frame in the display order or after the current frame in the display order.
[0122] In the present disclosure, the JMVD mode may refer to a JMVD with a regular full MV resolution (eg, the highest resolution, such as 1 / 4 pixel or 1 / 8 pixel), or a JMVD with AMVR.
[0123] In some embodiments, when the joint MVD coding mode is applied to encode a block in a frame (or a current frame), the scaling factor used in the joint MVD coding mode may be determined based on a list L0s of allowed or supported scaling factors. When the BCM mode is applied to encode a block, the weighting factor used in the BCM mode may be determined based on a list L0w of allowed or supported weighting factors. According to an embodiment of the present disclosure, when both the joint MVD coding mode and the BCM mode are applied to encode a block, the scaling factor used in the joint MVD coding mode may be determined based on a subset of the list L0s of scaling factors and / or the weighting factor used in the BCM mode may be determined based on a subset of the list L0w of weighting factors, for example, to reduce signaling costs and improve coding efficiency.
[0124] According to an embodiment of the present disclosure, when a joint MVD coding mode is selected to encode a block in a frame, and a scaling factor of a component (e.g., a first component and a second component) of at least one MVD (e.g., mvd_ref1 in equation (4)) associated with at least one reference frame (e.g., a first reference frame) of the block is a given value (also referred to as a predetermined scaling factor) (e.g., 1), a list L0w of weighting factors may be supported to perform weighted averaging of prediction samples from reference frames (e.g., multiple reference frames) of the block. The reference frames (e.g., multiple reference frames) of the block may include at least one reference frame of the block. Otherwise, when other scaling factors are selected for the joint MVD coding mode, a subset of the list L0w of weighting factors may be allowed. In an example, weighting factor information (e.g., a weighting factor index) may be signaled to indicate which weighting factor(s) in the list L0w of weighting factors (i) or (ii) may be selected to perform weighted averaging of prediction samples. The weighted averaging of the prediction samples may be performed by a BCW mode, such as described in equation (5) or equation (6). A list L0w of weighting factors may be predefined. A subset of the list L0w of weighting factors may be predefined.
[0125] In an embodiment, a block in a frame is encoded using a joint MVD coding mode and a BCW mode. The joint MVD coding mode may refer to a JOINT_NEWMV mode (e.g., with or without AMVR) or a JOINT_AMVDNEWMV mode (e.g., with or without AMVR). In an example, the reference frame includes a first reference frame from a reference frame list 0 and a second reference frame from a reference frame list 1, and the MVD of the block includes a first MVD associated with the first reference frame (e.g., MVD0 or mvd_ref0) and a second MVD associated with the second reference frame (e.g., MVD1 or mvd_ref1).
[0126] In an example, the joint MVD coding mode is JOINT_AMVDNEWMV mode, and equations (3) to (4) are used to determine the first MVD (e.g., mvd_ref0) and the second MVD (e.g., mvd_ref1), respectively. At least one corresponding reference frame may refer to the second reference frame, and at least one MVD may refer to the second MVD. Figure 7 , the scaling factors of the first component and the second component of the second MVD may be the same, for example, 1, 2 or 1 / 2. In the example, the predefined scaling factor is 1. If the scaling factor of the component of the second MVD is equal to the predefined scaling factor 1 (corresponding to Figure 7If the weighting factor of the BCW mode is an index 0 in the weighting factor list L0w, the weighting factor of the BCW mode can be determined based on the weighting factor list L0w. Otherwise, the weighting factor of the BCW mode can be determined based on a subset of the weighting factor list L0w. In the example, L0w includes 5 weights, such as -2, 3, 4, 5 and 10.
[0127] In an example, the joint MVD encoding mode is JOINT_NEWMV mode, and equations (3) to (4) are used to determine the first MVD (eg, mvd_ref0) and the second MVD, respectively. (e.g., mvd_ref1). For example, a first MVD (e.g., mvd_ref0) is determined directly from joint_mvd, and each component of a second MVD (e.g., mvd_ref1) is determined using equation (4) with a corresponding scaling factor (e.g., jmvd_scale). In an example, the predefined scaling factor is 1, and the scaling factors of the first component and the second component (e.g., the horizontal component and the vertical component) are 1 and 2 (e.g., Figure 8 1), and thus the scaling factors of the components of the second MVD include scaling factors (e.g., 2 for the vertical component) that are different from the predefined scaling factors (e.g., 1), the weighting factors for the BCW mode may be determined based on a subset of the list of weighting factors L0w. If the scaling factors of the first component and the second component (e.g., the vertical component and the horizontal component) are equal to 1 (corresponding to Figure 8 If index 0 in , the weighting factor of the BCW mode can be determined based on the list L0w of weighting factors.
[0128] Motion information associated with a corresponding reference frame of a block may be determined using a joint MVD coding mode based on a scaling factor of a component of at least one MVD associated with at least one corresponding reference frame of the block. For example, the motion information includes a first MV associated with a first reference frame and a second MV associated with a second reference frame. The first MV may be determined based on a first MVD (e.g., using equation (3)) and a first MVP, and the second MV may be determined based on a second MVD (e.g., using equation (4)) and a second MVP. The first MVP and the second MVP may be determined based on an index (e.g., a DRL index) and a list of MVPs. In an example, entries in the list of MVPs indicate the first MVP and the second MVP.
[0129] Based on the motion information associated with the corresponding reference frame of the block and the determined weighting factor w, the block can be reconstructed using the BCW mode. For example, a first reference block (or a first prediction block) (e.g., including the sample value indicated by P0) is determined based on the first MV, and a second reference block (or a second prediction block) (e.g., including the sample value indicated by P1) is determined based on the second MV. A dual prediction block (e.g., including the sample value indicated by P1) can be determined by the BCW mode (e.g., using equation (5) or (6)). bi-pred indicated sample value).
[0130] In an embodiment, when the scaling factor of the first component or the second component of one of the at least one MVD associated with at least one reference frame of the block is not equal to a given value (e.g., 1) of the joint MVD coding mode, only one weighting factor is allowed in the BCW mode. The one weighting factor may be predefined. In an example, signaling of the one weighting factor is not required. If the given value is 1, then Figure 7 In , if the index is 1 or 2 (e.g., JOINT_AMVDNEWMV mode), only one weighting factor is allowed; and in Figure 8 In the example, only one weighting factor is allowed if the index is 1, 2, 3 or 4 (e.g., JOINT_NEWMV mode). In the example, when the scaling factor of the first component or the second component of one MVD in at least one MVD associated with at least one reference frame of the block is not equal to a given value (e.g., 1) of the joint MVD coding mode, only equal weighting factors are allowed (e.g., the equal weighting factor is 4 in equation (5) or 8 in equation (6)). The equal weighting factors may be predefined. In the example, signaling of the equal weighting factors is not required.
[0131] In an embodiment, when the scaling factor of the first component or the second component of one of the at least one MVDs associated with at least one reference frame of the block is not equal to a given value (e.g., 1) of the joint MVD coding mode, a predefined lookup table may be used to determine the associated weighting factor for the weighted average of multiple prediction samples. Fig. 9 An exemplary predefined lookup table according to an embodiment of the present disclosure is shown. The first column of the lookup table shows the scaling factor pairs (two scaling factors) for the first component and the second component of the MVD. The second column of the lookup table shows the weighting factors corresponding to the corresponding scaling factor pairs in the first column. When the sum of the weights used to average the two prediction samples is 8, the weighting factors in the second column can be 3 and / or 5 (e.g., equation (5)). When the sum of the weights used to average the two prediction samples is 16, the weighting factors in the second column can be 6 and / or 10 (e.g., equation (6)).
[0132] In an embodiment, when the scaling factor of the first component or the second component of one MVD in at least one MVD associated with at least one reference frame of the block is not equal to a given value (e.g., 1) of the joint MVD coding mode, only unequal weighting factors may be used in the weighted average of multiple prediction samples, and equal weighting factors are not allowed. In an example, a subset of the list of weighting factors includes only unequal weighting factors. For example, in the weighted average of multiple prediction samples (e.g., two prediction samples), only one or more groups of unequal weighting factors may be used. Each group of unequal weighting factors may be predefined and may be used for each scaling factor pair of the first component and the second component of at least one MVD. In an embodiment, only 2 unequal weighting factors may be used to perform the weighted average of the two prediction samples. The 2 unequal weighting factors may be 3 or 5, for example, if the sum of the weights used to average the two prediction samples is 8 (e.g., equation (5)). The 2 unequal weighting factors may be 6 or 10, for example, if the sum of the weights used to average the two prediction samples is 16 (e.g., equation (6)).
[0133] A joint MVD coding mode is selected to encode a block. According to an embodiment of the present disclosure, when equal weighting factors are applied to each reference block of the block (e.g., equal weighting factors are used to average predicted samples from multiple reference frames), a list L0s of scaling factors may be allowed to scale components of at least one MVD associated with at least one reference frame (e.g., reference frame 0 and / or reference frame 1). Otherwise, when unequal weighting factors are used to average predicted samples from multiple reference frames, only a subset of the list L0s of scaling factors may be allowed. In an example, scaling factor information (e.g., scaling factor index) may be signaled to indicate which scaling factors in the list L0s of scaling factors (i) a subset of the list L0s of scaling factors or (ii) which scaling factors in the list L0s of scaling factors may be used in the joint MVD coding mode.
[0134] In an embodiment, when unequal weighting factors are used to average predicted samples from multiple reference frames, the scaling factors used to scale the first and second components (e.g., vertical and horizontal components) of at least one MVD may be limited to given predefined values (e.g., 1).
[0135] In an embodiment, when unequal weighting factors are used to average prediction samples from multiple reference frames, the scaling factors used to scale the first and second components (e.g., vertical and horizontal components) of at least one MVD are not allowed to be a given predefined value (e.g., 1).
[0136] In an embodiment, when the joint MVD coding mode is selected to encode the block, the context for signaling a weighting factor index, for example to mix (e.g., weighted average) multiple prediction samples (e.g., for CABAC), may rely on scaling factor information, such as scaling factors of components of at least one MVD associated with at least one reference frame. The weighting factor index may indicate (i) a list L0w of weighting factors or (ii) a weight in a subset of the list L0w of weighting factors.
[0137] In an embodiment, when the scaling factor information indicates that each of the scaling factors of the components of at least one MVD is a given value (or a predefined scaling factor, such as 1), the context used to signal the weighting factor index is a first context. Otherwise, when the scaling factor information indicates that at least one of the scaling factors is different from the given value, the context used to signal the weighting factor index is another set of contexts different from the first context (e.g., a second context).
[0138] In an embodiment, the context for signaling the weighting factor index may depend on whether the block is encoded in the joint MVD coding mode.In an example, the weighting factor index may indicate a weighting factor for mixing two prediction samples, for example by weighted averaging of the two prediction samples.
[0139] In an embodiment, a high-level syntax may be signaled at a high level, such as a sequence level, a frame level, a slice level, a super-block level, or a CTU level, to indicate whether all supported weighting factors (e.g., a list L0w of weight factors) or a subset of weighting factors (e.g., a subset of the list L0w of weight factors) are allowed to be used in the joint MVD coding mode when one of the scaling factors of a component of at least one MVD is not equal to a predetermined value, such as 1.
[0140] In an embodiment, the context (eg, for CABAC) of scaling factor information (eg, scaling factor index) used to signal scaling factors of components of at least one MVD in JMVD mode depends on weighting factor information indicating a weighting factor index.
[0141] Fig.10A flow chart summarizing a process (1000) according to an embodiment of the present disclosure is shown. The process (1000) may be used in a video / image decoder. In various embodiments, the process (1000) is performed by a processing circuit system, such as a processing circuit system that performs the functions of a video decoder (110), a processing circuit system that performs the functions of a video decoder (210), etc. In some embodiments, the process (1000) is implemented as software instructions, so when the processing circuit system executes the software instructions, the processing circuit system performs the process (1000). The process (1000) starts at (S1001) and proceeds to (S1010).
[0142] At (S1010), a bitstream including a frame (also referred to as a picture) may be received. Coding information of a block in the frame may indicate that the block is encoded using a joint MVD coding mode and a composite weighted prediction mode (or BCW mode). The coding information may also include scaling factor information of the joint MVD coding mode.
[0143] At (S1020), it may be determined whether the scaling factor information indicates that each of the scaling factors of the components of at least one MVD associated with at least one corresponding reference frame of the block is a predefined scaling factor (e.g., 1). If the scaling factor information indicates that each of the scaling factors of the components of at least one MVD associated with at least one corresponding reference frame of the block is a predefined scaling factor, the process (1000) proceeds to (S1030). Otherwise, if the scaling factor information indicates that at least one of the scaling factors is different from the predefined scaling factor, the process (1000) proceeds to (S1040).
[0144] At (S1030), the weighting factors for the BCW mode may be determined based on a list of weighting factors, such as L0w. In an example, the list of weighting factors is predefined and not signaled in the bitstream. In an example, the list is signaled in the bitstream. Process (1000) proceeds to (S1050).
[0145] At (S1040), the weighting factors for the BCW mode may be determined based on a subset of the list of weighting factors. In an example, the subset of the list is predefined and not signaled in the bitstream. In an example, the subset of the list is signaled in the bitstream. Process (1000) proceeds to (S1050).
[0146] At (S1050), the motion information associated with the corresponding reference frame of the block can be determined using the joint MVD coding mode and based on the scaling factor of the component of at least one MVD associated with at least one corresponding reference frame of the block. The reference frame of the block may include at least one corresponding reference frame. Then, the process proceeds to (S1099) and terminates.
[0147] The process (1000) may be adjusted appropriately. Steps in the process (1000) may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.
[0148] In an example, the block may be reconstructed using the BCW mode and based on motion information associated with a corresponding reference frame of the block and the determined weighting factors.
[0149] In an example, steps (S1020), (S1030) and (S1040) may be adjusted as follows. If the scaling factor information indicates that each of the scaling factors of the components of at least one MVD associated with at least one corresponding reference frame of the block is a predefined scaling factor, the weighting factors of the BCW mode are determined based on the list of weighting factors. Otherwise, if the scaling factor information indicates that at least one of the scaling factors is different from the predefined scaling factors, the weighting factors of the BCW mode may be determined based on a subset of the list of weighting factors.
[0150] In an embodiment, the scaling factor information indicates that at least one of the scaling factors is different from a predefined scaling factor, the subset of the list of weighting factors comprises only one weighting factor, and the determined weighting factor is the one weighting factor and is not signaled.
[0151] In an example, the one weighting factor is an equal weighting factor. A prediction block of the block is obtained by averaging reference blocks in the corresponding reference frame with an equal weighting factor for each reference block, such as described with reference to equation (4) where the equal weighting factor is 4. The block can be reconstructed based on the prediction block.
[0152] In an example, the weighting factor is determined based on a scaling factor of a component of at least one MVD, for example with reference to Fig. 9 For example, based on a weighting factor of a component of one MVD in at least one MVD (e.g., Fig. 9 ) and scaling factors (e.g. Fig. 9 ) between pairs of scaling factors in , for example, Fig. 9The weighting factor is determined by using a lookup table in (a). In an example, the reference frame includes a first reference frame having a first weighting factor and a second reference frame having a second weighting factor, and the sum of the first weighting factor and the second weighting factor is a constant. If the constant is 8, the weighting factor is 3 or 5, and the first weighting factor and the second weighting factor include 3 and 5. If the constant is 16, the weighting factor is 6 or 10, and the first weighting factor and the second weighting factor include 6 and 10. A prediction block of the block can be obtained by averaging the first reference frame having the first weighting factor and the second reference frame having the second weighting factor, and the block can be reconstructed based on the prediction block.
[0153] In an embodiment, a subset of the list of weighting factors includes only unequal weighting factors. The weighting factor may be determined as one of the unequal weighting factors. The prediction block of the block may be obtained by averaging the reference blocks in the corresponding reference frame with corresponding unequal weighting factors determined based on one of the unequal weighting factors, such as described in equation (5) where the weighting factor is not 4. In an example, the unequal weighting factors are two unequal weighting factors including the weighting factors, the reference blocks include a first reference block and a second reference block, and the prediction block is obtained by averaging the first reference block and the second reference block with the two unequal weighting factors, respectively. In an example, the two unequal weighting factors include (i) 3 and 5 or (ii) 6 and 10.
[0154] In an embodiment, the context for signaling the weighting factor index (eg, for CABAC) depends on the scaling factor information.The weighting factor may be determined based on (i) the weighting factor index and (ii) a list of weighting factors or a subset of the list of weighting factors.
[0155] In an example, based on the scaling factor information indicating that each of the scaling factors of the components of at least one MVD is a predefined scaling factor, the context used to signal a weighting factor index is a first context, and based on the scaling factor information indicating that at least one of the scaling factors is different from the predefined scaling factors, the context used to signal the weighting factor index is a second context different from the first context.
[0156] Fig.11A flow chart summarizing a process (1100) according to an embodiment of the present disclosure is shown. The process (1100) may be used in a video encoder. In various embodiments, the process (1100) is performed by a processing circuit system, such as a processing circuit system that performs the functions of a video encoder (103), a processing circuit system that performs the functions of a video encoder (303), etc. In some embodiments, the process (1100) is implemented as software instructions, so when the processing circuit system executes the software instructions, the processing circuit system performs the process (1100). The process (1100) starts at (S1101) and proceeds to (S1110).
[0157] At (S1110), it can be determined whether each of the scaling factors of the components of at least one MVD associated with at least one corresponding reference frame of the block is a predefined scaling factor. The scaling factors can be used in a joint MVD coding mode to encode the block in the frame. In addition, the BCW mode can be applied to encode the block. If each of the scaling factors of the components of at least one MVD associated with at least one corresponding reference frame of the block is determined to be a predefined scaling factor (e.g., 1), the process (1100) proceeds to (S1120). Otherwise, at least one of the scaling factors is different from the predefined scaling factor, and the process (1100) proceeds to (S1130).
[0158] At (S1120), the weighting factor of the BCW mode may be determined based on a list of weighting factors, such as L0w. The process (1100) proceeds to (S1140).
[0159] At (S1130), a weighting factor for the BCW mode may be determined based on a subset of the list of weighting factors.The process (1100) proceeds to (S1140).
[0160] At (S1140), the block may be encoded based on the BCW mode, for example, the block may be predicted using equation (5) or equation (6).
[0161] The scaling factor information of the joint MVD coding mode may be encoded to indicate the scaling factor. The scaling factor information may indicate a scaling factor of a component of at least one MVD associated with at least one corresponding reference frame of the block. In an example, the encoded scaling factor information is included in a video bitstream.
[0162] Then, the process (1100) proceeds to (S1199) and terminates.
[0163] The process (1100) may be adapted as appropriate. Steps in the process (1100) may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.
[0164] In an example, prior to (S1110), a scaling factor of a component of at least one MVD associated with at least one corresponding reference frame may be determined. Motion information associated with the corresponding reference frame of the block may be determined. The reference frame of the block includes at least one corresponding reference frame. At least one MVD associated with at least one corresponding reference frame of the block may be determined based on the motion information. A scaling factor of a component of at least one MVD associated with at least one corresponding reference frame may be determined, for example, based on the at least one MVD. In an example, the scaling factor is further determined based on a POC difference between the corresponding reference frame and a current frame including the block.
[0165] The corresponding embodiments described in the process (1000) can be applied to the process (1100) used in the encoding process.
[0166] Fig.12 A flow chart summarizing a process (1200) according to an embodiment of the present disclosure is shown. The process (1200) may be used in a video / image decoder. In a corresponding embodiment, the process (1200) is performed by a processing circuit system, such as a processing circuit system that performs the functions of a video decoder (110), a processing circuit system that performs the functions of a video decoder (210), etc. In some embodiments, the process (1200) is implemented as software instructions, so when the processing circuit system executes the software instructions, the processing circuit system performs the process (1200). The process (1200) starts at (S1201) and proceeds to (S1210).
[0167] At (S1210), a bitstream including a frame may be received. Coding information of a block in the frame may indicate that the block is encoded using a joint MVD coding mode and a BCW mode. The coding information may also include weighting factor information of the BCW mode.
[0168] At (S1220), it may be determined whether the weighting factor information indicates that an equal weighting factor is applied to each of the reference frames of the block. If the weighting factor information is determined to indicate that an equal weighting factor is applied to each of the reference frames of the block, the process (1200) proceeds to (S1230). Otherwise, if the weighting factor information is determined to indicate that unequal weighting factors are applied to the reference frames of the block, the process (1200) proceeds to (S1240).
[0169] At (S1230), a scaling factor of a component of at least one MVD associated with at least one corresponding one of the reference frames may be determined based on the list of scaling factors, such as L0s.The process (1200) proceeds to (S1250).
[0170] At (S1240), a scaling factor for a component of at least one MVD associated with at least one respective one of the reference frames may be determined based on the subset of the list of scaling factors.The process (1200) proceeds to (S1250).
[0171] At (S1250), motion information associated with a corresponding reference frame of the block may be determined using a joint MVD coding mode and based on the determined scaling factor.
[0172] At (S1260), the block may be reconstructed based on the motion information and weighting factor information associated with the corresponding reference frame of the block using the BCW mode.Then, the process (1200) proceeds to (S1299) and terminates.
[0173] The process (1200) may be adapted as appropriate. Steps in the process (1200) may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.
[0174] In an example, steps (S1220), (S1230) and (S1240) may be adjusted as follows: If the weighting factor information indicates that an equal weighting factor is applied to each of the reference frames of the block, the scaling factor of the component of at least one MVD associated with at least one corresponding reference frame of the reference frames may be determined based on the list of scaling factors. If the weighting factor information indicates that unequal weighting factors are applied to the reference frames of the block, the scaling factor of the component of at least one MVD may be determined based on a subset of the list of scaling factors.
[0175] In an embodiment, the weighting factor information indicates that unequal weighting factors are applied to reference frames of the block, a subset of the list of scaling factors includes only predefined scaling factors, and the determined scaling factor is equal to the predefined scaling factor and is not signaled. In an example, the predefined scaling factor is 1.
[0176] In an embodiment, the weighting factor information indicates that unequal weighting factors are applied to reference frames of the block, and the subset of the list of scaling factors does not include a predefined scaling factor (eg, 1).
[0177] In an embodiment, the context for signaling the scaling factor index (eg, for CABAC) depends on the weighting factor information.The scaling factor may be determined based on (i) the scaling factor index and (ii) a list of scaling factors or a subset of the list of scaling factors.
[0178] Fig.13A flow chart summarizing a process (1300) according to an embodiment of the present disclosure is shown. The process (1300) may be used in a video encoder. In various embodiments, the process (1300) is performed by a processing circuit system, such as a processing circuit system that performs the functions of a video encoder (103), a processing circuit system that performs the functions of a video encoder (303), etc. In some embodiments, the process (1300) is implemented as software instructions, so when the processing circuit system executes the software instructions, the processing circuit system performs the process (1300). The process (1300) starts at (S1301) and proceeds to (S1310).
[0179] At (S1310), it may be determined whether an equal weighting factor is applied to each of the reference frames of a block in a frame (or current frame). The block is to be encoded using the BCW mode. If it is determined that an equal weighting factor is applied to each of the reference frames of the block to encode the block, the process (1300) proceeds to (S1320). Otherwise, a non-equal weighting factor is applied to the reference frames of the block to encode the block, and the process (1300) proceeds to (S1330).
[0180] At (S1320), a scaling factor for a component of at least one MVD associated with at least one corresponding one of the reference frames may be determined based on the list of scaling factors (eg, L0s). The scaling factor is used for the joint MVD coding mode. The process (1300) proceeds to (S1340).
[0181] At (S1330), a scaling factor for a component of at least one MVD associated with at least one respective one of the reference frames may be determined based on a subset of the list of scaling factors.The process (1300) proceeds to (S1340).
[0182] At (S1340), the block may be encoded based on the BCW mode, for example, the block may be predicted using equation (5) or equation (6).
[0183] The weighting factor information of the BCW mode may be encoded to indicate the weighting factors (eg, equal weighting factors or unequal weighting factors). In an example, the encoded weighting factor information is included in a video bitstream.
[0184] Then, the process (1300) proceeds to (S1399) and terminates.
[0185] The process (1300) may be adapted as appropriate. Steps in the process (1300) may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.
[0186] In an example, before (S1310), the weighting factors used in BCW mode to encode the block may be determined based on a predefined list of weighting factors, such as L0w or a subset of L0w.
[0187] The various embodiments described in the process (1200) may be applied to the process (1300) used in the encoding process.
[0188] The embodiments in the present disclosure may be used alone or in combination in any order. In addition, each of the method (or embodiment), encoder, and decoder may be implemented by a processing circuit system (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium. In the present disclosure, the term block may be interpreted as a prediction block, a coding block, a coding unit (CU), etc.
[0189] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. Fig.14 A computer system (1400) suitable for implementing certain embodiments of the disclosed subject matter is shown.
[0190] Computer software may be encoded using any suitable machine code or computer language, which may be subjected to mechanisms such as assembly, compilation, linking, etc. to create code comprising instructions that may be executed directly by one or more computer central processing units (CPU), graphics processing units (GPU), etc., or by interpretation, microcode execution, etc.
[0191] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, etc.
[0192] Fig.14 The components for the computer system (1400) shown in the example are exemplary in nature and are not intended to suggest any limitation on the scope of use or functionality of computer software implementing embodiments of the present disclosure. Nor should the configuration of components be interpreted as having any dependency or requirement relating to any one or combination of components shown in the exemplary embodiment of the computer system (1400).
[0193] The computer system (1400) may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, tapping), visual input (e.g., gestures), olfactory input (not depicted). Human-machine interface devices may also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).
[0194] Input human-machine interface devices may include one or more of the following (only one of each is depicted): keyboard (1401), mouse (1402), touchpad (1403), touch screen (1410), data gloves (not shown), joystick (1405), microphone (1406), scanner (1407), camera (1408).
[0195] The computer system (1400) may also include certain human interface output devices. Such human interface output devices may stimulate one or more senses of a human user through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include: tactile output devices (e.g., tactile feedback via a touch screen (1410), a data glove (not shown), or a joystick (1405), but there may also be tactile feedback devices that do not function as input devices); audio output devices (e.g., speakers (1409), headphones (not depicted)); visual output devices (e.g., screens (1410), including CRT (Cathode Ray Tube, CRT) screens, LCD (Liquid Crystal Display, LCD) screens, plasma screens, OLED (Organic Light Emitting Diode, OLED) screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities - some of which may be able to output two-dimensional visual outputs or more than three-dimensional outputs, such as stereoscopic image outputs; virtual reality glasses (not depicted); holographic displays and cigarette cans (not depicted)); and printers (not depicted).
[0196] The computer system (1400) may also include storage devices and their associated media accessible to humans, such as optical media including CD / DVD ROM (Read Only Memory, ROM) / RW (1420) with CD / DVD and other media (1421), thumb drives (1422), removable hard drives or solid-state drives (1423), legacy magnetic media such as tapes and floppy disks (not depicted), dedicated ROM / ASIC (Application Specific Integrated Circuit, ASIC) / PLD (Programmable Logic Device, PLD) based devices such as security dongles (not depicted), etc.
[0197] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.
[0198] The computer system (1400) may also include an interface (1454) to one or more communication networks (1455). The network may be, for example, wireless, wired, optical. The network may also be local, wide, metropolitan, vehicle and industrial, real-time, delay-tolerant, etc. Examples of networks include: local area networks such as Ethernet; wireless LAN (Local Area Network, LAN); cellular networks including GSM (Global System for Mobile Communications, GSM), 3G (the Third Generation, 3G), 4G (the Fourth Generation, 4G), 5G (the Fifth Generation, 5G), LTE (Long Term Evolution, LTE), etc.; TV wired or wireless wide area digital networks including cable TV, satellite TV and terrestrial broadcast TV; vehicles and industries including CANBus, etc. Some networks typically require an external network interface adapter attached to some general data port or peripheral bus (1449) (such as, for example, a USB port of the computer system (1400)); other networks are typically integrated into the core of the computer system (1400) by attaching to a system bus as described below (e.g., to an Ethernet interface in a PC computer system or to a cellular network interface in a smartphone computer system). Using any of these networks, the computer system (1400) can communicate with other entities. Such communications can be one-way receive only (e.g., broadcast TV), one-way send only (e.g., a CAN bus to some CAN bus devices), or two-way, for example, to other computer systems using a local area digital network or a wide area digital network. Certain protocols and protocol stacks can be used on each of these networks and network interfaces as described above.
[0199] The above-mentioned human interface devices, human-accessible storage devices, and network interfaces may be attached to the core ( 1440 ) of the computer system ( 1400 ).
[0200] The core (1440) may include one or more central processing units (CPUs) (1441), graphics processing units (GPUs) (1442), dedicated programmable processing units in the form of field programmable gate areas (FPGAs) (1443), hardware accelerators for certain tasks (1444), graphics adapters (1450), etc. These devices, along with read-only memory (ROM) (1445), random access memory (1446), internal mass storage devices (1447) such as internal non-user accessible hard drives, SSDs (Solid State Drives, SSDs), etc., may be connected via a system bus (1448). In some computer systems, the system bus (1448) may be accessible in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices may be attached to the core's system bus (1448) directly or via a peripheral bus (1449). In an example, a screen (1410) may be connected to a graphics adapter (1450). The architecture of the peripheral bus includes PCI (Peripheral Component Interconnect / Interface, PCI), USB (Universal Serial Bus, USB), etc.
[0201] The CPU (1441), GPU (1442), FPGA (1443) and accelerator (1444) can execute certain instructions, which can be combined to form the computer code mentioned above. The computer code can be stored in ROM (1445) or RAM (Random Access Memory, RAM) (1446). Transient data can also be stored in RAM (1446), while permanent data can be stored in, for example, an internal large-capacity storage device (1447). Fast storage and retrieval of any storage device in the storage device can be achieved by using a cache memory, which can be closely associated with one or more CPUs (1441), GPUs (1442), large-capacity storage devices (1447), ROMs (1445), RAMs (1446), etc.
[0202] The computer readable medium may have computer code thereon for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or the medium and computer code may be of a type well known and available to those skilled in the art of computer software.
[0203] As an example and not limitation, a computer system (1400) having an architecture and in particular a core (1440) can provide functionality due to a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software contained in one or more tangible computer-readable media. Such a computer-readable medium can be a medium associated with a user-accessible mass storage device as described above, as well as certain storage devices of the core (1440) having non-transitory properties, for example, a core internal mass storage device (1447) or a ROM (1445). Software implementing various embodiments of the present disclosure can be stored in such a device and executed by the core (1440). Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software can enable the core (1440) and in particular the processor therein (including a CPU, GPU, FPGA, etc.) to perform a specific process or a specific part of a specific process described herein, including defining a data structure stored in the RAM (1446) and modifying such a data structure according to a process defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of hard-wired logic or otherwise embodied in circuitry (e.g., accelerator (1444)) that may replace or operate in conjunction with software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may include logic, and references to logic may also include software. Where appropriate, references to computer-readable media may include circuitry (e.g., an integrated circuit (IC)) storing software for execution, circuitry implementing logic for execution, or both. The present disclosure includes any suitable combination of hardware and software.
[0204] The use of "at least one of" or "one of" in this disclosure is intended to include any one or combination of the recited elements. For example, references to at least one of A, B, or C; at least one of A, B, and C; at least one of A, B, and / or C; and at least one of A through C are intended to include only A, only B, only C, or any combination thereof. References to one of A or B and one of A and B are intended to include A or B or (A and B). The use of "one of" does not exclude any combination of the recited elements where applicable (e.g., where the elements are not mutually exclusive).
[0205] Although the present disclosure has described several exemplary embodiments, there are changes, permutations, and various alternative equivalents that fall within the scope of the present disclosure. It will therefore be appreciated that those skilled in the art will be able to conceive of many systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure.
Claims
1. A method for video decoding in a video encoder, the method comprising: Receiving a bitstream including a frame, encoding information of a block in the frame indicating that the block is encoded using a joint motion vector difference (JMVD) encoding mode and a composite weighted prediction mode, the encoding information further comprising scaling factor information of the JMVD encoding mode; In response to the scaling factor information indicating that each of the scaling factors of the components of at least one MVD associated with at least one corresponding reference frame of the block is a predefined scaling factor, determining a weighting factor for the composite weighted prediction mode based on a list of weighting factors, the list being signaled in the bitstream; In response to the scaling factor information indicating that at least one of the scaling factors is different from the predefined scaling factors, determining the weighting factors for the composite weighted prediction mode based on a subset of the list of weighting factors, only the subset of the list being signaled in the bitstream; determining, using the JMVD coding mode, motion information associated with corresponding reference frames of the block based on the scaling factors of the components of the at least one MVD associated with the at least one corresponding reference frame of the block, the reference frames of the block comprising the at least one corresponding reference frame; as well as The block is reconstructed using the composite weighted prediction mode based on the motion information associated with the corresponding reference frame of the block and the determined weighting factors.
2. The method according to claim 1, wherein: The scaling factor information indicates that the at least one of the scaling factors is different from the predefined scaling factor, said subset of said list of weighting factors comprises only one weighting factor, and The determined weighting factor is the one weighting factor and is not signaled.
3. The method according to claim 2, wherein: The one weighting factor is an equal weighting factor, and The reconstruction includes obtaining a prediction block for the block by averaging the reference blocks in the corresponding reference frame with the equal weighting factor for each reference block; as well as The block is reconstructed based on the prediction block.
4. The method according to claim 2, wherein: The determining the weighting factor comprises determining the weighting factor based on the scaling factor of the component of the at least one MVD.
5. The method according to claim 4, wherein: The determining the weighting factor comprises determining the weighting factor based on a predefined relationship between the weighting factor and a scaling factor of a component of one of the at least one MVD.
6. The method according to claim 4, wherein: The reference frame includes a first reference frame having a first weighting factor and a second reference frame having a second weighting factor, and a sum of the first weighting factor and the second weighting factor is a constant, In response to the constant being 8, The weighting factor is 3 or 5, and The first weighting factor and the second weighting factor include 3 and 5, In response to the constant being 16, The weighting factor is 6 or 10, and The first weighting factor and the second weighting factor include 6 and 10, and The reconstruction includes obtaining a prediction block of the block by averaging the first reference frame having the first weighting factor and the second reference frame having the second weighting factor, and The block is reconstructed based on the prediction block.
7. The method according to claim 1, wherein: said subset of said list of weighting factors comprises only unequal weighting factors; The determining of the weighting factor comprises determining the weighting factor to be one of the unequal weighting factors; and The reconstruction includes obtaining a prediction block of the block by averaging reference blocks in the corresponding reference frame using corresponding unequal weighting factors determined based on the one of the unequal weighting factors; as well as The block is reconstructed based on the prediction block.
8. The method according to claim 7, wherein: The unequal weighting factors are two unequal weighting factors including the weighting factor, The reference blocks include a first reference block and a second reference block, and The obtaining of the prediction block comprises averaging the first reference block and the second reference block using the two unequal weighting factors respectively.
9. The method according to claim 8, wherein: The two unequal weighting factors include (i) 3 and 5 or (ii) 6 and 10.
10. The method according to claim 1, wherein: A context for signaling a weighting factor index depends on the scaling factor information; Determining the weighting factor of the composite weighted prediction mode based on the list of weighting factors comprises determining the weighting factor based on the weighting factor index and the list of weighting factors; as well as The determining the weighting factor of the composite weighted prediction mode based on the subset of the list of weighting factors comprises determining the weighting factor based on the weighting factor index and the subset of the list of weighting factors.
11. The method according to claim 10, wherein: indicating that each of the scaling factors of the components of the at least one MVD is the predefined scaling factor based on the scaling factor information, for signaling that the context of the weighting factor index is a first context, and Based on the scaling factor information indicating that the at least one of the scaling factors is different from the predefined scaling factor, the context for signaling the weighting factor index is a second context different from the first context.
12. A method for performing video decoding in a video encoder, the method comprising: Receiving a bitstream including a frame, encoding information of a block in the frame indicating that the block is encoded using a joint motion vector difference (MVD) coding mode and a composite weighted prediction mode, and the encoding information further indicating weighting factor information of the composite weighted prediction mode; In response to the weighting factor information indicating that an equal weighting factor is applied to each of the reference frames of the block, determining a scaling factor for a component of at least one MVD associated with at least one corresponding one of the reference frames based on a list of scaling factors; determining, in response to the weighting factor information indicating that unequal weighting factors are applied to the reference frame of the block, the scaling factors for the components of the at least one MVD based on a subset of the list of scaling factors; determining, using the joint MVD coding mode, motion information associated with the corresponding reference frame for the block based on the determined scaling factor; as well as The block is reconstructed based on the motion information and the weighting factor information associated with the corresponding reference frame of the block using the composite weighted prediction mode.
13. The method according to claim 12, wherein: The weighting factor information indicates that the unequal weighting factors are applied to the reference frame of the block, said subset of the list of scaling factors comprises only predefined scaling factors, and The determined scaling factor is equal to the predefined scaling factor and is not signaled.
14. The method according to claim 13, wherein: The predefined scaling factor is 1.
15. The method according to claim 12, wherein: The weighting factor information indicates that the unequal weighting factors are applied to the reference frame of the block, and The subset of the list of scaling factors does not include predefined scaling factors.
16. The method according to claim 15, wherein: The predefined scaling factor is 1.
17. The method according to claim 12, wherein: A context for signaling a scaling factor index depends on said weighting factor information; The determining the scaling factor based on the list of scaling factors comprises determining the scaling factor based on the scaling factor index and the list of scaling factors; as well as The determining the scaling factor based on the subset of the list of scaling factors comprises determining the scaling factor based on the scaling factor index and the subset of the list of scaling factors.
18. A device for video decoding, comprising A processing circuit system configured to: Receiving a bitstream including a frame, encoding information of a block in the frame indicating that the block is encoded using a joint motion vector difference (JMVD) encoding mode and a composite weighted prediction mode, the encoding information further comprising scaling factor information of the JMVD encoding mode; In response to the scaling factor information indicating that each of the scaling factors of the components of at least one MVD associated with at least one corresponding reference frame of the block is a predefined scaling factor, determining a weighting factor for the composite weighted prediction mode based on a list of weighting factors, the list being signaled in the bitstream; In response to the scaling factor information indicating that at least one of the scaling factors is different from the predefined scaling factor, determining a weighting factor for the composite weighted prediction mode based on a subset of the list of weighting factors, only the subset of the list being signaled in the bitstream; determining, using the JMVD coding mode, motion information associated with corresponding reference frames of the block based on the scaling factors of the components of the at least one MVD associated with the at least one corresponding reference frame of the block, the reference frames of the block comprising the at least one corresponding reference frame; and The block is reconstructed using the composite weighted prediction mode based on the motion information associated with the corresponding reference frame of the block and the determined weighting factors.
19. The apparatus according to claim 18, wherein: The scaling factor information indicates that the at least one of the scaling factors is different from the predefined scaling factor, said subset of said list of weighting factors comprises only one weighting factor, and The determined weighting factor is the one weighting factor and is not signaled.
20. The apparatus of claim 19, wherein: The one weighting factor is an equal weighting factor, and The processing circuit is configured to: obtaining a prediction block for the block by averaging reference blocks in the corresponding reference frame with the equal weighting factor for each reference block; and The block is reconstructed based on the prediction block.
Citation Information
Patent Citations
Block based weighting factor for joint motion vector difference coding mode
US20240314343A1
Cited By
Data processing method, computing unit, electronic device, storage medium and program product
CN121350397A
