Method and apparatus for encoding / decoding video
By adapting the length of the motion compensation interpolation filter in video encoding, selecting the appropriate filter according to the position and motion difference of the video block, the problem of inappropriate filter length in inter prediction is solved, and video encoding and decoding is improved Performance and efficiency.
Patent Information
- Application Number
- CN202380031390.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-30
- Filing Date
- 2023-03-24
- Publication Date
- 2024-11-15
AI Technical Summary
The existing video encoding technology has the problem of inappropriate motion compensation filter length in inter-frame prediction, resulting in imbalance in reconstruction accuracy and computational complexity. Especially when the filter acquires samples from another moving area, the longer filter does not expect.
Using an adaptive motion compensation interpolation filter, select the appropriate filter length based on the position, motion difference and encoding mode of the video block, for example by selecting the 8-tap filter of VVC and the 12-tap filter of ECM, or creating asymmetric filters to accommodate discontinuous positions.
It improves the accuracy and efficiency of inter-frame prediction, reduces the computational complexity, adapts to the motion characteristics of different video contents, and improves the performance of video encoding and decoding.
Smart Images

Figure CN118975258A_ABST
Abstract
Description
Technical Field
[0001] The present embodiments generally relate to methods and apparatus for video encoding or decoding.Some embodiments relate to methods and apparatus for video encoding or decoding using an adaptive interpolation motion compensation filter. Background Art
[0002] In order to achieve high compression efficiency, image and video coding schemes usually use prediction and transformation to balance the spatial and temporal redundancy in video content. Typically, intraprediction or interprediction is used to exploit intra-frame or inter-frame picture correlation, and then the difference between the original block and the predicted block (usually expressed as a prediction error or prediction residual) is transformed, quantized, and entropy encoded. In order to reconstruct the video, the compressed data is decoded by the inverse process corresponding to entropy coding, quantization, transformation, and prediction. Summary of the invention
[0003] According to a first aspect, a method for encoding a video is provided, wherein at least one motion compensation interpolation filter is determined for a block of the video, and a prediction block is determined based on the at least one motion compensation interpolation filter and a reference block. The block is encoded based at least on the prediction block.
[0004] A device for encoding a video is provided. The device includes one or more processors, wherein the one or more processors are configured to encode a block of the video. To this end, the one or more processors are configured to determine at least one motion compensation interpolation filter for the block, determine a prediction block based on the at least one motion compensation interpolation filter and a reference block, and encode the block based on at least the prediction block.
[0005] According to another aspect, a method for decoding a video is provided, wherein at least one motion compensation interpolation filter is determined for a block of the video, and a prediction block is determined based on the at least one motion compensation interpolation filter and a reference block. The block is decoded based at least on the prediction block.
[0006] A device for decoding a video is provided. The device includes one or more processors, wherein the one or more processors are configured to decode a block of the video. To this end, the one or more processors are configured to determine at least one motion compensation interpolation filter for the block, determine a prediction block based on the at least one motion compensation interpolation filter and a reference block, and decode the block based on at least the prediction block.
[0007] One or more embodiments also provide a computer program including instructions, which, when executed by one or more processors, cause the one or more processors to perform an encoding method or a decoding method according to any embodiment described herein. One or more embodiments also provide a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to the above method. One or more embodiments also provide a computer-readable storage medium having stored thereon a bitstream generated according to the above method. One or more embodiments also provide a method and apparatus for sending or receiving a bitstream generated according to the above method. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] Figure 1 A block diagram of a system is shown in which aspects of the present embodiments may be implemented.
[0009] Figure 2 A block diagram of an embodiment of a video encoder is shown.
[0010] Figure 3 A block diagram of an embodiment of a video decoder is shown.
[0011] Figure 4 Affine motion models based on control points are shown, a 4-parameter affine model on the left and a 6-parameter affine model on the right.
[0012] Figure 5 The affine motion vector field for each sub-block is shown.
[0013] Figure 6 Shown are spatial neighboring blocks used in sub-block temporal motion vector prediction mode (SbTMVP) in VVC.
[0014] Figure 7 An example of deriving a sub-CU motion field by applying motion shifts from spatial neighbors and scaling motion information from corresponding collocated sub-CUs (sub-coding units) in SbTMVP mode is shown.
[0015] Figure 8 Examples of frequency responses of a 12-tap interpolation filter (proposed) and a VVC interpolation filter at half-pel phase are shown.
[0016] Fig. 9 An example of a method for encoding a block of a video according to an embodiment is shown.
[0017] Fig.10 An example of a method for decoding a block of a video according to an embodiment is shown.
[0018] Fig.11An example of sub-block positions in a coding unit (CU) is shown.
[0019] Fig.12 An example of a method of MCIF selection based on motion difference according to an embodiment is shown.
[0020] Fig.13 An example of a method of MCIF selection based on coding modes of neighbor blocks according to an embodiment is shown.
[0021] Fig.14 An example of a method of MCIF selection based on a 4-parameter affine model condition according to an embodiment is shown.
[0022] Fig.15 An example of creation of an asymmetric filter according to an embodiment is shown, where the left portion of the filter is shorter than the right portion of the filter.
[0023] Fig.16 An example of creation of an asymmetric filter according to an embodiment is shown, where the right portion of the filter is shorter than the left portion of the filter.
[0024] Fig.17 An example of a method for asymmetric filter selection according to an embodiment is shown.
[0025] Fig.18 Two remote devices communicating over a communication network are shown according to one example of the present principles.
[0026] Fig.19 The syntax of a signal according to an example of the present principles is shown. DETAILED DESCRIPTION
[0027] The present application describes a number of aspects, including tools, features, embodiments, models, methods, etc. Many aspects in these aspects are specifically described, and at least in order to show individual features, are usually described in a manner that may sound limited. However, this is to describe the purpose clearly and does not limit the application or scope of these aspects. In fact, all different aspects can be combined and interchanged to provide other aspects. In addition, aspects can also be combined and interchanged with the aspects described in the early applications.
[0028] The aspects described and contemplated in this application can be implemented in many different forms. Figure 1 , Figure 2 and Figure 3 Some embodiments are provided, but other embodiments are contemplated and are intended for Figure 1 , Figure 2 and Figure 3The discussion does not limit the scope of implementation. At least one aspect generally relates to video encoding and decoding, and at least one other aspect generally relates to sending a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having stored thereon instructions for encoding or decoding video data according to any of the methods described, and / or computer-readable storage media having stored thereon bitstreams generated according to any of the methods described.
[0029] In this application, the terms "reconstruction" and "decoding" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image", "picture" and "frame" may be used interchangeably.
[0030] Various methods are described herein, and each method includes one or more steps or actions for realizing the method.Unless the correct operation of the method requires a specific order of steps or actions, otherwise the order and / or use of specific steps and / or actions can be modified or combined.In addition, terms such as "first", "second" etc. can be used to modify elements, components, steps, operations, etc. in various embodiments, such as, for example, "first decoding" and "second decoding".Unless specifically required, the use of such terms does not mean the sequencing of the operation to modification.Therefore, in this example, the first decoding does not need to be performed before the second decoding, and can occur in, for example, before, during or in a time period overlapping with the second decoding.
[0031] Figure 1 A block diagram of an example of a system in which various aspects and embodiments can be implemented is shown. System 100 can be implemented as a device including various components described below, and is configured to perform one or more aspects described in this application. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected household appliances and servers. The elements of system 100 can be embodied in a single integrated circuit, multiple ICs and / or discrete components, either alone or in combination. For example, in at least one embodiment, the processing and encoder / decoder elements of system 100 are distributed over multiple ICs and / or discrete components. In various embodiments, system 100 is coupled to other systems or other electronic devices via, for example, a communication bus or by a dedicated input and / or output port communication. In various embodiments, system 100 is configured to implement one or more aspects described in this application.
[0032] The system 100 includes at least one processor 110 configured to execute instructions loaded therein for implementing, for example, various aspects described in the present application. The processor 110 may include embedded memory, input-output interfaces, and various other circuits known in the art. The system 100 includes at least one memory 120 (e.g., a volatile memory device and / or a non-volatile memory device). The system 100 includes a storage device 140, which may include a non-volatile memory and / or a volatile memory, including but not limited to an EEPROM, a ROM, a PROM, a RAM, a DRAM, an SRAM, a flash memory, a magnetic disk drive, and / or an optical disk drive. As a non-limiting example, the storage device 140 may include an internal storage device, an attached storage device, and / or a network accessible storage device.
[0033] The system 100 includes an encoder / decoder module 130, which is configured to, for example, process data to provide encoded video or decoded video, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents a (multiple) module that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. In addition, the encoder / decoder module 130 may be implemented as a separate element of the system 100, or may be incorporated into the processor 110 as a combination of hardware and software known to those skilled in the art.
[0034] Program code to be loaded to the processor 110 or the encoder / decoder 130 to perform various aspects described in this application may be stored in the storage device 140 and subsequently loaded onto the memory 120 for execution by the processor 110. According to various embodiments, one or more of the processor 110, the memory 120, the storage device 140, and the encoder / decoder module 130 may store one or more of various items during the execution of the processes described in this application. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.
[0035] In some embodiments, memory internal to the processor 110 and / or encoder / decoder module 130 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 110 or the encoder / decoder module 130) is used for one or more of these functions. The external memory may be memory 120 and / or storage device 140, such as dynamic volatile memory and / or non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store the operating system of the television. In at least one embodiment, fast external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations, such as for MPEG-2 (MPEG refers to Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard developed by the JVET Joint Video Experts Group).
[0036] Input may be provided to the elements of system 100 via various input devices, as indicated at block 105. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives an RF signal transmitted over the air, for example, by a broadcaster, (ii) a component (COMP) input terminal (or a set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Figure 1 Other examples not shown include composite video.
[0037] In various embodiments, the input device of block 105 has associated corresponding input processing elements known in the art. For example, the RF section may be associated with elements suitable for the following operations: (i) selecting a desired frequency (also referred to as selecting a signal, or band limiting a signal to a certain frequency band), (ii) down-converting the selected signal, (iii) again band-limiting to a narrower frequency band to select a signal band that (for example) may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a multiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal transmitted by wired (for example, cable) medium, and filter to the desired frequency band again by filtering, down-conversion and perform frequency selection. Various embodiments rearrange the order of above-mentioned (and other) elements, remove some in these elements, and / or add other elements that perform similar or different functions. Adding element can include inserting element between existing elements, for example, inserting amplifier and analog-to-digital converter. In various embodiments, the RF part includes antenna.
[0038] In addition, the USB and / or HDMI terminals may include respective interface processors for connecting the system 100 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented, for example, within a separate input processing IC or within the processor 110 as desired. Similarly, various aspects of USB or HDMI interface processing may be implemented within a separate interface IC or within the processor 110 as desired. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 110 and the encoder / decoder 130, which operate in conjunction with memory and storage elements to process the data streams as desired for presentation on an output device.
[0039] The various components of system 100 may be disposed within an integrated housing where the various components may be interconnected and data may be transmitted therebetween using suitable connection means 115, such as an internal bus known in the art, including an I2C bus, wiring, and a printed circuit board.
[0040] The system 100 includes a communication interface 150 that can communicate with other devices via a communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to send and receive data through the communication channel 190. The communication interface 150 may include, but is not limited to, a modem or a network card, and the communication channel 190 may be implemented in, for example, a wired and / or wireless medium.
[0041] In various embodiments, a Wi-Fi network such as IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers) is used to stream data to the system 100. The Wi-Fi signal of these embodiments is received by a communication channel 190 and a communication interface 150 suitable for Wi-Fi communication. The communication channel 190 of these embodiments is usually connected to an access point or a router, which provides access to an external network including the Internet to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box to provide streaming data to the system 100, and the set-top box passes data through the HDMI connection of the input block 105. Other embodiments use the RF connection of the input block 105 to provide streaming data to the system 100. As described above, various embodiments provide data in a non-streaming mode. In addition, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.
[0042] The system 100 can provide output signals to various output devices, including a display 165, a speaker 175, and other peripherals 185. The display 165 of various embodiments includes, for example, one or more of a touch screen display, an organic light emitting diode (OLED) display, a curved display, and / or a foldable display. The display 165 can be used in a television, a tablet computer, a laptop computer, a cellular phone (mobile phone), or other devices. The display 165 can also be integrated with other components (for example, in a smart phone), or separate (for example, an external monitor for a laptop). In various embodiments, the other peripherals 185 include one or more of a stand-alone digital video disk (or digital versatile disk) (DVR, for both terms), a disk player, a stereo system, and / or a lighting system. Various embodiments use one or more peripherals 185 that provide functions based on the output of the system 100. For example, a disk player performs the function of playing the output of the system 100.
[0043] In various embodiments, control signals are transmitted between the system 100 and the display 165, speaker 175, or other peripheral device 185 using signaling such as AV.Link, CEC, or other communication protocols that enable device-to-device control with or without user intervention. The output devices can be communicatively coupled to the system 100 via dedicated connections through respective interfaces 160, 170, and 180. Alternatively, the output devices can be connected to the system 100 using a communication channel 190 via the communication interface 150. The display 165 and speaker 175 can be integrated into a single unit with other components of the system 100 in an electronic device (e.g., a television). In various embodiments, the display interface 160 includes a display driver, such as a timing controller (T Con) chip.
[0044] For example, if the RF portion of input 105 is part of a separate set-top box, the display 165 and speaker 175 may alternatively be separate from one or more of the other components. In various embodiments where the display 165 and speaker 175 are external components, the output signals may be provided via dedicated output connections, including, for example, an HDMI port, a USB port, or a COMP output.
[0045] The embodiments may be performed by computer software implemented by the processor 110 or by hardware or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 120 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as, as a non-limiting example, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. As a non-limiting example, the processor 110 may be of any type suitable for the technical environment and may include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.
[0046] Figure 2 An example of a block-based hybrid video encoder 200 is shown. Variations of the encoder 200 are contemplated, but for clarity, the encoder 200 is described below without describing all contemplated variations.
[0047] Before being encoded, the video sequence may undergo pre-encoding processing (201), for example, applying a color transform to an input color picture (e.g., conversion from RGB 4:4:4 to YCbCr4:2:0), or performing a remapping of input picture components in order to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of color components), or resizing the picture (e.g., downscaling). Metadata may be associated with the pre-processing and appended to the bitstream.
[0048] In encoder 200, a picture is encoded by an encoder element as described below. The picture to be encoded is divided (202) and processed in units such as CUs. Each unit is encoded using, for example, intra or inter mode. When a unit is encoded in intra mode, it performs intra prediction (260). In inter mode, motion estimation (275) and compensation (270) are performed. The encoder decides (205) which of intra mode or inter mode to use to encode the unit, and indicates the intra / inter decision by, for example, a prediction mode flag. The encoder can also mix (263) intra prediction results and inter prediction results, or mix results from different intra / inter prediction methods. For example, a prediction residual is calculated by subtracting (210) a prediction block from an original image block.
[0049] The motion refinement module (272) uses the already available reference pictures in order to refine the motion field of the block without referring to the original block. The motion field of a region can be considered as the set of motion vectors for all pixels in the region. If the motion vectors are sub-block based, then the motion field can also be represented as the set of motion vectors for all sub-blocks in the region (all pixels within a sub-block have the same motion vector, and the motion vector can vary from sub-block to sub-block). If a single motion vector is used for the region, then the motion field of the region can also be represented by the single motion vector (the same motion vector is used for all pixels in the region).
[0050] The prediction residual is then transformed (225) and quantized (230). The quantized transform coefficients, along with motion vectors and other syntax elements, are entropy encoded (245) to output a bitstream. The encoder may skip the transform and apply quantization directly to the untransformed residual signal. The encoder may bypass both the transform and quantization, i.e. the residual is directly encoded without applying the transform or quantization process.
[0051] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (255) to reconstruct the image block. An in-loop filter (265) is applied to the reconstructed picture to perform, for example, deblocking / SAO (sample adaptive offset) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (280).
[0052] Figure 3 A block diagram of a video decoder 300 is shown. In the decoder 300, the bitstream is decoded by the decoder elements as described below. The video decoder 300 generally performs the same Figure 2 The decoding process is the reverse of the encoding process described in . The encoder 200 also typically performs video decoding as part of encoding the video data.
[0053] In particular, the input to the decoder includes a video bitstream, which may be generated by the video encoder 200. The bitstream is first entropy decoded (330) to obtain transform coefficients, motion vectors, and other encoding information. Picture partition information indicates how the picture is partitioned. Thus, the decoder can partition (335) the picture according to the decoded picture partition information. The transform coefficients are dequantized (340) and inverse transformed (350) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (355) to reconstruct the image block.
[0054] The prediction block may be obtained (370) from an intra prediction (360) or a motion compensated prediction (i.e., inter prediction) (375). The decoder may blend (373) intra prediction results and inter prediction results, or blend results from multiple intra / inter prediction methods. Prior to motion compensation, the motion field may be refined (372) by using already available reference pictures. An in-loop filter (365) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (380).
[0055] The decoded picture may further undergo post-decoding processing (385), such as an inverse color transform (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping, which performs the inverse of the remapping process performed in the pre-encoding process (201), or resizing (e.g., upscaling) the reconstructed picture. The post-decoding processing may use metadata derived in the pre-encoding process and signaled in the bitstream.
[0056] In some embodiments, Figure 2 and Figure 3 Also shown is an encoder / decoder in which an encoder that uses a technique similar to VVC, such as that developed by JVET (Joint Video Development Team), or a HEVC or VVC standard is improved. The various methods and other aspects described in this application can be used to modify a module, such as Figure 2 The motion compensation module (270) of the video encoder 200 shown in FIG. Figure 3 The motion compensation module (375) of the video decoder 300 is shown. In addition, the present aspects are not limited to VVC or HEVC, and can be applied to, for example, other standards and recommendations, whether pre-existing or developed in the future, and extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise specified or technically excluded, the aspects described in this application can be used alone or in combination.
[0057] Inter-frame prediction is a coding tool in video compression that uses a reference picture of the video to predict the current block of the current picture for encoding / decoding. The encoder selects the best block in the reference frame after applying a motion model (e.g., translation or sub-block based motion wrapping). For example, the best block can be understood in the sense of rate / distortion for predicting the current block. In the following, the selected block is referred to as a reference block.
[0058] In current video codecs like VVC, there are two types of motion compensation (MC) filters available: an 8-tap filter based on DCT-IF and a smoother filter for 1 / 2 MC. In new video codec software, even larger MC filters (e.g., 12-tap filters) are used.
[0059] Some video codecs also propose several filters (such as normal, smooth and sharpen), possibly combined for horizontal and vertical directions.
[0060] In recent codecs, MC filters have become longer, improving the accuracy of reconstruction. However, in some cases, longer filters are undesirable: for example, when the filter takes samples from another moving area.
[0061] High-precision (1 / 16 pixel (pel)) motion compensation and motion vector storage
[0062] VVC increases MV accuracy to 1 / 16 luma sample to improve prediction efficiency for slow motion video. This higher motion accuracy is particularly helpful for video content with local changes and non-translational motion, such as in affine mode (warping). In order to generate fractional position samples with higher MV accuracy, HEVC's 8-tap luma interpolation filter and 4-tap chroma interpolation filter have been extended to 16 phases for luma and 32 phases for chroma. The extended filter group is applied to the MC process of inter-coded CUs (coding units) other than CUs in affine mode. For affine mode, a set of 6-tap luma interpolation filters with 16 phases is used to reduce computational complexity and memory bandwidth savings.
[0063] In VVC, the highest precision of the explicitly signaled motion vectors of non-affine CUs is one-quarter luma sample. In some inter prediction modes (such as affine mode), motion vectors can be signaled with 1 / 16 luma sample precision. In all inter-coded CUs with implicitly inferred MVs, MVs are derived with 1 / 16 luma sample precision, and motion compensated prediction is performed with 1 / 16 sample precision. In terms of internal motion field storage, all motion vectors are stored with 1 / 16 luma sample precision.
[0064] For temporal motion field storage used by TMVP (Temporal Motion Vector Prediction) and SbTMVP (Sub-block Temporal Motion Vector Prediction), motion field compression is performed at a granularity of 8×8 size, compared to 16×16 size in HEVC.
[0065] Affine Motion Compensated Prediction
[0066] In HEVC, only the translational motion model is applied to motion compensated prediction (MCP). In the real world, there are many kinds of motion, such as zooming in / out, rotation, perspective motion, and other irregular motions. In VVC, block-based affine transformation motion compensated prediction allows warping with motion compensation to be performed. Figure 4 As shown, the affine motion field of a block is described by the motion information of two control points (4-parameter affine model shown in the left part) or three control point motion vectors (6-parameter affine model shown in the right part).
[0067] For the 4-parameter affine motion model, the motion vector (mv) at the sample position (x, y) in the block (Cur) is x ,mv y ) is exported as:
[0068]
[0069] For the 6-parameter affine motion model, the motion vector at the sample position (x, y) in the block (Cur) is derived as:
[0070]
[0071] Where (mv 0x ,mv 0y ) is the upper left control point v 0 The motion vector, (mv 1x ,mv 1y ) is the upper right control point v 1 The motion vector of 2x ,mv 2y ) is the lower left control point v 2 is the motion vector of , and H and W are the block sizes (height and width).
[0072] To simplify motion compensated prediction, block-based affine transform prediction is applied. To derive the motion vector for each 4×4 luminance sub-block, such as Figure 5As shown, the motion vector of the center sample of each subblock is calculated according to the above equation and rounded to 1 / 16 fractional accuracy. The motion compensated interpolation filter is then applied to generate a prediction for each subblock using the derived motion vector. The subblock size of the chroma component is also set to 4×4. The MV of the 4×4 chroma subblock is calculated as the average of the MVs of the upper left and lower right luminance subblocks in the collocated 8×8 luminance region.
[0073] As done for translational motion inter prediction, there are also two affine motion inter prediction modes: affine merge mode and affine AMVP (Advanced Motion Vector Prediction) mode.
[0074] Sub-block based temporal motion vector prediction
[0075] VVC supports a sub-block based temporal motion vector prediction (SbTMVP) method. Similar to the temporal motion vector prediction (TMVP) in HEVC, SbTMVP uses the motion field in the collocated picture to improve the motion vector prediction and merge mode of the CU in the current picture. The same collocated image used by TMVP is used for SbTVMP. SbTMVP differs from TMVP in the following aspects. TMVP predicts motion at the CU level, but SbTMVP predicts motion at the sub-CU level. Although TMVP obtains the temporal motion vector from the collocated block in the collocated picture (the collocated block is the lower right or center block relative to the current CU), SbTMVP applies a motion offset before obtaining the temporal motion information from the collocated picture, where the motion offset is obtained from the motion vector from one of the spatially neighboring blocks of the current CU.
[0076] Figure 6 and Figure 7 The SbTMVP process is shown in FIG. SbTMVP predicts the motion vector of a sub-CU within the current CU in two steps. In the first step, check Figure 6 The spatial neighbor A1 in . If A1 has a motion vector that uses the collocated picture as its reference picture, then that motion vector is selected as the motion offset to be applied. If no such motion is identified, then the motion offset is set to (0, 0).
[0077] In the second step, the motion offset identified in the first step is applied (i.e., added to the coordinates of the current block) to Figure 7 The collocated picture shown obtains sub-CU level motion information (motion vector and reference index). Figure 7The example in assumes that the motion offset is set to the motion of block A1. Then, for each sub-CU, the motion information of its corresponding block in the collocated picture (the minimum motion grid covering the center sample) is used to derive the motion information of the sub-CU. After the motion information of the collocated sub-CU is identified, it is converted into a motion vector and reference index of the current sub-CU in a manner similar to the TMVP process of HEVC, where temporal motion scaling is applied to align the reference picture of the temporal motion vector with the reference picture of the current CU.
[0078] In VVC, a sub-block based merge list containing a combination of both SbTMVP candidates and affine merge candidates is used for signaling of sub-block based merge mode. The SbTMVP mode is enabled / disabled by a sequence parameter set (SPS) flag. If the SbTMVP mode is enabled, the SbTMVP predictor is added as the first entry in the list of sub-block based merge candidates, followed by the affine merge candidates. The size of the sub-block based merge list is signaled in the SPS, and the maximum allowed size of the sub-block based merge list in VVC is 5.
[0079] The sub-CU size used in SbTMVP is fixed to 8×8, and like the affine merge mode, the SbTMVP mode is only applicable to CUs with both width and height greater than or equal to 8.
[0080] The encoding logic for the additional SbTMVP merge candidates is the same as that for other merge candidates, ie, for each CU in a P or B slice, an additional RD check is performed to decide whether to use the SbTMVP candidate.
[0081] Interpolation filter in VVC
[0082] Table 1 below shows the 8-tap luma interpolation filter coefficients f for each 1 / 16 fractional sample position p L [p] specification.
[0083]
[0084] Table 1
[0085] Interpolation filter in ECM software
[0086] In the ECM software, the 8-tap interpolation filter used in VVC is replaced by a 12-tap filter. The interpolation filter is derived from a sinc function whose frequency response is cut off at the Nyquist frequency and clipped by a cosine window function. Table 2 below gives the filter coefficients for all 16 phases of the 12-tap interpolation filter. Figure 8 Comparing the frequency responses of the interpolation filter and the VVC interpolation filter, both are at half-pixel phase.
[0087]
[0088]
[0089] Table 2
[0090] According to one aspect of the present principles, there is provided a method for encoding or decoding video in which an adapted motion compensation filter is used for inter-frame prediction.
[0091] Fig. 9 An example of a method 900 for encoding a block of a video according to an embodiment is shown. The block will be encoded using inter-frame prediction, for example using a combination of Figure 2 The encoder described in the description. It is assumed here that a reference block has been identified for the current block to be encoded. The reference block is a block area in the reference picture that is used by the current block for encoding, and the block area in the reference picture is determined by the motion vector associated with the current block.
[0092] At 910, a motion compensation interpolation filter (MCIF) is determined for the current block. In some variants, when the MCIF is separable, the horizontal MCIF and the vertical MCIF are determined separately. Some embodiments for determining the MCIF of the current block are described below. In some embodiments, the MICF is adapted according to the current block discontinuity condition. At 920, a prediction block is determined based on the determined MCIF and a reference block from a reference picture. The reference block is interpolated using the determined MCIF to provide a prediction block. At 930, the current block is encoded using the prediction block.
[0093] Fig.10 An example of a method 1000 for decoding a block of a video according to an embodiment is shown. The block is encoded using inter-frame prediction and, for example, using a reference Figure 3 In a similar manner to the encoder, at 1010, a MCIF is determined for a block to be decoded. At 1020, a prediction block is determined based on the determined MCIF and a reference block, and at 1030, the prediction block is used to decode / reconstruct the current block.
[0094] According to the present principles, in order to improve inter-frame prediction, a motion compensated interpolation filter (MCIF) is adapted based on the discontinuity condition described below. Fig. 9 910 or Fig.10 At 1010, the length of the MCIF is adapted. For example, this can be done by selecting the MCIF from a set of filters with different lengths, such as an 8-tap filter for VVC and a 12-tap filter for ECM. However, other variations and filters are possible.
[0095] In other variants, the response of the filter is adapted, typically using a smoother alternative filter for sub - blocks at discontinuity positions.
[0096] Location - based selection
[0097] According to one embodiment, the selection of the MCIF is based on the position of the block in a region larger than the block. When the block is a sub - block of a larger block or coding unit (CU) with multiple sub - blocks, the selection of the MCIF is based on the position of the sub - block in the coding unit.
[0098] Fig.11 A 32x32 CU composed of 4×4 sub - blocks is shown. Assume that the sub - block is the unit entity for motion compensation. In this embodiment, the filter for motion compensation is adapted according to the sub - block position. For the central sub - block ( Fig.11 the white square on it), the default filter is used, such as an N - tap filter, where N is an integer. For sub - block "a", since vertical filtering obtains samples outside the CU, possibly in a region that does not belong to the same "object", the vertical filtering uses a shorter filter, such as an M - tap filter, where M is an integer and M < N. The horizontal filter is the default filter (N - tap filter). For the sub - blocks in "c", the situation is exactly the opposite: the horizontal filter is shorter (M - tap), and the vertical filter is the default filter (N - tap). For the sub - blocks in "b", both horizontal and vertical are shorter than the default.
[0099] For example, the default filter is the 12 - tap filter described above, and the shorter one is the 8 - tap filter. Other filters can be used.
[0100] In this embodiment, when the block is on the inner boundary of the coding unit, at least one of the horizontal MCIF or vertical MCIF has a shorter length than the MCIF used for the block when the block is inside the coding unit and not on the inner boundary of the coding unit (i.e., when the block is Fig.11 the white sub - block in it).
[0101] In Fig.11 the inner boundary has the width of one sub - block. In other variants, the inner boundary can be thicker and have a width of more than one sub - block. In this variant, the frequency band where the MCIF is shorter is more than 1 sub - block.
[0102] In another variant, several short filters are used, and when approaching the boundary of the block in one or two directions, the filter becomes shorter. In this variant, the length of at least one of the horizontal MCIF or vertical MCIF varies with the distance of the block from the center of the coding unit.
[0103] Motion - based filter selection
[0104] In another embodiment, the MCIF is selected based on a motion difference between the motion of the block and the motion of at least one of the horizontal and vertical neighbors of the block.
[0105] In this embodiment, the concept is generalized to any 4×4 sub-block based on the motion difference with its sub-block neighbors, such as Fig.12 shown.
[0106] The length of the filter is adapted according to the motion difference with its horizontal (respectively vertical) neighbors. When no motion is available, the motion difference is assumed to be zero (null).
[0107] Method 1200 is an example of a method for MCIF selection based on the motion difference between the motion of the block and the adjacent blocks. At 1210, the motion vector (ux, uy) of the current block is obtained. At 1220, the motion vector (lx, ly) of the block to the left of the block is obtained, and if available, the motion vector (rx, ry) of the block to the right of the block is obtained. At 1230, the difference between the motion vectors of the current block and its neighbors is compared with a threshold value th, for example as follows, if |ux-lx| is above a threshold, or if |ux-rx| is above a threshold, or if |uy-ly| is above a threshold, or if |uy-ry| is above a threshold, then at 1240, a short MCIF is selected for the horizontal direction. Otherwise, at 1250, a default filter is selected for the horizontal direction. The same logic can be applied to the vertical direction. At 1220, the motion vector (tx, ty) of the block at the top of the block and the motion vector (bx, by) of the block at the bottom of the block are obtained, if available. At 1270, the difference between the motion vectors is compared with a threshold value th. When the difference is above the threshold for at least one of the x or y coordinates, a short MCIF is selected for the vertical direction at 1280. Otherwise, a default filter is selected for the vertical direction at 1290.
[0108] When no motion is available, such as for right and bottom blocks, the difference is considered to be zero. In some cases, the motion of right and bottom sub-blocks is available for sub-blocks within a coding unit with affine or SbTMVP-like sub-block motion. For sub-blocks at the right or bottom boundary of a coding unit, the motion of the right and bottom neighbors is not available.
[0109] In a variant, for such sub-blocks (sub-blocks at the right or bottom boundary of the coding unit), the short filter is always selected.
[0110] In one variation, filter selection is done using neighboring reconstructed pixels of the current block for encoding or decoding. In this variation, a determination is made to determine whether an edge exists in the current block, for example by applying edge detection to the neighboring reconstructed pixels of the current block. When an edge is determined to exist, a filter shorter than a default filter is selected to filter the reference block.
[0111] In another variant, all filters are applied to the neighboring templates of the current block and the filter that allows the best reconstruction of the neighboring templates is selected for the current block.
[0112] For example, a filter is considered to be the best filter in terms of the quality reconstruction provided by the filter under consideration. Therefore, the selection of the filter in this variant is performed by selecting the filter that provides the minimum distortion between the pixels of the neighboring template of the current block and the pixels of the neighboring template of the reference block filtered by the filter under consideration. The neighboring template is for example a template of neighboring reconstructed pixels included in a strip to the left of the current block and in a strip at the top of the current block, such as for example a template from a template matching method.
[0113] In a variation, only sub-blocks at the boundary of a coding unit use the above embodiment. The boundary may have a width of one or more sub-blocks.
[0114] Mode-based selection
[0115] In another embodiment, the MCIF is selected based on the coding mode of at least one neighboring block of the block. In this variant, the filter selection is based on the mode of the neighboring (sub) blocks, e.g. Fig.13 As shown. The process starts at 1310. At 1320, the coding mode C0 of the left neighbor of the block and the coding mode C1 of the right neighbor of the block are obtained. At 1330, it is determined whether at least one of C0 or C1 is intra-coded. When at least one of C0 and C1 is intra-coded, at 1340, a short MCIF is selected for the horizontal direction. Otherwise, at 1350, a default filter is selected for the horizontal direction. The same logic can be applied to the vertical direction. At 1360, the coding mode C0 of the top neighbor of the block and the coding mode C1 of the bottom neighbor of the block are obtained. At 1370, it is determined whether at least one of C0 or C1 is intra-coded. When at least one of C0 and C1 is intra-coded, at 1380, a short MCIF is selected for the vertical direction. Otherwise, at 1390, a default filter is selected for the vertical direction.
[0116] In this embodiment, depending on the mode of the neighbors, the length of the filter is adapted.When no mode is available, the mode is assumed to be non-intra mode.
[0117] In a variant, other patterns are compared, such as the presence of residuals (non-zero transform coefficients) in neighboring subblocks. When residuals exist for left or right neighboring subblocks, a filter shorter than the default filter is selected for the horizontal direction. The same logic applies to the vertical direction with top and bottom neighbor blocks.
[0118] In a variation, only sub-blocks at block boundaries of a coding unit use the above embodiment.
[0119] In one variant, a combination of at least two of the above conditions (position, motion, mode) is used to determine the MCIF. In this variant, when at least one of the above conditions is true, the short filter is selected.
[0120] Deblocking filter condition selection
[0121]
[0122] Table 3: Derivation of deblocking filter strength for luma samples
[0123] Table 3 above shows the deblocking filter strength derivation for luma samples.
[0124] In this embodiment, the same logic is used again to select the MCIF for the block. The condition for selecting a filter shorter than the default filter can be inferred from at least one of the following: whether the neighboring block is intra-coded, or whether the neighboring block has a residual (i.e., non-zero transform coefficients), or the motion vectors of the current block and its neighbors are different, and the reference pictures of the current block and its neighbors are different.
[0125] When at least one condition is met, a short filter may be selected.
[0126] Affine model-based selection
[0127] According to another embodiment, when encoding a block in an affine model based coding mode, the MCIF is selected based on controlling the motion difference between motion vectors.
[0128] For affine inter prediction, the filter used for each subblock depends on the amplitude of the non-translational part of the motion. When the non-translational part has a large amplitude, the difference in motion between each subblock is large. Instead of testing the difference in motion between each subblock of an affine block, a test is performed on the entire block based on the motion model encoded in CPMV(v0, v1) for an affine 4-parameter model or (v0, v1, v2) for an affine 6-parameter model, as Fig.14 shown. Fig.14An example of a method 1400 for MCIF selection based on a 4-parameter affine model condition according to one embodiment is shown. At 1410 and 1420, the top left, respectively top right control motion vectors (v0x, v0y) of the block are obtained. At 1430, the magnitude between the control point motion vectors is checked against a threshold th. When the magnitude is above the threshold for at least one coordinate, then at 1440, a short MCIF is selected, otherwise at 1450, a default MCIF is used.
[0129] The same logic can also be applied to the control point motion vectors cpmvv2 when available for a 6-parameter affine motion model.
[0130] In one variation, the non-default MCIF selected at 1440 is a smoother filter rather than a shorter filter.
[0131] In a variation, only the sub-blocks on the boundary of the coding unit use the above embodiment.
[0132] Asymmetric MCIF
[0133] In another embodiment, the MCIF adapts the discontinuity location more specifically by considering not only the direction (horizontal or vertical) but also the left / right (or top / bottom) discontinuity.
[0134] In this embodiment, Fig. 9 910 or Fig.10 At 1010, a MCIF is determined for the block by creating a new MCIF from two filters having different lengths.
[0135] Phase 1 / 16 Filter Example
[0136] In the following, an example of an asymmetric filter structure using an 8-tap filter and a 12-tap filter is described. This example is for illustrative purposes of the present principles. Other filters with the same or other lengths may also be used.
[0137]
[0138] Table 4
[0139] Table 4 above gives examples of 8-tap (top original) and 12-tap (bottom original) filtering for phase 1 / 16 MCIF using the same scaling.
[0140] To create a filter with a shorter left part, two filters are cascaded, using the short filter for the left part and the long filter for the right part (see Fig.15 ). Adjust the center coefficient to obtain a unit filter (the sum of all coefficients should be 1, which is quantized to 256 here).
[0141] The same logic applies to creating Fig.16 The filter shown has a shorter right section.
[0142] Asymmetric filter use
[0143] exist Fig.17 , an example of a method 1700 for horizontal filter selection is provided.
[0144] At 1710, a motion vector (ux, uy) of the current block is obtained. At 1720, a motion vector (lx, ly) of a block to the left of the current block and a motion vector (rx, ry) of a block to the right of the current block are obtained. At 1730, an asymmetric filter is selected based on a condition.
[0145] This is provided in Table 5 below.
[0146] or |ux-lx|>th or |uy-|y|>th |ux-rx|>th or |uy-ry|>th long Fake Fake Long on the left, short on the right Fake real Short on the left, long on the right real Fake short real real
[0147] Table 5: Examples of asymmetric filter selection criteria
[0148] The conditions to be checked (|ux-lx|>th or |uy-1y|>th) are shown in the first raw data of Table 5. As shown in Table 5, when both conditions are false, a long filter or a default filter is selected for the module. When both conditions are true, a short filter is selected. Otherwise, an asymmetric filter is selected, and depending on the condition, either the left part or the right part is shorter than the other part.
[0149] By checking the top and bottom sub-blocks instead of the left and right, Fig.17 The same logic as shown in Table 5 applies to the vertical filter selection.
[0150] The same logic extends to other conditions for filter selection.
[0151] grammar
[0152] At CU / region / slice / picture / sequence level, a flag (sps_adapted_mcif) may be encoded to signal whether the adapted filter is used. It may also be inherited at CU level. The following table shows an example of syntax for signaling whether any one of the embodiments of the MICF that allows the use of adaptation is enabled.
[0153] Sequence Parameter Set RBSP Syntax
[0154]
[0155] Slice Header Syntax
[0156]
[0157] In the slice header of non-intra slices, a flag (sh_adapted_mcif) that controls the use of tools is signaled.
[0158] exist Fig.18 In the embodiment shown, in a transmission environment between two remote devices A and B on a communication network NET, device A includes a processor associated with a memory RAM and a ROM, which is configured to implement the following Figure 1-17 The method for encoding video is described, and device B includes a processor associated with memory RAM and ROM, which is configured to implement as follows Figure 1-17 The method for decoding video.
[0159] According to one example, the network is a broadcast network adapted to broadcast / transmit encoded data representing a video from device A to decoding devices including device b.
[0160] The signal intended to be transmitted by device A carries at least one bitstream comprising coded data representing a video. The bitstream may be generated from any of the embodiments of the present principles.
[0161] Fig.19 An example of the syntax of such a signal transmitted via a packet-based transport protocol is shown. Each transmitted packet P comprises a header H and a payload PAYLOAD. In some embodiments, the payload PAYLOAD may comprise encoded video data encoded according to any of the above-described embodiments.
[0162] Various implementations involve decoding. "Decoding" as used in this application may include, for example, all or part of a process performed on a received coded sequence to produce a final output suitable for display. In various embodiments, such a process includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such a process also or alternatively includes a process performed by a decoder of the various implementations described in this application, for example, decoding resampling filter coefficients, resampling decoded pictures.
[0163] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding, and in another embodiment, "decoding" refers to the entire reconstructed picture process including entropy decoding. Based on the context of the specific description, whether the phrase "decoding process" is intended to refer specifically to a subset of operations or to a broader decoding process will be clear and is considered to be well understood by those skilled in the art.
[0164] Various implementations involve encoding. In a manner similar to the discussion above about "decoding", "encoding" as used in this application may include, for example, all or part of a process performed on an input video sequence to produce an encoded bitstream. In various embodiments, such a process includes one or more processes typically performed by an encoder, such as partitioning, differential encoding, transforming, quantization, and entropy encoding. In various embodiments, such a process also or alternatively includes a process performed by an encoder of the various implementations described in this application, such as determining resampling filter coefficients, resampling decoded pictures.
[0165] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. Based on the context of the specific description, whether the phrase "encoding process" is intended to refer specifically to a subset of operations or to a broader encoding process will be clear and is considered to be well understood by those skilled in the art.
[0166] Note that the syntax elements used here are descriptive terms. Therefore, they do not exclude the use of other syntax element names.
[0167] This disclosure has described various information that can be transmitted or stored, such as, for example, syntax. The information can be packaged or arranged in various ways, including, for example, ways that are common in video standards, such as putting the information in an SPS, PPS, NAL unit, header (e.g., a NAL unit header or a slice header), or SEI message. Other ways are also available, including, for example, ways that are common to system-level or application-level standards, such as putting the information in one or more of the following:
[0168] a. SDP (Session Description Protocol), a format for describing multimedia communication sessions, used for the purpose of session announcement and session invitation, such as described in RFC, and used in conjunction with RTP (Real-time Transport Protocol) transport.
[0169] b. DASH MPD (Media Presentation Description) descriptor, used, for example, in DASH and transmitted over HTTP, a descriptor is associated with a representation or a set of representations to provide additional features to the content representation.
[0170] c. RTP header extensions, such as used during RTP streaming.
[0171] d. ISO Base Media File Format, such as that used by OMAF, and uses object-oriented building blocks defined by a unique type identifier and a length, also referred to as "atoms" in some specifications.
[0172] e. HLS (HTTP Live Streaming) manifests transmitted over HTTP. A manifest may be associated with, for example, a version or set of versions of content to provide characteristics of the version or set of versions.
[0173] When a figure is represented as a flow chart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow chart of the corresponding method / process.
[0174] Some embodiments relate to rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between rate and distortion is usually considered, usually given a constraint on computational complexity. Rate-distortion optimization is usually formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are different approaches to solving the rate-distortion optimization problem. For example, the method can be based on extensive testing of all coding options, including all considered modes or coding parameter values, using a complete evaluation of their coding costs and the associated distortion of the reconstructed signal after encoding and decoding. Faster methods can also be used to save coding complexity, in particular, calculating approximate distortion based on a prediction or prediction residual signal rather than a reconstructed signal. A mixture of these two methods can also be used, such as by using approximate distortion only for some possible coding options and full distortion for other coding options. Other methods only evaluate a subset of possible coding options. More generally, many methods employ any of a variety of techniques to perform optimization, but optimization is not necessarily a complete evaluation of coding costs and associated distortion.
[0175] The implementations and aspects described herein can be implemented in, for example, methods or processes, devices, software programs, data streams, or signals. Even if discussed only in the context of a single implementation (e.g., discussed only as a method), the implementation of the features discussed can also be implemented in other forms (e.g., devices or programs). Devices can be implemented in, for example, appropriate hardware, software, and firmware. Methods can be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, cellular phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate information communication between end users.
[0176] Reference to "one embodiment" or "an embodiment" or "an implementation" or "an implementation" and other variations thereof mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in an implementation" or "in an implementation", as well as any other variations appearing in various places throughout the application, are not necessarily all referring to the same embodiment.
[0177] Furthermore, the present application may refer to "determining" various pieces of information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.
[0178] Furthermore, the present application may refer to "accessing" various pieces of information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.
[0179] Furthermore, the present application may involve "receiving" various pieces of information. Like "accessing," receiving is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, during operations such as, for example, storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "receiving" is generally involved in one way or another.
[0180] It should be understood that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," any of the following uses of " / ," "and / or," and "at least one of" are intended to include selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to include selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). As will be apparent to one of ordinary skill in this and related arts, this can be extended to many of the items listed.
[0181] In addition, as used herein, the word "signal" particularly relates to indicating something to a corresponding decoder. For example, in some embodiments, an encoder signals a specific one of a plurality of resampled filter coefficients. Thus, in one embodiment, the same parameters are used on both the encoder side and the decoder side. Thus, for example, the encoder can send (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. On the contrary, if the decoder already has specific parameters as well as other parameters, signaling can be used without sending (implicit signaling) to simply allow the decoder to know and select specific parameters. By avoiding the transmission of any actual function, bit saving is achieved in various embodiments. It should be understood that signaling can be implemented in a variety of ways. For example, in various embodiments, one or more grammatical elements, flags, etc. are used to signal information to the corresponding decoder. Although the verb form of the word "signal" is mentioned above, the word "signal" can also be used as a noun here.
[0182] It will be apparent to one of ordinary skill in the art that implementations may generate a variety of signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for executing the method, or data generated by one of the described implementations. For example, a signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is well known, a signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor readable medium.
[0183] A number of embodiments are described herein. Features of these embodiments may be provided individually or in any combination across various claim categories and types. In addition, embodiments may include one or more of the following features, devices, or aspects, individually or in any combination across various claim categories and types:
[0184] • Encode / decode the video, which includes adapting the motion compensation interpolation filter based on the discontinuity condition of the block.
[0185] • Encoding / decoding a block of a video, which includes adapting a motion compensated interpolation filter based on the position of the block in the coding unit.
[0186] • Encoding / decoding a block of video, which includes adapting a motion compensated interpolation filter based on the motion difference between the block and its neighboring blocks.
[0187] • Encoding / decoding a block of video including adapting a motion compensated interpolation filter based on the coding modes of the block's neighbors.
[0188] • Encoding / decoding a block of video, which includes adapting a motion compensated interpolation filter based on neighboring reconstructed pixels of the block.
[0189] • Encoding / decoding a block of a video, comprising adapting a motion compensated interpolation filter based on controlling motion differences between motion vectors when encoding the block in a coding mode based on an affine model.
[0190] • Encoding / decoding a block of video including creating an asymmetric motion compensated interpolation filter based on a discontinuity condition for the block.
[0191] • Encoding / decoding a block of video which includes creating an asymmetric motion compensated interpolation filter from two filters with different lengths.
[0192] • Encoding / decoding a video, comprising adapting a motion compensated interpolation filter based on a discontinuity condition of the block in response to determining that the block is at a boundary of a coding unit.
[0193] Encoding / decoding a video, comprising, in response to determining that the block is at a boundary of a coding unit, creating an asymmetric motion compensated interpolation filter based on a discontinuity condition of the block.
[0194] • Encode / decode the video including signaling information to enable / disable the adaptive motion compensation interpolation filter.
[0195] • A bitstream or signal comprising one or more of said syntax elements or variants thereof.
[0196] • A bitstream or signal including syntax conveying information generated according to any of the described embodiments.
[0197] • Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal comprising one or more of said syntax elements or variations thereof.
[0198] • Creating and / or transmitting and / or receiving and / or decoding according to any of the described embodiments.
[0199] • A method, process, apparatus, medium storing instructions, medium storing data, or signal according to any of the described embodiments.
[0200] • A TV, set-top box, cell phone, tablet, or other electronic device that performs decoding of video according to any of the described embodiments.
[0201] • A TV, set-top box, cell phone, tablet, or other electronic device that performs decoding of the video according to any of the described embodiments, and displays (e.g., using a monitor, screen, or other type of display) the resulting video.
[0202] A TV, set-top box, cell phone, tablet, or other electronic device that selects (e.g., using a tuner) a channel to receive a signal including encoded video, and performs decoding of the video according to any of the described embodiments.
[0203] A TV, set-top box, cell phone, tablet, or other electronic device that receives over the air (e.g., using an antenna) a signal including the encoded video and performs decoding of the video according to any of the described embodiments.
Claims
1. A method comprising decoding a block of a video, comprising: determining at least one motion compensated interpolation filter for the block, wherein a length of the motion compensated interpolation filter is adapted based on a condition on the block, determining a prediction block based on the at least one motion compensated interpolation filter and a reference block, The block is decoded based at least on the prediction block.
2. An apparatus comprising one or more processors, wherein the one or more processors are configured to decode a block of a video, wherein decoding the block comprises: determining at least one motion compensated interpolation filter for the block, wherein a length of the motion compensated interpolation filter is adapted based on a condition on the block, determining a prediction block based on the at least one motion compensated interpolation filter and a reference block, The block is decoded based at least on the prediction block.
3. A method comprising encoding a block of a video, comprising: determining at least one motion compensated interpolation filter for the block, wherein a length of the motion compensated interpolation filter is adapted based on a condition on the block, determining a prediction block based on the at least one motion compensated interpolation filter and a reference block, The block is encoded based at least on the prediction block.
4. An apparatus comprising one or more processors, wherein the one or more processors are configured to encode a block of a video, wherein encoding the block comprises: determining at least one motion compensated interpolation filter for the block, wherein a length of the motion compensated interpolation filter is adapted based on a condition on the block, determining a prediction block based on the at least one motion compensated interpolation filter and a reference block, The block is encoded based at least on the prediction block.
5. The method according to any one of claims 1 or 3, or the device according to any one of claims 2 or 4, wherein: Determining at least one motion compensated interpolation filter for the block includes selecting the at least one motion compensated interpolation filter from a set of motion compensated interpolation filters having different lengths.
6. The method or apparatus of claim 5, wherein selecting the at least one motion compensated interpolation filter is based on a position of the block in a region larger than the block.
7. The method or apparatus of claim 5, wherein the block is a subblock in a coding unit having a plurality of subblocks, and selecting the at least one motion compensated interpolation filter is based on a position of the block in the coding unit.
8. The method or device according to any one of claims 5 to 7, wherein: Selecting the at least one motion compensated interpolation filter comprises: selecting a horizontal motion compensation interpolation filter for filtering the reference block in the horizontal direction, and A vertical motion compensation interpolation filter is selected for filtering the reference block in the vertical direction.
9. The method or apparatus according to claim 8, wherein: When the block is located at an inner boundary of the coding unit, a length of at least one of a horizontal motion compensation interpolation filter or a vertical motion compensation interpolation filter is shorter than a length of a motion compensation interpolation filter selected for the block when the block is within the coding unit but not at the inner boundary of the coding unit.
10. The method or apparatus according to claim 9, wherein: When the block is not at an inner boundary of the coding unit, a motion compensation interpolation filter selected for the block is a 12-tap filter.
11. The method or apparatus according to claim 9 or 10, when the block is located at an inner boundary of the coding unit, at least one of the horizontal motion compensation interpolation filter or the vertical motion compensation interpolation filter selected for the block is an 8-tap filter.
12. The method or apparatus according to any one of claims 9 to 11, wherein: The inner boundary of the coding unit is a band having a width of at least one sub-block.
13. The method or apparatus according to any one of claims 8 to 11, wherein: The block is a subblock of a coding unit having a plurality of subblocks, and a length of at least one of a horizontal motion compensation interpolation filter or a vertical motion compensation interpolation filter varies with a distance of the block from a center of the coding unit.
14. The method or apparatus according to claim 13, wherein: The closer the block is to the center of the coding unit, the shorter the length of the motion compensation interpolation filter.
15. The method or apparatus according to any one of claims 5 to 14, wherein: Selecting the motion compensated interpolation filter is based on a motion difference between the motion of the block and the motion of at least one of the horizontal and vertical neighbors of the block.
16. The method or apparatus of any one of claims 8-15, wherein a horizontal motion compensation interpolation filter selected for filtering a reference block in a horizontal direction is based on a motion difference between a motion of the block and a motion of at least one of a left or right neighboring block of the block, and wherein a vertical motion compensation interpolation filter selected for filtering a reference block in a vertical direction is based on a motion difference between a motion of the block and a motion of at least one of a top or bottom neighboring block of the block.
17. The method or apparatus of any one of claims 5-15, wherein the motion compensation interpolation filter is selected based on neighboring reconstructed pixels of the block.
18. The method or apparatus of claim 17, wherein in response to determining that an edge exists in the block, a motion compensation interpolation filter having a shorter length than a default motion compensation interpolation filter is selected.
19. The method or apparatus of claim 17, wherein all filters in the set are applied to neighboring templates of the block, and a motion compensated interpolation filter that provides minimum distortion for the neighboring templates is selected for the block.
20. The method or apparatus of any one of claims 5-19, wherein selecting the motion compensated interpolation filter is based on an encoding mode of at least one neighboring block of the block.
21. A method or apparatus according to any one of claims 8-20, wherein the horizontal motion compensation interpolation filter selected for filtering a reference block in a horizontal direction is based on whether at least one of the left or right neighboring blocks of the block is intra-frame coded, and wherein the vertical motion compensation interpolation filter selected for filtering the reference block in a vertical direction is based on whether at least one of the top or bottom neighboring blocks of the block is intra-frame coded.
22. The method or apparatus according to any one of claims 5 to 21, wherein: The motion compensated interpolation filter is selected following the same logic as the deblocking filter logic.
23. The method or apparatus according to any one of claims 5 to 22, wherein: When the block is encoded in an affine model-based encoding mode, the motion compensation interpolation filter is selected based on controlling the motion difference between motion vectors.
24. The method according to any one of claims 1, 3 or 5-23, or the apparatus according to any one of claims 2 or 4-23, wherein: Determining at least one motion compensated interpolation filter for the block includes creating a new motion compensated interpolation filter from at least two filters having different lengths.
25. The method or apparatus of claim 24, wherein: The new motion compensated interpolation filter is an asymmetric filter.
26. The method or apparatus of claim 25, wherein: One of the left portion or the right portion of the asymmetric filter is shorter than the right portion or the left portion.
27. The method or apparatus according to any one of claims 24 to 26, wherein: A new motion compensated interpolation filter is created by cascading at least two filters.
28. The method or apparatus of claim 27, wherein the center coefficients are adjusted to obtain a unit filter.
29. The method of any one of claims 1, 3, or 5-28, or the apparatus of any one of claims 2, or 4-28, wherein adapting the motion compensated interpolation filter or selecting the motion compensated interpolation filter or creating the new motion compensated interpolation filter is responsive to determining that the block is a sub-block of a coding unit and the block is located at an inner boundary of the coding unit.
30. A method according to any one of claims 1, 3 or 5-28, or an apparatus according to any one of claims 2 or 4-28, wherein the method further comprises, or the one or more processors are further configured to decode or encode information enabling or disabling the use of an adapted motion compensation interpolation filter determined according to any one of claims 5-29.
31. A bitstream comprising encoded data representing a video, formed by performing the method of any one of claims 3 or 5-30.
32. A bitstream according to claim 31, further comprising information enabling or disabling the use of an adapted motion compensated interpolation filter determined according to any one of claims 5-29.
33. A computer readable medium comprising a bitstream according to claim 31 or 32.
34. A computer-readable storage medium having stored thereon instructions for causing one or more processors to perform the method of any one of claims 1, 3, or 5-30.
35. A computer program product comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1, 3 or 5-30.
36. An apparatus comprising: The device according to any one of claims 2 or 5-30; as well as At least one of: (i) an antenna configured to receive a signal, the signal including data representing a video, (ii) a band limiter configured to limit the received signal to a frequency band including the data representing the video, or (iii) a display configured to display the decoded video.
37. The device of claim 36, comprising a TV, a cell phone, a tablet or a set-top box.
38. An apparatus comprising: an access unit configured to access data comprising a signal storing a bit stream according to claim 31 or 32, A transmitter is configured to transmit the accessed data.
39. A method comprising accessing data comprising a signal storing a bit stream according to claim 31 or 32, and transmitting the accessed data.
