METHOD AND APPARATUS FOR DECODING VIDEO DATA AND RELATED MEMORY

Abstract

A video decoder is configured by a bidirectional inter-predictive block, using a first mv, to locate a first predictive block in a first reference image; locate a second predictive block in a second reference image using a second mv; determine a first bidirectional optical flow (bio) momentum quantity for a first sub-block of the first predictive block; determine a first final predictive sub-block for the video data block based on the first bio momentum quantity; determine a second bio momentum quantity for a second sub-block of the first predictive block; determine a second final predictive sub-block for the video data block based on the second bio momentum quantity; and based on the first final predictive sub-block and the second final predictive sub-block, determine a predictive block.

Description

1 / 78 METHOD AND APPARATUS FOR DECODING VIDEO DATA AND RELATED MEMORY

[0001] This application claims the benefit of: U.S. Provisional Patent Application No. 62 / 442,357, filed January 4, 2017; and U.S. Provisional Patent Application No. 62 / 445,152, filed January 11, 2017, the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] This revelation refers to video encoding. BACKGROUND

[0003] Digital video capabilities can be incorporated into a wide range of equipment, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording equipment, digital media players, video game equipment, video game consoles, mobile phones or satellite phones, so-called “smartphones,” video teleconferencing equipment, streaming video equipment, and the like. Digital video equipment implements video compression techniques, such as those described in the standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265 High Efficiency Video Coding (HEVC), and extensions of such standards. Video equipment can transmit, Petition 870250009304, dated 04 / 02 / 2025, page 13 / 21 2 / 78 to receive, encode, decode and / or store digital video information more effectively by implementing such video encoding techniques.

[0004] Video coding techniques include spatial (intra-image) and / or temporal (inter-image) prediction to reduce or remove the redundancy inherent in video sequences. For block-based video coding, a video slice (i.e., a video frame or a portion of a video frame) can be partitioned into video blocks, and may also be referred to as a block tree, coding units (CUs), and / or coding nodes. The video blocks in an intra-coded slice (I) of an image are coded using spatial prediction with respect to reference samples in neighboring blocks in the same image. Video blocks in an intercoded slice (P or B) of an image can utilize spatial prediction with respect to reference samples in neighboring blocks within the same image, or temporal prediction with respect to reference samples in other reference images. Images can be referred to as frames, and reference images can be referred to as reference frames.

[0005] Spatial or temporal prediction results in a predictive block for a block to be encoded. Residual data represents pixel differences between the original block to be encoded and the predictive block. An inter-encoded block is encoded according to a motion vector that indicates a block of reference samples that form the predictive block, and with residual data that indicates the difference between the encoded block and the block. Petition 870200158161, dated 12 / 17 / 2020, page 13 / 115 3 / 78 predictive. An intra-coded block is encoded according to an intra-coding mode and residual data. For further compression, the residual data can be transformed from the pixel domain to the transform domain, resulting in residual coefficients, which can then be quantized. The quantized transform coefficients, initially arranged in a two-dimensional array, can be scanned to produce a one-dimensional vector of transform coefficients, and entropy coding can be applied to achieve even greater compression. SUMMARY

[0006] In general, this disclosure describes techniques related to bidirectional optical flow (BIO) in video coding. The techniques in this disclosure can be used in conjunction with existing video codecs, such as High Efficiency Video Coding (HEVC), or be an efficient coding tool for future video coding standards.

[0007] According to an example of this disclosure, a method of decoding video data includes determining a block of encoded video data using a bidirectional interprediction mode; determining a first motion vector (MV) for the block, in which the first MV points to a first reference image; determining a second MV for the block, in which the second MV points to a second reference image, the first reference image being different from the second reference image; using the first MV, which locates a first predictive block in Petition 870200158161, dated 12 / 17 / 2020, p. 14 / 115 4 / 78 first reference image; use the second MV, which locates a second predictive block in the second reference image; determine a first bidirectional optical flow (BIO) momentum for a first subblock of the first predictive block; determine a first final predictive subblock for the video data block based on the first subblock of the first predictive block, a first subblock of the second predictive block, and the first BIO momentum; determine a second BIO momentum for a second subblock of the first predictive block; determine a second final predictive subblock for the video data block based on the second subblock of the first predictive block, a second subblock of the second predictive block, and the second BIO momentum; determine a final predictive block for the video data block based on the first final predictive subblock and the second final predictive subblock;and output a video data image comprising a decoded version of the video data block.

[0008] According to another example of this disclosure, a device for decoding video data includes a memory configured to store video data; and one or more processors configured to determine that a block of video data is encoded using a bidirectional interprediction mode; determine a first motion vector (MV) for the block, wherein the first MV points to a first reference image; determine a second MV for the block, wherein the second MV points to a second reference image, the first reference image being the first reference image. Petition 870200158161, dated 12 / 17 / 2020, page 15 / 115 5 / 78 different from the second reference image; locate a first predictive block in the first reference image using the first MV; locate a second predictive block in the second reference image using the second MV; determine a first bidirectional optical flow (BIO) momentum for a first subblock of the first predictive block; determine a first final predictive subblock for the video data block based on the first subblock of the first predictive block, a first subblock of the second predictive block, and the first BIO momentum; determine a second BIO momentum for a second subblock of the first predictive block; determine a second final predictive subblock for the video data block based on the second subblock of the first predictive block, a second subblock of the second predictive block, and the second BIO momentum;Determine a final predictive block for the video data block based on the first final predictive sub-block and the second final predictive sub-block; and output a video data image comprising a decoded version of the video data block.

[0009] According to another example of this revelation, a computer-readable storage medium stores instructions that, when executed by one or more processors, cause one or more processors to determine that a block of video data be encoded using a bidirectional interpretation mode; determine a first motion vector (MV) for the block, in which the first MV points to a first reference image; determine a second MV for the Petition 870200158161, dated 12 / 17 / 2020, page 16 / 115 6 / 78 block, in which the second MV points to a second reference image, the first reference image being different from the second reference image; locate a first predictive block in the first reference image using the first MV; locate a second predictive block in the second reference image using the second MV; determine a first bidirectional optical flow (BIO) momentum for a first subblock of the first predictive block; determine a first final predictive subblock for the video data block based on the first subblock of the first predictive block, a first subblock of the second predictive block, and the first BIO momentum; determine a second BIO momentum for a second subblock of the first predictive block;Determine a second final predictive sub-block for the video data block based on the second sub-block of the first predictive block, a second sub-block of the second predictive block, and the second BIO motion quantity; determine a final predictive block for the video data block based on the first final predictive sub-block and the second final predictive sub-block; and output a video data image comprising a decoded version of the video data block.

[0010] According to another example of this disclosure, an apparatus for decoding video data includes a device for determining a block of video data that is encoded using a bidirectional interprediction mode; a device for determining a first motion vector (MV) for the block, in which the first MV points to a first reference image; a Petition 870200158161, dated 12 / 17 / 2020, page 17 / 115 7 / 78 device for determining a second MV for the block, wherein the second MV points to a second reference image, the first reference image being different from the second reference image; a device for locating a first predictive block in the first reference image using the first MV; a device for locating a second predictive block in the second reference image using the second MV; a device for determining a first bidirectional optical flow (BIO) quantity of motion for a first sub-block of the first predictive block; a device for determining a first final predictive sub-block for the video data block based on the first sub-block of the first predictive block, a first sub-block of the second predictive block, and the first BIO quantity of motion; a device for determining a second BIO quantity of motion for a second sub-block of the first predictive block;A device for determining a second final predictive sub-block for the video data block based on the second sub-block of the first predictive block, a second sub-block of the second predictive block, and the second BIO motion quantity; a device for determining a final predictive block for the video data block based on the first final predictive sub-block and the second final predictive sub-block; and a device for outputting a video data image comprising a decoded version of the video data block.

[0011] Details of one or more aspects of the techniques are presented in the attached drawings and in the description that follows. Other resources, objects and Petition 870200158161, dated 12 / 17 / 2020, page 18 / 115 8 / 78 The advantages of these techniques will be evident from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram illustrating an exemplary video encoding and decoding system that can utilize techniques for bidirectional optical flow.

[0013] Figure 2 is a conceptual diagram illustrating an example of one-sided motion estimation (ME), such as a block matching algorithm (BMA) running for motion-compensated frame rate upconversion (MC-FRUC).

[0014] Figure 3 is a conceptual diagram illustrating an example of bilateral ME, such as a BMA performed for MC-FRUC.

[0015] Figure 4A shows MV spatial neighbor candidates for the merge mode.

[0016] Figure 4B shows neighboring candidates Spatial MV for AMVP modes. Example of a

[0017] Figure 5A shows a TMVP candidate.

[0018] Figure 5B shows an example of MV scaling.

[0019] Figure 6 shows an example of optical flow path.

[0020] Figure 7 shows an example of BIO for an 8x4 block.

[0021] Figure 8 shows an example of modified BIO for an 8x4 block.

[0022] Figures 9A and 9B show illustrations Petition 870200158161, dated 12 / 17 / 2020, page 19 / 115 9 / 78 examples of sub-blocks where OBMC applies.

[0023] Figures 10A-10D show examples of OBMC weightings.

[0024] Figure 11 shows an example for the BIO proposed for an 8x4 block.

[0025] Figures 12A-12D show examples of Simplified BIO proposed at OBMC.

[0026] Figure 13 shows an example of a weighting function for a 4x4 sub-block with a 5x5 window.

[0027] Figure 14 is a block diagram illustrating an example of a video encoder.

[0028] Figure 15 is a block diagram illustrating an example of a video decoder that can implement techniques for bidirectional optical streaming.

[0029] Figure 16 is a flowchart that illustrates an example of the operation of a video decoder, according to a technique of this disclosure. DETAILED DESCRIPTION

[0030] In general, the techniques of this revelation are related to improvements in bidirectional optical flow (BIO) video coding techniques. BIO can be applied during motion compensation. As originally proposed, BIO is used to modify predictive sample values ​​for bi-predictive intercoded blocks based on an optical flow path in order to determine better predictive blocks, such as predictive blocks closer to an original video data block. The various techniques of this revelation can be applied, alone or in any combination, to determine when Petition 870200158161, dated 12 / 17 / 2020, page 20 / 115 10 / 78 and whether BIO should be run when predicting blocks of video data, for example, during motion compensation.

[0031] As used in this disclosure, the term video encoding refers generically to video encoding or video decoding. Similarly, the term video encoder can generically refer to a video encoder or a video decoder. Furthermore, certain techniques described in this disclosure relating to video decoding may also apply to video encoding, and vice versa. For example, video encoders and video decoders are often configured to perform the same process or reciprocal processes. In addition, video encoders typically perform video decoding as part of the processes for determining how to encode video data. Therefore, unless explicitly stated otherwise, it should not be assumed that a technique described relating to video decoding cannot also be performed by a video encoder, or vice versa.

[0032] This disclosure may also use terms such as current layer, current block, current image, current slice, etc. In the context of this disclosure, the term current is intended to identify a block, figure, slice, etc., that is currently encoded, as opposed to, for example, previously or already encoded blocks, images and slices, or even already encoded blocks, images and slices.

[0033] In general, an image is divided into blocks, each of which can be encoded. Petition 870200158161, dated 12 / 17 / 2020, page 21 / 115 11 / 78 predictively. A video encoder can predict a current block using prediction techniques (using data from the image that includes the current block), prediction techniques (using data from an image previously encoded relative to the image that includes the current block), or other techniques such as intra-block copy, palette mode, dictionary mode, etc. Intraprediction includes both one-way and two-way prediction.

[0034] For each inter-predicted block, a video encoder can determine a motion information set. The motion information set can contain motion information for forward and backward prediction directions. Here, forward and backward prediction directions are two prediction directions of a bidirectional prediction mode. The terms “forward” and “backward” do not have a geometric meaning. Instead, the terms generally correspond to whether reference images should be displayed before (“backward”) or after (“forward”) the current image. In some examples, the “forward” and “backward” prediction directions might correspond to reference image list 0 (RefPicList0) and reference image list 1 (RefPicList1) of a current image.When only one list of reference images is available for an image or slice, only RefPicList0 is available, and the movement information for each block of a slice always refers to an image from RefPicList0 (for example, it's forward).

[0035] In some cases, a motion vector Petition 870200158161, dated 12 / 17 / 2020, page 22 / 115 12 / 78 together with a corresponding reference index can be used in a decoding process. Such a motion vector with and associated with the reference index is denoted as a unipredictive motion information set.

[0036] For each prediction direction, the motion information contains a reference index and a motion vector. In some cases, for simplicity, a motion vector itself may be referenced in a way that assumes the motion vector has an associated reference index. A reference index can be used to identify a reference image in the current reference image list (RefPicList0 or RefPicList1). A motion vector has a horizontal (x) and a vertical (y) component. In general, the horizontal component indicates a horizontal displacement within a reference image, relative to the position of a current block in a current image, needed to locate an x-coordinate of a reference block, while the vertical component indicates a vertical displacement within the reference image, relative to the position of the current block, needed to locate a y-coordinate of the reference block.

[0037] Picture order count (POC) values ​​are widely used in video coding standards to identify the display order of an image. While there are cases where two images within an encoded video sequence may have the same POC value, this is not usually the case within an encoded video sequence. Thus, POC values Petition 870200158161, dated 12 / 17 / 2020, page 23 / 115 13 / 78 of the images are generally unique and therefore can uniquely identify the corresponding images. When multiple encoded video sequences are present in a bitstream, images that have the same POC value may be closer to each other in terms of decoding order. The POC values ​​of the images are typically used for constructing the reference image list, deriving reference image sets, such as in HEVC, and motion vector scaling.

[0038] E. Alshina, A. Alshina, J.-HK Choi, A. Saxena, M. Budagavi, Performance investigation of known tools for next-generation video coding, ITU - Telecommunications Standardization Sector, STUDY GROUP 16, Question 6, Video Coding Expert Group (VCEG), VCEG-AZO5, June 2015, Warsaw, Poland (hereinafter, Alshina 1), and A. Alshina, E. Alshina, T. Lee, Bidirectional optical flow to improve motion compensation, Video Coding Symposium (PCS), Nagoya, Japan, 2010 (hereinafter, Alshina 2) described a method called bidirectional optical flow (BIO). BIO is based on pixel-level optical flow. According to Alshina 1 and Alshina 2, BIO is applied only to blocks that have forward and backward prediction. BIO, as described in Alshina 1 and Alshina 2, is summarized below:

[0039] Given a pixel value It at time t, the first-order Taylor expansion of the pixel value is / t= / to+^(í-tO) (A) Petition 870200158161, dated 12 / 17 / 2020, page 24 / 115 14 / 78

[0040] Ito is on the path of movement of It. That is, the movement of Ito to It is considered in the formula.

[0041] Under the assumption of optical flow: DLDY dy_

[0042] Regarding e, depending on the speed of movement, Vxo and Vyo can be used to represent them.

[0043] Then, equation (B) becomes — ho bx01 / ^.Q · (t t0) Gy0· l / y0· (t t0) (C)

[0044] Assume a forward reference at to and a backward reference at ti, and that t0 - t = t - ti = At ​​= 1

[0045] So: ^tO bXQ ' Vj.0 (tt(j) byQ · VyQ · (t tg) / fQ + GXQ ' Vx04“ GyO ' VyQ It=Λι—GX1VX1(t — tx) — GylVyl(t — = ltl— GX1VX1— Gyl Vyl It= + / tl , (^O'tvQ —^11 Ά1 •^yo~^yi'^yi) (D)

[0046] Vxo = Vxi = Vxe Vyo = Vy is also assumed, provided the motion is along the trajectory. Then equation (D) would become Petition 870200158161, dated 12 / 17 / 2020, p. 25 / 115 15 / 78 ho+!ti I tcxo 0.1)hc+(Gyo ^yl)'^y _ ^to+tl । AQc'tx+AGy Vy 2 2 ~ 2 2 (E) where AGX= Gxo ~ Gxi, hGy= Gyo ~ Gyi can be calculated based on reconstructed references. Since ho+^tl AGx-Vx+AGy-Vy is the regular bi-prediction, 2 is called, from now on, the BIO shift for convenience.

[0047] Vxe Vysão are derived in both the encoder and the decoder, minimizing the following distortion: + (Gxo + θχΐ) ' Vx+ (Gyo + Gyl) P0 -block = min

[0048] With Vxe Vyderivatives, the final prediction of the block is calculated with equation (E) . Vxe Vy are called BIO motion for convenience.

[0049] In general, a video encoder performs BIO during motion compensation. That is, after the video encoder determines a motion vector for a current block, the video encoder produces a predictive block for the current block using motion compensation relative to the motion vector. In general, the motion vector identifies the location of a reference block relative to the current block in a reference image. When performing BIO, a video encoder modifies the motion vector on a per-pixel basis for the current block. That is, instead of recovering Petition 870200158161, dated 12 / 17 / 2020, page 26 / 115 16 / 78 each pixel of the reference block as a block unit, according to BIO, the video encoder determines per-pixel modifications in the motion vector for the current block and constructs the reference block in such a way that the reference block includes reference pixels identified by the motion vector and the per-pixel modification for the corresponding pixel of the current block. Thus, BIO can be used to produce a more accurate reference block for the current block.

[0050] Figure 1 is a block diagram illustrating an example of a video encoding and decoding system 10 that can utilize techniques for bidirectional optical stream. As shown in Figure 1, the system 10 includes a source device 12 that generates encoded video data to be subsequently decoded by a destination device 14. In particular, the source device 12 sends the video data to the destination device 14 via a computer-readable medium 16.Source device 12 and destination device 14 may comprise any of a wide range of devices, including desktop computers, notebook computers (such as, for example, laptops), tablet computers, set-top boxes, telephone devices such as so-called smartphones, so-called smart devices, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices or the like. In some cases, source device 12 and destination device 14 may be equipped for wireless communication. Petition 870200158161, dated 12 / 17 / 2020, page 27 / 115 17 / 78

[0051] The destination device 14 may receive the encoded video data to be decoded by means of the computer-readable medium 16. The computer-readable medium 16 may comprise any type of medium or device capable of moving the encoded video data from the source device 12 to the destination device 14. In one example, the computer-readable medium 16 may comprise a communication medium to enable the source device 12 to transmit encoded video data directly to the destination device 14 in real time. The encoded video data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to the destination device 14. The communication medium may comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines.The communication medium can be part of a packet-based network, such as a local area network, a wide area network, or a global network like the Internet. The communication medium may include routers, switches, base stations, or other equipment that may be useful for facilitating communication between the source device 12 and the destination device 14.

[0052] In some examples, encoded data can be transmitted from the output interface 22 to a storage device. Similarly, the encoded data can be accessed from the storage device via the input interface. The storage device can include any of several media of Petition 870200158161, dated 12 / 17 / 2020, page 28 / 115 18 / 78 distributed or locally accessed data storage, such as a hard drive, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other digital storage media for storing encoded video data. In another example, the storage device may correspond to a file server or other intermediate storage device that can store the encoded video generated by the source device 12. The destination device 14 can access stored video data from the storage device via streaming or download. The file server can be any type of server capable of storing encoded video data and transmitting that encoded video data to the destination device 14.Typical file servers include a web server (for a website, for example), an FTP server, network-attached storage (NAS) devices, or a local disk drive. The target device 14 can access the encoded video data through any standard data connection, including an internet connection. This may include a wireless channel (a Wi-Fi connection, for example), a wired connection (such as DSL, cable modem, etc.), or a combination of both that is suitable for accessing encoded video data stored on a file server. The transmission of encoded video data from the storage device may be a streaming transmission, a download transmission, or a combination thereof.

[0053] The techniques described in this disclosure Petition 870200158161, dated 12 / 17 / 2020, page 29 / 115 19 / 78 can be applied to video encoding in support of various multimedia applications, such as over-the-air television broadcasts, cable television broadcasts, satellite television broadcasts, streaming video transmissions from the Internet such as dynamic adaptive streaming over HTTP (DASH), digital video that is encoded on a data storage medium, decoding of digital video stored on a data storage medium, or other applications. In some examples, the system 10 can be configured to support unidirectional or bidirectional video transmission to support applications such as video streaming, video replay, video broadcasting, and / or video telephony.

[0054] In the example in Figure 1, the source device 12 includes a video source 18, a video encoder 20, and an output interface 22. The destination device 14 includes an input interface 28, a video decoder 30, and a display device 32. According to this disclosure, the video encoder 20 of the source device 12 can be configured to apply bidirectional optical stream techniques. In other examples, a source device and a destination device may include other components or arrangements. For example, the source device 12 may receive video data from an external video source 18, such as an external camera. Similarly, the destination device 14 may interface with an external display device instead of including an integrated display device. Petition 870200158161, dated 12 / 17 / 2020, page 30 / 115 20 / 78

[0055] System 10 illustrated in Figure 1 is merely an example. Techniques for bidirectional optical flow can be performed by any digital video encoding and / or decoding device. Although generally the techniques of this revelation are performed by a video encoding device, the techniques can also be performed by a video encoder / decoder, typically referred to as a CODEC. Furthermore, the techniques of this revelation can be performed by a video preprocessor. Source device 12 and destination device 14 are merely examples of encoding devices in which source device 12 generates decoded video data for transmission to destination device 14. In some examples, devices 12, 14 may operate in a substantially symmetrical manner so that each of devices 12, 14 includes video encoding and decoding components.Consequently, system 10 can support single or bidirectional video transmission between video devices 12, 14, for example, for video streaming, video replay, video broadcasting, or video telephony.

[0056] The video source 18 of the source device 12 may include a video capture device, such as a video camera, a video file containing previously captured videos, and / or a video feed interface for receiving video from a video content provider. Alternatively, the video source 18 may generate computer graphics-based data as the video source or a combination of live videos, Petition 870200158161, dated 12 / 17 / 2020, page 31 / 115 21 / 78 archived videos or computer-generated videos. In some cases, if the video source 18 is a video camera, the source device 12 and the destination device 14 may form so-called camera phones or video phones. As mentioned above, however, the techniques described in this disclosure can be applied to video encoding in general and can be applied to wireless and / or wired applications. In each case, the video, pre-captured or computer-generated, can be encoded by the video encoder 20. The encoded video information can be transmitted through the output interface 22 to a computer-readable medium 16.

[0057] Computer-readable media 16 may include transient media, such as a wireless broadcast or wired network transmission (or non-transient storage media, for example) such as a hard disk, a flash drive, a compact disc, a digital video disc, a Blu-ray disc, or computer-readable media. In some examples, a network server (not shown) may receive encoded video data from the source device 12 and send the encoded video data to the destination device 14, by means of network transmission, for example. Similarly, a computing device in a media production facility, such as a disc stamping facility, may receive encoded video data from the source device 12 and produce a disc containing the encoded video data. Therefore, computer-readable media 16 may be understood as including one or more computer-readable media in various forms, in various examples. Petition 870200158161, dated 12 / 17 / 2020, page 32 / 115 22 / 78

[0058] The input interface 28 of the destination device 14 receives information from the computer-readable medium 16. The information from the computer-readable medium 16 may include syntax information defined by the video encoder 20, which may be used by the video decoder 30, including syntax elements that describe characteristics and / or processing of the video data. The display device 32 displays the decoded video data to the user and may include any of several display devices, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, or an organic light-emitting diode (OLED) display, or another type of display device.

[0059] The video encoder 20 and the video decoder 30 may operate according to one or more video coding standards, such as ITU-T H.264 / AVC (Advanced Video Coding) or High Efficiency Video Coding (HEVC), also referred to as ITU-T H.265. H.264 is described in the International Telecommunication Union, “Advanced video coding for general audiovisual services,” SERIES H: SYSTEMS OF MULTIMEDIA AND AUDIOVISUAL, Audiovisual Services Infrastructure - Motion Video Coding, H.264, June 2011. H.265 is described in the International Telecommunication Union, "High-Efficiency Video Coding", SERIES H: MULTIMEDIA AND AUDIOVISUAL SYSTEMS, Audiovisual Services Infrastructure - Motion Video Coding, April 2015. The techniques of this disclosure can also be applied to any other encoding of Petition 870200158161, dated 12 / 17 / 2020, page 33 / 115 23 / 78 previous video or upcoming standards as an efficient encoding tool.

[0060] Other video coding standards include ITU-T H.261, ISO / IEC MPEG-1 Visual, ITU-T H.262 or ISO / IEC MPEG-2 Visual, ITU-T H.263, ISO / IEC MPEG-4 Visual and the Scalable Video Coding (SVC) and Multi-View Video Coding (MVC) extensions of H.264, as well as HEVC extensions such as the Range Extension, Multi-View Extensions (MV-HEVC) and Scalable Extension (SHVC). On April 20, 2005, the Video Coding Expert Group (VCEG) initiated a new research project aimed at a new generation of video coding standards. The reference software is called HM-KTA.

[0061] The ITU-T VCEG (Q6 / 16) and the ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) are currently studying the potential need for standardization of upcoming video coding technology with a compression capability that significantly exceeds that of the current HEVC standard (which includes current HEVC extensions and short-term extensions for screen content coding and high dynamic range coding). The groups are working together on this exploration activity in a collaborative effort known as the Joint Video Exploration Team (JVET) to evaluate compression technology designs proposed by their experts in this area. The JVET first met between October 19 and 21, 2015. The latest version of the reference software, namely Joint Exploration Model 3 (JEM3), can be downloaded from: https: / / jvet.hhi.fraunhofer.de / svn / svn Petition 870200158161, dated 12 / 17 / 2020, page 34 / 115 24 / 78 A description of the Joint Exploration Test Model 3 (JEM3) algorithm could be found in JVET-D1001. (HMJEMSoftware / tags / HM-16.6-JEM-4.0 / )

[0062] Certain video encoding techniques, such as H.264 and HEVC, which are related to the techniques in this disclosure, are described in this disclosure. Certain techniques in this disclosure may be described with reference to H.264 and / or HEVC to aid understanding, but the techniques described are not necessarily limited to H.264 or HEVC and may be used in conjunction with other encoding standards and other encoding tools.

[0063] Although not shown in Figure 1, in some respects, the video encoder 20 and the video decoder 30 may be integrated with an audio encoder and decoder, and may include appropriate MUX-DEMUX units, or other hardware and software, to process audio and video encoding into a common data stream or separate data streams. If applicable, the MUX-DEMUX units may conform to the ITU H.223 multiplexer protocol or other protocols, such as the User Datagram Protocol (UDP).

[0064] In HEVC and other video encoding standards, a video sequence typically includes a series of images. Images may also be referred to as frames. An image may include three sample arrays, denoted as Sl, SCb, and SCr. Sl is a two-dimensional array (i.e., a block) of luma samples. Scb is a two-dimensional array of samples. Petition 870200158161, dated 12 / 17 / 2020, page 35 / 115 25 / 78 chrominance CB. Scr is a two-dimensional array of chrominance Cr samples. Chrominance samples may also be referred to here as chroma samples. In other cases, an image may be monochromatic and may include only an array of luma samples.

[0065] To generate an encoded representation of an image, the video encoder 20 can generate a set of coding tree units (CTUs). Each of the CTUs can be a coding tree block of luma samples, two corresponding coding tree blocks of chroma samples, and syntax structures used to encode the samples of the coding tree blocks. In monochrome images or images that have three separate color planes, a CTU can comprise a single coding block and syntax structures used to encode the samples of the coding block. A coding tree block can be an NxN block of samples. A CTU can also be referred to as a block tree, larger coding unit (LCU). HEVC CTUs can be broadly analogous to the macroblocks of other video coding standards, such as H.264 / AVC.However, a CTU is not necessarily limited to a specific size and may include one or more encoding units (CUs). A slice may include an integer number of CTUs ordered consecutively in the scan.

[0066] A CTB contains a quad-tree whose nodes are encoding units. The size of a CTB can vary from 16x16 to 64x64 in the main HEVC profile (although technically 8x8 CTB sizes may be supported). Petition 870200158161, dated 12 / 17 / 2020, page 36 / 115 26 / 78 A coding unit (CU) could be the same size as a CTB, although it is as small as 8x8. Each coding unit is encoded with a mode. When a CU is intercoded, the CU can be partitioned into 2 or 4 prediction units (PUs) or become just one PU when another partition does not apply. When two PUs are present in a CU, the two PUs can be rectangles half the size or two rectangle sizes ¼ or ¾ the size of the CU.

[0067] To generate a coded CTU, the video encoder 20 can recursively perform quad-tree partitioning on the coding tree blocks of a CTU in order to divide the coding tree blocks into coding blocks, hence the name “coding tree units”. A coding block is an NxN block of samples. A CU may comprise a luma sample coding block and two corresponding chroma sample coding blocks from an image that has a luma sample array, a Cb sample array with a Cr sample array, and syntax structures used to encode the coding block samples. In monochrome images or images that have three separate color planes, a CU may comprise a single coding block and syntax structures used to encode the coding block samples.

[0068] Video encoder 20 can partition a coding block of a CU into one or more prediction blocks. A prediction block is a rectangular (i.e., square or non-square) block of samples. Petition 870200158161, dated 12 / 17 / 2020, page 37 / 115 27 / 78 in which the same prediction is applied. A prediction unit (PU) of a CU may comprise one luma sample prediction block, two corresponding chroma sample prediction blocks, and syntax structures used to predict the prediction block samples. In monochrome images or images that have three separate color planes, a PU may comprise a single prediction block and syntax structures used to predict the prediction block samples. The video encoder 20 may generate predictive luma, Cb and Cr blocks for the luma, Cb and Cr prediction blocks of each PU of the CU.

[0069] Video encoder 20 can use intraprediction or interprediction to generate the predictive blocks for a PU. If video encoder 20 uses intraprediction to generate the predictive blocks of a PU, video encoder 20 can generate the PU's predictive blocks based on decoded samples of the image associated with the PU. If video encoder 20 uses interprediction to generate the predictive blocks of a PU, video encoder 20 can generate the PU's predictive blocks based on decoded samples of one or more images other than the image associated with the PU. When the CU is intercoded, a motion information set can be present for each PU. In addition, each PU can be encoded with a unique prediction mode to derive the motion information set.

[0070] After video encoder 20 generates predictive luma, Cb and Cr blocks for one or more PUs of a CU, video encoder 20 can generate a block Petition 870200158161, dated 12 / 17 / 2020, p. 38 / 115 28 / 78 residual luma for the CU. Each sample in the CU residual luma block indicates the difference between a luma sample in one of the CU's predictive luma blocks and a corresponding sample in the CU's original luma coding block. Additionally, video encoder 20 can generate a Cb residual block for the CU. Each sample in the CU's Cb residual block can indicate the difference between a Cb sample in one of the CU's predictive Cb blocks and a corresponding sample in the CU's original Cb coding block. Video encoder 20 can also generate a Cr residual block for the CU. Each sample in the CU's Cr residual block can indicate the difference between a Cr sample in one of the CU's predictive Cr blocks and a corresponding sample in the CU's original Cr coding block.

[0071] In addition, the video encoder 20 can use quad-tree partitioning to decompose the luma, Cb, and Cr residual blocks of a CU into one or more luma, Cb, and Cr transform blocks. A transform block can be a rectangular block of samples to which the same transform is applied. A transform unit (TU) of a CU can comprise a luma sample transform block, two corresponding chroma sample transform blocks, and syntax structures used to transform the samples of the transform blocks. Thus, each TU of a CU can be associated with a luma transform block, a Cb transform block, and a Cr transform block. The luma transform block associated with the TU can be a sub-block of the luma residual block of the CU. The Cb transform block can be a sub-block of the Cb residual block of the CU. The transform block Petition 870200158161, dated 12 / 17 / 2020, page 39 / 115 29 / 78 Cr can be a sub-block of the residual block Cr of the CU. In monochrome images that have three separate color planes, a TU can comprise a single block of transforms and syntax structures used to transform the samples of the transform block.

[0072] Video encoder 20 can apply one or more transforms to a luma transform block of a TU in order to generate a luma coefficient block for the TU. A coefficient block can be a two-dimensional array of transform coefficients. A transform coefficient can be a scalar quantity. Video encoder 20 can apply one or more transforms to a Cb transform block of a TU in order to generate a Cb coefficient block for the TU. Video encoder 20 can apply one or more transforms to a Cr transform block of a TU in order to generate a Cr coefficient block for the TU.

[0073] After generating a block of coefficients (a luma coefficient block, a Cb coefficient block, or a Cr coefficient block, for example), video encoder 20 can quantize the coefficient block. Quantization generally refers to a process in which transform coefficients are quantized to possibly reduce the amount of data used to represent the transform coefficients, thus achieving additional compression. After video encoder 20 quantizes a block of coefficients, video encoder 20 can entropy-encode syntax elements that indicate the quantized transform coefficients. For example, the encoder Petition 870200158161, dated 12 / 17 / 2020, page 40 / 115 30 / 78 of video 20 can perform Context-Adaptive Binary Arithmetic Coding (CABAC) on the syntax elements that indicate the quantized transform coefficients.

[0074] Video encoder 20 can transmit a bitstream that includes a bit sequence forming a representation of encoded images and related data. The bitstream may comprise a sequence of NAL units. A NAL unit is a syntax structure containing an indication of the data type in the NAL unit and bytes containing that data in the form of an RBSP interspersed as needed with emulation prevention bits. Each NAL unit includes a NAL unit header and encapsulates an RBSP. The NAL unit header may include a syntax element indicating a NAL unit type code. The NAL unit type code specified by the NAL unit header of a NAL unit indicates the type of the NAL unit. An RBSP may be a syntax structure containing an integer number of bytes that is encapsulated within a NAL unit. In some cases, an RBSP includes zero bits.

[0075] Different types of NAL units can encapsulate different types of RBSP. For example, a first type of NAL unit can encapsulate an RBSP for a PPS, a second type of NAL unit can encapsulate an RBSP for an encoded slice, a third type of NAL unit can encapsulate an RBSP for SEI messages, and so on. NAL units that encapsulate RBSPs for video encoding data (as opposed to RBSPs for parameter sets and SEI messages) can be referred to as NAL VCL units. Petition 870200158161, dated 12 / 17 / 2020, page 41 / 115 31 / 78

[0076] Video decoder 30 can receive a bitstream generated by video encoder 20. In addition, video decoder 30 can parse the bitstream to obtain syntax elements from the bitstream. Video decoder 30 can reconstruct images from the video data based, at least in part, on the syntax elements obtained from the bitstream. The process for reconstructing the video data can generally be complementary to the process performed by video encoder 20. In addition, the video decoder can quantize by inversion coefficient blocks associated with TUs of the current CU. Video decoder 30 can perform inverse transforms on the coefficient blocks to reconstruct transform blocks associated with TUs of the current CU.The Video Decoder 30 can reconstruct the current CU's coding blocks by adding samples from the predictive blocks for the current CU's PUs to corresponding samples from the transform blocks of the current CU's TUs. By reconstructing the coding blocks for each CU of an image, the Video Decoder 30 can reconstruct the image.

[0077] According to the techniques of this disclosure, the video encoder 20 and / or the video decoder 30 can also perform BIO techniques during motion compensation, as discussed in more detail below.

[0078] The video encoder 20 and the video decoder 30 can each be implemented as any one of several encoder circuits. Petition 870200158161, dated 12 / 17 / 2020, page 42 / 115 Suitable components such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. Each of the video encoder 20 and video decoder 30 may be included in one or more encoders and decoders, both of which may be integrated as part of a combined encoder / decoder (CODEC) in a respective device. A device including the video encoder 20 and video decoder 30 may comprise an integrated circuit, a microprocessor, and / or a wireless communication device, such as a mobile phone.

[0079] Figure 2 is a conceptual diagram illustrating an example of one-sided motion estimation (ME), such as a block matching algorithm (BMA) performed for motion-compensated frame rate upconversion (MC-FRUC). In general, a video encoder (such as video encoder 20 or video decoder 30) performs one-sided ME to obtain motion vectors (MVs), such as MV 112, by searching for the best matching block (such as reference block 108) from reference frame 102 to current block 106 of current frame 100. Then, the video encoder interpolates an interpolated block 110 along the motion path of motion vector 112 in interpolated frame 104. That is, in the example in Figure 2, motion vector 112 passes through the midpoints of current block 106, reference block 108, and interpolated block 110. Petition 870200158161, dated 12 / 17 / 2020, page 43 / 115 33 / 78

[0080] As shown in Figure 2, three blocks in three frames are involved following the motion trajectory. Although the current block 106 in the current frame 100 belongs to a coded block, the best matching block in the reference frame 102 (i.e., the reference block 108) need not entirely belong to a coded block (i.e., the best matching block may not fall on a coded block boundary, but instead may overlap such a boundary). Similarly, the interpolated block 110 in the interpolated frame 104 does not need to entirely belong to a coded block. Consequently, overlapping block regions and unfilled regions (holes) may occur in the interpolated frame 104.

[0081] To handle overlaps, simple FRUC algorithms can involve averaging and overwriting the overlapping pixels. Additionally, holes can be covered by pixel values ​​from a reference or current frame. However, these algorithms can result in blocking and blurring artifacts. Consequently, motion field segmentation, successive extrapolation using the discrete Hartley transform, and image retouching can be used to handle holes and overlaps without increasing blocking and blurring artifacts.

[0082] Figure 3 is a conceptual diagram illustrating an example of bilateral ME, such as a BMA implemented for MC-FRUC. Bilateral ME is another solution (in MC-FRUC) that can be used to avoid the problems caused by overlaps and gaps. One Petition 870200158161, dated 12 / 17 / 2020, page 44 / 115 A 34 / 78 video encoder (such as a 20 video encoder and / or a 30 video decoder), which performs bilateral ME, obtains MVs 132, 134 by passing through the interpolated block 130 of the interpolated frame 124 (which is intermediate between the current frame 120 and the reference frame 122) using temporal symmetry between the current block 126 of the current frame 120 and the reference block 128 of the reference frame 122. As a result, the video encoder does not generate overlaps and holes in the interpolated frame 124. Assuming that the current block 126 is a block that the video encoder processes in a specific order, for example, as in the case of video encoding, a sequence of such blocks would cover the entire intermediate image without overlap. For example, in the case of video encoding, the blocks can be processed in the order of decoding. Therefore, such a method may be more suitable if the ideas of FRUC can be considered within a video coding framework.

[0083] S.-F. Tu, OC Au, Y. Wu, E. Luo and CH Yeun, “A Novel Framework for Frame Rate Upconversion by Predictive Variable Block Size Motion Estimated Optical Flow”, Congress The International Conference on Image Signal Processing (CISP), 2009, described a hybrid block-level motion estimation and pixel-level optical flow method for frame rate upconversion. Tu stated that the hybrid scene was better than either individual method.

[0084] In the HEVC standard, there are two inter-prediction modes, called merge mode (with jump mode considered a special case of the mode of Petition 870200158161, dated 12 / 17 / 2020, page 45 / 115 35 / 78 blending) and the Advanced Motion Vector Prediction (AMVP) mode. In either AMVP or blending mode, a video encoder maintains a list of MV candidates for various motion vector predictors. A video encoder determines the motion vector(s) for a specific PU, as well as reference indices in blending mode, by selecting a candidate from the list of MV candidates.

[0085] In HEVC, the MV candidate list contains up to 5 candidates for the merge mode and only two candidates for the AMVP mode. Other encoding standards may include more or fewer candidates. A merge candidate may contain a set of motion information, such as motion vectors corresponding to both reference image lists (list 0 and list 1) and reference indices. A video decoder receives a merge candidate identified by a merge index, and the video decoder predicts a current PU using the identified reference image(s) and motion vector(s). However, for the AMVP mode, for each potential prediction direction from list 0 or list 1, a reference index needs to be explicitly flagged, along with an MV predictor (MVP) index for the MV candidate list, since the AMVP candidate contains only one motion vector.In AMVP mode, the predicted motion vectors can also be refined.

[0086] A candidate merge corresponds to a complete set of motion information, while a candidate AMVP contains only a motion vector. Petition 870200158161, dated 12 / 17 / 2020, page 46 / 115 36 / 78 for a specific prediction direction and reference index. Candidates for both modes are similarly derived from the same neighboring blocks, temporal and spatial.

[0087] Figure 4A shows candidates for spatial neighbor MV for the merge mode and Figure 4B shows candidates for spatial neighbor MV for AMVP modes. The spatial MV candidates are derived from the neighboring blocks shown in Figures 4A and 4B, for a specific PU (PU0), although the methods that generate the candidates from the blocks are different for the AMVP and merge modes.

[0088] In merge mode, up to four spatial MV candidates can be derived with the orders shown in Figure 4A. The ordering is as follows: left (0), top (1), top right (2), bottom left (3), and top left (4), as shown in Figure 4A. If all spatial MV candidates 0-3 are available and unique, then the video encoder may not include motion information for the top left block in the candidate list. If, however, one or more spatial MV candidates 0-3 are unavailable or not unique, then the video encoder may include motion information for the top left block in the candidate list.

[0089] In AVMP mode, neighboring blocks are divided into two groups: the left group, consisting of blocks 0 and 1, and the top group, consisting of blocks 2, 3, and 4, as shown in Figure 4B. For each group, the potential candidate in a neighboring block refers to the same Petition 870200158161, dated 12 / 17 / 2020, page 47 / 115 The reference image indicated by the signaled reference index, 37 / 78, has the highest priority to be chosen to form a final candidate for the group. It is possible that all neighboring blocks do not contain a motion vector pointing to the same reference image. Therefore, if such a candidate cannot be found, the first available candidate will be ranked to form the final candidate, so that differences in temporal distance can be compensated for.

[0090] Figure 5A shows an example of a TMVP candidate and Figure 5B shows an example of MV scheduling. The temporal motion vector predictor (TMVP) candidate, if enabled and available, is added to the MV candidate list after the spatial motion vector candidates. The motion vector derivation process for the TMVP candidate is the same for both merge and AMVP modes; however, the target reference index for the TMVP candidate in merge mode is always set to 0.

[0091] The location of the primary block for TMVP candidate derivation is the lower right block outside the PU, placed as shown in Figure 5A, like a T-block, to compensate for the tendency of the blocks above and to the left to generate spatial neighbor candidates. However, if this block is located outside the current CTB line or motion information is not available, the block will be replaced by a central PU block.

[0092] The motion vector for the TMVP candidate is derived from the co-located PU of the image. Petition 870200158161, dated 12 / 17 / 2020, page 48 / 115 38 / 78 placed, indicated at the slice level. The motion vector for the co-localized PU is called the placed MV. Similar to the temporal direct mode in the AVC, to derive the candidate motion vector TMVP, the co-localized MV needs to be scaled to compensate for temporal distance differences, as shown in Figures 5B.

[0093] HEVC also uses motion vector scaling. The value of the motion vectors is assumed to be proportional to the distance between the images at presentation time. A motion vector associates two images, the reference image and the image containing the motion vector (notably the contained image). When one motion vector is used to predict another motion vector, the distance between the contained image and the reference image is calculated based on the POC values.

[0094] For a motion vector to be predicted, the image containing the motion vector and the reference image associated with the motion vector may be different. Therefore, a new distance (based on POC) is calculated and the motion vector is scaled based on these two POC distances. For a spatial neighbor candidate, the images containing the two motion vectors are the same, while the reference images are different. In HEVC, motion vector scaling applies to both TMVP and AMVP for neighboring, temporal, and spatial candidates.

[0095] HEVC also uses artificial motion vector candidate generation. If a list of motion vector candidates is not complete, artificial motion vector candidates are generated and Petition 870200158161, dated 12 / 17 / 2020, page 49 / 115 39 / 78 are inserted at the end of the list until all available entries in the motion vector candidate list have a candidate. In merge mode, there are two types of artificial motion vector candidates: combined candidate, derived only for B slices, and zero candidate, used only for AMVP if the first type does not provide enough artificial candidates. For each pair of candidates already in the candidate list that has the necessary motion information, the bidirectional combined motion vector candidates are derived by combining the motion vector of the first candidate that refers to an image in list 0 and the motion vector of a second candidate that refers to an image in list 1.

[0096] HEVC also uses a pruning process for candidate insertion. Candidates from different blocks may happen to be the same, which decreases the efficiency of a candidate list for AMVP / merging. A pruning process can be applied to solve this problem. A pruning process compares a candidate with others in the current candidate list to avoid inserting an identical candidate. To reduce complexity, only a limited number of pruning processes can be applied instead of comparing each potential candidate with all others. As an example, a video encoder might apply a pruning process to neighboring, temporal, and spatial candidates, but not to artificially generated candidates.

[0097] Aspects of bidirectional optical flow in the JEM will now be described. Figure 6 shows an example of an optical flow path. The BIO uses a refinement of Petition 870200158161, dated 12 / 17 / 2020, pp. 50 / 115 40 / 78 pixel-mode motion that runs on top of block-mode motion compensation in a biprediction case. As BIO compensates for refined motion within the block, enabling BIO can effectively result in increasing the block size for motion compensation. Sample-level motion refinement does not require exhaustive searching or signaling, but instead uses an explicit equation that provides a refined motion vector for each sample.

[0098] Let I <k>the luminance value from k (k = 0, 1) after the movement of the compensation block, and di^ / dy ~ . . . . . ' ' 'The horizontal and vertical components of the gradient I(k), respectively. Assuming that the optical flux is valid, the motion vector field (vx, vy) is given by an equation 5I{k} / dí + vxdI{k} / d)c+vvdI{k} / õy = 0. (1)

[0099] Combining the optical flux equation with Hermite interpellation for the motion path of each sample, a unique third-order polynomial is obtained that combines both function values ​​I <k>and derivatives AK A^ at the extremes. The value of this polynomial at t=0 is the prediction of BIO: prec^10=A / 2-{l^ + / ^ Hv* jãc — T^dl^ / ck)+vv}dy— τϋ5Ι^ (2)

[0100] Here, το and Ti denote the distance to reference frames, as shown in Figure 6. The distances, To and Ti, are calculated based on the POC for RefO and Refl: Petition 870200158161, dated 12 / 17 / 2020, pp. 51 / 115 41 / 78 το = POC (current) - POC (RefO), τι POC (Refl) POC (current). If both predictions come from the same time direction (either from the past or from the future), then the signs are different το x τι < 0. In this case, BIO is applied only if the prediction does not come from the same point in time (το τι), both referenced regions have non-zero motion (MVxo, MVyo, MVxi, MVyi y 0) and the block motion vectors are proportional to the time distance (MVx0 / MVxi = MVy0 / MVyi = το / τι).

[0101] The motion vector field (vx, vy), also referred to as the BIO momentum, is determined by minimizing the difference Δ between the values ​​at points A and B (intersection of the motion path and reference image planes in Figure 6). The model uses only the first linear term of the local Taylor expansion for Δ: Δ = ( / °) +vx[r1õl^ / βκ+τ09Ι^όκ)+νγ(τ\0Ι® / dy + ίθ (3)

[0102] All values ​​in (1) depend on the sample location (!', j'), which has been omitted so far. Assuming that the motion is consistent in the local environment, then Δ within (2M + 1) x (2M + 1) of the square window Ω centered on the current predicted point (1, j) can be minimized as follows: (vx,vj = argmin £a2[ / ', / ] (4) ''ry [í'j]eQ

[0103] For this optimization problem, the simplified solution that performs the first minimization in the directions Petition 870200158161, dated 12 / 17 / 2020, page 52 / 115 42 / 78 vertical and horizontal can be used, resulting in: vx= (si +r) > clip3 y—thBIO, thBI0, — J:θ (5) vy= (s5+r)>m? clip3 (-thBIO, thBIO, -: 0(6) where, 5j = Σ (τβΐ^ / δχ + r0dl^j dxf; s3= ^τ^Ι^ / dx + Toõl^ / dx\ .s2= ^(ηάΖ^ / άν + τ0dl^ / dxfa\dl® / dy + r0dl^ / ôy\ [<'.7kn55 = ς(γι0 / (1) / ^+γο0 / (ο7^)2'56=ς7(1)-^0)Χλ^(1) / ^+^(0) / ^) [í'j]ef2 [ι',. / kn (7)

[0104] To avoid division by zero or very small value, the regularization parameters rem are introduced in equations (2), (3). r = 500 4d-8(8) m = 700 4d-8(9) Here, d is the internal bit depth of the video input.

[0105] In some cases, the MV refinement of BIO may not be reliable due to noise or irregular motion. Therefore, in BIO, the magnitude of the MV refinement is clipped to the threshold determined by thBIO. The threshold value is determined based on whether all reference images of the current image are all from one direction. If all reference images of the images Petition 870200158161, dated 12 / 17 / 2020, p. 53 / 115 If the current 43 / 78 of the image is from one direction, the limit value can be set to 12 x 214-d; otherwise, the limit can be set to 12 x 213-d.

[0106] The gradients for BIO are calculated simultaneously with the motion compensation interpellation using operations consistent with the HEVC motion compensation process (2D separable FIR). The input for this 2D separable FIR is the same reference image sample as for the fractional motion and position compensation process (fracX, fracY), according to the fractional part of the block motion vector. In the case of the vertical gradient dl / dx, the first gradient filter is applied vertically using BIOfilterG, corresponding to the fractional position fracY with a descaling change d-8, then the signal shift is performed using BlOfilterS in the horizontal direction corresponding to the fractional position fracX with a descaling change d-8. The length of the interpellation filter for calculating gradients BIOfilterG and signal shift BIOfilterF is shorter (6 taps) in order to maintain reasonable complexity.Table 1 shows the filters used to calculate gradients for different fractional positions of the block motion vector in BIO. Table 2 shows the interpellation filters used to generate prediction signals in BIO.

[0107] Figure 7 shows an example of gradient calculation for an 8x4 block. For 8x4 blocks, a video encoder searches for compensated motion predictors and calculates the HOR / VER gradients of all of them. Petition 870200158161, dated 12 / 17 / 2020, page 54 / 115 44 / 78 pixels within the current block, as well as the two outer rows of pixels, because the vx and vy resolution for each pixel requires the gradient of HOR / VER values ​​and motion compensation predictors of the pixels within the Ω window centered on each pixel, as shown in equation (4). In JEM, the size of this window is set to 5x5. Therefore, a video encoder needs to fetch the motion compensation predictors and calculate the gradients for the two outer rows of pixels. Table 1: Filters for gradient calculation in BIO Position by Fractional Gradient Interpolation Filter (BIOfilterG) 0 {8, -39, -3, 46, -17, 5} 1 / 16 {8, -32, -13, 50, -18, 5} 1 / 8 {7, -27, -20, 54, -19, 5} 3 / 16 {6, -21, -29, 57, -18, 5} ¼ {4, -17, -36, 60, -15, 4} 5 / 16 {3, -9, -44, 61, -15, 4} 3 / 8 {1, -4, -48, 61, -13, 3} 7 / 16 {0, 1, -54, 60, -9, 2} U {1, 4, -57, 57, -4, 1} Table 2: Interpolation filters for generating prediction signals in BIO Position by Fractional Interpolation Filter for Prediction (BlOfilterS) Sign of 0 {0, 0, 64, 0, 0, 0} 1 / 16 {1, -3, 64, 4, -2, 0} 1 / 8 {1, -6, -62, 9, -3, 1} 3 / 16 {2, -8, 60, 14, -5, 1} ¼ {2, -9, -57, 19, -7, 2} 5 / 16 {3, -10, 53, 24, -8, 2} 3 / 8 {3, -11, 50, 29, -9, 2} 7 / 16 {3, -11, 44, 35, -10, 3} U {1, -7, 38, 38, -7, 1} Petition 870200158161, dated 12 / 17 / 2020, pp. 55 / 115 45 / 78

[0108] In JEM, BIO is applied to all predicted bidirectional blocks when the two predictions are from different reference images. When the IOL is available for a CU, BIO is unavailable.

[0109] Figure 8 shows an example of a modified BIO for the 8x4 block proposed in JVET-D0042. At the 4th JVET meeting, a proposal from JVET-D0042 (A. Alshina, E. Alshina, AHG6: On BIO Memory Bandwidth, JVET-D0042, October 2016) was submitted to modify the BIO operations and reduce the memory access bandwidth. In this proposal, motion-compensated predictors and gradient values ​​for pixels outside the current block are not required. Furthermore, the vx and vy resolution for each pixel is modified to utilize the motion-compensated predictors and gradient values ​​of all pixels within the current block, as shown in Figure 8. In other words, the square window Ω in equation (4) is modified to a window that is equal to the current block. Furthermore, a weighting factor w(iT, j') is considered to derive vx and vy. The w(i', j') is a function of the position of the central pixel (i, j) and the positions of the pixels (!', j') within the window. 5] = 'faõlw / ãc + r0 õl^ / dx)2;s3 = + / dx\ S2= / & + r, dl^ / õx^õl® jcy + r0dl{o} / dy\ lúki V = \^.i,J,)(rldI{i) / dy+r0õI(-0} / dyf-,s6 = -!{0}\T}dI[l} / õy + T0dIín} / cy) [<'T]εΩ [<'5]εΩ (10)

[0110] Aspects of Overlapping Block Motion Compensation (OBMC) in JEM will now be described. The Petition 870200158161, dated 12 / 17 / 2020, pp. 56 / 115 46 / 78 OBMC has been used for early generations of video standards, such as H.263. In JEM, OBMC runs for all Motion Compensated (MC) block boundaries, except the right and bottom boundaries of a CU. Furthermore, OBMC can be applied to both luma and chroma components. In JEM, an MC block corresponds to a coding block. When a CU is encoded with sub-CU mode (which includes subCU blending, Affine mode, and FRUC, as described in J. Chen, E. Alshina, G.J. Sullivan, J.-R. Ohm, J. Boyce, "Description of the Joint Testing Algorithm of Test Model 4," JVETD1001, October 2016), each sub-block of the CU is an MC block. To process CU boundaries uniformly, the OBMC runs at the sub-block level for all MC block boundaries, where the sub-block size is equal to 4x4, as illustrated in Figures 9A and 9B.

[0111] When OBMC is applied to the current sub-block, in addition to the current motion vectors, the motion vectors of four connected neighboring sub-blocks, if available and not identical to the current motion vector, are also used to derive the prediction block for the current sub-block. These various prediction blocks based on various motion vectors are combined to generate the final prediction signal for the current sub-block.

[0112] As shown in Figure 10, the prediction block based on motion vectors of a neighboring subblock is denoted as Pn, with N indicating an index for the adjacent subblocks above, below, to the left, and to the right, and the prediction block based on motion vectors of the current subblock is denoted as Pc. When Pn Petition 870200158161, dated 12 / 17 / 2020, page 57 / 115 47 / 78 is based on the movement information of a neighboring sub-block that contains the same movement information for the current sub-block; the OBMC does not run from Pn. Otherwise, each pixel of Pn is added to the same pixel in Pc, that is, four rows / columns of Pn are added to Pc. The weighting factors are {1 / 4, 1 / 8, {1 / 16, 1 / 32} are used for Pn and the weighting factors {3 / 4, 7 / 8, 15 / 16, 31 / 32} are used for Pc. The exception is small MC blocks (i.e., when the height or width of the encoding block is equal to 4 or a CU is encoded with sub-CU mode), for which only two rows / columns of Pn are added to Pc. In this case, the weighting factors {1 / 4, 1 / 8} are used for Pn and the weighting factors {3 / 4, 7 / 8} are used for the Pc. For the Pn generated based on vertically (horizontally) neighboring sub-block motion vectors, pixels in the same row (column) as PN are added to Pc with the same weighting factor. BIO can also be applied to the derivation of the Pn prediction block.

[0113] In JEM, for a CU with a size less than or equal to 256 luma samples, a CU level flag is displayed to indicate whether or not OBMC is applied to the current CU. For CUs with a size greater than 256 luma samples or not encoded with AMVP mode, OBMC is applied by default. In the encoder, when OBMC is applied to a CU, its impact is taken into account during the motion estimation stage. The prediction signal, which uses motion information from the upper and left neighboring blocks, is used to compensate for the upper and left limits of the signal. Petition 870200158161, dated 12 / 17 / 2020, page 58 / 115 48 / 78 original of the current CU, and then the normal motion estimation process is applied.

[0014] Although BIO potentially provides more than 1% reduction in Bjontegaard-Delta bitrate (BD-rate) in JEM4.0, BIO also potentially introduces significant computational complexity and may require increased memory bandwidth for both the encoder and decoder. This disclosure describes techniques that can potentially reduce the computational complexity and memory bandwidth requirements associated with BIO. As an example, according to the techniques in this disclosure, a video encoder can determine a quantity of BIO motion, such as the vx and vy values ​​described above, at a sub-block level and use that determined quantity of BIO motion to modify sample values ​​of a predictive block on a sample-by-sample basis.Therefore, the techniques of this discovery can improve video encoders and video decoders, allowing them to achieve the coding gains of BIO without incurring the substantial processing and memory costs required for existing BIO implementations.

[0115] Based on equation (4), this revelation introduces techniques to reduce the complexity of the BIO by redefining the window Ω. Such techniques can, for example, be performed by video encoder 20 (such as, for example, motion estimation unit 42 and / or motion compensation unit 44) or by video decoder 30 (such as, for example, compensation unit of Petition 870200158161, dated 12 / 17 / 2020, page 59 / 115 49 / 78 motion 72). The Ω window is defined as any block within the current block that covers the current pixel with size MxN, where M and N are any positive integers. In one example, the current block is divided into non-overlapping sub-blocks, and the Ω window is defined as the sub-block that covers the current pixel. In another example, as shown in Figure 11, the sub-block is defined as the smallest block for motion vector storage that covers the current pixel. In HEVC and JEM, the smallest block size is 4x4. In another example, the size of the Ω window is adaptive according to encoding information, such as the current block size and encoding modes. When the current block size is larger, a larger Ω window can be used. When the current block is encoded as a sub-block mode, such as sub-CU fusion, Affine mode, and FRUC, the Ω window is defined as the sub-block.

[0116] Figure 11 shows an example of the proposed BIO for an 8x4 block, according to the techniques of this disclosure, with a window Ω for pixels A, B, and C. According to the techniques of this disclosure, equal weightings can be used to solve vx and vy, as shown in equation (7). In another example, different weightings can be used to solve vx and vy, as shown in equation (10). Unequal weightings can be a function of the distances between the central pixel and the associated pixels. In yet another example, the weighting can be calculated using a bilateral approach, as described for example at https: / / en.wikipedia.org / wiki / Bilateral_filter. Furthermore, lookup tables can be used to store Petition 870200158161, dated 12 / 17 / 2020, pages 60 / 115 50 / 78 all weighting factors for each pixel for the Ω window in equation (7).

[0117] In another example, when deriving PN to OBMC, BIO is run only for partial pixels when deriving predictors using neighboring motions. In one example, BIO is completely disabled for all pixels in the PN derivation. In yet another example, BIO is applied only to pixels in the two outer rows, as shown in Figures 12A-12D.

[0118] In addition, for each block, how many BIO lines are applied can be explicitly signaled at the SPS / PPS slice level. If BIO is disabled or partially disabled, it can also be explicitly signaled at the SPS / PPS slice level.

[0119] On the other hand, how many BIO lines are applied can be implicitly based on certain encoding conditions, such as CU mode (sub-block or non-sub-block mode) or block size or the combination of other tools, such as signaled Lighting Compensation (IC) flagging. Whether BIO is disabled or partially disabled can also be implicitly derived based on certain conditions, such as CU mode (sub-block or non-sub-block mode) or block size or the combination of other tools, such as signaled IC flagging.

[0120] Figures 12A-12D show examples of simplified BIO proposed in OBMC according to the techniques of this disclosure, in which x represents the derived predictor without BIO and o represents the derived predictor with BIO. The refinement of the motion vector from the BIO can be Petition 870200158161, dated 12 / 17 / 2020, pp. 61 / 115 51 / 78 based on blocks. Let the block size be M-by-N, a weighting function can be used to provide different scaling factors for pixels from different locations during the calculation of the terms in equation (7). When solving equations (5) and (6), the interpellated pixels and their gradient values ​​gathered from the entire block can be used to solve vx and vy together, instead of solving vx and vy individually for each pixel position.

[0121] In one example, the window size Ω can be defined as an operational window centered on each pixel location, and the average value is used by summing the values ​​of all locations. Specifically,1õl(1] / dx + T^ / õxj / N· k=0 t=0 |rt',· ^=Σ Σ(Γ,δ / (1) / 5χ + / õy + q, õl^ / dy) / N; t=o [í',. / kgr*5 =Σ+=Σ Y^}-I^õI^ / õy+r0ÕI^ / õy) / N k=oi=o (Π) where N is the number of pixels in each sub-block and Ok is the window defined for each pixel. In an example, Ok can be the 5x5 window defined in the current BIO drawing for each pixel, and therefore the weighting function can be determined in advance. Figure 13 shows an example of the weighting function used for the 4x4 sub-block with a 5x5 window. Figure 13 shows an example of a weighting function for a 4x4 sub-block with a 5x5 window. Petition 870200158161, dated 12 / 17 / 2020, pp. 62 / 115 52 / 78

[0122] In another example, the weighting function can be sent in SPS, PPS, or slice header. To reduce signaling costs, a set of predefined weighting functions can be stored, and only the indices of the weighting functions need to be signaled.

[0123] In another example, the refined motion vector can be found using pixels located in the central part of the sub-block. The gradient values ​​of the central pixels can be calculated using the interpolation filter and an M-by-N size window can be applied to the interpolated pixels to provide different weights to the central pixels in order to calculate the s1-s6 variables in equation (7). In one example, the gradient values ​​of the central pixels can be calculated and the average value of the central pixels can be used (equal weight window). In another example, a median filter can be used to select the representative pixels to calculate the s1-s6 variables in equation (7).

[0124] In JVET-D0042, when solving by BIO offset(s), the window size for each pixel can be modified to be the entire current block, which potentially adds computational complexity to the current drawing when a current block is greater than or equal to 8x4. The worst-case scenario of the modifications is that a 128x128 window is used for the accumulation of gradients and predictors for each pixel within a 128x128 block.

[0125] Furthermore, when sub-blocks within a CU share the same MV or an inter-coded CU is divided into smaller sub-blocks for compensation of Petition 870200158161, dated 12 / 17 / 2020, pp. 63 / 115 53 / 78 motion (MC), the JEM-4.0 provides the flexibility to either execute MC and BIO for each sub-block in parallel or execute MC and BIO for the largest aggregate block of sub-blocks with the same MV in a single time effort. Either way, the JEM-4.0 provides identical encoding results. However, the modified BIO in the JVET-D0042 uses a block size-dependent gradient calculation and weighting factors such that executing MC and BIO for two neighboring similar motion blocks, together or separately, can lead to different results. To avoid different results, it must be specified that the decoder should execute MC and BIO at the block level or at a specific sub-block level. Such a restriction can be very stringent and undesirable for the practical implementation of codecs.

[0126] Based on equation (4), the complexity of the BIO can be further reduced by redefining the Ω window. Two types of Ω window are defined; one is the non-overlapping window and the other is a sliding window. For the non-overlapping window type, the current block is divided into non-overlapping subblocks and the Ω window is defined as the subblock that covers the current pixel, as shown in Figure 11. For the sliding window type, the Ω window is defined as a block centered on a current pixel, as shown in Figure 7.

[0127] For both types of Ω window, the size of the Ω window can be determined using different methods, as illustrated below. From then on, it can be assumed that the Ω window is a rectangular block with size MxN, where M and N can be any number. Petition 870200158161, dated 12 / 17 / 2020, pp. 64 / 115 54 / 78 is a non-negative integer, such as (4x4, 8x8, 16x16, 8x4, and so on). The Ω window is not limited to a rectangular shape and can have any other shape, traditionally such as a diamond shape. The techniques described can also be applied to shapes other than rectangular, if applicable.

[0128] The window size can be fixed or variable and can be predetermined or signaled in the bitstream. When the size is signaled, the size can be signaled in the sequence parameter set (SPS), in the image parameter set (PPS), in the slice header, or at the CTU level. The window size can be jointly determined by the motion-compensated (MC) block size using the equation below. Horizontal window size M = min (M, MC_Size); Vertical window size N = min (N, MC_Size).

[0129] In one example, the motion-compensated (MC) block is purely dependent on encoding information, such as the current block size and encoding modes. For example, the motion-compensated (MC) block is defined as the entire CU when the current CU is encoded with non-sub-block modes, such as sub-CU merge, Affine mode, and FRUC. The motion-compensated (MC) block is configured as a sub-block when sub-block modes are used, such as sub-CU merge, Affine, and FRUC, regardless of whether the sub-blocks have the same motion information.

[0130] In another example, the motion-compensated (MC) block is defined as the sample block Petition 870200158161, dated 12 / 17 / 2020, pages 65 / 115 55 / 78 within a CU that has the same MVs. In this case, the motion-compensated (MC) block is defined as the entire CU when the current CU is encoded with non-sub-block modes, such as sub-CU merge mode, Affine, and FRUC. When a CU is encoded with sub-block modes, such as sub-CU merge mode, Affine mode, and FRUC, the subblocks with the same motion information are merged as a motion-compensated (MC) block with a given sub-block scan order.

[0131] Adaptive sizing: the Ω window size adapts according to encoding information, such as the current block size and encoding modes. In one example, the Ω window is set to the entire current block or a quarter of the current block when the current block is encoded as non-sub-block modes, such as sub-CU merging, Affine mode, and FRUC; and the Ω window is set to sub-block when the current block is encoded as a sub-block mode. The adaptive window size can be jointly determined by the motion-compensated (MC) block size using the equation below. Horizontal window size M = mim (M, MC_Size); Vertical window size N = min(N, MC_Size).

[0132] For the various techniques to determine the window size Ω, a high-level size limitation can be included for user-friendly hardware or software implementation. For example, the window size must be less than or equal to the maximum Transform Unit (TU) size allowed in the video codec system. In another example, the window size must be greater than or Petition 870200158161, dated 12 / 17 / 2020, pages 66 / 115 56 / 78 equal to the smallest MC block, such as 4x4.

[0133] To further simplify BIO-related operations, this disclosure introduces techniques for performing BIO as a post-processing step, after all motion compensation prediction is complete. Specifically, after conventional MC is completed, OBMC can be applied to generate better predictors for the current block. Based on the final predictor, BIO is then applied using the motion information of the current block to further refine the predictor. For example, for gradient calculation in BIO, the motion of the entire block can be used. In another example, for each sub-block, the average motion vector from OBMC can be used. In yet another example, for each sub-block, the median motion vector (for each dimension individually) can be used.

[0134] The weighting functions can be designed differently when considering block-based derivation of BIO motion vector refinement. Equal weights can be used for any of the methods mentioned above. Alternatively, more weights can be placed towards the center of the window. In one example, the weights can be calculated by the inverse distance (which includes, but is not limited to, L1 norm or L2 norm) between the center of the window and the pixel.

[0135] Figure 8 is a block diagram showing an example of a video encoder 20 that can implement techniques for bidirectional optical streaming. The video encoder 20 can perform intra-encoding and Petition 870200158161, dated 12 / 17 / 2020, pp. 67 / 115 57 / 78 Intercoding of video blocks within video slices. Intracoding relies on spatial prediction to reduce or remove spatial redundancy in video within a given video frame or image. Intercoding relies on temporal prediction to reduce or remove temporal redundancy in video within adjacent frames or images of a video sequence. Intra-mode (I-mode) can refer to any of several spatial coding modes. Inter-modes, such as one-way prediction (P-mode) or two-way prediction (B-mode), can refer to any of several temporal coding modes.

[0136] As shown in Figure 14, the video encoder 20 receives video data and stores the received video data in the video data memory 38. The video data memory 38 can store video data to be encoded by the components of the video encoder 20. The video data stored in the video data memory 38 can be obtained, for example, from the video source i8. The reference image memory 64 can be a reference image memory that stores reference video data for use in encoding video data by video encoder 20, for example, in intra- or inter-encoding modes. The video data memory 38 and reference image memory 64 can be formed by any of a series of memory devices, such as dynamic random access memory (DRAM), which includes synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices.Video data memory 38 and image memory. Petition 870200158161, dated 12 / 17 / 2020, pp. 68 / 115 58 / 78 reference 64 can be provided by the same memory device or separate memory devices. In several examples, the video data memory 38 may be on the chip with other video encoder components 20, or without a chip in relation to these components.

[0137] The video encoder 20 receives the current video block within a video frame to be encoded. In the example in Figure 14, the video encoder 20 includes a mode selection unit 40, a reference frame memory 64 (which may also be referred to as a decoded image store (DPB)), an adder 50, a transform processing unit 52, a quantization unit 54, and an entropy encoding unit 56. The mode selection unit 40, in turn, includes a motion compensation unit 44, a motion estimation unit 42, an intraprediction unit 46, and a partitioning unit 48. For video block reconstruction, the video encoder 20 also includes an inverse quantization unit 58, an inverse transform unit 60, and an adder 62.A deblock filter (not shown in Figure 14) can also be included to filter block boundaries in order to remove blocking artifacts from reconstructed videos. If desired, the deblock filter would typically filter the output of adder 62. Additional filters (in-mesh or post-mesh) can also be used in addition to the deblock filter. Such filters are not shown for brevity, but, if desired, they can filter the output of the adder. Petition 870200158161, dated 12 / 17 / 2020, pp. 69 / 115 59 / 78 adder 50, (like a mesh filter).

[0138] During the encoding process, the video encoder 20 receives a video frame or slice to be encoded. The frame or slice can be divided into several video blocks. The motion estimation unit 42 and the motion compensation unit 44 perform inter-predictive encoding of the received video block with respect to one or more blocks in one or more reference frames in order to obtain temporal prediction. The intraprediction unit 46 can alternatively perform intra-predictive encoding of the received video block with respect to one or more neighboring blocks in the same frame or slice of the block to be encoded in order to obtain spatial prediction. The video encoder 20 can perform several encoding passes, for example, to select the appropriate condition mode for each video data block.

[0139] Furthermore, partition unit 48 can partition video data blocks into sub-blocks, based on the evaluation of previous partitioning schemes in previous encoding passes. For example, partition unit 48 can initially partition a frame or slice into LCUs and partition each of the LCUs into sub-CUs based on rate distortion analysis (such as, for example, rate distortion optimization). Mode selection unit 40 can also produce a quad-tree data structure indicating the partitioning of an LCU into sub-CUs. The leaf-node CUs of the quad-tree can include one or more PUs and one or more TUs. Petition 870200158161, dated 12 / 17 / 2020, pp. 70 / 115 60 / 78

[0140] The mode selection unit 40 can select one of the prediction modes, intra or inter, such as, for example, based on error results, and provides the predictive block for the adder 50 to generate residual data and for the adder 62 to reconstruct the encoded block for use as a reference frame. The mode selection unit 40 also provides syntax elements, such as motion vectors, intramode indicators, partition information and other syntax information, for the entropy encoding unit 56.

[0141] The motion estimation unit 42 and the motion compensation unit 44 may be highly integrated, but are shown separately for conceptual purposes. The motion estimation performed by the motion estimation unit 42 is the process of generating motion vectors, which estimate the motion for video blocks. A motion vector, for example, might indicate the displacement of a PU of a video block within the current video frame or image relative to a predictive block within a reference frame (or other encoded unit) relative to the current block that is encoded within the current frame (or other encoded unit). A predictive block is a block that is found to closely match the block to be encoded, in terms of pixel difference, which can be determined by the sum of absolute differences (SAD), the sum of squared differences (SSD), or other difference metrics.In some examples, video encoder 20 can calculate integer values ​​for sub-pixel positions of reference images stored in frame memory. Petition 870200158161, dated 12 / 17 / 2020, pp. 71 / 115 61 / 78 reference 64. For example, video encoder 20 can interpolate quarter-pixel position values, eighth-pixel position values, or other fractional pixel position values ​​from the reference image. Therefore, motion estimation unit 42 can perform a motion search with respect to both full pixel positions and fractional pixel positions and transmit a motion vector with fractional pixel precision.

[0142] The motion estimation unit 42 calculates a motion vector for a PU of a video block in an intercoded slice by comparing the position of the PU with the position of a predictive block of a reference image. The reference image can be selected from a first list of reference images (List 0) or a second list of reference images (List 1), each of which identifies one or more reference images stored in the reference frame memory 64. The motion estimation unit 42 sends the calculated motion vector to the entropy encoding unit 56 and the motion compensation unit 44.

[0143] The motion compensation performed by motion compensation unit 44 may involve searching for or generating the predictive block based on the motion vector determined by motion estimation unit 42. Again, motion estimation unit 42 and motion compensation unit 44 may be functionally integrated in some instances. Upon receiving the motion vector for the PU of the current video block, motion compensation unit 44 may locate the predictive block that the motion vector indicates. Petition 870200158161, dated 12 / 17 / 2020, pp. 72 / 115 62 / 78 in one of the reference image lists. The adder 50 forms a residual video block by subtracting the pixel values ​​of the predictive block from the pixel values ​​of the actual video that is encoded, forming pixel difference values, as discussed below. In general, the motion estimation unit 42 performs motion estimation with respect to luma components, and the motion compensation unit 44 uses motion vectors calculated based on luma components for both chroma and luma components. The mode selection unit 40 can also generate syntax elements associated with the video blocks and the video slice for use by the video decoder 30 in decoding the video blocks of the video slice.

[0144] Furthermore, motion compensation unit 44 can be configured to perform any or all of the techniques of this disclosure (alone or in any combination). Although discussed in relation to motion compensation unit 44, it should be understood that mode selection unit 40, motion estimation unit 42, partitioning unit 48 and / or entropy encoding unit 56 can also be configured to perform certain techniques of this disclosure, alone or in combination with motion compensation unit 44. In one example, motion compensation unit 44 can be configured to perform the BIO techniques discussed herein.

[0145] The intraprediction processing unit 46 can intra-predict or calculate the current block, as an alternative to the inter-prediction performed. Petition 870200158161, dated 12 / 17 / 2020, pp. 73 / 115 63 / 78 by motion estimation unit 42 and motion compensation unit 44, as described above. In particular, intraprediction unit 46 can determine the intraprediction mode to be used to encode the current block. In some examples, intraprediction unit 46 can encode the current block using several intraprediction modes, for example, during separate encoding passes, and intraprediction unit 46 (or mode selection unit 40, in some examples) can select an appropriate intraprediction mode to be used from the tested modes.

[0146] For example, intraprediction processing unit 46 can calculate rate distortion values ​​using a rate distortion analysis for the various intraprediction modes tested, and select the intraprediction mode that has the best rate distortion characteristics among the tested modes. Rate distortion analysis generally determines the degree of distortion (or error) between an encoded block and an original unencoded block that was encoded to produce the encoded block, as well as the bit rate (i.e., the number of bits) used to produce the encoded block. Intraprediction processing unit 46 can calculate ratios from the distortions and rates for the various encoded blocks in order to determine which intraprediction mode presents the best rate distortion value for the block.

[0147] After selecting an intraprediction mode for a block, intraprediction unit 46 can provide information indicating the mode of Petition 870200158161, dated 12 / 17 / 2020, pp. 74 / 115 64 / 78 intraprediction selected for the block to the entropy encoding unit 56. The entropy encoding unit 56 can encode information indicating the selected intraprediction mode. The video encoder 20 can include configuration data in the transmitted bitstream, which may include a series of prediction mode index tables and a series of modified intraprediction mode tables (also referred to as codeword mapping tables), encoding context definitions for various blocks, and indications of the most probable intraprediction mode, an intraprediction mode index table, and a modified intraprediction mode index table to be used for each of the contexts.

[0148] The video encoder 20 forms a residual video block by subtracting the prediction data from the mode selection unit 40 from the original video block that is encoded. The adder 50 represents the component or components that perform this subtraction operation. The transform processing unit 52 applies a transform, such as a discrete cosine transform (DCT) or a conceptually similar transform, to the residual block, producing a video block comprising residual transform coefficient values. The transform processing unit 52 can perform other transforms that are conceptually similar to the DCT. Wavelet transforms, integer transforms, sub-band transforms, or other types of transforms can also be used. In any case, the transform processing unit 52 applies the Petition 870200158161, dated 12 / 17 / 2020, pp. 75 / 115 65 / 78 transformed to the residual block, producing a block of residual transform coefficients. The transform can convert the residual information from a pixel value domain into a transform domain, such as a frequency domain. The transform processing unit 52 can send the resulting transform coefficients to the quantization unit 54. The quantization unit 54 quantizes the transform coefficients in order to further reduce the bit rate; the quantization process can reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be modified by adjusting a quantization parameter.

[0149] After quantization, the entropy coding unit 56 entropy encodes the quantized transform coefficients. For example, the entropy coding unit 56 can perform context-adaptive variable-length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy coding (PIPE), or another entropy coding technique. In the case of context-based entropy coding, the context can be based on neighboring blocks. After entropy coding by the entropy coding unit 56, the encoded bitstream can be transmitted to another device (the video decoder 30, for example) or archived for later transmission or retrieval. Petition 870200158161, dated 12 / 17 / 2020, pp. 76 / 115 66 / 78

[0150] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel domain. In particular, adder 62 adds the reconstructed residual block to the motion-compensated prediction block produced by the motion compensation unit 44 or the intraprediction processing unit 46 to produce a reconstructed video block for storage in the reference frame memory 64. The reconstructed video block can be used by the motion estimation unit 42 and the motion compensation unit 44 as a reference block to intercode a block in a subsequent video frame.

[0151] Figure 15 is a block diagram illustrating an example of a video decoder 30 that can implement techniques for bidirectional optical flow. In the example in Figure 15, the video decoder 30 includes an entropy decoding unit 70, a motion compensation unit 72, an intraprediction unit 74, an inverse quantization unit 76, an inverse transform unit 78, a reference frame memory 82, and an adder 80. The video decoder 30 can, in some examples, perform a decoding pass that is generally complementary to the encoding pass described with respect to the video encoder 20 (FIG. 14). The motion compensation unit 72 can generate prediction data based on the motion vectors received from the entropy decoding unit 70, while the intraprediction unit 74 can generate data from Petition 870200158161, dated 12 / 17 / 2020, pp. 77 / 115 67 / 78 prediction based on intraprediction mode indicators received from the entropy decoding unit 70.

[0152] During the decoding process, the video decoder 30 receives a stream of encoded video bits representing video blocks of an encoded video slice and associated syntax elements from the video encoder 20. The video decoder 30 stores the received encoded video bit stream in the video data memory 68. The video data memory 68 can store video data, such as an encoded video bit stream, to be decoded by the components of the video decoder 30. The video data stored in the video data memory 68 can be obtained, for example, through a computer-readable medium 16, storage medium, or from a local video source, such as a camera, or by accessing physical data storage media. The video data memory 85 can form a coded image store (CPB) that stores encoded video data from an encoded video bit stream.Reference image memory 82 may be a reference image memory that stores reference video data for use in decoding video data by the video decoder 30, for example, in intra- or inter-coding modes. Video data memory 68 and reference image memory 82 may be formed by any of a series of memory devices, such as DRAM, SDRAM, MRAM, RRAM, or other types of memory devices. Video data memory 68 and reference image memory 82 may be provided by the same memory device or... Petition 870200158161, dated 12 / 17 / 2020, pp. 78 / 115 68 / 78 separate memory device. In several examples, the 68 video data memory may be on the chip with other 30 video decoder components, or off the chip in relation to such components.

[0153] During the decoding process, the video decoder 30 receives a bitstream of encoded video representing video blocks of an encoded video slice and related syntax elements from the video encoder 20. The entropy decoding unit 70 of the video decoder 30 entropy-decodes the bitstream to generate quantized coefficients, motion vectors or intraprediction mode split indicators and other syntax elements. The entropy decoding unit 70 outputs the motion vectors and other syntax elements to the motion compensation unit 72. The video decoder 30 can receive the syntax elements at the video slice level and / or at the video block level.

[0154] When the video slice is encoded as an intra-coded slice (I), the intraprediction unit 74 can generate prediction data for a video block of the current video slice based on the signaled intraprediction mode and previously decoded block data from the current frame or image. When the video frame is encoded as an inter-coded slice (B, P, or GPB, for example), the motion compensation unit 72 produces predictive blocks for a video block of the current video slice based on motion vectors and other syntax elements received from the entropy decoding unit 70. Predictive blocks can be produced Petition 870200158161, dated 12 / 17 / 2020, pp. 79 / 115 69 / 78 from one of the reference images within one of the reference image lists. The video decoder 30 can construct the reference frame lists, List 0 and List 1, using predefined construction techniques based on reference images stored in the reference frame memory 82.

[0155] Motion compensation unit 72 determines prediction information for a video block from the current video source by parsing motion vectors and other syntax elements, and uses the prediction information to produce the predictive blocks for the current video block that is decoded. For example, motion compensation unit 72 uses some of the received syntax elements to determine a prediction mode (intra- or inter-prediction, for example) used to encode the video blocks of the video slice, the type of inter-prediction slice (B-slice, P-slice, or GPB-slice, for example), construction information for one or more of the reference image lists for the slice, motion vectors for each intercoded video block of the slice, the inter-prediction condition for each intercoded video block of the slice, and other information to decode the video blocks in the current video slice.

[0156] The motion compensation unit 72 can also perform interpolation based on interpolation filters. The motion compensation unit 72 can use the interpolation filters used by the video encoder 20 during the encoding of video blocks to calculate interpolated values ​​for sub-pixels of Petition 870200158161, dated 12 / 17 / 2020, pages 80 / 115 70 / 78 is the integer number of reference blocks. In this case, the motion compensation unit 72 can determine the interpolation filters used by the video encoder 20 from the received syntax elements and use the interpolation filters to produce predictive blocks.

[0157] In addition, motion compensation unit 72 can be configured to perform any or all of the techniques of this disclosure (alone or in any combination). For example, motion compensation unit 72 can be configured to perform the BIO techniques discussed herein.

[0158] The inverse quantization unit 76 quantizes by inversion, for example, it dequantizes the quantized transform coefficients presented in the bitstream and decoded by the entropy decoding unit 70. The inverse quantization process may include the use of a quantization parameter QPY calculated by the video decoder 30 for each video block in the video slice in order to determine the degree of quantization and, similarly, the degree of inverse quantization that should be applied.

[0159] The inverse transform unit 78 applies an inverse transform, such as an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process, to the transform coefficients in order to produce residual blocks in the pixel domain.

[0160] After the motion compensation unit 72 generates the predictive block for the video block Petition 870200158161, dated 12 / 17 / 2020, pages 81 / 115 Based on motion vectors and other syntax elements, the video decoder 30 forms a decoded video block by summing the residual blocks from the inverse transform unit 78 to the corresponding predictive blocks generated by the motion compensation unit 72. The adder 80 represents the component or components that perform this summation operation. If desired, a deblocking filter can also be applied to filter the decoded blocks in order to remove blocking artifacts. Other mesh filters (either in the encoding mesh or after the encoding mesh) can also be used to smooth transitions between pixels or otherwise improve video quality. The decoded video blocks in a given frame or image are then stored in the reference frame memory 82 which stores the reference images used in subsequent motion compensation.The reference image memory 82 also stores decoded video for later presentation on a display device, such as display device 32 in Figure 1. For example, the reference image memory 82 can store decoded images.

[0161] Figure 16 is a flowchart illustrating an example of the operation of a video decoder for decoding video data according to a technique of this disclosure. The video decoder described in relation to Figure 16 may, for example, be a video decoder, such as video decoder 30, for the production of displayable decoded video, or it may be a video decoder implemented in a video encoder, such as the Petition 870200158161, dated 12 / 17 / 2020, pp. 82 / 115 72 / 78 video encoder decoding mesh 20, which includes an inverse quantization unit, an inverse transformation processing unit 60, an adder 62 and a reference image memory 64, as well as parts of the mode selection unit 40.

[0162] According to the techniques in Figure 16, the video decoder determines that a block of video data be encoded using a bidirectional interpretation mode (200). The video decoder determines a first motion vector for the block that points to a first reference image (202). The video decoder determines a second MV for the block that points to a second reference image, the first reference image being different from the second reference image (204). The video decoder uses the first MV to locate a first predictive block in the first reference image (206). The video decoder uses the second MV to locate a second predictive block in the second reference image (208).

[0163] The video decoder determines a first BIO quantity of motion for a first subblock of the first predictive block (210). The first subblock may be different from a coding unit, a prediction unit, and a transform unit for the block. To determine the first BIO quantity of motion, the video decoder may, in some instances, determine the first BIO quantity of motion based on samples in the first subblock and samples outside the first subblock, and in other instances, determine the first BIO quantity of motion based only on Petition 870200158161, dated 12 / 17 / 2020, pp. 83 / 115 73 / 78 samples in the first sub-block. The first BIO momentum quantity may include, for example, a motion vector field that includes a horizontal component and a vertical component.

[0164] The video decoder determines a first final predictive sub-block for the video data block based on the first sub-block of the first predictive block, a first sub-block of the second predictive block, and the first BIO motion value (212). To determine the first final predictive sub-block for the video data block based on the first sub-block of the first predictive block, the first sub-block of the second predictive block, and the first BIO motion quantity, the video decoder can determine the first final predictive sub-block using, for example, equation (2) above.

[0165] The video decoder determines a second BIO quantity of motion for a second subblock of the first predictive block (214). The second subblock may be different from a coding unit, a prediction unit, and a transform unit for the block. To determine the second BIO quantity of motion, the video decoder may, in some instances, determine the second BIO quantity of motion based on samples in the second subblock and samples outside the second subblock, and in another instance, determine the second BIO quantity of motion based only on samples in the second subblock. The second BIO quantity of motion may, for example, include a motion vector field that includes a horizontal component and a vertical component. Petition 870200158161, dated 12 / 17 / 2020, pp. 84 / 115 74 / 78

[0166] The video decoder determines a second final predictive sub-block for the video data block based on the second sub-block of the first predictive block, a second sub-block of the second predictive block and the second BIO motion value (216). The video decoder can, for example, determine the second final predictive sub-block using, for example, equation (2) to determine the second final predictive sub-block for the video data block based on the second sub-block of the first predictive block, a second sub-block of the second predictive block and the second BIO motion volume.

[0167] The video decoder determines a final predictive block for the video data block based on the first final predictive sub-block and the second final predictive sub-block (218). The video decoder can, for example, add residual data to the final predictive block to determine a reconstructed block for the video data block. The video decoder can also perform one or more filtering processes on the reconstructed video data block.

[0168] The video decoder produces a video data image comprising a decoded version of the video data block (220). When decoding is performed as part of a decoding circuit of a video encoding process, the video decoder may, for example, produce the image by storing the image in a reference image memory and the video decoder may use the image as a reference image in encoding another image from the video data. When the video decoder is a Petition 870200158161, dated 12 / 17 / 2020, pages 85 / 115 The 75 / 78 video decoder is configured to produce displayable decoded video, so the video decoder can, for example, send the image from the video data to a display device.

[0169] It should be recognized that, depending on the example, certain acts or events of any of the techniques described here may be executed in a different sequence, may be added, merged, or left out entirely (not all acts or events described are necessary for the practice of the techniques, for example). Furthermore, in certain examples, the acts or events may be executed concurrently, such as, for example, through processing with multiple execution streams, processing with interrupts, or multiple processors, and not sequentially.

[0170] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored or transmitted as one or more instructions in code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media that correspond to a tangible medium such as a data storage medium or a communication medium that includes any means that facilitates the transfer of a computer program from one place to another, according to a communication protocol, for example. In this way, computer-readable media may Petition 870200158161, dated 12 / 17 / 2020, pages 86 / 115 76 / 78 generally correspond to (1) tangible, non-transient, computer-readable storage media or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more processors or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0171] By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. In addition, any connection is appropriately termed a computer-readable medium. For example, if instructions are transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave are included in the definition of medium.It should be understood, however, that computer-readable storage media and media of... Petition 870200158161, dated 12 / 17 / 2020, pages 87 / 115 77 / 78 Data storage does not include connections, carrier waves, signals or other transient media, but instead is directed towards tangible, non-transient storage media. Disc (disk and disc, in the original) as used here includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where discs usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of these elements must be included within the scope of computer-readable media.

[0172] The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term processor used here may refer to any of the preceding structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some respects, the functionality described herein may be presented within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined CODEC. In addition, the techniques may be fully implemented in one or more logic circuits or elements.

[0173] The techniques of this disclosure can be implemented in a wide variety of devices or appliances, including a cordless telephone, a Petition 870200158161, dated 12 / 17 / 2020, pages 88 / 115 78 / 78 integrated circuit (IC) or a set of ICs (a set of chips, for example). Several components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require execution by different hardware units. Instead, as described above, several units can be combined into a CODEC hardware unit or provided by a collection of interoperable hardware units, which include one or more processors described above, together with suitable software and / or firmware.

[0174] Several examples have been described. These and other examples are within the scope of the following claims. Petition 870200158161, dated 12 / 17 / 2020, pp. 89 / 115< / k> < / k>

Claims

1 / 7 CLAIMS 1. A method for decoding video data, the method characterized in that it comprises: determining that a block of video data is encoded using a bidirectional interprediction mode; determining a first motion vector, MV, for the block, wherein the first MV points to a first reference image; determining a second MV for the block, wherein the second MV points to a second reference image, the first reference image being different from the second reference image; using the first MV, which locates a first predictive block in the first reference image; using the second MV, which locates a second predictive block in the second reference image; for a first 4x4 sub-block of the first predictive block, determining a first bidirectional optical flow motion quantity, BIO, based on gradient values ​​of samples from the first 4x4 sub-block;To determine a first final 4x4 predictive sub-block for the video data block based on the first 4x4 sub-block of the first predictive block, a first 4x4 sub-block of the second predictive block, and the first BIO motion quantity, wherein determining the first final 4x4 predictive sub-block for the video data block based on the first 4x4 sub-block of the first predictive block, the first 4x4 sub-block of the second predictive block, and the first BIO motion quantity comprises, for each sample of the first final 4x4 predictive sub-block, determining a sample value of the first final 4x4 predictive sub-block based on colocalized samples of the first 4x4 sub-block of the first predictive block and the first Petition 870250009304, dated 04 / 02 / 2025, p. 14 / 21 2 / 7 sub-block 4x4 of the second predictive block using the first BIO momentum quantity;For a second 4x4 sub-block of the first predictive block, determine a second BIO momentum quantity, based on gradient values ​​of samples in the second 4x4 sub-block; To determine a second final 4x4 predictive sub-block for the video data block based on the second 4x4 sub-block of the first predictive block, a second 4x4 sub-block of the second predictive block, and the second BIO quantity of motion, wherein determining the second final 4x4 predictive sub-block for the video data block based on the second 4x4 sub-block of the first predictive block, the second 4x4 sub-block of the second predictive block, and the second BIO quantity of motion comprises, for each sample of the second final 4x4 predictive sub-block, determining a sample value of the second final 4x4 predictive sub-block based on colocalized samples of the second 4x4 sub-block of the first predictive block and the second 4x4 sub-block of the second predictive block using the second BIO quantity of motion;Based on the first final predictive 4x4 sub-block and the second final predictive 4x4 sub-block, determine a final predictive block for the video data block; and output a video data image comprising a decoded version of the video data block.

2. Method, according to claim 1, characterized in that determining the first BIO momentum comprises determining the first BIO momentum based only on samples in the first 4x4 sub-block.

3. Method, according to claim 1, characterized by the fact that Petition 870250009304, dated 04 / 02 / 2025, page 15 / 21 3 / 7 that determining the second BIO momentum quantity comprises determining the second BIO momentum quantity based only on samples in the second sub-block.

4. Method, according to claim 1, characterized in that the first BIO momentum quantity comprises a motion vector field comprising a horizontal component and a vertical component.

5. Method, according to claim 1, characterized in that the first 4x4 sub-block is different from one encoding unit, one prediction unit, and one transform unit for the block.

6. Method, according to claim 1, characterized in that it further comprises: adding residual data to the final predictive block to determine a reconstructed block for the video data block.

7. Method, according to claim 1, characterized in that determining the first final 4x4 predictive sub-block for the video data block based on the first 4x4 sub-block of the first predictive block, the first 4x4 sub-block of the second predictive block, and the first BIO momentum comprises determining the first final 4x4 predictive sub-block according to the equation: prec^lo = 1 / 2 · ( / °) ++ νΛ. · {rxdl^ / õx—rtídl^ / âr)+vv · (τβΐ^ / õy - roôl^ / dy)) wherein: predBio comprises a sample value of the first final predictive sub-block; (0) I comprises a sample value of the first 4x4 sub-block of the first predictive block; (1) I includes a sample value from the first 4x4 sub-block of Petition 870250009304, dated 04 / 02 / 2025, page.16 / 21 4 / 7 second predictive block; Vx comprises a horizontal component of the first BIO momentum; Vy comprises a vertical component of the first BIO momentum; to comprises a distance to the first reference image, and Ti comprises a distance to the second reference image.

8. A method according to claim 1, characterized in that the decoding method is performed as part of a decoding loop of a video encoding process, and wherein emitting the video data image comprising the decoded version of the video data block comprises storing the video data image comprising the decoded version of the video data block in a reference image memory, the method further comprising: using the video data image comprising the decoded version of the video data block as a reference image in encoding another video data image.

9. Method, according to claim 1, characterized in that emitting the video data image comprising the decoded version of the video data block comprises emitting the video data image comprising the decoded version of the video data block to a display device.

10. Computer-readable memory characterized in that it stores instructions which, when executed by one or more processors, cause the one or more processors to perform the method as defined in any one of claims 1 to 9.

11. Apparatus for decoding video data, the apparatus characterized in Petition 870250009304, dated 04 / 02 / 2025, page 17 / 21 5 / 7, by the fact that it comprises: means for determining that a block of video data is encoded using a bidirectional interprediction mode; means for determining a first motion vector, MV, for the block, wherein the first MV points to a first reference image; means for determining a second MV for the block, wherein the second MV points to a second reference image, the first reference image being different from the second reference image; means for locating a first predictive block in the first reference image using the first MV; means for locating a second predictive block in the second reference image using the second MV;means to determine a first quantity of bidirectional optical flux momentum, BIO, for a first 4x4 subblock of the first predictive block, based on gradient values ​​of samples from the first 4x4 subblock;means to determine a first final predictive 4x4 subblock for the video data block based on the first 4x4 subblock of the first predictive block, a first 4x4 subblock of the second predictive block, and the first BIO motion quantity, wherein determining the first final predictive 4x4 subblock for the video data block based on the first 4x4 subblock of the first predictive block, the first 4x4 subblock of the second predictive block, and the first BIO motion quantity comprises, for each sample of the first final predictive 4x4 subblock, determining a sample value of the first final predictive 4x4 subblock based on colocalized samples of the first 4x4 subblock of the first predictive block and the first 4x4 subblock of the second predictive block using the first quantity of Petition 870250009304, of 04 / 02 / 2025, page 18 / 21 6 / 7 BIO movement;means to determine a second BIO momentum quantity for a second 4x4 sub-block of the first predictive block, based on gradient values ​​of samples from the second 4x4 sub-block;means to determine a second final predictive 4x4 subblock for the video data block based on the second 4x4 subblock of the first predictive block, a second 4x4 subblock of the second predictive block, and the second BIO quantity of motion, wherein determining the second final predictive 4x4 subblock for the video data block based on the second 4x4 subblock of the first predictive block, the second 4x4 subblock of the second predictive block, and the second BIO quantity of motion comprises, for each sample of the second final predictive 4x4 subblock, determining a sample value of the second final predictive 4x4 subblock based on colocalized samples of the second 4x4 subblock of the first predictive block and the second 4x4 subblock of the second predictive block using the second BIO quantity of motion; means to determine a final predictive block for the video data block based on the first final predictive 4x4 sub-block and the second final predictive 4x4 sub-block;and means for outputting a video data image comprising a decoded version of the video data block.

12. Apparatus, according to claim 11, characterized in that the apparatus comprises a wireless communication device, further comprising a receiver configured to receive encoded video data; and preferably wherein the wireless communication device comprises a telephone apparatus and wherein the receiver is configured to demodulate, in accordance with a wireless communication standard, a signal comprising encoded video data.

13. Apparatus, according to claim 11, characterized in that the apparatus comprises a wireless communication device, further comprising a transmitter configured to transmit encoded video data; and preferably wherein the wireless communication device comprises a telephone apparatus and wherein the transmitter is configured to modulate, according to a wireless communication standard, a signal comprising encoded video data. Petition 870250009304, dated 04 / 02 / 2025, pp. 20 / 21

Images