Reference frame motion field selection for wedge mode blocks

Abstract

A compound prediction block for a current block is obtained based on a weight mask, a first motion vector, and a second motion vector. The current block is partitioned into units based on a motion vector granularity. For each of the units, the first motion vector, the second motion vector, or both of the first motion vector and the second motion vector is stored in association with the each unit based on a respective portion of the weight mask that is co-extensive with each unit.

Description

REFERENCE FRAME MOTION FIELD SELECTION FOR WEDGE MODEBLOCKSCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and the benefit of U.S. Provisional Patent Application U.S. Serial No. 63 / 667,881, filed July 05, 2024, the entire disclosure of which is incorporated herein by reference.BACKGROUND

[0002] Digital video streams may represent video using a sequence of frames or still images. Digital video can be used for various applications including, for example, video conferencing, high-definition video entertainment, video advertisements, or sharing of usergenerated videos. A digital video stream can contain a large amount of data and consume a significant amount of computing or communication resources of a computing device for processing, transmission, or storage of the video data. Various approaches have been proposed to reduce the amount of data in video streams, including encoding or decoding techniques.SUMMARY

[0003] One aspect of the disclosed implementations relates to a method, including: obtaining a compound prediction block for a current block based on a weight mask, a first motion vector, and a second motion vector; partitioning the current block into units based on a motion vector granularity; and for each of the units, storing, in association with the each unit and based on a respective portion of the weight mask that is co-extensive with the each unit, the first motion vector, the second motion vector, or both of the first motion vector and the second motion vector.

[0004] One aspect of the disclosed implementations relates to the method, where storing, in association with the each unit and based on the respective portion of the weight mask that is co-extensive with the each unit, the first motion vector, the second motion vector, or both of the first motion vector and the second motion vector includes: on a condition that the respective portion of the weight mask for the each unit has an average weight greater than or equal to a first threshold, storing the first motion vector in association with the each unit; on acondition that the average weight is less than or equal to a second threshold, storing the second motion vector in association with the each unit; and on a condition that the average weight is between the first threshold and the second threshold, storing both of the first motion vector and the second motion vector in association with the each unit.

[0005] One aspect of the disclosed implementations relates to the method, where storing, in association with the each unit and based on the respective portion of the weight mask that is co-extensive with the each unit, the first motion vector, the second motion vector, or both of the first motion vector and the second motion vector includes: determining a count of pixels of the each unit associated with the first motion vector; and storing, in association with the each unit, the first motion vector, the second motion vector, or both of the first motion vector and the second motion vector based on the count.

[0006] One aspect of the disclosed implementations relates to the method, further including: obtaining the weight mask based on an index associated with the weight mask decoded from a compressed bitstream.

[0007] One aspect of the disclosed implementations relates to the method, further including: encoding an index associated with the weight mask in a compressed bitstream.

[0008] One aspect of the disclosed implementations relates to the method, where a size of the each unit is 8x8.

[0009] One aspect of the disclosed implementations relates to the method, further including: updating a reference frame motion field with the stored motion vectors for each unit of the current block.

[0010] One aspect of the disclosed implementations relates to the method, where the weight mask is generated based on an angle of a wedge that separates the current block into two parts, each part being associated with one of the first motion vector and the second motion vector.

[0011] One aspect of the disclosed implementations relates to the method, where obtaining the compound prediction block includes: generating a first prediction block using the first motion vector and a first reference frame; generating a second prediction block using the second motion vector and a second reference frame; and combining the first prediction block and the second prediction block using the weight mask to generate the compound prediction block.

[0012] One aspect of the disclosed implementations relates to the method, where the weight mask is associated with a wedge that divides the current into two parts, where the weight mask includes a soft transition region between the two parts of the current block, andwhere storing the motion vectors for each unit is based on a distribution of weights within the transition region.

[0013] These and other aspects of the present disclosure are disclosed in the following detailed description of the implementations, the appended claims and the accompanying figures. It will be appreciated that aspects can be implemented in any convenient form. For example, aspects may be implemented by appropriate computer programs which may be carried on appropriate carrier media which may be tangible carrier media (e.g. disks) or intangible carrier media (e.g. communications signals). Aspects may also be implemented using suitable apparatus which may take the form of programmable computers running computer programs arranged to implement the methods and / or techniques disclosed herein. Aspects can be combined such that features described in the context of one aspect may be implemented in another aspect.BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The description herein makes reference to the accompanying drawings described below, wherein like reference numerals refer to like parts throughout the several views.

[0015] FIG. 1 is a schematic of a video encoding and decoding system.

[0016] FIG. 2 is a block diagram of an example of a computing device that can implement a transmitting station or a receiving station.

[0017] FIG. 3 is a diagram of a typical video stream to be encoded and subsequently decoded.

[0018] FIG. 4 is a block diagram of an encoder according to implementations of this disclosure.

[0019] FIG. 5 is a block diagram of a decoder according to implementations of this disclosure.

[0020] FIG. 6 is an illustration of examples of portions of a video frame.

[0021] FIG. 7 is an illustration of an example of a grid of blocks of a motion field.

[0022] FIG. 8 is a diagram of an example of a reference frame buffer.

[0023] FIG. 9 illustrates a codebook of possible partition lines in wedge-based compound prediction.

[0024] FIG. 10 illustrates a use case demonstrating the application of wedge-based compound prediction.

[0025] FIG. 11 is a flowchart of a technique for coding a current block.

[0026] FIG. 12 shows an example illustrating the concept of associating motion vectors with units within a block.DETAILED DESCRIPTION

[0027] As mentioned above, compression schemes related to coding video streams may include breaking images into blocks and generating a digital video output bitstream (i.e., an encoded bitstream) using one or more techniques to limit the information included in the output bitstream. A received bitstream can be decoded to re-create the blocks and the source images from the limited information. Encoding a video stream, or a portion thereof, such as a frame or a block, can include using temporal or spatial similarities in the video stream to improve coding efficiency. For example, a current block of a video stream may be encoded based on identifying a difference (residual) between the previously coded pixel values, or between a combination of previously coded pixel values, and those in the current block.

[0028] Encoding using spatial similarities is known as intra prediction. Intra prediction attempts to predict the pixel values of a block of a frame of a video stream using reconstructed peripheral pixels. Peripheral pixels are in the same frame as the block but are outside the block. A prediction block resulting from intra prediction is referred to herein as an intra predictor. Intra prediction may be performed along a direction of prediction where each direction can correspond to an intra prediction mode. The intra prediction mode can be signaled by an encoder to a decoder.

[0029] Encoding using temporal similarities is known as inter prediction. Inter prediction attempts to predict the pixel values of a block using a possibly displaced block or blocks from a temporally nearby frame (i.e., reference frame) or frames. A temporally nearby frame is a frame that appears earlier or later in time in the video stream than the frame of the block being encoded. A prediction block resulting from inter prediction is referred to herein as inter predictor. Inter prediction is performed using a motion vector (MV). An MV used to generate a prediction block refers to a frame (referred to as a reference frame) other than a current frame. Reference frames can be located before or after the current frame in the sequence of the video stream. Some codecs use up to eight reference frames, which can be stored in a frame buffer. The MV can point to a location within one of the reference frames stored in the frame buffer.

[0030] Two predictor blocks can be combined to form a compound prediction block for a current block or region of a video image. A compound prediction block can be created by combining two or more prediction blocks determined using, for example, the aforementionedprediction methods (i.e., inter and / or intra prediction). For example, a compound prediction block can be combination of a first prediction block and a second prediction block which can be two intra prediction blocks (i.e., intra+intra), an intra prediction block and an inter prediction block (i.e., intra+inter) or two inter prediction blocks (i.e., inter+inter).

[0031] In the case of inter+inter, compound inter prediction can employ a first MV to obtain a first prediction block from a first reference frame, and a second MV to obtain a second prediction block from a second reference frame. The reference frames can both be in the past, both in the future, or some combination thereof. The two prediction blocks are then combined to obtain a compound prediction block. Different ways of combining the two prediction blocks are possible.

[0032] One compound prediction mode used (e.g., implemented) by certain codecs is referred to as the wedge-based compound prediction (or, simply, the wedge mode). The wedge-based compound prediction implements a specific way of combining two prediction blocks. In the wedge mode, a “wedge” (e.g., a straight or oblique line) that separates the current block into two parts is identified. Each part of the current block may be better predicted by one of the MVs than by the other MV. A weight mask is selected (e.g., generated, identified, or calculated) in accordance with an angle and an offset of the wedge, and then the compound prediction block is generated based on the given weight mask and the two prediction blocks.

[0033] The angle defines the orientation of the wedge line. In some implementations, the offset may refer to a displacement parameter that defines where the wedge line intersects the block boundary (e.g., shifting the wedge line left or right across the block). In some implementations, the offset may provide additional granularity in selecting the orientation of the wedge mask, effectively refining the angle or applying an adjusted variant of a base angle mask. Both interpretations refer to selecting or positioning a specific wedge pattern from a codebook of candidate masks.

[0034] An encoder selects the wedge from a set of wedges and transmits an index (e.g., an identifier of the wedge) to the decoder in a compressed bitstream. The decoder decodes the index from the compressed bitstream. The index of the wedge corresponds to (e.g., identifies) the weight mask. In an addition to the wedge index, the encoder transmits, and the decoder decodes, the two MVs required to generate the two prediction blocks used in the wedge mode.

[0035] As a separate concept, MVs used in the prediction of a current block are stored into a reference frame motion field, which may be used for MV prediction and other codingtools. The reference frame motion field is in units of blocks having a minimal block unit size of MxN (e.g., where M and N are positive integers that may be equal (e.g., 8x8) or different (e.g., 8x16)). Each unit stores the MV(s) of that NxN block. This minimal block unit size defines what is referred to as the motion vector granularity.

[0036] Accordingly, “motion vector granularity” refers to the smallest spatial unit size (e.g., 8x8 pixels) for which motion vectors are individually calculated and stored in the reference frame motion field. Each unit in the motion field corresponds to a spatial region of the video frame and stores the motion vector(s) associated with the portion of the coding block that overlaps that unit . When a coding block is larger than the minimal block unit size, the motion field units that the coding block covers will all be updated (e.g., associated) with the MV(s) of the coding block. To illustrate, assume that a coding block is 16x16 pixels and that the minimal block unit size is 8x8 pixels. In this case, the coding block covers four 8x8 units in the motion field. Consequently, the MV(s) of the 16x16 coding block will be stored in (e.g., in association with) each of the four 8x8 units.

[0037] To give a more concrete example, a decoder may determine that a current block is to be decoded using an inter-prediction mode. The decoder obtains (e.g., selects, infers, or decodes) at least one MV to generate a prediction block of the block. The decoder may decode a residual block from a compressed bitstream. The residual block is added to the prediction block to obtain the current block (e.g., a reconstructed block that may further filtered, as described herein). The at least one MV is then associated with each block unit of the current block.

[0038] In the case of the wedge-based compound prediction, storing both MVs in association with each unit of a block is inefficient and wasteful. This inefficiency arises because, in the wedge mode, and depending on the weight mask used, some portions of the current block (or the compound prediction block) may be influenced solely by one of the MVs (e.g., by one of the prediction blocks), other portions may be influenced solely by the other MV, and other portions will be influenced, to different extents, by both MVs. When combining the two prediction blocks, particularly around the pixels near the wedge, the influence of both prediction blocks is considered based on the wedge mask. Storing both MVs for each unit within the coding block leads to unnecessary and wasteful storage, as the actual impact of each motion vector is limited to its corresponding region.

[0039] To illustrate, consider a 32x32 current block divided by a wedge into two regions. The wedge creates a boundary where both MVs influence the prediction, but only near the boundary. If each of the sixteen 8x8 units within the block redundantly stores both motionvectors (MVs), then 32 MVs will be stored (i.e., two MVs per 8x8 unit), even though most of the units do not require both for accurate prediction. Storing both MVs with every minimal block unit leads to increased memory usage and computational overhead without providing additional predictive accuracy.

[0040] Implementations according to the disclosure solve problems such as these by storing each of the two MVs, in wedge-based compound prediction, only with respect to the relevant minimal block units, thus optimizing storage and computational resources. Instead of saving both MVs to every unit, the MV(s) most suitable for a unit of a coding block are saved with that part of the block. Since each unit can cover an MxN (e.g., 8x8) area, it is still possible that it would cover an area that can benefit from both MVs.

[0041] In some implementations, a compound prediction block for a current block is obtained based on a weight mask, a first motion vector, and a second motion vector; the current block is partitioned into units based on a motion vector granularity; and, for each of the units, the first motion vector, the second motion vector, or both of the first motion vector and the second motion vector is(are) stored in association with the each unit based on a respective portion of the weight mask that is co-extensive with (e.g., corresponding to the spatial boundaries of, or having the same spatial extent as) the each unit.

[0042] Further details of techniques for reference frame motion field selection for wedge mode blocks are described herein with initial reference to a system in which they can be implemented. FIG. 1 is a schematic of a video encoding and decoding system 100. A transmitting station 102 can be, for example, a computer having an internal configuration of hardware such as that described in FIG. 2. However, other implementations of the transmitting station 102 are possible. For example, the processing of the transmitting station 102 can be distributed among multiple devices.

[0043] A network 104 can connect the transmitting station 102 and a receiving station 106 for encoding and decoding of the video stream. Specifically, the video stream can be encoded in the transmitting station 102, and the encoded video stream can be decoded in the receiving station 106. The network 104 can be, for example, the Internet. The network 104 can also be a local area network (LAN), wide area network (WAN), virtual private network (VPN), cellular telephone network, or any other means of transferring the video stream from the transmitting station 102 to, in this example, the receiving station 106.

[0044] The receiving station 106, in one example, can be a computer having an internal configuration of hardware such as that described in FIG. 2. However, other suitableimplementations of the receiving station 106 are possible. For example, the processing of the receiving station 106 can be distributed among multiple devices.

[0045] Other implementations of the video encoding and decoding system 100 are possible. For example, an implementation can omit the network 104. In another implementation, a video stream can be encoded and then stored for transmission at a later time to the receiving station 106 or any other device having memory. In one implementation, the receiving station 106 receives (e.g., via the network 104, a computer bus, and / or some communication pathway) the encoded video stream and stores the video stream for later decoding. In an example implementation, a real-time transport protocol (RTP) is used for transmission of the encoded video over the network 104. In another implementation, a transport protocol other than RTP may be used (e.g., a Hypertext Transfer Protocol-based (HTTP-based) video streaming protocol).

[0046] When used in a video conferencing system, for example, the transmitting station 102 and / or the receiving station 106 may include the ability to both encode and decode a video stream as described below. For example, the receiving station 106 could be a video conference participant who receives an encoded video bitstream from a video conference server (e.g., the transmitting station 102) to decode and view and further encodes and transmits his or her own video bitstream to the video conference server for decoding and viewing by other participants.

[0047] FIG. 2 is a block diagram of an example of a computing device 200 that can implement a transmitting station or a receiving station. For example, the computing device 200 can implement one or both of the transmitting station 102 and the receiving station 106 of FIG. 1. The computing device 200 can be in the form of a computing system including multiple computing devices, or in the form of one computing device, for example, a mobile phone, a tablet computer, a laptop computer, a notebook computer, a desktop computer, and the like.

[0048] A processor 202 in the computing device 200 can be a conventional central processing unit. Alternatively, the processor 202 can be another type of device, or multiple devices, capable of manipulating or processing information now existing or hereafter developed. For example, although the disclosed implementations can be practiced with one processor as shown (e.g., the processor 202), advantages in speed and efficiency can be achieved by using more than one processor.

[0049] A memory 204 in computing device 200 can be a read only memory (ROM) device or a random-access memory (RAM) device in an implementation. However, othersuitable types of storage device can be used as the memory 204. The memory 204 can include code and data 206 that is accessed by the processor 202 using a bus 212. The memory 204 can further include an operating system 208 and application programs 210, the application programs 210 including at least one program that permits the processor 202 to perform the techniques described herein. For example, the application programs 210 can include applications 1 through N, which further include a video coding application that performs, amongst other things, reference frame motion field selection for wedge mode blocks, as described herein. The computing device 200 can also include a secondary storage 214, which can, for example, be a memory card used with a mobile computing device. Because the video communication sessions may contain a significant amount of information, they can be stored in whole or in part in the secondary storage 214 and loaded into the memory 204 as needed for processing.

[0050] In some implementations, the code and data 206, the operating system 208, and the application programs 210 may be stored on a non-transitory computer-readable storage medium. The term “non-transitory” excludes transitory signals and refers to media such as hard drives, flash memory, ROM, and other physical storage devices capable of storing executable instructions. Such a non-transitory computer-readable storage medium may contain instructions that, when executed by the processor 202, cause the computing device 200 to perform any of the methods or processes described herein, including reference frame motion field selection for wedge mode blocks. The secondary storage 214 is a non-transitory computer-readable storage medium that can store code and data, including machine-readable instructions that, when executed by the processor 202, cause the computing device 200 to perform one or more of the techniques described herein.

[0051] The computing device 200 can also include one or more output devices, such as a display 218. The display 218 may be, in one example, a touch sensitive display that combines a display with a touch sensitive element that is operable to sense touch inputs. The display 218 can be coupled to the processor 202 via the bus 212. Other output devices that permit a user to program or otherwise use the computing device 200 can be provided in addition to or as an alternative to the display 218. When the output device is or includes a display, the display can be implemented in various ways, including by a liquid crystal display (LCD), a cathode-ray tube (CRT) display, or a light emitting diode (LED) display, such as an organic LED (OLED) display.

[0052] The computing device 200 can also include or be in communication with an image-sensing device 220, for example, a camera, or any other image-sensing device 220now existing or hereafter developed that can sense an image such as the image of a user operating the computing device 200. The image-sensing device 220 can be positioned such that it is directed toward the user operating the computing device 200. In an example, the position and optical axis of the image-sensing device 220 can be configured such that the field of vision includes an area that is directly adjacent to the display 218 and from which the display 218 is visible.

[0053] The computing device 200 can also include or be in communication with a soundsensing device 222, for example, a microphone, or any other sound-sensing device now existing or hereafter developed that can sense sounds near the computing device 200. The sound-sensing device 222 can be positioned such that it is directed toward the user operating the computing device 200 and can be configured to receive sounds, for example, speech or other utterances, made by the user while the user operates the computing device 200.

[0054] Although FIG. 2 depicts the processor 202 and the memory 204 of the computing device 200 as being integrated into one unit, other configurations can be utilized. The operations of the processor 202 can be distributed across multiple machines (wherein individual machines can have one or more processors) that can be coupled directly or across a local area or other network. The memory 204 can be distributed across multiple machines such as a network-based memory or memory in multiple machines performing the operations of the computing device 200. Although depicted here as one bus, the bus 212 of the computing device 200 can be composed of multiple buses. Further, the secondary storage 214 can be directly coupled to the other components of the computing device 200 or can be accessed via a network and can comprise an integrated unit such as a memory card or multiple units such as multiple memory cards. The computing device 200 can thus be implemented in a wide variety of configurations.

[0055] FIG. 3 is a diagram of an example of a video stream 300 to be encoded and subsequently decoded. The video stream 300 includes a video sequence 302. At the next level, the video sequence 302 includes a number of adjacent frames 304. While three frames are depicted as the adjacent frames 304, the video sequence 302 can include any number of adjacent frames 304. The adjacent frames 304 can then be further subdivided into individual frames, for example, a frame 306. At the next level, the frame 306 can be divided into a series of planes or segments 308. The segments 308 can be subsets of frames that permit parallel processing, for example. The segments 308 can also be subsets of frames that can separate the video data into separate colors. For example, a frame 306 of color video data can includea luminance plane and two chrominance planes. The segments 308 may be sampled at different resolutions.

[0056] Whether or not the frame 306 is divided into segments 308, the frame 306 may be further subdivided into blocks 310, which can contain data corresponding to, for example, 16x16 pixels in the frame 306. The blocks 310 can also be arranged to include data from one or more segments 308 of pixel data. The blocks 310 can also be of any other suitable size such as 4x4 pixels, 8x8 pixels, 16x8 pixels, 8x16 pixels, 16x16 pixels, or larger. Unless otherwise noted, the terms block and macroblock are used interchangeably herein.

[0057] FIG. 4 is a block diagram of an encoder 400 according to implementations of this disclosure. The encoder 400 can be implemented, as described above, in the transmitting station 102, such as by providing a computer software program stored in memory, for example, the memory 204. The computer software program can include machine instructions that, when executed by a processor such as the processor 202, cause the transmitting station 102 to encode video data in the manner described in FIG. 4. The encoder 400 can also be implemented as specialized hardware included in, for example, the transmitting station 102. In one particularly desirable implementation, the encoder 400 is a hardware encoder.

[0058] The encoder 400 has the following stages to perform the various functions in a forward path (shown by the solid connection lines) to produce an encoded or compressed bitstream 420 using the video stream 300 as input: an intra / inter prediction stage 402, a transform stage 404, a quantization stage 406, and an entropy encoding stage 408. The encoder 400 may also include a reconstruction path (shown by the dotted connection lines) to reconstruct a frame for encoding of future blocks. In FIG. 4, the encoder 400 has the following stages to perform the various functions in the reconstruction path: a dequantization stage 410, an inverse transform stage 412, a reconstruction stage 414, and a loop filtering stage 416. Other structural variations of the encoder 400 can be used to encode the video stream 300.

[0059] When the video stream 300 is presented for encoding, respective adjacent frames 304, such as the frame 306, can be processed in units of blocks. At the intra / inter prediction stage 402, respective blocks can be encoded using intra-frame prediction (also called intraprediction) or inter- frame prediction (also called inter-prediction). In any case, a prediction block can be formed. In the case of intra-prediction, a prediction block may be formed from samples in the current frame that have been previously encoded and reconstructed. In the case of inter-prediction, a prediction block may be formed from samples in one or more previously constructed reference frames.

[0060] Next, the prediction block can be subtracted from the current block at the intra / inter prediction stage 402 to produce a residual block (also called a residual). The transform stage 404 transforms the residual into transform coefficients in, for example, the frequency domain using block-based transforms. The quantization stage 406 converts the transform coefficients into discrete quantum values, which are referred to as quantized transform coefficients, using a quantizer value or a quantization level. For example, the transform coefficients may be divided by the quantizer value and truncated.

[0061] The quantized transform coefficients are then entropy encoded by the entropy encoding stage 408. The entropy-encoded coefficients, together with other information used to decode the block (which may include, for example, syntax elements such as used to indicate the type of prediction used, transform type, motion vectors, a quantizer value, or the like), are then output to the compressed bitstream 420. The compressed bitstream 420 can be formatted using various techniques, such as variable length coding (VLC) or arithmetic coding. The compressed bitstream 420 can also be referred to as an encoded video stream or encoded video bitstream, and the terms will be used interchangeably herein.

[0062] The reconstruction path (shown by the dotted connection lines) can be used to ensure that the encoder 400 and a decoder 500 (described below with respect to FIG. 5) use the same reference frames to decode the compressed bitstream 420. The reconstruction path performs functions that are similar to functions that take place during the decoding process (described below with respect to FIG. 5), including dequantizing the quantized transform coefficients at the dequantization stage 410 and inverse transforming the dequantized transform coefficients at the inverse transform stage 412 to produce a derivative residual block (also called a derivative residual). At the reconstruction stage 414, the prediction block that was predicted at the intra / inter prediction stage 402 can be added to the derivative residual to create a reconstructed block. The loop filtering stage 416 can be applied to the reconstructed block to reduce distortion such as blocking artifacts.

[0063] Other variations of the encoder 400 can be used to encode the compressed bitstream 420. In some implementations, a non-transform-based encoder can quantize the residual signal directly without the transform stage 404 for certain blocks or frames. In some implementations, an encoder can have the quantization stage 406 and the dequantization stage 410 combined in a common stage.

[0064] FIG. 5 is a block diagram of a decoder 500 according to implementations of this disclosure. The decoder 500 can be implemented in the receiving station 106, for example, by providing a computer software program stored in the memory 204. The computer softwareprogram can include machine instructions that, when executed by a processor such as the processor 202, cause the receiving station 106 to decode video data in the manner described in FIG. 5. The decoder 500 can also be implemented in hardware included in, for example, the transmitting station 102 or the receiving station 106.

[0065] The decoder 500, similar to the reconstruction path of the encoder 400 discussed above, includes in one example the following stages to perform various functions to produce an output video stream 516 from the compressed bitstream 420: an entropy decoding stage 502, a dequantization stage 504, an inverse transform stage 506, an intra / inter prediction stage 508, a reconstruction stage 510, a loop filtering stage 512, and a deblocking filtering stage 514. Other structural variations of the decoder 500 can be used to decode the compressed bitstream 420.

[0066] When the compressed bitstream 420 is presented for decoding, the data elements within the compressed bitstream 420 can be decoded by the entropy decoding stage 502 to produce a set of quantized transform coefficients. The dequantization stage 504 dequantizes the quantized transform coefficients (e.g., by multiplying the quantized transform coefficients by the quantizer value), and the inverse transform stage 506 inverse transforms the dequantized transform coefficients to produce a derivative residual that can be identical to that created by the inverse transform stage 412 in the encoder 400. Using header information decoded from the compressed bitstream 420, the decoder 500 can use the intra / inter prediction stage 508 to create the same prediction block as was created in the encoder 400 (e.g., at the intra / inter prediction stage 402).

[0067] At the reconstruction stage 510, the prediction block can be added to the derivative residual to create a reconstructed block. The loop filtering stage 512 can be applied to the reconstructed block to reduce blocking artifacts. Other filtering can be applied to the reconstructed block. In this example, the deblocking filtering stage 514 is applied to the reconstructed block to reduce blocking distortion, and the result is output as the output video stream 516. The output video stream 516 can also be referred to as a decoded video stream, and the terms will be used interchangeably herein. Other variations of the decoder 500 can be used to decode the compressed bitstream 420. In some implementations, the decoder 500 can produce the output video stream 516 without the deblocking filtering stage 514.

[0068] FIG. 6 is an illustration of examples of portions of a video frame 600, which may, for example, be the frame 306 shown in FIG. 3. The video frame 600 includes a number of 64x64 coding tree units (CTUs), such as four 64x64 CTUs 610 in two rows and two columns in a matrix or Cartesian plane, as shown. Each 64x64 CTU 610 may include up to four 32x32coding units (CUs) 620. Each 32x32 CU 620 may include up to four 16x16 CUs 630. Each 16x16 CU 630 may include up to four 8x8 CUs 640. Each 8x8 CU 640 may include up to four 4x4 CUs 650. Each 4x4 CU 650 may include 16 pixels, which may be represented in four rows and four columns in each respective CU in the Cartesian plane or matrix.

[0069] In some implementations, the video frame 600 may include CTUs larger than 64x64 and / or CUs smaller than 4x4. Subject to features within the video frame 600 and / or other criteria, the video frame 600 may be partitioned into various arrangements. Although one arrangement of CUs is shown, any arrangement may be used. Although FIG. 6 shows NxN CTUs and CUs, in some implementations, NxM CTUs and / or CUs may be used, wherein N and M are different numbers. For example, 32x64 CTUs, 64x32 CTUs, 16x32 CUs, 32x16 CUs, or any other size may be used. In some implementations, Nx2N CTUs or CUs, 2NxN CTUs or CUs, or a combination thereof, may be used.

[0070] The pixels may include information representing an image captured in the video frame 600, such as luminance information, color information, and location information. In some implementations, a block, such as a 16x16 pixel block as shown, may include a luminance block 660, which may include luminance pixels 662; and two chrominance blocks 670, 680, such as a U or Cb chrominance block 670, and a V or Cr chrominance block 680. The chrominance blocks 670, 680 may include chrominance pixels 690. For example, the luminance block 660 may include 16x16 luminance pixels 662 and each chrominance block 670, 680 may include 8x8 chrominance pixels 690 as shown.

[0071] In some implementations, coding the video frame 600 may include ordered blocklevel coding. Ordered block-level coding may include coding CUs of the video frame 600 in an order, such as raster-scan order, wherein CUs may be identified and processed starting with a CTU in the upper left comer of the video frame 600, or portion of the video frame 600, and proceeding along rows from left to right and from the top row to the bottom row, identifying each CU in turn for processing. For example, the 64x64 CTU in the top row and left column of the video frame 600 may be the first CTU coded and the 64x64 CTU immediately to the right of the first CTU may be the second CTU coded. The second row from the top may be the second row coded, such that the 64x64 CTU in the left column of the second row may be coded after the 64x64 CTU in the rightmost column of the first row.

[0072] In some implementations, coding a CTU of the video frame 600 may include using quad-tree coding, which may include coding smaller CUs within a CTU in raster-scan order. For example, the 64x64 CTU shown in the bottom left corner of the portion of the video frame 600 may be coded using quad-tree coding wherein the top left 32x32 CU may becoded, then the top right 32x32 CU may be coded, then the bottom left 32x32 CU may be coded, and then the bottom right 32x32 CU may be coded. Each 32x32 CU may be coded using quad-tree coding wherein the top left 16x16 CU may be coded, then the top right 16x16 CU may be coded, then the bottom left 16x16 CU may be coded, and then the bottom right 16x16 CU may be coded. Each 16x16 CU may be coded using quad-tree coding wherein the top left 8x8 CU may be coded, then the top right 8x8 CU may be coded, then the bottom left 8x8 CU may be coded, and then the bottom right 8x8 CU may be coded. Each 8x8 CU may be coded using quad-tree coding wherein the top left 4x4 CU may be coded, then the top right 4x4 CU may be coded, then the bottom left 4x4 CU may be coded, and then the bottom right 4x4 CU may be coded. In some implementations, 8x8 CUs may be omitted for a 16x16 CU, and the 16x16 CU may be coded using quad-tree coding wherein the top left 4x4 CU may be coded, then the other 4x4 CUs in the 16x16 CU may be coded in raster- scan order.

[0073] In some implementations, coding the video frame 600 may include encoding the information included in the original version of the image or video frame by, for example, omitting some of the information from that original version of the image or video frame from a corresponding encoded image or encoded video frame. For example, the coding may include reducing spectral redundancy, reducing spatial redundancy, or a combination thereof. Reducing spectral redundancy may include using a color model based on a luminance component (Y) and two chrominance components (U and V or Cb and Cr), which may be referred to as the YUV or YCbCr color model, or color space. Using the YUV color model may include using a relatively large amount of information to represent the luminance component of a portion of the video frame 600, and using a relatively small amount of information to represent each corresponding chrominance component for the portion of the video frame 600. For example, a portion of the video frame 600 may be represented by a high-resolution luminance component, which may include a 16x16 block of luma samples, and by two lower resolution chrominance components, each of which represents the portion of the image as an 8x8 block of chroma samples. A sample may indicate a value, for example, a value in the range from 0 to 255, and may be stored or transmitted using, for example, eight bits. Although this disclosure is described in reference to the YUV color model, another color model may be used. Reducing spatial redundancy may include transforming a CU into the frequency domain using, for example, a discrete cosine transform. For example, a unit of an encoder may perform a discrete cosine transform using transform coefficient values based on spatial frequency.Although described herein with reference to matrix or Cartesian representation of the video frame 600 for clarity, the video frame 600 may be stored, transmitted, processed, or a combination thereof, in a data structure such that pixel values and / or luma and chroma samples may be efficiently represented for the video frame 600. For example, the video frame 600 may be stored, transmitted, processed, or any combination thereof, in a two-dimensional data structure such as a matrix as shown, or in a one-dimensional data structure, such as a vector array. Furthermore, although described herein as showing a chrominance subsampled image where U and V have half the resolution of Y, the video frame 600 may have different configurations for the color channels thereof. For example, referring still to the YUV color space, full resolution may be used for all color channels of the video frame 600. In another example, a color space other than the YUV color space may be used to represent the resolution of color channels of the video frame 600.

[0074] FIG. 7 is an illustration of an example of a grid of blocks 700 of a motion field. The grid of blocks 700 includes a number of blocks arranged in a two-dimensional grid. Each of the blocks includes one or more units of pixels (i.e., minimal block units), each of which is an MxN portion of the motion field co-located with a corresponding MxN portion of a frame with which the motion field is associated, in which M and N are positive integers and may be the same or different. In the example shown, the blocks are indicated with bolded boundaries and the units of pixels are shown within respective ones of the blocks. For example, a block 702 is shown as including 16 units of pixels. The block 702 is illustrated as having been predicted using the wedge-based compound prediction based on a wedge 704. As such, two motion vectors ( MV0 and MV1) were used in generating the block 702.

[0075] For each unit of pixels, one or more motion vectors can be stored, as further described herein. In the example shown, at least one of the two motion vectors can be stored for each unit of pixels in which a first motion vector may be stored in a first slot of the unit of pixels and a second motion vector may be stored in a second slot of the unit of pixels. Each slot includes information identifying the motion vector, such as based on a location within a reference frame at which the motion vector points. The location within the reference frame may, for example, refer to a block, unit of pixels, or other portion of that reference frame. The information stored within a slot identifies the corresponding reference frame by frame index. Thus, as shown, each portion may include two slots in which each slot may store one motion vector, and, more particularly, in which each slot stores data indicative of a reference frame to which the motion vector points and a location within that reference frame at which the motion vector points.

[0076] Conventionally, because a block is typically predicted as a whole, all of the units of pixels within the block are usually associated with the same motion vector or vectors. However, as described herein, some of the units will be associated with MVO, some of the units will be associated with MV1, and some of the units (e.g., those around the wedge 704) will be associated with both of MVO and MV1. As shown, the first motion vector MVO points to a location (1, 2) in reference frame index 0 and the second motion vector MV1 points to a location (-1, -2) in a reference frame index 16. FIG. 7 illustrates that only MVO is associated with (e.g., stored in association with) units 706, 708; that both of MVO and MV1 are associated with units 710, 712; and that only MV1 is associated with unit 714.

[0077] FIG. 8 is a diagram of an example of a reference frame buffer 800. As described above with respect to inter prediction, a motion vector identifies a location within a reference frame from which a prediction block is generated. Reference frames can be stored in a reference frame buffer, such as the reference frame buffer 800.

[0078] A current frame, or a current block or region of a frame, can be encoded using a reference frame such as a “last frame,” which is the adjacent frame immediately before the current frame in the video sequence. When video frames are encoded out of order (i.e., not in the sequence that they appear in the video stream), motion information from video frames in the past or future can be included in the candidate reference motion vectors. Encoding video frames can occur, for example, using so-called “alternate reference frames” that are not temporally neighboring to the frames coded immediately before or after them. An alternate reference frame can be a synthesized frame that does not appear in the input video stream, or a duplicate of a frame from the stream that is retained for prediction purposes. An alternate frame may not be displayed following decoding. Such a frame can resemble a video frame in the non-adjacent future. Another example in which out of order encoding may occur is through the use of a so-called “golden frame,” which is a reconstructed video frame that may or may not be neighboring to a current frame and is stored in memory for use as a reference frame until replaced, e.g., by a new golden frame.

[0079] The reference frame buffer 800 stores reference frames used to code blocks of frames of a video sequence. For example, the reference frame buffer 800 can include a last frame LAST_FRAME 802, a golden frame GOLDEN_FRAME 804, and an alternative reference frame ALTREF_FRAME 806. A reference frame buffer can include more, fewer, or other reference frames. The reference frame buffer 800 is shown as including eight reference frames. However, a reference frame buffer can include more or fewer than eight reference frames.

[0080] The last frame LAST_FRAME 802 can be, for example, the adjacent frame immediately before the current frame in the video sequence. The golden frame GOLDEN_FRAME 804 can be, for example, a reconstructed video frame for use as a reference frame that may or may not be adjacent to the current frame. The alternative reference frame ALTREF_FRAME 806 can be a future frame that is not temporally adjacent to the current frame; it may serve as a backward reference frame if used in reverse temporal prediction.

[0081] The reference frames stored in the reference frame buffer 800 can be used to identify motion vectors for predicting blocks of frames to be encoded or decoded. Different reference frames may be used depending on the type of prediction used to predict a current block of a current frame. For example, when compound prediction is used, multiple frames, such as one for forward prediction (e.g., LAST_FRAME 802 or GOLDEN_FRAME 804) and one for backward prediction (e.g., ALTREF_FRAME 806) can be used for predicting the current block.

[0082] There may be a finite number of reference frames that can be stored within the reference frame buffer 800. As shown in FIG. 8, the reference frame buffer 800 can store up to eight reference frames. Although three of the eight spaces in the reference frame buffer 800 are used by the LAST_FRAME 802, the GOLDEN_FRAME 804, and the ALTREF_FRAME 806, five spaces remain available to store other reference frames.

[0083] In particular, one or more available spaces in the reference frame buffer 800 may be used to store a second last frame LAST2_FRAME and / or a third last frame LAST3_FRAME as additional forward reference frames, in addition to the LAST_FRAME 802. A backward frame BWDREF_FRAME 808 can be stored as an additional backward prediction reference frame, in addition to ALTREF_FRAME 806. The BWDREF_FRAME can be closer in relative distance to the current frame than the ALTREF_FRAME 806, for example.

[0084] In one example, the pair of {LAST_FRAME, BWDREF_FRAME] can be used to generate a compound predictor for coding the current block. In this example, LAST_FRAME is a “nearest” forward reference frame for forward prediction, and BWDREF_FRAME is a “nearest” backward reference frame for backward prediction.

[0085] A current block is predicted based on a prediction mode. In the case of inter prediction, the prediction mode may be selected from one of multiple inter-prediction modes, including the wedge mode, using one or more reference frames of the reference frame buffer 800 including, for example, the LAST_FRAME 802, the GOLDEN_FRAME 804, theALTREF_FRAME 806, or any other reference frame. The prediction mode of the current block can be transmitted from an encoder, such as the encoder 400 of FIG. 4, to a decoder, such as the decoder 500 of FIG. 5, in an encoded bitstream, such as the compressed bitstream 420 of FIGS. 4-5. A bitstream syntax can support three categories of inter prediction modes in an example. These inter prediction modes can include a mode (referred to herein as the ZERO_MV mode) in which a block from the same location within a reference frame as the current block is used as the prediction block, a mode (referred to herein as the NEW_MV mode) in which a motion vector is transmitted to indicate the location of a block within a reference frame to be used as the prediction block relative to the current block, or a mode (referred to herein as the REF_MV mode and comprising a NEAR_MV or NEAREST_MV mode) in which no motion vector is transmitted and the current block uses the last or second- to-last non-zero motion vector used by neighboring, previously coded blocks to generate the prediction block. The previously coded blocks may be those coded in the scan order, e.g., a raster or other scan order, before the current block. Inter-prediction modes may be used with any of the available reference frames. NEAREST_MV and NEAR_MV can refer to the most and second most likely motion vectors for the current block obtained by a survey of motion vectors in the context for a reference. The reference can be co-located motion vectors in the previous frame.

[0086] FIG. 9 illustrates a codebook 900 of possible partition lines in wedge-based compound prediction. Wedge-based compound prediction partitions a current block into two based on one partition line of the codebook 900. While the lines shown in FIG. 9 are straight, that need not be the case. At least some of the lines can be oblique (i.e., not straight). Each partition of the block can be separately predicted (i.e., can have a separate prediction block) using inter prediction. The separate predictor blocks are then combined to form the compound (final) prediction block having the same dimensions as the current block being predicted according to the partition line(s). The compound prediction block can be a smooth- weighted or a cliff- weighted combination of the first predictor block and the second predictor block.

[0087] A respective weight mask can be associated with (e.g., identified or calculated based on) each of the partition lines of the codebook 900. The weight mask can be generated in accordance with the angle and, optionally, an offset of the wedge. The encoder may signal an index of the wedge line. In some implementations, the encoder may additionally signal an offset adjustment to the mask location. As such, a mask of a certain angle can be further panned (such as more towards the right of the block edge) based on the offset adjustment.

[0088] In some implementations of wedge-based compound prediction, for each block size, a set of (e.g.,16) predefined two-dimensional weighting arrays are available (e.g., hard coded in the codec). Each weighting array corresponds to a predefined wedge partitioning pattern, in which two wedge partitions are specified along a certain edge direction and position. In each array, the weights, m(x, y), are arranged to reflect a predefined wedge partitioning pattern.

[0089] For samples located entirely within one of the two wedge partitions, the weights are typically set to a maximum value (MAX). In an example, MAX can be 64. For samples located entirely within the other wedge partition, the weights are typically set to 0. Moreover, near the transition edge, there is a gradual change from MAX to 0, with MAX / 2 (e.g., 32) at the actual edge, and the weights may be derived from predefined look-up tables that define the soft transition zone for each wedge pattern. More generally, the weight mask is associated with a wedge that divides the current into two parts, the weight mask includes a soft transition region between the two parts of the current block, and storing the motion vectors for each unit is based on a distribution of weights within the transition region.

[0090] In some implementations, two bitstream syntax elements may be used to signal wedge-based prediction: a wedge_index, which specifies the selected wedge partitioning pattern (e.g., 0-15), and a wedge_sign, which determines the assignment of prediction blocks to the two wedge partitions.

[0091] Accordingly, the wedge predictor uses one of the preset masks to split the block into two partitions, where one partition is filled with pixels from one reference block based on a first MV and the other partition is filled with pixels from another reference block based on a second MV. The first partition and the second partition are stitched together, using the weight mask, to obtain the compound prediction block. As mentioned above, the compressed bitstream includes the chosen wedge index from the set of available 16 wedges. To mitigate spurious artifacts, soft-cliff- shaped 2D wedge masks are employed.

[0092] The compound wedge-based predictor is obtained using equation (1):

[0093] That is, the compound prediction block P is defined as the weighted sum of a first prediction block Pi and a second prediction block P2, where m(i, j) is a weighting factor of a weight mask.

[0094] FIG. 10 illustrates a use case 1000 demonstrating the application of wedge-based compound prediction. The use case 1000 includes a current block 1002 of a current frame(not shown). The wedge-based compound prediction mode is designed to provide a better prediction along object boundaries, where a first part 1004 of the current block 1002 belonging to an object 1006 may be moving in a different direction than a second part 1008 belonging to another object or the background of the current frame.

[0095] A first prediction block is generated using a first MV 1010 and a second prediction block is generated using a second MV 1012. A weight mask 1014, generated (e.g., obtained or selected) based on a wedge 1016, is then used to combine the first prediction block and the second prediction block to obtain the compound prediction block.

[0096] To further describe some implementations in greater detail, reference is next made to examples of techniques which may be performed for reference frame motion field selection for wedge mode blocks. FIG. 11 is a flowchart of a technique 1100 for coding a current block. The technique 1100 can be implemented, for example, as a software program that may be executed by computing devices such as transmitting station 102 or receiving station 106. The software program can include machine-readable instructions that may be stored in a memory such as the memory 204 or the secondary storage 214, and that, when executed by a processor, such as the processor 202 of FIG. 2, may cause the computing device to perform the technique 1100.

[0097] The technique 1100 may be implemented in whole or in part in the intra / inter prediction stage 402 of the encoder 400 of FIG. 4 and / or the intra / inter prediction stage 508 of the decoder 500 of FIG. 5. When implemented by an encoder, coding means encoding in a compressed bitstream, such as the compressed bitstream 420 of FIG. 4; and when implemented by a decoder, coding means decoding from a compressed bitstream, such as the compressed bitstream 420 of FIG. 5. The technique 1100 can be implemented using specialized hardware or firmware. Multiple processors, memories, or both, may be used.

[0098] At 1102, a compound prediction block for the current block is obtained based on a weight mask, a first motion vector ( MV0), and a second motion vector ( MV1). The weight mask is used to generate the compound prediction block by combining predictions generated using M V0 and MV1.

[0099] When implemented by an encoder, the encoder may determine that the current block is to be predicted using the wedge-based compound prediction and may identify MV0 and MV 1. As can be appreciated, the encoder may make the selections based on a ratedistortion analysis. As described above with respect to FIG. 4, the encoder may obtain a residual (e.g., a difference between the current block and the compound prediction block. The encoder encodes, in the compressed bitstream, the prediction mode of the current block,syntax elements that enable the decoder to obtain (e.g., derive, select, etc.) MVO and MV1, the residual, and an index of the wedge used in the wedge -based compound prediction. As described with respect to FIG. 5, the decoder uses the prediction mode, MVO, MV1, and wedge to obtain the compound prediction block. The decoder then reconstructs the current block based on the compound prediction block.

[0100] At 1104, the current block is partitioned into units based on a motion vector granularity. The motion vector granularity can be determined based on a predefined block size, for example, 8x8 pixels. The partitioning ensures that the MVs are stored in association with appropriately sized units within the reference frame motion field.

[0101] At 1106, for each of the units, the technique 1100 stores, in association with the each unit and based on a respective portion of the weight mask co-extensive with each unit, the first motion vector ( MVO), the second motion vector ( MV1), or both of MVO and MV1. In an example, this process involves determining the appropriate MV(s) to store based on the average weight of the respective portion of the weight mask for each unit. In this context, “co-extensive” means that the respective portion of the weight mask corresponds exactly to the spatial boundaries of the each unit and that the respective portion of the weight mask covers the same area as the each unit. For each unit, the weight mask values are accumulated for every pixel location within the unit to calculate the average mask weight.

[0102] In an example, the storing may include, if (e.g., on a condition that) the respective portion of the weight mask for a unit has an average weight greater than or equal to a first threshold, the first motion vector is stored in association with the unit; if the average weight is less than or equal to a second threshold, the second motion vector in stored in association with the unit; and if the average weight is between the first threshold and the second threshold, then both of MVO and MV1 are stored in association with the unit. The first threshold and the second threshold can be empirically derived. In an example, these thresholds can be empirically derived to optimize the trade-off between memory savings and potential minor prediction accuracy degradation, often tuned based on rate-distortion performance metrics across various video content and block sizes. In an example, in the case that the weights of the weight mask are in the range [0, 63], then the first threshold can be set to 4 and the second threshold can be 60. Said another way, if the maximum weight value of the weight mask is MAX, then the first threshold may be set to a first percent (e.g., approximately 6%) of MAX, and the second threshold may be set to approximately a second percent (e.g., 94%) of MAX. However, other first and second thresholds are possible.

[0103] FIG. 12 shows an example 1200 illustrating the concept of associating MVs with units within a block 1202. The block 1202 is assumed to have been obtained using wedgebased compound prediction. The block 1202 is 8x8 pixels in size, and for purposes of the example 1200, the minimal block unit size (i.e., the MV granularity) is assumed to be 4x4 pixels. Consequently, the block 1202 is divided into four units: (i.e., units 1204, 1206, 1208, and 1210). The block 1202 is obtained using two MVs (not shown) and a weight mask 1212. Since the MV granularity is 4x4, the weight mask 1212, as shown, thus includes four mask portions 1214, 1216, 1218, and 1220, each co-extensive with the respective minimal block units 1204, 1206, 1208, and 1210 of the block 1202.

[0104] The weight mask 1212 defines a wedge pattern that separates the block 1202 into two partitions (not shown). The specific values of the weight mask 1212 or the location and direction of the wedge are not necessary for the understanding of the example 1200 and the disclosure. For each minimal block unit, the weight mask values determine which of MV 1 and / or MV1 to store in association with the unit of the block 1202. Based on the calculated average mask weight for each unit (i.e., for each mask portion), it can be determined which MV(s) to store in the motion field according to the following logic: If the average mask weight is greater than or equal to a first threshold (THRESHOLD_1), then only MV1 is stored; If the average mask weight is less than or equal to a second threshold (THRESHOLD_0), then only MV0 is stored; otherwise, both MV1 and MV0 are stored for the unit. For the sake of this example, assume that THRESHOLD_1 = 48 and THRESHOLD !) = 16.

[0105] As such, mask portion 1214, which corresponds to unit 1204, has an average mask weight of 58, indicating that only MV1 is stored for this unit (as it exceeds THRESHOLD_1=48). The mask portion 1216, which corresponds to unit 1206, has an average mask weight of 18, suggesting that both MVs are stored for this unit (as it falls between THRESHOLD ! and THRESHOLD.O: 48 > 18 > 16). The mask portion 1218, which corresponds to unit 1208, has an average mask weight of 6, indicating that only MV0 is stored for this unit (as the average, 6, is less than THRESHOLD_0=16). Finally, the mask portion 1220, which corresponds to unit 1210, has an average mask weight of 23, that both MVs are stored for this unit (as it falls between THRESHOLD.1 and THRESHOLD_0). Accordingly, the associations of MVs to units is shown in a summary 1222.

[0106] Referring again to FIG. 11, in an example, the storing can be based on the number of pixels of the unit associated with each of the first and second motion vectors. That is, the storing can be based on the numbers of pixels of the unit that are on one side vs. the otherside of the wedge. As such, a count of pixels of the each unit associated with the first motion vector can be determined, and storing, in association with the each unit, the first motion vector, the second motion vector, or both of the first motion vector and the second motion vector can be based on the count. Pixels with weight values greater than MAX / 2 may be counted as associated with the first motion vector, pixels with weight values less than MAX / 2 may be counted as associated with the second motion vector, and pixels with weight values equal to MAX / 2 may be counted toward both motion vectors, neither motion vector, or processed according to a predefined tie-breaking rule, which, in one example, may be random selection.

[0107] For example, if a count of pixels on one side of the wedge for the respective portion of the weight mask for the each unit is greater than or equal to a first threshold (e.g., 95% of the pixels), the first motion vector is stored in association with the unit; if the count of pixels on the one side of the wedge is less than or equal to a second threshold (e.g., 5% of the pixels), the second motion vector is stored in association with each unit; otherwise, both of the first motion vector and the second motion vector are stored in association with the unit.

[0108] The weight mask can be obtained based on an index associated with the weight mask decoded from a compressed bitstream. The index is used to identify or generate the weight mask required for the current block. The index associated with the weight mask can be encoded in a compressed bitstream by the encoder, which the decoder then decodes.

[0109] The weight mask can be generated based on the angle (and, optionally, an offset) of a wedge that separates the current block into two parts, each part being associated with one of the motion vectors. The wedge helps in providing better prediction along object boundaries where parts of the block may be moving in different directions.

[0110] The reference frame motion field is updated with the stored motion vectors for each unit of the current block, optimizing storage and computational resources, such as described with respect to FIG. 7. By selectively storing the motion vectors in this manner, the method reduces redundancy and increases efficiency in motion vector storage and retrieval.

[0111] The motion field generated according to implementations of this disclosure can be used by other tools or processes of codecs as a person skilled in the art can appreciate. To illustrate, motion vectors stored at the smallest spatial unit level (e.g., 8x8 blocks) may serve as candidates for motion vector prediction (MVP) modes such as NEAREST_MV or REF_MV, guide merge mode decisions, and enable motion vector reuse for temporally or spatially collocated blocks. The motion field may also support skip and merge modes, in which blocks inherit motion information from neighboring units without explicitlytransmitting motion vectors, and can facilitate motion refinement or adaptive search window selection for subsequent motion estimation. Additionally, stored motion vectors can be used by overlapped block motion compensation (OBMC) techniques, which incorporate neighboring motion vectors to create smoother transitions at block boundaries and reduce blocking artifacts.

[0112] For simplicity of explanation, the technique 1100 of FIG. 11 is depicted and described as a series of steps or operations. However, the steps or operations in accordance with this disclosure can occur in various orders and / or concurrently. Additionally, other steps or operations not presented and described herein may be used. Furthermore, not all illustrated steps or operations may be required to implement a technique in accordance with the disclosed subject matter.

[0113] The aspects of encoding and decoding described above illustrate some examples of encoding and decoding techniques. However, it should be understood that encoding and decoding, as those terms are used in the claims, could mean compression, decompression, transformation, or any other processing or change of data.

[0114] Some implementations are described below as numbered examples (Example A, B, C, etc.). These examples are provided as examples only and do not limit the other implementations disclosed herein.

[0115] Example A is a method that includes obtaining a compound prediction block for a current block based on a weight mask, a first motion vector, and a second motion vector; partitioning the current block into units based on a motion vector granularity; and for each of the units, storing, in association with each unit and based on a respective portion of the weight mask that is co-extensive with each unit, the first motion vector, the second motion vector, or both of the first motion vector and the second motion vector.

[0116] Example B is the method of Example A where storing, in association with each unit and based on the respective portion of the weight mask that is co-extensive with each unit, the first motion vector, the second motion vector, or both of the first motion vector and the second motion vector includes: on a condition that the respective portion of the weight mask for each unit has an average weight greater than or equal to a first threshold, storing the first motion vector in association with each unit; on a condition that the average weight is less than or equal to a second threshold, storing the second motion vector in association with each unit; and on a condition that the average weight is between the first threshold and the second threshold, storing both of the first motion vector and the second motion vector in association with each unit.

[0117] Example C is the method of Example A where storing, in association with each unit and based on the respective portion of the weight mask that is co-extensive with each unit, the first motion vector, the second motion vector, or both of the first motion vector and the second motion vector includes: determining a count of pixels of each unit associated with the first motion vector; and storing, in association with each unit, the first motion vector, the second motion vector, or both of the first motion vector and the second motion vector based on the count.

[0118] Example D is the method of Example A further including obtaining the weight mask based on an index associated with the weight mask decoded from a compressed bitstream.

[0119] Example E is the method of Example A further including encoding an index associated with the weight mask in a compressed bitstream.

[0120] Example F is the method of Example A where a size of each unit is 8x8.

[0121] Example G is the method of Example A further including updating a reference frame motion field with the stored motion vectors for each unit of the current block.

[0122] Example H is the method of Example A where the weight mask is generated based on an angle of a wedge that separates the current block into two parts, each part being associated with one of the first motion vector and the second motion vector.

[0123] Example I is the method of Example A where obtaining the compound prediction block includes: generating a first prediction block using the first motion vector and a first reference frame; generating a second prediction block using the second motion vector and a second reference frame; and combining the first prediction block and the second prediction block using the weight mask to generate the compound prediction block.

[0124] Example J is the method of Example A where the weight mask is associated with a wedge that divides the current block into two parts, where the weight mask includes a soft transition region between the two parts of the current block, and where storing the motion vectors for each unit is based on a distribution of weights within the transition region.

[0125] Example K is a device that includes a processor that is configured to perform the method of any one of Example A to Example J.

[0126] Example L is a device that includes a memory and a processor. The processor is configured to execute instructions stored in the memory to perform the method of any one of Example A to Example J.

[0127] Example M is one or more non-transitory computer-readable storage media that include executable instructions that, when executed by a processor, facilitate performance ofoperations, including operations that perform the method of any one of Example A to Example J.

[0128] Example N is one or more non-transitory computer-readable storage media having stored thereon an encoded bitstream, where the encoded bitstream is configured for decoding by a decoder performing the method of any one of Example A to Example E or Example G to Example J.

[0129] Example O is one or more non-transitory computer-readable storage media having stored thereon an encoded bitstream, where the encoded bitstream is generated by an encoder performing the method of any one of Example A to Example D and Example F to Example J.

[0130] The word “example” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “example” is not necessarily to be construed as being preferred or advantageous over other aspects or designs. Rather, use of the word “example” is intended to present concepts in a concrete fashion. As used in this application, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless specified otherwise or clearly indicated otherwise by the context, the statement “X includes A or B” is intended to mean any of the natural inclusive permutations thereof. That is, if X includes A; X includes B; or X includes both A and B, then “X includes A or B” is satisfied under any of the foregoing instances. In addition, the articles “a” and “an” as used in this application and the appended claims should generally be construed to mean “one or more,” unless specified otherwise or clearly indicated by the context to be directed to a singular form. Moreover, use of the term “an implementation” or the term “one implementation” throughout this disclosure is not intended to mean the same embodiment or implementation unless described as such.

[0131] Implementations of the transmitting station 102 and / or the receiving station 106 (and the algorithms, methods, instructions, etc., stored thereon and / or executed thereby, including by the encoder 400 and the decoder 500) can be realized in hardware, software, or any combination thereof. The hardware can include, for example, computers, intellectual property (IP) cores, application- specific integrated circuits (ASICs), programmable logic arrays, optical processors, programmable logic controllers, microcode, microcontrollers, servers, microprocessors, digital signal processors, or any other suitable circuit. In the claims, the term “processor” should be understood as encompassing any of the foregoing hardware, either singly or in combination. The terms “signal” and “data” are used interchangeably. Further, portions of the transmitting station 102 and the receiving station 106 do not necessarily have to be implemented in the same manner.- l-

[0132] Further, in one aspect, for example, the transmitting station 102 or the receiving station 106 can be implemented using a general-purpose computer or general-purpose processor with a computer program that, when executed, carries out any of the respective methods, algorithms, and / or instructions described herein. In addition, or alternatively, for example, a special purpose computer / processor can be utilized which can contain other hardware for carrying out any of the methods, algorithms, or instructions described herein.

[0133] The transmitting station 102 and the receiving station 106 can, for example, be implemented on computers in a video conferencing system. Alternatively, the transmitting station 102 can be implemented on a server, and the receiving station 106 can be implemented on a device separate from the server, such as a handheld communications device. In this instance, the transmitting station 102, using an encoder 400, can encode content into an encoded video signal and transmit the encoded video signal to the communications device. In turn, the communications device can then decode the encoded video signal using a decoder 500. Alternatively, the communications device can decode content stored locally on the communications device, for example, content that was not transmitted by the transmitting station 102. Other suitable transmitting and receiving implementation schemes are available. For example, the receiving station 106 can be a generally stationary personal computer rather than a portable communications device, and / or a device including an encoder 400 may also include a decoder 500.

[0134] Further, all or a portion of implementations of the present disclosure can take the form of a computer program product accessible from, for example, a computer-usable or computer-readable medium. A computer-usable or computer-readable medium can be any device that can, for example, tangibly contain, store, communicate, or transport the program for use by or in connection with any processor. The medium can be, for example, an electronic, magnetic, optical, electromagnetic, or semiconductor device. Other suitable mediums are also available.

[0135] The above-described embodiments, implementations, and aspects have been described in order to facilitate easy understanding of this disclosure and do not limit this disclosure. On the contrary, this disclosure is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims, which scope is to be accorded the broadest interpretation as is permitted under the law so as to encompass all such modifications and equivalent arrangements.

Claims

What is claimed is:

1. A method, comprising: obtaining a compound prediction block for a current block based on a weight mask, a first motion vector, and a second motion vector; partitioning the current block into units based on a motion vector granularity; and for each of the units, storing, in association with the each unit and based on a respective portion of the weight mask that is co-extensive with the each unit, the first motion vector, the second motion vector, or both of the first motion vector and the second motion vector.

2. The method of claim 1, wherein storing, in association with the each unit and based on the respective portion of the weight mask that is co-extensive with the each unit, the first motion vector, the second motion vector, or both of the first motion vector and the second motion vector comprises: on a condition that the respective portion of the weight mask for the each unit has an average weight greater than or equal to a first threshold, storing the first motion vector in association with the each unit; on a condition that the average weight is less than or equal to a second threshold, storing the second motion vector in association with the each unit; and on a condition that the average weight is between the first threshold and the second threshold, storing both of the first motion vector and the second motion vector in association with the each unit.

3. The method of claim 1, wherein storing, in association with the each unit and based on the respective portion of the weight mask that is co-extensive with the each unit, the first motion vector, the second motion vector, or both of the first motion vector and the second motion vector comprises: determining a count of pixels of the each unit associated with the first motion vector; and storing, in association with the each unit, the first motion vector, the second motion vector, or both of the first motion vector and the second motion vector based on the count.

4. The method of any one of claims 1 to 3, further comprising:obtaining the weight mask based on an index associated with the weight mask decoded from a compressed bitstream.

5. The method of any one of claims 1 to 3, further comprising: encoding an index associated with the weight mask in a compressed bitstream.

6. The method of any one of claims 1 to 3, wherein a size of the each unit is 8x8.

7. The method of any one of claims 1 to 3, further comprising: updating a reference frame motion field with the stored motion vectors for each unit of the current block.

8. The method of any one of claims 1 to 3, wherein the weight mask is generated based on an angle of a wedge that separates the current block into two parts, each part being associated with one of the first motion vector and the second motion vector.

9. The method of claim 1, wherein obtaining the compound prediction block comprises: generating a first prediction block using the first motion vector and a first reference frame; generating a second prediction block using the second motion vector and a second reference frame; and combining the first prediction block and the second prediction block using the weight mask to generate the compound prediction block.

10. The method of any one of claims 1 to 3 or claim 9, wherein the weight mask is associated with a wedge that divides the current into two parts, wherein the weight mask includes a soft transition region between the two parts of the current block, and wherein storing the motion vectors for each unit is based on a distribution of weights within the transition region.

11. A device, comprising: a processor that is configured to perform the method of any one of claim 1 to claim12. A device, comprising: a memory; and a processor, the processor configured to execute instructions stored in the memory to perform the method of any one of claim 1 to claim 10.

13. A non-transitory computer-readable storage medium, comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising operations that perform the method of any one of claim 1 to claim 10.

14. A non-transitory computer-readable storage medium having stored thereon an encoded bitstream, wherein the encoded bitstream is configured for decoding by a decoder performing the method of any one of claim 1 to claim 5 or claim 7 to claim 10.

15. A non-transitory computer-readable storage medium having stored thereon an encoded bitstream, wherein the encoded bitstream is generated by an encoder performing the method of any one of claim 1 to claim 4 and claim 6 to claim 10.

Images