Affine motion prediction for video coding

Abstract

An example method includes obtaining, for a current block of video data, a value of a motion vector (MV) of an affine motion model of a neighboring block of video data, deriving, from the value of the MV of the affine motion model of the neighboring block, a value of a predictor of a MV of an affine motion model of the current block, decoding, from a video bitstream, a representation of a difference between the value of the MV of the affine motion model of the current block and the value of the predictor, determining the value of the MV of the affine motion model of the current block from the value of the predictor and the decoded difference, determining a predictor block of video data based on the determined value of the MV of the affine motion model of the current block, and reconstructing the current block based on the predictor block.

Description

[0001] This application claims the benefit of U.S. Provisional Application No. 62 / 337,301, filed May 16, 2016, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] This disclosure relates to video coding. BACKGROUND

[0003] Digital video capabilities can be incorporated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video gaming devices, video game consoles, cellular or satellite radio telephones, so-called "smart phones," video teleconferencing devices, video streaming devices, and the like. Digital video devices implement video coding techniques, such as those described in video coding standards. Video devices can transmit, receive, encode, decode, and / or store digital video information more efficiently by implementing such video coding techniques.

[0004] Some video coding standards are defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4, Part 10, Advanced Video Coding (AVC), including its Scalable Video Coding (SVC) and Multiview Video Coding (MVC) extensions, ITU-T H.265, also referred to as High Efficiency Video Coding (HEVC), and extensions of these standards. Recently, the Joint Collaboration Team on Video Coding (JCT-VC) of ITU-T Video Coding Experts Group (VCEG) and ISO / IEC Motion Picture Experts Group (MPEG) has completed the design of a new video coding standard, namely, High Efficiency Video Coding (HEVC). A recent HEVC specification draft, and hereinafter referred to as HEVC WD, can be obtained at itu.int / rec / T-REC-H.265-201504-S / en. A range extension of HEVC, namely HEVC-Rext, is also being developed by JCT-VC. A recent Working Draft (WD) of the range extension, hereinafter referred to as RExt WD6, can be obtained from phenix.int-evry.fr / jct / doc_end_user / documents / 16_San%20Jose / wg11 / JCTVC-P1005-vl.zip.

[0005] Video decoding techniques involve spatial (intra-picture) prediction and / or temporal (inter-picture) prediction to reduce or remove inherent redundancy in video sequences. For block-based video decoding, video slices (e.g., video frames or portions of video frames) can be segmented into video blocks, which, in some techniques, may also be referred to as tree blocks, decoding units (CUs), and / or decoding nodes. Video blocks in an intra-frame decoded (I) slice of a picture are encoded using spatial prediction relative to reference samples in adjacent blocks within the same picture. Video blocks in an inter-frame decoded (P or B) slice of a picture can use spatial prediction relative to reference samples in adjacent blocks within the same picture or temporal prediction relative to reference samples in other reference pictures. A picture may be referred to as a frame, and a reference picture may be referred to as a reference frame.

[0006] Spatial or temporal prediction generates a predictive block for the block to be decoded. Residual data represents the pixel difference between the original block and the predictive block. Inter-frame decoded blocks are encoded based on the motion vector of the block pointing to the reference sample forming the predictive block and the residual data indicating the difference between the decoded block and the predictive block. Intra-frame decoded blocks are encoded according to the intra-frame decoding mode and the residual data. For further compression, the residual data can be transformed from the pixel domain to the transform domain to generate residual transform coefficients, which can then be quantized. The quantized transform coefficients, initially arranged in a two-dimensional array, can be scanned to generate a one-dimensional vector of transform coefficients, and entropy decoding can be applied to achieve even more compression. Summary of the Invention

[0007] In one example, a method for decoding video data includes: obtaining, through one or more processors of a video decoder, values ​​of motion vectors (MVs) of affine motion models for adjacent video data blocks; deriving, through the one or more processors, values ​​of predictors of the MVs of the affine motion model for the current video data block from the values ​​of the MVs of the affine motion models for adjacent video data blocks; through the one or more processors, representing a difference between the values ​​of the motion vectors of the affine motion model for the current video data block and the values ​​of the predictors from the encoded video bitstream; determining, through the one or more processors, the values ​​of the motion vectors of the affine motion model for the current video data block from the values ​​of the predictors and the decoded difference; determining predictor blocks of video data based on the determined values ​​of the motion vectors of the affine motion model for the current video data block; and reconstructing the current video data block based on the predictor blocks of video data.

[0008] In another example, a method for encoding video data includes: determining, by one or more processors of a video encoder, the values ​​of motion vectors of an affine motion model for a current video data block, wherein the motion vectors of the affine motion model identify predictive sub-blocks of video data for the current video data block; obtaining, by the one or more processors, the values ​​of motion vectors of affine motion models for adjacent video data blocks; deriving, by the one or more processors and the values ​​of motion vectors of affine motion models from adjacent video data blocks, the values ​​of predictive sub-blocks of motion vectors of the current video data block; and encoding, by the one or more processors and the representation of the difference between the values ​​of motion vectors of the affine motion model for the current video data block and the values ​​of the predictive sub-blocks in an encoded video bitstream.

[0009] In another example, the apparatus for decoding video data blocks includes: a memory configured to store video data; and one or more processing units implemented in a circuit. In this example, the one or more processing units are configured to: obtain the values ​​of motion vectors of affine motion models of adjacent video data blocks for the current video data block; derive the values ​​of predictors of the motion vectors of the affine motion model of the current video data block from the values ​​of the motion vectors of the affine motion models of adjacent video data blocks; represent the difference between the values ​​of the motion vectors of the affine motion model of the current video data block decoded from the encoded video bitstream and the values ​​of the predictors; determine the values ​​of the motion vectors of the affine motion model of the current video data block from the values ​​of the predictors and the decoded difference; determine predictor blocks of video data based on the determined values ​​of the motion vectors of the affine motion model of the current video data block; and reconstruct the current video data block based on the predictor blocks of video data.

[0010] In another example, the apparatus for encoding video data blocks includes: a memory configured to store video data; and one or more processing units implemented in a circuit. In this example, the one or more processing units are configured to: determine the values ​​of motion vectors of an affine motion model for the current video data block, the motion vectors of the affine motion model identifying predictor blocks of video data in the current video data block; obtain the values ​​of motion vectors of affine motion models for adjacent video data blocks; derive the values ​​of predictors of motion vectors of the current video data block's affine motion model from the values ​​of motion vectors of motion vectors of adjacent video data blocks; and encode a representation of the difference between the values ​​of motion vectors of the current video data block's affine motion model and the values ​​of the predictors in the encoded video bitstream.

[0011] In another example, an apparatus for encoding or decoding video data includes: means for obtaining values ​​of motion vectors of affine motion models of adjacent video data blocks for a current video data block; means for deriving values ​​of predictors of motion vectors of the current video data block's affine motion model from the values ​​of motion vectors of adjacent video data blocks; means for obtaining a difference between the values ​​of motion vectors of the current video data block's affine motion model and the values ​​of the predictors; means for determining each of the values ​​of motion vectors of the current video data block's affine motion model from the values ​​of the predictors and the decoded difference; and means for identifying predictor blocks of video data based on the determined values ​​of motion vectors of the current video data block's affine motion model.

[0012] In another example, a computer-readable storage medium stores instructions that, when executed, cause one or more processors of a video encoder or video decoder to perform the following operations: obtaining values ​​of motion vectors of affine motion models of adjacent video data blocks for a current video data block; deriving values ​​of predictors of motion vectors of the current video data block's affine motion model from the values ​​of motion vectors of the adjacent video data blocks; obtaining the difference between the values ​​of motion vectors of the current video data block's affine motion model and the values ​​of the predictors; determining each of the values ​​of motion vectors of the current video data block's affine motion model from the values ​​of the predictors and the decoded difference; and identifying predictor blocks of video data based on the determined values ​​of motion vectors of the current video data block's affine motion model.

[0013] Details of one or more examples are set forth in the following figures and embodiments. Other features, objectives, and advantages will be apparent from the description, figures, and claims. Attached Figure Description

[0014] Figure 1 A block diagram illustrating an example video encoding and decoding system that can be configured to perform the techniques of the present invention.

[0015] Figure 2 A block diagram illustrating an example of a video encoder that can be configured to perform the techniques of the present invention.

[0016] Figure 3 A block diagram illustrating an example of a video decoder that can be configured to perform the techniques of the present invention.

[0017] Figure 4A and 4B A conceptual diagram illustrating spatial adjacency candidates in High Efficiency Video Decoding (HEVC).

[0018] Figure 5 A conceptual diagram illustrating a two-point motion vector affine with four affine parameters.

[0019] Figure 6 A conceptual diagram illustrating the affine inter-frame mode.

[0020] Figure 7A and 7B A conceptual diagram illustrating the candidates for affine merging patterns.

[0021] Figure 8 A conceptual diagram illustrating a six-parameter affine motion model according to one or more techniques of the present invention.

[0022] Figure 9 A conceptual diagram illustrating the affine motion vector evaluation according to one or more techniques of the present invention.

[0023] Figure 10 A conceptual diagram illustrating Overlapping Block Motion Compensation (OBMC) in H.263.

[0024] Figure 11A and 11B A conceptual diagram illustrating the OBMC on top of HEVC.

[0025] Figure 12A and 12B This is a conceptual diagram illustrating the sub-blocks that can be applied to the OBMC.

[0026] Figure 13 A flowchart illustrating an example method for performing affine motion compensation via a video encoder (e.g., during the video encoding process) according to one or more techniques of the present invention.

[0027] Figure 14 A flowchart illustrating an example method for performing affine motion compensation via a video decoder (e.g., during the video decoding process) according to one or more techniques of the present invention. Detailed Implementation

[0028] Generally, this invention describes techniques related to the decoding (e.g., encoding or decoding) of affine motion information of video data blocks. In current video decoding standards, only translational motion models are applied to motion-compensated prediction (MCP). When a translational motion model is applied to MCP, a video decoder (e.g., a video encoder or video decoder) can use a single two-dimensional motion vector (MV) for the current block, which indicates the displacement between the current video data block and the corresponding predicted sub-block of the video data. The MV can be two-dimensional because each MV can have an x-component indicating the horizontal displacement between the current video data block and the predicted sub-block of the video data, and a y-component indicating the vertical displacement between the current video data block and the predicted sub-block of the video data. As discussed in further detail below, in current video decoding standards such as HEVC, there are two inter-frame prediction modes, called merge (skipping special cases considered as merges) and Advanced Motion Vector Prediction (AMVP) mode. In the merge mode, the value of the MV of the current block is directly inherited from the value of the MV candidate, which can be the value of the MV of the adjacent blocks of the current block. In contrast, in the AMVP mode, the value of the MV candidate can be further improved. In detail, a video decoder can use a signal to represent the difference between the value of the MV candidate and the value of the MV of the current block. This difference can be called the motion vector difference (MVD).

[0029] However, many types of motion exist besides translation, such as zooming, zooming, rotation, perspective, and other irregular motions. Applying only the translational motion model to MCP in these test sequences with irregular motion can affect prediction accuracy and lead to low decoding efficiency. For example, using only the translational motion model can produce a predicted block that does not match the properly decoded original block. Therefore, the size of the residual data (i.e., the value representing the pixel difference between the original block to be decoded and the predicted block) can increase, which reduces decoding efficiency.

[0030] ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) are investigating the potential need for standardization of future video decoding technologies with compression capabilities significantly exceeding those of the current HEVC standard (including its current and recent extensions for screen content decoding and high dynamic range decoding). These groups are working together on this exploration (known as the Joint Video Exploration Group (JVET)) to evaluate compression technology designs proposed by their experts in this field. JVET has published a Joint Exploration Model (JEM), which describes decoding features in a coordinated test model study as potential enhanced video decoding technologies beyond the capabilities of HEVC. In the JEM, an affine motion model is proposed for use in MCP. From February 20 to 26, 2016, in San Diego, USA, the 2nd meeting of the Joint Video Exploration Group (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11 was held. The document, JVET-B1001_v3 (hereinafter “JEM Test Model”), and the latest algorithm description of JEM, “Algorithm Description of Joint Exploration Test Model 2”, are available at phenix.it-sudparis.eu / jvet / doc_end_user / documents / 2_San%20Diego / wg11 / JVET-B1001-v3.zip.

[0031] When using an affine motion model for MCP, the video decoder can apply multiple motion vectors to the current block. These multiple motion vectors collectively indicate the affine transformation (e.g., translation, scaling, reflection, rotation, etc.) between the current video data block and its corresponding predicted sub-block. For example, the affine motion model may include a first two-dimensional motion vector indicating the displacement between the top-left corner of the current block and the top-left corner of the corresponding predicted sub-block, and a second two-dimensional motion vector indicating the displacement between the top-right corner of the current block and the top-right corner of the corresponding predicted sub-block. The motion vectors in the affine motion model can be referred to as control point motion vectors (CPMVs) and can be referenced to positions on the current block (i.e., control points). For example, a two-dimensional motion vector indicating the displacement between the top-left corner of the current block and the top-left corner of the corresponding predicted sub-block can be referred to as the top-left CPMV of the current block. As discussed in further detail below, in the JEM test model, there are two inter-frame prediction modes: affine inter-frame (e.g., AF_INTER) and affine merging (e.g., AF_MERGE).

[0032] In affine merging mode, the value of each CPMV of the current block is directly derived from the CPMV of the single neighboring block of the current block that is being decoded using the affine motion model. In other words, in affine merging mode, only the CPMV of the neighboring blocks is changed to the CPMV of the current block, and there is no flexibility to change or adjust the affine model parameters. Specifically, it is impossible to modify the value of CPMV using MVD.

[0033] In affine inter-frame mode, the value of each CPMV for the current block is individually derived based on the MV and MVD values ​​of the blocks adjacent to the corresponding control point. The MV value on which the CPMV is based can be called the Control Point Motion Vector Predictor (CPMVP). As an example, the value of the top-left CPMV of the current block can be derived based on the MV and MVD of the left block, the top-left block, or the block above the top-left point of the current block. As another example, the value of the top-right CPMV of the current block can be derived based on the MV and MVD of the top-right block or the block above the top-right point of the current block.

[0034] In both the HEVC and JEM test models, the video encoder can represent the MVD syntax (i.e., syntax elements representing the values ​​of the MVD) in the bitstream using signals so that the MV can be reconstructed on the decoder side. The amount of data used to represent the MVD syntax using signals can be related to the magnitude of the MVD value. For example, more data may be needed to represent the MVD syntax for a relatively large MVD value compared to an MVD with a relatively small value.

[0035] However, current techniques that derive the value of each CPMV based on the MV values ​​of neighboring blocks corresponding to control points exhibit one or more drawbacks. For example, current techniques do not utilize the correlation between the affine motion model of the current block and the affine motion models of neighboring blocks.

[0036] According to one or more techniques of the present invention, a video decoder can determine the value of the motion vector of the affine motion model of the current video data block based on the values ​​of the motion vectors of the affine motion models of specific neighboring blocks of video data and the difference between the value of the motion vector of the current video data block and the value of the motion vector derived from the affine motion models of neighboring video data blocks. For example, the video decoder can use the CPMV of neighboring blocks as the CPMVP of the CPMV of the current block. Because the CPMV of neighboring blocks can be correlated with the CMPV of the current block, the difference (MVD) between the predictor (e.g., CPMVP) and the motion vector (e.g., CMPV) of the current block can be reduced. In this way, since the amount of data used to encode the difference can be proportional to the size of the difference, the technique of the present invention can improve the efficiency of video compression.

[0037] A four-parameter affine motion model has been developed in Huawei Technologies Co., Ltd.’s document “Affinetransform prediction for next generation video coding”, ITU-T SG 16 (2013 study cycle) Proposal 1016 (hereinafter “Proposal 1016”), which is available from itu.int / md / T13-SG16-C-1016 / en. Proposal 1016 introduces the four-parameter affine model shown in Equation (1) below.

[0038]

[0039] Where (v 0x v 0y ) is the top-left corner CPMV of the current block and (v 1x v 1y ) is the CPMV of the upper right corner of the current block. The affine motion model (also called the motion vector field (MVF)) can be represented by the following equation (2).

[0040]

[0041] The four-parameter affine model shown in equation (1) above may exhibit one or more drawbacks. Specifically, the four-parameter affine motion constrains the affine parameters of the x and y components, thereby forcing the x and y components to have symmetric proportional adjustment properties. However, this constraint may not be accurate in diverse video content.

[0042] According to one or more techniques of the present invention, the video decoder may selectively utilize a four-parameter affine motion model or a six-parameter affine motion model. For example, the video decoder may determine whether to use the four-parameter affine motion model shown above in Equation (1) or the six-parameter affine motion model shown below in Equation (3) to decode the current block.

[0043]

[0044] In some instances, the video decoder can determine which affine motion model to use based on explicit signaling. For example, the video decoder can decode syntax elements from the bitstream indicating whether the affine motion model used for the current video data block includes a four-parameter model or a six-parameter model. In some instances, syntax elements can be decoded in one or more of the Video Parameter Set (VPS), Sequence Parameter Set (SPS), Picture Parameter Set (PPS), and slice header referenced by the current video data block. In some instances, syntax elements can be decoded at the decoding unit (CU) level of the CU containing the current video data block.

[0045] The processing and / or signal representation requirements of a four-parameter model are lower than those of a six-parameter model. However, in some instances, a six-parameter model can produce a prediction block that better matches the block being decoded, which can reduce the size of the residual value. Therefore, in some instances, a video encoder can balance the processing and signal representation costs of encoding blocks using a six-parameter model with the benefits of reduced residual values ​​in the block, and can choose the more advantageous model. In this way, the technique of the present invention can further improve the efficiency of video compression using an affine motion model.

[0046] Figure 1 A block diagram illustrating an example video encoding and decoding system 10 that can utilize the affine motion compensation technique for performing the present invention. (See diagram below.) Figure 1 As shown, system 10 includes a source device 12 that provides encoded video data to be decoded by a destination device 14 at a later time. Specifically, source device 12 provides the video data to destination device 14 via computer-readable medium 16. Source device 12 and destination device 14 can include any of a wide range of devices, including desktop computers, notebook (i.e., laptop) computers, tablet computers, set-top boxes, mobile phones such as so-called "smart" phones, so-called "smart" boards, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, or the like. In some cases, source device 12 and destination device 14 may be equipped for wireless communication.

[0047] Destination device 14 may receive encoded video data to be decoded via computer-readable medium 16. Computer-readable medium 16 may include any type of media or device capable of moving encoded video data from source device 12 to destination device 14. In one example, computer-readable medium 16 may include communication media enabling source device 12 to transmit encoded video data directly to destination device 14 in real time. The encoded video data may be modulated according to a communication standard (e.g., a wireless communication protocol) and transmitted to destination device 14. Communication media may include any wireless or wired communication media, such as radio frequency (RF) spectrum or one or more physical transmission lines. Communication media may form part of a packet-based network (e.g., a local area network, a wide area network, or a global network, such as the Internet). Communication media may include routers, switches, base stations, or any other equipment suitable for facilitating communication from source device 12 to destination device 14.

[0048] In some instances, encoded data can be output from output interface 22 to a storage device. Similarly, encoded data can be accessed from the storage device via an input interface. The storage device can comprise any of a variety of distributed or locally accessible data storage media, such as hard disk drives, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded video data. In another instance, the storage device can correspond to a file server or another intermediate storage device capable of storing the encoded video generated by source device 12. Destination device 14 can access the stored video data from the storage device via streaming or downloading. The file server can be any type of server capable of storing and transmitting the encoded video data to destination device 14. Example file servers include web servers (e.g., for websites), FTP servers, network attached storage (NAS) devices, or local disk drives. Destination device 14 can access the encoded video data via any standard data connection, including an Internet connection. The data connection may include a wireless channel (e.g., Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both, suitable for accessing encoded video data stored on a file server. Transmission of encoded video data from the storage device may be streaming, downloading, or a combination thereof.

[0049] The technology of this invention is not limited to wireless applications or settings. It can be applied to video decoding to support any of a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, internet streaming video transmission (e.g., Dynamic Adaptive Streaming (DASH) via HTTP), digital video encoded to data storage media, decoding of digital video stored on data storage media, or other applications. In some instances, system 10 can be configured to support one-way or two-way video transmission to support applications (e.g., video streaming, video playback, video broadcasting, and / or video telephony applications).

[0050] exist Figure 1 In one example, source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Destination device 14 includes an input interface 28, a video decoder 30, and a display device 32. According to the invention, the video encoder 20 of source device 12 can be configured to apply affine motion compensation techniques for performing the invention. In other examples, the source and destination devices may include other components or arrangements. For example, source device 12 may receive video data from an external video source 18 (e.g., an external camera). Similarly, destination device 14 may interface with an external display device rather than including an integrated display device.

[0051] Figure 1The system 10 described herein is merely one example. The techniques used to perform the affine motion compensation of the present invention can be implemented by any digital video encoding and / or decoding device. Although the techniques of the present invention are generally implemented by video encoding devices, they can also be implemented by video encoders / decoders (commonly referred to as "CODECs"). Furthermore, the techniques of the present invention can also be implemented by video preprocessors. Source device 12 and destination device 14 are merely examples of decoding devices that generate decoded video data for transmission to destination device 14 from source device 12. In some instances, devices 12, 14 can operate in a generally symmetrical manner, such that each of devices 12, 14 includes video encoding and decoding components. Thus, system 10 can support one-way or two-way video transmission between video devices 12, 14 for purposes such as video streaming, video playback, video broadcasting, or video telephony.

[0052] The video source 18 of the source device 12 may include a video capture device, such as a camera, a video archive containing previously captured video, and / or a video feed interface for receiving video from a video content provider. Alternatively, the video source 18 may generate computer graphics-based data as source video, or a combination of live video, archived video, and computer-generated video. In some cases, if the video source 18 is a camera, then the source device 12 and the destination device 14 may form a so-called camera phone or video phone. However, as mentioned above, the techniques described in this invention are generally applicable to video decoding and to wireless and / or wired applications. In each case, the captured, pre-captured, or computer-generated video may be encoded by the video encoder 20. The encoded video information may then be output from the output interface 22 to the computer-readable medium 16.

[0053] Computer-readable media 16 may comprise transient media, such as wireless broadcasting or wired network transmissions, or storage media (i.e., non-transient storage media), such as hard disks, flash drives, compressed optical discs, digital video optical discs, Blu-ray discs, or other computer-readable media. In some instances, a network server (not shown) may receive encoded video data from source device 12 and, for example, provide the encoded video data to destination device 14 via network transmission. Similarly, a computing device in a media production facility (e.g., an optical disc stamping facility) may receive encoded video data from source device 12 and produce optical discs containing the encoded video data. Therefore, in various instances, it is understood that computer-readable media 16 may comprise one or more computer-readable media of various forms.

[0054] The input interface 28 of the destination device 14 receives information from the computer-readable medium 16. The information on the computer-readable medium 16 may include grammatical information defined by the video encoder 20, which is also used by the video decoder 30, and includes descriptive blocks and other characteristics and / or processed grammatical elements of the decoding unit. The display device 32 displays the decoded video data to the user and may include any of a variety of display devices, such as a cathode ray tube (CRT), liquid crystal display (LCD), plasma display, organic light-emitting diode (OLED) display, or another type of display device.

[0055] The video encoder 20 and video decoder 30 can operate according to video decoding standards, such as the High Efficiency Video Decoding (HEVC) standard, also known as ITU-T H.265. Alternatively, the video encoder 20 and video decoder 30 can operate according to other proprietary or industry standards (such as the ITU-T H.264 standard, alternatively known as MPEG-4, Part 10, Advanced Video Decoding (AVC)) or extensions of these standards. However, the technology of the present invention is not limited to any particular decoding standard. Other examples of video decoding standards include MPEG-2 and ITU-T H.263. Although Figure 1 Not shown, but in some respects, the video encoder 20 and video decoder 30 may each be integrated with the audio encoder and decoder, and may include appropriate MUX-DEMUX units or other hardware and software to handle the encoding of both audio and video in a common data stream or separate data streams. If applicable, the MUX-DEMUX unit may comply with the ITU H.223 multiplexer protocol or other protocols such as User Datagram Protocol (UDP).

[0056] The video encoder 20 and video decoder 30 can each be implemented as any of a variety of suitable encoder circuits, such as one or more microprocessors, processing circuitry (including fixed-function circuitry and / or programmable processing circuitry), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the technical portion is implemented in software, the device may store instructions for the software in a suitable non-transitory computer-readable medium, and execute the instructions in hardware using one or more processors to perform the technology of the invention. Each of the video encoder 20 and video decoder 30 may be included in one or more encoders or decoders, and any of the encoders or decoders may be integrated as part of a combined encoder / decoder (CODEC) in the respective device.

[0057] Generally, according to ITU-T H.265, video frames can be divided into a series of code tree units (CTUs) (or maximum code unit (LCU)) that can contain both luminance and chrominance samples. Alternatively, a CTU can contain monochrome data (i.e., luminance samples only). The syntax data within the bitstream defines the size of the CTU, which is the maximum code unit in terms of the number of pixels. A slice contains several consecutive CTUs in decoding order. A video frame can be divided into one or more slices. Each CTU can be split into code units (CUs) according to a quartet. Generally, the quartet data structure contains one node per CU, where the root node corresponds to the CTU. If a CU is split into four sub-CUs, then the node corresponding to that CU contains four leaf nodes, each of which corresponds to one of the sub-CUs.

[0058] Each node in a quartic tree data structure provides syntax data for the corresponding CU. For example, a node in a quartic tree may contain a split flag, indicating whether the CU corresponding to that node has split into child CUs. The syntax elements for a CU can be defined recursively and can depend on whether the CU has split into child CUs. If a CU is not further split, it is called a leaf CU. In this invention, even if there is no obvious split of the original leaf CU, the four child CUs of the leaf CU will also be called leaf CUs. For example, if a 16×16 CU is not further split, then the four 8×8 child CUs will also be called leaf CUs, even though the 16×16 CU has never split.

[0059] Except that CUs do not have size distinctions, they serve a similar purpose to macroblocks in the H.264 standard. For example, a CTU can be split into four child nodes (also called child CUs), and each child node can then become a parent node and split into four more child nodes. The final unsplit child node of a leaf node, called a quadtree, includes a decoding node, also called a leaf CU. The syntax data associated with the decoded bitstream can define the maximum number of times a CTU can be split (called the maximum CU depth) and can also define the minimum size of the decoding node. Therefore, the bitstream can also define a minimum decoding unit (SCU). This invention uses the term "block" to refer to a CU, prediction unit (PU), or transform unit (TU) in the context of HEVC, or any similar data structure in the context of other standards (e.g., macroblocks and their subblocks in H.264 / AVC).

[0060] A CU comprises a decoding node and associated prediction units (PUs) and transform units (TUs). The size of the CU corresponds to the size of the decoding node and is generally square in shape. The size of the CU can range from 8×8 pixels to the size of a CTU with a maximum size (e.g., 64×64 pixels or larger). Each CU may contain one or more PUs and one or more TUs. The syntax data associated with the CU may describe, for example, the segmentation of the CU into one or more PUs. The segmentation mode may differ between skip or direct mode coding, intra-frame prediction mode coding, or inter-frame prediction mode coding. PUs may be segmented into non-square shapes. The syntax data associated with the CU may also describe, for example, the segmentation of the CU into one or more TUs according to a quadtree. TUs may be square or non-square (e.g., rectangular) in shape.

[0061] The HEVC standard allows transformations based on Transform Units (TUs), which can differ for different CUs. A TU is typically sized based on the size of the PU (the segmented region of the CU) within a given CU, defined for segmenting the CTU, but this is not always the case. A TU is usually the same size as or smaller than the PU (or, for example, the segmented region of the CU in the case of intra-frame prediction). In some instances, a quadtree structure called a "Residual Quadtree" (RQT) can be used to further divide the residual samples corresponding to the CU into smaller units. The leaf nodes of the RQT can be called Transform Units (TUs). The pixel differences associated with the TU can be transformed to produce quantizable transform coefficients.

[0062] A leaf CU may contain one or more prediction units (PUs) when performing prediction using inter-frame prediction. Generally, a PU represents a spatial region corresponding to all or part of the corresponding CU and may contain data for retrieving and / or generating reference samples for the PU. In addition, the PU contains prediction-related data. When the CU is coded in an inter-frame mode, one or more PUs of the CU may contain data defining motion information, such as one or more motion vectors, or the PU may be decoded in a skip mode. The data defining the motion vectors of the PU may describe, for example, the horizontal component of the motion vector, the vertical component of the motion vector, the resolution of the motion vector (e.g., quarter-pixel accuracy or eighth-pixel accuracy), the reference picture to which the motion vector points, and / or a list of reference pictures of the motion vector (e.g., list 0 or list 1).

[0063] Leaf CUs can also be predicted using intra-frame patterns. Generally, intra-frame prediction involves predicting leaf CUs (or their segmented regions) using intra-frame patterns. The video decoder can select a set of previously decoded pixels adjacent to the leaf CU to predict the leaf CU (or its segmented regions).

[0064] A leaf CU may also contain one or more transform units (TUs). As discussed above, the transform unit can be specified using an RQT (also known as a TU quadtree structure). For example, a split flag can indicate whether a leaf CU is split into four transform units. Each TU can then be further split into other sub-TUs. When a TU is not further split, it can be called a leaf TU. Generally, for intra-frame decoding, all leaf TUs belonging to a leaf CU share the same intra-frame prediction mode. That is, the same intra-frame prediction mode is generally used to calculate the prediction values ​​of all TUs of the leaf CU. For intra-frame decoding, the video encoder can use the intra-frame prediction mode to calculate the residual value of each leaf TU as the difference between the portion of the CU corresponding to that TU and the original block. The TU is not limited to the size of the PU. Therefore, a TU can be larger or smaller than the PU. For intra-frame decoding, the segmented region of the CU or the CU itself can coexist with the corresponding leaf TU of the CU. In some instances, the maximum size of a leaf TU can correspond to the size of the corresponding leaf CU.

[0065] Furthermore, the TU of a leaf CU can also be associated with a corresponding quadtree data structure (called a residual quadtree (RQT)). That is, a leaf CU can contain a quadtree indicating how the leaf CU is partitioned into TUs. The root node of the TU quadtree generally corresponds to the leaf CU, while the root node of the CU quadtree generally corresponds to the CTU (or LCU). The unsplit TU of the RQT is referred to as a leaf TU. Generally, unless otherwise indicated, the terms CU and TU are used in this invention to refer to leaf CU and leaf TU, ​​respectively.

[0066] A video sequence typically comprises a series of video frames or images, beginning with a random access point (RAP) image. The video sequence may contain syntax data in a sequence parameter set (SPS), describing the characteristics of the video sequence. Each slice of an image may contain slice syntax data describing the encoding mode of that corresponding slice. The video encoder 20 typically operates on video blocks within individual video slices to encode video data. Video blocks may correspond to decoding nodes within a CU. Video blocks may have a fixed or variable size, and their size may vary depending on a specified decoding standard.

[0067] As an example, predictions can be performed for PUs of various sizes. Assuming a specific CU is 2N×2N, intra-frame prediction can be performed for PUs of 2N×2N or N×N size, and inter-frame prediction can be performed for symmetrical PUs of 2N×2N, 2N×N, N×2N, or N×N size. Asymmetric segmentation with inter-frame prediction can also be performed for PUs of 2N×nU, 2N×nD, nL×2N, and nR×2N size. In asymmetric segmentation, the CU is not segmented in one direction, while it is segmented into 25% and 75% segments in the other direction. The portion of the CU corresponding to the 25% segmentation is indicated by "n" plus an indication of "Up," "Down," "Left," or "Right." Therefore, for example, "2N×nU" refers to a 2N×2NCU horizontally segmented with a 2N×0.5N PU at the top and a 2N×1.5N PU at the bottom.

[0068] In this invention, "N×N" and "N multiplied by N" are used interchangeably to refer to the pixel size of a video block in terms of both vertical and horizontal dimensions, for example, 16×16 pixels or 16 by 16 pixels. Generally, a 16×16 block will have 16 pixels in the vertical direction (y = 16) and 16 pixels in the horizontal direction (x = 16). Similarly, an N×N block typically has N pixels in the vertical direction and N pixels in the horizontal direction, where N represents a non-negative integer value. Pixels in the block can be arranged in rows and columns. Furthermore, the block does not necessarily need to have the same number of pixels in both the horizontal and vertical directions. For example, a block may include N×M pixels, where M is not necessarily equal to N.

[0069] Picture order count (POC) is widely used in video decoding standards to identify the display order of pictures. Although it is possible for two pictures within a decoded video sequence to have the same POC value, this is generally not the case within a decoded video sequence. When there are multiple decoded video sequences in a bitstream, pictures with the same POC value are closer to each other in terms of decoding order. Picture POC values ​​are commonly used for constructing reference picture lists, such as deriving reference picture sets in HEVC, and scaling motion vectors.

[0070] Motion compensation in HEVC is used to generate predictors for the current inter-frame block. A quarter-pixel accuracy motion vector is used, and the pixel value at the fractional position is interpolated using adjacent integer pixel values ​​for both the luminance and chrominance components.

[0071] In HEVC, for each block, a set of motion information is available. This set of motion information contains motion information for both the forward and backward prediction directions. Here, the forward and backward prediction directions are two prediction directions in a bidirectional prediction mode, and the terms "forward" and "backward" do not necessarily have a geometric meaning; in fact, they correspond to the reference image list 0 (RefPicList0) and reference image list 1 (RefPicList1) of the current image. When only one reference image list is available for an image or slice, only RefPicList0 is available, and the motion information for each block of the slice is always forward.

[0072] For each predicted direction, the motion information must contain a reference index and a motion vector. In some cases, for simplicity, the motion vector can be referenced by assuming that it itself has an associated reference index. The reference index is used to identify the reference image in the current list of reference images (RefPicList0 or RefPicList1). The motion vector has horizontal and vertical components.

[0073] In the HEVC standard, there are two inter-frame prediction modes for prediction units (PUs): merge (skipping special cases considered as merges) and Advanced Motion Vector Prediction (AMVP) mode. In either AMVP or merge mode, a motion vector (MV) candidate list is maintained for multiple motion vector predictors. The motion vector of the current PU (and the reference index in merge mode) is generated by obtaining a candidate from the MV candidate list.

[0074] The MV candidate list contains up to five candidates for the merge mode and only two candidates for the AMVP mode. A merge candidate may contain a set of motion information, such as motion vectors and reference indices corresponding to two reference image lists (list 0 and list 1). If the merge candidate is identified by the merge index, then the reference image is used for prediction of the current block and to determine the associated motion vector. However, in AMVP mode, for each potential prediction direction from list 0 or list 1, the reference index needs to be explicitly represented as a signal along with the MVP index for the MV candidate list, because AMVP candidates only contain motion vectors. In AMVP mode, the predicted motion vectors can be further improved.

[0075] As seen above, merged candidates can correspond to the entire set of motion information, while AMVP candidates may contain only a motion vector and a reference index for a specific prediction direction. Candidates for both modes are derived from the same spatially and temporally adjacent blocks in a similar manner. Further details regarding spatially adjacent candidates for merged and AMVP modes are referenced below. Figure 4A and 4B And so on.

[0076] The video encoder 20 and video decoder 30 can be configured to perform motion compensation using an affine motion model. For example, instead of using a translational motion model with only a single two-dimensional motion vector (i.e., as in HEVC), the video encoder 20 and video decoder 30 can utilize an affine motion model containing multiple motion vectors. Further details on using an affine motion model are discussed below.

[0077] After intra-frame predictive or inter-frame predictive decoding of the PU of the CU, the video encoder 20 may compute residual data of the TU of the CU. The PU may include syntax data describing the method or pattern of generating predictive pixel data in the spatial domain (also known as the pixel domain), and the TU may include coefficients in the transform domain after applying a transform (e.g., discrete cosine transform (DCT), integer transform, wavelet transform, or conceptually similar transform) to the residual video data. The residual data may correspond to the pixel difference between the pixels of the uncoded image and the predicted value corresponding to the PU. The video encoder 20 may form a TU containing quantized transform coefficients representing the residual data of the CU. That is, the video encoder 20 may compute the residual data (in the form of residual blocks), transform the residual blocks to produce blocks of transform coefficients, and then quantize the transform coefficients to form quantized transform coefficients. The video encoder 20 may form a TU containing quantized transform coefficients, as well as other syntax information (e.g., splitting information of the TU).

[0078] As mentioned above, after any transformation to produce transform coefficients, the video encoder 20 may perform quantization of the transform coefficients. Quantization generally refers to the process of quantizing transform coefficients to potentially reduce the amount of data used to represent the transform coefficients, thereby providing further compression. The quantization process may reduce the bit depth associated with some or all of the coefficients. For example, an n-bit value may be down-rounded to an m-bit value during quantization, where n is greater than m.

[0079] After quantization, the video encoder can scan the transform coefficients to generate a one-dimensional vector from a two-dimensional matrix containing the quantized transform coefficients. The scan can be designed to place higher-energy (and therefore lower-frequency) coefficients at the front of the array and lower-energy (and therefore higher-frequency) coefficients at the back. In some instances, the video encoder 20 can utilize a predefined scan order to scan the quantized transform coefficients to generate a serialized vector that can be entropy-encoded. In other instances, the video encoder 20 can perform adaptive scanning. After scanning the quantized transform coefficients to form a one-dimensional vector, the video encoder 20 can entropy-encode the one-dimensional vector, for example, according to context-adaptive variable-length decoding (CAVLC), context-adaptive binary arithmetic decoding (CABAC), syntax-based context-adaptive binary arithmetic decoding (SBAC), probabilistic interval partitioned entropy (PIPE) decoding, or another entropy coding method. The video encoder 20 can also entropy-encode the syntax elements associated with the encoded video data for use by the video decoder 30 to decode the video data.

[0080] To perform CABAC, the video encoder 20 can assign context within a context model to the symbol to be emitted. This context may involve, for example, whether the neighboring values ​​of the symbol are non-zero. To perform CAVLC, the video encoder 20 can select a variable-length code for the symbol to be emitted. Codewords in the VLC can be constructed such that relatively shorter codes correspond to more likely symbols, while longer codes correspond to less likely symbols. In this way, bit savings can be achieved using VLC compared to, for example, using an equal-length codeword for each symbol to be emitted. Probability determination can be based on the context assigned to the symbol.

[0081] In general, video decoder 30 performs a process that is largely similar to, but inverse of, the process performed by video encoder 20, to decode encoded data. For example, video decoder 30 dequantizes and inverse transforms the coefficients of the received TU to regenerate the residual block. Video decoder 30 uses a signal representation prediction mode (intra-frame prediction or inter-frame prediction) to form a predicted block. Video decoder 30 then combines the predicted block with the residual block (on a pixel-by-pixel basis) to regenerate the original block. Additional processing may be performed, such as performing a deblocking process to reduce visual artifacts along block boundaries. Additionally, video decoder 30 can use CABAC decoding syntax elements in a manner that is largely similar to, but inverse of, the CABAC encoding process of video encoder 20.

[0082] The video encoder 20 may further (e.g.) send syntax data (e.g., block-based syntax data, image-based syntax data, and sequence-based syntax data) to the video decoder 30 in the image header, block header, and slice header, or send other syntax data, such as sequence parameter sets (SPS), image parameter sets (PPS), or video parameter sets (VPS).

[0083] The video encoder 20 and video decoder 30 can each be implemented as any of a variety of suitable encoder or decoder circuits, such as one or more microprocessors, processing circuitry (including fixed-function circuitry and / or programmable processing circuitry), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic circuits, software, hardware, firmware, or any combination thereof. Each of the video encoder 20 and video decoder 30 can be included in one or more encoders or decoders, and any of the encoders or decoders can be integrated as part of a combined encoder / decoder (CODEC). Devices including the video encoder 20 and / or video decoder 30 can include integrated circuits, microprocessors, and / or wireless communication devices (e.g., cellular phones).

[0084] Figure 2 This is a block diagram illustrating an example of a video encoder 20 that can implement the affine motion compensation technique of the present invention. The video encoder 20 can perform intra-frame decoding and inter-frame decoding of video blocks within a video slice. Intra-frame decoding relies on spatial prediction to reduce or remove spatial redundancy in the video within a given video frame or image. Inter-frame decoding relies on temporal prediction to reduce or remove temporal redundancy in the video within adjacent frames or images of a video sequence. Intra-frame mode (I-mode) can refer to any of several spatial-based decoding modes. Inter-frame mode (e.g., one-way prediction (P-mode) or two-way prediction (B-mode)) can refer to any of several time-based decoding modes.

[0085] like Figure 2 As shown, video encoder 20 receives the current video block within the video frame to be encoded. Figure 2 In this example, the video encoder 20 includes a mode selection unit 40, a reference picture memory 64 (which may also be called a decoded picture buffer (DPB)), a summer 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The mode selection unit 40 further includes a motion compensation unit 44, a motion estimation unit 42, an intra-frame prediction unit 46, and a segmentation unit 48. For video block reconstruction, the video encoder 20 also includes an inverse quantization unit 58, an inverse transform unit 60, and a summer 62. It may also include a deblocking filter (…). Figure 2(Not shown) Block boundary filtering is used to remove block artifacts from the reconstructed video. If necessary, the deblocking filter will typically filter the output of summer 62. In addition to the deblocking filter, additional filters (in-loop or after-loop) may be used. These filters are not shown for simplicity, but if needed, they can filter the output of summer 50 (as in-loop filters).

[0086] During the encoding process, video encoder 20 receives video frames or slices to be decoded. Frames or slices can be divided into multiple video blocks. Motion estimation unit 42 and motion compensation unit 44 perform inter-frame predictive coding of the received video blocks relative to one or more blocks in one or more reference frames to provide temporal prediction. Intra-frame prediction unit 46 may alternatively perform intra-frame predictive coding of the received video blocks relative to one or more adjacent blocks in the same frame or slice as the block to be decoded to provide spatial prediction. Video encoder 20 may perform multiple decoding passes, for example, to select an appropriate decoding mode for each video data block.

[0087] Furthermore, segmentation unit 48 can segment blocks of video data into sub-blocks based on an evaluation of previous segmentation schemes in previous decoding passes. For example, segmentation unit 48 can first segment frames or slices into CTUs, and then segment each of the CTUs into sub-CUs based on bitrate-distortion analysis (e.g., bitrate-distortion optimization). Mode selection unit 40 can further generate a quadtree data structure indicating the segmentation of CTUs into sub-CUs. The leaf nodes of the quadtree, CUs, can contain one or more PUs and one or more TUs.

[0088] The mode selection unit 40 can select a prediction mode, either intra-frame or inter-frame (e.g., based on error results), and provides the resulting predicted block to the summer 50 to generate residual data and to the summer 62 to reconstruct the coded block for use as a reference frame. The mode selection unit 40 also provides syntax elements (e.g., motion vectors, intra-frame mode indicators, segmentation information, and other such syntax information) to the entropy coding unit 56.

[0089] The motion estimation unit 42 and motion compensation unit 44 can be highly integrated, but are described separately for conceptual purposes. Motion estimation performed by the motion estimation unit 42 is a process of generating motion vectors that estimate the motion of video blocks. For example, a motion vector may indicate the displacement of a video block's PU within the current video frame or picture relative to a predictive block within a reference picture (or other decoded unit) relative to the current block being decoded within the current picture (or other decoded unit). The predictive block is a block found to closely match the block to be decoded based on pixel differences, which can be determined by the sum of absolute differences (SAD), sum of squared differences (SSD), or other difference metrics. In some instances, the video encoder 20 may calculate values ​​for sub-integer pixel positions of a reference picture stored in the reference picture memory 64. For example, the video encoder 20 may interpolate values ​​for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of the reference picture. Therefore, the motion estimation unit 42 can perform motion search relative to full-pixel positions and fractional pixel positions and output motion vectors with fractional-pixel accuracy.

[0090] The motion estimation unit 42 calculates the motion vector of the PU in the inter-frame decoded slice by comparing the position of the PU with the position of a predictive block in a reference image. The reference images can be selected from a first list of reference images (list 0) or a second list of reference images (list 1), each of which identifies one or more reference images stored in the reference image memory 64. The motion estimation unit 42 sends the calculated motion vector to the entropy coding unit 56 and the motion compensation unit 44.

[0091] Motion compensation performed by motion compensation unit 44 may involve extracting or generating predictive blocks based on motion vectors determined by motion estimation unit 42. Again, in some instances, motion estimation unit 42 and motion compensation unit 44 may be functionally integrated. After receiving the motion vector of the PU for the current video block, motion compensation unit 44 may locate the predictive block to which the motion vector points in one of the reference image lists. Summer 50 forms a residual video block by subtracting the pixel values ​​of the predictive block from the pixel values ​​of the decoded current video block, thus forming a pixel difference, as discussed below. Generally, motion estimation unit 42 performs motion estimation relative to the luminance component, and motion compensation unit 44 uses the motion vector calculated based on the luminance component for both the chroma and luminance components. Mode selection unit 40 may also generate syntax elements associated with video blocks and video slices for use by video decoder 30 during the decoding of video slices.

[0092] Video encoder 20 can be configured to perform the above-mentioned functions. Figure 1Any of the various techniques of the present invention discussed herein will be described in more detail below. For example, motion compensation unit 44 may be configured to decode motion information of video data blocks using AMVP or merge mode according to HEVC, and / or may be configured to decode affine motion information or video data blocks using affine inter-frame mode or affine merge mode according to the techniques of the present invention.

[0093] As described above, as an alternative to inter-frame prediction performed by motion estimation unit 42 and motion compensation unit 44, intra-frame prediction unit 46 may perform intra-frame prediction for the current block. Specifically, intra-frame prediction unit 46 may determine the intra-frame prediction mode to be used for encoding the current block. In some instances, intra-frame prediction unit 46 may, for example, encode the current block using various intra-frame prediction modes during separate encoding passes, and intra-frame prediction unit 46 (or, in some instances, mode selection unit 40) may select an appropriate intra-frame prediction mode from test modes for use.

[0094] For example, intra-prediction unit 46 can use bitrate-distortion analysis for various tested intra-prediction modes to calculate bitrate-distortion values ​​and select the intra-prediction mode with the best bitrate-distortion characteristics among the tested modes. Bitrate-distortion analysis generally determines the amount of distortion (or error) between the coded block and the original, uncoded block (which is encoded to produce the coded block), and the bitrate (i.e., the number of bits) used to produce the coded block. Intra-prediction unit 46 can calculate a ratio based on the distortion and bitrate of different coded blocks to determine which intra-prediction mode exhibits the best bitrate-distortion value for the block.

[0095] After selecting an intra-prediction mode for a block, the intra-prediction unit 46 may provide information indicating the selected intra-prediction mode for the block to the entropy coding unit 56. The entropy coding unit 56 may encode the information indicating the selected intra-prediction mode. The video encoder 20 may include in the transmitted bitstream: configuration data, which may include multiple intra-prediction mode index tables and multiple modified intra-prediction mode index tables (also called codeword mapping tables); definitions of the coding contexts for various blocks; and indications of the most probable intra-prediction mode to be used for each of the contexts, the intra-prediction mode index tables, and the modified intra-prediction mode index tables.

[0096] Video encoder 20 forms a residual video block by subtracting the predicted data from mode selection unit 40 from the original video block being decoded. Summer 50 represents one or more components performing the subtraction operation. Transform processing unit 52 applies a transform (e.g., Discrete Cosine Transform (DCT) or a conceptually similar transform) to the residual block, thereby producing a video block including residual transform coefficient values. Wavelet transform, integer transform, subband transform, Discrete Sine Transform (DST), or other types of transforms can be used instead of DCT. In any case, transform processing unit 52 applies a transform to the residual block, thereby producing a transform coefficient block. The transform can convert residual information from the pixel domain to the transform domain, e.g., the frequency domain. Transform processing unit 52 can send the resulting transform coefficients to quantization unit 54. Quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process can reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be modified by adjusting the quantization parameters.

[0097] After quantization, the entropy coding unit 56 entropy decodes the quantized transform coefficients. For example, the entropy coding unit 56 can perform context-adaptive variable-length decoding (CAVLC), context-adaptive binary arithmetic decoding (CABAC), syntax-based context-adaptive binary arithmetic decoding (SBAC), probabilistic interval partitioning entropy (PIPE) decoding, or another entropy decoding technique. In the case of context-based entropy decoding, the context can be based on adjacent blocks. After entropy decoding by the entropy coding unit 56, the encoded bitstream can be transmitted to another device (e.g., video decoder 30) or archived for later transmission or retrieval.

[0098] The inverse quantization unit 58 and the inverse transform unit 60 apply inverse quantization and inverse transform respectively to reconstruct the residual blocks in the pixel domain. Specifically, the summer 62 adds the reconstructed residual blocks to the motion-compensated prediction blocks generated by the motion compensation unit 44 or the intra-frame prediction unit 46 to produce reconstructed video blocks for storage in the reference image memory 64. These reconstructed video blocks can be used by the motion estimation unit 42 and the motion compensation unit 44 as reference blocks for inter-frame decoding of blocks in subsequent video frames.

[0099] Figure 3 This is a block diagram illustrating an example of a video decoder 30 that can implement the affine motion compensation technique used to perform the present invention. Figure 3 In one example, the video decoder 30 includes an entropy decoding unit 70, a motion compensation unit 72, an intra-frame prediction unit 74, an inverse quantization unit 76, an inverse transform unit 78, a reference image memory 82, and a summer 80. In some instances, the video decoder 30 can perform functions substantially similar to those of the video encoder 20. Figure 2The decoding iterations are the inverse of the encoding iterations described above. Motion compensation unit 72 can generate prediction data based on motion vectors received from entropy decoding unit 70, while intra-frame prediction unit 74 can generate prediction data based on intra-frame prediction mode indicators received from entropy decoding unit 70.

[0100] During the decoding process, video decoder 30 receives from video encoder 20 a encoded video bitstream representing video blocks of encoded video slices and associated syntax elements. Entropy decoding unit 70 of video decoder 30 entropy decodes the bitstream to produce quantized coefficients, motion vectors or intra-prediction mode indicators, and other syntax elements. Entropy decoding unit 70 forwards the motion vectors and other syntax elements to motion compensation unit 72. Video decoder 30 may receive syntax elements at the video slice level and / or video block level.

[0101] When a video slice is decoded into an intra-frame decoded (I) slice, the intra-frame prediction unit 74 can generate prediction data for the video block of the current video slice based on the signal representing the intra-frame prediction mode and data from previously decoded blocks from the current frame or picture. When a video frame is decoded into an inter-frame decoded (i.e., B or P) slice, the motion compensation unit 72 generates predictive blocks for the video block of the current video slice based on motion vectors and other syntax elements received from the entropy decoding unit 70. Predictive blocks can be generated from one of the reference pictures in a reference picture list. The video decoder 30 can construct a reference frame list: list 0 and list 1, based on reference pictures stored in the reference picture memory 82 using a default construction technique.

[0102] Motion compensation unit 72 determines prediction information for video blocks in the current video slice by analyzing motion vectors and other syntax elements, and uses the prediction information to generate predictive blocks for properly decoded current video blocks. For example, motion compensation unit 72 uses some of the received syntax elements to determine the prediction mode (e.g., intra-frame or inter-frame prediction) for video blocks used to decode the video slice, the inter-frame prediction slice type (e.g., B-slice or P-slice), construction information for one or more of the reference picture lists for the slice, motion vectors for each inter-frame encoded video block of the slice, inter-frame prediction state for each inter-frame decoded video block of the slice, and other information for decoding video blocks in the current video slice.

[0103] Video decoder 30 can be configured to perform the above-mentioned functions. Figure 1Any of the various techniques of the invention discussed herein, as will be described in more detail below. For example, motion compensation unit 72 may be configured to perform motion vector prediction using AMVP or merging mode according to HEVC, and / or may be configured to perform affine motion information or video data blocks using affine inter-frame mode or affine merging mode according to the techniques of the invention. Entropy decoding unit 70 may decode representations of how motion information is decoded for one or more syntax elements of the current block.

[0104] The motion compensation unit 72 can also perform interpolation based on an interpolation filter. The motion compensation unit 72 can use an interpolation filter, such as that used by the video encoder 20 during video block encoding, to calculate the interpolated values ​​of sub-integer pixels of the reference block. In this case, the motion compensation unit 72 can determine the interpolation filter used by the video encoder 20 based on the received syntax elements and use the interpolation filter to generate a predictive block.

[0105] The dequantization unit 76 dequantizes (i.e., dequantizes) the quantized transform coefficients provided in the bitstream and decodes them by the entropy decoding unit 70. The dequantization process may include using the video decoder 30 to compute the quantization parameter QP for each video block in the video slice. Y To determine the appropriate level of quantization and, similarly, the appropriate level of inverse quantization.

[0106] The inverse transform unit 78 applies an inverse transform (e.g., inverse DCT, inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients in order to produce a residual block in the pixel domain.

[0107] After the motion compensation unit 72 generates a predictive block for the current video block based on motion vectors and other syntax elements, the video decoder 30 forms a decoded video block by summing the residual block from the inverse transform unit 78 with the corresponding predictive block generated by the motion compensation unit 72. The summer 80 represents the component or unit that performs the summation operation. If necessary, a deblocking filter can be applied to filter the decoded block to remove block artifacts. Other loop filters (in or after the decoding loop) can also be used to smooth pixel transitions or further improve video quality. The decoded video block in a given frame or image is then stored in a reference image memory 82, which stores reference images for subsequent motion compensation. The reference image memory 82 also stores the decoded video for later display on a display device (e.g., Figure 1 It is displayed on the display device 32).

[0108] Figure 4A and 4BThis is a conceptual diagram illustrating spatial adjacency candidates in High-Efficiency Video Decoding (HEVC). As discussed above, spatial MV candidates can be derived from neighboring blocks of a specific PU (PU0), but the method for generating candidates from blocks differs for merging and AMVP modes.

[0109] Figure 4A This describes how a video decoder can derive instances of spatial MV candidates in merge mode. In merge mode, the following can be used: Figure 4A The above numbers are used to derive up to four spatial MV candidates, in the following order: left (0), top (1), top right (2), bottom left (3), and top left (4), as shown above. Figure 4A As shown.

[0110] Figure 4B This describes how a video decoder can derive instances of spatial MV candidates in AVMP mode. In AVMP mode, adjacent blocks are divided into two groups: a left group consisting of block 0 and block 1, and an upper group consisting of block 2, block 3, and block 4, as shown below. Figure 4B As shown above. For each group, potential candidates in neighboring blocks of the same reference image indicated by a reference index represented by a signal have the highest priority to be selected to form the final candidates for the group. It is possible that none of the neighboring blocks contain motion vectors pointing to the same reference image. Therefore, if such candidates cannot be found, the first available candidates are scaled up to form the final candidates, thus compensating for the time distance difference.

[0111] Figure 5 A conceptual diagram illustrating the affine transformation of two-point motion vectors with four affine parameters. (See diagram below.) Figure 5 As shown, the CPMV of the top left corner 502 of the current block 500 is represented as v0 (v 0x v 0y And the CPMV of the top right corner 504 of the current block 500 is represented as v1 (v 1x v 1y As discussed above, the CMPV of the current block 500 can form a motion vector field (MVF) according to the above equation (2).

[0112] In the JEM test model, affine motion prediction is only applied to square blocks. As a natural extension, affine motion prediction can be applied to non-square blocks.

[0113] Figure 6A conceptual diagram illustrating the affine inter-frame mode is provided. For blocks (e.g., CU / PU) with a size equal to or greater than 16×16, a video decoder (e.g., video encoder 20 and / or video decoder 30) can apply the following affine inter-frame mode (AF_INTER). In some instances, if the current block (e.g., the current CU / PU) is in the affine inter-frame mode, the video decoder can represent the affine flag at the CU / PU level as a signal in the bitstream. The video decoder can construct a candidate motion vector list for the current block using motion vectors from adjacent valid reconstructed blocks. For example, as in the example... Figure 6 As shown, the candidate motion vector predictor for the top-left CPMV v0 can be selected from the motion vectors of blocks 602A, 602B, and 602C (i.e., the adjacent blocks that are in contact with the top-left corner of the current block 600). The video decoder can proportionally adjust the motion vectors from the adjacent blocks based on the relationship between the reference list and the POCs used for references of adjacent blocks, the POCs used for references of the current CU / PU, and the POCs of the current CU / PU. The video decoder can perform a similar method to select the candidate motion vector predictor for the top-right CPMV v1 from adjacent blocks 602D and 602E (i.e., the adjacent blocks that are in contact with the top-right corner of the current block 600). Thus, in some instances, the candidate list can be represented as {(v0,v1)|v0={v1,v2B,v2C ... 602A ,v 602B ,v 602C},v1={v 602D ,v 602E}}.

[0114] If the number of candidates in the candidate list is less than a threshold (e.g., two, three, or four), the video decoder can assign candidates for the AMVP to v0 and v1. The video decoder can use the bit rate-distortion optimization (RDO) cost of the current block to determine which (v0, v1) is selected as the control point motion vector prediction (CPMVP) for the current block. The video decoder can use a signal to represent an index in the bitstream to indicate the position of the CPMVP in the candidate list.

[0115] Based on the CPMVP of the current affine block, the video decoder can apply affine motion estimation to determine the CPMV. The video decoder can then decode a representation of the difference between the CPMV and CPMVP in the bitstream.

[0116] The video decoder can perform affine motion compensation prediction as described above to generate the residuals for the current block. The video decoder can transform and quantize the generated residuals for the current block, and decode the quantized residuals into the bitstream (e.g., in a manner similar to HEVC).

[0117] Figure 7A and 7BThis is a conceptual diagram illustrating a candidate affine merging mode. When the affine merging (AF_MERGE) mode is applied to the current block, a video decoder (e.g., video encoder 20 and / or video decoder 30) can obtain the first block decoded in affine mode from the valid adjacent reconstructed blocks of the current block. In some instances, the video decoder can parse adjacent reconstructed blocks in a specific selection order to obtain the first block decoded in affine mode. Figure 7A Explain the order in which instances were selected. For example... Figure 7A As shown, the selection order can be as follows: left block 702A, top block 702B, top right block 702C, bottom left block 702D to top left block 702E.

[0118] Figure 7B This indicates that the left block, selected in that order, is an instance of the first block decoded in affine mode. For example... Figure 7B As shown, the video decoder can derive motion vectors containing the top-left (v2), top-right (v3), and bottom-left (v4) corners of the CU / PU 704 of the selected block 1002A. The video decoder can determine / calculate the motion vector (v0) of the top-left corner and the motion vector (v1) of the top-right corner of the current block 700 based on the derived motion vectors (v2, v3, and v4) of the selected block.

[0119] The video decoder can determine the MVF of the current block 700 based on the CPMV v0 and v1 of the current block 700 according to the simplified affine motion model described in equation (2) above. The video decoder can apply the affine MCP using the MVF as described above.

[0120] To identify whether the current block is decoded in affine merge mode, the video decoder may signal an affine flag in the bitstream if at least one adjacent block is decoded in affine mode. If no affine block neighbor exists for the current block, the video decoder may omit decoding the affine flag or a decodeable affine flag in the bitstream to indicate that no affine block neighbor exists for the current block.

[0121] As discussed above, existing affine motion modeling methods (e.g., in the JEM test model and Proposal 1016) present several problems and / or have several drawbacks. As an example, in Proposal 1016, four-parameter affine motion imposes constraints on the affine parameters in MVx and MVy, thereby forcing MVx and MVy to have symmetric, proportionally adjustable properties. These constraints may not be accurate across diverse video content.

[0122] As another example, the affine merging pattern relies on a predefined checking order that primarily depends on the bottom left and top right corners. This predefined order places the top left corner with the lowest priority, and this corner information is used extensively in the following affine model derivation.

[0123] As another example, affine merging can inherit neighboring models simply by changing the adjacent corner MV to the current corner. There is no flexibility to change or adjust the parameters of the affine model when inheriting from a neighboring affine model.

[0124] According to one or more techniques of the present invention, a video decoder can decode instructions on how to identify the syntax elements of predicted sub-blocks of video data. For example, the video decoder can decode instructions on whether to use a four-parameter affine model or a six-parameter affine model to identify the syntax elements of predicted sub-blocks of video data for the current video data block. By enabling selection between a four-parameter affine model and a six-parameter affine model, the techniques of the present invention allow motion vectors to have asymmetric proportional adjustment properties, which improves decoding efficiency.

[0125] In some instances, the video decoder can decode syntax elements at the decoding unit (CU) level. For example, flags can be introduced at the CU level to indicate whether a four-parameter affine motion model or a six-parameter affine motion model is used for the current block in the CU.

[0126] In some instances, the video decoder can decode syntax elements in a skip mode syntax or a merge mode syntax referenced by the current video data block. For example, flags can be introduced in skip or merge modes to indicate whether a four-parameter affine motion model or a six-parameter affine motion model is used for the current block.

[0127] In some instances, the video decoder can decode syntax elements in the inter-frame mode syntax referenced by the current video data block. For example, flags can be introduced in the inter-frame mode (where the current block is neither skip mode nor merge mode) to indicate whether a four-parameter affine motion model or a six-parameter affine motion model is used for the current block.

[0128] In some instances, instead of simply indicating whether a four-parameter or six-parameter affine model is used to identify predicted sub-blocks of video data for the current video data block, the video decoder can decode syntax elements to indicate whether a single motion vector, a four-parameter affine model, a six-parameter affine model, or a switchable four / six-parameter affine model is used to identify predicted sub-blocks of video data for the current video data block. For example, a syntax element from the Sequence Parameter Set (SPS), Picture Parameter Set (PPS), and / or Slice header can be presented to signal which of the following is used for the current sequence / picture / slice: 1) Affine disabled, 2) 4-parameter affine, 3) 6-parameter affine, 4) 4 / 6 switchable affine. The syntax element can be decoded using unary, truncated unary, or fixed-length codewords.

[0129] In some instances, the video decoder can decode whether the number of parameters used in the affine model used to identify predictive sub-blocks of video data is a switchable enabled syntax element. For example, the video decoder can decode flags in the Sequence Parameter Set (SPS), Picture Parameter Set (PPS), and / or slice header to indicate whether a switchable affine model is enabled for referencing the SPS or PPS or slice header picture.

[0130] When the enabled syntax element indicates that the number of parameters used in the affine model for identifying predictive sub-blocks of video data is toggleable (e.g., when the enabled syntax element is a flag with a value of 1), the video decoder can decode the syntax element indicating whether a four-parameter or six-parameter affine model is used to identify predictive sub-blocks of video data for the current video data block as discussed above. For example, when the enabled syntax element indicates that the number of parameters used in the affine model for identifying predictive sub-blocks of video data is toggleable (e.g., when the enabled syntax element is a flag with a value of 1), both the four-parameter and six-parameter affine models are enabled, and an additional flag for each block can be signaled to indicate the use of either the four-parameter or six-parameter model.

[0131] When the number of parameters used in the affine model for identifying predicted sub-blocks of video data is not switchable (e.g., when the enabled syntax element is a flag with a value of 0), the video decoder can determine that a four-parameter affine model is used (i.e., in the case of using affine). In these instances, the video decoder can omit decoding the syntax element indicating whether a four-parameter or six-parameter affine model is used to identify predicted sub-blocks of video data in the current video data block.

[0132] In some instances, one or more of the syntax elements described above (i.e., affine parameter (four-parameter or six-parameter) flags and / or enable syntax elements) can be decoded using the CABAC context model depending on the affine parameter usage of adjacent blocks. In one instance, the current affine parameter context index CtxVal depends on the left and top adjacent blocks. If the left adjacent block is unavailable, not in affine mode, or in six-parameter affine mode, then leftCtx is set to 0; otherwise (if the left adjacent block is available and in six-parameter affine mode), leftCtx is set to 1. A similar calculation can be performed for the top adjacent block to obtain aboveCtx. Then, the CtxVal of the current block is set to equal leftCtx + aboveCtx. In this case, CtxVal is in the range [0,2] (0 and 2 are inclusive). Other variations of setting leftCtx (aboveCtx) are also possible. For example, if the left (top) adjacent block is unavailable or not affine decoded, then leftCtx(aboveCtx) is set to 0; if the left (top) adjacent block is using a four-parameter affine, then it is set to 1; if the left (top) adjacent block is using a six-parameter affine, then it is set to 2. In this case, CtxVal is in the range [0,4] (0 and 4 are inclusive).

[0133] In some instances, one or more of the syntax elements described above (i.e., affine parameter (four-parameter or six-parameter) flags and / or enable syntax elements) may be used to decode using the CABAC context model depending on the current block size, and a block size threshold may be used to distinguish different contexts. For example, context 0 is used for block sizes equal to or less than 16×16; while context 1 is used for block sizes greater than 16×16. The threshold may be predefined or represented as a signal in the bitstream. The block size may be specified individually or jointly by the width and height of the current block. For example, the size may be represented by a width × height value.

[0134] In some instances, one or more of the syntax elements described above (i.e., affine parameter (four-parameter or six-parameter) flags and / or enable syntax elements) can also be decoded using CABAC bypass mode without any context.

[0135] Figure 8A conceptual diagram illustrating a six-parameter affine motion model according to one or more techniques of the present invention is provided. A four-parameter affine model may contain two motion vectors, and a six-parameter affine model may contain three motion vectors. In some instances, such as when using a six-parameter affine motion model, a video decoder may decode three motion vector differences (MVDs) in the bitstream for inter-frame patterns. The three motion vector predictors may be generated from or derived from adjacent motion vectors. Adjacent motion vectors may or may not be affine motion vectors. For example, three motion vectors v0 (MV0), v1 (MV1), and v2 (MV2) in the three corners of the current block 800 may be as follows: Figure 8 The motion vectors shown are to be decoded. For predicting v0, motion vectors 802A (top left), 802B (top), and 802C (left) are possible candidates. Similarly, motion vectors 802D (top) and 802E (top right) are possible candidates for predicting v1, and motion vectors 802F (left) and 802G (bottom left) are possible candidates for predicting v2. In some instances, the first available candidate at each position in a predefined checking order is directly used as its predictor.

[0136] The three motion vector predictors can be selected from a list of combinations using verification, sorting, and deduplication schemes, with only the first K combinations used as possible predictors, where K>=1. In some instances, the video decoder can generate a complete combination of all predictors using adjacent available motion vectors. For example... Figure 8 As shown, there are a total of 3×2×2=12 combinations.

[0137] In the first step, for each combination, the video decoder can perform a verification check. If MV0 equals MV1 and MV0 equals MV2, then the combination is invalid; otherwise, the combination is valid. In the second step, the video decoder can perform sorting based on parameter similarity. For example, if the current block uses the following six-parameter affine mode (where a, b, c, d, e, and f are model parameters), then the affine motion model can be represented by equation (3) reproduced below.

[0138]

[0139] Using a six-parameter affine model, the three angular motion vectors can be represented as follows:

[0140]

[0141] Figure 9 A conceptual diagram illustrating the affine motion vector evaluation according to one or more techniques of the present invention is provided. To evaluate the correctness of the model, the present invention introduces a parameter called the estimation difference (ED). Meanwhile, in... Figure 9The two adjacent blocks MV highlighted in the adjacent blocks 902H and 902I, located at half the width and half the height respectively, can be used in the evaluation process. Therefore, there exists:

[0142]

[0143] Among all combinations, the top K smallest ED combinations can be selected as the final predictors. The following is an example of ED calculation:

[0144]

[0145] The video decoder can set ED to be equal to the sum of the four elements mentioned above.

[0146] ED=Δa+Δb+Δd+Δe (7)

[0147] In some instances, the video decoder can perform sorting based on affine motion vector similarity. In one instance, given three motion vectors, the video decoder can predict a fourth motion vector using a six-parameter affine model. The prediction difference can be added to the ED, and the top few combinations with the smallest ED can be selected as MV prediction candidates.

[0148] A four-parameter affine model can be used to generate motion vector predictors across other predictors. For example, given the first two reconstructed MVs, the video decoder can use a four-parameter affine model to generate a third MV predictor. For example, the MV predictor for MV2 can be derived based on MV0 and MV1 of the current block using the above equation (2).

[0149] In some instances, affine motion vector predictors can be generated from previously decoded affine motion vectors within the current frame. In one instance, a set of N (N>=0) affine motion vectors can be initialized at the beginning of each frame, and after decoding each affine block, the list is updated with the most recently decoded affine motion vectors, with the index represented by a signal to indicate the selected affine motion predictor in the list. The video decoder can use truncated unary codes or flag-plus-truncated unary codes to decode the index.

[0150] In some instances, a set of K (K>=0) affine model parameters is initialized at the beginning of each frame. After each affine block is decoded, the set of parameters is updated with the decoded affine model parameters. For example, in a six-parameter model, the video decoder can maintain a list of N vectors, where each vector is represented by a {a...} with six elements. i b i c i d i e i f i} represents this. Similarly, in four-parameter mode, the video decoder can maintain M vectors {a}.j b j c j d j A list of}. Note that M and N may or may not be the same.

[0151] In the techniques mentioned above, for affine inter-frame modes, the video decoder can individually derive motion vector predictors for each MV of the affine model using the MVs of its neighboring locations. According to one or more techniques of the present invention, when affine motion is used by neighboring blocks, the video decoder can extrapolate the affine motion of neighboring blocks using predictors of the affine motion models of all MVs of the current block, i.e., the predictors of MV0 and MV1 (and MV2 for the six-parameter model) of the current affine model, from the affine motion of the neighboring blocks, and then decode the MVD.

[0152] The different prediction methods mentioned above can be used in combination. For example, a flag or index can be used as a signal to indicate which MV prediction method is used. In some instances, predictors derived using the different prediction methods mentioned above are used to generate a list of MV predictor candidates, and a flag or index is used to indicate which candidate is used to predict the current affine motion model.

[0153] When using a four-parameter affine motion model, "MV0 and MV1" or "MV0 and MV2" (e.g., ...) Figure 8 The v0 and v1 or v0 and v2 shown can be used to represent the affine motion of the current CU / PU. When the width and height of the current CU / PU are different, a certain type of rule can be used to determine which pair of motion vectors to use.

[0154] In one instance, the pair MV0 and MV1 can be used when the width is greater than or equal to (or only greater than) the height, or when the width-to-height ratio is greater than a threshold; otherwise, the pair MV0 and MV2 can be used. The threshold can be block size dependent or width / height dependent.

[0155] The technology can be applied to both affine merge mode and affine inter-frame mode, or only to one of them, such as affine merge mode.

[0156] Video decoders can use a specific checking / evaluation order to select adjacent blocks (e.g., in merge mode). In some instances, a video decoder can use the following order to check adjacent blocks for affine merge mode: top -> left -> top left -> top right -> bottom left. This order corresponds to sequences such as D -> F -> A -> E -> G. Figure 9 The video decoder can apply checks in a predefined order until all five candidates have been checked when adjacent blocks are unavailable or are not affine decoded blocks.

[0157] In some instances, if no adjacent affine motion blocks are available, the video decoder may insert certain default, predefined, or pre-computed affine motion models as candidates for the merging pattern. The inserted model can be initialized to the image level and can be updated during runtime.

[0158] In some instances, if no valid adjacent affine model exists, the video decoder may perform the insertion of a default or predefined or pre-computed affine motion model after checking adjacent blocks in the order of "top -> left -> top left -> top right -> bottom left".

[0159] In some instances, a video decoder can decode an affine merge index to indicate which adjacent affine models are duplicated for the current block and truncated unary, or unary, or exponential Columbus, or Columbus family codewords, or concatenations of these are available for decoding the index.

[0160] Switchable four-parameter and six-parameter affine models are derived / inferred from other information. In some instances, the video decoder can derive affine parameters from inter-frame prediction direction information. For each block, if it is decoded using an inter-frame mode, the prediction reference frame index can come from refList0, or from refList1, or both refList0 and refList1. According to one or more techniques of the invention, when using unidirectional prediction (predicting from refList0 or from refList1), the video decoder can use a six-parameter affine model where three motion vector differences are decoded in the bitstream. When using bidirectional prediction (predicting from both refList0 and refList1), the video decoder can use a four-parameter affine model where two motion vector differences are decoded in the bitstream. In some of these instances, the video decoder may omit the explicit indication of whether to use a four-parameter or six-parameter affine model to identify the syntax elements of one or more predicted sub-blocks of video data in the current video data block.

[0161] According to one or more techniques of the present invention, for a bidirectional prediction block, when L1ZeroMVDFlag is enabled, the video decoder can implement a six-parameter affine model for refList1, even though the emitted MVD is not present. In this condition, the video decoder can generate motion-compensated predictors via a six-parameter affine model established through three motion vector predictors.

[0162] In some instances, affine parameters can be derived from neighboring blocks. If a majority of neighboring blocks use a four-parameter affine pattern, then the current block also uses a four-parameter affine model. Similarly, when a majority of neighboring blocks use a six-parameter affine model (the number of six-parameter affines is greater than the number of four-parameter affines), the current block also uses a six-parameter affine model. A counter can be used to count the number of neighboring blocks of a given cell size (for a 4×4 block) in determining the majority of neighboring affine usage. When no neighboring affine model exists, the six-parameter affine model is used as the default mode (alternately, the four-parameter affine model is used as the default). When the number of four-parameter affine models equals the number of six-parameter models, the six-parameter affine model is used as the default (alternately, the four-parameter affine model is used as the default).

[0163] The cross-frame determination of affine model flags and motion vectors. According to one or more techniques of the invention, the video decoder can use cross-frame affine motion model parameters instead of explicitly representing affine parameter flags (four- or six-parameter modes) or affine motion vector information with signals. In one instance, the current block inherits the affine parameter model flags from a co-located block. The co-located block comes from a previously decoded image at the same location but at the same time level. The co-located block and the current block may or may not have the same partition size. According to one or more techniques of the invention, the video decoder can examine all sub-blocks (in a 4×4 cell) in the co-located region, and most of the affine model is used for the current block. If no affine model exists in the co-located region, the video decoder can explicitly decode the four- or six-parameter switching flags. In some instances, a 6 (or 4) parameter affine is used as the default. In some instances, to reduce complexity, the first affine sub-block in the co-located region in raster scan order is examined and inherited by the current block.

[0164] In another instance, the current block directly inherits the affine motion model parameters {a, b, c, d, e, f} or {a, b, c, d} from the co-located block. The co-located blocks originate from the same location but are in previously decoded images with the same time level. The co-located block and the current block may or may not have the same partition size. According to one or more techniques of the invention, the video decoder can examine all sub-blocks (in a 4×4 cell) within the co-located region, and the current block inherits the motion model parameters of most of the affine region. If no affine pattern exists in the co-located region, the video decoder can explicitly decode a four- or six-parameter switching flag. In some instances, a six (or four)-parameter affine is used as the default. In some instances, to reduce complexity, the first affine sub-block in the co-located region in raster scan order is examined and inherited by the current block. In some instances, combinations of the above instances can be used together. The video decoder can decode flags to indicate whether the inheritance is used or at a different level (e.g., PU, ​​CU level, PPS, or SPS).

[0165] Affine motion compensation given affine parameter information. During reconstruction, given three motion vectors (e.g., angular motion vectors in the current block), a six-parameter affine model can be established by solving equation (4). Given the six-parameter model, the motion vector per pixel can be calculated by substituting the pixel position (x, y) into equation (3). To reduce the complexity of motion compensation, a motion vector can be used for each sub-block K×K, where K is an integer equal to or greater than 1. A representative motion vector can be calculated using the top-left pixel position within the K×K sub-block, or using the center position of the K×K sub-block. The size K can be explicitly represented by a signal, set as a default value, or calculated during operation based on whether the pixel group shares the same motion vector.

[0166] Affine motion vector decoding. Predictors from adjacent valid (verified according to the affine model) and decopied motion vectors can be used to identify / predict the current affine motion vector. Predictors from the most recently previously decopied decoded affine motion vector can be maintained to identify / predict the current affine motion vector. The number of predictors can be K, where K is an integer equal to or greater than 1. These predictors form an affine predictor list. K can be predefined or represented by signals in the bitstream.

[0167] In some instances, a combination of both of the techniques described above can be used to maintain a list of predictors. For example, a video decoder can use predictors from adjacent valid (verified according to the affine model) and decopied motion vectors, as well as predictors from the most recently previously decopied decoded affine motion vector, to identify / predict the current affine motion vector.

[0168] Video decoders can explicitly represent predictor indices with signals in the bitstream to indicate predictor usage. Three MVDs can be decoded in a six-parameter model, while two MVDs can be decoded in a four-parameter model.

[0169] MVD can use different binarization methods from traditional MVD decoding. In one instance, affine MVD is decoded using separate context modeling. In another instance, affine MVD decoding shares the same MVD decoding context modeling as traditional inter-frame MVD decoding (i.e., as in HEVC).

[0170] MVD can use a four-parameter or six-parameter affine model based on the relative positions within the block, employing different binarization methods for each MVD. In one instance, affine MVD can use a four-parameter or six-parameter affine model to decode using different contextualizations based on the relative positions within the block.

[0171] A flag can be signaled to indicate whether the MVD in both directions (X and Y) is zero for one or both of the affine motion vectors to further improve motion vector decoding. If the flag (AllZeroFlag) is 1, then a novel MVD decoding is introduced to jointly decode MVD_x and MVD_y. Specifically, if AllZeroFlag is 1, then both MVD_x and MVD_y are inferred to be zero; otherwise, if MVD_x is zero, then MVD_y must be non-zero. In this case, abs(MVD_y)-1 is decoded. In other words, for each motion vector, if AllZeroFlag is zero, then the flag AllZeroFlag is signaled before the two MVD decodings. For a four-parameter affine, two AllZeroFlags are decoded for each list; while for a six-parameter affine, three AllZeroFlags are decoded for each list.

[0172] In some instances, AllZeroFlag can be extended to represent all zero MVDs in both reference lists in a double prediction. For example, in a four-parameter affine, a total of two AllZeroFlags are decoded for both reference lists; in a six-parameter affine, a total of three AllZeroFlags are decoded for both reference lists.

[0173] Figure 10 This section illustrates an example of Overlapping Block Motion Compensation (OBMC). During the development of H.263, it was proposed to perform OBMC on 8×8 blocks, with the motion vectors of two adjacent, already connected 8×8 blocks used for the current block. For example, for the first 8×8 block in the current macroblock, in addition to the motion vector of the first 8×8 block, the upper and left adjacent motion vectors of the first 8×8 block are also applied to generate two additional prediction blocks. Similarly, for the second 8×8 block in the current macroblock, in addition to the motion vector of the second 8×8 block, the upper and right adjacent motion vectors of the second 8×8 block are also applied to generate two additional prediction blocks. For example, in… Figure 10 In the example, the motion vectors of blocks 1004A and 1004B can be used to generate an additional prediction block of 8×8 block 1002A for the 16×16 macroblock 1000, and the motion vectors of blocks 1006A and 1006B can be used to generate an additional prediction block of 8×8 block 1002B for the macroblock 1000. In this way, each pixel in the current 8×8 block can have three prediction blocks, and the weighted average of these three predictions can be used as the final prediction block.

[0174] When neighboring blocks are undecoded or decoded to intra-frame (i.e., neighboring blocks do not have usable motion vectors), the motion vector of the current 8×8 block is used as the neighboring motion vector. Simultaneously, for the third and fourth 8×8 blocks of the current macroblock (e.g., ...Figure 10 As shown in the image, the adjacent lower block is never used. For example, as... Figure 10 As shown in the example, the motion vector of block 1008B is not used to generate an additional prediction block for the 8×8 block 1002C because block 1008B is considered undecoded, and the motion vector of block 1010B is not used to generate an additional prediction block for the 8×8 block 1002D because block 1010B is considered undecoded. In other words, for each macroblock, no motion information from the macroblocks below it will be used during OBMC to reconstruct the pixels of the current macroblock.

[0175] Figure 11A and 11B This is a conceptual diagram illustrating the OBMC in HEVC. In HEVC, the OBMC was also proposed in U.S. Patent Application Publication No. 2013 / 0128974A1 and U.S. Patent Application Publication No. 2012 / 0177120A1 to smooth the PU boundary. Figure 11A and 11B Provide an example of the proposed method. Figure 11A and 11B In the diagram, the white areas are all first PU1102 (PU0), and the shaded areas are all second PU1104 (PU1). When a CU contains two (or more) PUs, lines / rows near the PU boundaries are smoothed using OBMC. For pixels marked "A" or "B" in PU0 1102 or PU1 1104, for example, two predicted values ​​are generated by applying the motion vectors of PU0 and PU1 respectively, and their weighted average is used as the final prediction.

[0176] Figure 12A and 12B This is a conceptual diagram illustrating the sub-blocks to which OBMC can be applied. In the Joint Exploration Model (JEM) reference software (available at https: / / jvet.hhi.fraunhofer.de / ), sub-PU-level OBMC is applied. OBMC is performed on all motion compensation (MC) block boundaries except for the right and bottom boundaries of the CU. Furthermore, it is applied to both the lightness and chroma components. In HEVC, MC blocks correspond to PUs. In JEM, each sub-block of a PU is an MC block when the PU is decoded via a sub-PU mode. To process CU / PU boundaries uniformly, OBMC is performed at the sub-block level for all MC block boundaries, where the sub-block size is set to equal to 4×4, as shown below. Figure 12A and 12B As explained in the text.

[0177] When OBMC is applied to the current sub-block, in addition to the current motion vector, the motion vectors of four adjacent sub-blocks, if available and different from the current motion vector, are also used to derive the prediction block for the current sub-block. These multiple prediction blocks based on multiple motion vectors are weighted to produce the final prediction signal for the current sub-block.

[0178] A predicted block based on the motion vectors of neighboring sub-blocks can be represented as P. N , where N indicates the indices of the adjacent upper, lower, left, and right sub-blocks. The predicted block based on the motion vector of the current block can be represented as P. C When P N Belongs to P C When the PU is the same (and therefore contains the same motion information), it does not originate from the P. N Execute OBMC. Otherwise, P N Each pixel is added to P C The same pixel in P N Four rows / columns were added to P C Weighting factors {1 / 4, 1 / 8, 1 / 16, 1 / 32} are used for P. N And the weighting factors {3 / 4, 7 / 8, 15 / 16, 31 / 32} are used for P C An exception is small MC blocks (i.e., when the PU size is equal to 8×4, 4×8, or when the PU uses ATMVP mode decoding), for which P N Only two columns / rows were added to P C In this situation, the weighting factors {1 / 4, 1 / 8} can be used for P. N And the weighting factors {3 / 4, 7 / 8} are used for P C For P generated based on the motion vectors of vertically (horizontally) adjacent sub-blocks N P N Pixels in the same column (row) are added to P along with the same weighting factor. C It should be noted that for PU boundaries, OBMC can be applied to each side of the boundary. For example, in Figure 12A and 12B In this approach, OBMC can be applied twice along the boundary between PU1 and PU2. First, OBMC, along with the MV of PU2, is applied to the shaded block along the inner boundary of PU1. Second, OBMC, along with the MV of PU1, is applied to the shaded block along the inner boundary of PU2. In contrast, OBMC can be applied only to one side of the CU boundary because we cannot change the already decoded CU when decoding the current CU.

[0179] Figure 13This is a flowchart illustrating an example method for performing affine motion compensation via a video encoder (e.g., during a video encoding process) according to one or more techniques of the present invention. For illustrative and explanatory purposes, regarding... Figure 1 and 2 Video encoder 20 description Figure 13 The method.

[0180] The video encoder 20 can receive the current video data block (1302) to be encoded. For example, the video encoder 20 can receive the raw pixel values ​​(e.g., RGB, CMYK, YUV, etc.) of the current image containing the video data of the current video data block from the video source 18. The segmentation unit 48 of the mode selection unit 40 of the video encoder 20 can divide the current image upwards into multiple blocks, one of which can be the current block.

[0181] Video encoder 20 can determine to encode the current video data block (1304) using affine motion prediction. For example, mode selection unit 40 can determine to encode the current video data block using an inter-frame prediction mode and select an affine motion model as the motion information prediction mode. Mode selection unit 40 can determine to use an inter-frame prediction mode based on multiple factors, such as the frame type of the current image (e.g., P-frame, I-frame, B-frame, etc.) and which prediction mode produces the lowest bit rate-distortion optimization (RDO) cost.

[0182] The video encoder 20 can encode an indication (1306) that the current block is encoded using affine motion prediction. For example, the mode selection unit 40 can cause the entropy coding unit 56 of the video encoder 20 to encode in the video bitstream one or more syntax elements indicating that the current block is encoded using an inter-frame prediction mode, one or more syntax elements indicating that the affine motion model is the motion information prediction mode of the current block, and / or one or more syntax elements indicating that the current block is encoded using an inter-frame prediction mode and that the affine motion model is the motion information prediction mode of the current block.

[0183] The video encoder 20 can determine the values ​​of the motion vectors (1308) of the affine motion model for the current video data block. For example, the motion estimation unit 42 and / or motion compensation unit 44 of the video encoder 20 can identify predicted sub-blocks of video data with pixel values ​​that closely match the pixel values ​​of the current video data block. The motion estimation unit 42 and / or motion compensation unit 44 can determine two or more motion vectors representing the affine transformation between the current video data block and the predicted sub-blocks of the video data.

[0184] As discussed above, in some instances, motion estimation unit 42 and / or motion compensation unit 44 may always use a four-parameter affine motion model containing two motion vectors to identify predicted sub-blocks. Similarly, in some instances, motion estimation unit 42 and / or motion compensation unit 44 may always use a six-parameter affine motion model containing three motion vectors to identify predicted sub-blocks. In other instances, motion estimation unit 42 and / or motion compensation unit 44 may selectively use a model containing two motion vectors (e.g., Figure 8 A four-parameter affine motion model containing three motion vectors (v0 and v1, also called MV0 and MV1). Figure 8 A six-parameter affine motion model of v0, v1, and v2 (also known as MV0, MV1, and MV2) is used to identify and predict sub-blocks.

[0185] In some instances, the video encoder 20 may encode an indication of whether the current block is decoded using a four-parameter model or a six-parameter model. For example, the motion estimation unit 42 and / or the motion compensation unit 44 may cause the entropy coding unit 56 to encode syntax elements in the encoded video bitstream indicating whether the affine motion model of the current video data block includes a four-parameter model or a six-parameter model. In some instances, the entropy coding unit 56 may encode syntax elements in one or more of the video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), or slice header referenced by the current video data block. In some instances, the entropy coding unit 56 may encode syntax elements at the decoding unit (CU) level of the CU containing the current video data block.

[0186] The video encoder 20 can select adjacent video data blocks (1310) with an affine motion model for the current video data block. For example, when encoding... Figure 8 When the current block is 800, the motion estimation unit 42 and / or the motion compensation unit 44 can evaluate in a specific order. Figure 8 The blocks 802A to 802G are selected, and the first block (e.g., the first block with an available affine motion model) is selected as the selected adjacent video data block in the specified order using affine motion compensation decoding. In some instances, the current video data block can be decoded using an affine inter-frame mode. In some instances, the selected adjacent video data block can be decoded using an affine inter-frame mode (e.g., AF_INTER) or an affine merging mode (e.g., AF_MERGE).

[0187] The video encoder 20 can obtain the predictor value (1312) of the motion vector of the affine motion model of the selected adjacent video data block. For example, the motion estimation unit 42 and / or the motion compensation unit 44 can obtain the value of the affine motion model of the selected adjacent video data block from the memory or storage device (e.g., reference image memory 64) of the video encoder 20. The motion estimation unit 42 and / or the motion compensation unit 44 can transform the value of the affine motion model of the selected adjacent video data block to the position of the current block to derive the predictor value. In other words, the motion estimation unit 42 and / or the motion compensation unit 44 can extrapolate the predictor value from the value of the affine motion model of the selected adjacent video data block. As an example, when the selected adjacent block is Figure 8 In the case of block 802F, video encoder 20 can obtain the values ​​of multiple motion vectors of block 802F (e.g., the CPMV value of block 802F) and transform the values ​​of the multiple motion vectors of block 802F to the position of the current block 800. As another example, when the selected adjacent block is Figure 8 In the case of block 802F, video encoder 20 can use the values ​​of the plurality of motion vectors of block 802F (e.g., the CPMV value of block 802F) as predictors.

[0188] The video encoder 20 can encode a representation (1314) in the encoded video bitstream of the difference between the values ​​of the motion vectors and the values ​​of the predictors of the affine motion model of the current video data block. For example, the motion estimation unit 42 and / or the motion compensation unit 44 can determine a corresponding motion vector difference (MVD) value for each corresponding motion vector of the affine motion model of the current block, representing the difference between the value of the corresponding motion vector of the affine motion model of the current block and the value of the corresponding predictor derived from the motion vector of the affine motion model of the selected neighboring block. As an example, if the values ​​of the motion vectors of the affine motion model of the current block are MV0 and MV1 and the values ​​of the predictors derived from the motion vectors of the affine motion model of the selected neighboring block are MVP0 and MVP1, the motion estimation unit 42 and / or the motion compensation unit 44 can determine a first MVD value as the difference between MV0 and MVP0, and a second MVD value as the difference between MV1 and MVP1. The motion estimation unit 42 and / or the motion compensation unit 44 can cause the entropy coding unit 56 to encode one or more syntax elements representing the determined MVD values ​​in the encoded video bitstream.

[0189] In some instances, the video encoder 20 may further encode residual data in the encoded video bitstream representing the pixel difference between the current block and a predicted sub-block identified by the affine motion model of the current block. The video encoder 20 may implement a decoder loop to reconstruct the pixel values ​​of the current block (e.g., for use in predicting future blocks). For example, the video encoder 20 may identify a predicted sub-block based on the affine motion model of the current block, obtain the pixel values ​​of the predicted sub-block from the reference image memory 64, and add residual values ​​to the pixel values ​​of the predicted sub-block to reconstruct the pixel values ​​of the current block.

[0190] Figure 14 This is a flowchart illustrating an example method for performing affine motion compensation via a video decoder (e.g., during the video decoding process) according to one or more techniques of the present invention. For illustrative and explanatory purposes, regarding... Figure 1 and 3 Video decoder 30 description Figure 14 The method.

[0191] The video decoder 30 can decode an indication (1402) that the current block is encoded using affine motion prediction. For example, the entropy decoding unit 70 can decode from the video bitstream one or more syntax elements indicating that the current block is encoded using inter-frame prediction mode, one or more syntax elements indicating that the affine motion model is the motion information prediction mode for the current block, and / or one or more syntax elements indicating that the current block is encoded using inter-frame prediction mode and that the affine motion model is the motion information prediction mode for the current block. The entropy decoding unit 70 can provide the value of the decoded syntax element to the motion compensation unit 72.

[0192] The video decoder 30 can select adjacent video data blocks (1404) with an affine motion model for the current video data block. For example, when decoding... Figure 8 When the current block is 800, the motion compensation unit 72 can evaluate in a specific order. Figure 8 The blocks 802A to 802G are selected, and the first block (e.g., the first block with an available affine motion model) is selected as the selected adjacent video data block in the specified order using affine motion compensation decoding. In some instances, the current video data block can be decoded using an affine inter-frame mode. In some instances, the selected adjacent video data block can be decoded using an affine inter-frame mode (e.g., AF_INTER) or an affine merging mode (e.g., AF_MERGE).

[0193] The video decoder 30 can obtain the value of the predictor (1406) derived from the motion vectors of the affine motion model of the selected adjacent video data blocks. For example, the motion compensation unit 72 can obtain the value of the affine motion model of the selected adjacent video data blocks from the memory or storage device (e.g., reference image memory 82) of the video decoder 30. The motion compensation unit 72 can transform the value of the affine motion model of the selected adjacent video data blocks to the position of the current block to derive the value of the predictor. In other words, the motion compensation unit 72 can extrapolate the value of the predictor from the value of the affine motion model of the selected adjacent video data blocks. As an example, when the selected adjacent block is... Figure 8 In the case of block 802F, the video decoder 30 can obtain the values ​​of multiple motion vectors of block 802F (e.g., the CPMV value of block 802F) and transform the values ​​of the multiple motion vectors of block 802F to the position of the current block 800. As another example, when the selected adjacent block is Figure 8 In the case of block 802F, video decoder 30 can use the values ​​of the plurality of motion vectors of block 802F (e.g., the CPMV value of block 802F) as predictors.

[0194] The video decoder 30 can decode from the encoded video bitstream a representation of the difference between the motion vector values ​​and predictor values ​​of the affine motion model of the current video data block (1408). For example, the entropy decoding unit 70 can decode from the encoded video bitstream a syntax element representing the difference between the corresponding motion vector values ​​of the affine motion model of the current block and the corresponding predictor values ​​derived from the motion vectors of the affine motion models of the selected neighboring blocks. As an example, if the motion vector values ​​of the affine motion model of the current block are MV0 and MV1, and the predictor values ​​derived from the motion vectors of the affine motion models of the selected neighboring blocks are MVP0 and MVP1, the entropy decoding unit 70 can decode a syntax element representing a first MVD value and a second MVD value, the first MVD value being the difference between MV0 and MVP0, and the second MVD value being the difference between MV1 and MVP1. The entropy decoding unit 70 can provide the values ​​of the decoded syntax elements to the motion compensation unit 72.

[0195] The video decoder 30 can determine the motion vector values ​​of the affine motion model for the current video data block based on the values ​​of the predictors and the decoded difference (1410). For example, the motion compensation unit 72 can add the value of MVP0 to the value of the first MVD value to determine the value of MV0 and add the value of MVP1 to the value of the second MVD value to determine the value of MV1.

[0196] The video decoder 30 can determine the predicted sub-blocks (1412) of the video data based on the determined values ​​of the motion vectors of the affine motion model of the current video data block. For example, the motion compensation unit 72 can obtain the pixel values ​​of the predicted sub-blocks identified by the affine motion model of the current video data block from the reference image memory 82.

[0197] The video decoder 30 can reconstruct the current video data block (1414) based on predicted sub-blocks of video data. For example, the entropy decoding unit 70 can decode residual data representing the pixel difference between the current block and the predicted sub-block identified by the affine motion model of the current block from the encoded video bitstream. The motion compensation unit 72 can add the residual values ​​to the pixel values ​​of the predicted sub-block to reconstruct the pixel values ​​of the current block.

[0198] It should be recognized that, depending on the instance, certain actions or events of any of the techniques described herein may be performed in a different sequence, may be added, combined, or may be omitted entirely (e.g., not all described actions or events are necessary for the practice of the technique). Furthermore, in some instances, actions or events may be performed simultaneously rather than sequentially, for example, via multithreading, interrupt handling, or multiple processors.

[0199] In one or more instances, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored on or transmitted via a computer-readable medium as one or more instructions or code, and executed by a hardware-based processing unit. The computer-readable medium may comprise a computer-readable storage medium (which corresponds to a tangible medium such as a data storage medium) or a communication medium, which includes, for example, any medium that facilitates the transfer of a computer program from one place to another according to a communication protocol. In this way, the computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium, such as a signal or carrier wave. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described herein. The computer program product may comprise a computer-readable medium.

[0200] By way of example, and not limitation, these computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, flash memory, or any other media that can be used to store desired program code in the form of instructions or data structures and that is accessible by a computer. Furthermore, any connection is properly referred to as computer-readable media. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, optical fiber, twisted pair, digital subscriber line (DSL), or wireless technology (such as infrared, radio, and microwave), then coaxial cable, optical fiber, twisted pair, DSL, or wireless technology (such as infrared, radio, and microwave) is included in the definition of media. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are in fact related to non-transient tangible storage media. As used herein, disks and optical discs include compact optical discs (CDs), laser discs, optical discs, digital audio-visual discs (DVDs), floppy disks, and Blu-ray discs, wherein disks typically reproduce data magnetically, while optical discs use lasers to reproduce data optically. The combination of the above should also be included in the scope of computer-readable media.

[0201] The instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, as used herein, the term "processor" can refer to any of the above-described structures or any other structures suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Furthermore, the techniques can be fully implemented within one or more circuit or logic elements.

[0202] The technology of this invention can be implemented in various devices or apparatuses, including wireless handsets, integrated circuits (ICs), or IC sets (e.g., chipsets). Various components, modules, or units are described in this invention to emphasize functional aspects of a device configured to perform the disclosed technology, but implementation by different hardware units is not necessarily required. Specifically, as described above, various units can be combined in a codec hardware unit, or provided through a collection of interoperable hardware units (comprising one or more processors as described above) combined with suitable software and / or firmware.

[0203] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A method for decoding video data, the method comprising: Using one or more processors of a video decoder and based on inter-frame prediction direction information of the current block of video data for the current image of video data, it is determined whether motion compensation for the current block of video data in the current image of video data should be performed using a four-parameter affine motion model defined by two motion vectors or a six-parameter affine motion model defined by three motion vectors, wherein the determination is performed without receiving explicit signaling regarding whether the motion compensation for the current block of video data should be performed using the four-parameter affine motion model or the six-parameter affine motion model, wherein determining whether motion compensation for the current block of video data in the current image of video data should be performed using a four-parameter affine motion model defined by two motion vectors or a six-parameter affine motion model defined by three motion vectors based on the inter-frame prediction direction information of the current block includes: When the inter-frame prediction direction information of the current block indicates unidirectional prediction, three motion vector differences are selected for the current block from the six-parameter affine motion model decoded in the bitstream; and When the inter-frame prediction direction information of the current block indicates bidirectional prediction, the difference between two motion vectors for the current block is selected as the four-parameter affine motion model decoded in the bit stream. Using the one or more processors, based on the values ​​of the motion vectors of the affine motion model of the previously decoded video data block in the current image, the values ​​of the predictors of the motion vectors of the selected affine motion model for the current block of video data are derived. The representation of the difference between the values ​​of the motion vectors of the affine motion model of the current block of video data and the values ​​of the predictors is decoded by the one or more processors and from the encoded video bitstream; The value of the motion vector of the affine motion model for the current block of video data is determined by the one or more processors from the value of the predictor and the decoded difference. Based on the determined values ​​of the motion vectors of the affine motion model for the current block of the video data, predictable sub-blocks of the video data are determined; and Based on the predicted sub-blocks of video data, the current block of video data is reconstructed.

2. The method according to claim 1, wherein, The current block of video data is decoded using an affine inter-frame mode.

3. The method according to claim 2, wherein, Adjacent video data blocks are decoded using either affine inter-frame mode or affine merge mode.

4. The method of claim 1, further comprising: For the current image, maintain the list of motion vectors from the affine motion model of the previously decoded video data block. The one or more processors decode, from the encoded video bitstream, instructions indicating which motion vectors in the list should be used as syntax elements of the predictors of the motion vectors for the current block of the video data in the affine motion model.

5. The method according to claim 1, further comprising: In response to determining that motion compensation for the current block of video data is to be performed using a four-parameter affine motion model defined by two motion vectors, the one or more processors determine whether to use the upper left and upper right motion vectors to represent the affine motion model of the current block, or to use the upper left and lower left motion vectors to represent the affine motion model of the current block.

6. The method according to claim 5, wherein, Determining whether to use the upper left and upper right motion vectors to represent the affine motion model of the current block, or to use the upper left and lower left motion vectors to represent the affine motion model of the current block, includes: Based on the ratio of the width to the height of the current block, it is determined whether to use the upper left and upper right motion vectors to represent the affine motion model of the current block, or to use the upper left and lower left motion vectors to represent the affine motion model of the current block.

7. The method according to claim 6, wherein, Determining whether to use the upper left motion vector and the upper right motion vector to represent the affine motion model of the current block, or to use the upper left motion vector and the lower left motion vector to represent the affine motion model of the current block, includes: If the width of the current block is greater than the height of the current block, determine that the affine motion model of the current block is represented by the upper left motion vector and the upper right motion vector; and If the height of the current block is greater than the width of the current block, the affine motion model of the current block is determined to be represented by the upper left motion vector and the lower left motion vector.

8. The method of claim 1, further comprising: The values ​​of motion vectors of affine motion models for adjacent video data blocks are obtained through one or more processors and for the current block of video data. Specifically, deriving the value of the predictor of the motion vector of the affine motion model for the current block of video data based on the value of the motion vector of the previously decoded video data block in the current image includes: deriving the value of the predictor of the motion vector of the affine motion model for the current block of video data based on the value of the motion vector of the affine motion model of the adjacent video data blocks.

9. The method according to claim 8, wherein, Obtaining the values ​​of the motion vectors of the affine motion model for the selected adjacent video data blocks includes: Evaluate adjacent video data blocks of the current video data block in a predefined order; and The first adjacent video data block among multiple adjacent video data blocks using affine motion compensation decoding is selected as the selected adjacent video data block.

10. An apparatus for decoding video data blocks, the apparatus comprising: A memory configured to store the video data; as well as One or more processing units, implemented in circuitry and configured to: Based on whether neighboring blocks of the current block of video data are predicted using a four-parameter affine motion model defined by two motion vectors or a six-parameter affine motion model defined by three motion vectors, the system determines whether motion compensation for the current block of the current frame of video data should be performed using either a four-parameter affine motion model defined by two motion vectors or a six-parameter affine motion model defined by three motion vectors. This determination is performed without receiving explicit signaling regarding whether motion compensation for the current block of video data should be performed using the four-parameter affine motion model or the six-parameter affine motion model. To determine whether motion compensation for the current block of video data in the current frame of video data should be performed using either a four-parameter affine motion model defined by two motion vectors or a six-parameter affine motion model defined by three motion vectors, based on inter-frame prediction direction information for the current block, the one or more processing units are configured to: When the inter-frame prediction direction information of the current block indicates unidirectional prediction, three motion vector differences are selected for the current block from the six-parameter affine motion model decoded in the bitstream; and When the inter-frame prediction direction information of the current block indicates bidirectional prediction, the difference between two motion vectors of the current block is selected as the four-parameter affine motion model decoded in the bit stream. Based on the values ​​of the motion vectors of the affine motion model of the previously decoded video data block in the current image, derive the values ​​of the predictors of the motion vectors of the selected affine motion model for the current video data block. From the encoded video bitstream, decode a representation of the difference between the value of the motion vector of the affine motion model of the current block of video data and the value of the predictor; The value of the motion vector of the affine motion model for the current block of video data is determined from the value of the predictor and the difference after decoding. Based on the determined values ​​of the motion vectors of the affine motion model for the current block of the video data, predictable sub-blocks of the video data are determined; and Based on the predicted sub-blocks of video data, the current block of video data is reconstructed.

11. The apparatus according to claim 10, wherein, The current block of video data is decoded using an affine inter-frame mode.

12. The apparatus according to claim 11, wherein, Adjacent video data blocks are decoded using either affine inter-frame mode or affine merge mode.

13. The apparatus according to claim 10, wherein, The one or more processing units are further configured to: For the current image, maintain the list of motion vectors from the affine motion model of the previously decoded video data block. From the encoded video bitstream, the decoder indicates which motion vectors in the list should be used as syntax elements of the predictor of the motion vectors for the current block of the video data in the affine motion model.

14. The apparatus according to claim 10, wherein, The one or more processing units are further configured to: In response to determining that motion compensation for the current block of video data is to be performed using a four-parameter affine motion model defined by two motion vectors, it is determined whether the affine motion model of the current block is represented by the upper left and upper right motion vectors, or by the upper left and lower left motion vectors.

15. The apparatus according to claim 14, wherein, To determine whether to use the upper left motion vector and the upper right motion vector to represent the affine motion model of the current block, or to use the upper left motion vector and the lower left motion vector to represent the affine motion model of the current block, the one or more processing units are configured to: Based on the ratio of the width to the height of the current block, it is determined whether to use the upper left and upper right motion vectors to represent the affine motion model of the current block, or to use the upper left and lower left motion vectors to represent the affine motion model of the current block.

16. The apparatus according to claim 15, wherein, To determine whether to use the upper left motion vector and the upper right motion vector to represent the affine motion model of the current block, or to use the upper left motion vector and the lower left motion vector to represent the affine motion model of the current block, the one or more processing units are configured to: If the width of the current block is greater than the height of the current block, determine that the affine motion model of the current block is represented by the upper left motion vector and the upper right motion vector; and If the height of the current block is greater than the width of the current block, it is determined that the affine motion model of the current block is represented by the upper left motion vector and the lower left motion vector.

17. The apparatus of claim 10, wherein the one or more processing units are further configured to: For the current block of video data, obtain the values ​​of the motion vectors of the affine motion model of the adjacent video data blocks. in, In order to derive the value of the predictor of the motion vector of the affine motion model for the current block of video data based on the value of the motion vector of the previously decoded video data block in the current image, the one or more processing units are configured to: derive the value of the predictor of the motion vector of the affine motion model for the current block of video data based on the value of the motion vector of the affine motion model of the adjacent video data blocks.

18. The apparatus according to claim 17, wherein, In order to obtain the values ​​of the motion vectors of the affine motion model for the selected adjacent video data blocks, the one or more processing units are configured to: The adjacent video data blocks of the current video data block are evaluated in a predefined order; as well as The first adjacent video data block among multiple adjacent video data blocks using affine motion compensation decoding is selected as the selected adjacent video data block.

19. The apparatus of claim 10, further comprising at least one of the following: A display configured to show reconstructed video data; or A camera configured to capture the video data.

20. The apparatus according to claim 19, wherein, The device includes one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

21. A computer-readable storage medium storing instructions, which, when executed, cause one or more processors of a video encoder or video decoder to perform the following operations: Based on the inter-frame prediction direction information of the current block of video data in the current image of the video data, it is determined whether motion compensation for the current block of video data in the current image of the video data should be performed using a four-parameter affine motion model defined by two motion vectors or a six-parameter affine motion model defined by three motion vectors. The determination is performed without receiving explicit signaling regarding whether the motion compensation for the current block of video data should be performed using the four-parameter affine motion model or the six-parameter affine motion model. The instructions that cause the one or more processors to determine whether the motion compensation for the current block of video data in the current frame of the current image should be performed using a four-parameter affine motion model defined by two motion vectors or a six-parameter affine motion model defined by three motion vectors include instructions that cause the one or more processors to perform the following operations: When the inter-frame prediction direction information of the current block indicates unidirectional prediction, three motion vector differences are selected for the current block from the six-parameter affine motion model decoded in the bitstream; and When the inter-frame prediction direction information of the current block indicates bidirectional prediction, the difference between two motion vectors for the current block is selected as the four-parameter affine motion model decoded in the bit stream. Based on the values ​​of the motion vectors of the affine motion model of the previously decoded video data block in the current image, derive the values ​​of the predictors of the motion vectors of the selected affine motion model for the current video data block. From the encoded video bitstream, decode a representation of the difference between the value of the motion vector of the affine motion model of the current block of video data and the value of the predictor; The value of the motion vector of the affine motion model for the current block of video data is determined from the value of the predictor and the difference after decoding. Based on the determined values ​​of the motion vectors of the affine motion model for the current block of the video data, predictable sub-blocks of the video data are determined; and Based on the predicted sub-blocks of video data, the current block of video data is reconstructed.

Images