Video Encoding and Decoding

Abstract

To provide a coding method, decoding method, and device for improving the coding efficiency of the affine mode.SOLUTION: The decoding method includes transmitting the affine mode in an encoded video stream, specifically determining a merge candidate list corresponding to adjacent blocks A1, B1, B0, A0, and B2 of the current block, encoding a context-coded flag from the data stream, meaning transmitting the affine mode of the current block, and determining the context variable of the flag on the basis of whether the adjacent blocks use the affine mode.SELECTED DRAWING: Figure 18

Description

[Technical Field]

[0001] The present invention relates to video encoding and decoding. [Background technology]

[0002] Recently, the Joint Video Experts Team (JVET), a joint team formed by MPEG and ITU-T Study Group 16's VCEG, began work on a new video coding standard called Versatile Video Coding (VVC). The goal of VVC is to provide significant improvements in compression performance over the existing HEVC standard (i.e., typically double the previous standard) and to be completed in 2020. Primary target applications and services include, but are not limited to, 360-degree and high dynamic range (HDR) video. Overall, the JVET evaluated responses from 32 parties using formal subjective testing conducted by an independent testing laboratory. Some proposals demonstrated compression efficiency gains of typically 40% or more compared to using HEVC. Particular effectiveness was demonstrated on ultra-high-definition (UHD) video test material. Therefore, we can expect compression efficiency gains well in excess of the 50% target for the final standard.

[0003] The JVET Search Model (JEM) uses all of the HEVC tools. An additional tool not present in HEVC is the use of "affine motion mode" when applying motion compensation. While motion compensation in HEVC is limited to translation, in reality there are many types of motion, such as zooming in / out, rotation, perspective motion, and other irregular movements. When using affine motion mode, more complex transformations are applied to blocks to attempt to more accurately predict such types of motion. However, using affine motion mode can increase the complexity of the encoding / decoding process and can also increase signal overhead. Therefore, a solution to at least one of the aforementioned problems is desirable. Summary of the Invention

[0004] A first aspect of the present invention provides a method for signaling a motion prediction mode for a portion of a bitstream, the method comprising: determining an inter prediction mode to be used for the portion of the bitstream; and signaling an affine motion mode depending on the inter prediction mode to be used for the portion of the bitstream. Optionally, the inter prediction mode to be used is determined based on a state of a skip flag in the portion of the bitstream. Optionally, affine mode is not enabled if the skip flag is present. Optionally, the method further comprises enabling merge mode if the affine mode is enabled. Optionally, affine mode is enabled if the inter prediction mode is Advanced Motion Vector Predictor (AMVP). Optionally, the determination is performed based on a high-level syntax flag, the high-level syntax flag indicating at least one of slice-level, frame-level, sequence-level, and coding tree unit (CTU)-level processing. Optionally, determining the inter prediction mode includes determining a mode of one or more blocks neighboring the current block.

[0005] In a second aspect of the present invention, there is provided a method for signaling a motion prediction mode in a bitstream, the method comprising: determining the modes of one or more neighboring blocks to a current block; and signaling an affine motion mode for the current block depending on the modes. Optionally, the neighboring blocks consist of only blocks A1 and B1. Alternatively, the neighboring blocks include blocks A2 and B3, preferably only blocks A2 and B3. Optionally, the method comprises enabling affine motion mode if one or both of the neighboring blocks use affine motion mode. Optionally, the neighboring blocks further include B0, A0, and B2. Optionally, the use of affine mode in the neighboring blocks is determined successively, and if one of the neighboring blocks uses affine mode, affine mode is enabled for the current block. Preferably, the series of neighboring blocks is A2, B3, B0, A0, B2.

[0006] In a third aspect of the present invention, there is provided a method for signaling a motion prediction mode for a portion of a bitstream, the method comprising: determining a list of merge candidates corresponding to blocks neighboring a current block; and enabling affine mode for the current block if one or more of the merge candidates use affine mode. Optionally, the list begins with blocks used to determine context variables associated with the block. Optionally, the list begins with blocks A2 and B3 in that order. Optionally, the list is A2, B3, B0, or A0, or B2 in that order. Optionally, affine mode is enabled for the current block if the neighboring blocks do not use merge mode. Optionally, affine mode is enabled for the current block if the neighboring blocks do not use merge skip mode. Optionally, signaling the affine mode includes inserting a context coding flag into the datastream, the context variable of the flag being determined based on whether the neighboring blocks use affine mode.

[0007] In a further aspect of the present invention, there is provided a method for signaling a motion prediction mode of a coded block in a bitstream, the method comprising: determining whether blocks neighboring the coded block in the bitstream use affine mode; and inserting a context coding flag into the bitstream, wherein a context variable of the context coding flag depends on the determination of whether blocks neighboring the coded block in the bitstream use affine mode. Optionally, the neighboring blocks include blocks A1 and B1. Optionally, if the mode of the block for which the motion prediction mode is enabled is merge mode, the neighboring blocks include blocks A1 and B1. Optionally, the context of the affine flag is obtained according to the following formula: Ctx = IsAffine(A1) + IsAffine(B1), where Ctx is the context variable of the affine flag and IsAffine is a function that returns 0 if the block is not an affine block and 1 if the block is affine.

[0008] In a fourth aspect of the present invention, there is provided a method for signaling a motion prediction mode of a coded block in a bitstream depending on whether neighboring blocks use merge mode and / or merge skip mode. In a fifth aspect of the present invention, there is provided a method for signaling a motion prediction mode in a bitstream, the method comprising: compiling a list of motion predictor candidates; and inserting an affine merge mode as a merge candidate. Optionally, the affine merge mode candidate is after the neighboring block motion vector in the list of merge candidates. Optionally, the affine merge mode candidate is before an alternative temporal motion vector predictor (ATMVP) candidate in the list of merge candidates. Optionally, the position (merge index) of the affine merge mode candidate in the list of candidates is fixed. Optionally, the position of the affine merge mode candidate in the candidate list is variable. Optionally, the location of the affine merge mode candidate is determined based on one or more of: a) the state of the skip flag; b) motion information of neighboring blocks; c) alternative temporal motion vector predictor (ATMVP) candidates; and d) whether the neighboring blocks use affine mode.

[0009] Optionally, if one or more of the following conditions are met: a) a skip flag is present; b) the motion information of neighboring blocks is equal; c) the ATMVP candidate contains only one motion information; and d) one or more neighboring blocks use affine mode, the affine merge mode is placed lower in the list of candidates (assigned a higher merge index). Optionally, the neighboring blocks include blocks A1 and B1. Optionally, the affine merge mode is placed lower in the list of candidates (assigned a higher merge index) than spatial motion vector candidates if one or more of the above conditions a) to d) are met. Optionally, the affine merge mode is placed lower (assigned a higher merge index) than temporal motion vector candidates if one or more of the above conditions a) to d) are met. Optionally, the affine merge mode is assigned a merge index related to the number of neighboring blocks that use affine mode. Optionally, the affine merge mode is assigned a merge index equal to 5 minus the amount of adjacent blocks using the affine mode among the five A1, B1, B0, A0, B2.

[0010] According to another aspect of the present invention, there is provided a method for signaling an affine motion mode in a video stream, the method including: determining whether an affine mode likelihood is used for a current block; compiling a motion candidate predictor list; and inserting an affine merge mode as a merge candidate depending on determining the likelihood of the affine mode for the current block. Optionally, the likelihood is determined based on at least one of: a) the state of a skip flag; b) motion information of neighboring blocks; and c) ATMVP candidates. Optionally, if one or more of the following conditions are met: a) the state of a skip flag; b) the motion information of neighboring blocks is equal; and c) the ATMVP candidate includes only one motion information, the affine merge mode is not inserted as a merge candidate. Optionally, the neighboring blocks include blocks A1 and B1. Optionally, the affine mode is signaled depending on characteristics of a device used to record the video corresponding to the bitstream being encoded.

[0011] Aspects of the present invention provide improved coding efficiency and / or reduced coding complexity compared to existing coding standards or proposals. In this way, more efficient and faster video encoding and / or decoding methods and systems are provided. Further aspects of the present invention relate to encoding and decoding methods using any of the above aspects. Yet another aspect of the present invention relates to an apparatus for signaling the use of affine modes in a bitstream representing coded video, as defined by claim 14. Yet another aspect of the present invention relates to an encoding unit and a decoding unit, as defined by claims 17 and 18, respectively. Yet another aspect of the present invention relates to a program, as defined by claim 19. The program may be provided on its own or may be carried on, by, or within a carrier medium. The carrier medium may be non-transitory, for example, a storage medium, in particular a computer-readable storage medium. The carrier medium may also be transitory, for example, a signal or other transmission medium. The transmission may be transmitted via any suitable network, including the Internet.

[0012] Yet further aspects of the invention relate to a peripheral device such as a camera or a mobile device as defined by claims 15 and 16. Optionally, the camera may further comprise zoom means, and may be adapted to indicate that the zoom means is operating when the zoom means is operating and in a signal affine mode dependent on the instruction. Optionally, the camera may further comprise pan means, and may be adapted to indicate that the pan means is operating when the pan means is operating and in an affine mode dependent on the instruction. Optionally, the mobile device may further comprise at least one position sensor adapted to sense a change in orientation of the mobile device, and may be adapted to communicate the affine mode dependent on the sensing of a change in orientation of the mobile device. Further features of the invention are characterized by other independent and dependent claims.

[0013] Any feature of one aspect of the present invention may be applied to other aspects of the present invention in any appropriate combination. In particular, method aspects may be applied to apparatus aspects, and vice versa. Furthermore, features implemented in hardware may be implemented in software, and vice versa. Any references to software and hardware features herein should be construed accordingly. Any apparatus features as described herein may also be provided as method features, and vice versa. As used herein, means-plus-function features may alternatively be expressed in terms of their corresponding structure, such as a suitably programmed processor and associated memory. It should also be understood that specific combinations of various features described and defined in any aspect of the present invention may be implemented and / or provided and / or used independently. [Brief explanation of the drawings]

[0014] Reference will now be made, by way of example, to the accompanying drawings in which:

[0015] [Figure 1]FIG. 1 is a diagram illustrating the coding structure used in HEVC. [Figure 2] FIG. 2 is a block diagram that schematically illustrates a data communications system in which one or more embodiments of the present invention may be implemented. [Figure 3] FIG. 3 is a block diagram illustrating components of a processing device in which one or more embodiments of the present invention may be implemented. [Figure 4] FIG. 4 is a flow chart illustrating steps of an encoding method according to an embodiment of the invention. [Figure 5] FIG. 5 is a flow chart illustrating the steps of a decoding method according to an embodiment of the present invention. [Figure 6a] FIG. 6(a) shows the spatial and temporal blocks used to generate the motion vector predictor. [Figure 6b] FIG. 6(b) shows the spatial and temporal blocks used to generate the motion vector predictor. [Figure 7] FIG. 7 shows simplified steps in the process of AMVP predictor set derivation. [Figure 8] FIG. 8 is a schematic diagram of the motion vector derivation process in merge mode. [Figure 9] FIG. 9 shows the division and temporal motion vector prediction of the current block. [Figure 10] Figure 10(a) shows the encoding of merge indexes for HEVC or when ATMVP is not enabled at the SPS level, and Figure 10(b) shows the encoding of merge indexes when ATMVP is enabled at the SPS level. [Figure 11] Figure 11(a) shows a simple affine motion field, and Figure 11(b) shows a more complex affine motion field. [Figure 12] FIG. 12 is a flowchart of the decoding process for some of the syntax elements associated with the coding modes. [Figure 13] FIG. 13 is a flowchart showing deriving merge candidates. [Figure 14]FIG. 14 is a flowchart showing the first embodiment of the present invention. [Figure 15] FIG. 15 is a flowchart showing the second embodiment of the present invention. [Figure 16] FIG. 16 is a flowchart showing the third embodiment of the present invention. [Figure 17] FIG. 17 is a flowchart showing the fourth embodiment of the present invention. [Figure 18] FIG. 18 is a flowchart showing the fifth embodiment of the present invention. [Figure 19] FIG. 19 is a schematic block diagram of a computing device for implementing one or more embodiments of the present invention. [Figure 20] FIG. 20 is a schematic block diagram of a computing device. [Figure 21] FIG. 21 is a diagram showing a network camera system. [Figure 22] FIG. 22 is a diagram illustrating a smartphone. DETAILED DESCRIPTION OF THE INVENTION

[0016] The present invention relates to improved propagation of affine motion modes, particularly determining when affine modes may result in improved coding efficiency and ensuring that affine modes are used and / or prioritized accordingly. Figure 1 illustrates a coding structure used in the High Efficiency Video Coding (HEVC) video standard. A video sequence 1 is composed of a series of digital images i. Each such digital image is represented by one or more matrices. The matrix coefficients represent pixels. Images 2 of the sequence may be divided into slices 3. A slice may, in some instances, constitute an entire image. These slices are divided into non-overlapping coding tree units (CTUs). A coding tree unit (CTU) is the basic processing unit of the High Efficiency Video Coding (HEVC) video standard and conceptually corresponds to the macroblock unit used in some previous video standards. A CTU is sometimes referred to as a maximum coding unit (LCU). A CTU has luma and chroma component parts, each of which is called a coding tree block (CTB). These different color components are not shown in Figure 1.

[0017] A CTU is typically 64 pixels by 64 pixels in size for HEVC, and may be 128 pixels by 128 pixels for VVC. Each CTU may be iteratively and sequentially divided into smaller, variable-size coding units (CUs) 5 using quadtree decomposition. A coding unit is a basic coding element and is composed of two types of subunits called prediction units (PUs) and transform units (TUs). The maximum size of a PU or TU is equal to the CU size. A prediction unit corresponds to the partitioning of a CU for the prediction of pixel values. Various different partitions of a CU into PUs are possible, including two different partitions into four square PUs and two rectangular PUs, as shown by 606. A transform unit is a basic unit that is subject to spatial transformation using a DCT. A CU may be divided into TUs based on a quadtree representation 607.

[0018] Each slice is embedded in one network abstraction layer (NAL) unit. Furthermore, the coding parameters of a video sequence are stored in a dedicated NAL unit called a parameter set. HEVC and H.264 / AVC use two types of parameter set NAL units: first, the sequence parameter set (SPS) NAL unit, which collects all parameters that do not change during the entire video sequence. Typically, it handles coding profiles, video frame sizes, and other parameters. Second, the picture parameter set (PPS) NAL unit contains parameters that can change from one picture (or frame) of a sequence to another. HEVC also includes the video parameter set (VPS) NAL unit, which contains parameters that describe the overall structure of the bitstream. The VPS is a new type of parameter set defined in HEVC and applies to all layers of the bitstream. A layer may contain multiple temporal sublayers; all Version 1 bitstreams are limited to a single layer. HEVC has specific hierarchical extensions for scalability and multiview, which allow multiple layers with a backward-compatible Version 1 base layer.

[0019] FIG. 2 illustrates a data communication system in which one or more embodiments of the present invention may be implemented. The data communication system includes a transmitting device, in this case a server 201, operable to transmit data packets of a data stream to a receiving device, in this case a client terminal 202, via a data communication network 200. The data communication network 200 may be a wide area network (WAN) or a local area network (LAN). Such a network may be, for example, a wireless network (Wi-Fi / 802.11a or b or g), an Ethernet network, an Internet network, or a hybrid network composed of multiple different networks. In certain embodiments of the present invention, the data communication system may be a digital television broadcasting system in which the server 201 transmits the same data content to multiple clients. The data stream 204 provided by the server 201 may consist of multimedia data representing video and audio data. The audio and video data streams may, in some embodiments of the present invention, be captured by the server 201 using a microphone and a camera, respectively. In some embodiments, the data stream may be stored on the server 201, received by the server 201 from another data provider, or generated by the server 201. The server 201 comprises, inter alia, an encoder for encoding the video and audio streams to provide a compressed bitstream for transmission, which is a more compact representation of the data presented as input to the encoder.

[0020] To obtain a better ratio of the quality of the transmitted data to the amount of the transmitted data, the compression of the video data may be, for example, according to the HEVC format or the H.264 / AVC format. The client 202 receives the transmitted bitstream and decodes the reconstructed bitstream to reproduce the video images on a display device and the audio data through a speaker. Although the example of Figure 2 considers a streaming scenario, it will be understood that in some embodiments of the present invention, data communication between the encoder and decoder may be performed using a media storage device, such as an optical disk. In one or more embodiments of the present invention, the video images are transmitted together with data representing a compensation offset for application to the reconstructed pixels of the image to provide filtered pixels in the final image.

[0021] 3 shows a schematic diagram of a processing device 300 configured to implement at least one embodiment of the present invention. The processing device 300 may be a device such as a microcomputer, a workstation, or a light portable device. The device 300 comprises a communication bus 313 connected to: a central processing unit 311 such as a microprocessor, denoted CPU; a read-only memory 306, denoted ROM, for storing computer programs for implementing the present invention; a random access memory 312, denoted RAM, for storing executable code of the methods of embodiments of the present invention and for storing registers adapted to record variables and parameters necessary for implementing the method for encoding a sequence of digital images and / or the method for decoding a bitstream according to embodiments of the present invention; and a communication interface 302 connected to a communication network 303 over which digital data to be processed are transmitted or received.

[0022] Optionally, the device 300 may also include the following components: a data storage means 304, such as a hard disk, for storing computer programs for implementing the methods of one or more embodiments of the present invention and data used or generated during the performance of one or more embodiments of the present invention; a disk drive 305 for a disk 306, the disk drive adapted to read data from or write data to said disk; a screen 309 for displaying data and / or serving as a graphical interface with a user by means of a keyboard 310 or any other indicating means.

[0023] The device 300 may be connected to various peripheral devices, such as, for example, a digital camera 320 or a microphone 308, each connected to an input / output card (not shown) for providing multimedia data to the device 300. A communication bus provides communication and interconnectivity between the various elements included in or connected to the device 300. The representation of the bus is not limiting, and in particular the central processing unit is operable to communicate instructions to any element of the device 300, either directly or by means of another element of the device 300.

[0024] The disk 306 may be replaced by any information carrier, such as a compact disk (CD-ROM), a ZIP disk or a memory card, rewritable or not, and in general by information storage means readable by a microcomputer or microprocessor, integrated in the device or not, possibly removable, and adapted to store one or more programs whose execution enables the method of encoding a sequence of digital images and / or the method of decoding a bitstream according to the invention to be implemented. The executable code may be stored either in the read-only memory 306, on the hard disk 304 or on a removable digital medium, such as the disk 306 as mentioned above. According to a variant, the executable code of the program may be received by means of the communication network 303, via the interface 302, to be stored in one of the storage means of the device 300, such as the hard disk 304, before being executed.

[0025] The central processing unit 311 is adapted to control and direct the execution of instructions or parts of the software code of a program or program according to the invention, instructions stored in one of the aforementioned storage means. At power-on, the program or programs stored in a non-volatile memory, for example on the hard disk 304 or on the read-only memory 306, are transferred to the random access memory 312, which then contains registers for storing the executable code of the program or programs, as well as variables and parameters necessary for carrying out the invention. In this embodiment, the device is a programmable device that uses software to carry out the invention. However, alternatively, the invention may be implemented in hardware (for example in the form of an application-specific integrated circuit or ASIC).

[0026] FIG. 4 shows a block diagram of an encoding unit according to at least one embodiment of the present invention. The encoding unit is represented by connected modules, each adapted to perform at least one corresponding step of a method for implementing at least one embodiment of encoding images of an image sequence according to one or more embodiments of the present invention, e.g., in the form of program instructions to be executed by the CPU 311 of the device 300. An original sequence of digital images i0 to in401 is received as input by the encoding unit 400. Each digital image is represented by a set of samples known as pixels. A bitstream 410 is output by the encoding unit 400 after implementing the encoding process. The bitstream 410 comprises a number of encoding units, or slices, each of which comprises a slice header comprising coded video data and a slice header for transmitting coded values ​​of coding parameters used to code the slice body.

[0027] The input digital images i0 to in401 are divided into blocks of pixels by module 402. The blocks correspond to image portions and may be of variable size (for example, 4x4, 8x8, 16x16, 32x32, 64x64, 128x128 pixels, as well as several rectangular block sizes). A coding mode is selected for each input block. Two families of coding modes are provided: coding modes based on spatial predictive coding (intra prediction) and coding modes based on temporal prediction (inter coding, merge skip). Possible coding modes are tested. Module 403 performs an intra prediction process in which a given block to be coded is predicted by a predictor calculated from pixels in the neighborhood of the block to be coded. An indication of the selected intra predictor and the difference between a given block and its predictor are coded to provide a residue if intra coding is selected.

[0028] Temporal prediction is performed by the motion estimation module 404 and the motion compensation module 405. First, a reference image is selected from the set of reference images 416, and the portion of the reference image, also called a reference region or image portion, which is the region closest to the predetermined block to be coded, is selected by the motion estimation module 404. The motion compensation module 405 then predicts the block to be coded using the selected region. The difference between the selected reference region and the predetermined block, also called a residual block, is calculated by the motion compensation module 405. The selected reference region is indicated by a motion vector. Thus, in both cases (spatial and temporal prediction), the residual is calculated by subtracting the prediction from the original block. In intra prediction performed by the module 403, the prediction direction is coded. In temporal prediction, at least one motion vector is coded. If inter prediction is selected, information about the motion vector and the residual block is coded. To further reduce the bit rate, assuming homogeneous motion, the motion vector is coded by a difference relative to the motion vector predictor. The motion vector predictors of the motion information predictor set are obtained from the motion vector field 418 by the motion vector prediction and coding module 417 .

[0029] The encoder 400 further comprises a selection module 406 for coding mode selection by applying a coding cost criterion, such as a rate-distortion criterion. To further reduce redundancy, a transform (e.g., DCT) is applied to the residual block by a transform module 407, and the resulting transformed data is quantized by a quantization module 408 and entropy coded by an entropy coding module 409. Finally, the coded residual block of the current block being coded is inserted into a bitstream 410. The encoder 400 also decodes the coded image to generate reference images for motion estimation of subsequent images. This allows the encoder and decoder receiving the bitstream to have the same reference frame. An inverse quantization module 411 performs inverse quantization of the quantized data, followed by an inverse transformation by an inverse transform module 412. A backward intra prediction module 413 uses the prediction information to determine which predictor to use for a given block, and a backward motion compensation module 414 actually adds the residual obtained by module 412 to a reference region obtained from a set of reference images 416.

[0030] Post-filtering is then applied by module 415 to filter the frame of reconstructed pixels. In an embodiment of the present invention, an SAO loop filter is used, in which a compensation offset is added to the pixel values ​​of the reconstructed pixels of the reconstructed image. Figure 5 shows a block diagram of a decoder 60 that can be used to receive data from an encoder according to an embodiment of the present invention. The decoder is represented by connected modules, each adapted to implement a corresponding step of a method performed by the decoder 60, e.g., in the form of program instructions executed by the CPU 311 of the device 300. The decoder 60 receives a bitstream 61 containing coding units, each consisting of a header containing information about coding parameters and a body containing the coded video data. As described with reference to Figure 4, the coded video data is entropy coded, and the motion vector predictor index is coded with a predetermined number of bits for a given block. The received coded video data is entropy decoded by module 62. The residual data is then inversely quantized by module 63, and then an inverse transform is applied by module 64 to obtain pixel values.

[0031] The mode data indicating the coding mode is also entropy decoded, and based on the mode, intra-type decoding or inter-type decoding is performed on the coded block of image data. For intra mode, an intra predictor is determined by the intra inverse prediction module 65 based on the intra prediction mode specified in the bitstream. If the mode is inter, motion prediction information is extracted from the bitstream to find the reference region to be used by the encoder. The motion prediction information consists of a reference frame index and a motion vector residual. The motion vector predictor is added to the motion vector residual to obtain a motion vector by the motion vector decoding module 70.

[0032] The motion vector decoding module 70 applies motion vector decoding to each current block coded by motion prediction. For the current block, the motion vector predictor index is used by module 66 to apply backward motion compensation once the actual value of the motion vector associated with the current block, which can be decoded, is obtained. The reference image portion indicated by the decoded motion vector is extracted from the reference image 68 to apply backward motion compensation 66. The motion vector field data 71 is updated with the decoded motion vector for use in inverse prediction of subsequent decoded motion vectors. Finally, a decoded block is obtained. Post-filtering is applied by the post-filtering module 67. A decoded video signal 69 is finally provided by the decoding unit 60.

[0033] (CABAC) HEVC uses several types of entropy coding, including context-adaptive binary arithmetic coding (CABAC), Golomb-Rice coding, or a simple binary representation called fixed-length coding. In most cases, a binary coding process is performed to represent different syntax elements. This binary coding process is also very specific and depends on the different syntax elements. Arithmetic coding represents syntax elements according to their current probability. CABAC is an extension of arithmetic coding that separates the probability of syntax elements according to a "context" defined by a context variable. This corresponds to conditional probability. The context variable can be derived from the current syntax values ​​of the top-left block (A2 in Figure 6b, as explained in more detail below) and the top-left block (B3 in Figure 6b) that have already been coded.

[0034] (Inter-coding) HEVC uses three different inter modes: inter mode, merge mode, and merge skip mode. The main difference between these modes is the data transmission in the bitstream. For motion vector coding, the current HEVC standard includes a contention-based scheme for motion vector prediction, which did not exist in previous versions of the standard. This means that several candidates compete against a distortion rate criterion at the encoder side to find the best motion vector predictor or best motion information for each inter or merge mode. An index corresponding to the best candidate for the best predictor or motion information is inserted into the bitstream. The decoder can derive the same set of predictors or candidates and use the best one according to the composite index. In the HEVC screen content extension, a new coding tool called intra block copy is transmitted as one of the three inter modes. The difference between IBC and the equivalent inter mode is performed by checking whether the reference frame is the current one. This can be implemented, for example, by checking the reference index of list L0 and inferring that if this is the last frame in that list, it is an intra block copy. Another way to implement is to compare the current picture order count with the reference frame: if they are equal, then it is an intra block copy.

[0035] The design of predictor and candidate derivation is important to achieve the best coding efficiency without disproportionately affecting complexity. In HEVC, two motion vector derivations are used: one for inter mode (Advanced Motion Vector Prediction (AMVP)) and one for merge mode (merge derivation process). These processes are described below. Figures 6a and 6b show spatial and temporal blocks that can be used to generate motion vector predictors in Advanced Motion Vector Prediction (AMVP) and merge mode of the HEVC encoding and decoding system, and Figure 7 shows simplified steps in the process of AMVP predictor set derivation. Two predictors, i.e., two spatial motion vectors in AMVP mode, are selected from the top block (indicated by the letter "B") and left block (indicated by the letter "A"), including the top corner block (block B2) and the left corner block (block A0), and one predictor is selected from the bottom right block (H) and center block (center) of the blocks arranged as shown in Figure 6a. Table 1 below outlines the nomenclature used when referring to blocks in current terms as shown in Figures 6a and 6b. This nomenclature is used as a shorthand notation, but it should be understood that other labeling systems may be used, especially in future versions of the standard.

[0036] [Table 1]

[0037] It should be noted that the "current block" may be variable in size, e.g., 4x4, 16x16, 32x32, 64x64, 128x128, or any size in between. The block dimensions are preferably a factor of two (i.e., 2^n x 2^m, where n and m are positive integers), as this results in more efficient use of bits when using binary encoding. The current block does not need to be square, although this is often the preferred embodiment for encoding complexity reasons. Referring to Figure 7, the first step aims to select a first spatial predictor (candidate 1, 706) from among the bottom-left blocks A0 and A1, whose spatial location is shown in Figure 6. To that end, these blocks are selected one after the other in a predetermined order (700, 702), and for each selected block, the following conditions are evaluated in a predetermined order (704), and the first block for which the conditions are met is set as the predictor, i.e., a motion vector from the same reference list and the same reference picture, a motion vector from the same reference picture as another reference list, a scaled motion vector from the same reference list and a different reference picture, or a scaled motion vector from the other reference list and a different reference picture.

[0038] If no value is found, the left predictor is deemed unavailable. This indicates that the associated block is intra-coded or does not exist. The next step aims to select a second spatial predictor (candidate 2, 716) from among the top right block B0, top block B1, and top left block B2, whose spatial locations are shown in FIG. 6. To do this, these blocks are selected one after the other in a predetermined order (708, 710, 712). For each selected block, the above conditions are evaluated in a predetermined order (714). The first block for which the above conditions are met is set as the predictor. Again, if no value is found, the top predictor is deemed unavailable. This indicates that the associated block is intra-coded or does not exist. In the next step (718), the two predictors are compared with each other if both are available, in order to eliminate one of them if they are equal (i.e., have the same motion vector value, the same reference list, the same reference index, and the same direction type). If only one spatial predictor is available, the algorithm looks for a temporal predictor in the next step.

[0039] The temporal motion predictor (Candidate 3, 726) is derived as follows: the bottom right (H, 720) location of the aligned block in the previous frame is first considered in the availability check module 722. If it does not exist, or if a motion vector predictor is not available, the center (Center, 724) of the aligned block is selected for checking. These temporal locations (Center and H) are shown in FIG. 6. In either case, scaling 723 is applied to the candidates so that the temporal distance between the current frame and the first frame matches the reference list. The motion predictor value is then added to the set of predictors. The number of predictors (Nb_Cand) is then compared (728) with the maximum number of predictors (Max_Cand). As mentioned above, the maximum number of motion vector predictors the AMVP derivation process needs to generate (Max_Cand) is 2 in the current version of the HEVC standard. If this maximum number is reached, a final list or set of AMVP predictors (732) is constructed. Otherwise, a zero predictor is added to the list 730. A zero predictor is a motion vector equal to (0,0).

[0040] As shown in Figure 7, the final list or set of AMVP predictors (732) is constructed from a subset of spatial motion predictors (700-712) and a subset of temporal motion predictors (720, 724). As mentioned above, a motion predictor candidate for merge or merge skip mode represents all necessary motion information: direction, list, reference frame index, and motion vector. An indexed list of candidates is generated by the merge derivation process. In the current HEVC design, the maximum number of candidates for both merge modes is equal to five (four spatial candidates and one temporal candidate). Figure 8 is a schematic diagram of the motion vector derivation process for merge mode. In the first step of the derivation process, five block locations are considered (800-808). These locations are the spatial locations indicated in Figure 3 by the criteria A1, B1, B0, A0, and B2. In the next step, the availability of spatial motion vectors is checked, and at most five motion vectors are selected (810). A predictor is considered available if it exists and if the block is not intra-coded.

[0041] Therefore, the motion vectors corresponding to the five blocks are selected as candidates according to the following conditions: if the "left" A1 motion vector (800) is available (810), i.e., if it exists and this block is not intra-coded, the "left" block motion vector is selected and used as the first candidate in the candidate list (814); if the "top" B1 motion vector (802) is available (810), the candidate "top" block motion vector is compared to the "left" A1 motion vector, if it exists (812). If the B1 motion vector is equal to the A1 motion vector, B1 is not added to the list of spatial candidates (814). Conversely, if the B1 motion vector is not equal to the A1 motion vector, B1 is added to the list of spatial candidates (814); if the "top right" B0 motion vector (804) is available (810), the "top right" motion vector is compared to the B1 motion vector (812). If the B0 motion vector is equal to the B1 motion vector, the B0 motion vector is not added to the list of spatial candidates (814). Conversely, if the B0 motion vector is not equal to the B1 motion vector, the B0 motion vector is added to the list of spatial candidates (814).

[0042] If the "bottom left" A0 motion vector (806) is available (810), it is compared with the A1 motion vector (812). If the A0 motion vector is equal to the A1 motion vector, it is not added to the list of spatial candidates (814). Conversely, if the A0 motion vector is not equal to the A1 motion vector, it is added to the list of spatial candidates (814). If the list of spatial candidates does not include four candidates, the availability of the "top left" B2 motion vector (808) is checked (810). If available, it is compared with the A1 and B1 motion vectors. If the B2 motion vector is equal to either the A1 or B1 motion vector, it is not added to the list of spatial candidates (814). Conversely, if the B2 motion vector is not equal to either the A1 or B1 motion vector, it is added to the list of spatial candidates (814).

[0043] At the end of this stage, the list of spatial candidates contains up to four candidates. For temporal candidates, two positions can be used: the bottom right position of the located block (designated H in FIG. 6 at 816) and the center of the located block (818). These positions are shown in FIG. 6. For the AMVP motion vector derivation process, the first step is to check the availability of a block in the H position (820). If it is not available, the availability of a block in the center position is checked (820). If at least one motion vector in these positions is available, the temporal motion vector can be scaled to the reference frame with index 0 (822), if necessary, for both lists L0 and L1, to create a temporal candidate (824) that is added to the list of merge motion vector predictor candidates, which is placed after the spatial candidate in the list. Lists L0 and L1 are two reference frame lists containing 0, one, or more reference frames.

[0044] If the number of candidates (Nb_Cand) is strictly less than the maximum number of candidates (826) (the value Max_Cand is signaled in the bitstream slice header and is equal to 5 in the current HEVC design) and if the current frame is of type B, a combined candidate is generated (828). The combined candidate is generated based on the available candidates in the list of merge motion vector predictor candidates. It mainly consists of combining the motion vector of one candidate in list L0 with the motion vector of one candidate in list L1. If the number of candidates (Nb_Cand) remains strictly less than the maximum number of candidates (Max_Cand) (830), zero motion candidates are generated (832) until the number of candidates in the merge motion vector predictor candidate list reaches the maximum number of candidates. At the end of this process, a list or set of merge motion vector predictor candidates is constructed (834). As shown in FIG. 8, a list or set of merge motion vector predictor candidates is constructed (834) from a subset of spatial candidates (800-808) and a subset of temporal candidates (816, 818).

[0045] Alternative Temporal Motion Vector Prediction (ATMVP) Alternative Temporal Motion Vector Prediction (ATMVP) is a specific type of motion compensation. Instead of considering only one motion vector for the current block from the temporal reference frame, each motion vector for each of the adjacent blocks is considered. This temporal motion vector prediction therefore gives the current block's partition along with the associated motion vector for each subblock, as shown in Figure 9. In the current VTM reference software, ATMVP is signaled as a merge candidate inserted into the list of merge candidates. When ATMVP is enabled at the SPS level, the maximum number of merge candidates is increased by one. Therefore, when this mode is disabled, six candidates are considered instead of five.

[0046] Furthermore, when this prediction is enabled at the SPS level, all bins of the merge index become contexts coded by CABAC. While in HEVC, or when ATMVP is not enabled at the SPS level, only the first bin is a coded context, and the remaining bins are bypass coded contexts. Figure 10(a) shows the coding of the merge index for HEVC, or when ATMVP is not enabled at the SPS level. This corresponds to unary maximal coding. Furthermore, the first bit is CABAC coded, and the other bits are bypass CABAC coded. Figure 10(b) shows the coding of the merge index when ATMVP is enabled at the SPS level. Furthermore, all bits are CABAC coded (from the first to the fifth bit). Note that each index has its own context; in other words, their probabilities are separated.

[0047] (Affine mode) In HEVC, only the translational motion model is applied for motion compensation prediction (MCP). In contrast, in the real world, there are many types of motion, such as zoom-in / zoom-out, rotation, perspective motion, and other irregular motions. In JEM, a simple affine transformation motion compensation prediction is applied. The general principles of affine mode are described below based on an extract from document JVET-G1001, presented at the JVET Conference in Turin on July 13-21, 2017. This entire document is incorporated herein by reference, as it describes other algorithms used in JEM. As shown in Figure 11(a), the affine motion field of a block is described by two control point motion vectors. The motion vector field (MVF) of a block is described by the following equation:

[0048]

number

[0049] where (v 0x , v 0y ) is the motion vector of the control point in the upper left corner, and (v 1x , v 1y ) is the motion vector of the control point in the upper right corner. To further simplify the motion compensation prediction, sub-block-based affine transformation prediction is applied. The sub-block size is derived as shown in Equation 2, where MvPre is the motion vector fractional precision (1 / 16 in JEM), and (v2x, v2y) is the motion vector of the bottom-left control point calculated according to Equation 1.

[0050]

number

[0051] After being derived according to Equation 2, M and N may be adjusted downward, if necessary, to become divisors of w and h, respectively. To derive the motion vector for each M×N sub-block, the motion vector for the center sample of each sub-block is calculated according to Equation 1 and rounded up to 1 / 16 fractional precision, as shown in Figure 6a. A motion compensated interpolation filter is then applied to generate a prediction for each sub-block with the derived motion vector.

[0052] Affine mode is a motion compensation mode as an inter mode (AMVP, merge, merge skip). Its principle is to generate one motion information per pixel according to two or three neighboring motion information. In the current VTM reference software, affine mode derives one motion information per 4x4 block, as shown in Figure 11(a). This mode is available for AMVP, and both merge modes are enabled by a flag. This flag is CABAC coded. In one embodiment, the context depends on the sum of the affine flags of the left block (position A2 in Figure 6b) and the top-left block (position B3 in Figure 6b).

[0053] Thus, in JEM, there are three possible context variables (0, 1 or 2) for the affine flag given by: Ctx = IsAffine(A2) + IsAffine(B3) Here, IsAffine(block) is a function that returns 0 if the block is not an affine block and 1 if the block is affine.

[0054] (Affine merge candidate derivation) In JEM, the affine merge mode (merge or merge skip) is derived from the first neighboring block that is affine between blocks at positions A1, B1, B0, A0, B2. These positions are shown in Figures 6a and 6b. However, how the affine parameters are derived is not fully defined, and the present invention aims to improve at least this point.

[0055] (Affine merge transfer) Figure 12 is a flowchart of a partial decoding process for some syntax elements related to coding modes. In this figure, the skip flag (1201), prediction mode (1211), merge flag (1203), merge index (1208), and affine flag (1207) may be decoded. For all CUs in an inter slice, the skip flag is decoded (1201). If the CU is not a skip (1202), the predicate mode (prediction mode) is decoded (1211). This syntax element indicates whether the current CU is in inter or intra mode. Note that if a CU is a skip (1202), its current mode is inter mode. For a CU (1212), the CU is coded in AMVP or merge mode. If the CU is inter (1212), the merge flag is decoded (1203). If the CU is a merge (1204) or a skip (1202), it is verified whether the affine flag (1206) needs to be decoded (1205). If the current CU is a 2Nx2N CU, which means that the height and width of the CU must be equal in the current VVC, this flag is decoded.

[0056] Furthermore, at least one adjacent CU A1 or B1 or B0 or ​​A0 or B2 must be coded in affine mode (merge or AMVP). Finally, the current CU must not be a 4x4 CU; by default, CU 4x4 is disabled in the VTM reference software. If this condition (1205) is false, it is ensured that the current CU is coded in classical merge mode or merge skip mode, and the merge index is decoded (1208). If the affine flag (1206) is set equal to 1 (1207), the CU is a merge affine CU or merge skip affine CU, and the merge index (1208) does not need to be decoded. Otherwise, the current CU is a classical (basic) merge or merge skip CU, and the merge index candidate (1208) is decoded. Otherwise, the current CU is a classical (basic) merge or merge skip CU, and the merge index candidate (1208) is decoded. In this specification, "transmitting" can mean inserting into or extracting from the bitstream one or more syntax elements that represent enabling or disabling of mode or other information.

[0057] (Merge candidate derivation) Figure 13 is a flowchart illustrating merge candidate derivation. This derivation builds on the HEVC merge list derivation shown in Figure 8. The main changes compared to HEVC are the addition of ATMVP candidates (1319, 1321, 1323), a full overlap check of candidates (1320, 1325), and a new candidate ordering. ATMVP prediction is set as a special candidate because it represents some motion information of the current CU. The value of the first subblock (top left) is compared with the temporal candidate, and if they are equal, the temporal candidate is not added to the merge list (1320). The ATMVP candidate is not compared with other spatial candidates. Contrary to the temporal candidate, which is compared with each spatial candidate already in the list (1325), if it is a overlap candidate, it is not added to the merge candidate list.

[0058] When a spatial candidate is added to the list, it is compared with other spatial candidates in the list, which is not the case in the final version of HEVC (1310). In the current VTM version, the list of merge candidates is set as the following order, as determined to provide the best results across the coding test conditions: A1 B1 B0 A0 ATMVP B2 ·Temporal ·combination ·Zero_MV

[0059] It is important to note that spatial candidate B2 is set after ATMVP candidates. Furthermore, when ATMVP is enabled at the slice level, the maximum number of candidate lists is six, not five. An object of the present invention is to signal an affine mode to a portion of a bitstream in an efficient manner, taking into account coding efficiency and complexity. It is also an object of the present invention to signal an affine mode in a manner that requires a minimal amount of structural modification to existing video coding frameworks. Exemplary embodiments of the present invention will now be described with reference to FIGS. 13-21 . It should be noted that embodiments may be combined unless otherwise specified; for example, certain combinations of embodiments may improve coding efficiency at the expense of increased complexity, which may be acceptable in certain use cases. In general, by modifying the syntax for signaling motion predictor modes to use affine modes when they are more likely to provide improved motion compensation, it is possible to improve coding efficiency at an acceptable increase in coding complexity.

[0060] (First embodiment) In a first embodiment, an affine motion prediction mode may be signaled (e.g., enabled or disabled) for a portion of a bitstream for at least one inter mode. The inter prediction mode used for the portion of the bitstream is determined, and the affine motion mode is signaled (enabled or disabled) depending on the inter prediction mode used for the portion of the bitstream. An advantage of this embodiment is that it results in better coding efficiency by removing unused syntax. Furthermore, it reduces the complexity of the encoder by avoiding some inter coding possibilities that do not need to be evaluated. Finally, on the encoder side, some affine flags that are CABAC coded do not need to be extracted from the bitstream, which improves the efficiency of the coding process. In the example of the first embodiment, skip mode is not enabled for affine mode. If a CU is a skipped CU (based on the state or presence of a skip flag in the data stream), this means that the affine flag does not need to be extracted from the bitstream. Figure 14 (which shares the same structure as Figure 12 and the corresponding description applies here) illustrates this example. 14, if the CU is a skip (1402), the affine flag (1406) is not decoded and the condition of 1405 is not evaluated. If the CU is a skip, the merge index is decoded (1406).

[0061] The benefit of this example is improved coding efficiency for sequences with a small amount of motion, rather than reduced coding efficiency for sequences with more motion. This is because skip mode is typically used when there is little or no motion, and as such, it is unlikely that affine mode is appropriate. As noted above, the complexity of the encoding and decoding process is also reduced. In a further example, affine merge skip mode can be enabled or disabled at a high level, for example, at the slice, frame, sequence, or CTU level. This may be determined based on a high-level syntax flag. In such a case, affine merge skip may be disabled for sequences or frames with little motion, and enabled when the amount of motion increases. The benefit of this further example is flexibility regarding the use of affine merge skip.

[0062] In one embodiment, the affine merge skip mode is not evaluated at the encoder side, and as a result, the bitstream does not include the affine merge skip mode. The advantage is that coding efficiency is observed, but smaller than in the first embodiment. For affine, the merge and skip mode may not be enabled (or affine is enabled only in AMVP). In a further example, the merge and merge skip mode is not enabled for affine mode. This means that if a CU is skipped or merged, the affine flag does not need to be extracted from the bitstream. Compared to Figure 14, in this embodiment, modules 1405, 1406, and 1407 have been removed. The advantage of this example is similar to the example immediately above. The advantage is improved coding efficiency for sequences with little motion, and the same coding efficiency for sequences with more motion. As mentioned above, the complexity of the encoding and decoding process is reduced.

[0063] A high-level syntax element signals that affine merging can be enabled. In yet another example, affine merge mode and merge skip mode may be enabled or disabled at a high level, such as at the slice, frame, sequence, or CTU level with one flag. In this case, affine merging may be disabled for sequences or frames with little motion, and enabled when the amount of motion increases. An advantage of this additional embodiment is flexibility regarding the use of affine skip. In another example, one flag is signaled for merge skip mode and one flag for merge mode. In another example, affine merge skip mode and merge mode are not evaluated at the encoder side. As a result, the bitstream does not include affine merge skip mode. An advantage is that coding efficiency can be observed.

[0064] (Second embodiment) In a second embodiment, signaling the affine mode of the current block depends on the modes of one or more neighboring blocks. There may be a correlation to how neighboring blocks are coded, which can be utilized to improve coding efficiency. In particular, if one or more specific neighboring blocks use an affine mode, it is more likely that the affine mode is appropriate for the current mode. In one embodiment, the number of candidates for affine merge or affine merge skip mode is reduced to only two candidates. An advantage of this embodiment is reduced complexity on the decoder side, since fewer affine flags are decoded for the merge mode and fewer comparisons and memory buffer accesses are required for the affine merge check condition (1205). Fewer affine merge modes on the encoder side need to be evaluated.

[0065] In one example of the second embodiment, one adjacent block and one adjacent block to the left above the current block (e.g., candidates A1 and B1, as shown in FIG. 6) are evaluated to determine whether the affine flag needs to be decoded and to derive affine merge candidates. The advantage of utilizing only these two positions for affine merging is coding efficiency similar to maintaining five candidates as current VTM implementations with reduced complexity. FIG. 15 illustrates this embodiment. In this figure, compared to FIG. 12, module 1505 is modified by checking only positions A1 and B1. In a further example of the second embodiment, only candidates A2 and B3, as shown in FIG. 6b, are evaluated to determine whether the affine flag needs to be decoded and to derive affine merge candidates. The advantage of this example is the same as the previous example, but it also reduces "worst-case" memory accesses compared to the previous example. In fact, positions A2 and B3 are the same as those used for affine flag context derivation. Indeed, for the affine flag, the context derivation depends on the neighboring blocks at positions A2 and B3 in Figure 6b. Consequently, when the affine flag needs to be decoded, the affine flag values ​​at A2 and B3 are already in memory for the context derivation of the current affine flag, and therefore no further memory access is required.

[0066] (Third embodiment) In the third embodiment, conveying the affine mode of the current block relies on a list of merge candidates corresponding to blocks adjacent to the current block. In one example of the third embodiment, the list starts with the block being used to determine the context variables associated with the current block, because the affine flag values ​​of such blocks are already in memory for context derivation of the current affine flag, and no further memory access is required. For example, the possible affine merge candidates are ordered as A2 or B3 or BO or AO or B2 (instead of A1 or B1 or BO or AO or B2), as shown in FIG. 6(b). This improves coding efficiency compared to the current VTM, and also limits the number of affine flags that need to be accessed for affine flag decoding for the worst-case scenario. In the current version, only 5 for module 1205 and 2 for affine flag context derivation, and in the current embodiment, only 5 as the affine flag values ​​of A2 and B3 are already in memory for context derivation of the current affine flag, so no further memory access is required.

[0067] A variation of the third embodiment relates to context alignment. Signaling the affine mode may include inserting a context encoding flag into the data stream, and a context variable for the flag is determined based on whether the neighboring block uses the affine mode. In an alternative example of the third embodiment, the positions considered for the context derivation of the affine flag are positions A1 and B1 instead of positions A2 and B3, as shown in FIG. 6b. In that case, the same advantages as in the previous example are obtained. This is another alignment between the context and the affine merge derivation. In that case, the context variable for the affine flag is obtained according to the following formula: Ctx = IsAffine(A1) + IsAffine(B1), where Ctx is the context of the affine flag and IsAffine is a function that returns 0 if the block is not an affine block and 1 if the block is affine. In this example, the affine flag values ​​of A1 and B1 are stored in memory for the current context derivation of the affine flag; as such, no further memory access is required.

[0068] In a further alternative, the positions considered for the context derivation of the affine flag are positions A1 and B1 instead of positions A2 and B3 when the current block is in merge mode (both merge modes). An additional advantage compared to the previous example is better coding efficiency. Indeed, for AMVP, there is no need for context derivation to be aligned to the derivation of the motion vector, since for AMVP, affine blocks are not considered for this derivation.

[0069] (Fourth embodiment) In the fourth embodiment, the affine mode signaling is performed depending on whether the neighboring block is in merge mode or not. In one example of the fourth embodiment, the candidates for affine merging (merge and skip) may be only affine AMVP candidates. Figure 17 illustrates this embodiment. An advantage of this embodiment is reduced coding complexity, since only a few affine flags need to be decoded without affecting coding efficiency. In a further example of the fourth embodiment, the candidates for affine merging (merge and skip) may be only affine AMVP candidates or merge affine candidates, but not affine merge skip. As with the previous example, an advantage of this example is reduced coding complexity, since only a few affine flags need to be decoded without affecting coding efficiency.

[0070] (Fifth embodiment) In the fifth embodiment, signaling the affine mode includes inserting the affine mode as a candidate motion predictor. In one example of the fifth embodiment, affine merge (and merge skip) is signaled as a merge candidate. In this case, modules 1205, 1206, and 1207 in FIG. 12 are removed. In addition, the maximum possible number of merge candidates is incremented so as not to affect the coding efficiency of the merge mode. For example, in the current VTM version, this value is set equal to 6, so when this embodiment is applied to the current version of VTM, the value becomes 7. This advantage simplifies the design of syntax elements for the merge mode because fewer syntax elements need to be decoded. In some situations, coding efficiency may be observed.

[0071] Two possibilities for implementing this example are now described below. The position of a candidate motion predictor indicates its likelihood of being selected, and as such, the higher it is placed in the list (lower index value), the higher its likelihood of being selected as a motion vector predictor. In the first example, the affine merge index always has the same position in the list of merge candidates. This means having a fixed merge idx value. For example, this value may be set equal to 5, since affine merge mode should represent complex motion, which is not the most likely content. An additional advantage of this embodiment is that the current block can be set as an affine block if the syntax elements are parsed / decoded / read, rather than just decoding the data itself. Consequently, the value can be used to determine the CABAC context of the affine flag used for AMVP. Therefore, the conditional probability should be improved for this affine flag, and coding efficiency will be better.

[0072] In the second example, affine merge candidates are derived along with other merge candidates. In this example, a new affine merge candidate is added to the list of merge candidates. Figure 18 shows this example. Compared to Figure 13, the affine candidates are the first affine neighboring blocks A1, B1, BO, AO, and B2 (1917). If the same conditions as 1205 in Figure 12 are valid (1927), a motion vector field generated using affine parameters is generated to obtain affine candidates (1929). The initial candidate list can have four, five, six, or seven candidates, depending on the use of ATMVP, temporal, and affine candidates. The order among all these candidates is important because more likely candidates should be processed first to ensure they are more likely to make the cut in the motion vector candidates. The preferred order is as follows: A1 B1 B0 A0 Affine Merge ATMVP B2 temporal combination Zero_MV

[0073] It is important to note that the affine merge is before the ATMVP mode but after the four major neighboring blocks. The advantage of placing the affine merge before the ATMVP candidate, compared to placing it after the ATMVP and temporal predictor, is improved coding efficiency. This improvement in coding efficiency depends on the GOP (group of pictures) structure and the quantization parameter (QP) setting of each picture within the GOP. However, for the most commonly used GOPs and QP settings, this order provides improved coding efficiency. A further advantage of this solution is the clean design of the merge and merge skip, both in terms of syntax and derivation. Furthermore, the affine candidate merge index can be changed according to the availability or value (duplicate check) of the previous candidate in the list, resulting in efficient transmission. In a further example, the affine merge index can be variable according to one or several conditions.

[0074] For example, the merge index or position in the list associated with an affine candidate varies according to a criterion. The principle is to set a low value for the merge index corresponding to an affine merge if the affine merge has a high probability of being selected (and a higher value if there is a low probability of being selected). The advantage of this example is improved coding efficiency thanks to optimal adaptation of the merge index when it is most likely to be used.

[0075] The criteria for selecting the position of the affine mode in the list of merge candidates include: a) When skip mode is enabled (skip flag status) One example of applying this criterion is that if the affine merge index has a value set equal to a high value (e.g., 5), or if the current merge is in merge skip mode, it is set after the spatial and temporal MVs. As described for the first embodiment, affine mode is unlikely to be selected for skip mode because it is unlikely to be in any large (or complex) motion. b) Motion information of neighboring blocks In one example of applying this criterion, if the affine merge index has a value set equal to a high value, or if the motion information of one block to the left and one block above (e.g., blocks A1 and B1) is similar or equal, it is set after spatial and temporal MV. If A1 has the same motion information as B1, there is a high probability that the motion information is constant for the current block. Therefore, the affine merge has a low probability of being selected.

[0076] c) ATMVP candidate In one example of applying this criterion, if the affine merge index has a value set equal to a high value, or if the ATMVP candidate contains only one motion information, it is set after the spatial and temporal MV. In that case, there is no subdivision in the previous frame of the positioned block. Therefore, it is preferable not to set the affine at a high position in the merge list, since there is a slight possibility that the current block content is in a non-constant motion state. d) When the neighboring blocks use affine mode In one example of applying this criterion, the affine merge index has a value set equal to a low value, or if two or more neighboring blocks are affine, it is set before the temporal prediction and far from the spatial predictor. In an additional example of applying this criterion, the affine merge index or affine position (idx) is set equal to idx=PN, where P is the lowest possible position for the affine merge index and N is the number of affine neighboring blocks. In one example, P is 5, N is 5, and the neighboring blocks are A1, B1, B0, A0, B2. Note that in this notation, the highest position has an index value of 0.

[0077] In this example, the affine merge index of a merge candidate position is set according to the probability associated with its neighboring blocks: thus, the first position if all neighboring positions are affine, and the fourth position if only one neighboring block is affine. It should be understood that the exemplary value "5" could be set to 6 or 7 to obtain similar coding efficiency. It should also be understood that combinations of these criteria are possible.

[0078] In another example of the fifth embodiment, the affine mode is signaled depending on the determination of the likelihood of the affine mode of the current block. In a specific example, the affine merge candidate is not added to the list of candidates, or there is no index corresponding to the affine merge according to the criteria. The principle of this example is to disable affine modes that are unlikely to be useful. The advantage of this example is improved coding efficiency thanks to optimal use of merge index bits.

[0079] Criteria for determining the likelihood of an affine mode being useful include: a) Skip flag status In one example of applying this criterion, if the current merge is in skip merge mode, no affine merge candidates are added. As explained in the first embodiment, there is a low probability that affine mode will be selected in skip mode. b) Motion information of neighboring blocks In one embodiment applying this criterion, an affine merge candidate is not added if the motion information of one block to the left and one block above (e.g., blocks A1 and B1) is similar or equal. If one block to the left and one block above (e.g., blocks A1 and B1) have the same motion information, there is a high probability that the motion information is constant relative to the current block. Therefore, the affine merge will be invalidated. c) ATMVP candidate In one embodiment applying this criterion, if the ATMVP candidate contains only one motion information, no affine merge candidate is added. In such an instance, it is preferable to disable affine at a higher position in the merge list, since there is a small chance that the current block content is within a non-constant motion. It should be understood that combinations of these criteria are possible.

[0080] (Implementation of an embodiment of the present invention) 20 is a schematic block diagram of a computing device 1300 for implementing one or more embodiments of the present invention. The computing device 1300 may be a device such as a microcomputer, a workstation, or a light portable device. The computing device 1300 includes a central processing unit (CPU) 1301, such as a microprocessor, having a communication bus connected to: a random access memory (RAM) 1302 for storing executable code for methods of embodiments of the present invention and registers adapted for recording variables and parameters required to implement methods for encoding or decoding at least a portion of an image according to embodiments of the present invention, the memory capacity of which may be expanded, for example, by an optional RAM connected to an expansion port; a read-only memory (ROM) 1303 for storing computer programs for implementing embodiments of the present invention; and a network interface (NET) 1304, typically connected to a communication network over which digital data to be processed is transmitted or received. The network interface (NET) 1304 may be a single network interface or may consist of a set of different network interfaces (e.g., wired and wireless interfaces, or different types of wired or wireless interfaces).

[0081] Data packets are written to the network interface for transmission or read from the network interface for reception under the control of a software application executing on the CPU 1301. A user interface (UI) 1305 may be used to receive input from a user or display information to a user. A hard disk (HD) 1306 may be provided as a mass storage device. An input / output module (IO) 1307 may be used to receive / transmit data from / to an external device such as a video source or display. Executable code may be stored in either the ROM 1303, the HD 1306, or a removable digital medium, such as a disk. According to a variant, the executable code of the program may be received by means of a communications network via the NET 1304 to be stored in one of the storage means of the communications device 1300, such as the HD 1306, before being executed. The CPU 1301 is adapted to control and direct the execution of instructions or portions of the program or software code of the program according to an embodiment of the present invention, the instructions of which are stored in one of the aforementioned storage means. After power-on, CPU 1301 may execute instructions for software applications from main RAM memory 1302, for example, after these instructions have been loaded from program ROM 1303 or HD 1306. Such software applications, when executed by CPU 1301, cause the steps of the method according to the present invention to be performed.

[0082] It will also be appreciated that according to other embodiments of the present invention, a decoding unit according to the aforementioned embodiments is provided in a user terminal such as a computer, a mobile phone (cell phone), a tablet or any other kind of device (e.g., a display device) that can provide / display content to a user. According to yet another embodiment, an encoding unit according to the aforementioned embodiments is provided in an image capture device that also comprises a camera, video camera or network camera (e.g., a closed circuit television or video surveillance camera) that captures and provides content for the encoding unit to encode. Two such examples are provided below with reference to Figures 21 and 22.

[0083] FIG. 21 illustrates a network camera system 2100 including a network camera 2102 and a client device 2104. The network camera 2102 includes an imaging unit 2106, an encoding unit 2108, a communication unit 2110, and a control unit 2112. The network camera 2102 and the client device 2104 are interconnected via a network 200 so as to be able to communicate with each other. The imaging unit 2106 includes a lens and an imaging element (e.g., a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS)) to capture an image of a subject and generate image data based on the image. The image may be a still image or a video image. The imaging unit may also include zoom means and / or pan means adapted to zoom or pan (either optically or digitally), respectively. The encoding unit 2108 encodes the image data using the encoding method described in the first to fifth embodiments. The encoding unit 2108 uses at least one of the encoding methods described in the first to fifth embodiments. For another example, the encoding unit 2108 can use a combination of the encoding methods described in the first to fifth embodiments.

[0084] The communication unit 2110 of the network camera 2102 transmits the encoded image data encoded by the encoding unit 2108 to the client device 2104. Furthermore, the communication unit 2110 receives commands from the client device 2104. The commands include commands for setting parameters for encoding by the encoding unit 2108. The control unit 2112 controls other units in the network camera 2102 according to the commands received by the communication unit 2110. The client device 2104 has a communication unit 2114, a decoding unit 2116, and a control unit 2118. The communication unit 2118 of the client device 2104 transmits commands to the network camera 2102. Furthermore, the communication unit 2118 of the client device 2104 receives the encoded image data from the network camera 2102. The decoding unit 2116 decodes the encoded image data by using the decoding method described in any of the first to fifth embodiments. As another example, the decoding unit 2116 can use a combination of the decoding methods described in the first to fifth embodiments.

[0085] The control unit 2118 of the client device 2104 controls other units within the client device 2104 in accordance with a user operation or command received by the communication unit 2114. The control unit 2118 of the client device 2104 controls the display device 2120 to display the image decoded by the decoding unit 2116. The control unit 2118 of the client device 2104 also controls the display device 2120 to display a GUI (Graphical User Interface) for specifying parameter values ​​of the network camera 2102, including parameters for encoding by the encoding unit 2108. The control unit 2118 of the client device 2104 also controls other units within the client device 2104 in accordance with a user operation input to the GUI displayed by the display device 2120. The control unit 2118 of the client device 2104 controls the communication unit 2114 of the client device 2104 to send a command to the network camera 2102 to specify parameter values ​​of the network camera 2102 in accordance with a user operation input to the GUI displayed by the display device 2120. The network camera system 2100 can determine whether the camera 2102 utilizes a zoom or pan while recording video, and such information may be used when encoding the video stream as a zoom or pan, while the shot can benefit from the use of an affine mode, which is well suited to encoding complex movements such as zooming, rotation, and / or stretching (which can be a side effect of panning, particularly if the lens is a "fisheye" lens).

[0086] FIG. 22 is a diagram illustrating a smartphone 2200. The smartphone 2200 includes a communication unit 2202, a decoding / encoding unit 2204, a control unit 2206, and a display unit 2208. The communication unit 2202 receives encoded image data via a network. The decoding unit 2204 decodes the encoded image data received by the communication unit 2202. The decoding unit 2204 decodes the encoded image data by using the decoding method described in the first to fifth embodiments. The decoding unit 2204 can use at least one of the decoding methods described in the first to fifth embodiments. For example, the encoding unit 2202 can use a combination of the decoding methods described in the first to fifth embodiments. The control unit 2206 controls other units within the smartphone 2200 in response to a user operation or command received by the communication unit 2202.

[0087] For example, the control unit 2206 controls the display device 2208 to display images decoded by the decoding unit 2204. The smartphone may further include an image recording device 2210 (e.g., a digital camera associated with the circuitry) for recording images or videos. Such recorded images or videos may be encoded by the decoding / encoding unit 2204 under the direction of the control unit 2206. The smartphone may further include a sensor 2212 adapted to sense the orientation of the mobile device. Such a sensor may include an accelerometer, gyroscope, compass, global positioning (GPS) unit, or similar position sensor. Such a sensor 2212 may determine whether the smartphone changes orientation, and such information may be used when encoding the video stream as changes in orientation, while video capture may benefit from the use of an affine mode, which is well suited to encoding complex movements such as rotations.

[0088] (Alternatives and Modifications) It will be appreciated that the object of the present invention is to ensure that affine modes are utilized in the most efficient way, and the particular examples given above relate to signaling the use of affine modes depending on the likelihood that they will be perceived as useful. A further example of this can be applied to the encoder when it is known that complex motion is being coded, for which affine transformations are particularly efficient. Examples of such cases include: a) Camera zoom in / out b) Portable cameras (e.g., cell phones) that change orientation during capture (i.e., rotational movement) c) Panning a "fisheye" lens camera (e.g. stretching / distorting parts of the image)

[0089] As such, the indication of complex motion during the recording process may be increased, making affine mode more likely to be used for slices, frame sequences, or the entire actual video stream. In a further example, affine mode may be more likely to be used depending on the features or functionality of the device used to record the video. For example, a mobile device is more likely to change orientation than a (say) fixed surveillance camera, so affine mode may be more appropriate for encoding video from the former. Examples of features or functionality include the presence / use of zoom means, the presence / use of position sensors, the presence / use of pan means, whether the device is handheld, or user selection on the device.

[0090] While the present invention has been described with reference to embodiments, it should be understood that the present invention is not limited to the disclosed embodiments. It will be understood by those skilled in the art that various changes and modifications can be made without departing from the scope of the present invention, as defined by the appended claims. All features disclosed herein (including the appended claims, abstract, and drawings), and / or all steps of any method or process so disclosed, may be combined in any combination, except combinations in which at least some of such features and / or steps are mutually exclusive. Each feature disclosed herein (including the appended claims, abstract, and drawings), unless otherwise stated, may be replaced by an alternative feature serving the same, equivalent, or similar purpose. Thus, unless otherwise stated, each feature disclosed is merely one example of a generic series of equivalent or similar features.

[0091] It should also be understood that any result of the above-described comparison, determination, evaluation, selection, execution, performance, or consideration, e.g., a selection made during encoding or filtering, may be indicated or determinable / inferable from data in the bitstream, e.g., in a flag or data indicating the result, such that the indicated or determined / inferred result may be used in processing instead of actually performing the comparison, determination, evaluation, selection, execution, implementation, or consideration, e.g., during a decoding process. In the claims, the word "comprises" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. The mere fact that different features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be advantageously used. Reference signs appearing in the claims are merely illustrative and shall not have a limiting effect on the scope of the claims.

Claims

1. 1. A method for decoding an image from a bitstream, comprising: decoding a first flag from the bitstream capable of indicating that a current block of the image is not skipped; decoding, from the bitstream, prediction mode information used to determine whether a prediction mode for the current block is an intra mode or an inter mode when the first flag indicates that the current block is not skipped; determining a prediction mode to be used for decoding the current block from a plurality of prediction modes including the intra mode and the inter mode based on the prediction mode information; generating a list of motion predictor candidates including candidates for sub-block affine prediction when the inter mode is determined as the prediction mode for the current block; the sub-block affine prediction candidates are arranged in the list as merge candidates below temporal motion vector candidates; The method, wherein the sub-block affine prediction uses two or three motion vectors to derive at least one motion vector for each sub-block of the current block.

2. selecting a sub-block merging mode using sub-block affine prediction for the current block; selecting the sub-block merging mode using sub-block affine prediction includes decoding a second flag from the bitstream using CABAC decoding; 2. The method of claim 1, wherein the context variable for the second flag is determined based on whether a first block adjacent to the current block uses sub-block affine prediction and whether a second block adjacent to the current block uses sub-block affine prediction.

3. 3. The method of claim 2, wherein the first block is located to the left of the current block and the second block is located above the current block.

4. 2. The method of claim 1, wherein, when the current block has a size of 16x16, the number of sub-blocks in the current block is 16, and in the sub-block affine prediction, one motion vector is derived for each sub-block of the current block using the two or three motion vectors.

5. The method of claim 1 , wherein the temporal motion vector candidates use motion vectors in blocks of an image different from an image containing the current block.

6. The method described in claim 1, characterized in that the current block is included in a coding tree unit (CTU), and the coding tree unit may have a size of 128x128.

7. The method described in claim 1, characterized in that when the first flag does not indicate that the current block is not skipped, the prediction mode information used to determine whether the prediction mode for the current block is the intra mode or the inter mode is not decoded from the bitstream.

8. 1. A method for encoding an image into a bitstream, comprising the steps of: determining a prediction mode to be used for encoding a current block of the image from a plurality of prediction modes, including an intra mode and an inter mode; generating a list of motion predictor candidates including candidates for sub-block affine prediction when the inter mode is determined as the prediction mode for the current block; the sub-block affine prediction candidates are arranged in the list as merge candidates below temporal motion vector candidates; The sub-block affine prediction derives at least one motion vector for each sub-block of the current block using two or three motion vectors; The bitstream includes a first flag capable of indicating that a current block of the image is not skipped, and further includes prediction mode information used to determine whether a prediction mode for the current block is an intra mode or an inter mode when the first flag indicates that the current block is not skipped. A method characterized by:

9. selecting a sub-block merging mode using sub-block affine prediction for the current block; selecting the sub-block merging mode using sub-block affine prediction includes encoding a flag from the bitstream using CABAC coding; 9. The method of claim 8, wherein the context variable for the flag is determined based on whether a first block adjacent to the current block uses sub-block affine prediction and whether a second block adjacent to the current block uses sub-block affine prediction.

10. 10. The method of claim 9, wherein the first block is located to the left of the current block and the second block is located above the current block.

11. 9. The method of claim 8, wherein when the current block has a size of 16x16, the number of sub-blocks in the current block is 16, and in the sub-block affine prediction, one motion vector is derived for each sub-block of the current block using the two or three motion vectors.

12. The method of claim 8 , wherein the temporal motion vector candidates use motion vectors in blocks of an image different from an image containing the current block.

13. A decoding device for decoding an image from a bitstream, comprising: means for decoding from the bitstream a first flag capable of indicating that a current block of the image is not skipped; means for decoding, from the bitstream, prediction mode information used to determine whether a prediction mode for the current block is an intra mode or an inter mode when the first flag indicates that the current block is not skipped; means for determining a prediction mode to be used for decoding the current block from a plurality of prediction modes including the intra mode and the inter mode based on the prediction mode information; means for generating a list of a plurality of motion predictor candidates including candidates for sub-block affine prediction when the inter mode is determined as the prediction mode for the current block; the sub-block affine prediction candidates are arranged in the list as merge candidates below temporal motion vector candidates; The decoding device, wherein the sub-block affine prediction derives at least one motion vector for each sub-block of the current block using two or three motion vectors.

14. 1. A coding device for coding an image into a bitstream, comprising: means for determining a prediction mode to be used for encoding a current block of said image from a plurality of prediction modes including an intra mode and an inter mode; means for generating a list of a plurality of motion predictor candidates including candidates for sub-block affine prediction when the inter mode is determined as the prediction mode for the current block; the sub-block affine prediction candidates are arranged in the list as merge candidates below temporal motion vector candidates; The sub-block affine prediction derives at least one motion vector for each sub-block of the current block using two or three motion vectors; The bitstream includes a first flag that can indicate that the current block of the image is not skipped, and further includes prediction mode information that is used to determine whether the prediction mode for the current block is intra mode or inter mode when the first flag indicates that the current block is not skipped.

15. A computer program product for causing a computer to carry out the method according to any one of claims 1 to 7.

16. A computer program product for causing a computer to carry out the method according to any one of claims 8 to 12.

Images