Video encoding and decoding method and device

Abstract

A video encoding and decoding method and apparatus are described herein. A merged shared region may be received, the merged shared region including a plurality of coding blocks. A shared merge candidate list is constructed for the merged shared region. The merged shared region is decoded based on the shared merge candidate list. At least one coding block using inter-frame coding in the merged shared region is processed, and a history-based motion vector prediction (HMVP) table is not updated using motion information of the at least one inter-frame coding block.

Description

[0001] Related Documents

[0002] This application claims priority to U.S. Provisional Application No. 62 / 793,872, filed on January 17, 2019, “Method of HMVP Buffer Update When Shared MergeList Is Used,” and U.S. Application No. 16 / 744,938, filed on January 16, 2020, “Method and Apparatus for Video Coding,” which are incorporated herein by reference in their entirety. Technical Field

[0003] The disclosed embodiments relate to the field of video coding and decoding. Background Art

[0004] The background description provided herein is intended to present the background of the present application. The work of the named inventors, the work described in this background section, and the contents within the scope of the embodiments of this specification may not be considered as prior art at the time of filing, and are not explicitly or implicitly admitted as prior art against the present application.

[0005] Currently, video encoding and decoding can be performed using inter-frame image prediction combined with motion compensation. Uncompressed digital video typically includes a series of images. For example, each image has a luminance sample and associated chrominance samples at a resolution of 1920×1080. The series of images may have a fixed or variable image rate (also referred to as a frame rate) of, for example, 60 images per second or 60Hz. Therefore, uncompressed video has significant bit rate requirements. For example, 1080p604:2:0 video with 8 bits per sample (1920×1080 luminance sample resolution at 60Hz frame rate) requires a bandwidth of nearly 1.5Gbit / s. One hour of such video requires more than 600GB of storage space.

[0006] One purpose of video encoding and decoding can be to reduce redundancy in the input video signal through compression. Compression helps reduce the bandwidth or storage space requirements mentioned above, in some cases by two orders of magnitude or more. Typically, lossless compression and lossy compression and combinations thereof can be used. Lossless compression refers to a technique that can reconstruct an exact copy of the original signal from the compressed original signal. When lossy compression is used, the reconstructed signal may be different from the original signal, but the distortion between the original signal and the reconstructed signal is small, so the reconstructed signal can achieve the desired purpose. Lossy compression is widely used in the video field. The amount of distortion allowed varies from application to application. For example, users of consumer live broadcast applications can tolerate higher distortion than users of TV program applications. The achievable compression ratio can reflect that the higher the allowable / tolerable distortion, the higher the compression ratio that can be produced.

[0007] Motion compensation may be a lossy compression technique and may be associated with the following technique: a block of sample data from a previously reconstructed image or part thereof (reference image) may be used to predict a newly reconstructed image or part of an image after being spatially shifted in a direction indicated by a motion vector (MV). In some cases, the reference image may be the same as the image currently being reconstructed. Each MV may have two dimensions, X and Y, or three dimensions, the third dimension indicating the reference image being used (indirectly, the third dimension may also be a temporal dimension).

[0008] In some video compression techniques, the MV applicable to a certain sample data region can be predicted based on other MVs, for example, the other MVs are related to another sample data region that is spatially adjacent to the region being reconstructed and whose decoding order is before the MV of the certain sample data region. In this way, the amount of data required to encode the MV can be greatly reduced, thereby eliminating redundancy and improving compression. For example, when encoding the input video signal from a camera (called the original video), there is a statistical possibility that multiple regions larger than a single MV region will move in a similar direction, so in some cases, similar motion vectors extracted from MVs of adjacent regions can be used for prediction, so MV prediction is very effective. In this way, the MV determined for a given region is similar or identical to the MV predicted based on the surrounding MVs, and after entropy coding, the number of bits representing the MV is less than the number of bits used when the MV is directly encoded. In some cases, MV prediction can be an embodiment of lossless compression of a signal (i.e., MV) extracted from the original signal (i.e., sample stream). In other cases, MV prediction itself may be lossy, for example, lossy due to rounding errors when calculating predicted values ​​based on several surrounding MVs.

[0009] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T H.265 Recommendation, "High Efficiency Video Coding", December 2016). Among the various MV prediction mechanisms provided by H.265, this application describes a technique referred to below as "spatial merging".

[0010] Referring to FIG. 1 , the current block (101) includes samples that have been discovered by the encoder during the motion search process, and the samples can be predicted based on the previous block of the same size that has generated a spatial offset. In addition, the MV can be derived from metadata associated with one or more reference images instead of encoding the MV directly. For example, the MV is derived from the metadata of the nearest reference image (in decoding order) using the MV associated with any of the five surrounding samples A0, A1 and B0, B1, B2 (corresponding to 102 to 106 respectively). In H.265, MV prediction can use the prediction value of the same reference image that the neighboring blocks are also using. Summary of the invention

[0011] Embodiments of the present disclosure provide a method for a video decoder to perform video decoding. A merged shared area may be received, the merged shared area including a plurality of coding blocks. A shared merge candidate list is constructed for the merged shared area. The merged shared area is decoded based on the shared merge candidate list. At least one coding block using inter-frame coding (hereinafter referred to as the inter-frame coding block) in the merged shared area is processed, and a history-based motion vector prediction (HMVP) table is not updated using motion information of the at least one inter-frame coding block.

[0012] In one embodiment, all inter-frame coding blocks in the merged shared area are processed, and the motion information of any inter-frame coding block is not used to update the HMVP table. In one embodiment, one or more coding blocks in the merged shared area that are inter-coded in merge mode or skip mode are processed, and the HMVP table is not updated using the motion information of the one or more coding blocks. In one embodiment, one or more of the following coding blocks in the merged shared area are processed, and the HMVP table is not updated with the motion information of the one or more coding blocks: (i) coding blocks that are inter-coded using the motion information of the merge candidates in the shared merge candidate list, or (ii) coding blocks that are inter-coded using the motion information determined based on the merge candidates in the shared merge candidate list. In one embodiment, one or more coding blocks in the merged shared area that are encoded according to the shared merge candidate list are processed, and the HMVP table is not updated with the motion information of the one or more coding blocks.

[0013] In one embodiment, one or more inter-frame coding blocks in the merged shared area are processed. If the inter-frame coding of the inter-frame coding block uses a merge candidate in the shared merge candidate list as the motion information of the one or more coding blocks, the HMVP table is not updated using the motion information of the one or more coding blocks.

[0014] In one embodiment, the HMVP table is updated using the motion information of the first inter-frame coding block in the merged shared area in the decoding order, and when other inter-frame coding blocks in the merged shared area are processed, the HMVP table is not updated using the motion information of the other inter-frame coding blocks. In one embodiment, the HMVP buffer is updated using the motion information of the last inter-frame coding block in the merged shared area in the decoding order, and when other inter-frame coding blocks in the merged shared area are processed, the HMVP buffer is not updated using the motion information of other inter-frame coding blocks.

[0015] Embodiments of the present disclosure provide a device for video decoding. The device may include a circuit for receiving a merged shared area, the merged shared area including a plurality of coding blocks. The circuit may be further used to construct a shared merge candidate list for the merged shared area, and decode the merged shared area based on the shared merge candidate list. At least one inter-frame coding block in the merged shared area is processed, and the motion information of the at least one inter-frame coding block is not used to update the HMVP table.

[0016] Each embodiment of the present disclosure further provides a non-volatile computer-readable medium storing instructions, which can be executed by a computer for video decoding so that the computer executes the video decoding method of this document. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to make the features, principles and various advantages of the disclosed subject matter more clear, the following detailed description of the specification and the accompanying drawings is given. In the accompanying drawings:

[0018] FIG. 1 is a schematic diagram of a current block and its surrounding spatial merging candidates according to an embodiment.

[0019] Figure 2 is a simplified schematic block diagram of a communication system of an embodiment.

[0020] Figure 3 is a simplified schematic block diagram of a communication system of an embodiment.

[0021] Figure 4 is a simplified schematic block diagram of a decoder of an embodiment.

[0022] Figure 5 is a simplified schematic block diagram of an encoder of an embodiment.

[0023] Figure 6 is a schematic block diagram of an encoder according to another embodiment.

[0024] Figure 7 is a schematic block diagram of a decoder according to another embodiment.

[0025] Figure 8 The diagram is a schematic diagram of merging subsequent positions of a merging mode according to an embodiment.

[0026] Fig. 9 The diagram is a schematic diagram of spatial neighboring blocks and temporal neighboring blocks of a current block according to an embodiment.

[0027] Fig. 10A is a schematic diagram of spatial neighboring blocks of an embodiment, and prediction information of a current block can be determined according to motion information of the spatial neighboring blocks using a sub-block-based temporal motion vector prediction method.

[0028] Fig. 10B The diagram is a schematic diagram of spatial neighboring blocks selected for a sub-block-based temporal motion vector prediction method according to an embodiment.

[0029] Fig.11A is a flowchart of a process of creating and updating a candidate motion information list using a history-based motion vector prediction method according to an embodiment.

[0030] Fig. 11B The present invention is a schematic diagram of updating candidate prediction information using a history-based motion vector prediction method according to an embodiment.

[0031] Fig.12 The present invention is a schematic diagram of determining starting points of two reference images associated with two reference image lists based on motion vectors of merge candidates in a merge with motion vector difference (MMVD) mode according to an embodiment.

[0032] Fig.13 FIG. 4 is a schematic diagram of predetermined points around two starting points to be evaluated in an MMVD mode according to an embodiment.

[0033] Figures 14A-14B Some examples of merging shared areas of an embodiment are shown.

[0034] Fig.15 is a list of inter prediction modes that may be used by some embodiments to encode or decode sub-blocks in a merged shared region.

[0035] Fig.16 is a general flow chart of a video decoding process for decoding a merged shared region in some embodiments.

[0036] Fig.17 is a schematic diagram of a computer system of an embodiment. DETAILED DESCRIPTION

[0037] I. Encoders and decoders for video encoding and decoding

[0038] Figure 2 2 is a simplified block diagram of a communication system (200) according to an embodiment disclosed in the present application. The communication system (200) includes a plurality of terminal devices, which can communicate with each other through, for example, a network (250). For example, the communication system (200) includes a first terminal device (210) and a second terminal device (220) interconnected through the network (250). Figure 2 In an embodiment of the present invention, the first terminal device (210) and the second terminal device (220) perform unidirectional data transmission. For example, the first terminal device (210) may encode video data (e.g., a video image stream captured by the terminal device (210)) for transmission to the second terminal device (220) via the network (250). The encoded video data is transmitted in the form of one or more encoded video code streams. The second terminal device (220) may receive the encoded video data from the network (250), decode the encoded video data to restore the video data, and display the video image based on the restored video data. Unidirectional data transmission is more common in applications such as media services.

[0039] In another embodiment, the communication system (200) includes a third terminal device (230) and a fourth terminal device (240) that perform bidirectional transmission of encoded video data, which can occur, for example, during a video conference. For bidirectional data transmission, each of the third terminal device (230) and the fourth terminal device (240) can encode video data (e.g., a video image stream collected by the terminal device) to transmit to the other terminal device of the third terminal device (230) and the fourth terminal device (240) through a network (250). Each of the third terminal device (230) and the fourth terminal device (240) can also receive the encoded video data transmitted by the other terminal device of the third terminal device (230) and the fourth terminal device (240), and can decode the encoded video data to restore the video data, and can display the video image on an accessible display device based on the restored video data.

[0040] exist Figure 2In the embodiment of the present invention, the first terminal device (210), the second terminal device (220), the third terminal device (230) and the fourth terminal device (240) may be servers, personal computers and smart phones, but the principles disclosed in the present application may not be limited thereto. The embodiments disclosed in the present application are applicable to laptop computers, tablet computers, media players and / or dedicated video conferencing equipment. The network (250) represents any number of networks that transmit encoded video data between the first terminal device (210), the second terminal device (220), the third terminal device (230) and the fourth terminal device (240), including, for example, wired (wired) and / or wireless communication networks. The communication network (250) may exchange data in circuit switching and / or packet switching channels. The network may include a telecommunications network, a local area network, a wide area network and / or the Internet. For the purpose of the present application, unless explained below, the architecture and topology of the network (250) may be irrelevant to the operation disclosed in the present application.

[0041] As an example, Figure 3 The video encoder and video decoder are shown in a streaming environment. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0042] The streaming system may include an acquisition subsystem (313), which may include a video source (301) such as a digital camera, which creates an uncompressed video image stream (302). In an embodiment, the video image stream (302) includes samples captured by a digital camera. Compared to the encoded video data (304) (or the encoded video code stream), the video image stream (302) is depicted as a thick line to emphasize the high data volume of the video image stream, and the video image stream (302) can be processed by an electronic device (320), which includes a video encoder (303) coupled to the video source (301). The video encoder (303) may include hardware, software, or a combination of hardware and software to implement or implement various aspects of the disclosed subject matter as described in more detail below. Compared to the video image stream (302), the encoded video data (304) (or the encoded video code stream (304)) is depicted as a thin line to emphasize the lower amount of data of the encoded video data (304) (or the encoded video code stream (304)), which can be stored on the streaming server (305) for future use. One or more streaming client subsystems, such as Figure 3The client subsystem (306) and the client subsystem (308) in the streaming server (305) can access the streaming server (305) to retrieve the copy (307) and the copy (309) of the encoded video data (304). The client subsystem (306) may include, for example, a video decoder (310) in the electronic device (330). The video decoder (310) decodes the incoming copy (307) of the encoded video data and generates an output video image stream (311) that can be presented on a display (312) (e.g., a display screen) or another presentation device (not depicted). In some streaming systems, the encoded video data (304), the video data (307), and the video data (309) (e.g., a video bitstream) can be encoded according to certain video encoding / compression standards. Examples of these standards include ITU-T H.265. In an embodiment, the video coding standard under development is informally referred to as Versatile Video Coding (VVC), and the present application can be used in the context of the VVC standard.

[0043] It should be noted that the electronic device (320) and the electronic device (330) may include other components (not shown). For example, the electronic device (320) may include a video decoder (not shown), and the electronic device (330) may also include a video encoder (not shown).

[0044] Figure 4 4 is a block diagram of a video decoder (410) according to an embodiment disclosed in the present application. The video decoder (410) may be provided in an electronic device (430). The electronic device (430) may include a receiver (431) (e.g., a receiving circuit). The video decoder (410) may be used to replace Figure 3 A video decoder (310) of an embodiment.

[0045] The receiver (431) may receive one or more encoded video sequences to be decoded by the video decoder (410); in the same or another embodiment, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of the other encoded video sequences. The encoded video sequence may be received from a channel (401), which may be a hardware / software link to a storage device storing the encoded video data. The receiver (431) may receive the encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams that may be forwarded to their respective consuming entities (not shown). The receiver (431) may separate the encoded video sequence from the other data. To prevent network jitter, a buffer memory (415) may be coupled between the receiver (431) and the entropy decoder / parser (420) (hereinafter referred to as "parser (420)"). In some applications, the buffer memory (415) is part of the video decoder (410). In other cases, the buffer memory (415) may be provided external to the video decoder (410) (not shown). In other cases, a buffer memory (not shown) is provided outside the video decoder (410) to prevent network jitter, for example, and another buffer memory (415) may be configured inside the video decoder (410) to handle broadcast timing, for example. When the receiver (431) receives data from a storage / forward device with sufficient bandwidth and controllability or from an isochronous network, it may not be necessary to configure the buffer memory (415), or the buffer memory may be made smaller. Of course, in order to use on a service packet network such as the Internet, a buffer memory (415) may also be required, and the buffer memory may be relatively large and have an adaptive size, and may be at least partially implemented in an operating system or a similar element (not shown) outside the video decoder (410).

[0046] The video decoder (410) may include a parser (420) to reconstruct symbols (421) from the encoded video sequence. The types of symbols include information used to manage the operation of the video decoder (410) and potential information used to control a display device such as a display device (412) (e.g., a display screen) that is not part of the electronic device (430) but can be coupled to the electronic device (430), such as Figure 4As shown in . The control information for the display device may be a parameter set fragment (not indicated) of Supplemental Enhancement Information (SEI message) or Video Usability Information (VUI). The parser (420) may parse / entropy decode the received coded video sequence. The encoding of the coded video sequence may be performed according to a video coding technique or standard, and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and the like. The parser (420) may extract a subgroup parameter set of at least one subgroup of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a Group of Pictures (GOP), a picture, a tile, a slice, a macroblock, a Coding Unit (CU), a block, a Transform Unit (TU), a Prediction Unit (PU), and the like. The parser (420) may also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[0047] The parser (420) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (415), thereby creating symbols (421).

[0048] Depending on the type of coded video picture or part of coded video picture (e.g., inter-frame and intra-frame pictures, inter-frame blocks and intra-frame blocks) and other factors, the reconstruction of the symbol (421) may involve multiple different units. Which units are involved and how they are involved can be controlled by subgroup control information parsed by the parser (420) from the coded video sequence. For the sake of brevity, such subgroup control information flow between the parser (420) and the multiple units below is not described.

[0049] In addition to the functional blocks already mentioned, the video decoder (410) can be conceptually subdivided into several functional units as described below. In a practical embodiment operating under commercial constraints, many of these units interact closely with each other and can be integrated with each other. However, for the purpose of describing the disclosed subject matter, the conceptual subdivision into the following functional units is appropriate.

[0050] The first unit is a sealer / inverse transform unit (451). The sealer / inverse transform unit (451) receives quantized transform coefficients as symbols (421) from the parser (420) and control information, including which transform method to use, block size, quantization factor, quantization scaling matrix, etc. The sealer / inverse transform unit (451) can output a block including sample values, which can be input into an aggregator (455).

[0051] In some cases, the output samples of the scaler / inverse transform unit (451) may belong to an intra-coded block; that is, a block that does not use predictive information from a previously reconstructed image, but may use predictive information from a previously reconstructed portion of a current image. Such predictive information may be provided by an intra-image prediction unit (452). In some cases, the intra-image prediction unit (452) generates surrounding blocks of the same size and shape as the block being reconstructed using reconstructed information extracted from a current image buffer (458). For example, the current image buffer (458) buffers a partially reconstructed current image and / or a fully reconstructed current image. In some cases, the aggregator (455) adds the prediction information generated by the intra-frame prediction unit (452) to the output sample information provided by the scaler / inverse transform unit (451) on a per-sample basis.

[0052] In other cases, the output samples of the scaler / inverse transform unit (451) may belong to an inter-frame coded and potentially motion compensated block. In this case, the motion compensated prediction unit (453) may access the reference image memory (457) to extract samples for prediction. After the extracted samples are motion compensated according to the symbols (421), these samples may be added to the output of the scaler / inverse transform unit (451) (in this case referred to as residual samples or residual signals) by the aggregator (455) to generate output sample information. The acquisition of the predicted samples by the motion compensated prediction unit (453) from the address in the reference image memory (457) may be controlled by a motion vector, and the motion vector is provided to the motion compensated prediction unit (453) in the form of the symbols (421), for example, including X, Y and reference image components. Motion compensation may also include interpolation of sample values ​​extracted from the reference image memory (457) when using sub-sample accurate motion vectors, motion vector prediction mechanisms, etc.

[0053] The output samples of the aggregator (455) may be used by various loop filtering techniques in a loop filter unit (456). The video compression techniques may include in-loop filter techniques that are controlled by parameters included in the encoded video sequence (also referred to as the encoded video bitstream) and that are available to the loop filter unit (456) as symbols (421) from the parser (420). However, in other embodiments, the video compression techniques may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of the encoded image or encoded video sequence, as well as to previously reconstructed and loop filtered sample values.

[0054] The output of the loop filter unit (456) may be a sample stream that may be output to a display device (412) and stored in a reference picture memory (457) for subsequent inter-picture prediction.

[0055] Once fully reconstructed, certain coded pictures may be used as reference pictures for future prediction. For example, once the coded picture corresponding to the current picture is fully reconstructed, and the coded picture is identified as a reference picture (e.g., by the parser (420)), the current picture buffer (458) may become part of the reference picture memory (457), and a new current picture buffer may be reallocated before beginning reconstruction of a subsequent coded picture.

[0056] The video decoder (410) may perform decoding operations according to a predetermined video compression technique, such as in the ITU-T H.265 standard. The encoded video sequence may conform to the syntax specified by the video compression technique or standard used in the sense that the encoded video sequence follows the syntax of the video compression technique or standard and the profile recorded in the video compression technique or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technique or standard as the only tools available for use under the profile. For compliance, the complexity of the encoded video sequence is also required to be within the range defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum image size, the maximum frame rate, the maximum reconstruction sampling rate (measured in, for example, mega samples per second), the maximum reference image size, etc. In some cases, the limits set by the hierarchy may be further limited by the Hypothetical Reference Decoder (HRD) specification and metadata of the HRD buffer management signaled in the encoded video sequence.

[0057] In an embodiment, the receiver (431) may receive additional (redundant) data along with the encoded video. The additional data may be part of the encoded video sequence. The additional data may be used by the video decoder (410) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0058] Figure 5 1 is a block diagram of a video encoder (503) according to an embodiment disclosed in the present application. The video encoder (503) is provided in an electronic device (520). The electronic device (520) includes a transmitter (540) (e.g., a transmission circuit). The video encoder (503) can be used to replace Figure 3 A video encoder (303) in an embodiment.

[0059] The video encoder (503) can be used to obtain the video source (501) (not Figure 5 In another embodiment, the video source (501) is a part of the electronic device (520) to receive video samples, and the video source can collect video images to be encoded by the video encoder (503). In another embodiment, the video source (501) is a part of the electronic device (520).

[0060] The video source (501) may provide a source video sequence in the form of a digital video sample stream to be encoded by the video encoder (503), wherein the digital video sample stream may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the video source (501) may be a storage device storing previously prepared videos. In a video conferencing system, the video source (501) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of separate images that are given motion when viewed sequentially. The image itself may be constructed as a spatial pixel array, wherein each pixel may include one or more samples depending on the sampling structure, color space, etc. used. The relationship between pixels and samples may be easily understood by those skilled in the art. The following description focuses on samples.

[0061] According to an embodiment, the video encoder (503) may encode and compress the images of the source video sequence into an encoded video sequence (543) in real time or under any other time constraints required by the application. Implementing the appropriate encoding speed is a function of the controller (550). In some embodiments, the controller (550) controls other functional units as described below and is functionally coupled to these units. For the sake of simplicity, the coupling is not marked in the figure. The parameters set by the controller (550) may include rate control related parameters (image skipping, quantizer, lambda value of rate distortion optimization technology, etc.), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (550) can be used to have other suitable functions that are related to the video encoder (503) optimized for a certain system design.

[0062] In some embodiments, the video encoder (503) operates in an encoding loop. As a simple description, in embodiments, the encoding loop may include a source encoder (530) (e.g., responsible for creating symbols, such as a symbol stream, based on an input image to be encoded and a reference image) and a (local) decoder (533) embedded in the video encoder (503). The decoder (533) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data (because in the video compression technology considered in this application, any compression between the symbols and the encoded video code stream is lossless). The reconstructed sample stream (sample data) is input to a reference image memory (534). Since the decoding of the symbol stream produces bit-accurate results that are independent of the decoder location (local or remote), the contents of the reference image memory (534) are also bit-accurate between the local encoder and the remote encoder. In other words, the reference image samples "seen" by the prediction part of the encoder are exactly the same as the sample values ​​that the decoder will "see" when using the prediction during decoding. This basic principle of reference image synchronicity (and the drift that occurs when synchronicity cannot be maintained, eg due to channel errors) is also used in some related techniques.

[0063] The operation of the "local" decoder (533) may be combined with, for example, Figure 4 The "remote" decoder described in detail for the video decoder (410) is identical. However, additional brief reference is made to Figure 4 , when symbols are available and the entropy encoder (545) and parser (420) are capable of losslessly encoding / decoding the symbols into an encoded video sequence, the entropy decoding portion of the video decoder (410), including the buffer memory (415) and the parser (420), may not be fully implemented in the local decoder (533).

[0064] At this point, it can be observed that any decoder technology other than the parsing / entropy decoding present in the decoder must also be present in the corresponding encoder in substantially the same functional form. For this reason, the present application focuses on decoder operation. The description of encoder technology can be simplified because encoder technology is mutually inverse to the decoder technology described comprehensively. A more detailed description is only needed in certain areas and is provided below.

[0065] During operation, in some embodiments, the source encoder (530) may perform motion compensated predictive coding. The motion compensated predictive coding predictively encodes an input image with reference to one or more previously encoded images from a video sequence designated as "reference images." In this manner, the encoding engine (532) encodes the differences between pixel blocks of the input image and pixel blocks of a reference image that may be selected as a prediction reference for the input image.

[0066] The local video decoder (533) may decode the encoded video data of the picture that may be designated as the reference picture based on the symbol created by the source encoder (530). The operation of the encoding engine (532) may be a lossy process. When the encoded video data is available at the video decoder ( Figure 5 When the video sequence is decoded at a remote location (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (533) replicates the decoding process that may be performed by the video decoder on the reference picture and may cause the reconstructed reference picture to be stored in the reference picture cache (534). In this way, the video encoder (503) may store a copy of the reconstructed reference picture locally that has common content (absent transmission errors) with the reconstructed reference picture to be obtained by the remote video decoder.

[0067] The predictor (535) may perform a prediction search for the encoding engine (532). That is, for a new image to be encoded, the predictor (535) may search the reference image memory (534) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that may serve as appropriate prediction references for the new image. The predictor (535) may operate on a pixel-by-pixel-block basis based on sample blocks to find a suitable prediction reference. In some cases, based on the search results obtained by the predictor (535), it may be determined that the input image may have prediction references taken from a plurality of reference images stored in the reference image memory (534).

[0068] The controller (550) may manage encoding operations of the source encoder (530), including, for example, setting parameters and subgroup parameters for encoding video data.

[0069] The outputs of all the above functional units may be entropy encoded in an entropy encoder (545). The entropy encoder (545) performs lossless compression on the symbols generated by the various functional units according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc., thereby converting the symbols into a coded video sequence.

[0070] The transmitter (540) may buffer the encoded video sequence created by the entropy encoder (545) in preparation for transmission over a communication channel (560), which may be a hardware / software link to a storage device where the encoded video data will be stored. The transmitter (540) may combine the encoded video data from the video encoder (503) with other data to be transmitted, such as encoded audio data and / or ancillary data streams (source not shown).

[0071] The controller (550) may manage the operation of the video encoder (503). During encoding, the controller (550) may assign a certain coded picture type to each coded picture, but this may affect the coding techniques that can be applied to the corresponding picture. For example, a picture may generally be assigned to any of the following picture types:

[0072] An intra picture (I picture) may be a picture that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow for different types of intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of the variations of I pictures and their corresponding applications and features.

[0073] Predictive pictures (P pictures), which may be pictures that may be encoded and decoded using intra prediction or inter prediction, which uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0074] Bidirectional predictive pictures (B pictures), which can be pictures that can be encoded and decoded using intra prediction or inter prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata for reconstruction of a single block.

[0075] The source picture may typically be spatially subdivided into blocks of samples (e.g., blocks of 4×4, 8×8, 4×8 or 16×16 samples) and coded block by block. These blocks may be predictively coded with reference to other (already coded) blocks, which are determined according to the coding allocation applied to the block's corresponding picture. For example, blocks of an I picture may be non-predictively coded, or they may be predictively coded (spatial prediction or intra prediction) with reference to already coded blocks of the same picture. Blocks of pixels of a P picture may be predictively coded by spatial prediction with reference to one previously coded reference picture or by temporal prediction. Blocks of a B picture may be predictively coded by spatial prediction with reference to one or two previously coded reference pictures or by temporal prediction.

[0076] The video encoder (503) may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (503) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in an input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard used.

[0077] In an embodiment, the transmitter (540) may transmit additional data when transmitting the encoded video. The source encoder (530) may include such data as part of the encoded video sequence. The additional data may include time / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0078] The captured video may be taken as a plurality of source images (video images) in a temporal sequence. Intra-image prediction (often simplified to intra-frame prediction) exploits spatial correlations in a given image, while inter-image prediction exploits (temporal or other) correlations between images. In an embodiment, a particular image being encoded / decoded is divided into blocks, and the particular image being encoded / decoded is referred to as the current image. When a block in the current image is similar to a reference block in a reference image that was previously encoded in the video and is still buffered, the block in the current image may be encoded by a vector called a motion vector. The motion vector points to a reference block in a reference image, and in the case where multiple reference images are used, the motion vector may have a third dimension that identifies the reference image.

[0079] In some embodiments, bidirectional prediction techniques may be used in inter-frame image prediction. According to the bidirectional prediction technique, two reference images are used, for example, a first reference image and a second reference image, both of which are before the current image in the video in decoding order (but may be in the past and future in display order, respectively). A block in the current image may be encoded by a first motion vector pointing to a first reference block in the first reference image and a second motion vector pointing to a second reference block in the second reference image. Specifically, the block may be predicted by a combination of the first reference block and the second reference block.

[0080] In addition, merge mode technology can be used in inter-picture prediction to improve coding efficiency.

[0081] According to some embodiments disclosed in the present application, predictions such as inter-frame image prediction and intra-frame image prediction are performed in units of blocks. For example, according to the HEVC standard, an image in a video image sequence is divided into coding tree units (CTUs) for compression, and the CTUs in the image have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU includes three coding tree blocks (CTBs), which are a luminance CTB and two chrominance CTBs. Furthermore, each CTU can be divided into one or more coding units (CUs) using a quadtree. For example, a 64×64 pixel CTU can be divided into a 64×64 pixel CU, or 4 32×32 pixel CUs, or 16 16×16 pixel CUs. In an embodiment, each CU is analyzed to determine the prediction type for the CU, such as an inter-frame prediction type or an intra-frame prediction type. In addition, depending on temporal and / or spatial predictability, the CU is divided into one or more prediction units (PUs). Typically, each PU includes a luminance prediction block (PB) and two chrominance PBs. In an embodiment, the prediction operation in encoding (encoding / decoding) is performed in units of prediction blocks. Taking the luminance prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luminance values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, and the like.

[0082] Figure 6 1 is a diagram of a video encoder (603) according to another embodiment of the present disclosure. The video encoder (603) is configured to receive a processed block (e.g., a prediction block) of sample values ​​within a current video image in a video image sequence and encode the processed block into an encoded image that is part of an encoded video sequence. In this embodiment, the video encoder (603) is configured to replace Figure 3A video encoder (303) in an embodiment.

[0083] In an HEVC embodiment, a video encoder (603) receives a matrix of sample values ​​for a processing block, such as a prediction block of 8×8 samples. The video encoder (603) uses, for example, rate-distortion (RD) optimization to determine whether to use intra mode, inter mode, or bidirectional prediction mode to encode the processing block. When encoding the processing block in intra mode, the video encoder (603) may use intra prediction techniques to encode the processing block into an encoded image; and when encoding the processing block in inter mode or bidirectional prediction mode, the video encoder (603) may use inter prediction or bidirectional prediction techniques to encode the processing block into an encoded image, respectively. In some video encoding techniques, the merge mode may be an inter-image prediction submode, in which a motion vector is derived from one or more motion vector predictors without the aid of an encoded motion vector component external to the predictor. In some other video encoding techniques, there may be a motion vector component applicable to the subject block. In an embodiment, the video encoder (603) includes other components, such as a mode decision module (not shown) for determining the processing block mode.

[0084] exist Figure 6 In an embodiment of the present invention, the video encoder (603) includes: Figure 6 Shown are an inter-frame encoder (630), an intra-frame encoder (622), a residual calculator (623), a switch (626), a residual encoder (624), a common controller (621), and an entropy encoder (625) coupled together.

[0085] The inter-frame encoder (630) is used to receive samples of a current block (e.g., a processing block), compare the block with one or more reference blocks in a reference image (e.g., blocks in a previous image and a subsequent image), generate inter-frame prediction information (e.g., redundant information description according to an inter-frame coding technique, motion vectors, merge mode information), and calculate an inter-frame prediction result (e.g., a predicted block) based on the inter-frame prediction information using any suitable technique. In some embodiments, the reference image is a decoded reference image decoded based on the encoded video information.

[0086] The intra-frame encoder (622) is used to receive samples of a current block (e.g., a processing block), compare the block with an encoded block in the same image in some cases, generate quantization coefficients after transformation, and in some cases also generate intra-frame prediction information (e.g., intra-frame prediction direction information according to one or more intra-frame coding techniques). In an embodiment, the intra-frame encoder (622) also calculates an intra-frame prediction result (e.g., a predicted block) based on the intra-frame prediction information and a reference block in the same image.

[0087] The general controller (621) is used to determine the general control data and control other components of the video encoder (603) based on the general control data. In an embodiment, the general controller (621) determines the mode of the block and provides a control signal to the switch (626) based on the mode. For example, when the mode is intra mode, the general controller (621) controls the switch (626) to select the intra mode result for use by the residual calculator (623) and controls the entropy encoder (625) to select the intra prediction information and add the intra prediction information to the bitstream; and when the mode is inter mode, the general controller (621) controls the switch (626) to select the inter prediction result for use by the residual calculator (623) and controls the entropy encoder (625) to select the inter prediction information and add the inter prediction information to the bitstream.

[0088] The residual calculator (623) is used to calculate the difference (residual data) between the received block and the prediction result selected from the intra-frame encoder (622) or the inter-frame encoder (630). The residual encoder (624) is used to operate based on the residual data to encode the residual data to generate a transform coefficient. In an embodiment, the residual encoder (624) is used to convert the residual data from the time domain to the frequency domain and generate a transform coefficient. The transform coefficient is then processed by quantization to obtain a quantized transform coefficient. In various embodiments, the video encoder (603) also includes a residual decoder (628). The residual decoder (628) is used to perform an inverse transform and generate decoded residual data. The decoded residual data can be used appropriately by the intra-frame encoder (622) and the inter-frame encoder (630). For example, the inter-frame encoder (630) can generate a decoded block based on the decoded residual data and the inter-frame prediction information, and the intra-frame encoder (622) can generate a decoded block based on the decoded residual data and the intra-frame prediction information. The decoded blocks are appropriately processed to generate a decoded picture, and in some embodiments, the decoded picture may be buffered in a memory circuit (not shown) and used as a reference picture.

[0089] The entropy encoder (625) is used to format the code stream to produce an encoded block. The entropy encoder (625) generates various information according to a suitable standard such as the HEVC standard. In an embodiment, the entropy encoder (625) is used to obtain general control data, selected prediction information (such as intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the code stream. It should be noted that according to the disclosed subject matter, when the block is encoded in the inter-frame mode or the merge sub-mode of the bidirectional prediction mode, there is no residual information.

[0090] Figure 7FIG. 7 is a diagram of a video decoder (710) according to another embodiment disclosed in the present application. The video decoder (710) is used to receive an encoded image as part of an encoded video sequence, and decode the encoded image to generate a reconstructed image. In an embodiment, the video decoder (710) is used to replace Figure 3 A video decoder (310) of an embodiment.

[0091] exist Figure 7 In one embodiment, the video decoder (710) includes: Figure 7 An entropy decoder (771), an inter-frame decoder (780), a residual decoder (773), a reconstruction module (774) and an intra-frame decoder (772) coupled together are shown in FIG.

[0092] The entropy decoder (771) can be used to reconstruct certain symbols from the encoded image, which represent syntax elements that constitute the encoded image. Such symbols may include, for example, a mode for encoding the block (e.g., intra-frame mode, inter-frame mode, bidirectional prediction mode, a merged sub-mode of the latter two, or another sub-mode), prediction information (e.g., intra-frame prediction information or inter-frame prediction information) that can identify certain samples or metadata for prediction by the intra-frame decoder (772) or the inter-frame decoder (780), respectively, residual information in the form of, for example, quantized transform coefficients, etc. In an embodiment, when the prediction mode is inter-frame or bidirectional prediction mode, the inter-frame prediction information is provided to the inter-frame decoder (780); and when the prediction type is an intra-frame prediction type, the intra-frame prediction information is provided to the intra-frame decoder (772). The residual information can be inverse quantized and provided to the residual decoder (773).

[0093] The inter-frame decoder (780) is used for receiving the inter-frame prediction information and generating an inter-frame prediction result based on the inter-frame prediction information.

[0094] The intra-frame decoder (772) is used to receive intra-frame prediction information and generate a prediction result based on the intra-frame prediction information.

[0095] The residual decoder (773) is used to perform inverse quantization to extract dequantized transform coefficients, and process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (773) may also require some control information (to obtain the quantizer parameter QP), and this information can be provided by the entropy decoder (771) (the data path is not indicated because this is only low-volume control information).

[0096] The reconstruction module (774) is used to combine the residual output by the residual decoder (773) with the prediction result (which can be output by the inter-frame prediction module or the intra-frame prediction module) in the spatial domain to form a reconstructed block, which can be part of the reconstructed image, which in turn can be part of the reconstructed video. It should be noted that other suitable operations such as deblocking operations can be performed to improve visual quality.

[0097] It should be noted that the video encoder (303), video encoder (503) and video encoder (603) and video decoder (310), video decoder (410) and video decoder (710) may be implemented using any suitable technology. In an embodiment, the video encoder (303), video encoder (503) and video encoder (603) and video decoder (310), video decoder (410) and video decoder (710) may be implemented using one or more integrated circuits. In another embodiment, the video encoder (303), video encoder (503) and video encoder (603) and video decoder (310), video decoder (410) and video decoder (710) may be implemented using one or more processors that execute software instructions.

[0098] II. Inter-frame prediction mode

[0099] In various embodiments, an image may be segmented into a plurality of blocks, for example, using a tree-based segmentation scheme. The plurality of blocks obtained by segmentation may be processed in different processing modes, for example, an intra-frame prediction mode, an inter-frame prediction mode (e.g., a merge mode, a skip mode, an advanced motion vector prediction (AVMP) mode), etc. An intra-frame coded block may be a block coded in an intra-frame prediction mode. Accordingly, an inter-frame coded block may be a block processed in an inter-frame prediction mode.

[0100] Examples of inter-frame prediction modes can be divided into two categories: (i) AMVP mode and (ii) merge / skip mode. Examples of merge / skip mode may include HEVC merge mode (or regular merge mode), current reference picture (CRP) mode (or intra block copy (IBC)) mode, triangulation mode, subblock-based temporal motion vector prediction (sbTMVP) mode, merge with motion vector difference (MMVD) mode, and affine merge mode. In some embodiments, AMVP mode and merge / skip mode can be combined with a history-based motion vector prediction (HMVP) table for encoding or decoding a coding block.

[0101] An example of inter prediction mode and use of the HMVP table are described below.

[0102] 1. Merge mode

[0103] When processing the block currently being processed (also referred to as the current block) in merge mode, a neighboring block can be selected from the spatial neighborhood or the temporal neighborhood of the current block. The current block can be merged with the selected neighboring block by sharing the same set of motion data (or motion information) with the selected neighboring block. This merge mode operation can be performed on a group of neighboring blocks so that the regions of these neighboring blocks can be merged together and share the same set of motion data. During transmission from the encoder to the decoder, an index indicating the motion data of the selected neighboring block can be transmitted for the current block instead of transmitting the entire set of motion data. In this way, the amount of data (bits) used to transmit motion information can be reduced, and the encoding and decoding efficiency can be improved.

[0104] In the above embodiments, the neighboring blocks providing motion data can be selected from a set of candidate positions. Multiple candidate positions related to the current block can be predefined. For example, the multiple candidate positions may include multiple spatial candidate positions and multiple temporal candidate positions. Each spatial candidate position is associated with a spatial neighboring block adjacent to the current block. Each temporal candidate position is associated with a temporal neighboring block located in another encoded image (e.g., a previously encoded image). The neighboring blocks overlapping with the candidate positions (referred to as candidate blocks) are a subset of all spatial neighboring blocks or temporal neighboring blocks of the current block. In this way, the candidate blocks can be evaluated to select the blocks to be merged, rather than using the entire set of neighboring blocks.

[0105] Figure 8Schematic diagram of candidate positions of an embodiment. From these candidate positions, a set of merge candidate positions can be selected to construct a merge candidate list. As shown in the figure, the current block (810) will be processed in merge mode. A set of candidate positions {A1, B1, B0, A0, B2, C0, C1} are defined for merge mode processing. Specifically, the candidate positions {A1, B1, B0, A0, B2} are spatial candidate positions, which represent the positions of candidate blocks in the same image as the current block (810). Relatively speaking, the candidate positions {C0, C1} are temporal candidate positions, which represent the positions of candidate blocks in another encoded image, and these positions are adjacent to or overlap with the same-position blocks of the current block (810). As shown in the figure, the candidate position C1 can be close to (e.g., adjacent to) the center of the current block (810).

[0106] In various embodiments, the candidate positions may be represented by sample blocks or samples. Figure 8 In , each candidate position is represented by a sample block, and the size of the sample block can be, for example, 4x4 samples. The size of such a sample block corresponding to the candidate position can be equal to or less than the minimum size of the allowed PB defined in the tree-based partitioning scheme used to generate the current block (810), for example, 4×4 samples. In this configuration, the block corresponding to the candidate position is always located within the coverage of a single adjacent PB. In another embodiment, a sample position (for example, the lower right sample in block A1, or the upper right sample in block A0) can be used to represent the candidate position. Such a sample is called a representative sample, and such a position is called a representative position.

[0107] In one embodiment, based on Figure 8 A merge mode process may be performed to select merge candidates from the candidate positions {A1, B1, B0, A0, B2, C0, C1} defined in , thereby constructing a candidate list. The candidate list may have a predefined maximum number of merge candidates, which is denoted by Cm. Each merge candidate in the candidate list may include a set of motion data that can be used for motion compensated prediction.

[0108] Merge candidates may be added to the candidate list in a certain order. For example, different merge candidates may have different probabilities of being selected, depending on how the merge candidates are derived. Merge candidates with a higher probability of being selected are located before merge candidates with a lower probability of being selected. Based on this order, each merge candidate is associated with an index (referred to as a merge index). In one embodiment, a merge candidate with a higher probability of being selected will have a smaller index value, so that the encoding and decoding of the corresponding index requires fewer bits.

[0109] In one embodiment, the merge candidate motion data may include horizontal motion vector displacement values ​​and vertical motion vector displacement values ​​of one or two motion vectors, one or two reference image indexes associated with the one or two motion vectors, and optionally, an identifier of a reference image list associated with the reference image index.

[0110] In one embodiment, a first number of merge candidates, Ca, are derived from the spatial candidate positions according to a predefined order, namely {A1, B1, B0, A0, B2}, and a second number of merge candidates, Cb=Cm-Ca, are derived from the temporal candidate positions, namely {C0, C1}. The numbers A1, B1, B0, A0, B2, C0, C1 used to represent the candidate positions may also be used to refer to the merge candidates. For example, the merge candidate obtained from the candidate position A1 may be referred to as the merge candidate A1.

[0111] In some cases, a merge candidate at a candidate position may be unavailable. For example, a candidate block at a candidate position may be intra-predicted to be located outside a strip or tile including the current block (810), or located in a different row from the current block (810) in a coding tree block (CTB). In some scenarios, a merge candidate at a candidate position may be redundant. For example, a neighboring block of the current block (810) may cover two candidate positions. Redundant merge candidates may be deleted from the candidate list (e.g., by performing a pruning process). When the total number of available merge candidates in the candidate list (after removing redundant candidates) is less than the maximum number of merge candidates Cm, additional merge candidates may be generated (e.g., according to a preconfigured rule) to fill the candidate list so that the candidate list can maintain a fixed length. For example, additional merge candidates may include combined dual prediction candidates and zero motion vector candidates.

[0112] After constructing the candidate list, an evaluation process may be performed at the encoder to select a merge candidate from the candidate list. For example, the rate-distortion performance corresponding to each merge candidate may be calculated, and a merge candidate with the best RD performance may be selected. Accordingly, a merge index associated with the selected merge candidate may be determined for the current block (810), and the merge index may be signaled to notify the decoder.

[0113] At the decoder, a merge index for the current block (810) may be received. As described above, a similar candidate list construction process may be performed to generate a candidate list identical to the candidate list generated at the encoder side. In some embodiments, after the candidate list is constructed, a merge candidate may be selected from the candidate list based on the received merge index without performing any further evaluation. The motion data of the selected merge candidate may be used for subsequent motion compensated prediction of the current block (810).

[0114] A skip mode is also used in some embodiments. For example, in skip mode, the current block can be predicted using the merge mode as described above to determine a set of motion data, however, no residual is generated in the process and the transform coefficients are not transmitted. A skip flag can be associated with the current block. The skip flag and the merge index indicating the relevant motion information of the current block can be signaled to the video decoder. For example, at the beginning of a CU in an inter-image prediction strip, a skip flag can be signaled, which implies the following: the CU contains only one PU (2Nx2N); the motion data is derived using the merge mode; and there is no residual data in the bitstream. On the decoder side, based on the skip flag, the prediction block can be determined based on the merge index used to decode the corresponding current block without adding residual information. Therefore, the various methods of video encoding and decoding with merge mode disclosed herein can be utilized in combination with the skip mode.

[0115] For example, in one embodiment, when the corresponding signal of the merge flag or skip flag in the codestream is true, a merge index is signaled to indicate which candidate in the merge candidate list is used to provide the motion vector of the current block. Up to four spatially adjacent motion vectors and up to one temporally adjacent motion vector may be added to the merge candidate list. The syntax element MaxMergeCandsNum is defined as the size of the merge candidate list. The syntax element MaxMergeVandsNum may be signaled in the codestream.

[0116] 2. Affine merge mode

[0117] Fig. 9 Schematic diagram of the spatial neighboring blocks and temporal neighboring blocks of the current block (or coding unit (CU)) (901) of an embodiment. As shown in the figure, the spatial neighboring blocks are represented as A0, A1, A2, B0, B1, B2 and B3 (902, 903, 907, 904, 905, 906 and 908 respectively), and the temporal neighboring block is represented as C0 (912). In some embodiments, the spatial neighboring blocks A0, A1, A2, B0, B1, B2 and B3 and the current block (901) belong to the same image. In some embodiments, the temporal neighboring block C0 belongs to a reference image and corresponds to a position outside the current block (901) and adjacent to the lower right corner of the current block (901).

[0118] In some embodiments, an affine model (e.g., a 6-parameter affine model or a 4-parameter affine model) may be used to derive the motion vector of the current block (901) and / or the motion vector of a sub-block of the current block. In some embodiments, the affine model has 6 parameters (e.g., a 6-parameter affine model) for describing the motion vector of the block. For example, the 6 parameters of the affine-encoded block may be obtained by three different positions of the block (e.g., Fig. 9 In another embodiment, the simplified affine model uses four parameters to describe the motion information of the affine coded block, which can be represented by two different positions of the block (for example, Fig. 9 It is represented by two motion vectors (also called two CPMVs) at the control points CP0 and CP1 at the upper left and upper right corners of the image.

[0119] An affine merge mode may be used to construct a motion information candidate list (also referred to as an affine merge candidate list) based on motion information of one or more spatially neighboring blocks and / or temporally neighboring blocks. In some embodiments, the affine merge mode may be applied when the width and height of the current block (901) are equal to or greater than 8 samples. According to the affine merge mode, the CPMV of the current block (901) may be determined based on the candidate motion information in the list. In some embodiments, the motion information candidate list may include up to five candidate CPMVs, and an index may be signaled to indicate which candidate CPMV is used by the current block.

[0120] In some embodiments, the affine merge candidate list may have three types of candidate CPVMs, including inherited affine candidates, constructed affine candidates, and zero MVs. Inherited affine candidates may be derived from the CPMVs of neighboring blocks by extrapolation. The constructed affine candidates may be derived by translating the MVs of neighboring blocks.

[0121] For example, there may be at most two inherited affine candidates, derived from the corresponding affine motion models of the neighboring blocks, including one block from the left neighboring blocks (A0 and A1) and one block from the upper neighboring blocks (B0, B1, and B2). For the candidate from the left, the neighboring blocks A0 and A1 may be checked in sequence, and the first available inherited affine candidate from the neighboring blocks A0 and A1 may be used as the affine candidate inherited from the left. For the candidate from the upper side, the neighboring blocks B0, B1, and B2 may be checked in sequence, and the first available inherited affine candidate from the neighboring blocks B0, B1, and B2 may be used as the affine candidate inherited from the upper side. In some embodiments, no pruning check is performed between two inherited affine candidates.

[0122] When a neighboring affine block is identified, the corresponding inherited affine candidate to be added to the affine merge list of the current block (901) can be derived from the control point motion vectors of the neighboring affine block. Fig. 9 In the example, if the adjacent block A1 is encoded using the affine mode, the upper left corner of the block A1 (control point CP0) can be obtained. A1 ), upper right corner (control point CP1A1 ) and the lower left corner (control point CP2 A1 ) motion vector. When block A1 is encoded using a 4-parameter affine model, it can be encoded based on the control point CP0 A1 and control point CP1 A1 The motion vector of the current block (901) is used to calculate the two CPMVs of the inherited affine candidate. When block A1 is encoded using a 6-parameter affine model, the control point CP0 A1 、Control point CP1 A1 and control point CP2 A1 The motion vector of the current block (901) is used to calculate the three CPMVs of the inherited affine candidates.

[0123] Furthermore, the constructed affine candidates can be derived by combining the neighboring translation motion information of each control point. The motion information of control points CP0, CP1, and CP2 is derived from the designated spatial neighboring blocks A0, A1, A2, B0, B1, B2, and B3.

[0124] For example, CPMV k (k=1, 2, 3, 4) represents the motion vector of the kth control point, where CPMV1 corresponds to control point CP0, CPMV2 corresponds to control point CP1, CPMV3 corresponds to control point CP2, and CPMV4 corresponds to the temporal control point of the temporally adjacent block C0. For CPMV1, the adjacent blocks B2, B3, and A2 can be checked in sequence, and the first available motion vector from the adjacent blocks B2, B3, and A2 is used as CPMV1. For CPMV2, the adjacent blocks B1 and B0 can be checked in sequence, and the first available motion vector from the adjacent blocks B1 and B0 is used as CPMV2. For CPMV3, the adjacent blocks A1 and A0 can be checked in sequence, and the first available motion vector from the adjacent blocks A1 and A0 is used as CPMV3. In addition, if available, the motion vector of the temporally adjacent block C0 can be used as CPMV4.

[0125] After obtaining the four control points CP0, CP1, CP2 and the temporal control points CPMV1, CPMV2, CPMV3 and CPMV4, an affine merge candidate list can be constructed to include affine merge candidates constructed in the following order: {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, and {CPMV1, CPMV3}. A combination of three CPMVs can form a 6-parameter affine merge candidate, and any combination of two CPMVs can form a 4-parameter affine merge candidate. In some embodiments, in order to avoid the motion scaling process, if the reference indexes of a set of control points are different, the corresponding CPMV combination can be discarded.

[0126] 3. Subblock-Based Temporal Motion Vector Prediction (SbTMVP) mode

[0127] Fig. 10A The present invention is a schematic diagram of a spatial neighboring block of an embodiment, which can be used to determine the predicted motion information of the current block (1011) based on the motion information of the spatial neighboring blocks using a sub-block based temporal MV prediction (SbTMVP) method. Fig. 10A A current block (1011) and its spatial neighboring blocks are shown, the spatial neighboring blocks being denoted as A0, A1, B0 and B1 (1012, 1013, 1014 and 1015 respectively). In some embodiments, the spatial neighboring blocks A0, A1, B0 and B1 and the current block (1011) belong to the same image.

[0128] Fig. 10B 1 is a schematic diagram of an embodiment of using the SbTMVP method to determine motion information of a sub-block of a current block (1011) based on a selected spatial neighboring block (e.g., block A1 in this non-limiting embodiment). In this embodiment, the current block (1011) is in a current image (1010), and the reference block (1061) is in a reference image (1060), which can be determined by a motion shift (or displacement) between the current block (1011) and the reference block (1061) indicated by a motion vector (1022).

[0129] In some embodiments, similar to the temporal motion vector prediction (TMVP) in HEVC, SbTMVP uses motion information in various reference sub-blocks in a reference image for a current block in a current image. In some embodiments, SbTVMP uses the same reference image as TMVP. In some embodiments, TMVP predicts motion information at the CU level, while SbTMVP predicts motion at the sub-CU level. In some embodiments, TMVP uses a temporal motion vector from a co-located block in a reference image, which corresponds to a position adjacent to the lower right corner or center of the current block, and SbTMVP uses a temporal motion vector from a reference block, which can be determined by performing a motion shift based on the motion vector of one of the spatial neighboring blocks of the current block.

[0130] For example, Fig. 10A As shown, neighboring blocks A1, B1, B0, and A0 can be checked sequentially during the SbTVMP process. Once a first spatial neighboring block having a motion vector that uses the reference image (1060) as its reference image is identified, such as block A1 having a motion vector (1022) pointing to reference block AR1 in the reference image (1060), the motion vector (1022) can be used to perform motion shifting. If no such motion vector is identified from the spatial neighboring blocks A1, B1, B0, and A0, the motion shift can be set to (0, 0).

[0131] After the motion shift is determined, a reference block (1061) may be determined based on the position of the current block (1011) and the determined motion shift. Fig. 10B In the present invention, the reference block (1061) can be further divided into 16 sub-blocks with reference motion information MRa to MRp. In some embodiments, the reference motion information of each sub-block in the reference block (1061) can be determined based on the minimum motion grid covering the center sample of the sub-block. The motion information may include a motion vector and a corresponding reference index. The current block (1011) can be further divided into 16 sub-blocks, and the motion information MVa to MVp of the sub-blocks in the current block (1011) can be derived from the reference motion information MRa to MRp in a manner similar to the TMVP process, and time scaling can be used in some embodiments.

[0132] The sub-block size used in the SbTMVP process may be fixed (or predetermined) or signaled. In some embodiments, the sub-block size used in the SbTMVP process may be 8×8 samples. In some embodiments, the SbTMVP process is only applicable to blocks whose width and height are equal to or greater than a fixed or signaled size (e.g., 8 pixels).

[0133] In one embodiment, a sub-block based merge list containing a combination of SbTMVP candidates and affine merge candidates is used to signal the sub-block based merge mode. The SbTMVP mode can be enabled or disabled by a sequence parameter set (SPS) flag. In some embodiments, when the SbTMVP mode is enabled, the SbTMVP candidate is added as the first entry in the sub-block based merge candidate list, followed by the affine merge candidates. In some embodiments, the maximum allowed sub-block based merge list size is set to 5. However, other sizes may be used in other embodiments.

[0134] In some embodiments, the encoding logic of the additional SbTMVP merge candidates is the same as other merge candidates. That is, for each block in a P slice or a B slice, an additional rate-distortion check may be performed to determine whether to use the SbTMVP candidate.

[0135] 4. History-Based Motion Vector Prediction (HMVP) mode

[0136] Fig.11A is a flow chart of an overview of a method (1100) for constructing and updating a motion information candidate list using a history-based MV prediction (HMVP) method in one embodiment.

[0137] In some embodiments, a motion information candidate list using the HMVP method may be constructed and updated during the encoding or decoding process. The list is also referred to as a history list. The history list may be stored in the form of an HMVP table or an HMVP buffer. When a new strip begins, the history list may be cleared. In some embodiments, whenever there is a non-affine block of inter-frame coding that has just been encoded or decoded, the associated motion information may be added as the last item in the history list as a new candidate HMVP. Therefore, before processing (encoding or decoding) the current block, a history list with candidate HMVPs may be loaded (S1112). The current block may be encoded or decoded using the candidate HMVPs in the history list (S1114). Then, the history list may be updated using the motion information used to encode or decode the current block (S1116).

[0138] Fig. 11B The diagram is a schematic diagram of updating a candidate motion information list using the HMVP method according to an embodiment. Fig. 11B A history list of size L is shown, and each candidate in the list can be identified by an index ranging from 0 to L-1. L is an integer equal to or greater than 0. Before encoding or decoding the current block, the history list (1120) includes L candidates HMVP0, HMVP1, HMVP2, ... HMVPm 、...、HMVP L-2 and HMVP L-1 , where m is an integer from 0 to L. After encoding or decoding the current block, the new entry HMVP C Add to history list.

[0139] In one embodiment, the size of the history list may be set to 6, indicating that up to 6 HMVP candidates may be added to the history list. C ), a constrained first-in-first-out (FIFO) rule can be used, wherein a redundancy check is first performed to determine whether there is a redundant HMVP in the history list. When no redundant HMVP is found, the first candidate HMVP is removed from the list ( Fig. 11B HMVP1 in the embodiment, whose index = 0), and then move the other candidate HMVPs forward, for example, reduce their indexes by 1. New candidate HMVP C Can be added as the last item in a list (e.g. Fig. 11B = L-1), as shown in the generated list (1130). On the other hand, if a redundant HMVP is found (e.g. Fig. 11B HMVP2) in the embodiment, the redundant HMVP in the history list is removed from the list, and all subsequent HMVP candidates are moved forward, for example, their indexes are reduced by 1. New candidate HMVP C Can be added as the last item in a list (e.g. Fig. 11B Index = L-1), as shown in the generated list (1140).

[0140] In some embodiments, HMVP candidates may be used to construct a merge candidate list. For example, the nearest few HMVP candidates in the list are sequentially checked and inserted after the TMVP candidates in the candidate list. In some embodiments, HMVP candidates may be pruned for spatial or temporal merge candidates instead of pruning sub-block motion candidates (i.e., SbTMVP candidates).

[0141] In some embodiments, in order to reduce the number of pruning operations, one or more of the following rules may be followed:

[0142] (a) The number of HMPV candidates to be checked, denoted by M, is set as follows:

[0143] M=(N<=4)? L:(8-N),

[0144] Wherein, N represents the number of available non-sub-block merge candidates, and L represents the number of available HMVP candidates in the history list.

[0145] (b) In addition, once the total number of available merge candidates is only one less than the maximum number of merge candidates indicated by the signal, the merge candidate list construction process according to the HMVP list can be terminated.

[0146] (c) In addition, the number of bi-predictive merging candidates used to derive the combination is reduced from 12 pairs to 6 pairs.

[0147] In some embodiments, HMVP candidates can be used in the AMVP candidate list construction process. The motion vectors of the last K HMVP candidates in the history list can be added after the TMVP candidates in the AMVP candidate list. In some embodiments, only HMVP candidates with the same reference image as the AMVP target reference image are added to the AMVP candidate list. The HMVP candidates are pruned. In some embodiments, K is set to 4, and the AMVP list size remains unchanged, for example, equal to 2.

[0148] 5. Pairwise Average Motion Vector Candidates

[0149] In some embodiments, pairwise average candidates can be generated by averaging predefined candidate pairs in the current merge candidate list. For example, in one embodiment, the predefined pairs can be defined as {(0, 1), (0, 2), (1, 2), (0, 3), (1, 3), (2, 3)}, where the number represents the merge index of the merge candidate list. For example, the average motion vector can be calculated separately for each reference image list. If two motion vectors to be averaged are available in one list, they can be averaged even when the two motion vectors point to different reference images. If only one motion vector is available, this one motion vector can be used directly. If no motion vector is available, in one embodiment, the corresponding pair can be skipped. In some embodiments, when constructing the merge candidate list, the combined candidate can be replaced by the pairwise average candidate.

[0150] 6. Merge with Motion Vector Difference (MMVD) mode

[0151] In some embodiments, a motion vector differential merge (MMVD) mode is used to determine a motion vector predictor for the current block. The MMVD mode can be used when the skip mode or merge mode is enabled. The MMVD mode reuses the merge candidates on the merge candidate list of the skip mode or merge mode. For example, the merge candidate selected from the merge candidate list can be used to provide a starting point at a reference image. The motion vector of the current block can be represented by a starting point and a motion offset, and the motion offset includes a motion amplitude and a motion direction relative to the starting point. On the encoder side, the selection of the merge candidate and the determination of the motion offset can be based on a search process (evaluation process). On the decoder side, the selected merge candidate and motion offset can be determined based on signaling from the encoder side.

[0152] The MMVD mode can reuse the merge candidate list constructed with the various inter-frame prediction modes described herein. In some embodiments, only candidates of the default merge type (e.g., MRG_TYPE_DEFAULT_N) in the merge candidate list are considered for the MMVD mode. Examples of merge candidates of the default merge type may include: (i) merge candidates used in the merge mode, (ii) merge candidates from the history buffer in the HMVP mode, and (iii) pairwise average motion vector candidates as described herein. In some embodiments, merge candidates in the affine mode or SbTMVP mode are not used for extension in the MMVD mode.

[0153] The base candidate index (IDX) can be used to define the starting point. For example, Table 1 shows a list of merge candidates (motion vector predictors (MVP)) associated with indices from 0 to 3. The merge candidate corresponding to the index of one of the base candidate indices can be determined from the list and used to provide a starting point.

[0154] Table 1. Basic candidate IDX

[0155]

[0156]

[0157] The distance index can be used to provide motion magnitude information. For example, multiple predefined pixel distances are shown in Table 2, each predefined pixel distance is associated with an index from 0 to 7. The pixel distance corresponding to the index having one of the distance indices can be determined from the multiple pixel distances and used to provide the motion magnitude.

[0158] Table 2. Distance IDX

[0159]

[0160] The direction index can be used to provide motion direction information. For example, four directions with indices from 00 to 11 (binary) are shown in Table 3. The direction corresponding to the index of one of the direction indices can be determined from the four directions and used to provide the direction of the motion offset relative to the starting point.

[0161] Table 3. Direction IDX

[0162] Directions to IDX 00 01 10 11 x-axis + - N / A N / A y-axis N / A N / A + -

[0163] The MMVD syntax elements may be transmitted in the codestream to signal a set of MMVD indexes in the MMVD mode, including basic candidate indexes, direction indexes, and distance IDX.

[0164] In some embodiments, the MMVD enable flag is signaled after the skip and merge flags for encoding and decoding the current block are sent. For example, when the skip and merge flags are true, the MMVD flag is parsed. In one embodiment, when the MMVD flag is equal to 1, the MMVD syntax element (the above-mentioned set of MMVD indexes) is parsed. In one embodiment, when the MMVD flag is not 1, a flag associated with another mode is parsed, such as an affine flag. When the affine flag is equal to 1, the current block is processed using the affine mode. In one embodiment, when the affine flag is not 1, the skip / merge index is parsed, thereby processing the current block with the skip / merge mode.

[0165] Fig.12 and Fig.13 This is an example of a search process in MMVD mode according to an embodiment of the present disclosure. By performing the search process, a set of MMVD indexes including basic candidate indexes, direction indexes and distance indexes can be determined for a current block (1201) in a current image (or current frame).

[0166] Fig.12 and Fig.13 A first motion vector (1211) and a second motion vector (1221) belonging to a first merge candidate are shown. The first merge candidate may be a merge candidate in a merge candidate list constructed for the current block (1201). The first motion vector (1211) and the second motion vector (1221) may be associated with two reference images (1202) and (1203) in reference image lists L0 and L1, respectively. Thus, the first merge candidate may be determined in reference images (1202) and (1203). Fig.13 There are two starting points (1311) and (1321) in .

[0167] In one embodiment, based on the starting points (1311) and (1321), a plurality of predefined points extending from the starting points (1311) and (1321) in the vertical direction (indicated by +Y or -Y) or the horizontal direction (indicated by +X and -X) in the reference images (1202) and (1203) may be evaluated. In one embodiment, a pair of points, such as a pair of points (1314) and (1324), or a pair of points (1315) and (1325), that are mirror images of each other with respect to the respective starting points (1311) or (1321) may be used to determine a pair of motion vectors that may form motion vector predictor candidates for the current block (1201). Those motion vector predictor candidates determined based on the predefined points around the starting points (1311) or (1321) may be evaluated.

[0168] In addition to the first merge candidate, other available or valid merge candidates in the merge candidate list of the current block (1201) may also be evaluated in a similar manner. In one embodiment, for a merge candidate of unidirectional prediction, only one prediction direction associated with one of the two reference picture lists is evaluated.

[0169] Based on the above evaluation, the best motion vector predictor candidate can be determined. Therefore, corresponding to the best motion vector predictor candidate, the best merge candidate can be selected from the merge list, and the motion direction and motion distance can also be determined. For example, based on the selected merge candidate and Table 1, the basic candidate index can be determined. Based on the selected motion vector predictor, such as the motion vector predictor corresponding to the predefined point (1315) (or (1325)), the direction and distance of the point (1315) relative to the starting point (1311) can be determined. According to Tables 2 and 3, the direction index and distance index can be determined accordingly.

[0170] It should be noted that the embodiments described above are only for illustrative purposes. In other embodiments, based on the motion vector expression method provided by the MMVD mode, the motion distance and motion direction can be defined in different ways. In addition, the evaluation process (search process) can be performed in different ways. For example, for a bidirectional prediction merge candidate, three types of prediction directions (e.g., L0, L1, and L0 and L1) can be evaluated based on a set of predefined distances and directions to select the best motion vector predictor. In another embodiment, the unidirectional prediction merge candidate can be converted into a bidirectional prediction merge candidate by mirroring or scaling, and then evaluated. In the above embodiments, additional syntax elements can be represented by signals to indicate the prediction direction (e.g., L0, L1, or L0 and L1) generated by the evaluation process.

[0171] As described above, the merge candidates in the merge candidate list are evaluated at the encoder to determine the basic candidates for the MMVD mode. At the decoder, the motion vector predictor can be selected from the merge candidate list using the basic candidate index as input. Therefore, the MMVD mode does not require an additional line buffer in addition to the line buffer for storing merge candidates.

[0172] III. Merged shared regions with shared merge lists

[0173] In some embodiments, when the size of the current block and / or the size of the sub-blocks within the current block meet certain conditions, a merge sharing region with a shared merge list is used. For example, multiple sub-CUs of a parent node in a CU partition tree can share a merge candidate list. In this way, it is possible to avoid constructing a merge candidate list separately for each sub-CU, which enables parallel processing of multiple CUs of small merge / skip coding within a merge sharing region. The parent node is referred to as a merge sharing node or a merge sharing region. The merge sharing node can also be treated as one of the sub-CUs, and a shared merge candidate list can be generated for the merge sharing node.

[0174] In the decoding parsing stage, it can be determined for each node inside the CTU whether the split-tree node can be regarded as a merged shared node. In one embodiment, when the parent node of the sub-CU meets the following criteria, the parent node can be determined as a merged shared node:

[0175] (a) The block size of the parent node is equal to or greater than a size threshold (e.g., 32 pixels, 64 pixels, etc.);

[0176] (b) Within the parent node, the size of a sub-CU is smaller than the size threshold.

[0177] Fig.14A An example of a merged shared region (1401)-(1404) defined based on a size threshold of 64 pixels is shown. The block size of each region in the merged shared region (1401)-(1404) is 64 pixels, which is equal to the size threshold. Each region in the merged shared region (1401)-(1404) includes at least one sub-CU whose size is smaller than the size threshold.

[0178] Fig. 14B Another example of a merged shared region (1410) defined based on a size threshold of 64 pixels is shown. The merged shared region (1410) includes three sub-blocks (1411)-(1413) having sizes of 32, 64, and 32 pixels, respectively. The merged shared region (1410) has a size of 128 pixels, which is greater than the size threshold, and includes two sub-blocks (1411) and (1413) having sizes less than the size threshold.

[0179] In other embodiments, the merged shared area may be determined based on different definitions. For example, the parent node of the sub-CU whose block size is smaller than the threshold may be determined as the merged shared area. Fig. 14B For example, when 64 pixels are used as the threshold, any one of the blocks 1411-1413 may be determined as a merged shared area.

[0180] In addition, in some embodiments, the merged shared node does not have any samples outside the image boundary. In the parsing stage, if the parent node meets the definition of the merged shared area, but some samples are outside the image boundary, the parent node is not used as a merged shared node. Next, the sub-CUs in the node can be evaluated to determine the merged shared area.

[0181] The shared merge candidate list algorithm can support translational merge / skip modes, such as normal merge mode, CRP mode (or IBC mode), triangulation mode, sbTMVP mode, MMVD mode, etc. In some embodiments, the affine merge mode is not used to encode blocks in the merged shared area. When these merge / skip modes are used in the merged shared area, they can be merged with the HMVP table. For these merge / skip modes, the operation of the shared merge candidate list algorithm is similar to that based on the regular merge candidate list. The shared merge candidate list algorithm only generates candidates for merged shared nodes and treats the merged shared nodes as sub-CUs.

[0182] Note that a sub-CU within a merge sharing region may be encoded using a coding mode other than merge / skip mode. For example, a CU within a merge sharing node may be encoded using intra mode or an inter mode other than merge / skip mode (e.g., AMVP mode).

[0183] IV. HMVP Table Update for CUs in Merged Shared Regions

[0184] In some embodiments, a merged shared area scheme is used to parallel process sub-blocks within the merged shared area. In addition, the merged shared area scheme can be combined with the HMVP scheme to improve inter-frame coding performance. For example, when a merged shared area is identified in the parsing stage of the decoder, a shared merge list can be constructed for the merged shared area. The construction of the merged shared area can include HMVP candidates from the HMVP table. The sub-blocks (or coding blocks) within the merged shared area can be encoded and decoded using an intra-frame prediction mode (intra-frame mode) or an inter-frame prediction mode (inter-frame mode). For each inter-frame coded sub-block, the corresponding motion information can be used to update the HMVP table.

[0185] However, according to some embodiments of the present disclosure, it may be unnecessary to perform an HMVP table update for each inter-coded sub-block (or coding block) within a merged shared area. For example, the sub-blocks within the merged shared area are close to each other. There is a high probability that adjacent inter-coded blocks have similar motion information. In addition, these sub-blocks within the merged shared area share the same merge list. As a result of the merge mode decoding process, different inter-coded sub-blocks are likely to use the same merge candidate as their motion information. Therefore, updating the HMVP table based on this similar motion information would be redundant. Instead, it may be more appropriate to perform an HMVP table update on a subset of those inter-coded blocks, and the computational cost associated with the HMVP table update may be saved.

[0186] Therefore, in some embodiments, when the merged shared area method and the HMVP method are combined, the update process of the HMVP table is constrained. For example, the HMVP table update is not performed every time an inter-frame coded sub-block is decoded, but only a subset of the inter-frame coded sub-blocks in the merged shared area triggers the update operation of the HMVP table. In this way, the cost of the HMVP table update can be avoided or reduced, and the encoding quality can be maintained.

[0187] In one embodiment, no HMVP table update is performed on inter-coded sub-blocks within the merged shared region.

[0188] In some embodiments, whether merging inter-coded sub-blocks within a shared region triggers an update of the HMVP table depends on which type of inter-prediction mode is used to encode / decode the corresponding sub-blocks.

[0189] Fig.15 A table (1500) of inter-frame prediction modes (inter-frame modes) that can be used within a merged shared region is shown. When one of these inter-frame modes is used to encode / decode a sub-block within a merged shared region, motion information can be generated. The motion information can be used to update the HMVP table. The inter-frame modes in table (1500) can be divided into two categories: merge / skip mode and AMVP mode.

[0190] In AMVP mode, an AMVP candidate list can be constructed to encode a sub-block. The AMVP candidate list may include a set of AMVP candidates selected from adjacent blocks of the sub-block. In one embodiment, an HMVP table can be used to construct the AMVP mode. Motion estimation can be performed to determine the motion information of the sub-block. The difference between the selected AMVP candidate and the motion information obtained by motion estimation can be signaled from the encoder to the decoder together with the index of the AMVP candidate selected from the AMVP candidate list. When the sub-block is decoded at the decoder, the original motion information is restored based on the difference and index represented by the signal. In some embodiments, the AMVP candidate list is constructed separately using the shared merge list corresponding to the sub-block. In some embodiments, the AMVP candidate list may be a subset of the corresponding shared merge list. In some embodiments, the corresponding shared merge list can be used as the AMVP candidate list.

[0191] The merge / skip modes in Table (1500) may include two groups of coding modes: Group I and Group II. Group I includes an inter-frame mode that uses merge candidates in a shared merge list as motion information for encoding or decoding sub-blocks (sub-CUs) within a merge shared area. Examples of inter-frame modes in Group I may include a regular merge mode, a CRP mode, and the like. When a sub-block within a merge shared area is encoded using the Group I inter-frame mode, a candidate in the shared merge list may be selected to provide motion information to determine prediction information for the sub-block. Subsequently, the difference (residual) between the prediction information and the sub-block may be encoded and sent. When the skip mode is used, no residual signal is generated or sent.

[0192] Group II includes an inter-frame mode for encoding or decoding sub-blocks within a merged shared area using motion information determined based on a merge candidate in a shared merge list. Examples of inter-frame modes in Group II may include a triangular partitioning mode, an sbTMVP mode, an MMVD mode, and the like. When a sub-block within a merged shared area with a shared merge list is encoded using a Group II inter-frame mode, motion information may be constructed based on the merge candidates in the shared merge list (e.g., a triangular partitioning mode). Alternatively, the merge candidates in the shared merge list may be used to determine a target block, and motion information of sub-blocks within the target block may be obtained to encode sub-blocks within the merged shared area (e.g., an sbTMVP mode). Alternatively, the merge candidates in the shared merge list may be used to provide a starting point on a reference image so as to search for motion information at a position adjacent to the starting point (e.g., MMVD). In the above case, the generated motion information may be different from the merge candidates in the shared merge list.

[0193] According to the classification of table (1500), in one embodiment, when the sub-blocks in the merged shared area are inter-coded using the merge / skip mode, the HMVP table is not updated after the sub-blocks are decoded. On the contrary, when the sub-blocks in the merged shared area are inter-coded using a coding mode other than the merge / skip mode such as the AMVP mode, the motion information obtained by decoding the sub-blocks is used to update the HMVP table.

[0194] As described herein, when the merge / skip mode is used to encode or decode a sub-block within a merge shared region, the merge / skip mode is based on a shared merge list. The shared merge list may include motion information that has recently been included in the HMVP table. The motion information generated by the merge / skip mode may be the same or similar to the merge candidates in the shared merge list, and therefore has a high probability of being included in the HMVP table. Therefore, updating the HMVP table with this information may be less efficient.

[0195] In contrast, in AMVP mode, the motion difference is determined and encoded. Thus, the motion information generated by AMVP mode may be different from the AMVP candidates in the AMVP candidate list (the AMVP candidate list may be similar to or different from the shared merge list). Therefore, it is unlikely that the motion information is already included in the HMVP table. It may be advantageous to update the HMVP table with this motion information.

[0196] In another embodiment, when the sub-block in the merged shared region is inter-coded using the Group I mode (e.g., the normal merge mode or the CRP mode), the motion information of the sub-frame is not used to update the HMVP table. On the contrary, when the sub-block in the merged shared region is inter-coded using an inter-frame mode other than the Group I mode, such as the Group II mode or the AMVP mode, the HMVP table is updated using the motion information obtained by decoding the sub-block.

[0197] As described herein, when a sub-block within a merged shared region is encoded or decoded using a group II inter-frame mode, the group II inter-frame mode determines motion information of the sub-block based on a merge candidate in a shared merge list, but the determined motion information may be different from the merge candidate in the shared merge list and may not be included in the HVMP table. Therefore, in some cases, updating the HMVP table using motion information obtained from the group II mode may achieve some coding gain.

[0198] In some embodiments, a sub-block is selected from a group of inter-coded sub-blocks in the merged shared area for updating the HMVP table. For example, there are multiple inter-coded sub-blocks, for example, sub-blocks that are encoded and decoded using the inter-frame mode in table (1500) in a coding / decoding order (e.g., raster scan order, zigzag order, etc.). The first or last sub-block can be selected to update the HMVP table according to the coding and decoding order. For other inter-coded sub-blocks, the HMVP table update is not performed. In this way, the update cost of the HMVP table can be reduced.

[0199] In various embodiments, for inter-frame coding blocks that are not in the merged shared area, the HMVP table may be updated using motion information corresponding to these inter-frame coding blocks.

[0200] Fig.16 An overview flow chart of the method (1600) of an embodiment of the present disclosure is shown. The method (1600) can be used to decode a merged shared area based on an HMVP table. In various embodiments, the method (1600) is performed by a processing circuit, such as a processing circuit in a terminal device (210), (220), (230) and (240), a processing circuit that performs the functions of a video decoder (310), a processing circuit that performs the functions of a video decoder (410), etc. In some embodiments, the method (1600) is implemented by software instructions, so when the processing circuit executes the software instructions, the processing circuit executes the method (1600). The method starts from (S1601) and proceeds to (S1610).

[0201] At (S1610), a merged shared region including a plurality of coding blocks is received. For example, the decoder may receive a bit stream of coding blocks obtained by image segmentation. In the parsing stage, the merged shared region may be determined based on a size threshold.

[0202] At (S1620), a shared merge candidate list may be constructed for decoding the merged shared region. The HMVP candidates in the HMVP table may be added to the shared merge candidate list. The shared merge candidate list may include other types of merge candidates, such as spatial merge candidates, temporal merge candidates, pair-wise average bi-prediction candidates, etc.

[0203] At (S1630), the merged shared area can be decoded based on the shared merge candidate list. For example, the coding blocks in the merged shared area can be inter-frame encoded or intra-frame encoded. For inter-frame coded blocks, motion information can be determined during the decoding process. However, in order to reduce the update cost of the HMVP table, not all inter-frame coded blocks will trigger the HMVP update operation.

[0204] In one embodiment, the HMVP update operation is not performed on the coding blocks in the merged shared area. In one embodiment, whether to update the HMVP table with the motion information of the coding blocks in the merged shared area may depend on the type of inter-frame prediction mode for encoding and decoding each coding block. In one embodiment, a coding block (for example, the first or last coding block in the encoding and decoding order) can be selected from the inter-frame coding blocks in the merged shared area, and its motion information can be used to update the HMVP table. The method (1600) can proceed to (S1699) and terminate at (S1699).

[0205] The above techniques can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. Fig.17 A computer system (1700) suitable for implementing some embodiments of the present application.

[0206] Computer software may be encoded using any suitable machine code or computer language, and may be generated using assembly, compilation, linking or similar mechanisms. These instruction codes may be directly executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or executed through operations such as code interpretation, microcode execution, etc.

[0207] These instructions may be executed in a variety of types of computers or computer components, including, for example, personal computers, tablets, servers, smart phones, gaming devices, Internet of Things devices, etc.

[0208] Fig.17 The components shown for the computer system (1700) are exemplary in nature and are not intended to limit the scope of use or functionality of computer software implementing the embodiments of the present application. The configuration of the components should not be interpreted as having any dependency or requirement on any component or combination of components in the exemplary embodiment of the computer system (1700).

[0209] The computer system (1700) may include certain human-computer interface input devices. Such human-computer interface input devices may be responsive to input from one or more human users through, for example, tactile input (such as keystrokes, swipes, data glove movements), audio input (such as voice, taps), visual input (such as gestures), olfactory input (not shown). Human-computer interface devices may also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (such as voice, music, ambient sounds), images (such as scanned images, photographic images obtained from a still image camera), and videos (such as two-dimensional video, including three-dimensional video including stereoscopic video).

[0210] The human-computer interface input device may include one or more of the following items (only one of each is depicted): keyboard (1701), mouse (1702), touchpad (1703), touch screen (1710), data gloves (not shown), joystick (1705), microphone (1706), scanner (1707), camera (1708).

[0211] The computer system (1700) may also include certain human-computer interface output devices. Such human-computer interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such human-computer interface output devices may include tactile output devices (e.g., tactile feedback through a touch screen (1710), a data glove (not shown), or a joystick (1705), but there may also be tactile feedback devices that are not used as input devices), audio output devices (such as speakers (1709), headphones (not shown)), visual output devices, and printers (not shown), wherein visual output devices such as screens (1710), virtual reality glasses (not shown), holographic displays, and smoke boxes (not shown), screens (1710) include cathode ray tube (CRT) screens, liquid crystal display (LCD) screens, plasma screens, organic light emitting diode (OLED) screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities, and some of these screens are capable of outputting two-dimensional visual outputs or more than three-dimensional outputs by means (such as stereo image output).

[0212] The computer system (1700) may also include human-accessible storage devices and their associated media, such as optical media (including CD / DVD ROM / RW (1720) with CD / DVD) or similar media (1721), thumb drives (1722), removable hard drives or solid-state drives (1723), traditional magnetic media (such as tapes and floppy disks (not shown)), dedicated ROM / ASIC / PLD-based devices (such as secure dongles (not shown)), and the like.

[0213] Those skilled in the art will also appreciate that the term "computer-readable media" used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0214] The computer system (1700) may also include an interface to connect one or more communication networks. The network may be, for example, a wireless network, a wired network, an optical network. The network may also be a local area network, a wide area network, a metropolitan area network, a vehicle network and an industrial network, a real-time network, a delay-tolerant network, and the like. Examples of networks include local area networks (such as Ethernet, wireless LAN), cellular networks (including global mobile communication systems (GSM), third generation mobile communication systems (3G), fourth generation mobile communication systems (4G), fifth generation mobile communication systems (5G), long term evolution (LTE), etc.), television wired or wireless wide area digital networks (including cable television, satellite television, and terrestrial broadcast television), vehicle and industrial networks (including CANBus), and the like. Some networks typically require an external network interface adapter that is connected to some universal data port or peripheral bus (1749) (such as a universal serial bus (USB) port of the computer system (1700)); others are typically integrated into the kernel of the computer system (1700) by connecting to the system bus as described below (for example, into the Ethernet interface of a personal computer system or into the cellular network interface of a smartphone computer system). By using any of these networks, the computer system (1700) can communicate with other entities. Such communication can be one-way, receive-only (e.g., broadcast TV), one-way send-only (e.g., a CAN bus to certain CAN bus devices), or two-way communication to other computer systems using local or wide area digital networks. Certain protocols and protocol stacks may be used on each of those networks and network interfaces as described above.

[0215] The above-mentioned human-machine interface device, human-accessible storage device and network interface can be connected to the kernel (1740) of the computer system (1700).

[0216] The core (1740) may include one or more central processing units (CPUs) (1741), graphics processing units (GPUs) (1742), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (1743), hardware accelerators for specific tasks (1744), etc. These devices, as well as read-only memory (ROM) (1745), random access memory (1746), internal mass storage (such as internal non-user accessible hard disk drives, SSDs) (1747), etc., may be interconnected via a system bus (1748). In some computer systems, the system bus (1748) may be accessible in the form of one or more physical plugs to enable expansion through additional CPUs, GPUs, etc. Peripheral devices may be connected to the core's system bus (1748) directly or through a peripheral bus (1749). Peripheral bus architectures include PCI, USB, etc.

[0217] The CPU (1741), GPU (1742), FPGA (1743) and accelerator (1744) can execute certain instructions, which can be combined to form the aforementioned computer code. The computer code can be stored in ROM (1745) or RAM (1746). Intermediate data can also be stored in RAM (1746), while permanent data can be stored in, for example, internal mass storage (1747). Fast storage and reading to any memory device can be achieved by using a cache memory, which can be closely associated with one or more CPUs (1741), GPUs (1742), mass storage (1747), ROM (1745), RAM (1746), etc.

[0218] The computer readable medium may have computer code thereon, and various computer-implemented operations may be performed on the computer code. The medium and computer code may be specially designed and constructed for the purpose of this application, or may be the medium and code well known and available to those skilled in the art of computer software.

[0219] As an example and not a limitation, a computer system having an architecture (1700), in particular a kernel (1740), can provide a processor (including a CPU, a GPU, an FPGA, an accelerator, etc.) to execute software in one or more tangible computer-readable media to implement functions. Such a computer-readable medium can be a medium associated with a user-accessible mass storage as described above, and some storage of a non-volatile kernel (1740), such as a kernel internal mass storage (1747) or a ROM (1745). The software implementing the various embodiments of the present application can be stored in such a device and executed by the kernel (1740). Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software can enable the kernel (1740), in particular the processor therein (including a CPU, a GPU, an FPGA, etc.), to perform a specific process or a specific part of a specific process described herein, including defining a data structure stored in RAM (1746), and modifying these data structures according to a software-defined process. In addition or in lieu thereof, a computer system may provide the same functionality as logic hardwired or other components in a circuit (e.g., accelerator 1744), which may replace software or operate in conjunction with software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may include logic and vice versa. Where appropriate, references to computer-readable media may include circuits (e.g., integrated circuits (ICs)) storing execution software, circuits including execution logic, or both. The present application includes any suitable combination of hardware and software.

[0220] Appendix A: Abbreviations

[0221] ASIC: Application-Specific Integrated Circuit

[0222] BMS:benchmark set

[0223] CANBus: Controller Area Network Bus

[0224] CD: Compact Disc

[0225] CPUs: Central Processing Units

[0226] CRT: Cathode Ray Tube

[0227] CTBs: Coding Tree Blocks

[0228] CTUs: Coding Tree Units

[0229] CU:Coding Unit

[0230] DVD:Digital Video Disc

[0231] FPGA: Field Programmable Gate Array

[0232] GOPs: Groups of Pictures

[0233] GPUs: Graphics Processing Units

[0234] GSM: Global System for Mobile communications

[0235] HEVC: High Efficiency Video Coding

[0236] HRD: Hypothetical Reference Decoder

[0237] IC: Integrated Circuit

[0238] JEM: joint exploration model

[0239] LAN: Local Area Network

[0240] LCD: Liquid-Crystal Display

[0241] LTE: Long-Term Evolution

[0242] MV: Motion Vector

[0243] OLED: Organic Light-Emitting Diode

[0244] PBs: Prediction Blocks

[0245] PCI: Peripheral Component Interconnect

[0246] PLD:Programmable Logic Device

[0247] PUs:Prediction Units

[0248] RAM: Random Access Memory

[0249] ROM: Read-Only Memory

[0250] SEI: Supplementary Enhancement Information

[0251] SNR:Signal Noise Ratio

[0252] SSD: solid-state drive

[0253] TUs:Transform Units, Transformation Unit

[0254] USB: Universal Serial Bus

[0255] VUI:Video Usability Information video availability information

[0256] VVC: versatile video coding

[0257] Although the present application has described a number of exemplary embodiments, various changes, permutations, and substitutions of the embodiments are within the scope of the present application. It should therefore be understood that those skilled in the art are capable of designing a variety of systems and methods that, although not explicitly shown or described herein, embody the principles of the present application and are therefore within the spirit and scope of the present application.

Claims

1. A video decoding method, characterized in that: include: Identifying a merged shared area, where the merged shared area includes a plurality of coding blocks, and the plurality of coding blocks includes at least one inter-frame coding block; Constructing a shared merging candidate list for the merged shared area; Based on the shared merge candidate list, decoding the merged shared area; and, The history-based motion vector prediction HMVP table is updated according to the motion information of the last inter-frame coding block in the merged shared area in the decoding order.

2. The method according to claim 1, characterized in that The updating of the history-based motion vector prediction HMVP table according to the motion information of the last inter-frame coding block in the merged shared area in the decoding order includes: A last sub-block is selected from a group of inter-frame coded sub-blocks in the merged shared area according to a coding and decoding order, and the HMVP table is updated according to motion information of the last sub-block.

3. The method according to claim 1, characterized in that The size of the HMVP table is set to 6.

4. The method according to claim 1, characterized in that: The updating of the history-based motion vector prediction HMVP table according to the motion information of the last inter-frame coding block in the merged shared area in the decoding order includes: The HMVP buffer is updated using the motion information of the last inter-frame coded block.

5. The method according to claim 1, characterized in that Further including: When the merged shared area includes a plurality of inter-frame coding blocks, when processing other inter-frame coding blocks except the last inter-frame coding block in the merged shared area, the motion information of the other inter-frame coding blocks is not used to update the HMVP table.

6. The method according to any one of claims 1 to 5, characterized in that The size of the merged shared area is equal to or greater than a size threshold, and the size of the coding block is smaller than the size threshold.

7. A video decoding device, characterized in that: include: An identification module, configured to identify a merged shared area, wherein the merged shared area includes a plurality of coding blocks, and the plurality of coding blocks includes at least one inter-frame coding block; A construction module, used to construct a shared merging candidate list for the merged shared area; A decoding module, configured to decode the merged shared area based on the shared merge candidate list; And, updating the history-based motion vector prediction HMVP table according to the motion information of the last inter-frame coding block in the merged shared area in the decoding order.

8. The device according to claim 7, characterized in that The decoding module is used to select a last sub-block from a group of inter-frame coded sub-blocks in the merged shared area according to a coding and decoding order, and update the HMVP table according to motion information of the last sub-block.

9. The device according to claim 7, characterized in that The size of the HMVP table is set to 6.

10. The device according to claim 7, characterized in that The decoding module is used to update the HMVP buffer using the motion information of the last inter-frame coding block.

11. The device according to claim 7, characterized in that The decoding module is further configured to, when the merged shared area includes a plurality of inter-frame coding blocks, not use motion information of other inter-frame coding blocks to update the HMVP table when processing other inter-frame coding blocks except the last inter-frame coding block in the merged shared area.

12. The device according to any one of claims 7 to 11, characterized in that The size of the merged shared area is equal to or greater than a size threshold, and the size of the coding block is smaller than the size threshold.

13. A computer device comprising a memory and a processor, wherein the memory stores instructions, characterized in that: The instructions are executed by the processor to implement the video decoding method according to any one of claims 1-6.

14. A non-volatile computer-readable storage medium storing instructions, characterized in that: The instructions are executed by a processor to implement the video decoding method according to any one of claims 1-6.

Images