Method, apparatus, electronic device and storage medium for video coding

Abstract

Provided are a method, device, electronic device, and storage medium for video coding. The method for video coding includes: obtaining a coding block of a video bitstream; determining whether joint motion vector difference (JMVD) is used for predicting the coding block; based on the determination that the JMVD is used for predicting the coding block, obtaining, from the video bitstream, a list containing multiple scaling factors and a joint motion vector difference; deriving a predicted scaling factor based on the list containing multiple scaling factors; deriving a motion vector difference (MVD) of a reference frame list based on at least the joint motion vector difference and the predicted scaling factor; and reconstructing the coding block based on the derived MVD.

Description

[0001] Cross-reference to related applications

[0002] This application is based on and claims priority to U.S. Provisional Application No. 63 / 332,563, filed April 19, 2022, and U.S. Application No. 17 / 982,171, filed November 7, 2022, the entire contents of which are expressly incorporated herein by reference. Technical Field

[0003] This disclosure relates to a set of advanced image and video encoding and decoding techniques, and more specifically, to an improved scheme for joint coding of motion vector difference (JMVD). Background Technology

[0004] AOMedia Video 1 (AV1) is an open video coding format designed for video transmission over the Internet. It was developed by the Open Media Consortium (AOMedia) as a successor to VP9. AOMedia is a consortium founded in 2015 that includes semiconductor companies, video-on-demand providers, video content producers, software development companies, and web browser vendors. Many components of the AV1 project are derived from previous research by consortium members. Individual contributors began experimenting with the technology platform several years ago: Xiph / Mozilla's Daala released the code in 2010, Google's experimental VP9 evolution project VP10 was announced on September 12, 2014, and Cisco's Thor released it on August 11, 2015. Building on the VP9 code, AV1 incorporates additional technologies, several of which were developed in these experimental forms. The first version 0.1.0 of the AV1 reference codec was released on April 7, 2016. The alliance released the AV1 stream specification and reference software-based encoders and decoders on March 28, 2018. On June 25, 2018, a final version 1.0.0 of the specification was released. On January 8, 2019, a final version 1.0.0 of the specification, including errata table 1, was released. The AV1 stream specification includes reference video codecs.

[0005] ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) released the H.265 / HEVC (High Efficiency Video Coding) standard in 2013 (Revision 1), 2014 (Revision 2), 2015 (Revision 3), and 2016 (Revision 4). Since then, they have been studying the potential need for standardization of future video coding technologies that significantly outperform HEVC in compression capabilities. In October 2017, they issued a joint call for proposals (CfP) on video compression capabilities exceeding HEVC. As of February 15, 2018, a total of 22 CfP responses were submitted for standard dynamic range (SDR), 12 for high dynamic range (HDR), and 12 for 360 video categories. In April 2018, all received CfP responses were evaluated at the 122nd MPEG / 10th Joint Video Exploration Group – Joint Video Experts Group (JVET) meeting. Following careful evaluation, JVET officially launched the standardization of next-generation video coding beyond HEVC, the so-called Universal Video Coding (VVC).

[0006] For JMVD, there are technical problems when assuming linear motion in the JMVD encoding mode, because motion between two reference frames may not always be linear, and for example, motion may become slower or faster from the backward reference frame to the forward reference frame. Therefore, technical solutions are needed to address this problem. Summary of the Invention

[0007] According to one aspect of some embodiments, a video decoding method executed by at least one processor is provided. The method includes: acquiring coded blocks of a video bitstream; determining whether Joint Motion Vector Difference Coding (JMVD) is used to predict the coded blocks; based on determining that the JMVD is used to predict the coded blocks, acquiring from the video bitstream a list containing multiple scaling factors and a joint motion vector difference; deriving a predicted scaling factor based on the list containing multiple scaling factors; deriving a motion vector difference (MVD) of a list of reference frames based at least on the joint motion vector difference and the predicted scaling factor; and reconstructing the coded blocks based on the derived MVD. A video encoding method is also provided, including: acquiring coded blocks of video data; determining whether Joint Motion Vector Difference Coding (JMVD) is used to predict the coded blocks; based on determining that the JMVD is used to predict the coded blocks, determining a list containing multiple scaling factors and a joint motion vector difference; deriving a predicted scaling factor based on the list containing multiple scaling factors; and deriving a motion vector difference (MVD) of a list of reference frames based at least on the joint motion vector difference and the predicted scaling factor; wherein the derived MVD is used to reconstruct the coded blocks.

[0008] According to other aspects of some embodiments, an apparatus, electronic device, and computer-readable storage medium consistent with the methods described above are also provided. Attached Figure Description

[0009] Other features, properties, and various advantages of the subject matter will become more apparent from the following detailed description and accompanying drawings.

[0010] Figure 1 These are simplified illustrations based on some embodiments;

[0011] Figure 2 These are simplified schematic diagrams based on some embodiments;

[0012] Figure 3 These are simplified schematic diagrams based on some embodiments;

[0013] Figure 4 These are simplified schematic diagrams based on some embodiments;

[0014] Figure 5 These are simplified schematic diagrams based on some embodiments;

[0015] Figure 6 These are simplified schematic diagrams based on some embodiments;

[0016] Figure 7 These are simplified schematic diagrams based on some embodiments;

[0017] Figure 8 These are simplified schematic diagrams based on some embodiments;

[0018] Figure 9A These are simplified schematic diagrams based on some embodiments;

[0019] Figure 9B These are simplified schematic diagrams based on some embodiments;

[0020] Figure 10A These are simplified schematic diagrams based on some embodiments;

[0021] Figure 10B These are simplified schematic diagrams based on some embodiments;

[0022] Figure 10C These are simplified schematic diagrams based on some embodiments;

[0023] Figure 11 These are simplified schematic diagrams based on some embodiments;

[0024] Figure 12 These are simplified schematic diagrams based on some embodiments;

[0025] Figure 13 This is a simplified flowchart based on some embodiments;

[0026] Figure 14 These are simplified schematic diagrams based on some embodiments; and

[0027] Figure 15 This is a schematic diagram based on some embodiments. Detailed Implementation

[0028] The proposed features discussed below can be used individually or in any combination in any order. Furthermore, embodiments can be implemented using processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium.

[0029] To address one or more different technical problems, according to some embodiments, a method and apparatus are provided. The apparatus includes: a memory configured to store computer program code; and one or more processors configured to access the computer program code and operate as instructed by the computer program code. The computer program code includes: first acquisition code configured to cause at least one hardware processor to acquire coded blocks of a video bitstream; determination code configured to cause at least one hardware processor to determine whether Joint Motion Vector Difference Coding (JMVD) is used to predict the coded blocks; second acquisition code configured to cause at least one hardware processor to acquire, based on the determination that the JMVD is used to predict the coded blocks, a list containing multiple scaling factors and a joint motion vector difference from the video bitstream; first derivation code configured to cause at least one hardware processor to derive predicted scaling factors based on the list containing multiple scaling factors; second derivation code configured to cause at least one hardware processor to derive motion vector difference (MVD) of a list of reference frames at least based on the joint motion vector difference and the predicted scaling factors; and reconstruction code configured to cause at least one hardware processor to reconstruct the coded blocks based on the derived MVD.

[0030] According to some embodiments, the computer program code further includes: generation code configured to cause at least one hardware processor to generate a plurality of prediction blocks for each scaling factor in the list containing a plurality of scaling factors; and signaling code configured to cause at least one hardware processor to determine, based on a cost criterion, the differences between corresponding prediction blocks among the plurality of prediction blocks; and to signal the selection of one of the plurality of scaling factors based on the differences between the corresponding prediction blocks, wherein the plurality of prediction blocks are generated based on corresponding scaling factors among the plurality of scaling factors.

[0031] According to some embodiments, the prediction blocks among the plurality of prediction blocks are generated based on motion vectors, the motion vectors being equal to the sum of the motion vector prediction and the scaled motion vector difference, the scaled motion vector difference being scaled by one of the plurality of scaling factors.

[0032] According to some embodiments, the cost criterion is based on at least one of the sum of absolute differences (SAD), the sum of mean square errors (SSE), and the sum of absolute transformation differences (SATD).

[0033] According to some embodiments, the computer program code includes: further signaling code configured to cause at least one hardware processor to signal at least one of a first index and a first flag among the plurality of scaling factors; and entropy encoding code configured to cause at least one hardware processor to entropy encode the first flag based on at least one of the plurality of scaling factors.

[0034] According to some embodiments, the computer program code includes: additional signaling code configured to cause at least one hardware processor to signal whether the signaling of at least one of the first index and the first flag is explicit or implicit in the code stream.

[0035] According to some embodiments, the computer program code includes: reordering code configured to cause at least one hardware processor to reorder the plurality of scaling factors; and additional signaling code configured to cause at least one hardware processor to signal the index of the reordered plurality of scaling factors.

[0036] According to some embodiments, the scaling factor of the prediction is derived based on at least one of the following: quantization step size of adjacent blocks, quantization parameters, block size, inter-frame prediction mode, motion vector difference (MVD) category, MVD resolution, reference image, motion vector prediction MVP index, and MVD scaling factor.

[0037] According to some embodiments, the computer program code includes: derivation code configured to cause at least one hardware processor to derive motion vector prediction MVP for at least one coded block following the coded block based on one of the plurality of scaling factors.

[0038] According to some embodiments, computer program code signals the selection of one of the plurality of scaling factors in at least one of the frame header, slice header, and sequence header.

[0039] Figure 1 A simplified block diagram of a communication system 100 according to an embodiment of the present disclosure is shown. The communication system 100 may include at least two terminals 102 and 103 interconnected via a network 105. For unidirectional data transmission, the first terminal 103 may encode video data locally for transmission to the second terminal 102 via the network 105. The second terminal 102 may receive the encoded video data from the first terminal 103 from the network 105, decode the encoded data, and display the recovered video data. Unidirectional data transmission is common in media service applications, etc.

[0040] Figure 1 A third terminal 101 and a fourth terminal 104 are shown, provided to support bidirectional transmission of encoded video, which may occur, for example, during a video conference. For bidirectional data transmission, each of the third terminal 101 and the fourth terminal 104 can encode video data acquired at a local location for transmission to the other terminal via network 105. Each of the third terminal 101 and the fourth terminal 104 can also receive encoded video data sent by the other terminal, can decode the encoded data, and can display the recovered video data on a local display device.

[0041] exist Figure 1 In the embodiments disclosed herein, the first terminal 101, the second terminal 102, the third terminal 103, and the fourth terminal 104 may be servers, personal computers, and smartphones, but the principles disclosed herein are not limited to these. The embodiments disclosed herein are applicable to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 105 refers to any number of networks that transmit encoded video data between the first terminal 101, the second terminal 102, the third terminal 103, and the fourth terminal 104, including, for example, wired (connected) and / or wireless communication networks. Network 105 may exchange data in circuit-switched and / or packet-switched channels. This network may include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this application, unless explained below, the architecture and topology of network 105 may be irrelevant to the operation of this application.

[0042] As an example, Figure 2 This illustration shows the placement of a video encoder and a video decoder in a streaming environment. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0043] The streaming system may include an acquisition subsystem 203, which may include a video source 201, such as a digital camera, to create, for example, an uncompressed video sample stream 213. This sample stream 213 may be characterized by a high data volume compared to an encoded video stream and may be processed by an encoder 202 coupled to the camera 201. The encoder 202 may include hardware, software, or a combination thereof to implement or implement aspects of the disclosed subject matter as described in more detail below. An encoded video stream 204 may be stored on a streaming server 205 for future use; this encoded video bitstream 204 may be characterized by a lower data volume compared to the sample stream. One or more streaming clients 212 and 207 may access the streaming server 205 to retrieve copies 208 and 206 of the encoded video stream 204. Client 212 may include a video decoder 211 that decodes the incoming copy of the encoded video stream 208 and creates an outgoing video sample stream 210 that may be presented on a display 209 or other presentation device (not shown). In some streaming systems, video streams can be encoded using 204, 206, and 208 bitrates according to certain video coding / compression standards. Examples of these standards have been mentioned above, and they are further described in this article.

[0044] Figure 3 This is a block diagram of a video decoder 300 according to an embodiment disclosed in this application.

[0045] Receiver 302 may receive one or more encoded video sequences to be decoded by decoder 300; in the same embodiment or another embodiment, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of the others. Encoded video sequences may be received from channel 301, which may be a hardware / software link to a storage device storing the encoded video data. Receiver 302 may receive encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams that may be forwarded to their respective user entities (not indicated). Receiver 302 may separate the encoded video sequences from other data. To prevent network jitter, buffer memory 303 may be coupled between receiver 302 and entropy decoder / parser 304 (hereinafter referred to as "parser 304"). Buffer memory 303 may not be necessary, or it may be made smaller, when receiver 302 receives data from a store / forward device with sufficient bandwidth and controllability, or from an isochronous synchronization network. Of course, in order to be used on business packet networks such as the Internet, a buffer memory 303 may also be required, which may be relatively large and have an adaptive size.

[0046] The video decoder 300 may include a parser 304 to reconstruct symbols 313 from an encoded video sequence. These symbols may include information for managing the operation of the video decoder 300, and potential information for controlling a display device such as a display 312, which is not a necessary component of the decoder but may be coupled to it. Control information for the display device may be a fragment of a parameter set (not indicated) of Supplemental Enhancement Information (SEI) messages or video availability information. The parser 304 may perform parsing / entropy decoding on the received encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser 304 may extract a subset of parameters from the encoded video sequence for use in the video decoder, based on at least one parameter corresponding to a group. Subgroups can include Group of Pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), prediction units (PU), and so on. The entropy decoder / parser can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[0047] The parser 304 can perform entropy decoding / parsing operations on the video sequence received from the buffer 303 to create symbol 313. The parser 304 can receive encoded data and selectively decode specific symbols 313. In addition, the parser 304 can determine whether to provide specific symbols 313 to the motion compensation prediction unit 306, the scaler / inverse transform unit 305, the intra-frame prediction unit 307, or the loop filter 311.

[0048] Depending on the type of encoded video frames or a subset of encoded video frames (e.g., inter-frame and intra-frame frames, inter-frame and intra-frame blocks) and other factors, the reconstruction of symbol 313 may involve multiple different units. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the encoded video sequence by parser 304. For brevity, the flow of such subgroup control information between parser 304 and the various units described below is not described.

[0049] In addition to the functional blocks already mentioned, the decoder 300 can be conceptually subdivided into several functional units as described below. In practical embodiments operating under commercial constraints, many of these units interact closely with each other and can be integrated with one another. However, for the purposes of describing the disclosed subject matter, it is appropriate to conceptually subdivide them into the functional units described below.

[0050] The first unit is the scaler / inverse transform unit 305. The scaler / inverse transform unit 305 receives quantization transform coefficients as symbols 313 from the parser 304, along with control information including the transform mode used, block size, quantization factor, and quantization scaling matrix. The scaler / inverse transform unit 305 can output blocks containing sample values, which can be input into the aggregator 310.

[0051] In some cases, the output samples of the scaler / inverse transform unit 305 may belong to intra-coded blocks; that is, blocks that do not use predictive information from previously reconstructed images, but can use predictive information from previously reconstructed portions of the current image. Such predictive information may be provided by the intra-picture prediction unit 307. In some cases, the intra-picture prediction unit 307 uses surrounding reconstructed information extracted from the current (partially reconstructed) image 309 to generate surrounding blocks of the same size and shape as the block being reconstructed. In some cases, the aggregator 310 adds the predictive information generated by the intra-picture prediction unit 307 to the output sample information provided by the scaler / inverse transform unit 305 based on each sample.

[0052] In other cases, the output samples of the scaler / inverse transform unit 305 may belong to inter-frame coding and latent motion compensation blocks. In this case, the motion compensation prediction unit 306 can access the reference image memory 308 to extract samples for prediction. After motion compensation is performed on the extracted samples according to symbol 313, these samples can be added by the aggregator 310 to the output of the scaler / inverse transform unit (referred to in this case as residual samples or residual signals) to generate output sample information. The motion compensation prediction unit's retrieval of prediction samples from addresses in the reference image memory can be controlled by motion vectors, and these motion vectors are available to the motion compensation prediction unit in the form of symbol 313, which, for example, includes X, Y, and reference image components. Motion compensation may also include interpolation of sample values ​​extracted from the reference image memory when using subsample precise motion vectors, motion vector prediction mechanisms, etc.

[0053] The output samples of aggregator 310 can be employed by various loop filtering techniques in loop filter unit 311. Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video bitstream, and these parameters can be used as symbols 313 from parser 304 in loop filter unit 311. However, in other embodiments, video compression techniques may also respond to metadata acquired during decoding of a previous (in decoding order) portion of an encoded picture or encoded video sequence, and to previously reconstructed and loop-filtered sample values.

[0054] The output of the loop filter unit 311 can be a sample stream, which can be output to the display device 312 and stored in the reference image memory 308 for subsequent inter-frame image prediction.

[0055] Once fully reconstructed, certain encoded images can be used as reference images for future predictions. Once the encoded images have been fully reconstructed and are identified as reference images (e.g., by parser 304), the current reference image 309 can become part of the reference image buffer 308, and new current image memory can be reallocated before the reconstruction of subsequent encoded images begins.

[0056] The video decoder 300 can perform decoding operations according to predetermined video compression techniques, such as those documented in the ITU-T H.265 standard. The encoded video sequence may conform to the syntax specified by the video compression technique or standard used; in this sense, it follows the syntax of the video compression technique or standard, as specified in the video compression technique document or standard, and specifically in the summary document therein. For compliance, the complexity of the encoded video sequence is also required to be within the limits defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum image size, maximum frame rate, maximum reconstruction sampling rate (measured, for example, megasamples per second), maximum reference image size, etc. In some cases, the limitations set by the hierarchy can be further defined by the Hypothetical Reference Decoder (HRD) specification and metadata managed by the HRD buffer, which is represented by signals in the encoded video sequence.

[0057] In this embodiment, receiver 302 may receive additional (redundant) data along with the encoded video. The additional data may be a portion of the encoded video sequence. The additional data may be used by video decoder 300 to properly decode the data and / or more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0058] Figure 4 This is a functional block diagram of a video encoder 400 according to an embodiment of the present disclosure.

[0059] Encoder 400 can receive video samples from video source 401 (not part of the encoder), which can capture video images to be encoded by encoder 400.

[0060] Video source 401 can provide a source video sequence in the form of a digital video sample stream encoded by video encoder 303. This digital video sample stream can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 YCrCb, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, video source 401 can be a storage device storing previously prepared video. In a video conferencing system, video source 401 can be a camera capturing local image information as a video sequence. Video data can be provided as multiple individual pictures, which are given motion when viewed sequentially. The pictures themselves can be constructed as spatial pixel arrays, where each pixel can include one or more samples, depending on the sampling structure, color space, etc., used. Those skilled in the art can easily understand the relationship between pixels and samples. The following focuses on describing samples.

[0061] According to an embodiment, encoder 400 can encode and compress images of a source video sequence into an encoded video sequence 410 in real time or under any other time constraints required by the application. Implementing an appropriate encoding rate is a function of controller 402. Controller 402 controls and is functionally coupled to other functional units described below. For simplicity, coupling is not shown in the figures. Parameters set by the controller may include rate control related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. Other functions of controller 402 can be readily identified by those skilled in the art, as they may relate to a video encoder 400 optimized for a particular system design.

[0062] Some video encoders (503) operate within an encoding loop readily recognized by those skilled in the art. In a simplified description, the encoding loop may include the encoding portion of encoder 402 (hereinafter referred to as the “source encoder”) (responsible for creating symbols based on the input image to be encoded and the reference image) and a (local) decoder 406 embedded in encoder 400. Decoder 406 creates sample data that the (remote) decoder also creates (because in the video compression techniques considered in this application, any compression between the symbols and the encoded video stream is lossless). The reconstructed sample stream is input to reference image memory 405. Since the decoding of the symbol stream produces bit-precise results independent of the decoder location (local or remote), the contents of the reference image buffer are also bit-precisely corresponding between the local encoder and the remote encoder. In other words, the reference image samples “seen” by the encoder’s prediction portion are exactly the same sample values ​​that the decoder will “see” during the prediction process. This fundamental principle of reference image synchronization (and the drift that occurs when synchronization cannot be maintained, for example, due to channel errors) is well known to those skilled in the art.

[0063] The operation of the “local” decoder 406 can be combined with, for example, the above. Figure 3 The "remote" decoder 300 is described in detail. However, a brief additional reference is provided. Figure 4 When symbols are available and the entropy encoder 408 and the parser 304 are able to encode / decode the symbols into an encoded video sequence without loss, the entropy decoding portion of the decoder 300, including the channel 301, receiver 302, buffer 303 and parser 304, may not be fully implemented in the local decoder 406.

[0064] It can then be observed that any decoder technique other than parsing / entropy decoding, which exists in the decoder, must also exist in the corresponding encoder in essentially the same functional form. The description of encoder techniques can be simplified because encoder techniques are inverses of the fully described decoder techniques. More detailed descriptions are only required in certain areas, and are provided below.

[0065] As part of its operation, the source encoder 403 can perform motion-compensated predictive coding. The motion-compensated predictive coding predictively encodes the input frame with reference to one or more previously encoded frames from the video sequence designated as "reference frames." In this manner, the encoding engine 407 encodes the differences between pixel blocks in the input frame and pixel blocks in the reference frame, which can be selected as a prediction reference for the input frame.

[0066] The local video decoder 406 can decode encoded video data of frames that can be designated as reference frames, based on symbols created by the source encoder 403. The encoding engine 407 can operate as a lossy process. When encoded video data can be decoded by the video decoder (… Figure 4 When the source video sequence (not shown) is decoded, the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder 406 replicates the decoding process, which can be performed by the video decoder on the reference frame, and allows the reconstructed reference frame to be stored in the reference image memory 405. In this way, the encoder 400 can locally store a copy of the reconstructed reference frame that shares common content (no transmission errors) with the reconstructed reference frame to be acquired by the remote video decoder.

[0067] Predictor 404 can perform a prediction search against encoding engine 407. That is, for a new frame to be encoded, predictor 404 can search the reference image memory 405 for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that can serve as appropriate prediction references for the new image. Predictor 404 can operate pixel-by-pixel based on the sample blocks to find suitable prediction references. In some cases, based on the search results obtained by predictor 404, it can be determined that the input image may have prediction references obtained from multiple reference images stored in the reference image memory 405.

[0068] The controller 402 can manage the encoding operations of the source encoder 403, including, for example, setting parameters and subgroup parameters for encoding video data.

[0069] The outputs of all the aforementioned functional units can be entropy encoded in the entropy encoder 408. The entropy encoder performs lossless compression on the symbols generated by the various functional units using techniques known to those skilled in the art, such as Huffman coding, variable-length coding, and arithmetic coding, thereby converting the symbols into an encoded video sequence.

[0070] Transmitter 409 can buffer the encoded video sequence created by entropy encoder 408, thereby preparing it for transmission via communication channel 411, which may be a hardware / software link to a storage device that will store the encoded video data. Transmitter 409 can combine the encoded video data from source encoder 403 with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0071] The controller 402 manages the operation of the encoder 400. During encoding, the controller 402 can assign a specific encoded image type to each encoded image, but this may affect the encoding techniques applicable to the corresponding images. For example, images can typically be assigned to any of the following frame types:

[0072] An intra-frame picture (I-picture) is a picture that can be encoded and decoded without using any other pictures in the sequence as a prediction source. Some video codecs allow different types of intra-frame pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are familiar with variations of I-pictures and their corresponding applications and characteristics.

[0073] A predictive image (P-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most one motion vector and reference index to predict sample values ​​for each block.

[0074] A bidirectional predictive image (B-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most two motion vectors and a reference index to predict sample values ​​for each block. Similarly, multiple predictive images can use more than two reference images and associated metadata to reconstruct a single block.

[0075] Source images are typically spatially subdivided into multiple sample blocks (e.g., 4×4, 8×8, 4×8, or 16×16 sample blocks), and encoded block by block. These blocks can be predictively encoded with reference to other (already encoded) blocks, determined based on the encoding assignment of the corresponding images applied to the blocks. For example, blocks of an I-image can be non-predictively encoded, or the blocks can be predictively encoded (spatial prediction or intra-frame prediction) with reference to already encoded blocks of the same image. Pixel blocks of a P-image can be predictively encoded with reference to a previously encoded reference image via spatial prediction or temporal prediction. Blocks of a B-image can be predictively encoded with reference to one or two previously encoded reference images via spatial prediction or temporal prediction.

[0076] The video encoder 400 can perform encoding operations according to predetermined video coding techniques or standards, such as those specified in ITU-T H.265 Recommendation. In operation, the video encoder 400 can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

[0077] In this embodiment, the transmitter 409 may transmit additional data while transmitting encoded video. The source encoder 403 may include such data as part of the encoded video sequence. Additional data may include temporal / spatial / SNR enhancement layers, redundant images and slices, other forms of redundant data, Supplementary Enhancement Information (SEI) messages, fragments of Visual Usability Information (VUI) parameter sets, etc.

[0078] Figure 5 The intra-prediction modes used in HEVC and JEM are shown. To capture arbitrary edge directions presented in natural video, the number of directional intra-prediction modes is expanded from 33, as used in HEVC, to 65. Additional directional modes in JEM, built on top of HEVC, are also shown. Figure 1 (b) is depicted as a dashed arrow, and the planar mode and DC mode remain the same. These denser directional intra-prediction modes are applicable to all block sizes and to both luma and chroma intra-prediction. Figure 5 As shown, the directional intra-prediction mode associated with an odd-numbered intra-prediction mode index, indicated by a dashed arrow, is called the odd-numbered intra-prediction mode. The directional intra-prediction mode associated with an even-numbered intra-prediction mode index, indicated by a solid arrow, is called the even-numbered intra-prediction mode. In this document, as... Figure 5 The directional intra-prediction mode indicated by the solid or dashed arrows in the image is also called the angle mode.

[0079] In JEM, a total of 67 intra-prediction modes are used for lumen intra-prediction. To encode intra-modes, a most probable mode (MPM) list of size 6 is constructed based on the intra-modes of neighboring blocks. If an intra-mode is not from the MPM list, a signal flag is sent to indicate whether the intra-mode belongs to the selected modes. In JEM-3.0, there are 16 selected modes, uniformly chosen as the four-angle mode. In JVET-D0114 and JVET-G0060, 16 secondary MPMs are derived to replace the uniformly selected modes.

[0080] Figure 6 The diagram illustrates N reference layers used for intra-frame orientation mode. There are block unit 611, segment A 601, segment B 602, segment C 603, segment D 604, segment E 605, segment F 606, first reference layer 610, second reference layer 609, third reference layer 608, and fourth reference layer 607.

[0081] In HEVC and JEM, as well as some other standards such as H.264 / AVC, the reference samples used to predict the current block are limited to the most recent reference row (or column). In multi-reference row intra-prediction methods, for intra-directional modes, the number of candidate reference rows (or columns) increases from 1 (i.e., most recent) to N, where N is an integer greater than or equal to 1. Figure 6 Taking a 4×4 prediction unit (PU) as an example, the concept of a multi-row intra-frame orientation prediction method is illustrated. The intra-frame orientation mode can arbitrarily select one of N reference layers to generate a prediction value. In other words, a prediction value p(x,y) is generated from one of the reference samples S1, S2, ..., SN. A signal flag is sent to indicate which reference layer is selected for the intra-frame orientation mode. If N is set to 1, the intra-frame orientation prediction method is the same as the traditional method in JEM 2.0. Figure 6 In this context, reference rows 610, 609, 608, and 607 consist of six segments 601, 602, 603, 604, 605, and 606, plus a top-left reference sample. In this document, the reference layer is also referred to as a reference row. The coordinates of the top-left pixel within the current block cell are (0,0), and the coordinates of the top-left pixel in the first reference row 610 are (-1,-1).

[0082] In JEM, for the luma component, neighboring samples used for intra-prediction sample generation are filtered before the generation process. Filtering is controlled by the given intra-prediction mode and transform block size. If the intra-prediction mode is DC or the transform block size is equal to 4×4, neighboring samples are not filtered. If the distance between the given intra-prediction mode and the vertical (or horizontal) mode is greater than a predetermined threshold, the filtering process is enabled. For neighboring sample filtering, a [1, 2, 1] filter and a bilinear filter are used.

[0083] Position-dependent intra prediction combination (PDPC) is an intra prediction method that combines unfiltered boundary reference samples with HEVC-type intra predictions of filtered boundary reference samples. Each predicted sample pred[x][y] at (x, y) is calculated as follows:

[0084] (Equation 2-1)

[0085] Among them, R x,-1 R -1,y Let R represent the unfiltered reference samples located at the top and left of the current sample (x, y), respectively, and R... -1,-1 This represents the unfiltered reference sample located at the top left corner of the current block. The weights are calculated as follows:

[0086] (Equation 2-2)

[0087] (Equation 2-3)

[0088] (Equation 2-4)

[0089] (Equation 2-5)

[0090] Figure 7 Figure 700 illustrates this, where the DC mode PDPC weights the (0, 0) and (1, 0) positions within a 4×4 block using (wL, wT, wTL). If the PDPC is applied to DC, planar, horizontal, and vertical intra-frame modes, no additional boundary filters, such as the HEVC DC mode boundary filter or the horizontal / vertical mode edge filter, are required. Figure 7 The diagram illustrates the definitions of reference samples Rx,-1, R-1,y, and R-1,-1 for PDPC applied to the top-right diagonal mode. The predicted sample pred(x', y') is located at (x', y') within the predicted block. The coordinates x of the reference sample Rx,-1 are given by: x = x' + y' + 1, and similarly, the coordinates y of the reference sample R-1,y are given by: y = x' + y' + 1.

[0091] Figure 8 The diagram 800 illustrates Local Illumination Compensation (LIC), based on a linear model of illumination variation using scaling factor a and offset b. The coding unit (CU) is adaptively enabled or disabled for each inter-frame mode coding.

[0092] When LIC is applied to CU, the least squares error method is used to derive parameters a and b by using the neighboring samples of the current CU and the reference samples corresponding to those neighboring samples. More specifically, as Figure 8 As shown, neighboring samples (2:1 subsampling) of the CU and corresponding samples in the reference image (identified by motion information of the current CU or subCU) are used. IC parameters are derived and applied to each prediction direction respectively.

[0093] When encoding a CU in merge mode, the LIC flag is copied from the adjacent block in a manner similar to motion information copying in merge mode; otherwise, the CU is signaled to the LIC flag to indicate whether LIC should be applied.

[0094] Figure 9AThe diagram illustrates intra-prediction mode 900 used in HEVC. HEVC has a total of 35 intra-prediction modes, with mode 10 being a horizontal mode, mode 26 a vertical mode, and modes 2, 18, and 34 diagonal modes. The intra-prediction modes are signaled using three most probable modes (MPMs) and 32 remaining modes.

[0095] Figure 9B The illustration shows a total of 87 intra-prediction modes in a VVC implementation, where mode 18 is a horizontal mode, mode 50 is a vertical mode, and modes 2, 34, and 66 are diagonal modes. Modes -1 to -10 and modes 67 to 76 are referred to as Wide-Angle Intra Prediction (WAIP) modes.

[0096] Based on the PDPC expression, a linear combination of the intra-frame prediction mode (DC, plane, angle) and the reference sample is used to predict the predicted sample pred(x,y) located at position (x,y):

[0097]

[0098] Where Rx,-1 and R-1,y represent the reference samples located at the top and left of the current sample (x, y), respectively, and R-1,-1 represents the reference sample located at the top left corner of the current block.

[0099] For DC mode, for blocks with dimensions width and height, the weights are calculated as follows:

[0100] ,

[0101] Where nScale = (log2(width) – 2 + log2(height) – 2 + 2) >> 2, wT represents the weighting factor of the reference sample located in the upper reference row with the same horizontal coordinate, wL represents the weighting factor of the reference sample located in the left reference row with the same vertical coordinate, and wTL represents the weighting factor of the top-left reference sample of the current block. nScale specifies the rate at which the weighting factor decreases along the axis (wL decreases from left to right or wT decreases from top to bottom), i.e., the rate of decrease of the weighting factor, and in the current design it is the same along the x-axis (from left to right) and y-axis (from top to bottom). And 32 represents the initial weighting factor of the adjacent sample, and this initial weighting factor is also the top (left or top-left) weight assigned to the top-left sample in the current CB, and the weighting factor of the adjacent sample during PDPC should be equal to or less than this initial weighting factor.

[0102] For planar mode, wTL = 0, while for horizontal mode, wTL = wT, and for vertical mode, wTL = wL. PDPC weights can be calculated using only addition and shifting. The value of pred(x,y) can be calculated in a single step using Equation 1.

[0103] The methods presented herein can be used individually or in any combination in any order. Furthermore, each of the methods (or embodiments), encoders, and decoders can be implemented by a processing circuitry system (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-volatile computer-readable medium. In the following text, the term "block" can be interpreted as a prediction block, a coding block, or a coding unit (i.e., a CU).

[0104] Figure 10A The illustration shows an example 1000 of block partitioning using QTBT, and Figure 10B The diagram illustrates the corresponding tree representation 1001. Solid lines indicate quadtree partitions, and dashed lines indicate binary tree partitions. In each partition (i.e., non-leaf) node of a binary tree, a flag is signaled to indicate which partition type (i.e., horizontal or vertical) is used, where 0 indicates a horizontal partition and 1 indicates a vertical partition. For quadtree partitions, it is not necessary to indicate the partition type because quadtree partitions always divide blocks horizontally and vertically to produce four sub-blocks of equal size.

[0105] In HEVC, the CTU is partitioned into CUs using a quadtree structure represented as a coding tree to accommodate various local characteristics. At the CU level, a decision is made on whether to use inter-frame (temporal) or intra-frame (spatial) prediction to encode the picture region. Each CU can be further partitioned into one, two, or four PUs based on the PU partitioning type. Within a PU, the same prediction process is applied, and relevant information is transmitted to the decoder based on the PU. After obtaining residual blocks by applying a prediction process based on the PU partitioning type, the CU can be partitioned into transform units (TUs) according to another quadtree structure (such as the coding tree of the CU). One of the key features of the HEVC architecture is its multiple partitioning concepts, including CUs, PUs, and TUs.

[0106] According to embodiments, the QTBT structure eliminates the concept of multiple partition types, namely, the separation of CU, PU, ​​and TU concepts, and supports greater flexibility in the shape of the CU partition. In the QTBT block structure, the CU can have a square or rectangular shape. Figure 11In flowchart 1100, according to an exemplary embodiment, the coding tree unit (CTU) or CU obtained at S11 is first partitioned by a quadtree structure at S12. At S14, it is further determined whether the leaf nodes of the quadtree are partitioned by a binary tree structure; if so, then at S15, for example... Figure 10C As described, there are two types of partitioning in binary tree partitioning: symmetric horizontal partitioning and symmetric vertical partitioning. The leaf nodes of a binary tree are called coding units (CUs), and this partitioning is used for prediction and transform processing without any further partitioning. This means that CUs, PUs, and TUs have the same block size in the QTBT coding block structure. In VVC, CUs sometimes consist of coding blocks (CBs) of different color components; for example, in the case of P and B slices in a 4:2:0 chroma format, a CU contains one luma CB and two chroma CBs. And sometimes they consist of CBs of a single component; for example, in the case of I slices, a CU contains only one luma CB or only two chroma CBs.

[0107] According to the embodiment, the following parameters are defined for the QTBT partitioning scheme:

[0108] - CTU size: The size of the root node of the quadtree, the same concept as in HEVC.

[0109] - MinQTSize: The minimum allowed size of a quadtree leaf node.

[0110] - MaxBTSize: The maximum allowed size of the root node of the binary tree.

[0111] - MaxBTDepth: The maximum allowed depth of the binary tree, and

[0112] -MinBTSize: The minimum allowed size of a binary leaf node.

[0113] In one example of a QTBT partitioning structure, the CTU size is set to 128×128 luminance samples with two corresponding 64×64 chroma sample blocks, the MinQTSize (where QT is a quadtree) is set to 16×16, the MaxBTSize is set to 64×64, the MinBTSize (for width and height) is set to 4×4, and the MaxBTDepth is set to 4. At S12 or S15, the quadtree partitioning is first applied to the CTU to generate quadtree leaf nodes. Quadtree leaf nodes can have sizes ranging from 16×16 (i.e., MinQTSize) to 128×128 (i.e., the CTU size). If a leaf quadtree node is 128×128, it will not be further partitioned by the binary tree because, as checked at S14, its size exceeds the MaxBTSize (i.e., 64×64). Otherwise, at S15, the leaf quadtree node can be further partitioned by the binary tree. Therefore, the leaf node of the quadtree is also the root node of the binary tree, and its binary tree depth is 0. When the binary tree depth reaches MaxBTDepth (i.e., 4), no further partitioning is considered at S14. When the binary tree node has a width equal to MinBTSize (i.e., 4), no further horizontal partitioning is considered at S14. Similarly, when the binary tree node has a height equal to MinBTSize, no further vertical partitioning is considered at S14. As discussed below regarding the syntax describing QT / TT / BT sizes, signaling is provided at S16 to process, for example, the leaf nodes of the binary tree. The leaf nodes of the binary tree are further processed by prediction and transformation processing at S17, and similar to what is discussed herein regarding such prediction and transformation processing, no further partitioning is required. According to an exemplary embodiment, Figure 11 As shown, this signaling can also be provided at S13 after S12. In JEM, the maximum CTU size is a luminance sample of 256×256.

[0114] Furthermore, according to the embodiments, the QTBT scheme supports the ability / flexibility of having separate QTBT structures for luma and chroma. Currently, for P and B slices, the luma and chroma coding tree blocks (CTBs) in a CTU share the same QTBT structure. However, for I slices, the luma CTB is partitioned into CUs using a QTBT structure, and the chroma CTB is partitioned into chroma CUs using another QTBT structure. This means that the CUs in the I slice consist of coding blocks for the luma component or coding blocks for the two chroma components, and the CUs in the P or B slices consist of coding blocks for all three color components.

[0115] In HEVC, inter-frame prediction for small blocks was restricted to reduce memory accesses for motion compensation, resulting in no bidirectional prediction support for 4×8 and 8×4 blocks, and no inter-frame prediction support for 4×4 blocks. These restrictions were removed in QTBT, implemented in JEM-7.0.

[0116] Figure 10C A simplified block diagram 1100 illustrates the included multi-type-tree (MTT) structure 1002, which is a combination of the illustrated quadtree (QT) with nested binary trees (BT) and triplet / ternary trees (TT) (QT / BT / TT). First, the CTU or CU is recursively partitioned into square blocks using QT. Then, each QT leaf can be further partitioned using BT or TT, where BT and TT partitions can be recursively applied and interleaved, but further QT partitions cannot be applied. In all relevant proposals, TT uses a 1:2:1 ratio to vertically or horizontally divide the rectangular blocks into three blocks (thus avoiding widths and heights that are not powers of 2). To prevent partition contention, as... Figure 10C As shown in the simplified figure 1002, additional partitioning constraints are typically imposed on the MTT, with QT / BT / TT block partitioning performed in the VVC relative to blocks 1103 (quaternary), 1104 (binary, JEM), and 1105 (ternary) to avoid duplicate partitions (e.g., prohibiting vertical / horizontal binary partitioning on intermediate partitions caused by vertical / horizontal ternary partitioning). Further constraints can be set to the maximum depth of BT and TT.

[0117] The key advantage of this ternary partitioning (such as the ternary block 1105 mentioned above) is that, as a complement to quadtree and binary tree partitioning, ternary tree partitioning can capture objects located at the center of the block, while quadtrees and binary trees are always partitioned along the center of the block, and the width and height of the proposed ternary tree partitions are always powers of 2, so no additional transformations are required.

[0118] The design of a two-level tree is primarily to reduce complexity. Theoretically, the time complexity of traversing a tree is T. D , where T represents the number of partition types and D is the depth of the tree.

[0119] Figure 11Example 1100 shows block partitioning in VP9 and AV1, where the example coding tree unit (CTU) 1111 of VP9 shows that VP9 uses a 4-way partition tree starting from a 64×64 level 1112 down to a 4×4 level 1113, with some additional restrictions on blocks of 8×8 and below, as shown in the upper half of level 1113. Note that partitions designated as R refer to recursion, as the same partition tree is repeated at a lower scale until the lowest 4×4 level is reached. The example CTU 1104 of AV1 not only extends the partition tree to a 10-way structure 1116, but also increases the maximum size (referred to as a superblock in VP9 / AV1 terminology) to start from a 128×128 level 1115. Note that this includes 4:1 / 1:4 rectangular partitions that do not exist in VP9. And no rectangular partition can be further subdivided. Furthermore, AV1 adds more flexibility for using partitions below 8×8 levels in the sense that 2×2 chroma inter-frame prediction becomes possible in some cases.

[0120] In HEVC, coding tree units (CTUs) can be partitioned into coding units (CUs) using a quadtree structure represented as a coding tree to accommodate various local characteristics. At the CU level, a decision is made on whether to use inter-frame (temporal) or intra-frame (spatial) prediction to encode a region of the image. Each CU can be further partitioned into one, two, or four prediction units (PUs) based on the PU partitioning type. Within a PU, the same prediction process is applied, and relevant information is transmitted to the decoder based on the PU. After obtaining residual blocks by applying prediction processes based on the PU partitioning type, the CUs can be partitioned into transform units (TUs) according to another quadtree structure (such as the coding tree of the CU). One of the key features of the HEVC structure is its multiple partitioning concepts, including CUs, PUs, and TUs. In HEVC, a CU or TU can simply be square in shape, while a PU can be a square or rectangular shape for blocks used for inter-frame prediction. In HEVC, a coding block can be further partitioned into four square sub-blocks, and a transform is performed on each sub-block (i.e., TU). Each TU can be recursively divided (using quadtrees) into smaller TUs, a process known as Residual Quad-Tree (RQT). Furthermore, at image boundaries, HEVC employs implicit quadtree partitioning, ensuring that blocks maintain the quadtree partitioning until their size fits the image boundary.

[0121] also, Figure 12Example 1200 related to a merge mode with motion vector difference (MMVD) according to an exemplary embodiment is shown. For example, a motion vector difference merge mode (MMVD) is introduced in VVC in addition to a merge mode (where implicitly deduced motion information is directly used for the generation of predicted samples for the current CU). The MMVD flag can be signaled immediately after sending a skip flag and a merge flag to specify whether the MMVD mode is used for the CU. Furthermore, in MMVD, after selecting a merge candidate, the merge candidate is further refined by signaled motion vector difference (MVD) information, such that the further information includes a merge candidate flag, an index specifying the motion amplitude, and an index indicating the motion direction. In the MMVD mode, one of the first two candidates in the merge list is selected as the MV basis. The merge candidate flag can be signaled to specify which one to use.

[0122] The distance index specifies the amplitude of motion and indicates a predefined offset from the starting point. Figure 12 L0 reference 1201 and L1 reference 1202 are shown, where offsets are added to the horizontal or vertical component of the starting MV. The relationship between the distance index and the predefined offset is specified in Table 1.

[0123] Table 1 – Relationship between Distance Index and Predefined Offset

[0124]

[0125] According to an exemplary embodiment, the direction index represents the direction of the MVD relative to the starting point. The direction index can represent the four directions shown in Table 2 below. The meaning of the MVD symbol can vary depending on the information of the starting MV. For example, when the starting MV is a one-way predictive MV or a two-way predictive MV, and both lists point to the same side of the current image (i.e., both reference image order counts (POCs) are greater than the current image's POC, or both are less than the current image's POC), the symbol in Table 2 specifies the symbol of the MV offset added to the starting MV. And / or when the starting MV is a two-way predictive MV, and the two MVs point to different sides of the current image (i.e., one reference POC is greater than the current image's POC, and the other reference POC is less than the current image's POC), and the difference in POCs in list 0 is greater than the difference in POCs in list 1, the symbol in Table 2 specifies the symbol of the MV offset added to the list0 MV component of the starting MV, and the symbol of the list1 MV has the opposite value. Otherwise, if the difference of POC in list 1 is greater than the difference of POC in list 0, then the sign in table 2 specifies the sign of the MV offset of the list 1 MV component added to the starting MV, and the sign of list 0 MV has the opposite value.

[0126] According to an exemplary embodiment, the MVD can be scaled based on the difference in POCs in each direction. If the difference in POCs in the two lists is the same, no scaling is required. Otherwise, if the difference in POCs in list 0 is greater than the difference in POCs in list 1, the MVD of list 1 is scaled. If the difference in POCs of L1 is greater than that of L0, the MVD of list 0 is scaled in the same manner. If the initial MV is unidirectionally predicted, the MVD is added to the available MVs.

[0127] Table 2 – Signs of MV Offsets Specifyed by Direction Index

[0128]

[0129] According to an exemplary embodiment, symmetric MVD encoding can exist, where the MVD can be scaled based on the difference in POCs in each direction. If the differences in POCs in the two lists are the same, no scaling is required. Otherwise, if the difference in POCs in list 0 is greater than the difference in POCs in list 1, the MVD of list 1 is scaled. If the difference in POCs of L1 is greater than that of L0, the MVD of list 0 is scaled in the same manner. If the initial MV is unidirectionally predicted, the MVD is added to the available MVs.

[0130] Furthermore, according to an exemplary embodiment, in VVC, in addition to normal unidirectional and bidirectional prediction mode MVD signaling, a symmetric MVD mode for bidirectional MVD signaling can also be applied. In the symmetric MVD mode, the reference image indices of list-0 and list-1, as well as the motion information of the MVD in list-1, are not signaled but derived. The decoding process of the symmetric MVD mode is as follows:

[0131] 1. At the slice level, derive the variables BiDirPredFlag, RefIdxSymL0, and RefIdxSymL1 as follows:

[0132] If mvd_l1_zero_flag is 1, then set BiDirPredFlag to equal 0.

[0133] Otherwise, if the most recent reference image in list-0 and the most recent reference image in list-1 form a forward and backward reference image pair or a backward and forward reference image pair, then BiDirPredFlag is set to 1, and both list-0 and list-1 reference images are short-term reference images. Otherwise, BiDirPredFlag is set to 0.

[0134] 2. At the CU level, if the CU is bidirectional predictive coded and BiDirPredFlag is equal to 1, the symmetric mode flag is explicitly signaled to indicate whether symmetric mode is used.

[0135] Furthermore, when the symmetry mode flag is true, only mvp_l0_flag, mvp_l1_flag, and MVD0 are explicitly signaled. The reference indices of list-0 and list-1 are set to be equal to the reference image pair, respectively. MVD1 is set to be equal to (−MVD0).

[0136] According to an exemplary embodiment, inter-frame mode coding may exist in CWG-B018, wherein in AV1, for each coded block in an inter-frame coded frame, if the mode of the current block is not skip mode but inter-frame coding mode, another flag is signaled to indicate whether a single reference mode or a composite reference mode is used for the current block, wherein in the single reference mode, a prediction block is generated by a motion vector, while in the composite reference mode, a prediction block is generated by a weighted average of two prediction blocks derived from two motion vectors.

[0137] For example, in the case of a single reference mode, the following modes can be signaled:

[0138] NEARMV—Uses one of the motion vector prediction values ​​(MVPs) from the list indicated by the Dynamic Reference List (DRL) index.

[0139] NEWMV—uses one of the motion vector prediction values ​​(MVPs) in the list signaled via DRL index as a reference and applies the increment to the MVP.

[0140] GLOBALMV—Uses motion vectors based on frame-level global motion parameters.

[0141] Furthermore, for composite reference modes, the following modes can be signaled:

[0142] NEAR_NEARMV — One of the motion vector prediction values ​​(MVPs) in the list that are signaled via DRL index.

[0143] NEAR_NEWMV — Uses one of the motion vector prediction values ​​(MVPs) in the list signaled via DRL index as a reference and sends an incremental MV for the second MV.

[0144] NEW_NEARMV — Uses one of the motion vector prediction values ​​(MVPs) in the list signaled via DRL index as a reference and sends an incremental MV for the first MV.

[0145] NEW_NEWMV — Uses one of the motion vector prediction values ​​(MVPs) in the list that are signaled via DRL index as a reference and sends an increment for both MVs.

[0146] GLOBAL_GLOBALMV — Based on their frame-level global motion parameters, use the MV from each reference.

[0147] Furthermore, according to an exemplary embodiment, motion vector difference encoding may also exist in AV1, where AV1 allows 1 / 8 pixel motion vector precision (or accuracy), and uses the following syntax to signal the motion vector differences in reference frame list 0 or list 1:

[0148] mv_joint specifies which components of the motion vector difference are non-zero:

[0149] 0 indicates that there is no non-zero MVD along the horizontal or vertical direction.

[0150] 1 indicates that non-zero MVD exists only in the horizontal direction.

[0151] 2 indicates that non-zero MVD exists only in the vertical direction, and

[0152] 3 indicates that there is non-zero MVD in both the horizontal and vertical directions.

[0153] mv_sign specifies whether the motion vector difference is positive or negative.

[0154] mv_class specifies the class of motion vector difference (as shown in Table 3, a higher class means that the motion vector difference has a larger amplitude).

[0155] Table 3: Amplitude Categories for Motion Vector Difference

[0156]

[0157] mv_bit specifies the integer part of the offset between the motion vector difference and the starting amplitude for each MV category.

[0158] mv_fr specifies the first two decimal bits of the motion vector difference, and

[0159] mv_hp specifies the third decimal bit of the motion vector difference.

[0160] Furthermore, according to an exemplary embodiment, an adaptive MVD resolution may exist in CWG-B092, wherein for NEW_NEARMV and NEAR_NEWMV modes, the accuracy of the MVD depends on the associated category and magnitude of the MVD.

[0161] First, fractional MVDs are allowed only when the MVD magnitude is equal to or less than one pixel. Second, only one MVD value is allowed when the value of the associated MV category is equal to or greater than MV_CLASS_1, and for MV categories 1 (MV_CLASS_1), 2 (MV_CLASS_2), 3 (MV_CLASS_3), 4 (MV_CLASS_4), or 5 (MV_CLASS_5), the MVD values ​​in each MV category are derived as 4, 8, 16, 32, and 64, respectively. The allowed MVD values ​​in each MV category are illustrated in Table 4.

[0162] Table 4: Adaptive MVD in each MV amplitude category

[0163]

[0164] Furthermore, if the current block is encoded in NEW_NEARMV or NEAR_NEWMV mode, one context is used to signal mv_joint or mv_class. Otherwise, another context is used to signal mv_joint or mv_class.

[0165] According to an exemplary embodiment, joint MVD coding (JMVD) may also exist in CWG-B092, where a new inter-frame coding mode named JOINT_NEWMV can be applied to indicate whether the MVDs of the two reference lists are jointly signaled. If the inter-frame prediction mode is equal to the JOINT_NEWMV mode, the MVDs of reference list 0 and reference list 1 are jointly signaled. Therefore, only one MVD (named joint_mvd) can be signaled and transmitted to the decoder, and the incremental MVs of reference list 0 and reference list 1 are derived from joint_mvd.

[0166] The JOINT_NEWMV mode can be signaled together with the NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV, and GLOBAL_GLOBALMV modes. According to the exemplary embodiment, no additional context is added.

[0167] Furthermore, when the JOINT_NEWMV mode is signaled and the POC distances between the two reference frames and the current frame are different, the MVD is scaled for reference list 0 or reference list 1 based on the POC distance. Specifically, the distance between reference list 0 and the current frame is labeled as td0, and the distance between reference list 1 and the current frame is labeled as td1. If td0 is equal to or greater than td1, then joint_mvd is directly used for reference list 0, and the MVD of reference list 1 is derived from joint_mvd based on equation (1).

[0168] Equation (1)

[0169] Otherwise, if td1 is equal to or greater than td0, then joint_mvd is used directly for reference list 1, and the mvd of reference list 0 is derived from joint_mvd based on equation (2).

[0170] Equation (2)

[0171] According to an exemplary embodiment, CWG-C011 also includes improvements to the adaptive MVD resolution, where a new inter-frame coding mode, named AMVDMV, can be added to a single reference case. When the AMVDMV mode is selected, this selection indicates that AMVD is applied to the signal MVD. A flag named amvd_flag can be added in the JOINT_NEWMV mode to indicate whether AMVD is applied to the joint MVD coding mode. And when the adaptive MVD resolution is applied to the joint MVD coding mode, the MVD of two reference frames can be jointly signaled, and the accuracy of the MVD is implicitly determined by the MVD amplitude. Otherwise, the MVD of two (or more) reference frames is jointly signaled, and other MVD coding can be applied.

[0172] Furthermore, additional improvements can be made because, for JMVD, it is assumed that there is linear motion between the backward and forward reference frames. However, when selecting the JMVD coding mode for a block, a joint MVD is signaled for both reference frames, and the MVDs of the two reference frames are derived from the joint MVD based on the distance between the reference frames and the current frame. However, the motion between the two reference frames may not always be linear, because, for example, the motion from the backward reference frame to the forward reference frame may become slower or faster.

[0173] As discussed herein, the orientation of the reference frame is determined by whether the reference frame is displayed before or after the current frame. And according to an exemplary embodiment, see [link to exemplary embodiment]. Figure 13The diagram illustrates flowchart 1300, where at S130, encoded blocks of video data are acquired, and at S131, when it is determined that a JMVD mode is selected for an encoded block, a predefined scaling factor list can be used to derive the MVD of reference lists 0 and / or 1 from the signaled joint MVD. According to an exemplary embodiment, a bilateral matching-based method is used to derive the predicted scaling factor from the predefined scaling factor list. At S133, in an exemplary embodiment, for each candidate scaling factor in the predefined list, prediction blocks P0 1402 and P1 1403 are generated, as shown below. Figure 14 As shown in Example 1400 for the current block 1401, MV is equal to the sum of MVP and the scaling MVD, and then at S134 the difference between P0 and P1 is calculated and measured by the cost criterion, and at S135 the candidate scaling factor with the minimum cost produced by this measurement is used as the predicted scaling factor (pred_scale_factor).

[0174] According to an exemplary embodiment, the cost criteria include, but are not limited to, the sum of absolute differences (SAD), the sum of mean squared errors (SSE), and the sum of absolute transformation differences (SATD) features.

[0175] At S136, if it is determined that the index or flag (scaled_jmvd_flag) of the candidate scaling factor is used to signal, then at S137, the index or flag (scaled_jmvd_flag) of the candidate scaling factor is signaled, and at S142, the predicted scaling factor is used as the context for the entropy encoding of scaled_jmvd_flag.

[0176] Optionally, at S138, if it is determined that the index or flag (scaled_jmvd_flag) of the candidate scaling factor is explicitly signaled, then at S139, the index or flag (scaled_jmvd_flag) of the candidate scaling factor is explicitly signaled together with the pred_scale_factor used to reorder the candidate scaling factors, and then the index of the reordered candidate scaling factor is signaled, and encoded according to the above JMVD at S142.

[0177] Optionally, at S140, if it is determined that instead of signaling the index or flag of the candidate scaling factor (scaled_jmvd_flag), pred_scale_factor is directly used as the scaling factor for deriving the final MVD for generating the prediction block, then at S141, pred_scale_factor is directly used as the scaling factor for deriving the final MVD for generating the prediction block, and then encoded according to the above JMVD at S142.

[0178] Alternatively, other encoding processes, such as any of the processes described above for the encoded block obtained at S130, can be implemented at S132.

[0179] According to an exemplary embodiment, the scaling process of MVD is the same as that described in US 63 / 328,062, filed April 6, 2022, which is incorporated herein in its entirety.

[0180] According to an exemplary embodiment, the index or flag of the candidate scaling factor can be explicitly signaled in the bitstream or implicitly deduced from the bilateral matching method (pred_scale_factor), and at S142, the choice between explicit signaling and implicit deduction is also signaled.

[0181] According to an exemplary embodiment, the predicted scaling factor can be derived from a different set of candidate scaling factors compared to a set of candidate scaling factors that are explicitly signaled. Furthermore, for example, the explicitly signaled candidate scaling factors may include a finite number of candidates (e.g., 1, ½, 2), but the predicted scaling factor can be derived from a larger set of candidates (e.g., 1, 1 / 8, 2 / 8, 3 / 8, 4 / 8, ..., 15 / 8, 16 / 8, ..., 32 / 8).

[0182] According to an exemplary embodiment, for motion vector prediction (MVP), such as at S130 after S142, only the MV of a neighboring block derived using MVD with an explicitly signaled scaling factor or a default scaling factor (e.g., 1) can be used to derive the MVP of a subsequent coding block. Otherwise, if the MV of such a neighboring block is derived using MVD with a scaling factor derived using bilateral matching, then the MV of that neighboring block is not used to derive the MVP of such a subsequent coding block.

[0183] According to an exemplary embodiment, candidate scaling factors are derived based on encoded information, which includes, but is not limited to, any one or more of the following: quantization step size or quantization parameters of adjacent blocks, block size, inter-frame prediction mode, MVD category, MVD resolution, reference image, MVP index, and MVD scaling factor.

[0184] According to an exemplary embodiment, candidate scaling factors are signaled using high-level syntax, including but not limited to any one of the frame header, slice header, and sequence header.

[0185] The techniques described above can be implemented as computer software that uses computer-readable instructions and is physically stored in one or more computer-readable media or in one or more specially configured hardware processors. For example, Figure 15A computer system 1500 suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0186] Computer software can be coded using any suitable machine code or computer language. Machine code or computer language can be created through assembly, compilation, linking or similar mechanisms to include instructions that can be executed directly by a computer's central processing unit (CPU), graphics processing unit (GPU) or through interpretation, microcode execution, etc.

[0187] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0188] Figure 15 The components shown for computer system 1500 are exemplary in nature and are not intended to impose any limitation on the scope or functionality of computer software implementing embodiments of this disclosure. The configuration of the components should also not be construed as having any dependency or requirement on any component or combination thereof illustrated in the exemplary embodiments of computer system 1500.

[0189] Computer system 1500 may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from one or more human users through, for example, tactile input (such as keystrokes, swipes, or movement with a data glove), audio input (such as speech or tapping), visual input (such as gestures), or olfactory input (not shown). Human-machine interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (such as speech, music, or ambient sound), images (such as scanned images or photographic images acquired from a still image camera), and video (such as two-dimensional video or three-dimensional video, including stereoscopic video).

[0190] The input human-machine interface device may include one or more of the following (only one of each is depicted): keyboard 1501, mouse 1502, touchpad 1503, touch screen 1510, joystick 1505, microphone 1506, scanner 1508, and camera 1507.

[0191] Computer system 1500 may also include certain human-machine interface output devices. Such human-machine interface output devices can stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (such as tactile feedback of touch screen 1510 or joystick 1505, but may also include tactile feedback devices that are not used as input devices), audio output devices (such as speakers 1509, headphones (not depicted)), visual output devices (such as screen 1510, including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without tactile feedback capability—some of which are capable of outputting two-dimensional or more than three-dimensional visual output in a manner such as stereoscopic output; virtual reality glasses (not depicted), holographic displays, and smoke canisters (not depicted)), and printers (not depicted).

[0192] Computer system 1500 may also include human-accessible storage devices and their associated media, such as optical media including media such as CD / DVD ROM / RW 1520 having CD / DVD 1511, thumb drives 1522, removable hard disk drives or solid-state drives 1523, conventional magnetic media such as magnetic tapes and floppy disks (not depicted), devices based on dedicated ROM / ASIC / PLD such as security dongles (not depicted), etc.

[0193] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other volatile signals.

[0194] Computer system 1500 may also include an interface 1599 to one or more communication networks 1598. Network 1598 may be, for example, wireless, wired, or optical. Networks may also be local, wide area, metropolitan area, vehicular and industrial, real-time, latency-tolerant, etc. Examples of networks 1598 include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicular and industrial networks including CAN buses, etc. Some networks 1598 typically require external network interface adapters (such as, for example, USB ports of computer system 1500) to be attached to certain general-purpose data ports or peripheral buses (1550 and 1551); other networks are typically integrated into the core of computer system 1500 by being attached to system buses as described below (e.g., Ethernet interface to a PC computer system or cellular network interface to a smartphone computer system). Using any of these networks 1598, computer system 1500 can communicate with other entities. Such communication can be unidirectional (receive-only, e.g., broadcasting TV), unidirectional (transmit-only, e.g., CANbus to certain CANbus devices), or bidirectional, e.g., to other computer systems using local area digital networks or wide area digital networks. Certain protocols and protocol stacks can be used on each of those networks and network interfaces described above.

[0195] The aforementioned human-machine interface devices, human-accessible storage devices, and network interfaces can be attached to the core 1540 of the computer system 1500.

[0196] Core 1540 may include one or more central processing units (CPUs) 1541, graphics processing units (GPUs) 1542, dedicated programmable processing units in the form of field-programmable gate arrays (FPGAs) 1543, hardware acceleration 1544 for certain tasks, etc. These devices, along with read-only memory (ROM) 1545, random access memory 1546, and internal mass storage 1547 such as internal non-user-accessible hard disk drives (SD drives) or SSDs, can be connected via system bus 1548. In some computer systems, system bus 1548 may be accessed as one or more physical connectors to allow for expansion by adding CPUs, GPUs, etc. Peripheral devices may be attached directly or via peripheral bus 1551 to the core's system bus 1548. Peripheral bus architectures include PCI, USB, etc.

[0197] CPU 1541, GPU 1542, FPGA 1543, and accelerator 1544 can execute certain instructions, and combinations of these instructions can constitute the aforementioned computer code. This computer code can be stored in ROM 1545 or RAM 1546. Transient data can also be stored in RAM 1546, while permanent data can be stored, for example, in internal mass storage 1547. Fast storage and retrieval of any memory device can be enabled by using cache memory, which can be closely associated with one or more CPUs 1541, GPU 1542, mass storage 1547, ROM 1545, RAM 1546, etc.

[0198] Computer-readable media may have computer code thereon for performing various computer-implemented operations. The media and computer code may be those specifically designed and constructed for the purposes of this disclosure, or they may be of types known and available to those skilled in the art of computer software.

[0199] As a non-limiting example, a computer system 1500 having an architecture, and particularly a core 1540, can provide functionality as a result of one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as described above, as well as certain memories of the core 1540 having non-volatile properties, such as internal mass storage 1547 or ROM 1545. Software implementing various embodiments of this disclosure can be stored in such devices and executed by the core 1540. Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software can cause the core 1540, and particularly the processors therein (including CPUs, GPUs, FPGAs, etc.), to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM 1546 and modifying such data structures according to software-defined processes. Alternatively or as an alternative, the computer system may provide functionality, as a result of hard-wired logic or otherwise embodied in circuitry (e.g., accelerator 1544), which may replace or operate with software to perform the specific process or a specific portion of the specific process described herein. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may include circuitry storing software for execution (such as integrated circuits (ICs)), circuitry embodying logic for execution, or both. This disclosure includes any suitable combination of hardware and software.

[0200] The proposed methods can be used individually or in combination in any order. Furthermore, each of the methods (or embodiments), encoders, and decoders can be implemented using processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium. In the following text, the term "block" can be interpreted as a prediction block, coding block, or coding unit, i.e., a CU.

[0201] While this disclosure describes several exemplary embodiments, variations, arrangements, and various alternative equivalents exist, all of which fall within the scope of this disclosure. Therefore, those skilled in the art will be able to design numerous systems and methods that, while not expressly shown or described herein, embody the principles of this disclosure and are therefore within its spirit and scope.

Claims

1. A method for video decoding, characterized in that, The method includes: Obtain the encoded blocks of the video stream; Determine whether Joint Motion Vector Difference Coding (JMVD) is used to predict the coded block; Based on the determination that the JMVD is used to predict the coded block, a list containing multiple scaling factors and joint motion vector difference are obtained from the video bitstream; The predicted scaling factor is derived based on the list containing multiple scaling factors; Based at least on the joint motion vector difference and the predicted scaling factor, the motion vector difference (MVD) of the reference frame list is derived; The coded block is reconstructed based on the derived MVD.

2. The video decoding method according to claim 1, characterized in that, Further includes: Generate multiple prediction blocks for each scaling factor in the list containing multiple scaling factors; as well as Based on cost criteria, the differences between corresponding prediction blocks in the plurality of prediction blocks are determined; Based on the differences between the corresponding prediction blocks, a signal is sent to notify the selection of one of the plurality of scaling factors; The plurality of prediction blocks are generated based on the corresponding scaling factors among the plurality of scaling factors.

3. The video decoding method according to claim 2, characterized in that, The prediction blocks among the plurality of prediction blocks are generated based on motion vectors, wherein the motion vectors are equal to the sum of the motion vector prediction and the scaled motion vector difference, and the scaled motion vector difference is scaled by one of the plurality of scaling factors.

4. The video decoding method according to claim 2, characterized in that, The cost criterion is based on at least one of the sum of absolute differences (SAD), the sum of mean square errors (SSE), and the sum of absolute transformation differences (SATD).

5. The video decoding method according to claim 1, characterized in that, At least one of the first index and first flag of one of the plurality of scaling factors is signaled; as well as The first flag is entropy encoded based on at least one of the plurality of scaling factors.

6. The video decoding method according to claim 5, characterized in that, The signaling is sent to indicate whether the signaling of at least one of the first index and the first flag is explicit or implicit in the code stream.

7. The video decoding method according to claim 1, characterized in that, The multiple scaling factors are reordered; and The indexes of multiple scaling factors that have been reordered are signaled.

8. The video decoding method according to claim 1, characterized in that, Further includes: The scaling factor of the prediction is derived based on at least one of the following: quantization step size of adjacent blocks, quantization parameters, block size, inter-frame prediction mode, motion vector difference (MVD) category, MVD resolution, reference image, motion vector prediction MVP index, and MVD scaling factor.

9. The video decoding method according to claim 1, characterized in that, Further includes: Based on one of the plurality of scaling factors, derive the motion vector prediction MVP for at least one subsequent coding block.

10. The video decoding method according to claim 1, characterized in that, The selection of one of the plurality of scaling factors is signaled in at least one of the frame header, slice header, and sequence header.

11. An electronic device, characterized in that, include: At least one memory is configured to store computer program code; At least one processor is configured to access the computer program code and operate according to the instructions of the computer program code to implement the method as described in any one of claims 1-10.

12. A video decoding apparatus, characterized in that, The device includes: The first acquisition unit acquires the encoded blocks of the video stream; The determination unit determines whether the Joint Motion Vector Difference Coding (JMVD) is used to predict the coded block; The second acquisition unit, based on determining that the JMVD is used to predict the coded block, acquires from the video bitstream a list containing multiple scaling factors and joint motion vector difference; The first derivation unit derives the predicted scaling factor based on the list containing multiple scaling factors; The second derivation unit derives the motion vector difference (MVD) of the reference frame list based at least on the joint motion vector difference and the predicted scaling factor. The reconstruction unit reconstructs the coded block based on the derived MVD.

13. A non-transitory computer-readable storage medium, characterized in that, The system stores program code that, when executed by a computer, causes the computer to perform the method as described in any one of claims 1-10.

Images