Bilateral Matching for Synthesis Reference Mode

Abstract

The various implementations described herein include methods and systems for coding video. In one aspect, the method includes receiving a current coding block within a current image frame from a video bitstream and determining that the current coding block is to be predicted in a joint motion vector difference (JMVD) mode. The method further includes, in the JMVD mode, determining that the current coding block is associated with a forward prediction block and a backward prediction block, and refining a motion vector difference for at least the forward prediction block based on a cost criterion of a difference between the forward prediction block and the backward prediction block to determine a first refined motion vector difference. The method further includes generating a forward prediction block based on at least the first refined motion vector difference and determining motion compensation data for the current coding block based on the forward and backward prediction blocks.

Description

Technical Field

[0001] Related Applications This application is a continuation of and claims priority to U.S. Patent Application No. 18 / 196,372, filed on May 11, 2023, entitled "Bilateral Matching for Compound Reference Mode", and also claims priority to U.S. Provisional Patent Application No. 63 / 353,788, filed on Jun. 20, 2022, entitled "Bilateral Matching for Compound Reference Mode", the entire contents of which are incorporated herein by reference.

[0002] The disclosed embodiments generally relate to video coding, including, but not limited to, systems and methods for refining motion vector differences associated with two reference prediction blocks of a coding block in a bilateral matching scheme for inter-prediction of video data.

Background Art

[0003] Digital video is supported by a variety of electronic devices such as digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smartphones, video teleconference devices, video streaming devices. The electronic devices transmit and receive digital video data via a communication network or communicate in other ways, and / or store digital video data in a storage device. Since the bandwidth capacity of the communication network is limited and the memory resources of the storage device are limited, video coding may be used to compress the video data according to one or more video coding standards before the video data is communicated or stored.

[0004] Multiple video codec standards have been developed. For example, video coding standards include AOMedia Video 1 (AV1), Versatile Video Coding (VVC), Joint Exploration test Model (JEM), High-Efficiency Video Coding (HEVC / H.265), Advanced Video Coding (AVC / H.264), and Moving Picture Expert Group (MPEG) coding. Video coding generally utilizes prediction methods (e.g., inter prediction, intra prediction, etc.) that exploit redundancies inherent in video data. The purpose of video coding is to compress video data into a form that uses a lower bitrate while avoiding or minimizing a degradation in video quality.

[0005] HEVC, also known as H.265, is a video compression standard designed as part of the MPEG-H project. ITU-T and ISO / IEC published the HEVC / H.265 standard in 2013 (version 1), 2014 (version 2), 2015 (version 3), and 2016 (version 4). Versatile Video Coding (VVC), also known as H.266, is a video compression standard intended as a successor to HEVC. ITU-T and ISO / IEC published the VVC / H.266 standard in 2020 (version 1) and 2022 (version 2). AV1 is an open video coding format designed as an alternative to HEVC. On January 8, 2019, an effective version 1.0.0 including Errata1 of this specification was released. SUMMARY OF THE INVENTION MEANS FOR SOLVING THE PROBLEM

[0006] As described above, encoding (compression) reduces bandwidth and / or storage space requirements. As will be described in detail below, both reversible compression and irreversible compression can be employed. Reversible compression refers to a technique in which an exact copy of the original signal can be reconstructed from the compressed original signal via a decoding process. Irreversible compression refers to a coding / decoding process in which the original video information is not fully retained during coding and cannot be fully restored during decoding. When using irreversible compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original signal and the reconstructed signal is made small enough to be useful for the intended application of the reconstructed signal. The amount of acceptable distortion depends on the application. For example, users of certain consumer video streaming applications may tolerate higher distortion than users of movie or television broadcast applications. The compression ratio achievable by a particular coding algorithm can be selected or adjusted to reflect various distortion tolerances, and generally, the higher the distortion tolerance, the more likely it is that a coding algorithm that results in high loss and a high compression ratio is possible.

[0007] This disclosure describes refining the motion vector difference (MVD) of the motion vectors connecting a coding block and two reference prediction blocks in a bilateral matching method for inter-prediction of video data. Joint coding of MVDs is applied to generate motion compensation data for a current coding block based on two reference prediction blocks belonging to the same picture group (GOP) as the current coding block. In some situations, there is a linear motion between a backward reference block preceding the current coding block and a forward reference block following the current coding block. The joint MVD coding mode is implemented to signal joint MVDs for both of the two reference prediction blocks. For each reference block, respective MVDs are derived from the joint MVD based on the distance between the reference block and the current coding block. Conversely, in some situations, the motion between two reference frames is not linear and the speed of the motion optionally increases or decreases from the backward reference frame to the forward reference frame. In various embodiments of the present application, the bilateral matching method is applied to code a current coding block using a forward reference block identified by a forward motion vector and a backward reference block identified by a backward motion vector. Each of the forward and backward motion vectors includes a motion vector prediction (MVP) and an MVD. The MVD is scaled by a scale factor or changed by MVD refinement to produce a refined MVD. In some embodiments, the MVP and MVD of each motion vector are communicated between the encoder and the decoder, but the scale factor or MVD refinement of each motion vector is not signaled and needs to be determined separately at the encoder and the decoder.

[0008] According to some embodiments, a video coding method is provided. The method includes receiving, from a video bitstream, a current coding block within a current picture frame, and determining that the current coding block should be predicted in a joint motion vector difference (JMVD) mode. The method further includes, in response to the current coding block being predicted in the JMVD mode, determining that the current coding block is associated with two dual prediction blocks including a forward prediction block and a backward prediction block. The method further includes refining a motion vector difference for at least the forward prediction block based on a cost criterion of a difference between the forward prediction block and the backward prediction block to determine a first refined motion vector difference. The method further includes generating a forward prediction block based on at least the first refined motion vector difference, and determining motion compensation data for the current coding block based on the forward prediction block and the backward prediction block.

[0009] In some embodiments, the step of refining the motion vector difference for at least the forward prediction block further includes scaling the motion vector difference with a plurality of predefined scale factors to generate a plurality of scaled motion vector differences for the forward prediction block, determining that the difference between the forward prediction block and the backward prediction block corresponding to one of the plurality of scaled motion vector differences meets a cost criterion, identifying one of the plurality of scaled motion vector differences as a first refined motion vector difference, and determining the motion vector of the forward prediction block based on the motion vector predictor and the first refined motion vector difference. Alternatively, in some embodiments, the step of refining the motion vector difference for at least the forward prediction block further includes obtaining a plurality of predefined motion vector refinements for the motion vector difference of the forward prediction block, determining that the difference between the forward prediction block and the backward prediction block corresponding to one of the plurality of predefined motion vector refinements meets a cost criterion, identifying one of the plurality of predefined motion vector refinements as a first refined motion vector difference, and determining the motion vector of the forward prediction block based on the motion vector predictor, the motion vector difference, and the first refined motion vector difference.

[0010] According to some embodiments, a computing system such as a streaming system, a server system, a personal computer system, or other electronic device is provided. The computing system includes a control circuit and a memory storing one or more instruction sets. The one or more instruction sets include instructions for performing any of the methods described herein. In some embodiments, the computing system includes an encoder component and / or a decoder component.

[0011] According to some embodiments, a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium stores one or more sets of instructions for execution by a computing system. The one or more sets of instructions include instructions for performing any of the methods described herein.

[0012] Accordingly, devices and systems having a method for coding video are disclosed. Such methods, devices, and systems may complement or replace conventional methods, devices, and systems for video coding.

[0013] The features and advantages described herein are not necessarily all-inclusive, and in particular, some additional features and advantages will be apparent to those of ordinary skill in the art upon consideration of the drawings, specification, and claims provided in this disclosure. Further, note that the language used herein has been selected primarily for readability and for the purpose of explanation, and has not necessarily been selected to describe or limit the subject matter described herein.

[0014] To better understand the present disclosure, a more detailed description can be made by referring to the features of various embodiments, some of which are shown in the accompanying drawings. However, the accompanying drawings merely illustrate the relevant features of the present disclosure and should not necessarily be considered limiting since other effective features can be recognized by those of ordinary skill in the art upon reading the description of the present disclosure.

Brief Description of the Drawings

[0015]

Figure 1

Figure 2A

Figure 2B

Figure 3

Figure 4

Figure 5A

Figure 5B

Figure 6

Figure 7A

Figure 7B

Figure 8

[0016] By convention, the various features shown in the drawings are not necessarily drawn to scale, and like reference numbers can be used throughout the specification and drawings to indicate like features.

[0017] This disclosure describes refining the MVD of motion vectors connecting a coding block and two reference prediction blocks in a bilateral matching method for inter-prediction of video data. Joint coding of MVDs is applied to generate motion compensation data for a current coding block based on two reference prediction blocks, including a backward reference block preceding the current coding block within a GOP and a forward reference block following the current coding block within a GOP. In some situations, the motion between these two reference prediction blocks is linear. The joint MVD coding mode is implemented to signal the joint MVD for both of the two reference prediction blocks. For each reference block, the respective MVD is derived from the joint MVD based on the distance between the reference block and the current coding block. Conversely, in some situations, the motion between two reference frames is not linear, and the speed of motion optionally increases or decreases from the backward reference frame to the forward reference frame. In various embodiments of this application, the bilateral matching method is applied to code a current coding block using a forward reference block identified by a forward motion vector and a backward reference block identified by a backward motion vector. Each of the forward and backward motion vectors includes an MVP and an MVD. The MVD is scaled by a scale factor or changed by MVD refinement to produce a refined MVD. In some embodiments, the MVP and MVD of each motion vector are communicated in the bitstream of video data between the encoder and the decoder, but the scale factor or MVD refinement of each motion vector is not signaled and needs to be determined separately at the encoder and the decoder.

[0018] FIG. 1 is a block diagram showing a communication system 100 according to some embodiments. The communication system 100 includes a source device 102 and a plurality of electronic devices 120 (e.g., electronic devices 120-1 to electronic devices 120-m) communicatively coupled to each other via one or more networks. In some embodiments, the communication system 100 is a streaming system for use in video-enabled applications such as, for example, video conferencing applications, digital TV applications, and media storage and / or distribution applications.

[0019] The source device 102 includes a video source 104 (e.g., a camera component or a media storage) and an encoder component 106. In some embodiments, the video source 104 is a digital camera (e.g., configured to generate an uncompressed video sample stream). The encoder component 106 generates one or more coded video bitstreams from the video stream. The video stream from the video source 104 can be of a high data volume as compared to the encoded video bitstream 108 generated by the encoder component 106. The encoded video bitstream 108 has a lower data volume (less data) as compared to the video stream from the video source, and thus the encoded video bitstream 108 requires less bandwidth for transmission and less storage space for storage as compared to the video stream from the video source 104. In some embodiments, the source device 102 does not include the encoder component 106 (e.g., configured to transmit uncompressed video data to the network 110).

[0020] One or more networks 110 represent any number of networks that communicate information between a source device 102, a server system 112, and / or an electronic device 120, including, for example, wired (wired) and / or wireless communication networks. The one or more networks 110 may exchange data over circuit-switched channels and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet.

[0021] One or more networks 110 include a server system 112 (e.g., a distributed / cloud computing system). In some embodiments, the server system 112 is a streaming server or includes a streaming server configured to store and / or deliver video content (such as an encoded video stream from the source device 102). The server system 112 includes an encoder component 114 configured to encode and / or decode (e.g., video data). In some embodiments, the encoder component 114 includes an encoder component and / or a decoder component. In various embodiments, the encoder component 114 is instantiated as hardware, software, or a combination thereof. In some embodiments, the encoder component 114 is configured to decode an encoded video bitstream 108 and re-encode the video data using different encoding standards and / or methods to generate an encoded video data 116. In some embodiments, the server system 112 is configured to generate multiple video formats and / or encodings from the encoded video bitstream 108.

[0022] In some embodiments, server system 112 functions as a media - aware network element (MANE). For example, server system 112 may be configured to prune encoded video bitstream 108 to adapt potentially different bitstreams to one or more of electronic devices 120. In some embodiments, a MANE is provided separate from server system 112.

[0023] Electronic device 120 - 1 includes decoder component 122 and display 124. In some embodiments, decoder component 122 is configured to decode encoded video data 116 to generate an output video stream that can be rendered on a display or other type of rendering device. In some embodiments, one or more of electronic devices 120 do not include a display component (e.g., communicatively coupled to an external display device and / or including media storage). In some embodiments, electronic device 120 is a streaming client. In some embodiments, electronic device 120 is configured to access server system 112 to obtain encoded video data 116.

[0024] The source device and / or the plurality of electronic devices 120 may also be referred to as "terminal devices" or "user devices". In some embodiments, one or more of source device 102 and / or electronic devices 120 are instances of a server system, a personal computer, a portable device (e.g., a smartphone, a tablet, or a laptop), a wearable device, a video conferencing device, and / or other types of electronic devices.

[0025] In an exemplary operation of communication system 100, source device 102 transmits an encoded video bitstream 108 to server system 112. For example, source device 102 may encode a stream of pictures captured by the source device. Server system 112 receives encoded video bitstream 108 and may decode and / or encode encoded video bitstream 108 using coder component 114. For example, server system 112 may apply an optimal encoding to the video data for network transmission and / or storage. Server system 112 may transmit encoded video data 116 (e.g., one or more encoded video bitstreams) to one or more of electronic devices 120. Each electronic device 120 may decode encoded video data 116 to recover video pictures and optionally display them.

[0026] In some embodiments, the transmission described above is unidirectional data transmission. Unidirectional data transmission may be used, for example, in media serving applications. In some embodiments, the transmission described above is bidirectional data transmission. Bidirectional data transmission may be used, for example, in video conferencing applications. In some embodiments, encoded video bitstream 108 and / or encoded video data 116 are encoded and / or decoded according to any of the video coding / compression standards described herein, such as HEVC, VVC, and / or AV1.

[0027] FIG. 2A is a block diagram showing exemplary elements of an encoder component 106 according to some embodiments. The encoder component 106 receives a source video sequence from a video source 104. In some embodiments, the encoder component includes a receiver (e.g., transceiver) component configured to receive the source video sequence. In some embodiments, the encoder component 106 receives the video sequence from a remote video source (e.g., a video source that is a component of a device different from the encoder component 106). The video source 104 may provide the source video sequence in the form of a digital video sample stream that can be of any suitable bit depth (e.g., 8-bit, 10-bit, or 12-bit), any color space (e.g., BT.601 Y CrCb, or RGB), and any suitable sampling structure (e.g., Y CrCb 4:2:0 or Y CrCb 4:4:4). In some embodiments, the video source 104 is a storage device that stores previously captured / prepared video. In some embodiments, the video source 104 is a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures that convey motion when viewed in sequence. The pictures themselves may be organized as a spatial array of pixels, and each pixel may include one or more samples depending on the sampling structure, color space, etc. in use. One of ordinary skill in the art can easily understand the relationship between a pixel and a sample. The following description focuses on samples.

[0028] The encoder component 106 is configured to encode and / or compress pictures of the source video sequence into the coded video sequence 216 in real time or under other timing constraints required by the application. Implementing an appropriate coding speed is one function of the controller 204. In some embodiments, the controller 204 controls and is functionally coupled to other functional units described below. The parameters set by the controller 204 may include rate control related parameters (e.g., picture skip, quantizer, and / or lambda value of rate distortion optimization techniques), picture size, Group of Pictures (GOP) layout, maximum motion vector search range, etc. One of ordinary skill in the art can readily identify other functions of the controller 204 that may be related to the encoder component 106 optimized for a particular system design.

[0029] In some embodiments, the encoder component 106 is configured to operate in a coding loop. In a simplified example, the coding loop includes a source coder 202 (which, for example, is responsible for generating symbols such as a symbol stream based on an input picture and a reference picture to be coded) and a (local) decoder 210. The decoder 210 reconstructs the symbols to generate sample data, similar to a (remote) decoder when the compression between the symbols and the coded video bitstream is reversible. The reconstructed sample stream (sample data) is input into the reference picture memory 208. Since the decoding of the symbol stream results in bit-exact results regardless of the position of the decoder (local or remote), the content of the reference picture memory 208 is also bit-exact between the local encoder and the remote encoder. In this way, the prediction unit of the encoder interprets the same sample values as the reference picture samples that the decoder interprets when using prediction during decoding. This principle of reference picture synchronization (and, for example, the resulting drift if synchronization cannot be maintained due to channel errors) is known to those skilled in the art.

[0030] The operation of the decoder 210 can be the same as that of a remote decoder such as the decoder component 122, which will be described in detail below in relation to FIG. 2B. However, referring briefly to FIG. 2B, since the symbols are available and the encoding / decoding of the symbols into the coded video sequence by the entropy coder 214 and the parser 254 can be reversible, the entropy decoding portion of the decoder component 122, which includes the buffer memory 252 and the parser 254, may not be fully implemented in the local decoder 210.

[0031] What can be said at this point is that any decoder technology other than the parsing / entropy decoding existing within the decoder must also necessarily exist in a substantially identical functional form in the corresponding encoder. For this reason, the subject matter of the present disclosure focuses on the operation of the decoder. The description of the encoder technology can be omitted because the encoder technology is the reverse of the decoder technology described comprehensively. A more detailed description is necessary only in specific areas, which will be shown below.

[0032] As part of its operation, the source coder 202 can perform motion compensation prediction coding, which predictively codes the input frame by referring to one or more previously coded frames from the video sequence designated as the reference frame. In this way, the coding engine 212 codes the difference between the pixel block of the input frame and the pixel block of the reference frame that can be selected as a prediction reference for the input frame. The controller 204 can manage the coding operation of the source coder 202, including, for example, setting the parameters and subgroup parameters used for encoding the video data.

[0033] Decoder 210 decodes the coded video data of a frame that can be specified as a reference frame based on the symbols created by source coder 202. The operation of coding engine 212 may advantageously be an irreversible process. When the coded video data is decoded by a video decoder (not shown in FIG. 2A), the reconstructed video sequence can typically be a replica of the source video sequence with some errors. Decoder 210 can reproduce the decoding process that can be performed by a remote video decoder for the reference frame and store the reconstructed reference frame in reference picture memory 208. In this way, encoder component 106 locally stores a copy of the reconstructed reference frame having common content as the reconstructed reference frame that would be obtained by a remote video decoder (without transmission errors).

[0034] Predictor 206 may perform a prediction search for coding engine 212. That is, for a new frame to be coded, predictor 206 may seek specific metadata such as sample data (as candidate reference pixel blocks) or reference picture motion vectors, block shapes, etc. that can serve as appropriate prediction references for the new picture, and search reference picture memory 208. Predictor 206 may operate on sample blocks for each pixel block to find an appropriate prediction reference. In some cases, the input picture may have prediction references drawn from a plurality of reference pictures stored in reference picture memory 208 as determined by the search results obtained by predictor 206.

[0035] The output of all the aforementioned functional units may undergo entropy coding within the entropy coder 214. The entropy coder 214 converts the symbols generated by the various functional units into a coded video sequence by reversibly compressing the symbols according to techniques known to those skilled in the art (e.g., Huffman coding, variable-length coding, and / or arithmetic coding).

[0036] In some embodiments, the output of the entropy coder 214 is coupled to a transmitter. The transmitter may be configured to buffer the coded video sequence generated by the entropy coder 214 and prepare for transmission via a communication channel 218, which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter may be configured to merge the coded video data from the source coder 202 with other data to be transmitted, such as coded audio data and / or an auxiliary data stream (source not shown). In some embodiments, the transmitter may transmit additional data along with the encoded video. The source coder 202 may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, supplementary enhancement information (SEI) messages, visual user utility information (VUI) parameter set fragments, and the like.

[0037] Controller 204 may manage the operation of encoder component 106. During coding, controller 204 may assign a specific coded picture type to each coded picture, which may affect the coding technique applied to each picture. For example, a picture may be assigned as an intra picture (I picture), a predicted picture (P picture), or a bi-directionally predicted picture (B picture). An intra picture may be coded and decoded without using any other frame in the sequence as a source of prediction. Some video codecs allow for various types of intra pictures, such as, for example, an independent decoder refresh (IDR) picture. Those skilled in the art are familiar with those variations of I pictures as well as their respective uses and characteristics, and thus they will not be repeated here. A predicted picture may be coded and decoded using intra prediction or inter prediction that uses up to one motion vector and a reference index to predict the sample values of each block. A bi-directionally predicted picture may be coded and decoded using intra prediction or inter prediction that uses up to two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple predicted pictures may use three or more reference pictures and associated metadata for the reconstruction of a single block.

[0038] The source picture is generally spatially subdivided into a plurality of sample blocks (e.g., blocks of 4×4 samples, 8×8 samples, 4×8 samples, or 16×16 samples each), and can be coded block by block. The blocks can be coded predictively by referring to other (already coded) blocks as determined by the coding assignment applied to each block of the picture. For example, blocks of an I picture may be coded non-predictively, or may be coded predictively by referring to already coded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of a P picture can be coded non-predictively via spatial prediction or via temporal prediction referring to one previously coded reference picture. Pixel blocks of a B picture can be coded non-predictively via spatial prediction or via temporal prediction referring to one or two previously coded reference pictures.

[0039] Video can be captured in time series as a plurality of source pictures (video pictures). Intra picture prediction (often abbreviated as intra prediction) uses the spatial correlation within a given picture, and inter picture prediction uses the (temporal or other) correlation between pictures. In one example, a particular picture being encoded / decoded, called the current picture, is divided into blocks. When a block within the current picture is similar to a reference block within a reference picture that has been previously coded and is still buffered in the video, the block within the current picture can be coded by a vector called a motion vector. The motion vector points to the reference block within the reference picture and can have a third dimension identifying the reference picture if multiple reference pictures are being used.

[0040] The encoder component 106 can perform a coding operation according to a predetermined video coding technique or standard, such as any of those described herein. In its operation, the encoder component 106 can perform various compression operations, including a predictive coding operation that utilizes the temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard being used.

[0041] FIG. 2B is a block diagram showing exemplary elements of a decoder component 122 according to some embodiments. The decoder component 122 of FIG. 2B is coupled to a channel 218 and a display 124. In some embodiments, the decoder component 122 includes a transmitter configured to transmit data to the display 124 (e.g., via a wired or wireless connection) and is coupled to a loop filter 256.

[0042] In some embodiments, decoder component 122 includes a receiver configured to be coupled to channel 218 and receive data from channel 218 (e.g., via a wired or wireless connection). The receiver may be configured to receive one or more coded video sequences to be decoded by decoder component 122. In some embodiments, the decoding of each coded video sequence is independent of other coded video sequences. Each coded video sequence may be received from channel 218, which may be a hardware / software link to a storage device storing the encoded video data. The receiver may receive the encoded video data along with other data, such as coded audio data and / or auxiliary data streams, and that data may be transferred to their respective usage entities (not shown). The receiver may separate the coded video sequence from the other data. In some embodiments, the receiver receives additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by decoder component 122 for decoding the data and / or for more accurately reconstructing the original video data. The additional data may be in the form of, for example, temporal layers, spatial layers, or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0043] According to some embodiments, decoder component 122 includes buffer memory 252, a parser 254 (which may also be referred to as an entropy decoder), a scaler / inverse transform unit 258, an intra-picture prediction unit 262, a motion compensation prediction unit 260, an aggregator 268, a loop filter unit 256, a reference picture memory 266, and a current picture memory 264. In some embodiments, decoder component 122 is implemented as an integrated circuit, a series of integrated circuits, and / or other electronic circuits. In some embodiments, decoder component 122 is implemented at least partially in software.

[0044] Buffer memory 252 is coupled between channel 218 and parser 254 (e.g., to counter network jitter). In some embodiments, buffer memory 252 is separate from decoder component 122. In some embodiments, a separate buffer memory is provided between the output of channel 218 and decoder component 122. In some embodiments, in addition to buffer memory 252 within decoder component 122 (e.g., configured to handle layout timing), a separate buffer memory is provided external to decoder component 122 (e.g., to counter network jitter). When receiving data from a storage / transfer device with sufficient bandwidth and controllability or from an isochronous network, buffer memory 252 may be unnecessary or can be made small. For use in a best-effort packet network such as the Internet, buffer memory 252 may be required, and buffer memory 252 may be relatively large, advantageously of an adaptable size, and may be implemented at least partially in an operating system or similar element (not shown) external to decoder component 122.

[0045] Parser 254 is configured to reconstruct symbol 270 from the coded video sequence. The symbol may include, for example, information used to manage the operation of decoder component 122 and / or information for controlling a rendering device such as display 124. The control information for the rendering device may be in the form of a Supplementary Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). Parser 254 analyzes (entropy decodes) the coded video sequence. The coding of the coded video sequence can follow a video coding technology or standard, and can follow principles well-known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context dependence, etc. Parser 254 can extract a set of subgroup parameters regarding at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. The subgroup can include, for example, a Group of Pictures (GOP), picture, tile, slice, macroblock, coding unit (CU), block, transform unit (TU), prediction unit (PU), etc. Parser 254 may also extract information such as transform coefficients, quantization parameter values, motion vectors, etc. from the coded video sequence.

[0046] The reconstruction of symbol 270 can involve multiple different units depending on the type of the coded video picture or a portion thereof (such as inter-picture and intra-picture, inter-block and intra-block, etc.), and other factors. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the video sequence coded by parser 254. Such a flow of subgroup control information between parser 254 and the following multiple units is not depicted for the sake of explanation.

[0047] In addition to the function blocks already mentioned, decoder component 122 can be conceptually subdivided into several functional units as described below. In an actual implementation operating under commercial constraints, many of these units interact closely with each other and can be at least partially integrated with each other. However, for the purpose of explaining the disclosed subject matter, the conceptual subdivision into the following functional units is maintained.

[0048] Scaler / inverse transform unit 258 receives the quantized transform coefficients and control information as symbol 270 (e.g., which transform to use, block size, quantization coefficients, and / or quantization scaling matrix) from parser 254. Scaler / inverse transform unit 258 can output a block containing sample values that can be input to aggregator 268.

[0049] In some cases, the output samples of the scaler / inverse transform unit 258 are related to blocks that are intra-coded, i.e., blocks that do not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information can be provided by the intra-picture prediction unit 262. The intra-picture prediction unit 262 can generate a block of the same size and shape as the block being reconstructed, using surrounding already-reconstructed information fetched from the current (partially reconstructed) picture in the current picture memory 264. The aggregator 268 can add, for each sample, the prediction information generated by the intra-picture prediction unit 262 to the output sample information provided by the scaler / inverse transform unit 258.

[0050] In other cases, the output samples of the scaler / inverse transform unit 258 are related to inter-coded and potentially motion-compensated blocks. In such cases, the motion-compensation prediction unit 260 can access the reference picture memory 266 to fetch the samples used for prediction. After motion-compensating the samples fetched according to the symbol 270 related to the block, these samples can be added by the aggregator 268 to the output of the scaler / inverse transform unit 258 to generate the output sample information (in this case, called residual samples or a residual signal). The address in the reference picture memory 266 where the motion-compensation prediction unit 260 fetches the prediction samples can be controlled by a motion vector. The motion vector can be available to the motion-compensation prediction unit 260, for example, in the form of a symbol 270 that can have X, Y, and reference picture components. Motion compensation can also include interpolation of the fetched sample values, a motion vector prediction mechanism, etc., from the reference picture memory 266 when an exact motion vector with sub-samples is used.

[0051] The output samples of the aggregator 268 can undergo various loop filtering techniques in the loop filter unit 256. The video compression technology is controlled by parameters included in the coded video bitstream and can include in-loop filter techniques made available in the loop filter unit 256 as symbols 270 from the parser 254, but can also respond to meta information obtained during the decoding of previous (in decoding order) portions of the coded picture or coded video sequence, or to previously reconstructed and loop-filtered sample values.

[0052] The output of the loop filter unit 256 can be output to a rendering device such as the display 124 and can be made into a sample stream that can also be stored in the reference picture memory 266 for use in future inter-picture prediction.

[0053] When a particular coded picture is completely reconstructed, it can be used as a reference picture for future prediction. When the coded picture is completely reconstructed and the coded picture is identified as a reference picture (e.g., by the parser 254), the current reference picture can become part of the reference picture memory 266, and a new current picture memory can be reallocated before starting the reconstruction of the next coded picture.

[0054] The decoder component 122 can perform a decoding operation according to a predetermined video compression technology that can be documented by a standard such as any of the standards described in this specification. The coded video sequence can conform to the syntax specified by the video compression technology or standard being used, in the sense that it is faithful to the syntax of the video compression technology or standard, as specified in the video compression technology document or standard, specifically the profile document therein. Also, in order to conform to some video compression technologies or standards, the complexity of the coded video sequence can be within the range defined by the level of the video compression technology or standard. In some cases, the level restricts, for example, the maximum picture size, the maximum frame rate, the maximum reconstructed sample rate (measured, for example, in megasamples per second), the maximum reference picture size, etc. The limits imposed by the level can, in some examples, be further restricted by the specifications of the Hypothetical Reference Decoder (HRD) and the metadata for HRD buffer management signaled in the coded video sequence.

[0055] Figure 3 is a block diagram showing a server system 112 according to some embodiments. The server system 112 includes a control circuit 302, one or more network interfaces 304, a memory 314, a user interface 306, and one or more communication buses 312 for interconnecting these components. In some embodiments, the control circuit 302 includes one or more processors (e.g., a CPU, a GPU, and / or a DPU). In some embodiments, the control circuit includes one or more field programmable gate arrays (FPGAs), hardware accelerators, and / or one or more integrated circuits (e.g., application specific integrated circuits).

[0056] The network interface 304 may be configured to form an interface with one or more communication networks (e.g., wireless, wired, and / or optical networks). The communication networks can be local, wide area, metropolitan, vehicle and industrial, real-time, delay tolerant, etc. Examples of communication networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., and wired or wireless wide area digital networks for television including cable television, satellite television, and terrestrial television, and vehicle and industrial including CANBus. Such communication can be unidirectional, receive only (e.g., broadcast television), transmit only unidirectionally (e.g., CANbus to a specific CANbus device), or bidirectional (e.g., using a local or wide area digital network to other computer systems). Such communication can include communication to one or more cloud computing networks.

[0057] The user interface 306 includes one or more output devices 308 and / or one or more input devices 310. The input device 310 can include one or more of a keyboard, mouse, trackpad, touch screen, data glove, joystick, microphone, scanner, camera, etc. The output device 308 can include one or more of an audio output device (e.g., speaker), a visual output device (e.g., display or monitor), etc.

[0058] Memory 314 may include a high-speed random access memory (such as DRAM, SRAM, DDR RAM, and / or other random access solid-state memory devices) and / or a non-volatile memory (such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, and / or other non-volatile solid-state storage devices). Memory 314 may optionally include one or more storage devices that are remotely located from control circuit 302. Memory 314, or alternatively a non-volatile solid-state memory device within memory 314, includes a non-transitory computer-readable storage medium. In some embodiments, memory 314, or the non-transitory computer-readable storage medium of memory 314, stores the following programs, modules, instructions, and data structures, or subsets or supersets thereof: ● An operating system 316 that processes various basic system services and includes procedures for performing hardware-dependent tasks; ● A network communication module 318 used to connect server system 112 to other computing devices via one or more network interfaces 304 (e.g., via wired and / or wireless connections); ● A coding module 320 that performs various functions related to encoding and / or decoding data such as video data. In some embodiments, coding module 320 is an instance of coder component 114. Coding module 320 includes, but is not limited to, one or more of the following: 〇 A decoding module 322 for performing various functions related to decoding encoded data, as described above with respect to decoder component 122; and 〇 An encoding module 340 for performing various functions related to encoding data, as described above with respect to encoder component 106; and ● A picture memory 352 for storing pictures and picture data for use, for example, with a coding module 320. In some embodiments, the picture memory 352 includes one or more of a reference picture memory 208, a buffer memory 252, a current picture memory 264, and a reference picture memory 266.

[0059] In some embodiments, the decoding module 322 includes a syntax analysis module 324 (configured to perform various functions as described above with respect to the parser 254), a conversion module 326 (configured to perform various functions as described above with respect to the scalar / inverse transform unit 258), a prediction module 328 (configured to perform various functions as described above with respect to the motion compensation prediction unit 260 and / or the intra-picture prediction unit 262), and a filter module 330 (configured to perform various functions as described above with respect to the loop filter 256).

[0060] In some embodiments, the encoding module 340 includes a code module 342 (configured to perform various functions as described above with respect to the source coder 202 and / or the coding engine 212), and a prediction module 344 (configured to perform various functions as described above with respect to the predictor 206). In some embodiments, the decoding module 322 and / or the encoding module 340 includes a subset of the modules shown in FIG. 3. For example, a shared prediction module is used by both the decoding module 322 and the encoding module 340.

[0061] Each of the identified modules stored in the memory 314 corresponds to a set of instructions for performing the functions described herein. The modules identified above (e.g., sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus, various subsets of these modules may be combined or otherwise rearranged in various embodiments. For example, the coding module 320 optionally does not include separate decoding and encoding modules, but rather uses the same set of modules to perform both sets of functions. In some embodiments, the memory 314 stores a subset of the modules and data structures identified above. In some embodiments, the memory 314 stores additional modules and data structures not described above, such as an audio processing module.

[0062] In some embodiments, the server system 112 includes a web server, a Hypertext Transfer Protocol (HTTP) server, a File Transfer Protocol (FTP) server, a Common Gateway Interface (CGI) script, a PHP Hyper-text Preprocessor (PHP), an Active Server Pages (ASP), a Hyper Text Markup Language (HTML), an Extensible Markup Language (XML), Java, JavaScript, an Asynchronous JavaScript and XML (AJAX), XHP, Javelin, a Wireless Universal Resource File (WURFL), and web pages and applications implemented using the same.

[0063] FIG. 3 shows a server system 112 according to some embodiments, but FIG. 3 is not a structural schematic diagram of the embodiments described herein and is intended as a functional description of various features that may exist in one or more server systems. In practice, and as will be recognized by those skilled in the art, it is possible for separately shown items to be combined and for some items to be separated. For example, some of the items separately shown in FIG. 3 can be implemented on a single server, and a single item can be implemented by one or more servers. The actual number of servers used to implement the server system 112 and how the functions are allocated among them will vary from implementation to implementation and, optionally, will depend in part on the amount of data traffic that the server system processes during peak and average usage periods.

[0064] FIG. 4 is a schematic diagram of a bilateral matching configuration 400 that associates a current coding block 404 within a GOP with two reference prediction blocks 408A and 408B located within the same GOP based on two different motion vectors 410, according to some embodiments. A GOP includes a sequence of image frames. The sequence of image frames includes a current image frame 402 that further includes the current coding block 404. The current coding block 404 is encoded based on prediction data of one or more coding blocks of one or more reference images within the GOP. In some embodiments, bilateral matching is applied to encode the current coding block 404. For example, the current coding block 404 is encoded based on prediction data of two reference prediction blocks 408A and 408B of two reference frames 406A and 406B within the GOP. In one example, the current coding block 404 is temporally positioned between a first prediction block 408A (e.g., a forward or backward prediction block) of a first reference frame 406A and a second prediction block 408B (e.g., a backward or forward prediction block) of a second reference frame 406B. One of the reference prediction blocks 408A and 408B precedes the current coding block 404, and the other of the reference prediction blocks 408A and 408B follows the current coding block 404. Each reference frame 406A or 406B is either the very next frame after the current image frame 402 or is separated from the current image frame 402 by some number of frames. Specifically, the first prediction block 408A optionally precedes or follows the current coding block 404 and optionally is directly adjacent to the current coding block 404 or is separated from the current coding block 404 by some number of frames.

[0065] In some embodiments, the current coding block 404 is coded based on two reference prediction blocks 408A and 408B in translational motion. Alternatively, in some embodiments, the current coding block 404 is coded based on two reference prediction blocks 408A and 408B in warped motion. The motion vector 410A or 410B of the current coding block 404 is adjusted based on the warping motion for each sub-block or sample of the current coding block 404 before the motion vector 410A or 410B is used to code or predict a sub-block or sample based on the respective reference sub-blocks or samples of the prediction blocks 408A or 408B. In some embodiments associated with bilateral matching, the MVD of the motion vector 410A or 410B is adaptively determined in both translational and warped motion modes. The motion vector 410A or 410B of the current coding block 404 is a combination of a motion vector prediction MVP (e.g., MVP A or MVP B ) and an MVD (e.g., MVD A or MVD¬ B ). For each of the two reference prediction blocks 408, the MVP is fixed, the MVD is adaptively encoded, transferred in the bitstream, and decoded for the current coding block 404. In some embodiments, the MVD A or MVD B is determined based on a predefined MVD adaptation region (not shown) that includes the prediction blocks 408A or 408B identified based on the MVP on the reference frame 406A or 406B, respectively.

[0066] In some embodiments, the joint MVD coding mode is implemented to signal a joint MVD for both of the two reference prediction blocks 408A and 408B. For each prediction block 408A or 408B, respective MVDs (MVD A or MVD B ) are derived from the joint MVD, for example based on the distance between the reference block and the current coding block. In one example, the MVDs of the prediction blocks 408A and 408B are equal to each other.

[0067] In some embodiments, the current coding block 404 of the current image frame 402 has two reference prediction blocks including a first prediction block 408A and a second prediction block 408B. Based on a cost criterion of the difference between the first prediction block 408A and the second prediction block 408B, the MVD is refined for at least the first prediction block 408B, and a first refined MVD (RMVD A ) is determined. The first prediction block 408A is defined based on at least the first refined MVD (RMVD A ). The motion compensation data of the current coding block 404 is determined based on the first prediction block 408A and the second prediction block 408B. In some embodiments, the refined MVD for determining the first refined MVD (RMVD A ) includes a first MVD (MVD A ) associated with the first prediction block 408A. Alternatively, in some embodiments, the refined MVD for determining the first refined MVD (RMVD A ) includes a second MVD (MVD B ) used by the second prediction block 408B. In some embodiments, such MVD refinement is implemented in both the encoder 106 and the decoder 122. The first refined MVD (RMVD A ) is not transmitted together with the MVD and is determined separately during MVD refinement in both the encoder 106 and the decoder 122.

[0068] In some embodiments, the refinement of the MVD is implemented for the first prediction block 408A based on a first MVD refinement region 412A that includes a predicted block 408A identified on a first reference frame 406A based on a first MVP (MVP A ) and a first MVD (MVD A ).

[0069] The difference between prediction blocks 408A and 408B is determined and applied as a cost to further determine whether a cost criterion is met. In one example, a first refined MVD (e.g., RMVD A ) that meets the cost criterion is used to determine a first motion vector 410A for the current coding block 404. In some embodiments, the first refined MVD (RMVD A ) is determined for the first prediction block 408A of the first reference frame 406A and further applied to determine a second refined MVD (RMVD B ) for the second reference frame 406B. In one example, the second refined MVD (RMVD B ) is mirrored from the first refined MVD (RMVD A ) based on the distances between the first and second reference frames 406A and 406B and the current image frame 402. In some embodiments, the difference between the first prediction block 408A and the second prediction block 408B associated with the cost criterion includes one of a sum of absolute differences (SAD), a sum of squared errors (SAE), a sum of absolute transform differences (SATD), and a sum of mean removed SAD.

[0070] In some embodiments, the second motion vector 410B of the second prediction block 408B is fixed and the MVD (MVD A ) is refined for the first prediction block 408A based on the cost criterion.

[0071] In some embodiments, based on a cost criterion of a difference between a first prediction block 408A and a second prediction block 408B, an MVD (e.g., MVD A or MVD B ) is refined for the second prediction block 408B, and a second refined MVD (RMVD B ) is determined sequentially, or iteratively, for example, simultaneously with a refinement of a first refined MVD (RMVD A ) of the first prediction block 408A. The second prediction block 408B is generated based on the second refined MVD (RMVD B ). In some embodiments, the refinement of the MVD is implemented for the second prediction block 408B based on a second MVD refinement region 412B including a prediction block 408B identified on a second reference frame 406B based on a second MVP (MVP B ) and a second MVD (MVD B ).

[0072] In some embodiments, a current coding block 404 is used as a prediction block for a next coding block and is identified for the next coding block based on a next motion vector including a next MVD. The next MVD is not refined for encoding and decoding the next coding block. In other words, if a next MVD identifying the current coding block 404 needs to be refined during encoding and decoding of the next coding block, the current coding block is not used as a prediction block for the next coding block.

[0073] In some embodiments, a current image frame 402 includes a motion vector predictor that identifies an MVP. A first motion vector 410A is based on the motion vector predictor and a first refined MVD (RMVD A) is determined based on and applied to generate the first prediction block 408A. The second motion vector 410B is also determined based on the motion vector predictor and the MVD and is applied to determine the second prediction block 408B within the second reference frame 406B. In some embodiments, the MVD is refined for the first prediction block 408A according to (1) the determination that the current coding block 404 has one of a group of composite reference modes consisting of NEW_NEARMV, NEW_NEARMV_OPTL, NEAR_NEWMV, and NEAR_NEWMV_OPTL, or (2) the determination that the MVD is obtained for the second prediction block 408B and not obtained for the first prediction block 408A.

[0074] Specifically, in some embodiments associated with NEW_NEARMV or its variant NEW_NEARMV_OPTL, the motion vector predictor is signaled by an MVP index to select a motion vector prediction (MVP) within a predefined list of MVPs. The first MVD (MVD A ) is sent to the decoder 122 in the bitstream together with the motion vector predictor. In some embodiments associated with NEW_NEWMV or its variant NEW_NEWMV_OPTL, the motion vector predictor is signaled by an MVP index to select an MVP within a predefined list of MVPs. The first MVD (MVD A ) and the second MVD (MVD B ) are sent to the decoder 122 in the bitstream together with the motion vector predictor.

[0075] In some embodiments, according to the determination that the current coding block has one of the groups of combined reference modes consisting of NEAR_NEAR_MV and NEAR_NEAR_OPTL, the first prediction block 408A is identified among two reference prediction blocks 408A and 408B based on conditions before the MVD is refined for the first prediction block 408A. For example, according to the conditions, one of the two reference prediction blocks 408A and 408B closer to the current image frame 402 is selected as the first prediction block 408A for MVD refinement. The first MVD (MVD A ) refined to generate the first refined MVD (RMVD A ) optionally precedes or follows the current coding block 404. That is, MVD refinement may be applied to either the backward prediction block or the forward prediction block in the bilateral matching method of inter prediction of video data.

[0076] FIG. 5A is a schematic diagram of an exemplary bilateral matching configuration 500 in which the MVD associated with the reference prediction block 408A or 408B is refined according to some embodiments, and FIG. 5B shows an exemplary process 550 for refining the MVD associated with the reference prediction block 408A or 408B within the MVD refinement region 412A according to some embodiments. The current coding block 404 of the current image frame 402 has two reference prediction blocks including a first prediction block 408A and a second prediction block 408B. Based on the cost criterion of the difference between the first prediction block 408A and the second prediction block 408B, the MVD is refined at least for the first prediction block 408B, and the first refined MVD (RMVD A ) is determined. The first prediction block 408A is defined based at least on the first refined MVD (RMVD A ). The motion compensation data of the current coding block 404 is determined based on the first prediction block 408A and the second prediction block 408B. The first refined MVD (RMVDA ) The refined MVD for determining is optionally the first MVD (MVD A ) associated with the first prediction block, the second MVD (MVD B ) associated with the second prediction block 408B, or the joint MVD used in the joint MVD coding mode. Note that in some embodiments, such MVD refinement is implemented in both the encoder 106 and the decoder 122. The MVD is adaptively determined and transmitted to the decoder 122. The first refined MVD (RMVD A ) is not transmitted with the MVD and is determined separately during MVD refinement in both the encoder 106 and the decoder 122.

[0077] In some embodiments, the first prediction block 408A is the first motion vector MV A which is the sum of the first MVP (MVP A ) and the first refined MVD (RMVD A ). The MVD is scaled by a scale factor K A to generate the first refined MVD (RMVD A ). The scale factor K Ais identified in the exemplary MVD refinement process 550. Referring to FIG. 5B, in some embodiments, the MVD is refined with a plurality of predefined scale factors K to generate a plurality of scaled MVDs for the first prediction block 408A. For example, the plurality of scaled MVDs include PR1 corresponding to a first scale factor (K = 0.5), PD corresponding to a unit scale factor (K = 1), PR2 corresponding to a second scale factor (K = 1.5), and PR3 corresponding to a third scale factor (K = 2). In some embodiments, the plurality of predefined scale factors correspond to a scale factor range (e.g., 0.5 to 2), and the resulting prediction block 408A is controlled within the MVD refinement range 412A. The difference between the first prediction block 408A and the second prediction block 408B corresponding to one of the plurality of scaled MVDs (e.g., PD) meets the cost criterion. In some embodiments, the difference between the first prediction block 408A and the second prediction block 408B corresponding to one of the plurality of scaled MVDs (e.g., PR2 corresponding to K = 1.5) is smaller than the difference between the prediction blocks 408A and 408B corresponding to any other scaled MVD (e.g., PD, PR1, and PR3). One of the plurality of scaled MVDs is identified as the first refined MVD (RMVD A ) of the first prediction block 408A. By these means, the first motion vector 410A of the first prediction block 408B is further determined based on the motion vector predictor (e.g., MVP A ) and the first refined MVD (RMVD A ).

[0078] More specifically, in some embodiments, a plurality of differences between the first prediction block 408A and the second prediction block 408B are determined for a plurality of pre-defined scale factors. Each difference corresponds to one of the plurality of scaled MVDs of the first prediction block 408A. The plurality of differences are compared to each other to determine whether each difference satisfies a cost criterion. Further, in some embodiments, one or more pre-defined scale factors (e.g., K = 2) are more or less preferred than other scale factors. A first subset of the differences between the prediction blocks 408A and 408B is determined based on a first subset of the pre-defined scale factors of the MVD. Before comparing the plurality of differences, each of the first subset of differences is scaled by a respective difference factor. In some embodiments, the difference factor is less than 1, and the first subset of the pre-defined scale factors is more preferred compared to the remaining pre-defined scale factors. For example, the difference between the prediction blocks 408A and 408B corresponding to K = 2 is scaled by 0.8 and is likely to be smaller than the unscaled or differently scaled differences with a difference factor greater than 1. Conversely, in some embodiments, the difference factor is less than 1, and the first subset of the pre-defined scale factors is less preferred compared to the remaining pre-defined scale factors.

[0079] FIG. 6 is a diagram illustrating an exemplary process 600 for refining the MVD associated with the reference prediction block 408A based on a scale factor range and a scaling index, according to some embodiments. The current image frame 402 includes a scaling index for refining the MVD of the prediction block 408A. The scaling index identifies the scale factor K in the ordered sequence 602 of scale factors. The encoder 106 encodes the current image frame 402 with the scaling index. Based on the scaling index, the decoder 122 selects the scale factor K within the ordered sequence 602 of scale factors, and the selected scale factor K A is identified. The encoder 106 encodes the current image frame 402 with the scaling index. Based on the scaling index, the decoder 122 selects the scale factor K within the ordered sequence 602 of scale factors, and the selected scale factor K A is selected, and the selected scale factor K ARefine the MVD of the prediction block 408A based on this, and generate a motion vector corresponding to the prediction block 408A based on the refined first MVD (RMVD A ).

[0080] In some embodiments, the MVD is scaled by a plurality of predefined scale factors 604 (e.g., K = 1, 1 / 2, 1 / 3, 2, 1.5, 3, and 4) to generate a plurality of scaled MVDs for the first prediction block 408A. A plurality of differences between the first prediction block 408A and the second prediction block 408B are determined, and each difference corresponds to one of the respective predefined scale factors and one of the plurality of scaled MVDs of the first prediction block 408A. A subset of the predefined scale factors is arranged in an ordered sequence 602 of scale factors based on the plurality of differences. In some embodiments, the ordered sequence 604 of scale factors is not transmitted from the encoder 106 to the decoder 122. The current coding block 404 is associated with a scaling index (e.g., 3) that identifies the scale factor K A within the ordered sequence 604 of scale factors. Based on the scaling index, one of the subset of the predefined scale factors K A is selected to scale the MVD and generate the first refined MVD (RMVD A ).

[0081] FIG. 7A is a schematic diagram of another exemplary bilateral matching configuration 700 in which the MVD of reference prediction block 408A or 408B is refined according to some embodiments, and FIG. 7B shows another exemplary process 750 for refining the MVD of reference prediction block 408A or 408B within MVD refinement region 412A according to some embodiments. The current coding block 404 of the current image frame 402 has two reference prediction blocks including a first prediction block 408A and a second prediction block 408B. Each prediction block 408A or 408B optionally precedes or follows the current coding block 404 within the GOP. Based on the cost criterion of the difference between prediction blocks 408A and 408B, the MVD is refined for at least the first prediction block 408B to determine a first refined MVD (RMVD A ). The first prediction block 408A is defined based on at least the first refined MVD (RMVD A ) and is applied with the second prediction block 408B to determine the motion compensation data of the current coding block 404. In some embodiments, the MVD is adaptively determined and transmitted from the encoder 106 to the decoder 122 (FIG. 1), but the first refined MVD (RMVD A ) is not transmitted with the MVD. The first refined MVD (RMVD A ) is determined separately during MVD refinement in both the encoder 106 and the decoder 122.

[0082] In some embodiments, the first prediction block 408A is determined based on a first motion vector MV A that is the sum of a first MVP (MVP A ), a first MVD (MVD A ), and a first refined MVD (RMVD A ). The first prediction block 408A is the first motion vector MV A￢It is refined in the MVD refinement region 412A of the first reference frame 406A to identify. Referring to FIG. 7B, in some embodiments, to refine the MVD of the first prediction block 408A, a plurality of predefined motion vector refinements R1 to R6 are identified, for example, in the MVD refinement region 412. The difference between the first prediction block 408A corresponding to one of the plurality of predefined motion vector refinements R5 and the second prediction block 408B meets the cost criterion. In some embodiments, the difference between the first prediction block 408A corresponding to one of the plurality of predefined motion vector refinements (e.g., R5) and the second prediction block 408B is smaller than the difference between the prediction blocks 408A and 408B corresponding to any other motion vector refinement (e.g., R1 to R4 and R6). One of the plurality of predefined motion vector refinements is identified as the first refined MVD (RMVD A ) of the first prediction block 408A. By these means, the first motion vector 410A of the first prediction block 408B is further determined based on the motion vector predictor (e.g., MVP A ), the first MVD (MVD A ), and the first refined MVD (RMVD A ).

[0083] In some embodiments, the plurality of predefined motion vector refinements R1 to R6 are included in a given search region that surrounds the motion vector defined by the motion vector predictor and the MVD of the first prediction block 408A. For example, referring to FIG. 7B, the MVD refinement region 412A is a given search region centered at the position corresponding to the MVD, and the predefined motion vector refinements R1 to R6 are distributed in different directions of the MVD. Further, in some embodiments, a given search region (e.g., the MVD refinement region 412A) is restricted within an offset from the motion vector (e.g., within a range of 100 pixels from D corresponding to the MVD).

[0084] In some embodiments, a given search region (e.g., MVD refinement region 412A) is restricted to one or more predefined directions of motion vectors. For example, the center of each of motion vector refinements R1 - R6 is aligned with D corresponding to the MVD along a horizontal direction, a vertical direction, or a diagonal direction having an inclination angle (e.g., 45 degrees). Further, in some situations, a given search range of MVD refinement depends on the MVD. For example, according to the determination that the direction of the MVD is along one of the horizontal direction (e.g., R3) and the vertical direction, the MVD refinement direction of the first prediction block 408A is limited to one of the horizontal direction and the vertical direction. In some embodiments, according to the determination that the direction of the motion vector difference (e.g., MVD A ) is along a first direction (e.g., left), the MVD refinement direction of the first prediction block 408A is selected to be the same as (e.g., R3) or perpendicular to (e.g., R2) the first direction.

[0085] Furthermore, in some embodiments, a plurality of differences between the first prediction block 408A and the second prediction block 408B are determined. Each difference corresponds to one of a plurality of pre - defined motion vector refinements R1 - R6 of the first prediction block 408A. The plurality of differences are compared to each other to determine whether each difference meets a cost criterion. Further, in some embodiments, one or more pre - defined motion vector refinements are preferred over other motion vector refinements. A first subset of the differences between the prediction blocks 408A and 408B is determined based on a first subset of the pre - defined motion vector refinements. Before comparing the plurality of differences, each of the first subset of differences is scaled by a respective difference coefficient. In some embodiments, the difference coefficient is greater than 1 and the first subset of the pre - defined motion vector refinements is less preferred compared to the remaining pre - defined motion vector refinements. Conversely, in some embodiments, the difference coefficient is less than 1 and the first subset of the pre - defined motion vector refinements is more preferred compared to the remaining pre - defined motion vector refinements. In one example, the difference between the prediction blocks 408A and 408B corresponding to R5 is scaled by 0.8 and is likely to be smaller than differences that are not scaled or are scaled with a difference coefficient greater than 1.

[0086] FIG. 8 is a flow diagram illustrating a method of coding video according to some embodiments. Method 800 may be performed in a computing system (e.g., server system 112, source device 102, or electronic device 120) having a control circuit and a memory storing instructions for execution by the control circuit. In some embodiments, method 800 is performed by executing instructions stored in a memory of the computing system (e.g., coding module 320 of memory 314). In some embodiments, a current image frame 402 includes a current coding block 404 (802). When a joint MVD coding mode is selected when the MVDs are signaled together for two reference frame lists of the current coding block 404, bilateral matching (e.g., JVVD mode) is employed (804) to refine the MVD of one of two reference frame lists (e.g., reference prediction blocks 408A and 408B on reference frames 406A and 406B of FIG. 4) corresponding to the current coding block 404 (805).

[0087] In some embodiments, a list of predefined scale factors K (FIG. 5) is used to refine the MVD of a first prediction block 408A (e.g., a forward prediction block) from the signaled joint MVD. The scale factor K from the list of predefined scale factors K ATo derive it, bilateral matching is employed. In some embodiments, the prediction block 408A or 408B is generated with a motion vector equal to the sum of the motion vector prediction signaled to the decoder 122 for the prediction block 408A or 408B and the MVD. Conversely, for each candidate scale factor in a predefined list, the candidate prediction block 408B or 408A is generated using a candidate motion vector equal to the sum of the motion vector prediction for the MVD signaled for the other prediction block 408A or 408B and the scaled MVD. The difference between the two prediction blocks 408A and 408B is determined for a predefined list of scale factors K and measured by a cost criterion. The scale factor K￢ A (pred_scale_factor), the scale factor with the minimum cost (corresponding to the minimum difference) is used.

[0088] Thus, in some embodiments, at least for the first prediction block 408A, the MVD is refined (806) to determine a first refined MVD (RMVD￢ A ) based on the cost criterion of the difference between the first prediction block and the second prediction block. The first prediction block 408A is generated (808) based at least on the first MVD (RMVD A ) and used together with the second prediction block 408B (e.g., a reverse prediction block) to determine the motion compensation data for the current coding block 404 (810).

[0089] In some embodiments, the motion vector 410A (or 410B) of the prediction block 408A (or 408B) is equal to the sum of the MVP of one reference frame list and the signaled MVD. For each candidate scale factor in the pre - defined list 604 of scale factors, the motion vector 410B (or 410A) of the prediction block 408B (or 408A) is equal to the sum of the MVP for the other reference frame list and the scaled MVD, and the difference between the prediction blocks 408A and 408B is calculated and measured according to a cost criterion. A subset 602 of candidate scale factors associated with the smallest N cost values is optionally signaled and used as the available scale factors, where N is a positive integer (e.g., 1, 2, 3, 4, …, 8). In one example (FIG. 6), the full set of candidate scale factors in the pre - defined list 604 includes eight different scale factors, and a subset 602 of four scale factors associated with the smallest four cost values can be selected for the current coding block 404. An index 606 having a value range from 0 to 3 is signaled to indicate which of the four scale factors is used for the current coding block 404.

[0090] Alternatively, in some embodiments, the prediction block 408A or 408B is generated using a motion vector 410A or 410B that is equal to the sum of the motion vector prediction (MVP) and the MVD for one reference frame list. Then, for each MVD within the allowable / given search region 412A or 412B surrounding the MV of the other reference frame list, the prediction block 408B or 408A is generated using a motion vector 410B or 410A that is equal to the sum of the MVP, the MVD, and the refined MVD. The difference between the prediction blocks 408A and 408B is calculated and measured according to a cost criterion, and the refined MVD having the minimum cost is used as the refined MVD for the current coding block 404. Further, in one example, the search for the refined MVD is restricted to a particular direction and / or offset.

[0091] In some embodiments, the cost criteria include, but are not limited to, SAD (Sum of Absolute Differences), SSE (Sum of Squared Errors), SATD (Sum of Absolute Transform Differences), and Sum of Mean Removed SAD.

[0092] In some embodiments, the distortion cost or pre - defined scale factor at one or more positions is modified by a difference factor to make a position more or less preferable during comparison. If the factor is greater than 1, the position is less preferable. If the factor is less than 1, the position is more preferable. In other words, the MVD is refined based on a plurality of pre - defined scale factors K (Figure 5B) or a plurality of motion vector refinements R1 - R6 (Figure 7B). In some embodiments, the refined MVD is determined based on the difference between prediction blocks 408A and 408B corresponding to each scale factor. Alternatively, in some embodiments, the refined MVD is determined based on the difference between prediction blocks 408A and 408B corresponding to each motion vector refinement. In some embodiments, according to the determination that one of the pre - defined scale factors K or motion vector refinements R1 - R6 is preferable, the corresponding difference between prediction blocks 408A and 408B is scaled by a difference factor less than 1. Alternatively and conversely, in some embodiments, according to the determination that one of the pre - defined scale factors K or motion vector refinements R1 - R6 is less preferable, the corresponding difference between prediction blocks 408A and 408B is scaled by a difference factor greater than 1. In some embodiments, regardless of whether the difference factor is greater than or less than 1, the lower the difference factor, the more preferable the corresponding refined MVD. In some embodiments, the application of the difference factor reduces the computational complexity of MVD refinement.

[0093] In some embodiments, the refined MVD determined by bilateral matching (e.g., RMVD A) is not used for constructing the MVP list of additional coding blocks in the current image frame, the next image frame using the current image frame as a reference frame, or both. In other words, the current coding block 404 is reconstructed based on the refined MVD determined by bilateral matching and is not used as a prediction block for the next coding block that requires refinement of the next MVD that identifies the current coding block as a prediction block. The next coding block is optionally within the same current image frame 402 or within a separate frame within the same GOP as the current image frame 402.

[0094] In some embodiments, the search direction for MV refinement by bilateral matching depends on the direction of the MVD. In some embodiments, when the direction of the MVD is along the horizontal or vertical direction, the search direction for MV refinement by bilateral matching is also restricted to the horizontal or vertical direction. In some embodiments, the search direction for MV refinement by bilateral matching is substantially parallel (including parallel) or perpendicular to the direction of the MVD.

[0095] Furthermore, in some embodiments, the NEW_NEARMV (or its variant NEW_NEARMV_OPTL) mode or the NEAR_NEWMV (or its variant NEAR_NEWMV_OPTL) mode is selected for the current coding block 404. There is an MVD in one reference frame list, but no MVD is available in the other reference frame list. Bilateral matching is employed to refine the MVD of the reference frame list without an MVD.

[0096] In some embodiments, the NEAR_NEAR_MV (or its variant NEAR_NEAR_OPTL) mode is selected for the current coding block 404. Bilateral matching-based MVD refinement is applied to one of the motion vectors of the two prediction blocks 408A and 408B. Optionally, one of the motion vectors of the two prediction blocks 408A and 408B selected for refinement is predefined according to the conditions on the encoder 106 and the decoder 122. Optionally, one of the motion vectors of the two prediction blocks 408A and 408B selected for refinement is signaled between the encoder 106 and the decoder 122. For example, according to the conditions, one of the two prediction blocks 408A and 408B closer to the current image frame 402 is selected for MVD refinement.

[0097] FIG. 8 shows several logical steps in a particular order, but the steps that are not order-dependent may be rearranged, and other steps may be combined or split. Some rearrangements or other groupings not specifically mentioned will be apparent to those skilled in the art, and thus the orderings and groupings presented herein are not exhaustive. Further, it should be recognized that the steps may be implemented in hardware, firmware, software, or any combination thereof.

[0098] Here, some exemplary embodiments are referred to.

[0099] (A1)In one aspect, some embodiments include a method 800 for processing (e.g., decoding) video data. The method 800 includes receiving (802) a current coding block within a current image frame from a video bitstream, determining (804) that the current coding block should be predicted in a joint motion vector difference (JMVD) mode, determining (805) in response to the current coding block being predicted in the JMVD mode that the current coding block is associated with two dual prediction blocks including a forward prediction block and a backward prediction block, refining (806) a motion vector difference for at least the forward prediction block based on a cost criterion of a difference between the forward prediction block and the backward prediction block to determine a first refined motion vector difference, generating (808) a forward prediction block based on at least the first refined motion vector difference, and determining (810) motion compensation data for the current coding block based on the forward prediction block and the backward prediction block.

[0100] (A2)In some embodiments of A1, the step of refining the motion vector difference for at least the forward prediction block further includes scaling (812) the motion vector difference with a plurality of predefined scale factors to generate a plurality of scaled motion vector differences for the forward prediction block, determining that a difference between the forward prediction block and the backward prediction block corresponding to one of the plurality of scaled motion vector differences satisfies a cost criterion, identifying (814) one of the plurality of scaled motion vector differences as the first refined motion vector difference, and determining (816) a motion vector of the forward prediction block based on a motion vector predictor and the first refined motion vector difference.

[0101] (A3) Further, in some embodiments of A2, the step of refining the motion vector difference for at least the forward prediction block further includes determining a plurality of differences between the forward prediction block and the backward prediction block, comparing the plurality of differences, and determining whether each difference meets a cost criterion. Each difference corresponds to one of the plurality of scaled motion vector differences of the forward prediction block.

[0102] (A4) Further, in some embodiments of A3, the step of determining a plurality of differences includes determining a first subset of differences based on a first subset of predefined scale factors of the motion vector difference, and modifying the first subset of differences with a difference coefficient before comparing the plurality of differences. (A5) In some embodiments of A4, the difference coefficient is less than 1.

[0103] (A6) In some embodiments of A1, the step of refining the motion vector difference for at least the forward prediction block includes obtaining (818) a plurality of predefined motion vector refinements for the motion vector difference of the forward prediction block, determining that the difference between the forward prediction block and the backward prediction block corresponding to one of the plurality of predefined motion vector refinements meets a cost criterion, identifying (820) one of the plurality of predefined motion vector refinements as a first refined motion vector difference, and determining (822) the motion vector of the forward prediction block based on the motion vector predictor, the motion vector difference, and the first refined motion vector difference. (A7) Further, in some embodiments of A6, the plurality of predefined motion vector refinements are included in a given search region surrounding the motion vector defined by the motion vector predictor and the motion vector difference. (A8) Further, in some embodiments of A7, the given search region is restricted within an offset from the motion vector or in one or more predefined directions of the motion vector.

[0104] (A9) Further, in some embodiments of A6 - A8, the step of refining the motion vector difference for at least the forward prediction block further includes determining a plurality of differences between the forward prediction block and the backward prediction block, comparing the plurality of differences, and determining whether each difference meets a cost criterion. Each difference corresponds to one of a plurality of predefined motion vector refinements of the forward prediction block. (A10) Further, in some embodiments of A9, the step of determining the plurality of differences includes determining a first subset of differences based on a first subset of predefined scale factors of the motion vector difference, and modifying the first subset of differences with a difference coefficient before comparing the plurality of differences. (A11) In some embodiments of A10, the difference coefficient is greater than 1.

[0105] (A12) In some embodiments of A1, the current image frame includes a scaling index. The step of refining the motion vector difference for at least the forward prediction block further includes scaling the motion vector difference with a plurality of predefined scale factors to generate a plurality of scaled motion vector differences of the forward prediction block, determining a plurality of differences between the forward prediction block and the backward prediction block, arranging a subset of predefined scale factors in an ordered sequence of scale factors based on the plurality of differences, selecting one of the subset of predefined scale factors based on the scaling index, scaling the motion vector difference, and generating a first refined motion vector difference. Each difference corresponds to one of the plurality of scaled motion vector differences of the forward prediction block.

[0106] (A13) In some embodiments of A1 - A12, the difference between the forward prediction block and the backward prediction block associated with the cost criterion includes one of the sum of absolute differences (SAD), the sum of squared errors (SAE), the sum of absolute transform differences (SATD), and the sum of mean removed SAD.

[0107] (A14)In some embodiments of A1 to A13, the current image frame includes a motion vector predictor. Method 800 further includes determining a second motion vector based on the motion vector predictor and a motion vector difference, and generating a reverse prediction block within a second image frame based on the second motion vector. The step of generating a forward prediction block based on a first refined motion vector difference further includes determining a first motion vector based on the motion vector predictor and the first refined motion vector difference, and generating a forward prediction block based on the first motion vector.

[0108] (A15)In some embodiments of A1 to A14, method 800 further includes fixing the second motion vector of the reverse prediction block while refining the motion vector difference for the forward prediction block based on a cost criterion.

[0109] (A16)In some embodiments of A1 to A14, method 800 further includes refining the motion vector difference for the reverse prediction block based on a cost criterion of the difference between the forward prediction block and the reverse prediction block to determine a second refined motion vector difference. Method 800 further includes generating a reverse prediction block based on the second refined motion vector difference.

[0110] (A17)In some embodiments of A1 to A16, the current coding block is not used as a prediction block for a next coding block that requires refinement of the next motion vector difference of the current coding block.

[0111] (A18)In some embodiments of A1 to A17, method 800 further includes restricting the refinement direction of the forward prediction block to one of the horizontal direction and the vertical direction in accordance with a determination that the direction of the motion vector difference is along one of the horizontal direction and the vertical direction.

[0112] In some embodiments of A1 to A17, method 800 further includes selecting a refinement direction of a forward prediction block to be the same as or perpendicular to a first direction according to a determination that a direction of a motion vector difference is along the first direction.

[0113] In some embodiments of A1 to A19, a motion vector difference is refined for a forward prediction block according to (1) a determination that a current coding block has one of a group of composite reference modes consisting of NEW_NEARMV, NEW_NEARMV_OPTL, NEAR_NEWMV, and NEAR_NEWMV_OPTL, or (2) a determination that a motion vector difference is obtained for a backward prediction block and not obtained for a forward prediction block.

[0114] In some embodiments of A1 to A19, method 800 further includes identifying a forward prediction block from among two bi-prediction blocks based on a condition before refining a motion vector difference for the forward prediction block according to a determination that a current coding block has one of a group of composite reference modes consisting of NEAR_NEAR_MV and NEAR_NEAR_OPTL.

[0115] In other aspects, some embodiments include a computing system (e.g., server system 112) including a control circuit (e.g., control circuit 302) and a memory (e.g., memory 314) coupled to the control circuit, the memory storing one or more instruction sets configured to be executed by the control circuit, the one or more instruction sets including instructions for performing any of the methods described herein (e.g., A1 to A18 above).

[0116] In yet other aspects, some embodiments include a non-transitory computer-readable storage medium storing one or more sets of instructions for execution by a control circuit of a computing system, the one or more sets of instructions including instructions for performing any of the methods described herein (e.g., A1 - A18 above).

[0117] The terms "first", "second", etc. may be used herein to describe various elements, but it should be understood that these elements should not be limited by these terms. These terms are only used to distinguish one element from another.

[0118] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the scope of the claims. As used in the description of the embodiments and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It should also be understood that the terms "comprises" and / or "comprising", when used herein, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0119] As used herein, the term "if" can be interpreted, depending on the context, to mean "when" or "upon" or "in response to determining" or "in accordance with a determination" or "in response to detecting" that the stated precondition is true. Similarly, the phrases "when it is determined that [the stated precondition is true]" or "if [the stated precondition is true]" or "when [the stated precondition is true]" can be interpreted, depending on the context, to mean "upon determining" or "in response to determining" or "in accordance with a determination" or "upon detecting" or "in response to detecting" that the stated precondition is true.

[0120] The foregoing description has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the claims to the precise forms disclosed. Many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of operation and the practical application, and to enable others skilled in the art to best utilize the disclosure.

Description of Reference Numerals

[0121] 100 Communication system 102 Source device 104 Video source 106 Encoder, encoder component 108 Encoded video bitstream 110 Network 112 Server system 114 Coder component 116 Encoded video data 120 Electronic device 120-1 Electronic device 120-m Electronic device 122 Decoder, decoder component 124 Display 202 Source coder 204 Controller 206 Predictor 208 Reference picture memory 210 Decoder 212 Coding engine 214 Entropy coder 216 Encoded video sequence 218 Channel 252 Buffer memory 254 Parser 256 Loop filter, loop filter unit 258 Scaler / inverse transform unit 260 Motion compensation prediction unit 262 Intra-picture prediction unit 264 Current picture memory 266 Reference picture memory 268 Aggregator 270 Symbol 302 Control circuit 304 Network interface 306 User interface 308 Output device 310 Input device 312 Communication bus 314 Memory 316 Operating system 318 Network communication module 320 Coding module 322 Decoding module 324 Syntax analysis module 326 Conversion Module 328 Prediction Module 330 Filter Module 340 Encoding Module 342 Code Module 344 Prediction Module 352 Picture Memory 400 Bilateral Matching Configuration 402 Current Image Frame 404 Current Coding Block 406A First Reference Frame 406B Second Reference Frame 408A Reference Prediction Block, First Prediction Block 408B Reference Prediction Block, Second Prediction Block 410A Motion Vector, First Motion Vector 0410B Motion Vector, Second Motion Vector 412A First MVD Refinement Region 412B Second MVD Refinement Region 500 Bilateral Matching Configuration 550 MVD Refinement Process 600 Process 602 Ordered Sequence of Scale Factors, Subset of Scale Factors 604 Scale Factor, Ordered Sequence of Scale Factors, Predetermined List of Scale Factors 606 Exponent 700 Bilateral Matching Configuration 750 Process 800 Method

Claims

1. A method for decoding a video bitstream, performed by a decoder, The steps include receiving the current coding block in the current image frame from the video bitstream, The steps include determining that the current coding block should be predicted in joint motion vector difference (JMVD) mode, In response to the decision that the current coding block should be predicted in the JMVD mode, the step of determining that the current coding block is associated with two dual prediction blocks, including a forward prediction block and a reverse prediction block, A step of determining a first refined motion vector difference by refining the motion vector difference for at least the forward prediction block based on a cost criterion for the difference between the forward prediction block and the reverse prediction block, The steps include generating the forward prediction block based on at least the first refined motion vector difference, The steps include determining motion compensation data for the current coding block based on the forward prediction block and the reverse prediction block. Methods that include...

2. The step of refining the motion vector difference for at least the forward prediction block is as follows: The steps include scaling the motion vector difference by a plurality of predefined scaling factors to generate a plurality of scaled motion vector differences for the forward prediction block, The steps include determining whether the difference between the forward prediction block and the reverse prediction block corresponding to one of the plurality of scaled motion vector differences satisfies the cost criterion, The steps include identifying one of the plurality of scaled motion vector differences as the first refined motion vector difference, A step of determining the motion vector of the forward prediction block based on the motion vector predictor and the first refined motion vector difference, The method according to claim 1, further comprising:

3. The step of refining the motion vector difference for at least the forward prediction block is as follows: A step of determining a plurality of differences between the forward prediction block and the reverse prediction block, wherein each difference corresponds to one of the plurality of scaled motion vector differences of the forward prediction block. The steps include comparing the aforementioned multiple differences and determining whether each difference satisfies the cost criteria. The method according to claim 2, further comprising:

4. The step of determining the aforementioned multiple differences is, The steps include determining a first subset of the difference based on a first subset of predefined scaling factors of the motion vector difference, Before comparing the multiple differences, there is a step of correcting the first subset of the differences with a difference coefficient. The method according to claim 3, further comprising:

5. The step of refining the motion vector difference for at least the forward prediction block is as follows: The steps include obtaining a plurality of predefined motion vector refinements for the motion vector difference of the forward prediction block, A step of determining whether the difference between the forward prediction block and the reverse prediction block corresponding to one of the plurality of predefined motion vector refinements satisfies the cost criterion, The steps include identifying one of the plurality of predefined motion vector refinements as the first refined motion vector difference, A step of determining the motion vector of the forward prediction block based on the motion vector predictor, the motion vector difference, and the first refined motion vector difference. The method according to claim 1, further comprising:

6. The method according to claim 5, wherein the plurality of predefined motion vector refinements are included in a given search region that encloses the motion vector defined by the motion vector predictor and the motion vector difference.

7. The method according to claim 6, wherein the given search area is limited to an offset from the motion vector or to one or more predefined directions of the motion vector.

8. The step of refining the motion vector difference for at least the forward prediction block is as follows: A step of determining a plurality of differences between the forward prediction block and the reverse prediction block, wherein each difference corresponds to one of the plurality of predefined motion vector refinements of the forward prediction block. The steps include comparing the aforementioned multiple differences and determining whether each difference satisfies the cost criteria. The method according to claim 5, further comprising:

9. The step of determining the aforementioned multiple differences is, A step of determining a first subset of differences based on a first subset of predefined motion vector refinements, Before comparing the multiple differences, there is a step of correcting the first subset of the differences with a difference coefficient. The method according to claim 8, further comprising:

10. The current image frame includes a scaling index, and the step of refining the motion vector difference for at least the forward prediction block is: The steps include scaling the motion vector difference by a plurality of predefined scaling factors to generate a plurality of scaled motion vector differences for the forward prediction block, A step of determining a plurality of differences between the forward prediction block and the reverse prediction block, wherein each difference corresponds to one of the plurality of scaled motion vector differences of the forward prediction block. The steps include: arranging a predefined subset of scale factors based on the aforementioned multiple differences into an ordered sequence of scale factors; A step of selecting one of the predefined subset of scaling factors based on the scaling index to scale the motion vector difference and generate the first refined motion vector difference; The method according to claim 1, further comprising:

11. The current image frame includes a motion vector predictor, and the method, The steps include determining a second motion vector based on the motion vector predictor and the motion vector difference, The steps of generating the reverse prediction block in the second image frame based on the second motion vector, and It further includes, The step of generating the forward prediction block based on the first refined motion vector difference further includes the step of determining a first motion vector based on the motion vector predictor and the first refined motion vector difference, and the step of generating the forward prediction block based on the first motion vector. The method according to claim 1.

12. The step of fixing the second motion vector of the reverse prediction block while refining the motion vector difference for the forward prediction block based on the cost criteria. The method according to claim 1, further comprising:

13. A step of determining a second refined motion vector difference by refining the motion vector difference for the reverse prediction block based on the cost criterion of the difference between the forward prediction block and the reverse prediction block, The steps include generating the reverse prediction block based on the second refined motion vector difference, and The method according to claim 1, further comprising:

14. The method according to claim 1, wherein the motion vector difference is refined for the forward prediction block according to (1) a determination that the current coding block has one of the group of composite reference modes consisting of NEW_NEARMV, NEW_NEARMV_OPTL, NEAR_NEWMV, and NEAR_NEWMV_OPTL, or (2) a determination that the motion vector difference is obtained for the reverse prediction block but not for the forward prediction block.

15. The method according to claim 1, wherein the current coding block is not used as a prediction block for the next coding block which requires refinement of the next motion vector difference of the current coding block.

16. A step of restricting the refinement direction of the forward prediction block to the horizontal or vertical direction, according to the determination that the direction of the motion vector difference is along one of the horizontal or vertical directions. The method according to claim 1, further comprising:

17. A step of selecting the refinement direction of the forward prediction block to be the same as or perpendicular to the first direction, in accordance with the determination that the direction of the motion vector difference is along the first direction. The method according to claim 1, further comprising:

18. The step of identifying the forward prediction block from among the two biprediction blocks on condition, before refining the motion vector difference for the forward prediction block, according to the determination that the current coding block has one of the group of composite reference modes consisting of NEAR_NEAR_MV and NEAR_NEAR_OPTL. The method according to claim 1, further comprising:

19. A decoder configured to perform the method described in any one of claims 1 to 18.

20. A computer program for causing a computer to perform the method described in any one of claims 1 to 18.

Images