Improvements in bilateral matching and decoder-side motion vector correction under out-of-frame boundary conditions

Abstract

A video decoding method includes receiving an encoded video bitstream including encoded information of a current block in a current picture. The encoded information is used to indicate a bi-directional prediction with a modification. The correction determines an offset based on a plurality of calculations, and each calculation includes reference samples for a first reference image and a second reference image of the current image. The method also includes determining that a first calculation of the plurality of calculations uses a first reference sample as an out-of-boundary (OOB) prediction sample. The OOB prediction sample of the reference image is outside the frame boundary of the reference image. The method also includes performing the correction to determine the offset based on the plurality of calculations while excluding at least the first reference sample, and reconstructing the current block according to the bidirectional prediction having the offset.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority to U.S. Patent Application No. 18 / 587,718, filed on February 26, 2024, entitled “IMPROVEMENT OF BILATERAL MATCHING AND DECODER SIDE MOTION VECTOR REFINEMENT WITH OUT-OF-FRAME BOUNDARY CONDITION”, which claims the benefit of priority to U.S. Provisional Application No. 63 / 448,619, filed on February 27, 2023, entitled “Improvement of Bilateral Matching And DecoderSide Motion Vector Refinement With Out-Of-Frame Boundary Condition”. The disclosures of these two prior applications are incorporated herein by reference in their entirety. Technical Field

[0002] This disclosure describes embodiments generally related to video coding. Background Art

[0003] The background description provided herein is intended to present the background of the present disclosure as a whole. Certain works of the presently named inventors (i.e., works described in this background section) and the contents described in various aspects of this specification that have not become prior art before the filing date are never explicitly or implicitly admitted to be prior art of the present disclosure.

[0004] Image / video compression can help transmit image / video data between different devices, storage spaces, and networks with minimal quality degradation. In some examples, video codec technology can compress video based on spatial and temporal redundancy. In one example, a video codec can use a technique called intra-frame prediction, which can compress images based on spatial redundancy. For example, intra-frame prediction can use reference data from the current image being reconstructed to predict samples. In another example, a video codec can use a technique called inter-frame prediction, which can compress images based on temporal redundancy. For example, inter-frame prediction can use motion compensation to predict samples in the current image from a previously reconstructed image. Motion compensation can be indicated by a motion vector (MV). Summary of the invention

[0005] Aspects of the present disclosure include methods and apparatus for video encoding / decoding. In some examples, an apparatus for video decoding includes a processing circuit. In some examples, a video decoding method includes receiving an encoded video bitstream, the encoded video bitstream including encoded information of a current block in a current image. The encoded information is used to indicate a bidirectional prediction with a correction. The correction is based on multiple calculations to determine an offset, and each calculation includes reference samples of a first reference image and a second reference image for the current image. The method also includes determining that a first calculation of the multiple calculations uses a first reference sample as an out of boundary (OOB) prediction sample. The OOB prediction sample of the reference image is outside the frame boundary of the reference image. The method also includes performing the correction of determining the offset based on the multiple calculations while excluding at least the first reference sample, and reconstructing the current block according to the bidirectional prediction with the offset.

[0006] In some examples, the refinement includes a decoder side motion vector refinement (DMVR) that determines the motion vector offset based on calculation of a bilateral matching (BM) cost value associated with the motion vector offset candidate.

[0007] In some examples, the first calculation calculates a first BM cost as a distortion measure between corresponding reference samples in a pair of reference blocks associated with a first motion vector offset candidate, and a first reference block in the pair of reference blocks includes the first reference sample as the OOB prediction sample. The method includes discarding the first motion vector offset candidate for excluding the first calculation from the plurality of calculations. In one example, the method includes assigning a maximum BM cost value associated with the first motion vector offset candidate; and / or marking the first motion vector offset candidate as unavailable.

[0008] In some examples, the first motion vector offset candidate corresponds to an integer position that is a neighboring position of a specific position having a lowest BM cost, and the method includes, in response to discarding the first motion vector offset candidate, determining to use fractional block matching to replace secondary prediction of a fractional sample correction at the specific position, and performing the fractional sample correction for the specific position by calculating a BM cost value at the position corresponding to the fractional sample offset.

[0009] In some examples, the first calculation calculates a first BM cost as a distortion measure between corresponding reference samples in a pair of reference blocks associated with the first motion vector offset candidate, and a first reference block in the pair of reference blocks includes the first reference sample as the OOB prediction sample. To determine the offset, the method includes performing the first calculation while excluding the first reference sample from the distortion measure.

[0010] In some examples, the method includes normalizing the first BM cost based on a logarithm of the corresponding reference sample in the first calculation.

[0011] In some examples, to perform the first calculation, the method includes determining a plurality of reference sample pairs in the pair of reference blocks, each reference sample pair including at least a reference sample that is an OOB prediction sample, and determining whether a ratio of a number of pairs of the plurality of reference sample pairs to a total number of pairs of reference sample pairs in the pair of reference blocks is above a threshold. In response to the ratio being above the threshold, the method includes discarding the first motion vector offset candidate for excluding the first calculation from the plurality of calculations. In response to the ratio being below the threshold, the method includes performing the first calculation while excluding the plurality of reference sample pairs from the distortion measure.

[0012] In some examples, the current block corresponds to at least one of: a coding block; and / or, a sub-block of the coding block.

[0013] In some examples, the modification includes bi-directional optical flow (BDOF), which uses optical flow to calculate the prediction offset based on correlation calculation of gradients performed based on the reference samples of the first reference image and the second reference image.

[0014] According to another aspect of the present disclosure, a device is provided. The device includes a processing circuit. The processing circuit can be configured to perform any disclosed method for video decoding / encoding.

[0015] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform any disclosed method for video decoding / encoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Other features, properties and various advantages of the disclosed subject matter will become further apparent from the following detailed description and accompanying drawings, in which:

[0017] Figure 1 is a schematic diagram of an exemplary block diagram of a communication system (100).

[0018] Figure 2is a schematic diagram of an exemplary block diagram of a decoder.

[0019] Figure 3 is a schematic diagram of an exemplary block diagram of an encoder.

[0020] Figure 4 An exemplary schematic diagram of decoder-side motion vector correction based on bilateral matching is shown.

[0021] Figure 5 A schematic diagram showing some calculations in bidirectional optical flow (BDOF) in one example.

[0022] Figure 6 The search area in some examples is shown.

[0023] Figure 7 A flow chart outlining a decoding process according to some embodiments of the present disclosure is shown.

[0024] Figure 8 A flowchart outlining the encoding process according to some embodiments of the present disclosure is shown.

[0025] Fig. 9 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION

[0026] Figure 1 A block diagram of a video processing system (100) in some examples is shown. The video processing system (100) is an example of an application of the subject matter disclosed in this application, a video encoder and a video decoder in a streaming environment. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, streaming services, storing compressed video on digital media including CDs, DVDs, memory sticks, etc., and the like.

[0027] The video processing system (100) includes an acquisition subsystem (113) that may include a video source (101) such as a digital camera that creates, for example, an uncompressed video image stream (102). In one example, the video image stream (102) includes samples captured by the digital camera. The video image stream (102) is depicted as a thick line to emphasize that it has a high amount of data compared to the encoded video data (104) (or the encoded video bitstream). The video image stream (102) can be processed by an electronic device (120) that includes a video encoder (103) coupled to the video source (101). The video encoder (103) can include hardware, software, or a combination of hardware and software to implement or implement various aspects of the disclosed subject matter as described in more detail below. Compared to the video image stream (102), the encoded video data (104) (or the encoded video bitstream) is depicted as a thin line to emphasize that it has a lower amount of data. The encoded video data (104) may be stored on a streaming server (105) for future use. One or more streaming client subsystems, e.g. Figure 1 A client subsystem (106) and a client subsystem (108) in a streaming server (105) may access a streaming server (105) to retrieve a copy (107) of the encoded video data (104) and a copy (109) of the encoded video data (104). The client subsystem (106) may include, for example, a video decoder (110) in an electronic device (130). The video decoder (110) decodes the incoming copy (107) of the encoded video data and produces an output video image stream (111) that can be rendered / presented on a display (112) (e.g., a display screen) or another presentation device (not depicted). In some streaming systems, the encoded video data (104), the copy (107) of the encoded video data, and the copy (109) of the encoded video data (e.g., a video bitstream) may be encoded according to certain video codec / compression standards. Examples of such standards include ITU-T Recommendation H.265. In one example, the video codec standard under development is informally referred to as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of the VVC standard.

[0028] It should be noted that the electronic device (120) and the electronic device (130) may include other components (not shown). For example, the electronic device (120) may include a video decoder (not shown), and the electronic device (130) may also include a video encoder (not shown).

[0029] Figure 2An exemplary block diagram of a video decoder (210) is shown. The video decoder (210) may be disposed in an electronic device (230). The electronic device (230) may include a receiver (231) (e.g., a receiving circuit). The video decoder (210) may be used to replace Figure 1 A video decoder (110) is shown in an example.

[0030] The receiver (231) may receive one or more encoded video sequences, for example, included in a bitstream, to be decoded by the video decoder (210). In one embodiment, the encoded video sequences are received one at a time, wherein the decoding of each encoded video sequence is independent of the decoding of the other encoded video sequences. The encoded video sequence may be received from a channel (201), which may be a hardware / software link to a storage device storing the encoded video data. The receiver (231) may receive the encoded video data as well as other data, for example, encoded audio data and / or auxiliary data streams that may be forwarded to their respective consuming entities (not depicted). The receiver (231) may separate the encoded video sequence from the other data. To prevent network jitter, a buffer memory (215) may be coupled between the receiver (231) and the entropy decoder / parser (220) (hereinafter referred to as "parser (220)"). In some applications, the buffer memory (215) is part of the video decoder (210). In other cases, the buffer memory (215) may be provided external to the video decoder (210) (not shown). In other cases, a buffer memory (not shown) is provided outside the video decoder (210) to prevent network jitter, for example, and another buffer memory (215) may be configured inside the video decoder (210) to handle broadcast timing, for example. When the receiver (231) receives data from a storage / forwarding device with sufficient bandwidth and controllability or from an isochronous network, it may not be necessary to configure the buffer memory (215), or the buffer memory may be made smaller. Of course, in order to use on a service packet network such as the Internet, a buffer memory (215) may also be required, which may be relatively large and have an adaptive size, and may be at least partially implemented in an operating system or a similar element (not shown) outside the video decoder (210).

[0031] The video decoder (210) may include a parser (220) to reconstruct symbols (221) from the encoded video sequence. The categories of these symbols include information for managing the operation of the video decoder (210) and potential information for controlling a presentation device such as a presentation device (212) (e.g., a display screen) that is not part of the electronic device (230) but can be coupled to the electronic device (230), such as Figure 2As shown in . The control information for the presentation device may be a parameter set fragment (not indicated) of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI). The parser (220) may parse / entropy decode the received coded video sequence. The encoding of the coded video sequence may be performed according to a video coding technique or standard, and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and the like. The parser (220) may extract a subgroup parameter set for at least one subgroup of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include Groups of Picture (GOP), pictures, tiles, slices, macroblocks, Coding Units (CU), blocks, Transform Units (TU), Prediction Units (PU), and the like. The parser (220) may also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the encoded video sequence.

[0032] The parser (220) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (215), thereby creating symbols (221).

[0033] Depending on the type of coded video picture or part of coded video picture (such as: inter-frame picture and intra-frame picture, inter-frame block and intra-frame block) and other factors, the reconstruction of symbol (221) may involve multiple different units. Which units are involved and how they are involved can be controlled by subgroup control information parsed by parser (220) from the coded video sequence. For the sake of brevity, such subgroup control information flow between parser (220) and multiple units below is not depicted.

[0034] In addition to the functional blocks already mentioned, the video decoder (210) can be conceptually subdivided into several functional units as described below. In a practical implementation operating under commercial constraints, many of these units interact closely with each other and may be at least partially integrated with each other. However, for the purpose of describing the disclosed subject matter, the conceptual subdivision into the following functional units is appropriate.

[0035] The first unit is a scaler / inverse transform unit (251). The scaler / inverse transform unit (251) receives quantized transform coefficients as symbols (221) from the parser (220) and control information, including which transform method to use, block size, quantization factor, quantization scaling matrix, etc. The scaler / inverse transform unit (251) can output blocks including sample values, which can be input into an aggregator (255).

[0036] In some cases, the output samples of the scaler / inverse transform unit (251) may belong to an intra-coded block. An intra-coded block does not use predictive information from a previously reconstructed image, but may use predictive information from a previously reconstructed portion of a current image. Such predictive information may be provided by an intra-image prediction unit (252). In some cases, the intra-image prediction unit (252) generates a block of the same size and shape as the block being reconstructed using surrounding reconstructed information extracted from a current image buffer (258). For example, the current image buffer (258) buffers a partially reconstructed current image and / or a fully reconstructed current image. In some cases, the aggregator (255) adds the prediction information generated by the intra-frame prediction unit (252) to the output sample information provided by the scaler / inverse transform unit (251) on a per-sample basis.

[0037] In other cases, the output samples of the scaler / inverse transform unit (251) may belong to an inter-frame coded and potentially motion compensated block. In this case, the motion compensated prediction unit (253) may access the reference image memory (257) to extract samples for prediction. After the extracted samples are motion compensated according to the symbols (221) belonging to the block, these samples may be added to the output of the scaler / inverse transform unit (251) (in this case referred to as residual samples or residual signals) by the aggregator (255) to generate output sample information. The address in the reference image memory (257) from which the motion compensated prediction unit (253) extracts the predicted samples may be controlled by a motion vector, and the motion vector is provided to the motion compensated prediction unit (253) in the form of a symbol (221), which may have, for example, an X component, a Y component and a reference image component. Motion compensation may also include interpolation of sample values ​​extracted from the reference image memory (257) when using sub-sample accurate motion vectors, motion vector prediction mechanisms, and the like.

[0038] The output samples of the aggregator (255) may be used in a loop filter unit (256) for various loop filtering techniques. The video compression techniques may include in-loop filter techniques controlled by parameters included in the encoded video sequence (also referred to as the encoded video bitstream) and available to the loop filter unit (256) as symbols (221) from the parser (220). The video compression may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of an encoded picture or encoded video sequence, and to previously reconstructed and loop filtered sample values.

[0039] The output of the loop filter unit (256) may be a sample stream that may be output to a rendering device (212) and stored in a reference picture memory (257) for subsequent inter-picture prediction.

[0040] Once fully reconstructed, certain coded pictures may be used as reference pictures for future predictions. For example, once the coded picture corresponding to the current picture is fully reconstructed, and the coded picture is identified as a reference picture (e.g., by the parser (220)), the current picture buffer (258) may become part of the reference picture memory (257), and a new current picture buffer may be reallocated before starting reconstruction of a subsequent coded picture.

[0041] The video decoder (210) may perform decoding operations according to a predetermined video compression technology or standard such as ITU-T Recommendation H.265. The encoded video sequence may conform to the syntax specified by the video compression technology or standard used in the sense that the encoded video sequence follows the syntax of the video compression technology or standard and the profile recorded in the video compression technology or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technology or standard as the only tools available for use under the profile. For compliance, the complexity of the encoded video sequence is also required to be within the range defined by the hierarchy of the video compression technology or standard. In some cases, the hierarchy limits the maximum image size, the maximum frame rate, the maximum reconstruction sampling rate (measured in, for example, millions of samples per second), the maximum reference image size, etc. In some cases, the limits set by the hierarchy may be further limited by the Hypothetical Reference Decoder (HRD) specification and metadata of the HRD buffer management signaled in the encoded video sequence.

[0042] In one embodiment, the receiver (231) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The additional data may be used by the video decoder (210) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0043] Figure 3 An exemplary block diagram of a video encoder (303) is shown. The video encoder (303) is disposed in an electronic device (320). The electronic device (320) includes a transmitter (340) (e.g., a transmission circuit). The video encoder (303) can be used to replace Figure 1 A video encoder (103) in an example.

[0044] The video encoder (303) can be used to obtain the video source (301) (not Figure 3 In another example, the video source (301) is a part of the electronic device (320) to receive video samples, and the video source can capture video images to be encoded by the video encoder (303). In another example, the video source (301) is a part of the electronic device (320).

[0045] The video source (301) may provide a source video sequence in the form of a digital video sample stream to be encoded by the video encoder (303), which may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (301) may be a storage device for storing previously prepared videos. In a video conferencing system, the video source (301) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of separate images that are imparted with motion when viewed sequentially. The images themselves may be constructed as a spatial array of pixels, where each pixel may include one or more samples, depending on the sampling structure, color space, etc. used. The following description focuses on the samples.

[0046] According to an embodiment, the video encoder (303) may encode and compress images of a source video sequence into an encoded video sequence (343) in real time or under any other time constraints as required. Implementing an appropriate encoding speed is a function of the controller (350). In some embodiments, the controller (350) controls other functional units as described below and is functionally coupled to these units. For the sake of brevity, the coupling is not depicted in the figure. The parameters set by the controller (350) may include rate control related parameters (image skipping, quantizer, lambda value of rate distortion optimization techniques, etc.), image size, GOP layout, maximum motion vector search range, etc. The controller (350) may be used to have other suitable functions that are related to the video encoder (303) optimized for a certain system design.

[0047] In some embodiments, the video encoder (303) operates in an encoding loop. As a simple description, in one example, the encoding loop may include a source encoder (330) (e.g., responsible for creating symbols, such as a symbol stream, based on an input image to be encoded and a reference image) and a (local) decoder (333) embedded in the video encoder (303). The decoder (333) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data. The reconstructed sample stream (sample data) is input into a reference image memory (334). Since the decoding of the symbol stream results in a bit-accurate result that is independent of the decoder location (local or remote), the contents of the reference image memory (334) are also bit-accurate between the local encoder and the remote encoder. In other words, the reference image samples "seen" by the prediction part of the encoder are exactly the same as the sample values ​​that the decoder will "see" when using the prediction during decoding. This basic principle of reference image synchronization (and the drift that occurs when synchronization cannot be maintained, such as due to channel errors) is also used in some related technologies.

[0048] The operation of the "local" decoder (333) may be similar to that described above in conjunction with Figure 2 The "remote" decoder of the video decoder (210) described in detail is identical. However, additional brief reference is made to Figure 2 , when symbols are available and the entropy encoder (345) and parser (220) are capable of losslessly encoding / decoding the symbols into an encoded video sequence, the entropy decoding portion of the video decoder (210), including the buffer memory (215) and the parser (220), may not be fully implemented in the local decoder (333).

[0049] In one embodiment, decoder techniques other than parsing / entropy decoding present in a decoder are present in a corresponding encoder in the same or substantially the same functional form. Therefore, the disclosed subject matter focuses on decoder operation. The description of encoder techniques can be simplified because encoder techniques are mutually inverse to the decoder techniques described comprehensively. A more detailed description in some areas is provided below.

[0050] During operation, in some examples, the source encoder (330) may perform motion compensated predictive (MCP) coding. MCP coding predictively encodes an input image with reference to one or more previously encoded images from a video sequence designated as "reference images." In this manner, the encoding engine (332) encodes the difference between a pixel block of the input image and a pixel block of a reference image that may be selected as a prediction reference for the input image.

[0051] The local video decoder (333) may decode the encoded video data of the picture that may be designated as the reference picture based on the symbols created by the source encoder (330). The operation of the encoding engine (332) may advantageously be a lossy process. When the encoded video data is available at the video decoder ( Figure 3 When the video encoder (303) is decoded at a remote location (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (333) replicates the decoding process that may be performed by the video decoder on the reference picture and may cause the reconstructed reference picture to be stored in the reference picture memory (334). In this way, the video encoder (303) may locally store a copy of the reconstructed reference picture that has common content (absent transmission errors) with the reconstructed reference picture to be obtained by the remote video decoder.

[0052] The predictor (335) may perform a prediction search for the encoding engine (332). That is, for a new image to be encoded, the predictor (335) may search the reference image memory (334) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that may serve as appropriate prediction references for the new image. The predictor (335) may operate on a pixel block by pixel block basis to find a suitable prediction reference. In some cases, as determined based on the search results obtained by the predictor (335), the input image may have prediction references taken from multiple reference images stored in the reference image memory (334).

[0053] The controller (350) may manage encoding operations of the source encoder (330), including, for example, setting parameters and subgroup parameters for encoding video data.

[0054] The outputs of all the above functional units may be entropy encoded in an entropy encoder (345). The entropy encoder (345) applies lossless compression to the symbols generated by the various functional units according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc., thereby converting these symbols into a coded video sequence.

[0055] The transmitter (340) may buffer the encoded video sequence created by the entropy encoder (345) in preparation for transmission over a communication channel (360), which may be a hardware / software link to a storage device where the encoded video data will be stored. The transmitter (340) may combine the encoded video data from the video encoder (303) with other data to be transmitted, such as encoded audio data and / or ancillary data streams (source not shown).

[0056] The controller (350) may manage the operation of the video encoder (303). During encoding, the controller (350) may assign a certain coded picture type to each coded picture, but this may affect the coding techniques that can be applied to the corresponding picture. For example, a picture may generally be assigned to any of the following picture types:

[0057] Intra pictures (I pictures) can be encoded and decoded without using any other pictures in the sequence as a prediction source.Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures.

[0058] Predictive pictures (P pictures) can be encoded and decoded using intra prediction or inter prediction, which uses a motion vector and a reference index to predict the sample values ​​of each block.

[0059] Bidirectional predictive pictures (B pictures) can be encoded and decoded using intra prediction or inter prediction, which uses two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple prediction pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0060] The source picture may typically be spatially subdivided into a number of blocks of samples (e.g., each block of samples comprising 4×4, 8×8, 4×8 or 16×16 samples), and coded block by block. These blocks may be predictively coded with reference to other (already coded) blocks, which may be determined by the coding allocation of the corresponding picture applied to the block. For example, blocks of an I picture may be non-predictively coded, or these blocks may be predictively coded with reference to already coded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of a P picture may be predictively coded by spatial prediction with reference to one previously coded reference picture or by temporal prediction. Blocks of a B picture may be predictively coded by spatial prediction with reference to one or two previously coded reference pictures or by temporal prediction.

[0061] The video encoder (303) may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T Recommendation H.265. In operation, the video encoder (303) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in an input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard used.

[0062] In one embodiment, the transmitter (340) may transmit additional data when transmitting the encoded video. The source encoder (330) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0063] The captured video may be taken as a plurality of source images (video images) in a temporal sequence. Intra-image prediction (often simplified to intra-frame prediction) exploits spatial correlations in a given image, while inter-image prediction exploits (temporal or other) correlations between images. In one example, a particular image being encoded / decoded is divided into blocks, and the particular image being encoded / decoded is referred to as the current image. When a block in the current image is similar to a reference block in a reference image that was previously encoded in the video and is still buffered, the block in the current image may be encoded by a vector called a motion vector. The motion vector points to a reference block in a reference image, and in the case where multiple reference images are used, the motion vector may have a third dimension that identifies the reference image.

[0064] In some embodiments, bidirectional prediction techniques may be used in inter-frame image prediction. According to the bidirectional prediction technique, two reference images are used, such as a first reference image and a second reference image that are both before the current image in the video in decoding order (but may be in the past and future, respectively, in display order). A block in the current image may be encoded by a first motion vector pointing to a first reference block in the first reference image and a second motion vector pointing to a second reference block in the second reference image. The block may be predicted by a combination of the first reference block and the second reference block.

[0065] In addition, merge mode technology can be used in inter-frame image prediction to improve encoding and decoding efficiency.

[0066] According to some embodiments of the present disclosure, predictions such as inter-image prediction and intra-image prediction are performed in units of blocks. For example, according to the HEVC standard, an image in a video image sequence is divided into coding tree units (CTUs) for compression, and the CTUs in the image have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU includes three coding tree blocks (CTBs), which are a luminance CTB and two chrominance CTBs. Each CTU can be recursively split into one or more CUs using a quadtree. For example, a 64×64 pixel CTU can be split into a 64×64 pixel CU, or 4 32×32 pixel CUs, or 16 16×16 pixel CUs. In one example, each CU is analyzed to determine a prediction type for the CU, such as an inter-prediction type or an intra-prediction type. Depending on temporal and / or spatial predictability, the CU is split into one or more PUs. Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In one embodiment, the prediction operation in the codec (encoding / decoding) is performed in units of prediction blocks. Taking the luma prediction block as an example, the prediction block includes a matrix of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, and the like.

[0067] It should be noted that the video encoder (103) and the video encoder (303) and the video decoder (110) and the video decoder (210) may be implemented using any suitable technology. In one embodiment, the video encoder (103) and the video encoder (303) and the video decoder (110) and the video decoder (210) may be implemented using one or more integrated circuits. In another embodiment, the video encoder (103) and the video encoder (303) and the video decoder (110) and the video decoder (210) may be implemented using one or more processors that execute software instructions.

[0068] Aspects of the present disclosure provide techniques for bilateral matching and decoder-side motion vector correction with out-of-frame boundary conditions.

[0069] Various inter prediction modes can be used in video codecs. For example, in VVC, for an inter-predicted CU, motion parameters may include MV, one or more reference image indices, a reference image list usage index, and additional information for certain coding features of inter-prediction sample generation. Motion parameters may be signaled explicitly or implicitly. When a CU is encoded in skip mode, the CU may be associated with one PU, and no significant residual coefficients, encoded motion vector increments, MV differences (MVDs), or reference image indices may be required. A merge mode may be specified in which the motion parameters of the current CU are obtained from neighboring CUs (including spatial and / or temporal candidates) and optional additional information such as that introduced in VVC. The merge mode may be applied to inter-predicted CUs, not just skip mode CUs. In one example, an alternative to the merge mode is the explicit transmission of motion parameters, with MVs, corresponding reference image indices for each reference image list, reference image list usage flags, and other information explicitly signaled for each CU.

[0070] In some examples, such as in VVC, the VVC Test model (VTM) reference software includes one or more new and revised inter-frame prediction coding tools, such as extended merge prediction, merge motion vector difference (MMVD) mode, adaptive motion vector prediction (AMVP) mode with symmetric MVD signaling, affine MCP, subblock-based temporal motion vector prediction (SbTMVP), adaptive motion vector resolution (AMVR), motion field storage (1 / 16 luma sample MV storage and 8×8 motion field compression), bi-prediction with CU-level weight (BCW), bi-directional optical flow (BDOF), prediction refinement using optical flow (PROF), decoder side motion vector refinement (DMVR), combined inter and intra prediction (CIIP), geometric partitioning mode (geometricpartitioning mode, GPM) etc.

[0071] It should be noted that motion field storage can change for new standards. In one example, in order to use temporal motion vector prediction (TMVP) in merge mode and AMVP mode, VVC needs to store motion data in collocated reference frames and store motion data on an 8×8 block basis. In order to limit the required storage of MVs for temporal prediction, VVC uses a 10-bit mantissa exponent representation method to compress the stored MVs. In the development of super VVC reference software such as ECM5, the granularity of motion storage is reduced to a 4×4 block basis.

[0072] Several tools are described in detail in this disclosure.

[0073] In inter-image prediction, merge mode can be used to improve codec efficiency. In merge mode, motion vectors can be derived from neighboring blocks and used directly for motion compensation. To improve the accuracy of MVs in merge mode, decoder-side motion vector correction based on bilateral matching (BM) can be applied, such as in VVC. In bidirectional prediction operation, the corrected MV can be searched around the initial MV in reference image list L0 and reference image list L1. BM calculates the distortion between two candidate blocks in reference image list L0 and list L1.

[0074] Figure 4 FIG. 4 shows an exemplary schematic diagram of a decoder-side motion vector correction based on BM. Figure 4 As shown, the current image (402) may include a current block (408). The current image may have a first reference image (404) from a (reference image) list L0 and a second reference image (406) from a (reference image) list L1. For the current block (408), a pair of reference blocks are identified in the first reference image and the second reference image based on initial motion vectors MV0 and MV1. For example, the initial reference block (412) in the first reference image (404) may be located based on the initial motion vector MV0, and the initial reference block (414) in the second image (406) may be located based on the initial motion vector MV1. A search process may be performed around the initial MV0 in the first reference image (404) and the initial MV1 in the second reference image (406). For example, an adjustment MV is applied to the initial MV0 and MV1 in opposite directions. diff , to obtain MV candidates, such as MV0' and MV1'. Based on the MV candidates, a pair of candidate reference blocks are identified in the first reference image and the second reference image. For example, a candidate reference block (410) can be identified in the first reference image (404) based on MV0', and a candidate reference block (416) can be identified in the second reference image (406) based on MV1'. In some examples, bilateral matching (BM) refers to an operation for calculating a distortion measure between a pair of reference blocks of respective reference images of a current image, such as taking the sum of absolute differences (SAD) between a pair of reference blocks as a distortion measure for the pair of reference blocks. For example, the BM method calculates an initial SAD between a pair of initial reference blocks (412) and (414), and calculates a second SAD between a pair of candidate reference blocks (410) and (416). The initial SAD is associated with an initial MV (e.g., MV0 and MV1), and the second SAD is associated with an MV candidate (e.g., MV0' and MV1'). Similarly, the BM method can calculate the SADs of multiple MV candidates around the initial MV. The MV candidate with the lowest SAD may become the revised MV and used to generate a bidirectional prediction signal to predict the current block (408).

[0075] In some examples (e.g., VVC), the application of DMVR is restricted and only applies to CUs encoded with modes and features that meet certain conditions. If a block meets certain conditions, the DMVR algorithm is called. For example, the conditions (also called DMVR requirements or a set of DMVR conditions) may include: (1) CU-level merge mode with bidirectional prediction MV; (2) relative to the current image, one reference image is in the past, while the other reference image is in the future; (3) The distances from the two reference images to the current image (e.g., the picture order count (POC) difference) are the same; (4) Both reference images are short-term reference images; (5) CU has more than 64 brightness samples; (6) Both CU height and CU width are greater than or equal to 8 luma samples; (7) Weight index of bidirectional prediction (BCW) with CU-level weight is used to indicate equal weight; (8) Weighted prediction (WP) is not enabled for the current block; and (9) The current block does not use the combined inter and intra prediction (CIIP) mode.

[0076] It should be noted that the modified MV derived by the DMVR process is used to generate inter-frame prediction samples and can be used for temporal motion vector prediction for future image coding. In some examples, the original MV is used for deblocking and for spatial motion vector prediction for future CU coding.

[0077] In DVMR, the search point can be around the initial MV, and the MV offset obeys the MV difference mirror rule. Any point checked by DMVR and recorded as a candidate MV pair (MV0', MV1') obeys MV0'=MV0+MV_offset and MV1'=MV1-MV_offset. Where MV_offset represents the corrected offset between the initial MV (e.g., (MV0', MV1')) and the corrected MV in one of the reference images. In some examples, the corrected search range is two integer brightness samples of the initial MV. The search includes an integer sample offset search stage and a fractional sample correction stage.

[0078] In some examples (e.g., VVC), decoder-side motion vector modification (DMVR) is applied to CUs encoded in normal merge mode. The MV pairs obtained from normal merge candidates are used as input to the DMVR process. DMVR applies bilateral matching (BM) to modify the input MV pair {MV0, MV1} and uses the modified MV pair {MV refinedL0 ,MV refinedL1} for motion compensation prediction of luminance and chrominance components, such as Figure 4 The output MV of DMVR can be called a modified MV pair and can be expressed by formula (1):

[0079] The motion vector difference Δmv is applied to the input MV pair to obtain a modified MV pair by using the MVD mirroring property, because the input MV pair points to two different reference images, whose picture order counts (POCs) have equal differences with the current image, and the two reference images are in different temporal directions.

[0080] In some examples, DMVR can be applied at the sub-block level, dividing the luma coding block into 16×16 sub-blocks for MV correction processing. Δmv is derived independently for each sub-block.

[0081] In some examples, the motion vector correction search range is two integer luma samples from the initial MV. The motion vector search can be performed in two steps, such as a first step of an integer sample offset search phase (also known as integer precision motion search) and a second step of a fractional sample correction phase (also known as a fractional motion search step or a fractional sample offset search).

[0082] In some examples, a full search of 25 points can be applied to the integer sample offset search. The SAD of the initial MV pair is first calculated. If the SAD of the initial MV pair is less than a threshold, the integer sampling stage of DMVR is terminated. Otherwise, the SAD of the remaining 24 points is calculated and checked in raster scan order. The point with the smallest SAD can be selected as the output of the integer sample offset search stage. In order to reduce the loss of uncertainty in DMVR correction, the original MV (e.g., initial MV candidates MV0 and MV1) can be preferred during DMVR processing. The SAD between the reference blocks referenced by the initial MV candidates is reduced, for example, by 1 / 4 of the SAD value, so that the initial MV candidates are made preferred.

[0083] In some examples, fractional sample correction is performed after the integer sample search. In some examples, fractional sample correction is performed by using fractional sample offsets, such as using 1 / 2 pixel offsets in the vertical and horizontal directions. In some examples, in order to save computational complexity, fractional sample correction is derived by using a parameter error surface formula (also known as a quadratic prediction-based method) instead of performing an additional search using SAD comparison. Fractional sample correction is conditionally called based on the output of the integer sample search stage. For example, when the integer sample search stage terminates with a specific integer position (also known as the center) with the minimum SAD in the first iteration or the second iteration search, fractional sample correction can be further applied.

[0084] In the parametric error surface based sub-pixel offset estimation, the center position cost (the center position is the point with the minimum SAD in the integer sample offset search) and the costs of the four neighboring positions from the center (e.g., the costs from the center position (-1, 0), (0, -1), (1, 0), and (0, 1)) are used to fit a 2-D parabolic error surface formula, such as formula (2): E(x,y)=A(x min ) 2 +B(yy min ) 2 +C Formula (2) Among them, (x min ,y min ) corresponds to the fractional position with the minimum cost, and C corresponds to the minimum cost value. By solving the above formula using the cost values ​​of the five search points, (x min ,y min ): x min =(E(-1,0)-E(1,0)) / (2(E(-1,0)+E(1,0)-2E(0,0))) Formula (3) y min =(E(0,-1)-E(0,1)) / (2((E(0,-1)+E(0,1)-2E(0,0))) Formula (4) Among them, x min and min The value of can be automatically constrained between -8 and 8, because all cost values ​​are positive, with a minimum of E(0,0). These constraints correspond to half-pixel offsets with 1 / 16 pixel MV precision in VVC. The calculated score (x min ,y min) is added to the integer distance corrected MV to obtain the sub-pixel accurate corrected increment MV. In formula (3) and formula (4), E(-1,0), E(1,0), E(1,0), E(0,-1), and E(0,0) represent the cost values ​​at 5 points (the center position and four adjacent positions).

[0085] Bilinear interpolation and sample filling can be applied, such as in VVC. In VVC, the resolution of the MV can be 1 / 16 luma samples. Samples at fractional positions can be interpolated using an 8-tap interpolation filter. In DMVR, the search point can be offset by integer samples around the initial fractional-pixel MV, so it is necessary to interpolate samples at fractional positions for DMVR search processing. In order to reduce computational complexity, a bilinear interpolation filter can be used to generate fractional samples for search processing in DMVR. In another important effect, by using a bilinear filter with a 2-sample search range, DVMR does not access more reference samples than normal motion compensation processing. After the modified MV is obtained through the DMVR search process, a normal 8-tap interpolation filter can be applied to generate the final prediction. In order not to access more reference samples than normal MC processing, samples that may not be required for interpolation processing based on the original MV but may be required for interpolation processing based on the modified MV can be filled from available samples.

[0086] When the width and / or height of a CU is greater than 16 luma samples, the CU may be further partitioned into sub-blocks with a width and / or height equal to 16 luma samples. The maximum unit size of the DMVR search process may be limited to 16x16.

[0087] A technique called bidirectional optical flow (BDOF) can be used, for example, in VVC. BDOF was previously called BIO in JEM. BDOF in VVC can be a simpler version that requires less computation than the JEM version, especially in terms of the number of multiplications and multiplier size.

[0088] BDOF can be used to modify the bidirectional prediction signal of a CU at the 4×4 sub-block level. BDOF can be applied to a CU if it meets the following conditions (also called BDOF requirements or a set of BDOF conditions): (1) The CU is encoded using the “true” bidirectional prediction mode, that is, one of the two reference images precedes the current image in the display order, and the other follows the current image in the display order; (2) The distances from the two reference images to the current image (e.g., the picture order count (POC) difference) are the same; (3) Both reference images are short-term reference images; (4) The CU is not encoded using the affine mode or SbTMVP merge mode; (5) CU has more than 64 brightness samples; (6) Both CU height and CU width are greater than or equal to 8 luma samples; (7) BCW weight index indicates equal weights; (8) The current CU does not enable WP; and (9) The current CU does not use CIIP mode.

[0089] In some examples, BDOF is applied only to the luma component. As the name of BDOF indicates, the BDOF mode can be based on the concept of optical flow, which assumes that the motion of objects is smooth. For each 4×4 sub-block, the motion correction (v x ,v y ). The motion correction can then be used to adjust the sample values ​​of the bidirectional prediction in the 4x4 sub-block. BDOF may include the following steps:

[0090] First, the horizontal gradient of the two prediction signals from reference list L0 and reference list L1 can be calculated by directly calculating the difference between two adjacent samples: and vertical gradient k = 0, 1. The horizontal gradient and the vertical gradient can be provided in formula (5) and formula (6) as follows: Among them, k (k) (i,j) may be the sample value at coordinate (i,j) of the prediction signal in list k, k=0,1, and shift1 may be calculated based on the luma bit depth bitDepth, where shift1=max(6, bitDepth-6).

[0091] Then, the autocorrelation and cross-correlation of the gradients S1, S2, S3, S5, and S6 can be calculated according to the following formulas (7) to (11): S1=∑ (i,j)∈Ω Abs(ψ x (i,j)) Formula (7) S2=∑ (i,j)∈Ω ψ x (i,j)·Sign(ψ y (i,j)) Formula (8) S3=∑ (i,j)∈Ω θ(i,j)·Sign(ψ x (i,j)) Formula (9) S5=∑ (i,j)∈Ω Abs(ψy (i,j)) Formula (10) S6=∑ (i,j)∈Ω θ(i,j)·Sign(ψ y (i,j)) Formula (11) Among them, ψ x (i,j),ψ y (i, j) and θ(i, j) can be provided in formula (12)-formula (14), respectively. θ(i,j)=(I (1) (i,j)>>n b )-(I (0) (i,j)>>n b ) Formula (14) where Ω can be a 6×6 window around a 4×4 sub-block, and n a and n b The value can be set to min(1, bitDepth-11) and min(4, bitDepth-8) respectively.

[0092] The motion correction (vx,vx) can then be derived using the cross-correlation and autocorrelation terms by using the following equations (15) and (16): y ): in, and th′ BIO =2 max(5,BD-7) . is the floor function, and Based on the motion correction and gradient, the adjustment value can be calculated for each sample in the 4×4 sub-block based on formula (17):

[0093] Finally, the BDOF samples of the CU can be calculated by adjusting the bidirectional prediction samples according to the following formula (18): pred BDOF (x,y)=(I (0) (x,y)+I (1) (x,y)+b(x,y)+o offset )>>shift formula (18) The values ​​may be chosen so that the multipliers in the BDOF process do not exceed 15 bits, and the maximum bit width of the intermediate parameters in the BDOF process may be kept within 32 bits.

[0094] Figure 5A schematic diagram of some calculations in a bidirectional optical flow (BDOF) in an example is shown. In order to derive the gradient value, some prediction samples I in a list k (k = 0, 1) need to be generated outside the current CU boundary. (k) (i,j). Figure 5 As shown, BDOF in VVC can use an extended row / column (502) around the boundary (506) of the CU (504). In order to control the computational complexity of generating prediction samples outside the boundary, the extended area (e.g., Figure 5 ), and a normal 8-tap motion compensated interpolation filter may be used to generate prediction samples within the CU (e.g., Figure 5 ). The extended sample values ​​can only be used for gradient calculations. For the remaining steps in BDOF processing, if any samples and gradient values ​​outside the CU boundary are needed, the samples and gradient values ​​can be filled (e.g., repeated) from their nearest neighbors.

[0095] In some examples, a sample-based BDOF may be used instead of a block-based BDOF. In a sample-based BDOF, motion correction is performed on a sample-by-sample basis rather than on a block-by-block basis. x , v y ). The coding block is divided into 8×8 sub-blocks. For each sub-block, it is determined whether to apply BDOF by checking the SAD between two reference sub-blocks against a threshold. If it is decided to apply BDOF to a sub-block, a sliding 5×5 window is used for each sample in the sub-block and the existing BDOF process is applied to each sliding window to derive v x and v y . Apply the exported motion correction (v x , v y ) to adjust the bidirectional prediction sample value of the center sample of the window.

[0096] In some examples, multi-pass DMVR can be used. In one example, in the first pass, bilateral matching (BM) is applied to the coding block. In the second pass, BM is applied to each 16×16 sub-block within the coding block. In the third pass, the MV in each 8×8 sub-block is corrected by applying bidirectional optical flow (BDOF). The corrected MV is stored for spatial and temporal motion vector prediction.

[0097] Specifically, the first pass performs block-based bilateral matching MV correction. In the first pass, a revised MV is derived by applying BM to the coding block. Similar to decoder-side motion vector correction (DMVR), in a bidirectional prediction operation, a revised MV is searched around two initial MVs (MV0 and MV1) in the reference image lists L0 and L1. Based on the minimum bilateral matching cost between the two reference blocks in L0 and L1, revised MVs (MV0_pass1 and MV1_pass1) are derived around the initial MVs. The bilateral matching cost can be calculated by any suitable error measurement metric for measuring the error between the two reference blocks in L0 and L1. In one example, the bilateral matching cost includes a term that is the sum of absolute errors (SAD) between corresponding samples in the two reference blocks in L0 and L1.

[0098] BM can perform a local search to derive integer sample precision intDeltaMV. The local search applies a 3×3 square search pattern, looping in the horizontal search range [-sHor, sHor] and the vertical search range [-sVer, sVer], where the values ​​of sHor and sVer are determined by the block dimension and the maximum values ​​of sHor and sVer are 8.

[0099] The bilateral matching cost is calculated as: bilCost = mvDistanceCost + sadCost. When the block size cbW×cbH is greater than 64, the mean removed SAD (MRSAD) cost function is applied to remove the DC effect of the distortion between reference blocks. When the bilCost of the center point of the 3×3 search pattern has the minimum cost, the intDeltaMV local search is terminated. Otherwise, the current minimum cost search point becomes the new center point of the 3×3 search pattern, and the search for the minimum cost continues until the end of the search range is reached.

[0100] The existing fractional sample correction is further applied to derive the final deltaMV. Then, the corrected MV after the first pass is exported as: MV0_pass1 = MV0 + deltaMV formula (19) MV1_pass1 = MV1 – deltaMV formula (20)

[0101] In the second pass, subblock-based bilateral matching MV correction is performed. Specifically, in the second pass, the corrected MV is derived by applying BM to the 16×16 grid subblocks. For each subblock, the corrected MV is searched around the two MVs (MV0_pass1 and MV1_pass1) obtained on the first pass in the reference image lists L0 and L1. The corrected MVs (MV0_pass2(sbIdx2) and MV1_pass2(sbIdx2)) are derived based on the minimum bilateral matching cost between the two reference subblocks in L0 and L1.

[0102] For each sub-block, BM performs a full search to derive integer sample precision intDeltaMV. The full search has a search range of [-sHor, sHor] in the horizontal direction and a search range of [-sVer, sVer] in the vertical direction, where the values ​​of sHor and sVer are determined by the block dimension and the maximum values ​​of sHor and sVer are 8.

[0103] The bilateral matching cost is calculated by applying the cost factor to the sum of absolute transformed differences (SATD) cost between the two reference sub-blocks, such as: bilCost = satdCost × costFactor. In some examples, the search area (2 × sHor + 1) × (2 × sVer + 1) is divided into up to 5 diamond search areas.

[0104] Figure 6 The search area (600) in some examples is shown. The search area (600) is divided into five search areas (601)-(605). The shape of the search area is similar to a diamond.

[0105] In some examples, each search region is assigned a costFactor determined by the distance (intDeltaMV) between each search point and the starting MV, and each diamond region is processed in order starting from the center of the search region. In each region, the search points are processed in raster scan order starting from the upper left corner to the lower right corner of the region. When the minimum bilCost in the current search region is less than a threshold whose value is equal to sbW×sbH, the integer pixel (int-pel) full search is terminated, otherwise, the int-pel full search continues to the next search region until all search points are checked. In addition, if the difference between the previous minimum cost and the current minimum cost in the iteration is less than a threshold whose value is equal to the area of ​​the block, the search process is terminated.

[0106] In some examples, fractional sample correction, such as DMVR fractional sample correction in VVC, is further applied to derive the final deltaMV (sbIdx2). Then, the corrected MV of the second channel is derived as: MV0_pass2(sbIdx2) = MV0_pass1 + deltaMV(sbIdx2) Formula (21) MV1_pass2(sbIdx2) = MV1_pass1 – deltaMV(sbIdx2) Formula (22)

[0107] In the third pass, sub-block based bidirectional optical flow MV correction can be performed. Specifically, in the third pass, the corrected MV is derived by applying BDOF to the 8×8 grid sub-blocks. For each 8×8 sub-block, the BDOF correction is applied to derive the scaled Vx and Vy without clipping from the corrected MV of the parent-child block of the second pass. The derived bioMv(Vx, Vy) is rounded to 1 / 16 sample accuracy and clipped between -32 and 32. The corrected MVs (MV0_pass3(sbIdx3) and MV1_pass3(sbIdx3)) of the third pass are exported as: MV0_pass3(sbIdx3) = MV0_pass2(sbIdx2) + bioMv formula (23) MV1_pass3(sbIdx3) = MV0_pass2(sbIdx2) – bioMv formula (24)

[0108] It should be noted that out-of-frame boundary conditions may occur in bidirectional motion compensation. When a motion vector partially or completely points to an area outside the reference image, it is called out of boundary (OOB) or out of frame boundary (OOFB).

[0109] In bidirectional motion compensation, in some examples, out-of-boundary (OOB) prediction samples are discarded and only non-OOB prediction values ​​(when available) are used to generate the final prediction value. For example, let Pos_x i,j and Pos_y i,j Denote as the position of a predicted sample in the current block, and Recorded as the MV of the current block; Pos LeftBdry 、Pos RightBdry 、Pos TopBdry and Pos BottomBdryis the position of the four boundaries of the image. When at least one of the following formulas (25) to (28) is satisfied, the prediction sample is considered OOB (or OOFB): Wherein half_pixel is equal to 8, indicating a half-pixel sampling distance in a 1 / 16 pixel sampling accuracy. Formula (25) to Formula (28) may be referred to as an OOB (OOFB) condition or an OOB (OOFB) standard.

[0110] In some examples, after checking the OOB (OOFB) condition of each sample, the final prediction sample of the bidirectional block can be generated as in the following pseudo code:

[0111] It is important to note that when BCW is enabled, the OOB checking process also applies.

[0112] Note that in some examples, the sample adaptive bi-directional prediction process is only applied to prediction units where at least the reference block is first detected as partially or fully out of bounds. Therefore, the block level OOB criterion is checked first. If both prediction blocks are non-OOB, then the usual bi-directional prediction occurs.

[0113] Bidirectional motion compensation for samples in a current image is based on prediction samples in two reference images. During bidirectional prediction motion compensation, out-of-boundary (OOB) prediction samples are discarded, and only non-OOB prediction values ​​(prediction samples) (when available) are used to generate final prediction values ​​for samples in the current image. According to one aspect of the present disclosure, various bidirectional encoding and decoding tools are used to provide finer motion compensation to improve bidirectional motion compensation. For example, BDOF can provide finer motion compensation per sample, and DMVR can use bilateral matching to provide block-level or sub-block-level motion correction. Some bidirectional encoding tools can generate offset information based on reference samples of a reference image. For example, DMVR can calculate a cost value associated with a candidate MV offset by bilateral matching of reference samples of a reference image, and determine the optimal MV offset based on the cost value. In another example, BDOF can calculate an offset to the prediction (e.g., the adjustment amount in formula (17)) based on the motion correction and gradient calculated from the reference samples of the reference image. It should be noted that when OOB (OOFB) prediction is used in a bidirectional coding tool (such as using padding values ​​from adjacent samples as reference samples for OOB (OOFB)), such as determining the MV offset through bilateral matching processing in DMVR and performing the offset calculation in BDOF, the offset obtained by the bidirectional coding tool (such as the modified MV with OOB conditions in DMVR, the offset in BDOF) may be inaccurate and may result in coding losses.

[0114] Some aspects of the present disclosure provide techniques for improving the accuracy of offset calculations in bidirectional coding tools with OOB (OOFB) conditions, such as offset calculations in BDOF with OOB (OOFB) conditions, MV offsets processed by bilateral matching in DMVR with OOB (OOFB) conditions, and the like. In some examples, an encoder / decoder performs a bidirectional prediction with correction on a current block in a current image. The correction determines an offset based on multiple calculations, and each calculation includes reference samples of a first reference image and a second reference image for the current image. The encoder / decoder determines that a first calculation of the multiple calculations uses a first reference sample in a first reference image, and the first reference sample is an out-of-boundary (OOB) prediction sample. The OOB prediction sample of the reference image is outside a frame boundary of the reference image. The encoder / decoder performs the correction of determining the offset based on the multiple calculations while excluding at least the first reference sample, and reconstructs the current block according to the bidirectional prediction with the offset.

[0115] In the following description, terms such as Out of Boundary (OOB), Out of Frame Boundary (OOFB), Bilateral Matching (BM), Decoder-Side Motion Vector Correction (DMVR), Bidirectional Optical Flow (BDOF) are used, and some definitions of these terms are provided in the above description. In the following description, discarding a coding tool or operation means not allowing or not selecting such a tool or operation as a result.

[0116] According to one aspect of the present disclosure, when a bilateral matching block has an OOB (also referred to as OOFB) prediction sample, the bilateral matching block is discarded. More specifically, during bilateral matching, if at least one of a pair of blocks has an OOB (OOFB) prediction sample, the block pair will be discarded. For example, when bilateral matching is used, a pair of reference blocks (also referred to as a block pair or bilateral matching block) is identified in each reference image of the current image, for example based on a candidate motion vector, and the distortion between the two reference blocks can be calculated. In one example, when a reference block has a sample that satisfies an OOB condition, such as one of formulas (25)-(28), bilateral matching is not performed on the pair of reference blocks, in other words, the pair of reference blocks is discarded, and the candidate motion vector may not be allowed.

[0117] In some implementation examples, the relative cost value of the discarded block pair may be set to a maximum value or marked as unavailable. For example, bilateral matching is not performed, and the maximum cost value is assigned as the cost value of the pair of reference blocks (block pair).

[0118] In some examples, when at least one integer block match search around the best integer position has a maximum cost value or an unavailable cost value, fractional block matching is used instead of a method based on secondary prediction. For example, DMVR can perform a motion vector search in two steps, such as a first step of an integer precision motion search step and a second step of fractional sample correction. In the first step, 25 integer sample offset points are searched, and when a pair of reference blocks identified according to the integer sample offsets are available (there are no OOB prediction samples in the reference blocks), BM can be performed for the points of the integer sample offsets. However, when at least one reference block identified according to the integer sample offset has an OOB prediction sample, the maximum cost value or an unavailable cost value is assigned to the pair of reference blocks (also referred to as integer blocks) associated with the integer sample offset. The first step determines the best integer position with the lowest cost value (e.g., calculated by BM). In the second step, in some examples, when four neighboring integer positions around the best integer position are available (e.g., BM operations are performed), a method based on secondary prediction, such as using a parametric error surface formula, can be performed to save computational complexity. However, in some examples, when at least one neighboring position (neighboring integer offset position) around the best integer position (integer offset position) has a maximum cost value or an unavailable cost value, for example, because there is at least an OOB prediction sample in the reference block of the neighboring integer offset position, then fractional block matching (e.g., using fractional sample offsets to identify a reference block, using bilateral matching to determine a cost value for the reference block, and using the cost value to determine the best fractional position) may be used instead of a method based on secondary prediction.

[0119] In some examples, sub-block level OOB (OOFB) conditions are checked for sub-block based DMVR. In some examples, for a sub-block, during a first step of integer sample offset search of DMVR, when a pair of reference sub-blocks associated with an MV offset candidate has at least one OOB reference sub-block (e.g., includes at least one reference sample outside the boundary of a reference image), the MV offset candidate (and the pair of reference sub-blocks associated with the MV offset candidate) is not used (discarded). During a second step of fractional sample correction of the sub-block, when at least one neighboring integer position around the best integer position has a maximum cost value or an unavailable cost value, e.g., due to the presence of at least an OOB prediction sample in a reference sub-block at a neighboring integer position, fractional block matching (e.g., using fractional sample offsets to identify a reference sub-block, using bilateral matching to determine a cost value for the reference sub-block, and using the cost value to determine the best fractional position) may be used instead of a quadratic prediction based approach.

[0120] According to one aspect of the present disclosure, when the corresponding block / sub-block satisfies the OOB (OOFB) condition, the BDOF at the block level / sub-block level may be discarded. In some examples, in order to calculate an offset for applying the BDOF at the block level of the current block, a first reference sample in the first reference image and a second reference sample in the second reference image are used. When at least one of the first reference samples or one of the second reference samples is an OOB prediction sample, the BDOF is discarded at the block level and the current block is predicted without the BDOF.

[0121] In some examples, to calculate an offset for applying BDOF at a sub-block level for a current sub-block, a first reference sample in a first reference image and a second reference sample in a second reference image are used. When at least one of the first reference samples or one of the second reference samples is an OOB sample, the BDOF is discarded at the sub-block level and the current sub-block is predicted without the BDOF.

[0122] According to one aspect of the present disclosure, OOB prediction samples may be discarded during a bilateral matching process. More specifically, samples with OOB prediction in one of the bilateral matching blocks are not used for cost calculation. In order to make a fair cost comparison, a normalized bilateral matching cost is derived by dividing the cost by the number of non-OOB samples. In some examples, when bilateral matching is used, a pair of reference blocks (also referred to as a block pair or bilateral matching block) is identified in each reference image of the current image, for example based on a candidate motion vector (e.g., a combination of an initial MV and an MV offset candidate), and a distortion measure between the two reference blocks may be calculated. In one example, when a reference block has one or more samples that satisfy an OOB condition (such as one or more of formulas (25)-(28)), the bilateral matching cost of the pair of reference blocks is calculated without one or more samples of OOB. In some examples, the bilateral matching cost is normalized, for example based on the number of sample pairs used in the calculation of the bilateral matching cost. In one example, the bilateral matching cost is calculated as the distortion sum between multiple sample pairs (non-OOB samples) within the boundary of the reference image, such as the absolute error sum of multiple sample pairs. In the calculation of the bilateral matching cost, the bilateral matching cost is normalized based on the number of sample pairs (eg, averaged).

[0123] In some examples, the number of samples used to calculate the normalized BM cost can be the minimum (#L0 non-OOB samples, #L1 non-OOB samples). This can be applied to the case where both L0 and L1 prediction values ​​are OOB, as well as the case where only one of the L0 and L1 prediction values ​​is OOB. In an example, the reference block in the first reference image from the reference image list L0 has OOB samples, while the reference block in the second reference image from the reference image list L1 does not have OOB samples, so the number of non-OOB samples in the reference block of the first reference image (denoted as #L0 non-OOB samples) is less than the number of non-OOB samples in the reference block of the second reference image from the reference image list L1 (denoted as #L1 non-OOB samples), so the number of non-OOB samples in the reference block of the first reference image (#L0 non-OOB samples) is used in normalization.

[0124] According to one aspect of the present disclosure, when the ratio of OOB prediction samples in a bilateral matching block is greater than a predefined threshold, the bilateral matching block is discarded. More specifically, when the number of OOB prediction samples exceeds a predetermined ratio of all prediction samples used in bilateral matching, the block pair is discarded during bilateral matching. Example values ​​of the ratio include, but are not limited to, 1 / 2, 2 / 3. In some examples, when bilateral matching is used, a pair of reference blocks (also referred to as a block pair or bilateral matching block) is identified in each reference image of the current image, for example, based on a candidate motion vector (e.g., a combination of an initial MV and an MV offset candidate), and the distortion between the two reference blocks can be calculated. In one example, the reference block has one or more samples that satisfy an OOB condition (such as one or more of formulas (25)-(28)), and the one or more samples are referred to as OOB samples. When the number of OOB samples is less than or equal to a predetermined threshold, the bilateral matching cost of the pair of reference blocks is calculated without OOB samples. For example, the bilateral matching cost is normalized based on the number of sample pairs (e.g., non-OOB sample pairs) used in the calculation of the bilateral matching cost. However, when the number of OOB samples is greater than a predefined threshold (eg, the ratio of the number of OOB samples in the reference block to the total number of samples is greater than a ratio threshold), the pair of reference blocks may be discarded and the candidate motion vector is not further considered.

[0125] Figure 7A flow chart outlining a process (700) according to one embodiment of the present disclosure is shown. The process (700) may be used in a video decoder. In various embodiments, the process (700) is performed by a processing circuit, such as a processing circuit that performs the functions of a video decoder (110), a processing circuit that performs the functions of a video decoder (210), etc. In some embodiments, the process (700) is implemented in software instructions, so when the processing circuit executes the software instructions, the processing circuit performs the process (700). The process starts at (S701) and proceeds to (S710).

[0126] In (S710), a coded video bitstream is received. The coded video bitstream includes coded information of a current block in a current picture. The coded information is used to indicate a bidirectional prediction with a modification, the modification is based on multiple calculations to determine an offset, and each calculation includes reference samples of a first reference picture and a second reference picture for the current picture.

[0127] In (S720), it is determined that a first calculation in the plurality of calculations uses a first reference sample in a first reference picture, the first reference sample being an out-of-boundary (OOB) prediction sample. An OOB prediction sample of a reference picture is defined as a sample outside a frame boundary of the reference picture.

[0128] In (S730), the correction is performed, the correction determining the offset based on the plurality of calculations excluding at least the first reference sample.

[0129] In (S740), the current block is reconstructed according to the bidirectional prediction with the offset.

[0130] According to one aspect of the present disclosure, the modification includes a decoder-side motion vector modification (DMVR) that determines a motion vector offset based on calculation of a bilateral matching (BM) cost value associated with a motion vector offset candidate.

[0131] In some examples, the first calculation calculates a first BM cost as a distortion measure between corresponding reference samples in a pair of reference blocks associated with a first motion vector offset candidate, and a first reference block in the pair of reference blocks includes the first reference sample as the OOB prediction sample. The first motion vector offset candidate is discarded and thus the first calculation is excluded from the multiple calculations.

[0132] In order to discard the first motion vector offset candidate, in one example, the maximum BM cost value is assigned to the first motion vector offset candidate. In another example, the first motion vector offset candidate is marked as unavailable.

[0133] In some examples, the first motion vector offset candidate corresponds to an integer position that is a neighboring position of the specific position having a lowest BM cost. In one example, in response to discarding the first motion vector offset candidate, it is determined to use fractional block matching instead of fractional sample revised secondary prediction at the specific position. The fractional sample revision for the specific position is performed by calculating a BM cost value at the position corresponding to the fractional sample offset.

[0134] In some examples, the first calculation calculates a first BM cost as a distortion measure between corresponding reference samples in a pair of reference blocks associated with the first motion vector offset candidate, and a first reference block in the pair of reference blocks includes the first reference sample as the OOB prediction sample. The first calculation is performed excluding the first reference sample from the distortion measure. In some examples, the first BM cost is normalized based on a logarithm of the corresponding reference sample in the first calculation.

[0135] In some examples, a plurality of reference sample pairs in the pair of reference blocks may be determined, each reference sample pair including at least a reference sample that is an OOB prediction sample. Subsequently, it may be determined whether a ratio of the number of pairs of the plurality of reference sample pairs to the total number of pairs of reference sample pairs in the pair of reference blocks is above a threshold. In response to the ratio being above the threshold, the first motion vector offset candidate is discarded and the first calculation is thus excluded from the plurality of calculations. In response to the ratio being below the threshold, the first calculation is performed with the plurality of reference sample pairs excluded from the distortion measure.

[0136] According to one aspect of the present disclosure, the current block may be a coding block or may be a sub-block of a coding block.

[0137] According to one aspect of the present disclosure, the modification includes a bidirectional optical flow (BDOF) that uses optical flow to calculate a prediction offset based on a correlation calculation of gradients performed according to the reference samples of the first reference image and the second reference image.

[0138] Then, the process proceeds to (S799) and terminates.

[0139] The process (700) may be appropriately adjusted. Steps in the process (700) may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0140] Figure 8A flow chart outlining a process (800) according to one embodiment of the present disclosure is shown. The process (800) may be used in a video encoder. In various embodiments, the process (800) is performed by a processing circuit, such as a processing circuit that performs the functions of a video encoder (103), a processing circuit that performs the functions of a video encoder (303), etc. In some embodiments, the process (800) is implemented in software instructions, so when the processing circuit executes the software instructions, the processing circuit performs the process (800). The process starts at (S801) and proceeds to (S810).

[0141] In (S810), it is determined to encode a current block in a current image using bi-directional prediction with modification. The modification determines an offset based on a plurality of calculations, and each calculation includes reference samples of a first reference image and a second reference image for the current image.

[0142] In (S820), it is determined that a first calculation in the plurality of calculations uses a first reference sample in a first reference picture, the first reference sample being an out-of-boundary (OOB) prediction sample. An OOB prediction sample of a reference picture is defined as a sample outside a frame boundary of the reference picture.

[0143] In (S830), the correction is performed, the correction determining the offset based on the plurality of calculations while excluding at least the first reference sample.

[0144] In (S840), the current block is reconstructed according to the bidirectional prediction with the offset.

[0145] According to one aspect of the present disclosure, the modification includes a decoder-side motion vector modification (DMVR) that determines a motion vector offset based on calculation of a bilateral matching (BM) cost value associated with a motion vector offset candidate.

[0146] In some examples, the first calculation calculates a first BM cost as a distortion measure between corresponding reference samples in a pair of reference blocks associated with a first motion vector offset candidate, and a first reference block in the pair of reference blocks includes the first reference sample as the OOB prediction sample. The first motion vector offset candidate is discarded and thus the first calculation is excluded from the multiple calculations.

[0147] In order to discard the first motion vector offset candidate, in one example, the maximum BM cost value is assigned to the first motion vector offset candidate. In another example, the first motion vector offset candidate is marked as unavailable.

[0148] In some examples, the first motion vector offset candidate corresponds to an integer position that is a neighboring position of the specific position having a lowest BM cost. In one example, in response to discarding the first motion vector offset candidate, it is determined to use fractional block matching instead of fractional sample revised secondary prediction at the specific position. The fractional sample revision for the specific position is performed by calculating a BM cost value at the position corresponding to the fractional sample offset.

[0149] In some examples, the first calculation calculates a first BM cost as a distortion measure between corresponding reference samples in a pair of reference blocks associated with the first motion vector offset candidate, and a first reference block in the pair of reference blocks includes the first reference sample as the OOB prediction sample. The first calculation is performed excluding the first reference sample from the distortion measure. In some examples, the first BM cost is normalized based on a logarithm of the corresponding reference sample in the first calculation.

[0150] In some examples, a plurality of reference sample pairs in the pair of reference blocks may be determined, each reference sample pair including at least a reference sample that is an OOB prediction sample; and then, it may be determined whether a ratio of the number of pairs of the plurality of reference sample pairs to the total number of pairs of reference sample pairs in the pair of reference blocks is above a threshold. In response to the ratio being above the threshold, the first motion vector offset candidate is discarded and the first calculation is thus excluded from the plurality of calculations. In response to the ratio being below the threshold, the first calculation is performed while excluding the plurality of reference sample pairs from the distortion measure.

[0151] According to one aspect of the present disclosure, the current block may be a coding block or may be a sub-block of a coding block.

[0152] According to one aspect of the present disclosure, the modification includes a bidirectional optical flow (BDOF) that uses optical flow to calculate a prediction offset based on a correlation calculation of gradients performed according to the reference samples of the first reference image and the second reference image.

[0153] Then, the process proceeds to step (S899) and terminates.

[0154] The process (800) may be appropriately adjusted. Steps in the process (800) may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0155] The above techniques may be implemented as computer software via computer-readable instructions and physically stored in one or more computer-readable media. Fig. 9 A computer system (900) is shown that is suitable for implementing certain embodiments of the disclosed subject matter.

[0156] Computer software may be encoded in any suitable machine code or computer language, and may be assembled, compiled, linked, or otherwise used to create code comprising instructions that may be directly executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or executed through decoding, microcode, or the like.

[0157] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablets, servers, smartphones, gaming devices, IoT devices, and the like.

[0158] Fig. 9 The components shown for the computer system (900) are exemplary in nature and are not intended to impose any limitation on the scope of use or functionality of computer software implementing embodiments of the present disclosure. Nor should the configuration of components be interpreted as having any dependency or requirement on any one or combination of components shown in the exemplary embodiment of the computer system (900).

[0159] The computer system (900) may include certain human-computer interface input devices. Such human-computer interface input devices may respond to input from one or more human users through tactile input (e.g., keyboard input, sliding, data glove movement), audio input (e.g., sound, applause), visual input (e.g., gestures), olfactory input (not shown), etc. The human-computer interface device may also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (e.g., speech, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still image camera), and videos (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0160] The human-computer interface input device may include one or more of the following (only one of each is drawn): keyboard (901), mouse (902), touchpad (903), touch screen (910), data gloves (not shown), joystick (905), microphone (906), scanner (907), camera (908).

[0161] The computer system (900) may also include certain human-computer interface output devices. Such human-computer interface output devices may stimulate one or more human user senses through, for example, tactile output, sound, light, and smell / taste. Such human-computer interface output devices may include tactile output devices (e.g., tactile feedback through a touch screen (910), a data glove (not shown), or a joystick (905), but there may also be tactile feedback devices that are not used as input devices), audio output devices (e.g., speakers (909), headphones (not shown)), visual output devices (e.g., screens (910) including cathode ray tube (CRT) screens, liquid crystal (LCD) screens, plasma screens, organic light emitting diode (OLED) screens, each of which has or does not have a touch screen input function, each of which has or does not have a tactile feedback function - some of which can output two-dimensional visual output or output of more than three dimensions through means such as stereoscopic picture output; virtual reality glasses (not shown), holographic displays, and smoke boxes (not shown)) and printers (not shown).

[0162] The computer system (900) may also include human-accessible storage devices and their associated media, such as optical media including high-density read-only / rewritable optical disks (CD / DVD ROM / RW) (920) with CD / DVD or similar media (921), thumb drives (922), removable hard disk drives or solid state drives (923), traditional magnetic media such as tapes and floppy disks (not shown), special-purpose ROM / ASIC / PLD-based devices such as security software dongles (not shown), and the like.

[0163] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0164] The computer system (900) may also include an interface (954) to one or more communication networks (955). For example, the network may be wireless, wired, or optical. The network may also be a local area network, a wide area network, a metropolitan area network, an in-vehicle network, an industrial network, a real-time network, a delay-tolerant network, and the like. Examples of networks also include local area networks such as Ethernet, wireless local area networks, cellular networks (including GSM, 3G, 4G, 5G, LTE, etc.), television wired or wireless wide area digital networks (including cable television, satellite television, and terrestrial broadcast television), in-vehicle and industrial networks (including CANBus), and the like. Some networks typically require an external network interface adapter for connecting to some universal data port or peripheral bus (949) (e.g., a USB port of the computer system (900)); other networks are typically integrated into the kernel of the computer system (900) by connecting to a system bus as described below (e.g., an Ethernet interface integrated into a PC computer system or a cellular network interface integrated into a smart phone computer system). By using any of these networks, the computer system (900) can communicate with other entities. The communication can be one-way, for receiving only (e.g., broadcast television), one-way for sending only (e.g., CAN bus to certain CAN bus devices), or two-way, such as to other computer systems via a local or wide area digital network. Each of the above networks and network interfaces can use certain protocols and protocol stacks.

[0165] The above-mentioned human-machine interface devices, human-accessible storage devices, and network interfaces may be connected to the kernel (940) of the computer system (900).

[0166] The kernel (940) may include one or more central processing units (CPUs) (941), graphics processing units (GPUs) (942), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (943), hardware accelerators for specific tasks (944), graphics adapters (950), etc. These devices, as well as read-only memory (ROM) (945), random access memory (946), internal mass storage (e.g., internal non-user accessible hard disk drives, solid-state drives (SSDs)), etc.) (947), etc., may be connected via a system bus (948). In some computer systems, the system bus (948) may be accessed in the form of one or more physical plugs so that it may be expanded by additional central processing units, graphics processing units, etc. Peripheral devices may be directly attached to or connected to the kernel's system bus (948) via a peripheral bus (949). In one example, a screen (910) may be connected to a graphics adapter (950). The architecture of the peripheral bus includes a PCI peripheral controller interface, a USB universal serial bus, etc.

[0167] The CPU (941), GPU (942), FPGA (943) and accelerator (944) can execute certain instructions, which can be combined to form the above-mentioned computer code. The computer code can be stored in ROM (945) or RAM (946). Transition data can also be stored in RAM (946), while permanent data can be stored in, for example, internal mass storage (947). Fast storage and retrieval of any memory device can be achieved by using a cache memory, which can be closely associated with one or more CPUs (941), GPUs (942), mass storage (947), ROM (945), RAM (946), etc.

[0168] The computer readable medium may have computer code thereon for performing various computer-implemented operations. The medium and computer code may be specially designed and constructed for the purpose of the present application, or may be the medium and code well known and available to those skilled in the art of computer software.

[0169] As an example and not limitation, a computer system having architecture (900), in particular a kernel (940), can provide the function of executing software contained in one or more tangible computer-readable media as a processor (including a CPU, GPU, FPGA, accelerator, etc.). Such a computer-readable medium can be a medium associated with the above-mentioned user-accessible mass storage, as well as a specific memory of the kernel (940) having non-volatility, such as a kernel internal mass storage (947) or ROM (945). Software implementing various embodiments of the present disclosure can be stored in such a device and executed by the kernel (940). Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software can enable the kernel (940), in particular the processor therein (including a CPU, GPU, FPGA, etc.) to perform a specific process or a specific part of a specific process described herein, including defining a data structure stored in RAM (946) and modifying such a data structure according to a software-defined process. Additionally or alternatively, the computer system may provide functionality hardwired in logic or otherwise contained in circuitry (e.g., accelerator (944)) that may operate in place of or in conjunction with software to perform specific processes or specific portions of specific processes described herein. Where appropriate, references to software may include logic and vice versa. Where appropriate, references to computer-readable media may include circuitry (e.g., an integrated circuit (IC)) storing execution software, circuitry containing execution logic, or both. The present disclosure includes any suitable combination of hardware and software.

[0170] The use of "at least one" or "one" in this disclosure is intended to include any one or combination of the recited elements. For example, reference to at least one of A, B, or C; at least one of A, B, and C; at least one of A, B, and / or C; and at least one of A through C is intended to include only A, only B, only C, or any combination thereof. Reference to one of A or B and one of A and B is intended to include A, B, or (A and B). Where applicable, the use of "one" does not exclude any combination of the recited elements, for example when these elements are not mutually exclusive.

[0171] Although the present application has described a number of exemplary embodiments, various changes, arrangements and various equivalent substitutions of the embodiments are within the scope of the present application. Therefore, it should be understood that those skilled in the art can design a variety of systems and methods, which, although not explicitly shown or described herein, embody the principles of the present application and are therefore within the spirit and scope of the present application.

Claims

1. A video decoding method, characterized in that: The method comprises: Receiving an encoded video bitstream, the encoded video bitstream comprising encoded information for a current block in a current picture, the encoded information being used to indicate bidirectional prediction with a modification, the modification being based on a plurality of calculations to determine an offset, and each calculation comprising reference samples of a first reference picture and a second reference picture for the current picture; Determining that a first calculation of the plurality of calculations uses a first reference sample of the first reference picture, the first reference sample being an out-of-boundary (OOB) prediction sample, the OOB prediction sample of the reference picture being outside a frame boundary of the reference picture; performing the modifying, the modifying determining the offset based on the plurality of calculations while excluding at least the first reference sample; and The current block is reconstructed according to the bidirectional prediction with the offset.

2. The method according to claim 1, characterized in that The modification includes a decoder-side motion vector modification (DMVR) that determines a motion vector offset based on calculation of a bilateral matching (BM) cost value associated with a motion vector offset candidate.

3. The method according to claim 2, characterized in that The first calculation calculates a first BM cost as a distortion measure between corresponding reference samples in a pair of reference blocks associated with a first motion vector offset candidate, and a first reference block in the pair of reference blocks includes the first reference sample as the OOB prediction sample, and determining the offset further comprises: The first motion vector offset candidate is discarded for excluding the first calculation from the plurality of calculations.

4. The method according to claim 3, characterized in that Discarding the first motion vector offset candidate further comprises at least one of the following: assigning a maximum BM cost value associated with the first motion vector offset candidate; and / or The first motion vector offset candidate is marked as unavailable.

5. The method according to claim 3, characterized in that: The first motion vector offset candidate corresponds to an integer position, the integer position being a neighboring position of a specific position having a lowest BM cost, and the method further comprises: In response to discarding the first motion vector offset candidate, determining to use fractional block matching instead of fractional sample corrected secondary prediction at the specific location; and The fractional sample correction for the specific position is performed by calculating a BM cost value at the position corresponding to the fractional sample offset.

6. The method according to claim 2, characterized in that The first calculation calculates a first BM cost as a distortion measure between corresponding reference samples in a pair of reference blocks associated with a first motion vector offset candidate, and a first reference block in the pair of reference blocks includes the first reference sample as the OOB prediction sample, and determining the offset further comprises: The first calculation is performed excluding the first reference sample from the distortion measure.

7. The method according to claim 6, characterized in that The method further comprises: The first BM cost is normalized based on a logarithm of the corresponding reference sample in the first calculation.

8. The method according to claim 6, characterized in that The performing of the first calculation further comprises: Determine a plurality of reference sample pairs in the pair of reference blocks, each reference sample pair including at least a reference sample as an OOB prediction sample; determining whether a ratio of the number of pairs of the plurality of reference sample pairs to the total number of pairs of reference sample pairs in the pair of reference blocks is higher than a threshold; In response to the ratio being above the threshold, discarding the first motion vector offset candidate for excluding the first calculation from the plurality of calculations; and In response to the ratio being below the threshold, the first calculation is performed excluding the plurality of reference sample pairs from the distortion measure.

9. The method according to claim 1, characterized in that: The current block corresponds to at least one of the following: a coding block; and / or a sub-block of the coding block.

10. The method according to claim 1, characterized in that The modification includes bidirectional optical flow (BDOF) using optical flow to calculate prediction offsets based on correlation calculations of gradients from the reference samples of the first and second reference images.

11. A video decoding device, characterized in that: The apparatus comprises a processing circuit configured to: Receiving an encoded video bitstream, the encoded video bitstream comprising encoded information for a current block in a current picture, the encoded information indicating bidirectional prediction with a modification, the modification being based on a plurality of calculations to determine an offset, and each calculation comprising reference samples in a first reference picture and a second reference picture for the current picture; Determining that a first calculation of the plurality of calculations uses a first reference sample of the first reference picture, the first reference sample being an out-of-boundary (OOB) prediction sample, the OOB prediction sample of the reference picture being outside a frame boundary of the reference picture; performing the modifying, the modifying determining the offset based on the plurality of calculations while excluding at least the first reference sample; as well as The current block is reconstructed according to the bidirectional prediction with the offset.

12. The device according to claim 11, characterized in that The modification includes a decoder-side motion vector modification (DMVR) that determines a motion vector offset based on calculation of a bilateral matching (BM) cost value associated with a motion vector offset candidate.

13. The device according to claim 12, characterized in that The first calculation calculates a first BM cost as a distortion measure between corresponding reference samples in a pair of reference blocks associated with a first motion vector offset candidate, and a first reference block in the pair of reference blocks includes the first reference sample as the OOB prediction sample, and the processing circuit is configured to: The first motion vector offset candidate is discarded for excluding the first calculation from the plurality of calculations.

14. The device according to claim 13, characterized in that The processing circuit is configured to perform at least one of the following: assigning a maximum BM cost value associated with the first motion vector offset candidate; and / or The first motion vector offset candidate is marked as unavailable.

15. The device according to claim 13, characterized in that The first motion vector offset candidate corresponds to an integer position, the integer position being a neighboring position of a specific position having a lowest BM cost, and the processing circuit is configured to: In response to discarding the first motion vector offset candidate, determining to use fractional block matching instead of fractional sample corrected secondary prediction at the specific location; as well as The fractional sample correction for the specific position is performed by calculating a BM cost value at the position corresponding to the fractional sample offset.

16. The device according to claim 12, characterized in that The first calculation calculates a first BM cost as a distortion measure between corresponding reference samples in a pair of reference blocks associated with a first motion vector offset candidate, and a first reference block in the pair of reference blocks includes the first reference sample as the OOB prediction sample, and the processing circuit is configured to: The first calculation is performed excluding the first reference sample from the distortion measure.

17. The device according to claim 16, characterized in that The processing circuit is configured to: The first BM cost is normalized based on a logarithm of the corresponding reference sample in the first calculation.

18. The device according to claim 16, characterized in that The processing circuit is configured to: Determine a plurality of reference sample pairs in the pair of reference blocks, each reference sample pair including at least a reference sample as an OOB prediction sample; determining whether a ratio of the number of pairs of the plurality of reference sample pairs to the total number of pairs of reference sample pairs in the pair of reference blocks is higher than a threshold; responsive to the ratio being above the threshold, discarding the first motion vector offset candidate for excluding the first calculation from the plurality of calculations; as well as In response to the ratio being below the threshold, the first calculation is performed excluding the plurality of reference sample pairs from the distortion measure.

19. The device according to claim 11, characterized in that The current block corresponds to at least one of the following: a coding block; and / or a sub-block of the coding block.

20. The device according to claim 11, characterized in that The modification includes bidirectional optical flow (BDOF) using optical flow to calculate prediction offsets based on correlation calculations of gradients from the reference samples of the first and second reference images.

Images