Outbound check in video coding

Abstract

A method of encoding and decoding a block of pixels using an out-of-bound (OOB) check is provided. A video codec receives data to be encoded or decoded as a current block of a current picture of a video. A video codec identifies a first reference block in a first reference picture based on a first block vector of a current block. A video codec performs an OOB check on a first reference block with respect to a boundary of a subunit of a first reference picture. The video codec generates a predictor for the current block based on the first reference block and based on the OOB check. When a sample of the first reference block is OOB, this sample is not used for bidirectional motion compensation of the current block. The video codec encodes or decodes the current block by using the generated predictor.

Description

[0001] Cross-references

[0002] The present invention is a non-provisional application of and claims priority to U.S. Provisional Patent Application No. 63 / 369,087 filed on July 22, 2022. The entire contents of this U.S. Provisional Patent Application are incorporated herein by reference. Technical Field

[0003] The present invention relates to a video coding and decoding system, and in particular to a method for coding and decoding a pixel block using out-of-bound (OOB) checks. Background Art

[0004] Unless otherwise indicated herein, the approaches described in this section are not prior art to the claims listed below and are not admitted to be prior art by inclusion in this section.

[0005] High Efficiency Video Coding (HEVC) is an international video coding standard developed by the Joint Collaboration Group on Video Coding (JCT-VC). HEVC is based on a hybrid block-based motion compensation categorical DCT transform coding architecture. The basic unit of compression is called a coding unit (CU), which is a 2Nx2N pixel block. Each CU can be recursively split into four smaller CUs until a predefined minimum size is reached. Each CU contains one or more prediction units (PUs).

[0006] Versatile Video Coding (VVC) is the latest international video coding standard developed by ITU-T SG16 WP3 and ISO / IEC JTC1 / SC29 / WG11 Joint Video Experts Group (JVET). The input video signal is predicted from a reconstructed signal derived from a coded picture region. The prediction residual signal is processed by block transform. The transform coefficients are quantized and entropy coded along with other side information in the bitstream. The reconstructed signal is generated from the prediction signal and the reconstructed residual signal after inverse transforming the dequantized transform coefficients. The reconstructed signal is further processed by loop filtering to remove coding artifacts. The decoded pictures are stored in a frame buffer and used to predict future pictures in the input video signal.

[0007] In VVC, a coded picture is divided into non-overlapping square block regions represented by associated coding tree units (CTUs). The leaf nodes of the coding tree correspond to coding units (CUs). A coded picture can be represented by a set of slices, each slice containing an integer number of CTUs. Each CTU in a slice is processed in raster scan order. Bidirectionally predicted (B) slices can be decoded using intra prediction or inter prediction, with up to two motion vectors and reference indices to predict the sample values ​​of each block. Predicted (P) slices are decoded using intra prediction or inter prediction with up to one motion vector and reference index to predict the sample values ​​of each block. Intra (I) slices are decoded using only intra prediction.

[0008] A CTU can be divided into one or more non-overlapping coding units (CUs) using a quadtree (QT) with a nested multi-type tree (MTT) structure to accommodate various local motion and texture characteristics. A CU can be further split into smaller CUs using one of five partitioning types: quadtree partitioning, vertical binary tree partitioning, horizontal binary tree partitioning, vertical center-side ternary tree partitioning, and horizontal center-side ternary tree partitioning.

[0009] Each CU contains one or more prediction units (PUs). The prediction unit, together with the associated CU syntax, serves as the basic unit for sending prediction information. A specified prediction process is used to predict the values ​​of the relevant pixel samples within the PU. Each CU may contain one or more transform units (TUs) for representing prediction residual blocks. A transform unit (TU) consists of a transform block (TB) of luma samples and two corresponding chroma sample TBs, each TB corresponding to a residual block of samples from one color component. An integer transform is applied to the transform block. The values ​​of the quantized coefficients are entropy encoded in the bitstream along with other auxiliary information. The terms coding tree block (CTB), coding block (CB), prediction block (PB), and transform block (TB) are defined as 2-dimensional sample arrays that specify one color component associated with a CTU, CU, PU, ​​and TU, respectively. Therefore, a CTU consists of one luma CTB, two chroma CTBs, and related syntax elements. Similar relationships are valid for CUs, PUs, and TUs.

[0010] For each inter-predicted CU, motion parameters consisting of motion vector, reference picture index and reference picture list usage index, as well as additional information, are used for inter-prediction sample generation. Motion parameters can be signaled explicitly or implicitly. When a CU is encoded using skip mode, this CU is associated with one PU and has no significant residual coefficients, no encoded motion vector increments or reference picture indices. A merge mode is specified, whereby the motion parameters of the current CU are obtained from neighboring CUs, including spatial and temporal candidates and additional scheduling introduced in VVC. Merge mode can be applied to any inter-predicted CU. An alternative to merge mode is the explicit transmission of motion parameters, where motion vectors, corresponding reference picture indices for each reference picture list and reference picture list usage flags, as well as other required information, are explicitly signaled for each CU. Summary of the invention

[0011] The following summary is illustrative only and is not intended to be limiting in any way. That is, the following summary is provided to introduce the concepts, highlights, benefits, and advantages of the novel and non-obvious technologies described herein. Selected but not all embodiments are further described in the detailed description below. Therefore, the following summary is not intended to identify essential features of the claimed subject matter, nor is it intended to be used to determine the scope of the claimed subject matter.

[0012] Some embodiments of the present disclosure provide a method for encoding a pixel block using an out-of-bounds (OOB) check. A video codec receives data of a current block of a current picture to be encoded or decoded as a video. The video codec identifies a first reference block in a first reference picture based on a first block vector of the current block. The video codec performs an OOB check on the first reference block relative to a boundary of a subunit of the first reference picture. The subunit may be a slice, a tile, or a pipeline data unit of the first reference picture.

[0013] The video codec generates a predictor for the current block based on the first reference block and based on the OOB check. For example, when a sample of the first reference block is OOB, the sample is not used for bidirectional motion compensation of the current block. The encoder may identify a second reference block in a second reference picture based on a second vector of the current block, and may perform an OOB check on the second reference block. A predictor for bidirectional motion compensation may be generated based on the first and second reference blocks and based on the OOB check of the first and second reference blocks.

[0014] In some embodiments, the current block may be one of multiple sub-blocks of a larger block, and the OOB check of the first reference block is performed only on one representative pixel position in the first reference block. The representative pixel position may be a fractional position. The size of the sub-block may be determined based on the encoding tool used to encode or decode the current block.

[0015] For each sub-block, the encoder may identify a reference block in a reference picture, perform an OOB check on the identified reference block relative to a boundary of a sub-unit of the reference picture only at a representative pixel position and not at other pixel positions, and determine whether to perform bidirectional motion compensation for all samples of the sub-block based on the OOB check at the representative pixel position. The encoder may derive a corresponding motion vector from a plurality of motion vectors of the sub-block, and may use the corresponding motion vector to identify the reference block of the sub-block. In some embodiments, the corresponding motion vector of the sub-block is derived by calculating a minimum or maximum value of horizontal and vertical components of a plurality of motion vectors of the sub-block.

[0016] In some embodiments, when the sample of the first reference block is OOB, this sample is not used for bidirectional motion compensation of the current block. In some embodiments, the current block is one of several sub-blocks of a larger block, and when the sample at the representative pixel position is OOB, bidirectional motion compensation is not applied to the sub-block. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are included to provide a further understanding of the present disclosure and are incorporated into and constitute a part of the present disclosure. The accompanying drawings illustrate embodiments of the present disclosure and, together with the description, are used to explain the principles of the present disclosure. It is worth noting that the accompanying drawings are not necessarily drawn to scale, because in order to clearly illustrate the concepts of the present disclosure, some components may be shown as being out of proportion to the sizes in actual implementation.

[0018] Figure 1 Bi-directional prediction using out-of-bounds (OOB) reference blocks is shown.

[0019] Figure 2A Figure -B shows OOB checks performed based on different types of boundaries.

[0020] Figure 3 Performing OOB checking at a sub-block level is conceptually shown.

[0021] Figure 4 An example video encoder that may use out-of-bounds checking when performing a prediction mode is shown.

[0022] Figure 5 The portion of the video encoder that implements out-of-bounds checking for predictive coding is marked.

[0023] Figure 6 The process of using out-of-bounds checking when performing predictive coding of a pixel block is conceptually shown.

[0024] Figure 7 An example video decoder that may use out-of-bounds checking when performing prediction mode is shown.

[0025] Figure 8The portion of the video decoder that implements out-of-bounds checking for predictive coding is marked.

[0026] Fig. 9 The process of using out-of-bounds checking when performing predictive coding of a block of pixels is conceptually shown.

[0027] Fig.10 An electronic system for implementing some embodiments of the present disclosure is conceptually illustrated. DETAILED DESCRIPTION

[0028] In the detailed description below, many specific details are set forth by way of example to provide a thorough understanding of the relevant teachings. Any variation, derivation and / or extension based on the teachings described herein are within the scope of protection of the present disclosure. In some cases, well-known methods, processes, components and / or circuits related to one or more example implementations disclosed herein may be described at a relatively high level without detail to avoid unnecessarily obscuring aspects of the teachings of the present disclosure.

[0029] I. Out-of-Boundary (OOB) Check

[0030] A. OOB Check for Bidirectional Motion Compensation

[0031] An inter-predicted CU may have reference blocks that are partially or completely outside the reference picture. Figure 1 Bidirectional prediction using out-of-bounds (OOB) reference blocks is shown. In the figure, bidirectional motion compensation is performed to generate an inter-frame prediction block (predictor) of a current block 105 in a current picture 100. List 0 (L0) reference block 120 is partially beyond the boundary (OOB) of the reference block, and list 1 (L1) reference block 121 is completely located within the reference picture 111. When the video codec performs the prediction generation process 130, the OOB portion 140 of the reference block 120 is filled with repeated padding samples. The prediction generation process 130 uses the reference blocks 120 and 121 to generate a prediction block 135 for the current block 105. Since the OOB portion 140 of the reference block 120 is filled with repeated samples derived from boundary samples within the reference picture 110, the portion 135 of the motion compensation block corresponding to the OOB portion 140 can provide lower prediction efficiency.

[0032] In some embodiments, a unidirectional motion compensated sample is considered OOB when one of its reference samples is more than half a sample outside the reference picture. For each prediction sample in a bidirectional motion compensated block, when one of its unidirectional prediction samples is OOB and the other is non-OOB, the corresponding bidirectional sample is set equal to the non-OOB sample instead of using the average of the OOB and non-OOB samples.

[0033] In some embodiments, when more than one prediction block is combined, OOB prediction samples are discarded and non-OOB prediction samples are used to generate the final predictor. Specifically, let Pos_xi,j and Pos_yi,j represent the x and y positions of a prediction sample (i,j) in the current block, MVLX_xi,j and MVLX_yi,j (LX = L0 or L1) represent the MV of the current block; PosLeftBdry, PosRightBdry, PosTopBdry, PosBottomBdry are the positions of the four boundaries of the picture. A prediction sample is considered OOB when at least one of the following conditions is met:

[0034] (Pos_x i,j +MV LX _x i,j )>(Pos RightBdry +half_pixel),

[0035] (Pos_x i,j +MV LX _x i,j )<(Pos LeftBdry +half_pixel),

[0036] (Pos_y i,j +MV LX _y i,j )>(Pos BottomBdry +half_pixel),

[0037] (Pos_y i,j +MV LX _y i,j )>(Pos TopBdry +half_pixel)

[0038] Where half_pixel is equal to 8, indicating a half-pixel sampling distance at 1 / 16 pixel sampling accuracy. After checking the OOB condition of each sample, the final prediction sample of a bidirectional block is generated as follows:

[0039] If P L0 i,j is OOB and P L1 i,j is non-OOB, then P final i,j =

[0040] P L1 i,j

[0041] Otherwise if P L0 i,j Is non-OOB and PL1 i,j It is OOB, then

[0042] P final i,j =P L0 i,j

[0043] Otherwise P final i,j =(P L0 i,j +P L1 i,j +1)>>1

[0044] In some embodiments, an OOB check is performed on an intra-coded block or an IBC block. For example, the current block may be decoded by referencing a block vector of prediction samples in the current picture, and an OOB check may be performed on the referenced samples. Samples determined to be OOB by the OOB check may be replaced by padding samples, or, if the current block is encoded by multiple predictors (e.g., bi-directional prediction or multi-hypothesis), the OOB prediction samples will not be used, and collocated non-OOB samples of other predictors are used to generate the final predictor.

[0045] B. OOB check based on sub-image boundaries

[0046] In some embodiments, the boundary of the OOB check may be any one of a picture boundary, a sub-picture boundary, a tile boundary, a slice boundary, a virtual pipeline data unit (e.g., VPDU) boundary, or a virtual boundary. Figure 1 ,OOB check is performed based on picture boundaries.

[0047] Figure 2A -B shows an OOB check performed based on different types of boundaries. The OOB check is performed on the L0 and L1 reference blocks used for bidirectional motion compensation. As shown, the current block 210 in the current picture 200 is encoded using bidirectional motion compensation. The current block 210 has an L0 motion vector MV0 and an L1 motion vector MV1. MV0 points to the L0 reference block 215 in the L0 reference picture 220. MV1 points to the L1 reference block 216 in the L1 reference picture 230.

[0048] Figure 2AAn OOB check is shown that is performed on a slice boundary. The L0 reference picture 220 is divided into multiple slices (slice 0 to slice 3). The reference block 215 is located in slice 0, but a portion of the reference block 215 crosses the slice boundary into slice 1, so the OOB check fails. The video codec can exclude the OOB samples of the reference block 215 from the bidirectional motion compensation.

[0049] Figure 2B An OOB check is shown that is performed on a tile boundary. The L0 reference picture 220 is partitioned into multiple tiles (tile 0 to tile 3). Reference block 215 is located in tile 1, but a portion of reference block 215 crosses the tile boundary into tile 0, so the OOB check fails. The video codec can exclude the OOB samples of reference block 215 from bidirectional motion compensation.

[0050] C. Sub-block based OOB check

[0051] In some embodiments, the OOB check is applied at the sub-block level instead of the sample level. In some of these embodiments, the video codec uses a representative position in the sub-block and the corresponding MV (of the sub-block) to determine the OOB results for all positions in the sub-block. For example, for a 4x4 sub-block, the video codec can use the position (2,2) (assuming the position of the upper left corner is (0,0)) as the representative pixel position to determine the OOB results for all positions in the sub-block.

[0052] Figure 3 The OOB check is conceptually shown to be performed at the sub-block level. This figure shows a pixel block 310 with four 4×4 sub-blocks 321-324. Each sub-block has its own corresponding MV for referencing a reference block. In this example, the corresponding MVs of sub-blocks 321-324 point to reference sub-blocks 331-334, respectively. For each reference sub-block, the video codec performs an OOB check on a representative pixel position (displayed as a black box / dot in each reference sub-block). The OOB check result of a representative pixel position is used as the OOB result for the sub-block. The OOB is determined based on a boundary 305, which can be a picture boundary, a sub-picture boundary, a slice boundary, a tile boundary, or a virtual boundary.

[0053] In this example, for reference sub-block 331, the representative position is within boundary 305 (as is the case for the entire reference sub-block 331). Therefore, sub-block 321 is considered to be completely non-OOB, and bidirectional motion compensation can be performed for the sub-block.

[0054] For reference sub-block 332, the representative position is outside boundary 305. For the purpose of bidirectional motion compensation, the video codec may treat sub-block 322 as fully OOB (even though some portion 332 of the reference sub-block is still within boundary 305).

[0055] For reference block 333, the representative position is outside boundary 305 (as is the entire reference sub-block 333). Therefore, for the purpose of bidirectional motion compensation, sub-block 323 is considered to be completely OOB.

[0056] For reference sub-block 334, the representative position is inside boundary 305. For the purpose of bidirectional motion compensation, the video codec may treat sub-block 324 as completely non-OOB, even if some portion of reference sub-block 324 is outside boundary 305. In these cases, padding samples may be used for the portion of reference sub-block 334 that is outside boundary 305.

[0057] In some embodiments, a representative position for sub-block OOB checking may be a fractional position. For example, for a 4x4 sub-block, the position (1.5, 1.5) may be used. For some embodiments, the size of the sub-block may vary depending on the prediction mode or encoding tool used to encode the current block. For example, a 4x4 sub-block size may be used for affine mode, a 16x16 sub-block size may be used for DMVR mode, an 8x8 sub-block size may be used for normal bidirectional mode, etc.

[0058] In some embodiments, if a sub-block includes multiple MVs, the corresponding MV of the sub-block can be derived from the multiple MVs. For example, an 8x8 sub-block can have four different MVs, and the corresponding MV of the sub-block for OOB check is derived by calculating the minimum or maximum of the horizontal and vertical components of the four different MVs.

[0059] In some embodiments, more than one corresponding MV is used for OOB check. For example, in some embodiments, a first corresponding MV for the right and bottom borders and a second corresponding MV for the left and top borders are used for OOB check. In another example, an 8x8 sub-block may have 4 different MVs, and the MVs for the right and bottom borders are obtained by calculating the maximum of the horizontal and vertical components of the 4 different MVs, and the MVs for the left and top borders are obtained by calculating the minimum of the horizontal and vertical components of the 4 different MVs.

[0060] The aforementioned proposed method can be implemented in an encoder and / or a decoder. For example, the proposed method can be implemented in an inter-frame prediction module and / or an intra-frame block copy prediction module of an encoder and / or an inter-frame prediction module (and / or an intra-frame block copy prediction module) of a decoder.

[0061] II. Example Video Encoder

[0062] Figure 4 An example video encoder 400 using out-of-bounds checking when performing a prediction mode is illustrated. As shown, the video encoder 400 receives a video source 405 input video signal and encodes the signal into a bitstream 495. The video encoder 400 has several components or modules for encoding the signal from the video source 405, including at least some components selected from the following: a transform module 410, a quantization module 411, an inverse quantization module 414, an inverse transform module 415, an intra-frame estimation module 420, an intra-frame prediction module 425, a motion compensation module 430, a motion estimation module 435, a loop filter 445, a reconstructed picture buffer 450, an MV buffer 465, an MV prediction module 475, and an entropy encoder 490. The motion compensation module 430 and the motion estimation module 435 are part of the inter-frame prediction module 440.

[0063] In some embodiments, modules 410-490 are software instruction modules executed by one or more processing units (e.g., processors) of a computing device or electronic device. In some embodiments, modules 410-490 are hardware circuit modules implemented by one or more integrated circuits (ICs) of an electronic device. Although modules 410-490 are shown as separate modules, some modules can be combined into a single module.

[0064] The video source 405 provides an uncompressed raw video signal that presents pixel data for each video frame. A subtractor 408 calculates the difference between the raw video pixel data of the video source 405 and the predicted pixel data 413 from the motion compensation module 430 or the intra prediction module 425 as a prediction residual 409. The transform module 410 converts the difference (or residual pixel data or residual signal 408) into transform coefficients (e.g., by performing a discrete cosine transform or DCT). The quantization module 411 quantizes the transform coefficients into quantization data (or quantization coefficients) 412, which are encoded into a bitstream 495 by the entropy encoder 490.

[0065] The inverse quantization module 414 inversely quantizes the quantized data (or quantized coefficients) 412 to obtain transform coefficients, and the inverse transform module 415 inversely transforms the transform coefficients to generate a reconstructed residual 419. The reconstructed residual 419 is added to the predicted pixel data 413 to generate reconstructed pixel data 417. In some embodiments, the reconstructed pixel data 417 is temporarily stored in a line buffer (not shown) for intra-frame prediction and spatial MV prediction. The reconstructed pixels are filtered by the loop filter 445 and stored in the reconstructed picture buffer 450. In some embodiments, the reconstructed picture buffer 450 is a memory external to the video encoder 400. In some embodiments, the reconstructed picture buffer 450 is a memory internal to the video encoder 400.

[0066] The intra estimation module 420 performs intra prediction based on the reconstructed pixel data 417 to generate intra prediction data. The intra prediction data is provided to the entropy encoder 490 to be encoded into a bitstream 495. The intra prediction data is also used by the intra prediction module 425 to generate the predicted pixel data 413.

[0067] The motion estimation module 435 performs inter-frame prediction by generating MVs to refer to pixel data of a previously decoded frame stored in the reconstructed picture buffer 450. These MVs are provided to the motion compensation module 430 to generate predicted pixel data.

[0068] Instead of encoding the complete actual MV in the bitstream, the video encoder 400 uses MV prediction to generate a predicted MV, and the difference between the MV used for motion compensation and the predicted MV is encoded as residual motion data and stored in the bitstream 495.

[0069] The MV prediction module 475 generates a predicted MV based on a reference MV generated for a previously encoded video frame, i.e., a motion compensated MV for performing motion compensation. The MV prediction module 475 retrieves a reference MV from a previous video frame from the MV buffer 465. The video encoder 400 stores the MV generated for the current video frame in the MV buffer 465 as a reference MV for generating a predicted MV.

[0070] The MV prediction module 475 uses the reference MV to create a predicted MV. The predicted MV can be calculated by spatial MV prediction or temporal MV prediction. The entropy encoder 490 encodes the difference (residual motion data) between the predicted MV and the motion compensation MV (MC MV) of the current frame into a bitstream 495.

[0071] The entropy encoder 490 encodes various parameters and data into a bitstream 495 by using an entropy coding technique such as context adaptive binary arithmetic coding (CABAC) or Huffman coding. The entropy encoder 490 encodes various header elements, flags along with the quantized transform coefficients 412 and residual motion data as syntax elements into the bitstream 495. The bitstream 495 is in turn stored in a storage device or transmitted to a decoder over a network communication medium, such as a network.

[0072] The in-loop filter 445 performs a filtering or smoothing operation on the reconstructed pixel data 417 to reduce encoding artifacts, particularly at the boundaries of pixel blocks. In some embodiments, the filtering operation or smoothing operation performed by the in-loop filter 445 includes a sample adaptive offset (SAO), a deblocking filter (DBF), and / or an adaptive loop filter (ALF).

[0073] Figure 5The portion of the video encoder 400 that implements out-of-bounds checking for predictive coding is labeled. Specifically, the figure identifies components of the motion compensation module 430 of the video encoder 400.

[0074] As shown, the motion compensation module 430 has a candidate selector 510 that selects a block vector (including a motion vector and an intra prediction mode) from the MV buffer 465 and the intra prediction module 425. This selection is controlled by the motion estimation module 435. The selected block vector is checked by the OOB check module 530, which determines whether the reference block identified by the block vector is out of bounds. The selected block vector is also provided to the prediction generation module 520.

[0075] The OOB check module 530 performs an OOB check of the identified reference block based on boundaries of the video coding hierarchy. In some embodiments, the OOB check is based on picture boundaries of a reference picture from which the reference block is obtained. In some embodiments, the OOB check is based on boundaries of a sub-picture unit that includes (at least a portion of) the reference block, a sub-picture unit (e.g., a slice, a tile, or a pipeline data unit).

[0076] In some embodiments, the OOB check module 530 checks the out-of-bounds condition of each pixel position or each sample of the reference block. In some embodiments, the OOB check module 530 divides the current block into sub-blocks and performs an OOB check on each sub-block (e.g., 4x4 or 8x8). Specifically, for each sub-block, the OOB check module determines the corresponding motion vector and then performs an OOB check at the representative pixel position of the reference sub-block identified by the corresponding motion vector.

[0077] The prediction generation module 520 generates a predictor or prediction block for the current block as the prediction pixel data 413 by obtaining samples from the reconstructed picture buffer 450. The motion estimation module 435 selects one or more prediction coding tools (e.g., bi-directional prediction, GPM, CIIP, MHP, intra prediction, IBC, etc.) for the current block, and the prediction generation module 520 generates the predictor based on the selected prediction tool. The motion estimation module 435 also provides the prediction tool selection to the entropy encoder 490 for signaling in the bitstream 495.

[0078] The prediction generation module 520 also uses the OOB check results from the OOB check module 530 to determine whether to exclude certain reference samples identified by the selected block vector when performing blending to generate a predictor for bidirectional motion compensation. In some embodiments, sub-blocks determined to be OOB by the OOB check module 530 are excluded from blending for bidirectional motion compensation, while sub-blocks determined to be non-OOB are included in blending for bidirectional motion compensation.

[0079] Figure 6A process 600 for using out-of-bounds checking when performing predictive encoding of a pixel block is conceptually shown. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing encoder 400 perform process 600 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing encoder 400 performs process 600.

[0080] The encoder receives (at block 610) data of a current block to be encoded as a current picture of a video. The encoder identifies (at block 620) a first reference block in a first reference picture based on a first block vector of the current block. The encoder may also identify a second reference block in a second reference picture based on a second block vector of the current block.

[0081] The encoder performs (at block 630) an out-of-bounds (OOB) check on a first reference block relative to a boundary of a sub-unit of the first reference picture. The encoder may also perform an OOB check on a second reference block. The sub-unit may be a slice, a tile, or a pipeline data unit of the first reference picture.

[0082] In some embodiments, the current block may be one of multiple sub-blocks of a larger block, and the OOB check of the first reference block is performed only on one representative pixel position in the first reference block. The representative pixel position may be a fractional position. The size of the sub-block may be determined based on the encoding tool used to encode or decode the current block.

[0083] For each sub-block, the encoder may identify a reference block in a reference picture, perform an OOB check on the identified reference block relative to a boundary of a sub-unit of the reference picture only at a representative pixel position and not at other pixel positions, and determine whether to perform bidirectional motion compensation for all samples of the sub-block based on the OOB check at the representative pixel position. The encoder may derive a corresponding motion vector from a plurality of motion vectors of the sub-block, and may use the corresponding motion vector to identify the reference block of the sub-block. In some embodiments, the corresponding motion vector of the sub-block is derived by calculating a minimum or maximum value of horizontal and vertical components of a plurality of motion vectors of the sub-block.

[0084] The encoder generates (at block 640) a predictor for the current block based on the first reference block and based on the OOB check. The predictor for bidirectional motion compensation may be generated based on the first and second reference blocks and based on the OOB check of the first and second reference blocks. In some embodiments, when a sample of the first reference block is OOB, this sample is not used for bidirectional motion compensation of the current block. In some embodiments where the current block is one of several sub-blocks of a larger block, when the sample at the representative pixel position is OOB, bidirectional motion compensation is not applied to the sub-block.

[0085] The encoder (at block 650) encodes the current block using the generated predictor to generate a prediction residual.

[0086] III. Example Video Decoder

[0087] In some embodiments, the encoder may signal (or generate) one or more syntax elements in a bitstream so that the decoder may parse the one or more syntax elements from the bitstream.

[0088] Figure 7 An example video decoder 700 using out-of-bounds checking when performing prediction mode is illustrated. As shown, the video decoder 700 is an image decoding or video decoding circuit that receives a bitstream 795 and decodes the contents of the bitstream into pixel data of a video frame for display. The video decoder 700 has several components or modules for decoding the bitstream 795, including some components selected from an inverse quantization module 711, an inverse transform module 710, an intra-frame prediction module 725, a motion compensation module 730, a loop filter 745, a decoded picture buffer 750, an MV buffer 765, an MV prediction module 775, and a parser 790. The motion compensation module 730 is part of the inter-frame prediction module 740.

[0089] In some embodiments, modules 710-790 are software instruction modules executed by one or more processing units (e.g., processors) of a computing device. In some embodiments, modules 710-790 are hardware circuit modules implemented by one or more ICs of an electronic device. Although modules 710-790 are shown as separate modules, some modules may be combined into a single module.

[0090] The parser 790 (or entropy decoder) receives the bitstream 795 and performs initial parsing according to the syntax defined by the video coding or image coding standard. The parsed syntax elements include various header elements, flags, and quantization data (or quantization coefficients) 712. The parser 790 parses out the various syntax elements using entropy coding techniques such as context adaptive binary arithmetic coding (CABAC) or Huffman coding.

[0091] The inverse quantization module 711 inversely quantizes the quantized data (or quantized coefficients) 712 to obtain transform coefficients, and the inverse transform module 710 inversely transforms the transform coefficients 716 to obtain a reconstructed residual signal 719. The reconstructed residual signal 719 is added to the predicted pixel data 713 from the intra-frame prediction module 725 or the motion compensation module 730 to generate decoded pixel data 717. The decoded pixel data is filtered by the in-loop filter 745 and stored in the decoded picture buffer 750. As shown in the figure, in some embodiments, the decoded picture buffer 750 is a storage outside the video decoder 700. In some embodiments, the decoded picture buffer 750 is a storage inside the video decoder 700.

[0092] The intra prediction module 725 receives intra prediction data from the bitstream 795 and generates prediction pixel data 713 from decoded pixel data 717 stored in the decoded picture buffer 750. In some embodiments, the decoded pixel data 717 is also stored in a line buffer (not shown) for intra prediction and spatial MV prediction.

[0093] In some embodiments, the contents of decoded picture buffer 750 are used for display. Display device 755 either retrieves the contents of decoded picture buffer 750 for direct display or retrieves the contents of decoded picture buffer to a display buffer. In some embodiments, the display device receives pixel values ​​from decoded picture buffer 750 via pixel transfer.

[0094] The motion compensation module 730 generates predicted pixel data 713 from the decoded pixel data 717 stored in the decoded picture buffer 750 according to the motion compensation MV (MC MV). These motion compensation MVs are decoded by adding the residual motion data received from the bitstream 795 to the predicted MV received from the MV prediction module 775.

[0095] The MV prediction module 775 generates a predicted MV, for example, a motion compensated MV for performing motion compensation, based on a reference MV generated for decoding a previous video frame. The MV prediction module 775 retrieves a reference MV of a previous video frame from the MV buffer 765. The video decoder 700 stores the motion compensated MV generated for decoding a current video frame in the MV buffer 765 as a reference MV for generating a predicted MV.

[0096] The in-loop filter 745 performs filtering or smoothing operations on the decoded pixel data 717 to reduce coding artifacts, particularly at the boundaries of pixel blocks. In some embodiments, the filtering operations or smoothing operations performed by the in-loop filter 445 include sample adaptive offset (SAO), deblocking filter (DBF), and / or adaptive loop filter (ALF).

[0097] Figure 8 A portion of the video decoder 700 that implements out-of-bounds checking for predictive decoding is shown. Specifically, the figure illustrates components of the motion compensation module 730 of the video decoder 700.

[0098] As shown, the motion compensation module 730 has a candidate selector 810, which selects a block vector (including a motion vector and an intra prediction mode) from the MV buffer 765 and the intra prediction module 725. This selection is controlled by the entropy decoder 790. The block vector is checked by the OOB check module 830, which determines whether the reference block identified by the block vector is out of bounds. The selected block vector is also provided to the prediction generation module 820.

[0099] The OOB check module 830 performs an OOB check of the identified reference block based on boundaries of the video decoding hierarchy. In some embodiments, the OOB check is based on picture boundaries of a reference picture from which the reference block is obtained. In some embodiments, the OOB check is based on boundaries of a sub-picture unit that includes (at least a portion of) the reference block, a sub-picture unit (e.g., a slice, a tile, or a pipeline data unit).

[0100] In some embodiments, the OOB check module 830 checks the out-of-bounds condition of each pixel position or each sample of the reference block. In some embodiments, the OOB check module 830 divides the current block into sub-blocks and performs an OOB check on each sub-block (e.g., 4x4 or 8x8). Specifically, for each sub-block, the OOB check module determines the corresponding motion vector and then performs an OOB check at the representative pixel position of the reference sub-block identified by the corresponding motion vector.

[0101] The prediction generation module 820 generates a predictor or prediction block of the current block as the prediction pixel data 713 by obtaining samples from the decoded picture buffer 750. The entropy decoder 790 selects one or more prediction coding tools (e.g., bidirectional prediction, GPM, CIIP, MHP, intra prediction, IBC, etc.) for the current block, and the prediction generation module 820 generates the predictor based on the selected prediction tool. The entropy decoder 790 also signals the entropy decoder 790 in the bitstream 795.

[0102] The predictor generator module 820 also uses the OOB check results from the OOB check module 830 to determine whether to exclude certain reference samples identified by the selected block vector when performing blending to generate a predictor for bidirectional motion compensation. In some embodiments, sub-blocks determined to be OOB by the OOB check module 830 are excluded from blending for bidirectional motion compensation, while sub-blocks determined to be non-OOB are included in blending for bidirectional motion compensation.

[0103] Fig. 9Conceptually, process 900 is shown for using out-of-bounds checking when performing predictive coding of a pixel block. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing decoder 700 perform process 900 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing decoder 700 performs process 900.

[0104] The decoder receives (at block 910) data of a current block to be decoded as a current picture of a video. The decoder identifies (at block 920) a first reference block in a first reference picture based on a first block vector of the current block. The decoder may also identify a second reference block in a second reference picture based on a second block vector of the current block.

[0105] The decoder performs (at block 930) an out-of-bounds (OOB) check on a first reference block relative to a boundary of a sub-unit of the first reference picture. The decoder may also perform an OOB check on a second reference block. The sub-unit may be a slice, a tile, or a pipeline data unit of the first reference picture.

[0106] In some embodiments, when a sample of the first reference block is OOB, this sample is not used for bidirectional motion compensation of the current block. In some embodiments where the current block is one of several sub-blocks of a larger block, when the sample at the representative pixel position is OOB, bidirectional motion compensation is not applied to the sub-block.

[0107] In some embodiments, the current block may be one of multiple sub-blocks of a larger block, and the OOB check of the first reference block is performed only on one representative pixel position in the first reference block. The representative pixel position may be a fractional position. The size of the sub-block may be determined based on the encoding tool used to decode or decode the current block.

[0108] For each sub-block, the decoder may identify a reference block in a reference picture, determine a boundary of a sub-unit of the reference picture relative to the identified reference block at only a representative pixel position in the reference block and not at other pixel positions, and determine whether to perform bidirectional motion compensation for all samples of the sub-block at the representative pixel position based on an OOB check. The decoder may derive a corresponding motion vector from a plurality of motion vectors of the sub-block, and may use the corresponding motion vector to identify the reference block of the sub-block. In some embodiments, the corresponding motion vector of the sub-block is derived by calculating a minimum or maximum value of horizontal and vertical components of a plurality of motion vectors of the sub-block.

[0109] The decoder generates (at block 940) a predictor for the current block based on the first reference block and based on the OOB check. The predictor for bidirectional motion compensation may be generated based on the first and second reference blocks and based on the OOB check of the first and second reference blocks. In some embodiments, when a sample of the first reference block is OOB, this sample is not used for bidirectional motion compensation of the current block. In some embodiments where the current block is one of several sub-blocks of a larger block, when the sample at the representative pixel position is OOB, bidirectional motion compensation is not applied to the sub-block.

[0110] The decoder reconstructs the current block (at block 950) by using the generated predictor. The decoder may then provide the reconstructed current block for display as part of a reconstructed current picture.

[0111] IV. Example Electronic Systems

[0112] Many of the above features and applications are implemented as software processes that are specified as a set of instructions recorded on a computer-readable storage medium (also referred to as a computer-readable medium). When these instructions are executed by one or more computing or processing units (e.g., one or more processors, processor cores, or other processing units), they cause the processing units to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, flash drives, random access memory (RAM) chips, hard drives, erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), etc. Computer-readable media do not include carrier waves and electronic signals transmitted wirelessly or via wired connections.

[0113] In this specification, the term "software" is meant to include firmware residing in a read-only memory or an application stored in a magnetic storage device, which can be read into the memory for processing by the processor. In addition, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while retaining different software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of separate programs that implement the software inventions described herein together are within the scope of this disclosure. In some embodiments, when the software program is installed to run on one or more electronic systems, one or more specific machine implementations for executing and performing the operations of the software program are defined.

[0114] Fig.10An electronic system 1000 is conceptually illustrated to implement some embodiments of the present disclosure. The electronic system 1000 may be a computer (e.g., a desktop calculator, a personal calculator, a tablet calculator, etc.), a phone, a PDA, or any other type of electronic device. Such an electronic system includes various types of computer-readable media and interfaces for various other types of computer-readable media. The electronic system 1000 includes a bus 1005, a processing unit 1010, a graphics processing unit (GPU) 1015, a system memory 1020, a network 1025, a read-only memory 1030, a permanent storage device 1035, an input device 1040, and an output device 1045.

[0115] The bus 1005 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the electronic system 1000. For example, the bus 1005 communicatively connects the processing unit 1010 and the GPU 1015, the read-only memory 1030, the system memory 1020, and the permanent storage device 1035.

[0116] From these various memory units, the processing unit 1010 retrieves instructions to be executed and data to be processed in order to perform the processes of the present disclosure. In different embodiments, the processing unit may be a single processor or a multi-core processor. Some instructions are passed to and executed by the GPU 1015. The GPU 1015 may offload various calculations or supplement the image processing provided by the processing unit 1010.

[0117] The read-only memory (ROM) 1030 stores static data and instructions used by the processing unit 1010 and other modules of the electronic system. On the other hand, the permanent storage device 1035 is a read-write storage device. This device is a non-volatile storage unit that stores instructions and data even when the electronic system 1000 is turned off. Some embodiments of the present disclosure use a mass storage device (such as a magnetic disk or optical disk and its corresponding magnetic disk drive) as the permanent storage device 1035.

[0118] Other embodiments use removable storage devices (e.g., floppy disks, flash memory devices, etc. and their corresponding disk drives) as permanent storage devices. Like permanent storage device 1035, system memory 1020 is a read-write storage device. However, unlike storage device 1035, system memory 1020 is a volatile read-write memory, such as random access memory. System memory 1020 stores some instructions and data used by the processor at runtime. In some embodiments, processing according to the present disclosure is stored in system memory 1020, permanent storage device 1035 and / or read-only memory 1030. For example, various memory units include instructions for processing multimedia clips and some embodiments. From these different memory units, processing unit 1010 retrieves instructions to be executed and data to be processed in order to perform processing of some embodiments.

[0119] The bus 1005 is also connected to input and output devices 1040 and 1045. The input device 1040 enables a user to transmit information and select commands to the electronic system. The input device 1040 includes an alphanumeric keyboard and a pointing device (also referred to as a "cursor control device"), a camera (e.g., a webcam), a microphone or a similar device for receiving voice commands, etc. The output device 1045 displays images generated by the electronic system or outputs data in other ways. The output device 1045 includes a printer and a display device, such as a cathode ray tube (CRT) or a liquid crystal display (LCD), and a speaker or similar audio output device. Some embodiments include devices that function as both input and output devices, such as a touch screen.

[0120] Finally, if Fig.10 As shown, bus 1005 also couples electronic system 1000 to network 1025 via a network adapter (not shown). In this manner, the computer can be part of a network of computers, such as a local area network ("LAN"), a wide area network ("WAN") or an intranet, or a network of networks. Any or all components of electronic system 1500 may be used in conjunction with the present disclosure.

[0121] Some embodiments include electronic components, such as microprocessors, memory, and storage that store computer program instructions in machine-readable or computer-readable media (or alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, compact disk-read only (CD-ROM), compact disk-recordable (CD-R), compact disk-rewritable (CD-RW), digital versatile disk-read only (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini-SD card, micro SD card, etc.), magnetic and / or solid-state hard drives, read-only and recordable A computer readable medium may store a computer program executable by at least one processing unit and including a set of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as that generated by a compiler, and files including high-level code that is executed by a computer, an electronic component, or a microprocessor using an interpreter.

[0122] Although the above discussion primarily involves microprocessors or multi-core processors executing software, many of the above functions and applications are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuits themselves. In addition, some embodiments execute software stored in programmable logic devices (PLDs), ROM, or RAM devices.

[0123] As used in this specification and any claims of this application, the terms "computer", "server", "processor" and "memory" refer to electronic or other technical devices. These terms do not include people or groups of people. For the purposes of this specification, the terms display or display means display on an electronic device. As used in this specification and any claims of this application, the terms "computer readable medium", "computer readable medium" and "machine readable medium" are entirely limited to tangible physical objects that store information in a readable form. These terms do not include any wireless signals, wired download signals, and any other temporary signals.

[0124] Although the present disclosure has been described with reference to many specific details, those skilled in the art will recognize that the present disclosure may be implemented in other specific forms without departing from the spirit of the present disclosure. Figure 6 and Fig. 9 ) conceptually illustrate processes. The specific operations of these processes may not be performed in the exact order shown and described. Specific operations may not be performed in a continuous series of operations, and different specific operations may be performed in different embodiments. In addition, the process may be implemented using multiple sub-processes or as part of a larger macro process. Therefore, it will be understood by those skilled in the art that the present disclosure is not limited by the foregoing illustrative details, but is defined by the appended claims.

[0125] Additional notes

[0126] The subject matter described herein sometimes illustrates different components contained within or connected to different other components. It should be understood that the architectures depicted in this way are merely examples, and that many other architectures that implement the same functionality can actually be implemented. Conceptually, any arrangement of components that implement the same functionality is effectively "associated" to achieve the desired functionality. Therefore, any two components combined herein to achieve a particular functionality regardless of the architecture or intermediate components can be considered to be "associated" with each other to achieve the desired functionality. Similarly, any two components so associated may also be considered to be "operably connected" or "operably coupled" to each other to achieve the desired functionality, and any two components that can be so associated may also be considered to be "operably connected" or "coupled" to each other to achieve the desired functionality. Specific examples of operably coupled include, but are not limited to, physically matable and / or physically interactive components and / or wirelessly interactive and / or wirelessly interactive components and / or logically interactive and / or logically interactive components.

[0127] Furthermore, with respect to the use of substantially any plural and / or singular terms herein, those skilled in the art may translate from the plural to the singular and / or from the singular to the plural, depending on the context and / or application. For the sake of clarity, the various singular / plural arrangements may be expressly set forth herein.

[0128] Furthermore, those skilled in the art will appreciate that, in general, the terms used herein, and particularly the terms used in the appended claims, such as the bodies of the appended claims, are generally intended as "open" terms, e.g., the term "comprising" should be interpreted as "including but not limited to," and "having" should be interpreted as "at least having." Those skilled in the art will further appreciate that if a specific number of claim statements are intended to be introduced, such intent will be explicitly stated in the claim, and such intent is absent in the absence of such statement. For example, to aid understanding, the following appended claims may contain statements that use the introductory phrases "at least one" and "one or more" to introduce the claim statements. However, the use of such phrases should not be interpreted as implying that a claim introduced by the indefinite article "a" or "an" limits any particular claim that contains such introduced claim to an implementation that contains only one such statement, even when the same claim includes the introductory phrases "one or more" or "at least one" and an indefinite article such as "an," which should be interpreted as "at least one" or "one or more." The same applies to the use of definite articles to introduce claim statements. Furthermore, even if a specific number of the claims recited by the introduction is explicitly cited, a person skilled in the art will recognize that such a recitation should be interpreted as indicating at least the number of citations, e.g., a reference to "two iterations," without other modifiers, means at least two iterations, or two or more iterations. Furthermore, in those cases where the convention uses something like "at least one of A, B, and C, etc.", generally, such a construction is intended to be understood by a person skilled in the art in the sense of the convention, e.g., "a system having at least one of A, B, and C" would include, but is not limited to, systems having A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together, etc. In those cases where the convention uses something like "at least one of A, B, or C," generally, such a construction is intended to be understood by a person skilled in the art in the sense of the convention, e.g., "a system having at least one of A, B, or C" would include, but is not limited to, systems having A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together, etc. Those skilled in the art will further understand that in practice, any disjunctive words and / or phrases that appear as two or more alternative terms, whether in the specification, claims or drawings, should be understood to include the possibility of one term, one term or two terms. For example, the phrase "A or B" will be understood to include the possibility of "A" or "B" or "A and B".

[0129] It can be understood from the above that various embodiments of the present disclosure have been described herein for illustrative purposes, and various modifications may be made without departing from the scope and spirit of the present disclosure. Therefore, the various embodiments disclosed herein are not intended to be limiting, and the true scope and spirit are indicated by the attached claims.

Claims

1. A video encoding and decoding method, comprising: Receiving data of a current block of a current picture to be encoded or decoded as a video; identifying a first reference block in a first reference picture based on a first block vector of the current block; performing an out-of-bounds (OOB) check on a boundary of the first reference block relative to a subunit of the first reference picture; generating a predictor for the current block based on the first reference block and based on an OOB check; as well as The current block is encoded or decoded using the generated predictor.

2. The video encoding and decoding method according to claim 1, characterized in that: Also includes: identifying a second reference block in a second reference picture based on a second block vector of the current block; as well as performing an OOB check on the second reference block, Therein, the predictor is generated based on the first reference block and the second reference block and based on an OOB check of the first and second reference blocks.

3. The video encoding and decoding method according to claim 1, characterized in that: When samples of the first reference block are OOB, the samples are not used for bidirectional motion compensation of the current block.

4. The video encoding and decoding method according to claim 1, characterized in that: The sub-unit is one of a slice, a tile, or a pipeline data unit of the first reference picture.

5. The video encoding and decoding method according to claim 1, characterized in that: The current block is one of a plurality of sub-blocks of a larger block, wherein the OOB check of the first reference block is performed on only one representative pixel position in the first reference block.

6. The video encoding and decoding method according to claim 5, characterized in that: When the sample at the representative pixel position is OOB, bidirectional motion compensation is not applied to the sub-block.

7. The video encoding and decoding method according to claim 5, characterized in that: The representative pixel positions are fractional positions.

8. The video encoding and decoding method according to claim 5, characterized in that: Also for each sub-block of the larger block: identifying a reference block in a reference image; performing an OOB check on a boundary of the identified reference block relative to a sub-unit of a reference picture only at representative pixel positions in the reference block and not at other pixel positions; as well as Based on the OOB check of the representative pixel position, it is determined whether to perform bidirectional motion compensation on all samples of the sub-block.

9. The video encoding and decoding method according to claim 5, characterized in that: The size of the sub-block is determined based on a codec tool used to encode or decode the current block.

10. The video encoding and decoding method according to claim 5, characterized in that: A corresponding motion vector derived from the plurality of motion vectors of the sub-block is used to identify a reference block for the sub-block.

11. The video encoding and decoding method according to claim 10, characterized in that: The corresponding motion vector of the sub-block is derived by calculating the minimum or maximum of the horizontal and vertical components of the plurality of motion vectors of the sub-block.

12. An electronic device comprising: The video codec circuit is configured to perform operations including the following operations: Receiving data of a current block of a current picture to be encoded or decoded as a video; identifying a first reference block in a first reference picture based on a first block vector of the current block; performing an out-of-bounds (OOB) check on a boundary of the first reference block relative to a subunit of a first reference picture; generating a predictor for the current block based on the first reference block and based on an OOB check; as well as The current block is encoded or decoded using the generated predictor.

13. A video decoding method, comprising: Receiving data of a current block of a current picture to be decoded into a video; identifying a first reference block in a first reference picture based on a first block vector of the current block; performing an out-of-bounds (OOB) check on a boundary of the first reference block relative to a subunit of a first reference picture; generating a predictor for the current block based on the first reference block and based on an OOB check; and reconstructing the current block using the generated predictor.

Images