Method and apparatus for padding pictures in video coding

Abstract

A method and apparatus for filling out-of-boundary pixels is disclosed. According to the method, input data associated with a current block located at or near a picture boundary is received, where the input data includes prediction data and reconstruction residual data associated with the current block. An extended motion-compensated reconstructed block of the current block is generated based on the prediction data and the reconstructed residual data, wherein the extended motion-compensated reconstructed block of the current block is inter-coded and includes a padding region located outside the picture boundary and the reconstructed current block. At least one loop filter is applied to the extended motion compensated reconstructed block.

Description

[0001] Cross-references

[0002] The present invention is a non-provisional application and claims priority to U.S. Provisional Patent Application No. 63 / 369,090 filed on July 22, 2022. The entirety of the U.S. Provisional Patent Application is incorporated herein by reference. Technical Field

[0003] The present invention relates to filling pixels beyond the boundary in a video coding and decoding system ( out-o f -b oun d ary In particular, the present invention relates to an efficient method for generating padding samples during a pixel or block reconstruction phase. Background Art

[0004] Versatile Video Codec (VVC) is the latest international video codec standard (MPEG) developed by the Joint Video Experts Group (JVET) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group. The standard has been published as an ISO standard: ISO / IEC 23090-3:2021, Information technology-Coded representation of immersive media-Part 3:Versatile video coding, released in February 2021. VVC was developed by adding more codec tools to improve codec efficiency and handle various types of video sources, including 3D video signals, based on its predecessor HEVC (High Efficiency Video coding).

[0005] Figure 1AAn exemplary adaptive Inter / Intra video codec system including loop processing is described. For intra prediction, prediction data is derived from previously coded video data in the current picture (hereinafter also referred to as a picture). For inter prediction 112, motion estimation (abbreviated as ME) is performed on the encoder side and motion compensation (abbreviated as MC) is performed based on the result of ME to provide prediction data derived from other pictures and motion data. Switch 114 selects intra prediction 110 or inter prediction 112 and the selected prediction data is provided to adder 116 to form a prediction error, also referred to as a residual. The prediction error is then processed by transform (T) 118 and subsequent quantization (Q) 120. The transformed and quantized residual is then encoded by entropy encoder 122 to be included in the video bitstream corresponding to the compressed video data. The bitstream associated with the transform coefficients is then packaged with side information (e.g., motion and decoding modes associated with intra-frame prediction and inter-frame prediction) and other information (e.g., parameters associated with the loop filter applied to the underlying image area). Side information associated with intra-frame prediction 110, inter-frame prediction 112, and loop filter 130 is provided to entropy encoder 122, as shown in FIG. Figure 1A As shown. When using inter-frame prediction mode, one or more reference pictures must also be reconstructed at the encoder end. Therefore, the transformed and quantized residual is processed by inverse quantization (IQ) 124 and inverse transform (IT) 126 to recover the residual. The residual is then added back to the prediction data 136 at reconstruction (REC) 128 to reconstruct the video data. The reconstructed video data can be stored in a reference picture buffer 134 and used to predict other frames.

[0006] like Figure 1AAs shown, the input video data undergoes a series of processes in the encoding system. Due to the series of processes, the reconstructed video data from REC 128 may be subject to various impairments. Therefore, the loop filter 130 is often applied to the reconstructed video data before it is stored in the reference picture buffer 134 to improve the video quality. For example, a deblocking filter (DF), a sample adaptive offset (SAO), and an adaptive loop filter (ALF) may be used. It may be necessary to merge the loop filter information into the bitstream so that the decoder can correctly recover the required information. Therefore, the loop filter information is also provided to the entropy encoder 122 to be merged into the bitstream. Figure 1A In , a loop filter 130 is applied to the reconstructed video before the reconstructed samples are stored in a reference picture buffer 134 . Figure 1A The system in is intended to illustrate an exemplary structure of a typical video encoder. It may correspond to the High Efficiency Video Codec (HEVC) system, VP8, VP9, ​​H.264, or VVC.

[0007] like Figure 1B As shown, except for transform 118 and quantization 120, the decoder can use similar or identical functional blocks as the encoder, because the decoder only needs inverse quantization 124 and inverse transform 126. Instead of entropy encoder 122, the decoder uses entropy decoder 140 to decode the video bitstream into quantized transform coefficients and required coding information (such as ILPF information, intra-frame prediction information, and inter-frame prediction information). Intra-frame prediction 150 on the decoder side does not need to perform mode search. Instead, the decoder only needs to generate intra-frame prediction based on the intra-frame prediction information received from entropy decoder 140. In addition, for inter-frame prediction, the decoder only needs to perform motion compensation (MC 152) based on the inter-frame prediction information received from entropy decoder 140 without motion estimation.

[0008] According to VVC, similar to HEVC, the input picture is partitioned into non-overlapping square block areas called CTUs (Codec Tree Units). Each CTU can be divided into one or more smaller Codec Units (CUs). The generated CU partitions can be square or rectangular. In addition, VVC divides CTUs into Prediction Units (PUs) as units for applying prediction processing, such as inter-frame prediction, intra-frame prediction, etc.

[0009] In HEVC, reference pictures are extended by vertical padding of picture boundary samples. During VVC standardization, new methods of boundary padding were studied, which use inter-prediction based techniques or intra-prediction based techniques. In the present invention, an efficient padding technique is disclosed by filling pixels outside the boundary during the reconstruction phase. Summary of the invention

[0010] A method and apparatus for filling pixels outside a boundary are disclosed. According to the method, input data associated with a current block located at or near a picture boundary is received, wherein the input data includes prediction data and reconstruction residual data related to the current block. An extended motion compensation reconstruction block of the current block is generated based on the prediction data and the reconstruction residual data, wherein the extended motion compensation reconstruction block of the current block is inter-coded and includes a filling area located outside the picture boundary and a reconstructed current block. At least one loop filter is applied to the extended motion compensation reconstruction block.

[0011] In one embodiment, the current block corresponds to a 4x4 block at or near a picture boundary, and the extended motion compensated reconstruction block includes M padding lines beyond the picture boundary, where M is a positive integer. In another embodiment, the current block corresponds to a WxH block at or near a picture boundary, and if the current block is at or near a horizontal picture boundary, the extended motion compensated reconstruction block includes M padding lines beyond the horizontal picture boundary. If the current block is at or near a vertical picture boundary, the extended motion compensated reconstruction block includes M padding lines beyond the vertical picture boundary, where M, W, and H are positive integers. In another embodiment, the current block includes a wxh sub-block at or near a picture boundary, and the extended motion compensated reconstruction block includes an extended motion compensated reconstruction wxh sub-block, and if the wxh sub-block is located at or near a horizontal picture boundary, the extended motion compensated reconstruction wxh sub-block includes M padding lines beyond the horizontal picture boundary, or if the wxh sub-block is located at or near a vertical picture boundary, the extended motion compensated reconstruction wxh sub-block includes M padding lines beyond the vertical picture boundary, where M, w, and h are positive integers.

[0012] In one embodiment, the same interpolation filter associated with the motion compensation process is used to generate the padding area and the area inside the reconstructed current block. In one embodiment, a first interpolation filter associated with the motion compensation process used to generate the padding area has a shorter number of taps than a second interpolation filter associated with the motion compensation process used to generate the area inside the reconstructed current block.

[0013] In one embodiment, the same interpolation filter associated with the motion compensation process is used to generate all padding samples outside the picture boundaries. In one embodiment, the same interpolation filter corresponds to a predefined interpolation filter.

[0014] In one embodiment, the prediction mode associated with the motion compensation process for generating padding samples outside the picture boundary is set to a predefined value. In one embodiment, the predefined value corresponds to LIC, BDOF, BCW, filter type, multiple hypothesis or inter prediction direction.

[0015] In one embodiment, the same prediction mode associated with the motion compensation process is used to generate the padding area and the area inside the reconstructed current block. In one embodiment, the same prediction mode corresponds to LIC, BDOF, BCW, filter type or multiple hypothesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1A An exemplary adaptive inter / intra video coding system incorporating in-loop processing is shown.

[0017] Figure 1B Shows Figure 1A The corresponding decoder of the encoder in .

[0018] Figure 2 An example of a template region used to estimate the MDBP (Multi-Directional Boundary Filling) angle pattern is shown.

[0019] Figure 3 Examples of two MDBP template regions T1 and T2 based on the outermost and first pixel lines located outside the reference frame, respectively, are shown.

[0020] Figure 4 An example of border pixel filling using motion compensation according to JVET-K0117 is shown.

[0021] Figure 5 An example of a motion compensated border filling method is shown.

[0022] Figure 6 An example of exporting an M×4 padding block with left padding direction is shown.

[0023] Figure 7 An example of a filled area in a picture according to an embodiment of the present invention is shown.

[0024] Figure 8 Examples of a current picture, a reference picture, and a reference picture of the reference picture are shown.

[0025] Fig. 9 A flow chart of an exemplary video coding system for generating padding samples outside picture boundaries during a reconstruction phase according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0026] It is readily understood that, as generally described and illustrated in the accompanying drawings herein, the components of the present invention may be arranged and designed in a variety of different configurations. Therefore, the following more detailed description of the embodiments of the system and method of the present invention as shown in the figures is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention. References to "one embodiment", "an embodiment" or similar language throughout the specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present invention. Therefore, the phrases "in one embodiment" or "in one embodiment" that appear in various places throughout the specification do not necessarily all refer to the same embodiment.

[0027] In addition, the described features, structures or characteristics can be combined in any suitable manner in one or more embodiments. However, those skilled in the art will recognize that the present invention can be practiced without one or more specific details or using other methods, components, etc. In other cases, well-known structures or operations are not shown or described in detail to avoid blurring the various aspects of the present invention. The embodiments shown in the present invention will be better understood with reference to the accompanying drawings, in which the same parts are always represented by the same reference numerals. The following description is intended only by way of example, and simply describes certain selected embodiments of the apparatus and method consistent with the present invention claimed herein.

[0028] Multi-directional boundary padding (MDBP)

[0029] In JVET-J0014 (M. Albrecht et al., "Description of SDR, HDR, and 360° video coding technology proposal by Fraunhofer HHI", Joint Video Experts Team (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC1 / SC 29 / WG 11, 10th Meeting: San Diego, USA, April 10-20, 2018, document: JVET-J0014), multi-directional boundary padding (MDBP) is disclosed. Based on the codec block shape, a given motion vector and the number of interpolation filter taps, a specific area of ​​the reference frame is used for motion compensated prediction. In HEVC and JEM (Joint Exploration Model for Video Compression), if this reference sampling area is partially or completely outside the area of ​​the reconstructed reference frame, a vertical extension of the frame boundary pixels is used, which may not best approximate the predicted block. Better continuity can be achieved by extending the reference frame boundaries using different spatial prediction modes. Therefore, whenever the reference pixel area is partially or completely outside the area of ​​the reconstructed reference frame, Multi-Directional Boundary Filling (MDBP) uses angular intra prediction to extend the reference frame boundary.

[0030] In order to reduce the signaling cost of the used angular prediction mode, the best fit mode is estimated both at the encoder and decoder side. For the estimation, a template region is defined, which is located within the reconstructed reference frame, such as Figure 2 As shown. Figure 2 , a frame boundary line 210 and a reference area 220 located at the top side of the frame are shown, wherein pixels below the frame boundary line 210 are located within the frame, and pixels above the frame boundary line 210 are located outside the frame.

[0031] In addition, for each possible angular intra prediction mode, the prediction direction is rotated 180° to point to the available boundary pixels within the reference frame. The template region is then predicted from the neighboring boundary pixels and compared with the reconstructed reference frame pixels based on the SAD measurement. Finally, the angular prediction mode with the smallest template-based SAD measurement is selected to predict the reference pixel region outside the reference frame.

[0032] In order to use the available angular intra prediction with MDBP, some modifications have to be applied. First, for MDBP intra prediction, the boundary pixels are only available on a single side of the prediction region. Therefore, depending on the prediction direction, only half of the angular intra prediction modes are used, such as horizontal or vertical mode. Second, for the top and left boundaries of the reference frame, the angular intra prediction modes have to be rotated 180° before being applied to the MDBP boundary extension.

[0033] Figure 3 An example of providing a complete estimate of the entire reference pixel region 320 outside the reference frame is shown, along with two template regions (330 and 332) used in JVET-J0014. The first template region 330 is determined based on the outermost pixel line parallel to the reference frame boundary. Figure 3 As shown, a second template region 332 is determined based on a first pixel line outside a reference frame boundary, wherein a frame boundary line 310 is shown.

[0034] At the edge of the reference frame, the reference pixel region overlaps both sides of the frame border. Here MDBP is applied only on one side (the side that overlaps the frame border with the most pixels). The remaining side is filled with the already available vertical frame border fill.

[0035] Inter / Intra Boundary Padding

[0036] In HEVC, reference pictures are extended by vertical padding of samples at picture boundaries.

[0037] Inter-prediction based boundary filling uses motion compensated prediction to extend the area of ​​the reference picture. The boundary extension area is divided into 4xM or Mx4 sample blocks. Each block is filled through motion compensation using the motion information of the neighboring reference blocks. For boundary extension blocks without associated motion information and for boundary extension areas where the motion information points outside the reference picture, fall-back vertical filling is applied.

[0038] The padding method in JVET-K0363 (Yan Zhang et al., “CE4.5.2: Motion compensated boundary pixel padding”, Joint Video Experts Group (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 11th Meeting: Ljubljana, SI, July 10-18, 2018, Document: JVET-K0363) requires adding the average residual offset to the boundary extension sampling, while the padding method in JVET-K0117 (Minsoo Park et al., “CE4: Results on Reference picture boundary padding in J0025”, Joint Video Experts Group (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 11th Meeting: Ljubljana, SI, July 10-18, 2018, Document: JVET-K0117) supports bidirectional prediction for boundary extension sampling.

[0039] Such as JVET-J0012 (Rickard et al., "Description of SDR and HDR videocoding technology proposal by Ericsson and Nokia", Joint Video Experts Group (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 10th Meeting: San Diego, USA, April 10-20, 2018, Document: JVET-J0012)) The intra prediction based boundary filling proposed in (i) uses angular intra prediction to fill the area where the reference block is outside the reference picture. A detection method based on decoded image samples is used to select the applied angular intra prediction mode in the encoder and decoder.

[0040] In JVET-K0195, a coordinated boundary filling method using boundary filling based on inter-prediction and intra-prediction is disclosed, and experimental results are reported.

[0041] Inter-frame / intra-frame based border padding

[0042] JVET-K0195 proposes border filling based on inter / intra prediction, combining border filling based on per-picture inter-prediction with border filling based on per-reference intra-prediction. After generating border filling based on inter prediction, for each reference block that requires border filling samples, the number of border filling samples derived from vertical border filling is evaluated. If the number exceeds a threshold (e.g., 50% of the border filling samples), border filling based on intra prediction is used for the reference block.

[0043] In VVC, the outer area of ​​the reference image is filled by inferring the edge pixels of the image. In JVET-K0117, a method for filling the outer area of ​​the image with motion compensation based on the motion information of the edge pixels of the image is disclosed, such as Figure 4 As shown. Figure 4 In FIG. 4 , a boundary block 412 in the current frame 410 is shown and details of the padding 430 around the boundary block are shown. The corresponding boundary block 422 in the reference picture 420 is shown in FIG. Figure 4 , and the details of the corresponding boundary block 440 are shown in Figure 4 In the detail of the fill 430 around the boundary block, a boundary line 434 is shown. The pixels to the left of the boundary line of the reference area 432 are unavailable and need to be filled. The corresponding reference area 442 is located and used to derive the reference area 432, as shown in FIG. Figure 4 As indicated by the arrow in .

[0044] To use the motion information of edge pixels, each 4x4 block is checked at the border of the picture. If there is motion information in the block, the position of the block is checked in the reference picture of the block. If the position is within the image area, the neighboring area of ​​the reference area is checked to see if it is available.

[0045] The position of the neighboring area can be located in four directions: up, down, left, and right. The direction of the neighboring area is the same as the position of the filling area. For example, if the filling area is located on the left side of the picture, the inspection area is also on the left side of the pixel. The "inspection area" here means that the reference sample is located around the reference block. For example, if the left picture boundary filling is to be performed, the reference sample on the left side of the reference block is checked. In addition, the length of the side of the filling area that does not face the picture is determined by the distance between the pixel position of the reference picture and the position of the edge pixel or the size of the filling area. Select the shorter one. If the predetermined length is shorter than the size of the filling area, the rest of the area is filled with the extrapolated edge pixels of the external picture.

[0046] The available neighboring regions are derived through motion compensation. However, when the neighboring regions are not available or there is no information about the motion in the border blocks, the traditional filling method is performed. The block can have two pieces of information about the motion. In this case, each piece of information is used to create a filling image and the two images are integrated into one. In addition, the last pixel of each position is extrapolated to produce the top left, top right, bottom left, and right filling regions.

[0047] Motion compensated border pixel filling

[0048] In JVET-K0363, boundary pixel filling for motion compensation is disclosed. When performing motion compensation on the decoder side, it is possible that the motion vector points to a reference block that is partially or completely outside the reference slice. Without boundary filling, these pixels will not be available. Traditionally, reference slices are filled using a repeated filling method that repeats the outermost pixels a certain number of times in each of the four directions depending on the filling size. These filling pixels can only provide very limited information because the filled area is likely to contain no meaningful content compared to the pixels inside the boundary.

[0049] In JVET-K0363, a new boundary pixel filling method is introduced so that the filled area in the reference slice can provide more information. First, the motion vector is derived from the boundary 4x4 block in the current frame, such as Figure 5 As shown, the left side shows padding (510) and the right side shows MC padding (520) according to JVET-K0363. If the boundary 4x4 block is intra-coded or the motion vector is not available, repeated padding will be used. If the boundary 4x4 block is predicted using unidirectional inter prediction, the unique motion vector within the block will be used for motion compensated boundary pixel padding. Using the position of the boundary 4x4 block and its motion vector, the corresponding starting position can be calculated in the reference frame. From this starting position to the boundary of the reference slice in a given padding direction, 4xM or Mx4 image data can be obtained, where M is the distance between the horizontal / vertical coordinates of the boundary pixel position and the starting position, depending on the padding direction. Here in the CE test, M is forced to be less than 64. In the case of bidirectional inter prediction, only the motion vector, which points to the direction of the pixel position in the reference slice that is farther from the frame boundary in terms of padding, is used for motion compensated boundary pixel padding. The difference between the DC values ​​of the boundary 4x4 block in the current slice and its corresponding reference 4x4 block in the reference slice is used as an offset to filter the captured motion compensated image data before copying it to the padding area outside the image boundary.

[0050] In ECM-5.0, bidirectional prediction is performed in a way that avoids relying on reference samples outside reference picture boundaries (OOB) if possible.

[0051] To this end, in case of a bidirectionally predicted block with an OOB reference block in one of the two reference pictures, the OOB prediction samples are not used. If available in the other reference picture, the relevant part of the block is unidirectionally predicted based on non-OOB prediction samples.

[0052] However, for unidirectionally predicted blocks with OOB reference blocks or bidirectionally predicted blocks with OOB reference samples for reference pictures in both directions, repeated padding pixels are used instead of MC.

[0053] That is, in ECM-5.0, the picture is extended by an area of ​​size (maxCUwidth+16) around the picture in each direction of the picture boundary. The pixels in the extended area are obtained by repeated boundary padding. When the reference block used for unidirectional prediction is partially or completely outside the picture boundary (OOB), repeated padding pixels are used instead of motion compensation (MC).

[0054] In JVET-Z0130 (Zhi Zhang et al., “EE2-related: Motion compensation boundary padding”, Joint Video Experts Group (JVET) / WG 11 of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29, 26th meeting, via teleconference, April 20-29, 2022, document: JVET-Z0130), a method called motion compensated boundary padding replaces repeated boundary padding to improve coding efficiency.

[0055] In JVET-AA0096 (Fabricé Le Léannec et al., "EE2-2.2: Motion compensated picture boundary padding", Joint Video Experts Group (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 27th Meeting, by teleconference, July 13-22, 2022, document: JVET-AA0096), samples outside the picture boundary are derived by motion compensation instead of using only repetition padding as in ECM. In practice, the total padding region size is increased by 64 (Test 2.2a) or 16 (Test 2.2b) compared to ECM (Enhanced Compression Model). This is to keep MV cropping (implementation of repetition padding) and is not normative.

[0056] For motion compensated padding, the MV of the 4×4 boundary block is used to derive the M×4 or 4×M padding block. The value M is derived as the distance from the reference block to the picture boundary, as Figure 66, where an MC padding region 630 is added to the current picture 610 and a reference picture 620 is shown. For a 4x4 boundary block 612, the corresponding reference block 622 is located according to the motion vector 616. An Mx4 padding block 614 of the current picture and an Mx4 padding block 624 of the reference picture are shown. In addition, once the motion vector points to a location inside the reference picture boundary, M is set to at least 4. If the boundary block is intra-coded, MV is not available and M is set to 0. If M is less than 64, the remainder of the padding region is filled with repeated padding samples.

[0057] In the case of bidirectional inter prediction, only one prediction direction is used in MC boundary filling, whose motion vector points to pixel positions in the reference picture that are far away from the picture boundary in the filling direction.

[0058] The pixels in the MC padding block are corrected by an offset equal to the difference between the DC values ​​of the reconstructed boundary block and its corresponding reference block.

[0059] In order to further improve the encoding and decoding performance, a new filling method is proposed for picture boundary filling. Unlike the repeated filling in HEVC and VVC, filling and repeated filling based on intra-frame prediction, inter-frame prediction or a combination of the two are allowed in the picture boundary filling according to the present invention. For the filling method based on intra-frame prediction, the traditional intra-frame prediction method can be used to generate boundary filling samples, or the implicit method of the encoder and decoder side derivation method or other signaling method can also be performed. The filling method based on intra-frame prediction is applied before loop filtering (for example, in the CU reconstruction stage). For the filling method based on inter-frame prediction, during encoding and decoding, motion compensation is not performed after loop filtering, but a larger motion compensation block including filling samples is generated. Further operations can also be called during motion compensation. In addition, the reference picture of the reference picture of the current picture can also be used during the generation of filling samples.

[0060] In one embodiment, the padding samples are derived based on a certain intra prediction mode (e.g., planar mode). To generate such padding samples, both sides of the reference sample may be needed, but one side may not be used. Reference sample padding may also be applied to the reference sample, and intra prediction is performed to derive the padding samples.

[0061] In one embodiment, if the current block in the picture is an intra-frame mode codec block, the same intra-frame mode (e.g., the same intra-frame angle mode) can be used to generate the filling result of the samples outside the picture boundary. The reference sample for the intra-frame prediction of the block outside the boundary can be the reconstructed current sample or the reference sample for the intra-frame prediction of the current block. In one example, chroma intra-frame prediction can also be applied in a similar manner to the luminance block.

[0062] In one embodiment, if the current block uses intra-template matching prediction (Intra-TMP), intra-block copy (IBC), or intra-block copy with template matching (IBC-TM), intra-block copy can also be applied to the out-of-boundary block (OOB block). For example, the block vector (BV) of the current block is used to generate a predictor as a filler sample for the OOB block.

[0063] In another embodiment, a template-based intra mode derivation (TIMD) is performed to derive padding samples. Instead of using two template regions determined by some external pixel lines in JVET-J0014, one template region is used and the SAD between the predicted padding samples and the template region is calculated. A hybrid process of the predicted padding samples can also be performed.

[0064] In another embodiment, decoder-side intra mode derivation (DIMD) is performed to derive padding samples. To generate padding samples, histogram data is first calculated based on the current reconstructed samples using a Sobel filter. A prediction mode index is determined based on the histogram data, and the reconstructed samples are used to generate padding samples for the final prediction based on the selected prediction mode index. In one example, the boundary samples of the current block are used to derive the intra prediction mode through DIMD. The reconstructed samples of the current block can be used as reference samples to generate padding samples for the OOB block.

[0065] In another embodiment, between the filling samples and the reconstruction samples, a position dependent intra-prediction combination (PDPC) can be applied to resolve discontinuities. The PDPC process can be applied as in VVC, or applied differently with fewer or more rows, weaker weights or stronger weights at the filling samples.

[0066] In another embodiment, during encoding and decoding, for those blocks at the picture boundary, larger motion compensation blocks are generated, such as (M+4)x4 blocks or 4x(M+4) blocks, where M is the length of the padding samples. In another example, the padding samples can be generated with the entire CU as a (M+H)xW block or Wx(M+H) block, where M is the length of the padding samples, and W and H are the block width and height. In another example, the padding samples can be generated with sub-blocks of the current block, such as (M+h)xw blocks or wx(M+h) blocks, where M is the length of the padding samples, and w and h are the width and height. The sub-block size can be pre-defined, or can be different values ​​for different modes.

[0067] In one embodiment, when performing motion compensation, a check is performed to see if the current block / current sub-block is in a picture boundary. If so, additional reference samples are loaded (e.g., reference samples of (M+h)xw blocks or wx(M+h) blocks). OOB block samples are generated during the sampling phase of current block / current sub-block reconstruction.

[0068] In another embodiment, during motion compensation of padding samples, the same interpolation filter is used for both padding samples and blocks within a picture. Another embodiment is that the interpolation filter is the same for all padding samples outside a picture (eg, a predefined filter is used for padding samples).

[0069] In one embodiment, during motion compensation, the prediction mode for the OOB block is set to a predefined value. The prediction mode can be LIC, BDOF, BCW, filter type, multiple hypothesis, inter prediction direction, etc.

[0070] In another embodiment, during motion compensation, the prediction mode used for blocks inside the picture is also applied to OOB blocks. The prediction mode can be LIC, BDOF, BCW, filter type, multiple hypothesis, etc.

[0071] In another embodiment, during motion compensation, if local illumination compensation (LIC) is applied to blocks inside the picture, local illumination compensation is also applied to the padding samples. In another embodiment, if bidirectional optical flow (BDOF) is applied to blocks inside the picture, BDOF is also applied to the padding samples.

[0072] In one embodiment, shorter filter tap interpolation may be used for OOB sample MCs. For example, integer MCs, or 2-tap, or 4-tap, or 6-tap, or 8-tap filters are used for OOB sample MCs. The MVs of OOB block MCs may also be rounded to a coarser granularity.

[0073] In another embodiment, OOB samples can only be generated by using the same reference samples (for MC processing or for decoder side mode / MV derivation) of the current block / current sub-block within the picture boundary. Adding additional reference samples is not allowed.

[0074] In another embodiment, OOB samples can only be generated by using the same reference samples (for MC processing or for decoder side mode / MV derivation) plus a small predefined or adaptive number of samples of the current block / current sub-block that are within the picture boundary.

[0075] In another embodiment, after the padding samples are generated, further offset or compensation is applied to the padding samples. One method is to calculate the difference between the entire border block of the picture and the entire generated padding samples to derive the offset. Another method is to calculate the difference between the border block on one side of the border and the padding samples generated on the other side of the border to derive the offset.

[0076] In another embodiment, for Figure 7 The corner pixels (A, B, C and D) shown in the figure can generate padding samples according to different methods. For example, padding samples at A, B, C and D can be generated according to the upper left corner sample of the picture, the upper right corner sample of the picture, the lower left corner sample of the picture and the lower right corner sample of the picture. Another example is to generate Figure 7 After the filling samples in , the filling samples at A, B, C, and D are generated according to the weighted sum of the corresponding neighborhood filling samples (i.e., the two rectangular gray filling sampling areas 710). Another example is that when generating Figure 7 After the padding samples in , the padding samples in A, B, C, and D are generated directly from the adjacent padding samples (for example, area A is generated from the padding samples adjacent to its right).

[0077] In another embodiment, after generating the padding samples, a further padding operation is applied to make the padded frame rectangular in size, such as Figure 7 As shown in region E in FIG. Further filling operations can generate filling samples in region E according to different methods. An example is that the filling samples in region E are generated directly from the image boundary, or from Figure 7 The padding samples in are generated.

[0078] In another embodiment, when using boundary blocks for motion compensation, the reference block on the reference picture may be partially located outside the picture, such as Figure 8As shown. At this time, if the reference block in the reference picture is inter-coded, the reference block in the reference of the reference picture can be used. In one example, only the part outside the picture uses the reference block in the reference of the reference picture. In another example, when the reference block in the reference picture exceeds the picture boundary, the reference block in the reference of the reference picture is used to generate padding samples.

[0079] In another embodiment, if Figure 8 As shown, there may be two MVs (MV0 and MV1) in three pictures, where picture 810 corresponds to the current picture, picture 820 corresponds to the reference picture, and picture 830 corresponds to the reference picture of the reference picture. Block 812 corresponds to a boundary block in the current picture. The motion vector MV0 associated with block 812 points to a reference block 822 in the reference picture 820 (a portion of the reference block is outside the reference picture). The motion vector MV1 associated with reference block 822 points to another reference block 832 in the reference picture 830 of the reference picture 820. During the generation of padding samples, another reference block may be considered. In one example, we add two MVs (e.g., MV0 and MV1) to obtain another reference block in another reference picture or a reference picture of a reference picture. In another example, we average the two MVs to obtain another reference block in the reference picture or a reference picture of a reference picture.

[0080] Any of the aforementioned proposed sampling padding methods for out-of-bounds pixels may be implemented in the encoder and / or decoder. For example, any of the proposed sampling padding methods may be implemented in the predictor derivation module (e.g. Figure 1A inter-frame prediction 112 and / or intra-frame prediction 110) and reconstruction phase (e.g. Figure 1A 128 in the REC 128), and / or a predictor derivation module of a decoder (e.g., Figure 1B MC 152 and / or intra prediction 150) and reconstruction phase (e.g., Figure 1A Alternatively, any of the proposed methods may be implemented as a circuit coupled to the predictor derivation module and reconstruction stage of the encoder and / or the predictor derivation module and reconstruction stage of the decoder to provide the information required by the predictor derivation module. The filling method may also be implemented using executable software or firmware program code stored on a medium (e.g., a hard disk or flash memory) for use in a CPU (central processing unit) or a programmable device (e.g., a DSP (digital signal processor) or an FPGA (field programmable gate array)).

[0081] Fig. 9A flow chart of an exemplary video codec system for generating padding samples outside a picture boundary during a reconstruction phase according to an embodiment of the present invention is shown. The steps shown in the flow chart can be practiced as executable program codes on one or more processors (e.g., one or more CPUs) on the encoder side. The steps shown in the flow chart can also be implemented based on hardware, such as one or more electronic devices or processors arranged to perform the steps in the flow chart. According to the method, input data associated with a current block located at or near a picture boundary is received in step 910, wherein the input data includes prediction data and reconstruction residual data related to the current block. In step 920, an extended motion compensation reconstruction block of the current block is generated based on the prediction data and the reconstruction residual data, wherein the extended motion compensation reconstruction block of the current block is inter-coded and includes a padding area outside the picture boundary and a reconstructed current block. In step 930, at least one loop filter is applied to the extended motion compensation reconstruction block.

[0082] The flowchart shown is intended to illustrate an example of video coding according to the present invention. Those skilled in the art may modify each step, rearrange the steps, split the steps, or combine the steps to implement the present invention without departing from the spirit of the present invention. In the present disclosure, specific syntax and semantics have been used to illustrate examples of implementing embodiments of the present invention. A skilled person may practice the present invention by replacing syntax and semantics with equivalent syntax and semantics without departing from the spirit of the present invention.

[0083] The above description is given to enable those skilled in the art to practice the present invention provided in the context of specific applications and their requirements. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. Therefore, the present invention is not intended to be limited to the specific embodiments shown and described, but should be given the widest scope consistent with the principles and novel features disclosed herein. In the above detailed description, various specific details are shown in order to provide a thorough understanding of the present invention. However, it will be appreciated by those skilled in the art that the present invention can be put into practice.

[0084] The embodiments of the present invention as described above can be implemented with various hardware, software codes or a combination of the two. For example, the embodiments of the present invention can be one or more circuits integrated into a video compression chip or a program code integrated into a video compression software to perform the processing described herein. The embodiments of the present invention can also be a program code executed on a digital signal processor (DSP) to perform the processing described herein. The present invention can also be related to a variety of functions performed by a computer processor, a digital signal processor, a microprocessor or a field programmable gate array (FPGA). These processors can be configured to perform specific tasks according to the present invention by executing machine-readable software codes or firmware codes that define the specific methods embodied in the present invention. The software code or firmware code can be developed in different programming languages ​​and different formats or styles. The software code can also be compiled for different target platforms. However, different code formats, styles and languages ​​of software codes and other ways of configuring codes to perform tasks according to the present invention will not depart from the spirit and scope of the present invention.

[0085] The present invention can be implemented in other specific forms without departing from its spirit or essential characteristics. The described examples should be considered as illustrative and non-restrictive in all aspects. Therefore, the scope of the present invention is indicated by the appended claims rather than the aforementioned description. All changes within the meaning and scope of the equivalents falling within the claims are included within its scope.

Claims

1. A video encoding and decoding method, the method comprising: Receiving input data associated with a current block located at or near a picture boundary, wherein the input data includes prediction data and reconstructed residual data related to the current block; generating an extended motion compensated reconstructed block of the current block based on the prediction data and the reconstructed residual data, wherein the extended motion compensated reconstructed block of the current block is inter-coded and includes a padding area outside a boundary of the picture and the reconstructed current block; After said generating the extended motion compensated reconstructed block, at least one loop filter is applied to generate a filtered reconstructed block.

2. The method according to claim 1, characterized in that The current block corresponds to a 4×4 block at or near the picture boundary, and the extended motion compensated reconstructed block includes M padding lines beyond the picture boundary, where M is a positive integer.

3. The method according to claim 1, characterized in that The current block corresponds to a WxH block at or near the picture boundary, and if the current block is at or near a horizontal picture boundary, the extended motion compensated reconstructed block includes M padding lines beyond the horizontal picture boundary, and if the current block is at or near a vertical picture boundary, the extended motion compensated reconstructed block includes M padding lines beyond the vertical picture boundary, where M, W, and H are positive integers.

4. The method according to claim 1, characterized in that: The current block includes a wxh sub-block at or near the picture boundary, and the extended motion compensated reconstructed block includes an extended motion compensated reconstructed wxh sub-block, and wherein if the wxh sub-block is located at or near a horizontal picture boundary, the extended motion compensated reconstructed wxh sub-block includes M padding lines exceeding the horizontal picture boundary, or if the wxh sub-block is located at or near a vertical picture boundary, the extended motion compensated reconstructed wxh sub-block includes M padding lines exceeding the vertical picture boundary, wherein M, w, and h are positive integers.

5. The method according to claim 1, characterized in that The same interpolation filters associated with the motion compensation process are used to generate the padding area and the area inside the reconstructed current block.

6. The method according to claim 1, characterized in that A first interpolation filter associated with a motion compensation process used to generate the padding region has a shorter number of taps than a second interpolation filter associated with the motion compensation process used to generate the region within the reconstructed current block.

7. The method according to claim 1, characterized in that The same interpolation filters associated with the motion compensation process are used to generate all padding samples outside the picture boundaries.

8. The method according to claim 7, characterized in that The same interpolation filter corresponds to a predefined interpolation filter.

9. The method according to claim 1, characterized in that: The prediction mode associated with the motion compensation process for generating padding samples outside the picture boundary is set to a predefined value.

10. The method according to claim 9, characterized in that The predefined value corresponds to LIC, BDOF, BCW, filter type, multi-hypothesis or inter prediction direction.

11. The method according to claim 1, characterized in that: The same prediction mode associated with the motion compensation process is used to generate the padding area and the area inside the reconstructed current block.

12. The method according to claim 11, characterized in that The same prediction mode corresponds to LIC, BDOF, BCW, filter type or multiple hypothesis.

13. An apparatus for video encoding and decoding, the apparatus comprising one or more electronic devices or processors, arranged to: Receiving input data associated with a current block located at or near a picture boundary, wherein the input data includes prediction data and reconstructed residual data related to the current block; generating an extended motion compensated reconstructed block of the current block based on the prediction data and the reconstructed residual data, wherein the extended motion compensated reconstructed block of the current block is inter-coded and includes a padding area outside a boundary of the picture and the reconstructed current block; After generating the extended motion compensated reconstructed block, at least one loop filter is applied to generate a filtered reconstructed block.

Images