Method and apparatus for shared preloading area for affine prediction or motion compensation

Abstract

A video encoding or decoding method and apparatus includes receiving input data associated with a current block in a current picture, determining two or more coding configurations for affine prediction or motion compensation or a preloading region shared by two or more affine refinement iterations in a reference picture, loading reference samples in the preloading region, generating a predictor for the current block, and encoding or decoding the current block according to the predictor. The predictor associated with the affine refinement iteration or the coding configuration is generated based on some of the reference samples in the preloading region.

Description

Technical Field

[0001] This invention relates to inter-frame coding methods and apparatus for encoding or decoding video data. In particular, this invention relates to affine prediction and motion compensation in video coding systems. Background Technology

[0002] The Universal Video Coding (VVC) standard is a new video coding standard developed by video coding experts from the ITU-T Study Group in conjunction with the Joint Video Coding Collaboration Group (JCT-VC). In the VVC draft, an image is divided into one or more tile rows and one or more tile columns. A tile is a sequence of Coded Tree Units (CTUs) covering a rectangular region within the image, with specific tile columns and rows. Tile features are primarily used for parallel processing purposes, and sub-images formed by rectangular slices of tile segmentation can also be used for viewport-based 360-degree video applications. The processing order for image tile segmentation is in-image tile raster scanning followed by in-tile CTU raster scanning. Tiles break in-image prediction dependencies and entropy decoding dependencies. A slice consists of an integer number of consecutive complete CTU rows or an integer number of complete tiles within a single tile of the image. Slice features are primarily used for sub-image-level access and ultra-low latency. Each sub-image in an image contains one or more slices that collectively cover a rectangular region of the image, and each sub-image contains multiple complete CTUs. Sub-images within an image allow for independent encoding, decoding, and extraction of rectangular subsets of encoded image sequences, useful for viewport-related 360-degree video stream optimization and region of interest applications. Sub-images allow motion vectors of encoded blocks to point outside the sub-image, and even when the sub-image is extractable, sub-image boundaries can be padded as frame boundaries.

[0003] In the development of the VVC standard, block-based affine transformation motion compensation prediction has become an alternative inter-frame predictive coding tool to traditional translational motion compensation prediction. The affine motion field of a block is described by the motion information of two control points in a 4-parameter affine motion model or three control points in a 6-parameter affine motion model. For the 4-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived by the following equation.

[0004]

[0005] For a 6-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived by the following equation.

[0006]

[0007] Where (mv0x, mv0y) is the motion vector of the top left control point, (mv1x, mv1y) is the motion vector of the top right control point, and (mv2x, mv2y) is the bottom motion vector of the bottom control point.

[0008] The luminance coding block (CB) of the coding unit (CU) is divided into 4x4 sub-blocks. Figure 1 An example of the affine motion vector field for each 4x4 sub-block in a CU using affine motion compensation predictive coding is illustrated. To derive the motion vector for each 4x4 luma sub-block in the affine codec CU, the motion vector of the center sample of each sub-block is calculated according to the affine model shown in Equation (1) or (2) and rounded to 1 / 16 fractional precision. A motion compensation interpolation filter is applied to generate the prediction signal for each sub-block using the derived sub-block motion vectors. The sub-block size for the chroma component is also set to 4x4 samples. The MV of the 4x4 chroma sub-block is calculated as the average of the MVs of the four corresponding 4x4 luma sub-blocks.

[0009] There are two affine motion inter-frame prediction modes: affine merge mode and affine advanced motion vector prediction (AMVP) mode. Affine merge mode can be applied to control units (CUs) with a width and height greater than or equal to 8 samples. In this mode, the control point MV of the current CU is generated based on the motion information of spatially neighboring CUs. There can be up to five Control Point Motion Vector Predictor (CPMVP) candidates, and an index is issued to indicate which one should be used for the current CU. Three types of CPMVP candidates are used to form the affine merge candidate list: inherited affine merge candidates extrapolated from the control point MVs of neighboring CUs, constructed affine merge candidate CPMVPs derived using the translation MVs of neighboring CUs, and zero MV.

[0010] There are at most two inherited affine candidates, derived from the affine motion model of neighboring blocks: one from the left-side neighboring CU and one from the top-side neighboring CU. The scan order of the candidate positions for the left predictor is A0 212 then A1211, as follows. Figure 2 As shown. The scanning order of the candidate positions for the above predictors is B0 214, B1 213, and then B2 215, as follows. Figure 2 As shown. Only the first inherited candidate from each side is selected. Due to redundancy, two inherited candidates are not examined. If adjacent affine CUs are identified, their control point motion vectors are used to derive CPMVP candidates from the affine merging list of the current CU. Figure 3This explains the inheritance of control point motion vectors, such as Figure 3 As shown, when the adjacent lower left block A is encoded and decoded using an affine model, the motion vectors v2, v3, and v4 of the upper left, upper right, and lower left corners of the CU containing block A are affine. When the CU containing block A is encoded and decoded using a 4-parameter affine model, the two control points MV of the CU are calculated based on the motion vectors v2 and v3. When the CU containing block A is encoded and decoded using a 6-parameter affine model, the three control points MV of the CU are calculated based on the motion vectors v2, v3, and v4.

[0011] The constructed affine merging candidate is built by combining the neighboring translational motion information of each control point. The motion information of the control points comes from specified spatial and temporal neighbors, such as... Figure 4 As shown. CPMVk (k = 1, 2, 3, 4) represents the k-th control point. For CPMV1, the check order is B2, B3, then A2, using the MV of the first available block. For CPMV2, the check order is B1 then B0, and for CPMV3, the check order is A1 then A0. If available, the time MVP will be used as CPMV4.

[0012] After obtaining the MVs of the four control points, affine merge candidates are constructed based on motion information. These are constructed sequentially using the following combinations of control point MVs: {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}. A combination of three CPMVs constructs a 6-parameter affine merge candidate, and a combination of two CPMVs constructs a 4-parameter affine merge candidate. To avoid motion scaling, combinations of control point MVs are discarded if the reference indices of the control points are different. After checking the inherited and constructed affine merge candidates, if the list is still not full, a zero MV is inserted at the end of the list.

[0013] The affine AMVP mode can be applied to control units (CUs) with a width and height greater than or equal to 16 samples. An affine flag at the CU level is signaled in the video bitstream to indicate whether the affine AMVP mode is used, and another flag is signaled to indicate whether a 4-parameter or 6-parameter affine model is used. In affine AMVP mode, the difference between the control point MV of the current CU and its predicted value CPMVP is represented by a signal in the video bitstream. The affine AMVP candidate list is 2 in size and is generated sequentially using four types of control point MV candidates. The first type of control point MV candidate is an inherited affine AMVP candidate extrapolated from the control point MVs of neighboring CUs. The second type is an affine AMVP candidate CPMVP derived using the translation MVs of neighboring CUs. The third type of candidate control point MV is a translation MV from neighboring CUs, and the fourth is a zero MV. The checking order of inherited affine AMVP candidates is the same as that of inherited affine merge candidates. The only difference is that when generating inherited affine AMVP candidates, only affine CUs with the same reference picture as the current block are considered. No pruning process is applied when inserting inherited affine motion predictors into the candidate list. For example... Figure 4 As shown, the constructed AMVP candidates are derived from specified spatial neighbors. The same checking order is used to derive the constructed affine AMVP candidates and the affine merge candidates. Additionally, the reference picture indices of adjacent blocks are checked. The first block in the checking order is inter-frame encoded and has the same reference picture as the current CU. When the current CU is encoded using a 4-parameter affine model and both CPMVs are available, there is only one constructed AMVP candidate, and these two CPMVs are added as one candidate to the affine AMVP candidate list. When the current CU is encoded using a 6-parameter affine model and all three CPMVs are available, all three CPMVs are added as one candidate to the affine AMVP candidate list. Otherwise, the constructed AMVP candidate is set to unavailable. If the number of candidates in the affine AMVP candidate list is less than two after checking the inherited and constructed AMVP candidates, one or more available CPMVs are added as translation MVs to predict all control point MVs of the current CU. Finally, if the affine AMVP candidate list is still not full, zero MVs are used to fill the list.

[0014] Sub-block-based temporal motion vector prediction (SbTMVP) extends the sub-block concept of affine motion compensation prediction to CU-level temporal motion vector prediction in merged mode. SbTMVP allows motion information to be inherited from collocated reference images at the sub-block level. Each sub-block of a CU can have its own motion information without explicitly transferring block partitioning structure or motion information, thus achieving divergent motion compensation within the CU without any additional cost. Sub-block motion information derivation in SbTMVP includes deriving the displacement vector (DV) for the current CU, checking the availability of SbTMVP candidates, and deriving the central motion. Sub-block motion information is derived from the corresponding sub-block identified by the DV. The derived sub-block-level motion information is used for motion compensation of the CU's luma and chroma blocks (CBs). SbTMVP applies the DV to find the correspondence between a position or block in the current image and a position or block in the collocated image. If the left neighboring CU uses the collocated image as its reference image, the MV of the left neighboring CU is selected as the DV. If the adjacent CU on the left is not encoded in inter-frame prediction mode or the motion information does not point to the co-location image, the DV is set to (0, 0). The DV is then applied to the center position of the current CU to locate the displacement center position of the current CU, thereby locating the displacement center position in the co-location image. If the block containing the displacement center position has not undergone inter-frame encoding / decoding, the SbTMVP candidate is considered unavailable. Otherwise, the motion information for the center position of the current CU is obtained from the motion information of the block containing the offset center position in the co-location image. A maximum of two motion vectors can be derived, one for each list. When an SbTMVP candidate is available, the DV is used to find the corresponding sub-block in the co-location image for each sub-block of the current CU. The motion information of the sub-blocks in the current CU is derived from the motion information of the corresponding sub-block in the same way as the center motion is derived. If the corresponding sub-block of the current sub-block has not undergone inter-frame encoding / decoding, the motion information of the current sub-block is set to the center motion. SbTMVP is suitable for CUs with a width and height greater than or equal to 8 luminance samples. The sub-block size of SbTMVP is set to 8x8 to limit memory bandwidth consumption to the worst-case scenario in 8x8 bidirectional prediction. After applying DV, the position of the corresponding sub-block is clipped to the restricted area. Summary of the Invention

[0015] An embodiment of a video encoding or decoding method for a video encoding / decoding system receives input data associated with a current block in a current image, determines a preloaded region in a reference image for two or more encoding / decoding configurations for affine prediction or motion compensation, or for two or more affine refinement iterations, loads reference samples in the preloaded region into a memory buffer, generates a predictor for the current block, and encodes or decodes the current block based on the predictor. The predictor associated with two or more affine refinement iterations or encoding / decoding configurations is generated based on a corresponding portion of the reference samples in the preloaded region. According to some embodiments of the invention, the predictor associated with an affine refinement iteration or encoding / decoding configuration is generated based on a portion of the reference samples in the preloaded region, without loading other reference samples in the reference image. In various embodiments of the invention, the encoding / decoding configurations for affine prediction or motion compensation include combinations of affine one-way prediction, affine two-way prediction, 4-parameter affine prediction, 6-parameter affine prediction, inter-frame prediction, merged prediction, and SbTMVP. For example, two or more affine refinement iterations are associated with affine one-way prediction or affine two-way prediction performed on a reference image. In one embodiment, the preloaded region is determined by unidirectional prediction and reused by bidirectional prediction. In another embodiment, the preloaded region is determined by 4-parameter affine prediction or 4-parameter affine combined motion compensation and reused by 6-parameter affine prediction or 6-parameter affine combined motion compensation. In yet another embodiment, the preloaded region is determined by the encoding / decoding configuration of affine prediction or motion compensation and reused by SbTMVP. If any of the 4x4 sub-block motion vectors (MVs) of the SbTMVP points to a reference sample outside the preloaded region, the current block skips the SbTMVP or uses padding.

[0016] Some embodiments of the video encoding or decoding method skip search candidates when the required data for affine prediction or motion compensation is not within the preloaded region. Other embodiments of the video encoding or decoding method modify search candidates when the required data for affine prediction or motion compensation is not within the preloaded region. Search candidates are modified by modifying one or more affine control point motion vectors (CPMVs) of the search candidates. In one embodiment, the affine CPMV is modified by constraining the distance between reference regions of two adjacent affine 4x4 sub-blocks. For example, the distance between the reference regions of two adjacent affine 4x4 sub-blocks is restricted to less than one sample. In another embodiment, the affine CPMV is modified by constraining the overlapping region of two adjacent affine 4x4 sub-blocks. For example, each dimension of the overlapping region of two adjacent affine 4x4 sub-blocks is restricted to less than one column or one row. In yet another embodiment, the affine CPMV is modified by constraining the difference in center motion vectors (MVs) between two adjacent affine 4x4 sub-blocks within a defined range. Some embodiments that modify one or more affine CPMVs modify the affine model to ensure that the farthest CPMV is within the preloaded region. Modify one or more embodiments of the affine CPMV to clip one or more violating CPMVs within the preloaded region. Modify one or more embodiments of the affine CPMV to switch the affine mode to a translation mode using the affine center MV as the translation MV to derive a new reference region. Modify one or more embodiments of the affine CPMV to switch the affine mode to a translation mode using the affine center MV as the translation MV to derive a new reference region and move the new reference region to align with the boundary of the preloaded region.

[0017] According to some embodiments of the invention, the current block is a coding block (CB) or a coding unit (CU). In some embodiments, the size of the current block is the width multiplied by the height, and the size of the preloaded area is obtained by multiplying the width by 1.4 times and the height by 1.4 times.

[0018] Several aspects of this disclosure also provide an apparatus for a video encoding / decoding system. Some embodiments of this apparatus receive input video data associated with a current block in a current picture, determine a preloaded region in a reference picture for two or more encoding / decoding configurations for affine prediction or motion compensation, or for two or more affine refinement iterations, load reference samples from the preloaded region into a memory buffer, generate a predictor for the current block, and encode or decode the current block based on the predictor. The predictor associated with the two or more affine refinement iterations or encoding / decoding configurations is generated based on a corresponding portion of the reference samples in the preloaded region.

[0019] The following overview is illustrative only and is not intended to be limiting in any way. That is, it is provided to introduce the concepts, emphasis, benefits, and advantages of the novel and non-obvious techniques described herein. Choices, but not all, of implementations are further described in the detailed description below. Therefore, the following overview is not intended to identify the essential characteristics of the claimed subject matter, nor is it intended to determine the scope of the claimed subject matter. Attached Figure Description

[0020] Various embodiments of this disclosure will be described in detail by way of example with reference to the following accompanying drawings, wherein the same numerals denote the same elements, and wherein:

[0021] Figure 1 The illustration shows an example of the affine motion vector field of a block in a CU that is predicted by affine motion compensation using a 4-parameter affine model.

[0022] Figure 2 The diagram illustrates the positions of spatial and temporal candidates used to construct a merge candidate list for the regular merge pattern according to the VVC standard.

[0023] Figure 3 The illustration shows an example of control point motion vector inheritance used to construct an affine merge candidate list for the current block.

[0024] Figure 4 The spatial and temporal adjacent positions of the constructed affine candidates are shown by combining the adjacent translational motion information of each control point.

[0025] Figure 5 The diagram illustrates an irregular loading pattern associated with the loading of a reference sample for a 4x4 sub-block.

[0026] Figure 6 An embodiment of the invention is shown, which reuses a preloaded region with SbTMVP for affine prediction.

[0027] Figure 7A and Figure 7B The constraints applied to reduce the bandwidth of internal memory accesses according to some embodiments of the present invention are shown.

[0028] Figures 8A-8D Four examples of modifying CPMV when the required reference data exceeds the preloaded area are shown.

[0029] Figure 9 This is a flowchart illustrating the use of a shared preloaded region to encode or decode the current block for affine prediction or motion compensation, according to an embodiment of the present invention.

[0030] Figure 10An exemplary system block diagram of a video encoding system incorporating a video encoding method according to an embodiment of the present invention is shown.

[0031] Figure 11 An exemplary system block diagram of a video decoding system incorporating a video decoding method according to an embodiment of the present invention is shown. Detailed Implementation

[0032] It will be readily understood that, as generally described and illustrated in the accompanying drawings, the components of the present invention can be arranged and designed in a variety of different configurations. Therefore, the following more detailed description of embodiments of the systems and methods of the present invention illustrated in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention.

[0033] Throughout this specification, references to "one embodiment," "some embodiments," or similar language mean that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. Therefore, the phrases "in one embodiment" or "in some embodiments" appearing throughout this specification do not necessarily refer to the same embodiment, which may be implemented alone or in combination with one or more other embodiments. Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. However, those skilled in the art will recognize that the invention can be practiced without one or more specific details, or using other methods, components, etc. In other instances, well-known structures or operations are not shown or illustrated. Detailed description is provided to avoid obscuring aspects of the invention.

[0034] Affine CPMV refinement, based on the VVC standard, applies multiple CPMV refinement iterations to determine the final CPMV in affine bidirectional or unidirectional prediction for each reference image. The number of CPMV refinement iterations in affine bidirectional prediction for each reference image is 3, while the number of iterations in affine unidirectional prediction for each reference image is 5 if a 4-parameter affine motion model is used, and 4 if a 6-parameter affine motion model is used. The number of CPMV refinement iterations represents the number of motion compensations required during the CPMV refinement process. In each CPMV refinement iteration, the preloaded reference region (also called the preloaded region) used for motion compensation changes according to the updated CPMV, resulting in additional preload cycles and increased timing overhead. Different binding boxes of the preloaded region are used for unidirectional and bidirectional prediction. The reference region required to perform motion compensation for each 4x4 sub-block may partially overlap with the reference regions of other 4x4 sub-blocks, leading to irregular loading patterns of reference samples and wasted bandwidth, such as... Figure 5 As shown. Figure 5 This example illustrates the overlapping search area of ​​four 4x4 sub-blocks.

[0035] Shared Preloaded Regions for Affine CPMV Refinement Iterations or Codec Configurations Embodiments of the present invention determine and load preloaded regions for the required data used in affine prediction or motion compensation, and these preloaded regions can be reused in at least one other CPMV refinement iteration or another codec configuration. In one embodiment, the same preloaded region is used for unidirectional and bidirectional prediction in the same reference image. For example, a preloaded region used for unidirectional prediction is reused for bidirectional prediction. In another embodiment, the same preloaded region is used for 4-parameter and 6-parameter affine prediction and affine merging motion compensation. For example, a preloaded region used for 4-parameter affine prediction and affine merging motion compensation is reused for 6-parameter affine prediction and affine merging motion compensation. Embodiments that share preloaded regions for multiple CPMV refinement iterations or multiple codec configurations prevent bandwidth overhead caused by reloading reference samples in each CPMV refinement iteration or each codec configuration. In some embodiments, a single preloaded region in a reference image is shared for multiple codec configurations for affine prediction or motion compensation. Reference samples from a preloaded region in the reference image are loaded into a memory buffer, and portions of these reference samples are used by the codec configuration. In some embodiments, while portions of the reference samples in the preloaded region are reused by the corresponding codec configuration, other reference samples in the reference image can still be loaded and used by other codec configurations. In some embodiments, the preloaded region is reused by the codec configuration without loading other reference samples in the reference image. Different codec configurations include affine one-way prediction, bidirectional prediction, 4-parameter affine prediction, 6-parameter affine prediction, inter-frame prediction, merged prediction, and SbTMVP. In some embodiments that reuse the preloaded region, if the required data for affine prediction or motion compensation is not within the preloaded region, the search candidate is skipped or modified to be within the preloaded region. For example, the search candidate is modified by changing one or more affine CPMVs, thus the new search range is within the preloaded region. Some embodiments of modifying the affine CPMV include the reference range distance or center MV difference between two 4x4 sub-blocks within the constraint definition range, which will be described in the later "CPMV Adjustment Constraints" section, and some other embodiments of modifying the affine CPMV will be described in the later "Modifying the Affine CPMV to be within the Preloaded Region" section.

[0036] According to an embodiment of the invention, the affine preload region is set to twice the area of ​​the reference sample required for the translation mode. That is, the size of the affine preload region is 1.4 x (PU_width + 7) x 1.4 x (PU_height + 7), where PU_width and PU_height are the width and height of the current prediction unit (PU) under affine prediction. A 9x9 sample mode is loaded from the preload region each cycle for affine motion compensation. In one embodiment, the shared reference preload region is determined by the 4-parameter unidirectional prediction CPMV of the affine merging mode, and the shared reference preload region is used by unidirectional and bidirectional predictions of 4-parameter and 6-parameter affine modes. Reference samples in the preload region are preloaded into a memory buffer, such as P / D delivery static random access memory (SRAM), before the affine prediction is processed by the affine 4-parameter unidirectional prediction processing element (PE). In some embodiments of the invention, the size of the shared preload region for each reference image remains equal to 1.4 x (PU_width + 7) x 1.4 x (PU_height + 7).

[0037] To further reduce the number of preloading cycles in the SbTMVP during the motion compensation phase, according to some embodiments of the invention, the SbTMVP shares the same reference preloading region as the affine prediction. In one embodiment, if any of the 4x4 sub-blocks MV points to a reference sample outside the shared preloading region, the current PU skips the SbTMVP or uses padding. Figure 6 An example of determining the affine preload region and reusing the affine preload region for SbTMVP is shown. Figure 6 In this context, the affine preload region 64 is determined by the affine center MV of the current PU 620 in the current CTU 62. This affine preload region 64 is reused for SbTMVP execution because the reference samples in the affine preload region 64 are obtained based on each SbTMVP 4x4 sub-block MV of the current PU 620, where 640, 642, 644, and 646 are sub-block reference regions.

[0038] In some embodiments of the invention, a CPMV adjustment constraint is set to reduce the bandwidth of internal memory accesses. The CPMV adjustment constraint limits the distance between reference regions associated with two adjacent affine 4x4 sub-blocks to less than one sample, such as... Figure 7A As shown. That is, any adjacent affine 4x4 sub-blocks satisfy the following equation, where in this embodiment, w is the width of the 4x4 sub-block and h is the height of the 4x4 sub-block.

[0039]

[0040] In some other embodiments of the CPMV adjustment constraint, the overlapping region corresponding to the reference regions of two adjacent affine 4x4 sub-blocks is constrained. For example, the overlapping region associated with two adjacent affine 4x4 sub-blocks is less than one column or one row, such as... Figure 7B As shown. This CPMV adjustment constraint avoids complex selection signals in overlapping reference regions. That is, any adjacent affine 4x4 sub-blocks satisfy the following equation, where in this embodiment, w is the width of the 4x4 sub-block and h is the height of the 4x4 sub-block.

[0041]

[0042] In some embodiments of the CPMV adjustment constraint, according to an embodiment of the invention, the center MV difference between two adjacent affine 4x4 sub-blocks is constrained within a defined range [a, b].

[0043] In some embodiments where the affine CPMV is modified to reuse the preloaded region in CPMV refinement iterations within the preloaded region or to reuse the preloaded region for multiple codec configurations, one or a combination of modifications can be applied if the required reference data extends beyond the preloaded region. In one embodiment, the affine model is modified to ensure that the furthest CPMV remains within the preloaded region when using 4-parameter affine prediction; or when using 6-parameter affine prediction, violating CPMVs are clipped into the preloaded region. Figure 8A An example is shown where the affine model is modified according to this embodiment to ensure that the farthest CPMV is within the preloaded region. Figure 8B An example is shown where, according to this embodiment, the violated CPMV is directly modified to be within the preloaded region. Figure 8A In this process, the original reference region 802 is transformed into a new reference region 804 by modifying the 4-parameter affine model. Therefore, the new reference region 804 is entirely within the preloaded region 800, and its rectangular shape is preserved. Figure 8B In this embodiment, the violating CPMV is clipped into the preloaded region 810, changing the original reference region 812 into a new reference region 814. In another embodiment, the violating CPMV is clipped within the preloaded region, for example, using a translation mode when the affine reference region is flipped or extends beyond the preloaded region, such as... Figure 8C As shown. In Figure 8C In the process, when the original affine reference region 822 is flipped or moved away from the preloaded region 820, the affine mode first switches to the translation mode, and then the affine center MV 826 is adjusted to move the new reference region 824 to align with the boundary of the preloaded region 820. Figure 8DAnother embodiment of CPMV adjustment is shown. In this embodiment, when the affine reference region 832 is flipped or moves away from the preload region 830, the affine mode directly switches to the translation mode, and a new reference region 834 is derived by using the affine center MV 836 as the translation MV.

[0044] Representative flowcharts of embodiments of the present invention Figure 9 This is a flowchart illustrating an embodiment of a video encoding or decoding method using a shared preloaded region in multiple affine refinement iterations or for multiple codec configurations. In step S902, the video codec system receives input data for the current block in the current picture, and in step S904, determines a preloaded region in a reference picture for multiple affine refinement iterations or multiple codec configurations. In step S906, reference samples in the preloaded region are loaded into a memory buffer. In step S908, the video codec system generates a predictor for the current block. The predictor for the current block associated with the affine refinement iteration or codec configuration is generated based on some reference samples in the preloaded region, without loading other reference samples in the reference picture. In step S910, the current block is encoded or decoded using one of the predictors.

[0045] Exemplary video encoders and video decoders implementing the present invention can be implemented in the video encoder and / or video decoder. For example, the disclosed video encoding or decoding methods can be implemented in the inter-frame prediction module of the video encoder or the inter-frame prediction module of the video decoder. Alternatively, any disclosed method can be implemented as circuitry coupled to the inter-frame prediction module of the video encoder or the inter-frame prediction module of the video decoder to provide the information required by any module.

[0046] Figure 10An exemplary system block diagram of a video encoder 1000 for implementing one or more embodiments of the present invention is illustrated. The video encoder 1000 receives input video data of the current block in the current frame. An intra-frame prediction module 1010 provides intra-frame predictors based on reconstructed video data of the current frame. An inter-frame prediction module 1012 performs motion estimation (ME) and motion compensation (MC) to provide inter-frame predictors based on reference video data from other frames. In some embodiments of the invention, a preloaded region in each reference frame is determined for two or more affine refinement iterations or for two or more codec configurations for affine prediction or motion compensation. Some examples of codec configurations are affine one-way prediction, affine two-way prediction, 4-parameter affine prediction, 6-parameter affine prediction, inter-frame prediction, merge prediction, and SbTMVP. Reference samples in the preloaded region are loaded into a memory buffer of the inter-frame prediction module 1012. According to some embodiments of the invention, when the required data for affine prediction or motion compensation is not in the preloaded region, search candidates are skipped or modified. For example, one or more affine CPMVs of the search candidates are modified such that the new reference range is within the preloaded region. In some embodiments, the predictor of the current block in the affine refinement iteration is generated based on some reference samples in the preloaded region without loading other reference samples in the reference image into the memory buffer. In some other embodiments, the predictor of the current block of the codec configuration is generated based on some reference samples in the preloaded region without loading other reference samples in the reference image into the memory buffer. The intra-frame prediction module 1010 or the inter-frame prediction module 1012 uses switch 1014 to provide the selected predictor of the current block in the current image to adder 1016 to form a residual by subtracting the selected predictor from the original video data of the current block. For example, the current block is a coding unit (CU). The residual of the current block is further processed by transform module (T) 1018 and subsequent quantization module (Q) 1020. The transformed and quantized residual signals are then encoded by entropy encoder 1030 to form a video bitstream. The transform and quantization residual signals of the current block are processed by the inverse quantization module (IQ) 1022 and the inverse transform module (IT) 1024 to recover the prediction residuals. As shown in Figure 10, the residuals are recovered by adding back the selected predictors at the reconstruction module (REC) 1026 to produce reconstructed video data. The reconstructed video data can be stored in the reference picture buffer (Ref.Pict.Buffer) 1032 and used for prediction of other pictures. Due to the encoding process, the reconstructed video data from REC 1026 may be subject to various degradations; therefore, at least one loop processing filter (ILPF) 1028 is conditionally applied to the reconstructed video data before it is stored in the reference picture buffer 1032 to further improve image quality. A deblocking filter is an example of ILPF 1028.Syntax elements are provided to the entropy encoder 1030 to be incorporated into the video bitstream.

[0047] Figure 10 The video encoder 1000 corresponds to the video decoder 1100, such as... Figure 11 As shown. The video bitstream encoded by the video encoder is the input to the video decoder 1100, which is decoded by the video decoder (also called the entropy decoder) 1110 to parse and recover system information such as the transform-quantized residual signal of the current image. The decoding process of the decoder 1100 is similar to the reconstruction loop at the encoder 1000, except that the entropy decoder 1100 only needs motion compensation prediction in the inter-frame prediction module 1114. The entropy decoder 1110 receives the current image, and each block of the current image in the decoder 1100 is decoded by either the intra-frame prediction module 1112 or the inter-frame prediction module 1114. The switch 1116 selects the intra-frame predictor from the intra-frame prediction module 1112 or the inter-frame predictor from the inter-frame prediction module 1114 according to the decoded mode information. In some embodiments, a preloaded region in the reference image is determined and reference samples of the preloaded region are loaded into a memory buffer for use in multiple affine refinement iterations or for multiple codec configurations. According to some embodiments, one or more affine CPMVs are modified when the required data for affine prediction or motion compensation is not in the preloaded region. The transformed and quantized residual signal is recovered by the inverse quantization module (IQ) 1120 and the inverse transform module (IT) 1122. The IQ module 1120 is also called the dequantization module. The recovered residual signal is reconstructed by adding back the predictors in the reconstruction (REC) module 1118 to produce the reconstructed video. At least one in-loop processing filter (ILPF) 1124 further conditionally processes the reconstructed video to generate the final decoded video. A deblocking filter is an example of the ILPF 1124. If the currently decoded picture is a reference picture, the reconstructed video of the currently decoded picture is also stored in the reference picture buffer (Ref.Pict.Buffer) 1126 for later picture decoding commands.

[0048] Figure 10 and Figure 11Various components of the video encoder 1000 and video decoder 1100 can be implemented by hardware components, one or more processors configured to execute program instructions stored in memory, or a combination of hardware and processors. For example, a processor executes program instructions to control the reuse of preloaded regions for multiple affine refinement iterations or encoding / decoding configurations. The processor is equipped with one or more processing cores. In some examples, the processor executes program instructions to perform functions in some components of the encoder 1000 and decoder 1100, and memory electrically coupled to the processor is used to store program instructions, information corresponding to the reconstructed image of the blocks, and / or intermediate data during the encoding or decoding process. In some examples, the video encoder 1000 can signal information by including one or more syntax elements in the video bitstream, and the corresponding video decoder 1100 derives such information by parsing and decoding one or more syntax elements. In some embodiments, the memory includes a non-transitory computer-readable medium, such as semiconductor or solid-state memory, random access memory (RAM), read-only memory (ROM), hard disk, optical disk, or other suitable storage media. The memory can also be a combination of two or more of the non-transitory computer-readable media listed above, as illustrated in the figure. Referring to Figures 10 and 11, the encoder 1000 and decoder 1100 can be implemented in the same electronic device. Therefore, if implemented in the same electronic device, various functional components of the encoder 1000 and decoder 1100 can be shared or reused. For example, Figure 10 One or more of the reconstruction module 1026, inverse transform module 1024, inverse quantization module 1022, loop processing filter 1028, and reference image buffer 1032 can also be used as reconstruction module 1118, respectively in Figure 11 The inverse transform module 1122, inverse quantization module 1120, loop processing filter 1124, and reference image buffer 1126 are included.

[0049] Embodiments of more efficient video processing methods can be implemented in circuitry integrated into a video compression chip or in program code integrated into video compression software to perform the aforementioned processing. For example, encoding or decoding one or more blocks in the current frame via inter-frame coding can be implemented in program code that executes on a computer processor, digital signal processor (DSP), microprocessor, or field-programmable gate array (FPGA). These processors can be configured to perform specific tasks according to the invention by executing machine-readable software code or firmware code that defines the specific methods embodied in the invention.

[0050] The invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples are to be considered illustrative rather than limiting in all respects. Therefore, the scope of the invention is indicated by the appended claims rather than by the foregoing description. All variations within the equivalent meaning and scope of the claims should be included within their scope.

Claims

1. A method for encoding or decoding video data in a video encoding system, comprising: Receive input data associated with the current block in the current image; For two or more codec configurations for affine prediction and / or motion compensation, or for two or more affine refinement iterations, determine a shared preloaded region in a reference image; Load the reference samples in the preloaded region into the memory buffer; Generate predictors for the current block, where predictors associated with two or more affine refinement iterations or encoding / decoding configurations are generated based on corresponding portions of reference samples in the preloaded region; as well as Encode or decode the current block based on the predictor; In this context, when a reference sample in the preloaded region, which is loaded into the memory buffer and used by one of the two or more affine refinement iterations or one of the two or more codec configurations, is reused by another of the two or more affine refinement iterations or another of the two or more codec configurations, the already loaded reference sample is not reloaded.

2. The method for encoding or decoding video data according to claim 1, characterized in that, Predictors associated with two or more affine refinement iterations or encoding / decoding configurations are generated based on corresponding portions of reference samples in a preloaded region, without loading other reference samples in the reference image.

3. The method for encoding or decoding video data according to claim 2, characterized in that, The two or more encoding / decoding configurations for affine prediction or motion compensation include combinations of affine one-way prediction, affine two-way prediction, 4-parameter affine prediction, 6-parameter affine prediction, inter-frame prediction, merge prediction, and sub-block-based temporal motion vector prediction (SbTMVP).

4. The method for encoding or decoding video data according to claim 3, characterized in that, The two or more affine refinement iterations are associated with affine one-way prediction or affine two-way prediction performed on the reference image.

5. The method for encoding or decoding video data according to claim 3, characterized in that, The preloaded region is determined by unidirectional prediction and reused by bidirectional prediction.

6. The method for encoding or decoding video data according to claim 3, characterized in that, The preloaded region is determined by 4-parameter affine prediction or 4-parameter affine combined motion compensation and reused by 6-parameter affine prediction or 6-parameter affine combined motion compensation.

7. The method for encoding or decoding video data according to claim 3, characterized in that, The preloaded region is determined by the encoding / decoding configuration of affine prediction or motion compensation and is reused by SbTMVP.

8. The method for encoding or decoding video data according to claim 7, characterized in that, If any of the motion vectors (MVs) of the 4×4 sub-blocks of SbTMVP point to a reference sample outside the preloaded region, the current block skips SbTMVP or uses padding.

9. The method for encoding or decoding video data according to claim 1, characterized in that, Also includes: If the required data for affine prediction or motion compensation is not within the preloaded area, skip the search candidates.

10. The method for encoding or decoding video data according to claim 1, characterized in that, Also includes: When the required data for affine prediction or motion compensation is not within the preloaded region, modify the search candidates.

11. The method for encoding or decoding video data according to claim 10, characterized in that, Modifying the search candidate includes modifying one or more affine control point motion vectors (CPMVs) of the search candidate.

12. The method for encoding or decoding video data according to claim 11, characterized in that, The one or more affine CPMVs are modified by constraining the distance between the reference regions of two adjacent affine 4×4 sub-blocks.

13. The method for encoding or decoding video data according to claim 12, characterized in that, The distance between the reference regions of two adjacent affine 4×4 sub-blocks is limited to less than one sample.

14. The method for encoding or decoding video data according to claim 13, characterized in that, The one or more affine CPMVs are modified by constraining the overlapping regions of two adjacent affine 4×4 sub-blocks.

15. The method for encoding or decoding video data according to claim 14, characterized in that, Each dimension of the overlapping region of the two adjacent affine 4×4 sub-blocks is restricted to less than 1 column or 1 row.

16. The method for encoding or decoding video data according to claim 11, characterized in that, The one or more affine CPMVs are modified by constraining the difference in center motion vectors (MVs) between two adjacent affine 4×4 sub-blocks within a defined range.

17. The method for encoding or decoding video data according to claim 11, characterized in that, Modify one or more affine CPMVs by modifying the affine model to ensure that the furthest CPMV is within the preloaded region.

18. The method for encoding or decoding video data according to claim 11, characterized in that, The one or more affine CPMVs are modified by clipping one or more violating CPMVs into the preloaded area.

19. The method for encoding or decoding video data according to claim 11, characterized in that, The one or more affine CPMVs are modified to derive a new reference region by switching the affine mode to a translation mode that uses the affine center MV as the translation MV.

20. The method for encoding or decoding video data according to claim 11, characterized in that, The one or more affine CPMVs are modified by switching the affine mode to a translation mode that uses the affine center MV as the translation MV to derive a new reference region and move the new reference region to align with the boundary of the preloaded region.

21. The method for encoding or decoding video data according to claim 1, characterized in that, The current block size is width multiplied by height, and the size of the preloaded area is obtained by multiplying the width by 7 by 1.4 times and the height by 7 by 1.4 times.

22. An apparatus for performing video encoding or decoding, comprising: The computer processor is configured to receive video data; And by executing a computer processor-executable program for video encoding and decoding, including the following steps: Receive input data associated with the current block in the current image; For two or more codec configurations for affine prediction and / or motion compensation, or for two or more affine refinement iterations, determine a shared preloaded region in a reference image; Load the reference samples in the preloaded region into the memory buffer; Generate predictors for the current block, where predictors associated with two or more affine refinement iterations or encoding / decoding configurations are generated based on corresponding portions of reference samples in the preloaded region; as well as Encode or decode the current block based on the predictor; In this context, when a reference sample in the preloaded region, which is loaded into the memory buffer and used by one of the two or more affine refinement iterations or one of the two or more codec configurations, is reused by another of the two or more affine refinement iterations or another of the two or more codec configurations, the already loaded reference sample is not reloaded.

Images