Intra mode derivation for inter prediction coding units
By improving the MPM list construction method and deriving more accurate intra-frame modes using the inter-frame coding modes of adjacent blocks, and combining CIIP, GPM and TIMD technologies, the problem of increased mode notification overhead in intra-frame mode prediction is solved, thereby improving the coding efficiency and accuracy of video coding.
Patent Information
- Application Number
- CN202380032737.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-07
- Filing Date
- 2023-03-22
- Publication Date
- 2024-11-15
AI Technical Summary
Existing video coding techniques suffer from increased mode notification overhead and reduced gain in intra-frame mode prediction, especially when intra-frame coding blocks are surrounded by inter-frame coding blocks, resulting in inaccurate MPM list filling and low coding efficiency.
By improving the MPM list construction method, more accurate intra-frame modes are derived by utilizing the inter-frame coding modes of adjacent blocks. By combining CIIP, GPM, and TIMD technologies, the selection and padding of intra-frame prediction modes are optimized, signaling overhead is reduced, and coding efficiency is improved.
It improves the coding efficiency of video encoding, reduces mode notification overhead, and enhances the accuracy of intra-frame modes and the compression performance of coding blocks.
Smart Images

Figure CN118975233A_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims the benefit of EPO application serial number 22305483.4, filed on April 7, 2022, which is incorporated herein by reference in its entirety. Technical Field
[0003] At least one of the embodiments generally relates to a method or apparatus for video encoding or decoding, compression or decompression. Background Art
[0004] In order to achieve high compression efficiency, image and video coding schemes usually use prediction including motion vector prediction and transform to exploit spatial and temporal redundancy in video content. Typically, intra-frame or inter-frame prediction is used to exploit intra-frame or inter-frame correlation, and then the difference between the original image and the predicted image (usually expressed as prediction error or prediction residual) is transformed, quantized and entropy encoded. In order to reconstruct the video, the compressed data is decoded by the inverse process corresponding to entropy coding, quantization, transformation and prediction. Summary of the invention
[0005] At least one of the present embodiments generally relates to a method or apparatus for video encoding or decoding, and more particularly to a method or apparatus for improving coding efficiency derived from decoder-side intra-modes of surrounding reference pixels.
[0006] According to a first aspect, a method is provided. The method comprises the steps of: deriving an intra coding mode from one or more inter-coded neighboring blocks; conditionally adding the derived intra coding mode to a most probable mode list; and encoding at least a portion of a video block using the most probable mode list.
[0007] According to a second aspect, another method is provided. The method comprises the steps of: deriving an intra-coding mode from one or more inter-coded neighboring blocks; conditionally adding the derived intra-coding mode to a most probable mode list; and decoding at least a portion of a video block using the most probable mode list.
[0008] According to another aspect, a device is provided. The device includes a processor. The processor can be configured to encode a video block or decode data by performing any of the aforementioned methods.
[0009] According to another general aspect of at least one embodiment, a device is provided, comprising: an apparatus according to any embodiment of the decoding embodiment; and at least one of the following: (i) an antenna configured to receive a signal, the signal including a video block; (ii) a bandlimiter configured to limit the received signal to a frequency band including the video block; or (iii) a display configured to display an output representing the video block.
[0010] According to another general aspect of at least one embodiment, there is provided a non-transitory computer-readable medium containing data content generated according to any of the described encoding embodiments or variations.
[0011] According to another general aspect of at least one embodiment, there is provided a signal comprising video data generated according to any of the described encoding embodiments or variants.
[0012] According to another general aspect of at least one embodiment, a bitstream is formatted to include data content generated according to any of the described encoding embodiments or variations.
[0013] According to another general aspect of at least one embodiment, there is provided a computer program product comprising instructions which, when executed by a computer, cause the computer to perform any of the described decoding embodiments or variants.
[0014] These and other aspects, features and advantages of the general aspects will become apparent from the following detailed description of exemplary embodiments, which is to be read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 67 intra prediction modes in VVC and ECM are illustrated.
[0016] Figure 2 Examples of available IPM candidates are illustrated in (a)-(c), and an example of a GPM with inter- and intra-prediction is illustrated in (d).
[0017] Figure 3 The division method for the angle mode is illustrated.
[0018] Figure 4 An example of the source of the MPM intra mode from neighboring blocks is illustrated.
[0019] Figure 5 One embodiment of a method for performing the described aspects is illustrated.
[0020] Figure 6 An example of another embodiment of a method for performing the described aspects is illustrated.
[0021] Figure 7 An example of one embodiment of an apparatus for implementing the aspects is illustrated.
[0022] Figure 8 An example of a general video encoding or compression system is illustrated.
[0023] Fig. 9 An example of a general video decoding or decompression system is illustrated.
[0024] Fig.10 An example of a processor-based system for implementing the described aspects is illustrated. DETAILED DESCRIPTION
[0025] Intra prediction is a basic coding tool in video compression. The encoder selects the best prediction mode and signals its index to the decoder to perform the same prediction. Signaling the mode may add additional overhead and reduce the gain from the intra part. Therefore, a smart way to encode the mode is to select a set of most likely modes and thus reduce the signaling overhead if the mode is within the list. This is a classical method for signaling intra prediction modes, which is called MPM list based signaling, which is adopted in VVC and HEVC. This method is extended in ECM, where 2 MPM lists are used instead of one, where the first list contains 6 MPMs and the second list contains another 16 MPMs.
[0026] In ECM, two additional intra prediction modes are introduced. The first is called decoder-side intra mode derivation (DIMD), and the second is called template-based intra mode derivation (TIMD). In both modes, the reconstructed pixels (template pixels) around the current block in the top and left directions are used to derive the intra prediction mode. Specifically, in DIMD, the template of the reconstructed pixels is analyzed to infer the directionality of the template, and two directional modes are selected based on it. A prediction signal is generated based on mixing those two modes with a plane mode. On the other hand, in TIMD, the intra prediction mode is tested on the template of the reconstructed pixels, and two best modes (those modes that minimize the sum of absolute transform differences (SATD) between the template of the reconstructed pixels and their predictions) are selected. A prediction signal is generated based on mixing those two modes. In non-I-slices (B- or P-slices), it is possible that many intra-coded blocks are surrounded by inter-coded blocks. This leads to defects in filling the MPM list. This is because the MPM list is filled with intra modes of adjacent blocks. If the neighboring block is inter coded, a process called intra mode propagation is used, where the intra mode of the reference block is used if it is intra coded. Otherwise, some default mode is used.
[0027] Recently in ECM, two modes combine two inter-intra modes. First, combined inter-intra prediction (CIIP) and geometric partitioning mode (GPM). Specifically, CIIP derives the intra mode from its template pixels and combines this mode with the inter part, while GPM uses a similar intra-derived mode. This is different from the design of VVC, where CIIP only uses a predefined intra mode (planar) and GPM combines 2 inter modes, but no inter and intra modes.
[0028] Intra prediction modes are easily obtained with the help of GPM and can be used to populate the MPM list instead of using the mode of the reference samples. Similarly, intra prediction mode derivation can be used to infer the best intra prediction mode for neighboring inter blocks so that it will be used in the MPM list.
[0029] The main idea of the described aspects is to improve the MPM list construction by filling it with more accurate intra modes when neighboring blocks are inter coded.
[0030] Intra-mode coding with 67 intra-prediction modes
[0031] To capture the arbitrary edge directions present in natural video, the number of directional intra modes in VVC is extended from 33 as used in HEVC to 65. The new directional modes not present in HEVC are depicted as red dashed arrows in Figure 12, and the planar and DC modes remain the same. These denser directional intra prediction modes apply to all block sizes and both luma and chroma intra prediction.
[0032] In VVC, for non-square blocks, several conventional angular intra prediction modes are adaptively replaced with wide-angle intra prediction modes. Wide-angle intra prediction is described elsewhere in this specification.
[0033] In HEVC, each intra-coded block has a square shape and the length of each of its sides is a power of 2. Therefore, no division operation is required to generate the intra predictor using the DC mode. In VVC, blocks can have a rectangular shape, which in general requires the use of a division operation for each block. To avoid division operations for DC prediction, only the longer sides are used to calculate the average for non-square blocks.
[0034] In order to keep the complexity of the most probable mode (MPM) list generation low, an intra-mode coding method with 6 MPMs is used by considering two available adjacent intra-modes. The following three aspects are considered to construct the MPM list:
[0035] – Default intra mode
[0036] – Adjacent Intra Mode
[0037] – Intra-mode for export
[0038] Regardless of whether the MRL and ISP coding tools are applied, a unified 6-MPM list is used for intra blocks. The MPM list is constructed based on the intra modes of the left and above neighboring blocks. Assuming that the mode on the left is denoted as Left and the mode of the above block is denoted as Above, the unified MPM list is constructed as follows:
[0039] – When a neighboring block is not available, its intra mode is set to planar by default.
[0040] – If both Left and Above modes are non-angle modes, then:
[0041] – MPM list {Planar, DC, V, H, V-4, V+4}
[0042] – If one of the Left and Above modes is an angular mode and the other is a non-angular mode, then:
[0043] – Set the mode Max to the larger of the Left and Above modes
[0044] – MPM list {Planar, Max, Max-1, Max+1, Max––2, Max+2}
[0045] – If both Left and Above are angle modes and they are different, then:
[0046] – Set the mode Max to the larger of the Left and Above modes
[0047] – Set Mode Min to the smaller of Left and Above
[0048] – If Max–Min is equal to 1, then:
[0049] – MPM list {Planar, Left, Above, Min–1, Max+1, Min–2}
[0050] – Otherwise, if Max–Min is greater than or equal to 62, then:
[0051] – MPM list {Planar, Left, Above, Min+1, Max–1, Min+2}
[0052] – Otherwise, if Max–Min is equal to 2, then:
[0053] – MPM list {Planar, Left, Above, Min+1, Min–1, Max+1}
[0054] -otherwise:
[0055] – MPM list {Planar, Left, Above, Min–1, –Min+1, Max–1}
[0056] – If both Left and Above are angle modes and they are the same, then:
[0057] –MPM list {Planar, Left, Left-1, Left+1, Left–2, Left+2}
[0058] In addition, the first bin of the MPM index codeword is CABAC context coded. A total of three contexts are used, corresponding to whether the current intra block is MRL enabled, ISP enabled or a normal intra block.
[0059] During the 6MPM list generation process, pruning is used to remove repeated patterns so that only unique patterns can be included in the MPM list. For entropy coding of the 61 non-MPM patterns, truncated binary code (TBC) is used.
[0060] Decoder-side intra mode derivation (DIMD)
[0061] When DIMD is applied, two intra modes are derived from the reconstructed neighboring samples and those two predictors are combined with the planar mode predictor with weights derived from the gradients as described in JVET-00449. The division operations in the weight derivation are performed using the same lookup table (LUT) based integration scheme used by CCLM. For example, the division operations in the orientation calculation
[0062] Orient=G y / G x
[0063] It is calculated by the following LUT-based scheme:
[0064] x=Floor(Log2(Gx))
[0065] normDiff=((Gx<<4)>>x)&15
[0066] x+=(3+(normDiff!=0)?1:0)
[0067] Orient=(Gy*(DivSigTable[normDiff]|8)+(1<<(x-1)))>>x
[0068] in
[0069] DivSigTable
[16] ={0, 7, 6, 5, 5, 4, 4, 3, 3, 2, 2, 1, 1, 1, 1, 0}.
[0070] The derived intra modes are included in the main list of intra most probable modes (MPMs), so the DIMD process is performed before building the MPM list. The main derived intra modes of a DIMD block are stored with the block and used for MPM list construction for neighboring blocks.
[0071] Template-based fusion derived from intra-mode
[0072] For each intra prediction mode in the MPM, the SATD between the predicted and reconstructed samples of the template is calculated. The first two intra prediction modes with the smallest SATD are selected as TIMD modes. After applying the PDPC process, the two TIMD modes are fused with weights and the current CU is encoded using such weighted intra prediction. Position-dependent intra prediction combination (PDPC) is included in the derivation of TIMD modes.
[0073] The cost of the two chosen modes is compared to a threshold, and in the test a cost factor of 2 is applied as follows:
[0074] costMode2<2*costMode1.
[0075] If this condition is true, then fusion is applied, otherwise only mode 1 is used.
[0076] The weight of a mode is calculated based on its SATD cost as follows:
[0077] weight1=costMode2 / (costMode1+costMode2)
[0078] weight2=1-weight1
[0079] The division operation is performed using the same lookup table (LUT) based integration scheme used by CCLM.
[0080] GPM with inter-frame and intra-frame prediction
[0081] In GPM with inter and intra prediction, prediction samples are generated by weighting inter prediction samples and intra prediction samples for each GPM separation region. Inter prediction samples are derived by the same scheme as GPM in current ECM, while intra prediction modes are derived using the DIMD process. Figure 1 The GPM with intra and intra prediction shown in d is restricted in the proposed method to reduce the signaling overhead of IPM and avoid the increase in the size of the intra prediction circuit on the hardware decoder. In addition, direct motion vector and IPM storage is introduced on the GPM mixed region to further improve the coding performance.
[0082] CIIP in ECM
[0083] In CIIP mode, prediction samples are generated by weighting the inter prediction signal predicted using CIIP-TM merge candidate predictions and the intra prediction signal predicted using TIMD derived intra prediction modes. This method is only applicable to coding blocks with an area less than or equal to 1024.
[0084] In CIIP, the TIMD derivation method is used to derive intra prediction modes. Specifically, the intra prediction mode with the smallest SATD value in the TIMD mode list is selected and mapped to one of the 67 conventional intra prediction modes.
[0085] Furthermore, if the derived intra prediction mode is an angular mode, it is also proposed to modify the weights of the two tests (wIntra, wInter). For near-horizontal modes (2 <= angular mode index < 34), the current block is split vertically, as Figure 3 As shown in (a); for the near vertical mode (34 <= angle mode index <= 66), the current block is divided horizontally, such as Figure 3 As shown in (b).
[0086] Table 1 shows (wIntra, wInter) of different sub-blocks.
[0087] Table 1. Modification weights for angle mode.
[0088] Sub-block index (wIntra,wInter) 0 (6,2) 1 (5,3) 2 (3,5) 3 (2,6)
[0089] Using CIIP-TM, a CIIP-TM merge candidate list is established for the CIIP-TM mode. The merge candidates are refined by template matching. The CIIP-TM merge candidates are also reordered as regular merge candidates by the ARMC method. The maximum number of CIIP-TM merge candidates is equal to two.
[0090] Fill the MPM list with intra modes from neighboring blocks. Figure 4 Shown in.
[0091] AL: Upper left
[0092] A: Above
[0093] AR: Top right
[0094] L: Left
[0095] BL: lower left
[0096] These five neighbors are used by default to populate the MPM list. If any of them is inter-coded, the described general aspect proposes a different method for deriving its intra mode. In the prior art, this situation is handled by using propagated intra modes.
[0097] In case any neighbor (AL, A, AR, L, BL) is inter-coded, its intra mode is derived from its reference block. In the following embodiments, a new method is proposed to derive the intra modes of these blocks instead of using the current intra mode propagation method. The main advantage is that the derived intra mode is closer to the current mode, and therefore the MPM will encode the current intra mode more efficiently.
[0098] Example 1: Using CIIP intra-frame mode
[0099] If the neighboring block is coded with CIIP, its intra mode is available. It is proposed to use this mode to fill the MPM instead of the propagated intra mode. It should be noted that this mode corresponds to the TIMD mode, because TIMD is used for mode derivation in CIIP.
[0100] Example 2: Using GPM Intra Mode
[0101] In GPM, there is a mode where intra and inter are combined. Similar to the previous embodiment, this mode can be used to fill the MPM instead of the propagated intra mode. It should be noted that this mode corresponds to the DIMD mode, because DIMD is used for mode derivation in GPM.
[0102] Example 3: Performing DIMD / TIMD to derive the best prediction mode
[0103] If the neighboring block is neither GPM (with intra) nor CIIP, then the intra mode needs to be derived. As seen in the previous embodiments, the current mode is derived using TIMD (in CIIP) or DIMD (in GPM). Therefore, it is proposed to use any of them.
[0104] In other words, if the current neighboring block is not intra-coded, DIMD or TIMD is used to derive its intra-mode, and this mode is used in MPM list construction. It should be noted that this embodiment also considers the intra-block copy (IBC) mode. That is, if the neighboring block is IBC-coded, the TIMD or DIMD process is used to derive the best intra-mode.
[0105] Embodiment 4: Testing all possible intra modes on reconstructed inter blocks
[0106] Instead of performing DIMD or TIMD, it is proposed to infer the best intra prediction mode by testing all intra modes on reconstructed inter blocks around the current block. That is, if one of the neighboring blocks is inter coded and an MPM is to be constructed, the intra mode corresponding to the inter block is derived as follows:
[0107] 1- Generate a prediction signal for the currently considered inter block from the available intra modes
[0108] 2- Calculate the distance between the predicted signal and the reconstructed signal of the considered inter block.
[0109] 3- Select the mode with the smallest distance as the current intra mode for the considered inter block.
[0110] Figure 5 One embodiment of a method 500 according to the general aspects described herein is shown in FIG. The method starts at a start block 501, and control proceeds to block 510 for deriving an intra-coding mode from one or more inter-coded neighboring blocks. Control proceeds from block 510 to block 520 for conditionally adding the derived intra-coding mode to a most probable mode list. Control proceeds from block 520 to block 530 for encoding at least a portion of a video block using the most probable mode list.
[0111] Figure 6One embodiment of a method 600 according to the general aspects described herein is shown in FIG. 6. The method starts at a start block 601, and control proceeds to a block 610 for deriving an intra-coding mode from one or more inter-coded neighboring blocks. Control proceeds from block 610 to a block 620 for conditionally adding the derived intra-coding mode to a most probable mode list. Control proceeds from block 620 to a block 630 for decoding at least a portion of a video block using the most probable mode list.
[0112] Figure 7 An embodiment of an apparatus 700 for encoding, decoding, compressing or decompressing video data using any of the above methods or variations is shown. The apparatus includes a processor 710 and may be interconnected to a memory 720 via at least one port. Both the processor 710 and the memory 720 may also have one or more additional interconnections to connect to the outside.
[0113] The processor 710 is also configured to insert or receive information in a bitstream and use any of the aspects described for compression, encoding or decoding.
[0114] The embodiments described herein include various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are specifically described, and at least in order to illustrate the various characteristics, they are usually described in a manner that may sound restrictive. However, this is for the purpose of describing clearly and does not limit the application or scope of those aspects. In fact, all of the different aspects can be combined and interchanged to provide further aspects. In addition, the aspects can also be combined and interchanged with the aspects described in the earlier submissions.
[0115] The aspects described and contemplated in this application may be implemented in many different forms. Figure 8 , Fig. 9 and Fig.10 Some embodiments are provided, but other embodiments are contemplated, and Figure 8 , Fig. 9 and Fig.10 The discussion does not limit the breadth of implementation. At least one of the aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the methods.
[0116] In this application, the terms "reconstruction" and "decoding" can be used interchangeably, the terms "pixel" and "sample" can be used interchangeably, and the terms "image", "picture" and "frame" can be used interchangeably. Usually, but not necessarily, the term "reconstruction" is used at the encoder side, while "decoding" is used at the decoder side.
[0117] Various methods are described herein, and each of the methods includes one or more steps or actions for implementing the method. Unless a specific order of steps or actions is required for correct operation of the method, the order and / or use of specific steps and / or actions may be modified or combined.
[0118] Various methods and other aspects described herein may be used to modify modules (e.g., intra-frame prediction, entropy encoding and / or decoding modules (160, 360, 145, 330)) of the video encoder 100 and decoder 200, such as Figure 8 and Fig. 9 In addition, the present aspects are not limited to VVC or HEVC, and can be applied to, for example, other standards and recommendations (whether pre-existing or developed in the future) and extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise indicated or technically excluded, the aspects described in this application can be used alone or in combination.
[0119] Various numerical values are used in this application. The specific values are for illustrative purposes, and the aspects are not limited to these specific values.
[0120] Figure 8 An encoder 100 is illustrated. Variations of the encoder 100 are contemplated, but for clarity, the encoder 100 is described below without describing all contemplated variations.
[0121] Prior to encoding, the video sequence may undergo pre-encoding processing (101), such as applying a color transform to an input color picture (e.g., converting from RGB 4:4:4 to YCbCr4:2:0), or performing a remapping of input picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and attached to the bitstream.
[0122] In encoder 100, a picture is encoded by encoder elements as described below. The picture to be encoded is partitioned (102) and processed in units such as CUs. For example, each unit is encoded using intra mode or inter mode. When a unit is encoded in intra mode, the unit performs intra prediction (160). In inter mode, motion estimation (175) and compensation (170) are performed. The encoder decides (105) which of intra mode or inter mode to use to encode the unit, and indicates the intra / inter decision by, for example, a prediction mode flag. The prediction residual is calculated, for example, by subtracting (110) the predicted block from the original image block.
[0123] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, along with motion vectors and other syntax elements, are entropy encoded (145) to output a bitstream. The encoder may skip the transform and apply quantization directly to the untransformed residual signal. The encoder may bypass both the transform and quantization, i.e., encode the residual directly without applying the transform or quantization process.
[0124] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are de-quantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (155) to reconstruct the image block. An in-loop filter (165) is applied to the reconstructed picture to perform, for example, deblocking / SAO (sample adaptive offset) / ALF (adaptive loop filter) filtering to reduce coding artifacts. The filtered image is stored at a reference picture buffer (180).
[0125] Fig. 9 FIG. 2 is a block diagram of a video decoder 200. In the decoder 200, the bitstream is decoded by the decoder elements described below. The video decoder 200 generally performs the same operations as described above. Figure 8 The encoding pass described in is a decoding pass which is the inverse of the encoding pass described in . The encoder 100 also typically performs video decoding as part of encoding the video data.
[0126] In particular, the input to the decoder includes a video bitstream, which may be generated by the video encoder 100. First, the bitstream is entropy decoded (230) to obtain transform coefficients, motion vectors, and other encoding information. Picture partition information indicates how the picture is partitioned. Thus, the decoder can divide (235) the picture according to the decoded picture partition information. The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (255) to reconstruct the image block. The prediction block can be obtained (270) from intra-frame prediction (260) or motion compensated prediction (i.e., inter-frame prediction) (275). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored at a reference picture buffer (280).
[0127] The decoded picture may be further subjected to post-decoding processing (285), such as an inverse color transform (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping that performs the inverse of the remapping process performed in the pre-encoding process (101). The post-decoding process may use metadata derived in the pre-encoding process and signaled in the bitstream.
[0128] Fig.10 A block diagram of an example of a system in which various aspects and embodiments are implemented is illustrated. System 1000 can be implemented as a device including various components described below, and is configured to perform one or more aspects of the aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances and servers. The elements of system 1000 can be implemented individually or in combination in a single integrated circuit (IC), multiple ICs and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1000 is coupled to one or more other systems or other electronic devices via, for example, a communication bus or by a dedicated input port and / or output port communication. In various embodiments, system 1000 is configured to implement one or more aspects of the aspects described in this document.
[0129] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein for implementing, for example, the various aspects described in this document. The processor 1010 may include embedded memory, input-output interfaces, and various other circuits as known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which may include a non-volatile memory and / or a volatile memory, including but not limited to an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), a programmable read-only memory (PROM), a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a magnetic disk drive, and / or an optical disk drive. As a non-limiting example, the storage device 1040 may include an internal storage device, an attached storage device (including a removable and non-removable storage device), and / or a network-accessible storage device.
[0130] The system 1000 includes an encoder / decoder module 1030, which is configured to process data, for example, to provide encoded video or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents a module (one or more) that may be included in a device to perform encoding and / or decoding functions. As is well known, a device may include one or both of an encoding module and a decoding module. Additionally, the encoder / decoder module 1030 may be implemented as a separate element of the system 1000, or may be incorporated into the processor 1010 as a combination of hardware and software as known to those skilled in the art.
[0131] Program code to be loaded onto the processor 1010 or the encoder / decoder 1030 to perform various aspects described in this document may be stored in the storage device 1040 and subsequently loaded onto the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of the various entries during the execution of the processes described in this document. Such stored entries may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from processing equations, formulas, operations, and operation logic.
[0132] In some embodiments, memory internal to the processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or storage device 1040, such as dynamic volatile memory and / or non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store, for example, an operating system for a television. In at least one embodiment, a fast external dynamic volatile memory (such as RAM) is used as working memory for video encoding and decoding operations, such as for MPEG-2 (MPEG refers to Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard developed by the Joint Video Experts Group JVET).
[0133] Input to the elements of system 1000 may be provided through various input devices as indicated in block 1130. Such input devices include, but are not limited to: (i) a radio frequency (RF) section that receives an RF signal transmitted over the air, for example, by a broadcaster; (ii) a component (COMP) input terminal (or a set of COMP input terminals); (iii) a universal serial bus (USB) input terminal; and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Fig.10 Other examples not shown include composite video.
[0134] In various embodiments, the input device of block 1130 has associated corresponding input processing elements as known in the art. For example, the RF part can be associated with elements applicable to the following: (i) select the desired frequency (also referred to as selecting a signal, or limiting the signal band to a band), (ii) down-convert the selected signal, (iii) again band-limit to a narrower band to select the signal band that can be called a channel (for example) in some embodiments, (iv) demodulate the down-converted and band-limited signal, (v) perform error correction, and (vi) demultiplex to select the desired data packet stream. The RF part of various embodiments includes one or more elements for performing these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF part can include a tuner that performs various functions in these functions, including, for example, down-converting a received signal to a lower frequency (for example, an intermediate frequency or a near-baseband frequency) or down-converting to a baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal transmitted by wired (for example, cable) medium, and perform frequency selection by filtering, down-conversion and filtering to desired frequency band again.Various embodiments rearrange the order of (and other) elements described above, remove some elements in these elements, and / or add other elements of similar or different functions.Adding element can include inserting element between existing element, such as, inserting amplifier and analog-to-digital converter.In various embodiments, the RF part comprises antenna.
[0135] Additionally, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 1000 to other electronic devices across the USB and / or HDMI connections. It is to be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented, for example, in a separate input processing IC or in the processor 1010 as needed. Similarly, various aspects of USB or HDMI interface processing may be implemented, for example, in a separate interface IC or in the processor 1010 as needed. The demodulated, error corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 1010 and the encoder / decoder 1030, which operate in conjunction with the memory and storage elements to process the data stream as needed for presentation on the output device.
[0136] The various elements of system 1000 may be provided within an integrated housing within which the various elements may be interconnected and data transmitted therebetween using a suitable connection arrangement (e.g., an internal bus as known in the art, including an Inter-IC (I2C) bus, wiring, and printed circuit boards).
[0137] The system 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data through the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel 1060 may be implemented, for example, within a wired and / or wireless medium.
[0138] In various embodiments, a wireless network such as a Wi-Fi network (e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers)) is used to stream or otherwise provide data to the system 1000. The Wi-Fi signals of these embodiments are received through a communication channel 1060 and a communication interface 1050 suitable for Wi-Fi communication. The communication channel 1060 of these embodiments is typically connected to an access point or router that provides access to an external network (including the Internet) to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box to provide streaming data to the system 1000, which delivers data through the HDMI connection of the input block 1130. Still other embodiments use the RF connection of the input block 1130 to provide streaming data to the system 1000. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use a wireless network other than Wi-Fi, such as a cellular network or a Bluetooth network.
[0139] The system 1000 can provide output signals to various output devices, including a display 1100, a speaker 1110, and other peripheral devices 1120. The display 1100 of various embodiments includes, for example, one or more of a touch screen display, an organic light emitting diode (OLED) display, a curved display, and / or a foldable display. The display 1100 can be used for a television, a tablet computer, a laptop computer, a cellular phone (mobile phone), or another device. The display 1100 can also be integrated with other components (e.g., as in a smart phone), or separated (e.g., an external monitor for a laptop computer). In various examples of embodiments, other peripheral devices 1120 include one or more of an independent digital video disk (or digital versatile disk) (DVR, for both terms), a disk player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 1120 that provide functions based on the output of the system 1000. For example, a disk player performs the function of playing the output of the system 1000.
[0140] In various embodiments, control signals are transmitted between the system 1000 and the display 1100, speaker 1110, or other peripheral device 1120 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable device-to-device control with or without user intervention. Output devices may be communicatively coupled to the system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, output devices may be connected to the system 1000 using a communication channel 1060 via a communication interface 1050. In an electronic device such as, for example, a television, the display 1100 and speaker 1110 may be integrated into a single unit with other components of the system 1000. In various embodiments, the display interface 1070 includes a display driver such as, for example, a timing controller (T Con) chip.
[0141] For example, if the RF portion of input 1130 is part of a separate set-top box, the display 1100 and speaker 1110 may alternatively be separate from one or more of the other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signal may be provided via a dedicated output connection including, for example, an HDMI port, a USB port, or a COMP output.
[0142] The embodiments may be performed by computer software implemented by the processor 1010, or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. As a non-limiting example, the memory 1020 may be of any type suitable for the technical environment, and may be implemented using any appropriate data storage technology (such as an optical memory device, a magnetic memory device, a semiconductor-based memory device, a fixed memory, and a removable memory). As a non-limiting example, the processor 1010 may be of any type suitable for the technical environment, and may include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.
[0143] Various implementations involve decoding. "Decoding" as used in the present application may encompass, for example, all or part of a process performed on a received coded sequence to produce a final output suitable for display. In various embodiments, such a process includes one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such a process also includes or alternatively includes a process performed by a decoder of the various implementations described in the present application.
[0144] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or generally to a broader decoding process will be clear based on the context of the specific description and is considered to be well understood by those skilled in the art.
[0145] Various implementations involve encoding. In a manner similar to the discussion above about "decoding", "encoding" as used in this application can cover, for example, all or part of a process performed on an input video sequence to produce an encoded bitstream. In various embodiments, such a process includes one or more of the processes typically performed by an encoder, such as partitioning, differential encoding, transforms, quantization, and entropy encoding. In various embodiments, such a process also includes or alternatively includes a process performed by an encoder of the various implementations described in this application.
[0146] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or generally to a broader encoding process will be clear based on the context of the specific description and is considered to be well understood by those skilled in the art.
[0147] Note that the syntax elements as used herein are descriptive terms. Therefore, they do not exclude the use of other syntax element names.
[0148] When the figures are presented as flow charts, it should be understood that they also provide block diagrams of the corresponding apparatus. Similarly, when the figures are presented as block diagrams, it should be understood that they also provide flow charts of the corresponding methods / processes.
[0149] Various embodiments may relate to parameter models or rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between rate and distortion is often considered, which often takes into account constraints on computational complexity. It can be measured by a rate-distortion optimization (RDO) metric or by a least mean square (LMS), mean absolute error (MAE) or other such measurements. Rate-distortion optimization is generally expressed as minimizing a rate-distortion function that is a weighted sum of rate and distortion. There are different methods to solve the rate-distortion optimization problem. For example, the method may be based on an extended test of all coding options (including all considered modes or coding parameter values) with a complete evaluation of its coding cost and the associated distortion of the reconstructed signal after encoding and decoding. Faster methods can also be used to save coding complexity, particularly the calculation of approximate distortion based on a predicted or predicted residual signal rather than a reconstructed residual signal. A mixture of these two methods may also be used, such as by using approximate distortion for only some of the possible coding options and using full distortion for other coding options. Other methods evaluate only a subset of possible coding options. More generally, many approaches employ any of a variety of techniques to perform optimization, but optimization is not necessarily a complete assessment of both coding cost and associated distortion.
[0150] The implementations and aspects described herein may be implemented in, for example, a method or process, a device, a software program, a data stream, or a signal. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method), the implementation of the features discussed may also be implemented in other forms (e.g., a device or program). The device may be implemented in, for example, appropriate hardware, software, and firmware. The method may be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes communication devices, such as, for example, computers, cellular phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate the transmission of information between end users.
[0151] References to "one embodiment" or "an embodiment" or "an implementation" or "implementation" and other variations thereof mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearance of the phrases "in one embodiment" or "in an embodiment" or "in an implementation" or "in an implementation" and any other variations thereof appearing in various places throughout this application are not necessarily all referring to the same embodiment.
[0152] Additionally, the present application may involve "determining" various pieces of information. Determining information may include, for example, one or more of: estimating information, calculating information, predicting information, or retrieving information from a memory.
[0153] Furthermore, the present application may involve "accessing" various pieces of information. Accessing information may include, for example, one or more of: receiving information, retrieving information (e.g., retrieving information from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.
[0154] Additionally, the present application may involve "receiving" various pieces of information. As with "accessing," receiving is intended to be a broad term. Receiving information may include, for example, one or more of: accessing information or retrieving information (e.g., retrieving information from a memory). Furthermore, "receiving" is generally involved in one way or another during operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.
[0155] It is to be understood that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," the use of any of the following " / ," "and / or," and "at least one of..." is intended to encompass selecting only the first listed option (A), or selecting only the second listed option (B), or selecting both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such wording is intended to encompass selecting only the first listed option (A), or selecting only the second listed option (B), or selecting only the third listed option (C), or selecting only the first listed option and the second listed option (A and B), or selecting only the first listed option and the third listed option (A and C), or selecting only the second listed option and the third listed option (B and C), or selecting all three options (A and B and C). As will be apparent to one of ordinary skill in this and related arts, this may be extended to as many items as listed.
[0156] In addition, among other things, as used herein, the word "signaling" also points to a corresponding decoder to indicate something. For example, in some embodiments, the encoder signals a specific one of multiple transforms, coding modes or flags. In this way, in an embodiment, the same transform, parameter or mode is used at both the encoder side and the decoder side. Therefore, for example, the encoder can transmit (explicitly signal) specific parameters to the decoder so that the decoder can use the same specific parameters. On the contrary, if the decoder already has specific parameters and other parameters, signaling can be used without transmitting (implicitly signaling) to allow only the decoder to know and select specific parameters. By avoiding the transmission of any actual function, bit saving is achieved in various embodiments. It is to be understood that signaling can be implemented in a variety of ways. For example, in various embodiments, one or more grammatical elements, flags, etc. are used to signal information to the corresponding decoder. Although the verb form of the word "signaling" is mentioned above, the word "signal" can also be used as a noun in this article.
[0157] As will be apparent to one of ordinary skill in the art, implementations may generate a variety of signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for executing a method or data generated by one of the described implementations. For example, a signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is well known, a signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor readable medium.
[0158] The previous sections describe multiple embodiments across various claim categories and types. Features of these embodiments may be provided individually or in any combination. In addition, embodiments may include one or more of the following features, devices, or aspects, individually or in any combination across various claim categories and types:
[0159] One embodiment includes predicting a video block using a most probable mode list that includes intra-coded modes from neighboring inter-coded blocks.
[0160] One embodiment includes the prediction described above, wherein a motion model is used to determine the intra coding mode
[0161] One embodiment includes the above method, wherein the intra coding mode is determined using combined inter-intra prediction, geometric partitioning mode, template-based intra mode derivation, or decoder-side intra mode derivation.
[0162] An embodiment includes the above method wherein a motion model is used to determine the reference frame or the motion vector.
[0163] Some embodiments include any of the methods described above, wherein an index is signaled to indicate a motion model, reference frame, or motion vector to be used for encoding / decoding.
[0164] Some embodiments include any of the methods described above that use reference samples surrounding the motion compensated block from a reference frame.
[0165] One embodiment includes a bitstream or signal that includes one or more syntax elements to perform the above functions or variations thereof.
[0166] One embodiment comprises a bitstream or signal comprising syntax conveying information generated according to any of the described embodiments.
[0167] An embodiment comprises creating and / or transmitting and / or receiving and / or decoding according to any of the described embodiments.
[0168] One embodiment includes a method, process, apparatus, medium storing instructions, medium storing data, or signal according to any of the described embodiments.
[0169] One embodiment comprises inserting syntax elements in the signaling that enable the decoder to determine decoding information in a manner corresponding to that used by the encoder.
[0170] One embodiment comprises creating and / or transmitting and / or receiving and / or decoding a bitstream or signal comprising one or more of the described syntax elements or variations thereof.
[0171] One embodiment comprises a TV, set-top box, cell phone, tablet computer or other electronic device that performs the transformation method(s) according to any of the described embodiments.
[0172] One embodiment includes a TV, set-top box, cellular phone, tablet computer, or other electronic device that performs (one or more) transformation methods according to any of the described embodiments to determine and display (e.g., using a monitor, screen, or other type of display) the resulting image.
[0173] One embodiment includes a TV, set-top box, cellular phone, tablet computer, or other electronic device that selects, band limits, or tunes (e.g., using a tuner) a channel to receive a signal including an encoded image and performs (one or more) transformation methods in accordance with any of the described embodiments.
[0174] One embodiment includes a TV, set-top box, cellular phone, tablet computer, or other electronic device that receives over the air (eg, using an antenna) a signal including an encoded image and performs the transformation method(s).
Claims
1. A method for encoding a video block, comprising: deriving an intra coding mode using a template intra mode derivation process; When the current neighboring block is inter-coded, adding the derived intra-coding mode to a most probable mode list; and, At least a portion of the video block is encoded using the most probable mode list.
2. A device comprising: A processor configured to execute: deriving an intra coding mode using a template intra mode derivation process; When the current neighboring block is inter-coded, adding the derived intra-coding mode to a most probable mode list; and, At least a portion of the video block is encoded using the most probable mode list.
3. A method comprising: deriving an intra coding mode using a template intra mode derivation process; When the current neighboring block is inter-coded, adding the derived intra-coding mode to a most probable mode list; and, At least a portion of the video block is decoded using the most probable mode list.
4. A device comprising: A processor configured to execute: deriving an intra coding mode using a template intra mode derivation process; When the current neighboring block is inter-coded, adding the derived intra-coding mode to a most probable mode list; and, At least a portion of the video block is decoded using the most probable mode list.
5. The method according to claim 1 or 3, or the apparatus according to claim 2 or 4, wherein the derived intra coding mode is obtained from neighboring blocks using a combined inter-intra prediction coding mode.
6. A method according to claim 1 or 3 or 5, or an apparatus according to claim 2 or 4 or 5, wherein the derived intra-frame coding mode is obtained from a geometric partitioning mode in which inter-frame coding and intra-frame coding are combined.
7. The method according to claim 1 or 3 or 5 to 6, or the apparatus according to claim 2 or 4 or 5 to 6, wherein the derived intra-frame coding mode is derived from a template-based intra-frame mode.
8. The method according to claim 1, 3 or 5 to 7, or the apparatus according to claim 2, 4 or 5 to 7, wherein the derived intra-coding mode is derived from a decoder-side intra-coding mode.
9. The method or apparatus of claim 8, wherein the neighboring blocks are not intra-coded.
10. A method according to claim 1, 3 or 5 to 9, or an apparatus according to claim 2, 4 or 5 to 9, wherein the derived intra coding mode is obtained by testing intra modes on reconstructed inter blocks surrounding the video block.
11. A method or apparatus according to claim 10, wherein the derived intra coding mode is obtained by selecting the mode having the smallest distance between the prediction signal and the reconstructed version of the considered inter block.
12. A device comprising: The device according to claim 4; as well as At least one of: (i) an antenna configured to receive a signal, the signal comprising a video block; (ii) a band limiter configured to limit the received signal to a frequency band comprising the video block; and (iii) a display configured to display an output representing the video block.
13. A non-transitory computer-readable medium, comprising data content for playback using a processor, the data content being generated according to the method of any one of claims 1 and 5 to 11 or generated by the apparatus of any one of claims 2 and 5 to 11.
14. A signal comprising video data for playback using a processor, the video data being generated according to the method of any one of claims 1 and 5 to 11 or generated by the apparatus of any one of claims 2 and 5 to 11.
15. A computer program product comprising instructions which, when said program is executed by a computer, cause said computer to perform the method according to any one of claims 1, 3 and 5 to 11.
