Conversion selection of implicit multiple conversion selection

Abstract

To provide method and apparatus for video encoding or decoding.SOLUTION: An encoding method includes a step of setting syntax information indicating use of a multiple conversion selection mode when an implicit conversion selection mode is used to encode at least one video block in a bitstream, a step of encoding the at least one video block by using at least one conversion corresponding to the syntax information, and a step of storing, communicating, or transmitting a bitstream including the encoded at least one video block. A decoding method includes a step of analyzing the bitstream for syntax information indicating use of a multiple conversion selection mode when an implicit conversion selection mode is used to decode a video block, and a step of decoding at least one video block in the bitstream by using at least one conversion of the multiple conversion selection mode corresponding to the information when the implicit transform selection mode is used.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] Technical Field At least one of the present embodiments generally relates to a method or apparatus for video encoding or decoding. [Background technology]

[0002] background To achieve high compression efficiency, image and video coding schemes typically employ prediction (including spatial and / or motion vector prediction) and transform to exploit spatial and temporal redundancy within video content. Typically, intra- or inter-prediction is used to exploit intra- or inter-frame correlation, and then the difference between the original image and the predicted image (often represented by prediction error or prediction residual) is transformed, quantized, and entropy coded. To reconstruct the video, the compressed data is decoded by an inverse process corresponding to entropy coding, quantization, transform, and prediction. Many coding tools can be used in the coding and decoding (including transform and inverse transform) processes. Summary of the Invention

[0003] overview The shortcomings and disadvantages of the prior art may be addressed by the general aspects described herein, which are directed towards block shape adaptive intra prediction during encoding and decoding.

[0004] According to a first aspect, a method is provided, the method including: setting syntax information indicating use of a multiple transform selection mode when an implicit transform selection mode is used to encode at least one video block in a bitstream; encoding the at least one video block by using at least one transform corresponding to the syntax information; and storing, communicating, or transmitting the bitstream including the encoded at least one video block.

[0005] According to a second aspect, a method is provided, the method including: parsing a bitstream for syntax information indicating use of a multiple transform selection mode when an implicit transform selection mode is used to decode a video block; and, when the implicit transform selection mode is used, decoding at least one video block in the bitstream by using at least one transform of the multiple transform selection mode corresponding to the syntax information.

[0006] According to another aspect, an apparatus is provided, the apparatus including a processor, the processor may be configured to encode blocks of video or to decode a bitstream by performing any of the methods described above.

[0007] According to another general aspect of at least one embodiment, there is provided a device including an apparatus according to any of the decoding embodiments; and at least one of (i) an antenna configured to receive a signal including the video block, (ii) a band limiter configured to limit the received signal to a band of frequencies including the video block, or (iii) a display configured to display an output representing the video block.

[0008] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided that includes data content generated in accordance with any of the described coding embodiments or variations.

[0009] According to another general aspect of at least one embodiment, there is provided a signal including video data generated according to any of the described encoding embodiments or variants.

[0010] According to another general aspect of at least one embodiment, a bitstream is formatted to include data content generated according to any of the described encoding embodiments or variants.

[0011] According to another general aspect of at least one embodiment, there is provided a computer program product including instructions that, when executed by a computer, cause the computer to perform any of the above-described coding embodiments or variations.

[0012] These and other aspects, features and advantages of the general aspects will become apparent from the following detailed description of illustrative embodiments, which is to be read in connection with the accompanying drawings. [Brief explanation of the drawings]

[0013] BRIEF DESCRIPTION OF THE DRAWINGS [Figure 1] 1 shows a standard and common video compression scheme. [Figure 2] 1 shows a standard and common video compression scheme. [Figure 3] 1 shows an exemplary flowchart of a proposed method for selecting a transform pair depending on the LFNST flag. [Figure 4] 1 shows an exemplary flowchart of a proposed method for selecting a transform pair depending on the LFNST flag and the MIP flag. [Figure 5] 10 shows an exemplary flowchart of a proposed method for selecting a transform pair depending on an LFNST CU flag and an MIP CU flag. [Figure 6] 10 shows an exemplary flowchart of a proposed method for selecting a transform pair depending on the LFNST CU flag. [Figure 7] 1 illustrates one embodiment of an encoding method under the general aspects described. [Figure 8] 1 illustrates one embodiment of a decoding method under the general aspects described. [Figure 9] 1 illustrates an embodiment of an apparatus for encoding or decoding using intra-prediction mode extensions. [Figure 10] 1 illustrates an exemplary processor arrangement in which the described embodiments may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0014] Detailed Description The following general aspects are in the field of video compression. They aim to improve the compression efficiency of an upcoming video compression standard: Versatile Video Coding (VVC). The Versatile Video Coding (VVC) test model (VTM) is a standardization effort to improve video compression. Specifically, these aspects affect the transform design of this standard, where compression defects are newly introduced due to the adoption of two tools: the Low-Frequency Non-Separable Transform (LFNST) and Matrix-based intra prediction (MIP).

[0015] In addition to the DCT2, the VVC transform design includes other transforms: DCT8 and DST7 (collectively known as multiple transform selection (MTS)). The encoder may select one combination of the defined transforms (DCT2, DST7 and DCT8). There are the following possible pairs of horizontal and vertical transforms: - {DCT2,DCT2} - {DST7,DCT7} - {DST7,DCT8} - {DCT8,DST7} - {DCT8,DCT8}

[0016] The MTS is controlled by an SPS (Sequence Parameter Set) flag that can deactivate the coder, and only the pair {DCT2, DCT2} is considered.

[0017] When the MTS flag is enabled, a coding gain of about 1.7% over the MTS-off case is achieved, while longer coding times are required due to RDO (Rate Distortion Optimization) competition between multiple candidates, which is time consuming. Alternatively, another mode of transform signaling (named MTSImplicit) has recently been adopted in VVC. In this mode, the transform selection is predefined (implicit) depending on the dimensions of the transform block. In the VVC specification, the transform selection for a luma block by MTSImplicit is as follows: trTypeHor=(Width>=4 && Width<=16)?DST7:DCT2 trTypeVer=(Height>=4 && Height<=16)?DST7:DCT2

[0018] The same selection mechanism is used for intra-subpartioning (ISP). This type of selection generates new pairs that are combinations of DCT2 and DST7. The following table summarizes the cases in which new pairs arise:

[0019] [Table 1]

[0020] Recently, two new coding tools were adopted into the JVET (Joint Video Experts Team) standard. The first is the LFNST (Low-Frequency Non-Separable Transform), formerly known as NSST (Non-Separable Secondary Transform). This is an efficient tool that accounts for redundancy in the transform domain and applies another non-separable transform to produce a sparse representation of the residual that is then quantized and coded. In the VVC test model VTM-5.0 version, two LFNST transform sets are defined: one transform set for large residual blocks and the other for small residual blocks. For each set, four groups of transforms are defined, each containing two transform matrices. The group selection depends on the intra-prediction mode, and the transform index is coded in the bitstream.

[0021] Unlike the linear transforms, which are DCT2, DST7, and DCT8, LFSNT is a trained transform. In other words, the basic functions of LFSNT are obtained by learning from a large dataset of residual data. The nature of the dataset and the training method are not explained in any public document.

[0022] The second tool is matrix-based intra prediction (MIP). The second tool is a trained tool for predicting a block from its neighboring reference samples. The second tool can be considered an extension to existing intra prediction, where the prediction signal is generated by multiplying the reference samples by a matrix rather than by simple duplication along the prediction direction. Similar to LFNST, the matrices used for multiplication are trained matrices with no description of the dataset used to obtain them.

[0023] To understand the interaction of these tools with the MTSImplicit transform design, simulations were performed using VTM-5.0 with common test conditions. For simplicity, simulation results with one frame intra-coding are presented. The anchor is VTM-5.0 with MTS disabled (--MTS=0), and the test is the implicit transform design (--MTS=0 -MTSImplicit=1). The results are shown in the table below.

[0024] [Table 2]

[0025] As can be seen, instead of the predicted gain of the implicit conversion design, there is a loss of 0.3%. Another test is performed to verify the coding gain when the two new tools are deactivated; that is, the anchors and tests are the same, but LFNST and MIP are set to zero. The results are provided in the table below.

[0026] [Table 3]

[0027] Therefore, it can be concluded that the predicted gain is maintained when the new tool is switched off.

[0028] To understand the impact of each tool, the two tools are deactivated individually. The table below presents the results of switching only LFNST to zero and leaving MIP activated.

[0029] [Table 4]

[0030] And the table below is for when MIP is off and LFNST is off.

[0031] [Table 5]

[0032] From the simulation results, it can be observed that MIP and / or LFNST are incompatible with the implicit transformation design. However, the impact of LFNST is much more severe than that of MIP. The reason for the incompatibility is that these tools are trained with residuals (which do not have the new pairwise transformations specified in the MTSImplicit tool). In fact, MTSImplicit incurs a loss of coding efficiency when used in combination with these two tools.

[0033] The general aspects described solve the problem of MTSImplicit loss and improve the transformation design of ImplicitMTS by considering the interaction between the tools LFNST and MIP. The MTSImplicit design does not fit the newly added tools LFNST and MIP. To solve this problem, the simplest way is to deactivate implicit selection when these tools are activated.

[0034] Embodiment 1: Implicit MTS with DCT2 when LFNST and / or MIP are activated In VTM5, the choice of MTSImplicit transformation is described as follows: trTypeHor=(Width>=4 && Width<=16)?DST7:DCT2 trTypeVer=(Height>=4 && Height<=16)?DST7:DCT2

[0035] To resolve the interaction issue between LFNST and MIP, the SPS flag of MIP (sps_mip_flag) and / or the SPS flag of LFNST (lfnst_enabled_flag) need to be checked: if(lfnst_enabled_flag) { trTypeHor=DCT2 trTypeVer=DCT2 } { trTypeHor=(Width>=4 && Width<=16)?DST7:DCT2 trTypeVer=(Height>=4 && Height<=16)?DST7:DCT2 }

[0036] The corresponding flow chart is shown in Figure 3.

[0037] To demonstrate the advantages of this method, a simulation was performed with VTM 5.0 (with the new method as a test (both with the following options: --MTS=0 and --MTSImplicit=1)) under the same conditions as those described in Table 2 (1-frame analysis). The following results were obtained:

[0038] [Table 6]

[0039] An additional condition was also tested: if LFNST is deactivated, the following results are obtained:

[0040] [Table 7]

[0041] Finally, when the MIP is deactivated, the following results are obtained:

[0042] [Table 8]

[0043] Another way to implement the method is to consider both the LFNST flag and the MIP flag: if(lfnst_enabled_flag && sps_mip_flag) { trTypeHor=DCT2 trTypeVer=DCT2 } { trTypeHor=(Width>=4 && Width<=16)?DST7:DCT2 trTypeVer=(Height>=4 && Height<=16)?DST7:DCT2 }

[0044] The corresponding flow chart is shown in Figure 4.

[0045] In a variation of this embodiment, if LFNST or MIP is activated for the current CU, the implicit MTS becomes DCT2, DCT2.

[0046] Embodiment 2: Implicit MTS with DCT2 when LFNST and / or MIP are used Besides checking the SPS flag, we can also look at the current coding unit (CU) and see if MIP or LFNST is used. For this case, the implicit mts selection is DCT2. In VTM5.0, lfnstIdx is the index of the LFNST with values ​​between 0 and 2. Zero indicates no LFNST. For MIP, the flag mipFlag indicates whether MIP is used. So the modification is as follows: if(CU::lfnstIdx || cu::mipFlag) { trTypeHor=DCT2 trTypeVer=DCT2 } { trTypeHor=(Width>=4 && Width<=16)?DST7:DCT2 trTypeVer=(Height>=4 && Height<=16)?DST7:DCT2 }

[0047] The corresponding flow chart is shown in FIG.

[0048] The difficulty with this method is that the transform selection depends on the selected prediction mode. Generally, it would be preferable to not have this dependency. However, since the current LFNST is intra-mode dependent, this method is acceptable.

[0049] Similarly, we can consider only the cases where LFNST is used: if(CU::lfnstIdx) { trTypeHor=DCT2 trTypeVer=DCT2 } { trTypeHor=(Width>=4 && Width<=16)?DST7:DCT2 trTypeVer=(Height>=4 && Height<=16)?DST7:DCT2 }

[0050] The corresponding flow chart is shown in FIG.

[0051] In a variation of this embodiment, if LFNST or MIP is activated for the current CU, the implicit MTS becomes DCT2, DCT2.

[0052] A one-frame analysis is also performed here, with the anchor being VTM5.0 (the new method as test (both with the following options: --MTS=0 and -MTSImplicit=1)), giving the following results:

[0053] [Table 9]

[0054] To demonstrate the advantages over the method of embodiment 1, one frame analysis was performed with embodiment 1 as anchor and the present method as test. The following results are obtained:

[0055] [Table 10]

[0056] One embodiment of a method 700 under the general aspects described herein is shown in Figure 7. The method begins at start block 701, and control passes to block 710 for setting syntax information indicating the use of a multiple transform selection mode if an implicit transform selection mode is used for the encoding of at least one video block in a bitstream. Control passes from block 710 to block 720 for encoding the at least one video block by using at least one transform corresponding to the syntax information. Control passes from block 720 to block 730 for storing, communicating, or transmitting a bitstream including the encoded at least one video block.

[0057] One embodiment of a method 800 under the general aspects described herein is shown in Figure 8. The method begins at start block 801, and control passes to block 810 for parsing a bitstream for syntax information indicating the use of a multiple transform selection mode when an implicit transform selection mode is used to decode a video block. Control passes from block 810 to block 820 for decoding at least one video block in the bitstream by using at least one transform of a multiple transform selection mode that corresponds to the syntax information when an implicit transform selection mode is used.

[0058] 9 shows one embodiment of an apparatus 900 for compressing, encoding, or decoding video by using a coding or decoding tool. The apparatus includes a processor 910 and may be interconnected via at least one port to a memory 920. Both the processor 910 and the memory 920 may also have one or more additional interconnections to external connections.

[0059] The processor 910 is also configured to insert information into or receive information in a bitstream by either compressing, encoding or decoding it using various coding tools.

[0060] This application describes various aspects, including tools, features, embodiments, models, techniques, and the like. Many of these aspects are described with specificity, and often in a manner that may sound limiting, at least to illustrate individual features. However, this is for purposes of clarity in the description and therefore does not limit the applicability or scope of these aspects. In fact, all of the various aspects may be combined or interchanged to provide further aspects. Furthermore, various aspects may be combined or interchanged with aspects described in previous applications.

[0061] Aspects described and contemplated in this application may be implemented in many different forms. While Figures 1, 2, and 10 provide some embodiments, other embodiments are contemplated, and thus the discussion of Figures 1, 2, and 10 does not limit the scope of the embodiments. At least one of the aspects relates generally to video encoding and decoding, and at least one other aspect relates generally to transmitting a generated or encoded bitstream. These and other aspects may be embodied as a method, an apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the described methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the described methods.

[0062] In this application, the terms "reconstructed" and "decoded" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image", "picture" and "frame" may be used interchangeably. Typically, but not necessarily, the term "reconstructed" is used on the encoder side, while the term "decoded" is used on the decoder side.

[0063] Various methods are described herein, each of which includes one or more steps or acts for implementing the described method. Unless a specific order of steps or acts is required for the proper operation of the method, the order and / or use of specific steps and / or acts may be modified or combined.

[0064] Various methods and other aspects described in this application may be used to modify modules, such as the intra-prediction, entropy coding, and / or decoding modules (160, 360, 145, 330) of video encoder 100 and decoder 200 as shown in FIGS. 1 and 2 . Furthermore, the aspects are not limited to VVC or HEVC and thus may be applied, for example, to other standards and recommendations (whether existing or developed in the future) and extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise indicated or technically precluded, the aspects described in this application may be used individually or in combination.

[0065] Various numerical values ​​are used in this application. The specific values ​​are for illustrative purposes, and therefore the described aspects are not limited to these specific values.

[0066] 1 shows an encoder 100. Although variations of this encoder 100 are contemplated, encoder 100 is described below for clarity without describing all possible variations.

[0067] Before being encoded, a video sequence may go through a pre-encoding process (101): for example, applying a color transformation (e.g., from RGB 4:4:4 to YCbCr 4:2:0) to the input color picture, or remapping the input picture components (e.g., using histogram equalization of one of the color components) to obtain a signal distribution that is more resistant to compression. Metadata may be associated with the pre-processing and attached to the bitstream.

[0068] In encoder 100, a picture is coded by the encoder elements as described below. The picture to be coded is divided (102) and processed, for example, in units of CUs. Each unit is coded, for example, using either intra mode or inter mode. If a unit is coded in intra mode, encoder 100 performs intra prediction (160). In inter mode, motion estimation (175) and motion compensation (170) are performed. The encoder (105) determines whether intra mode or inter mode should be used to code the unit and indicates the intra / inter decision, for example, with a prediction mode flag. A prediction residual is calculated, for example, by subtracting (110) the predicted block from the original image block.

[0069] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, as well as motion vectors and other syntax elements, are entropy coded (145) to output a bitstream. The encoder may skip this transform and apply quantization directly to the untransformed residual signal. The encoder may bypass both the transform and quantization, i.e., the residual is coded directly without applying the transform or quantization processes.

[0070] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residual. An image block is reconstructed by combining the decoded prediction residual with the predicted block (155). An in-loop filter (165) is applied to the reconstructed image to perform deblocking / SAO (Sample Adaptive Offset) filtering, for example, to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0071] Figure 2 shows a block diagram of a video decoder 200. In decoder 200, the bitstream is decoded by decoder elements as described below. Video decoder 200 typically performs a reciprocal decoding path to the encoding path described in Figure 1, and encoder 100 also typically performs video decoding as part of encoding the video data.

[0072] In particular, the decoder input includes a video bitstream, such as may be generated by video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other coded information. Picture partition information indicates how the image is divided. Thus, the decoder may partition the picture according to the decoded picture partition information (235). The transform coefficients are dequantized (240) and then inverse transformed (250) to decode the prediction residual. An image block is reconstructed by combining the decoded prediction residual with a prediction block (255). The prediction block may be obtained from intra prediction (260) or motion-compensated prediction (i.e., inter prediction) (270). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0073] The decoded picture may further go through a post-decoding process (285) (such as an inverse color transform (e.g., YCbCr 4:2:0 to RGB 4:4:4) or an inverse remapping that reverses the remapping process performed in the pre-encoding process (101)). The post-decoding process may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0074] FIG. 10 shows a block diagram of an example system in which various aspects and embodiments may be implemented. System 1000 may be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include various electronic devices, such as, but not limited to, personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, servers, etc. Elements of system 1000 (alone or in combination) may be embodied in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1000 is communicatively coupled to one or more other systems or other electronic devices, for example, via a communication bus or via dedicated input and / or output ports. In various embodiments, system 1000 is configured to perform one or more of the aspects described herein.

[0075] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein, for example, to implement various aspects described herein. The processor 1010 may include embedded memory, input / output interfaces, and various other circuitry known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which may include non-volatile and / or volatile memory, including, but not limited to, electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash, magnetic disk drives, and / or optical disk drives. The storage device 1040 may include, by way of non-limiting example, an internal storage device, an attached storage device (including a removable or non-removable storage device), and / or a network-accessible storage device.

[0076] System 1000 includes an encoder / decoder module 1030 configured to process data, for example, to provide encoded video or decoded video. The encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents a module that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. Additionally, the encoder / decoder module 1030 may be implemented as a separate element of system 1000 or may be incorporated within processor 1010 as a combination of hardware and software, as is known to those skilled in the art.

[0077] Program code loaded into the processor 1010 or the encoder / decoder 1030 to perform various aspects described herein may be stored in the storage device 1040 and then loaded into the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more various items during performance of the processes described herein. Such stored items include, but are not limited to, input video, decoded video, or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and arithmetic logic.

[0078] In some embodiments, memory internal to the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and to provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be either the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be the memory 1020 and / or the storage device 1040 (e.g., dynamic volatile memory and / or non-volatile flash memory). In some embodiments, the external non-volatile flash memory is used to store the operating system of the television (for example). In at least one embodiment, a high-speed external dynamic volatile memory such as RAM is used as working memory for video coding and decoding operations such as MPEG2 (MPEG refers to the Moving Picture Expert Group, MPEG2 is also known as ISO / IEC 13818, 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard being developed by JVET (Joint Video Experts Team)).

[0079] Input to the elements of system 1000 may be provided via various input devices shown in block 1130. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives RF signals transmitted wirelessly, for example by a broadcaster, (ii) a component (COMP) input terminal (or a set of COMP input terminals), (iii) a Universal Serial Bus (USB) input terminal, and / or (iv) a High Definition Multimedia Interface (HDMI) input terminal. Other examples not shown in FIG. 10 include composite video.

[0080] In various embodiments, the input devices of block 1130 have associated respective input processing elements as known in the art. For example, the RF section may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or bandlimiting a signal to a band of frequencies), (ii) downconverting the selected signal, (iii) bandlimiting again to a narrow band of frequencies to select (for example) a signal frequency band, which in particular embodiments may be referred to as a channel, (iv) demodulating the downconverted, bandlimited signal, (v) performing error correction, and (vi) demultiplexing to select a desired stream of data packets. The RF section of various embodiments includes one or more elements (e.g., a frequency selector, a signal selector, a bandlimiter, a channel selector, a filter, a downconverter, a demodulator, an error corrector, and a demultiplexer) to perform these functions. The RF section may include, for example, a tuner to perform various of these functions, including downconverting a received signal to a lower frequency (e.g., an intermediate frequency or near-baseband frequency) or to baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted over a wired (e.g., cable) medium and perform frequency selection by filtering, downconverting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements (e.g., inserting amplifiers and analog-to-digital converters, etc.). In various embodiments, the RF section includes an antenna.

[0081] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting system 1000 to other electronic devices over USB and / or HDMI connections. It should be understood that various aspects of the input processing (e.g., Reed-Solomon error correction) may be implemented, for example, within a separate input processing IC or within processor 1010, as desired. Similarly, aspects of the USB or HDMI interface processing may be realized within a separate interface IC or within processor 1010, as desired. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements (including, for example, encoder / decoder 1030 operating in combination with processor 1010 and memory and storage elements) to process the data stream as needed for presentation on an output device.

[0082] The various elements of system 1000 may be provided within a unified housing, where the various elements may be interconnected and data may be transmitted therebetween using suitable connection arrangements (e.g., internal buses known in the art, including Inter-IC (I2C) bus, wires, and printed circuit boards).

[0083] System 1000 includes a communication interface 1050 that enables communication with other devices over a communication channel 1060. Communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data over communication channel 1060. Communication interface 1050 may include, but is not limited to, a modem or a network card, and communication channel 1060 may be implemented within a wired and / or wireless medium, for example.

[0084] Data is streamed or otherwise provided to system 1000 in various embodiments by using a wireless network such as a Wi-Fi network (e.g., IEEE 802.11, where IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal in these embodiments is received over communication channel 1060 and communication interface 1050 adapted for Wi-Fi communication. Communication channel 1060 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, enabling streaming applications and other over-the-top communications. Other embodiments provide streamed data to system 1000 by using a set-top box that delivers data over an HDMI connection in input block 1130. Still other embodiments provide streamed data to system 1000 by using an RF connection in input block 1130. As noted above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi (e.g., cellular networks, Bluetooth networks).

[0085] System 1000 may provide output signals to various output devices, including a display 1100, speakers 1110, and other peripheral devices 1120. Display 1100 in various embodiments includes, for example, one or more of a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. Display 1100 may be for a television, a tablet, a laptop, a cellular phone (mobile phone), or other device. Display 1100 may also be integrated with other components (e.g., components in a smartphone) or may be separate (e.g., an external monitor for a laptop). Other peripheral devices 1120, in various example embodiments, include one or more of a standalone digital video disc (or digital versatile disc) (both referred to as a DVR), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 1120 to provide functionality based on the output of system 1000. For example, a disc player functions to play the output of the system 1000 .

[0086] In various embodiments, control signals are communicated between system 1000 and display 1100, speaker 1110, or other peripheral device 1120 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable device-to-device control with or without user intervention. Output devices may be communicatively coupled to system 1000 through dedicated connections via respective interfaces 1070, 1080, and 1090. Alternatively, output devices may be connected to system 1000 using communication channel 1060 via communication interface 1050. Display 1100 and speaker 1110 may be integrated into a single unit with other components of system 1000 within an electronic device such as a television. In various embodiments, display interface 1070 includes a display driver, such as a timing controller (T Con) chip.

[0087] The display 1100 and speakers 1110 may alternatively be separate from one or more of the other components, for example if the RF portion of the input block 1130 is part of a separate set-top box. In various embodiments where the display 1100 and speakers 1110 are external components, the output signal may be provided via a dedicated output connection including, for example, an HDMI port, a USB port, or a COMP output.

[0088] Some embodiments may be implemented by the processor 1010, by computer software implemented in hardware, or by a combination of hardware and software. By way of non-limiting example, embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type appropriate to the technical environment and may be implemented using any suitable data storage technology, including, by way of non-limiting example, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor 1010 may be of any type appropriate to the technical environment and may include, by way of non-limiting example, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0089] Various implementations involve decoding. As used herein, "decoding" may encompass all or part of the processing performed on a received encoded sequence, e.g., to generate a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder (e.g., entropy decoding, inverse quantization, inverse transform, and differential decoding). In various embodiments, such processes also, or alternatively, include processes performed by the decoders of various embodiments described herein.

[0090] As another example, in one embodiment "decoding" refers to entropy decoding only, in another embodiment "decoding" refers to differential decoding only, and in another embodiment "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or generally to the broader decoding process will be clear based on the context of the particular description and, therefore, is believed to be well understood by one of ordinary skill in the art.

[0091] Various implementations involve encoding. In a manner similar to the above discussion regarding "decoding," "encoding," as used herein, may encompass all or some of the processes performed on, for example, an input video sequence to generate an encoded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder (e.g., segmentation, differential encoding, transform, quantization, and entropy coding). In various embodiments, such processes also, or alternatively, include processes performed by the encoders of various implementations described herein.

[0092] As another example, in one embodiment, "encoding" refers to entropy encoding only, in another embodiment, "encoding" refers to differential encoding only, and in another embodiment, "encoding" refers to a combination of differential and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or generally to the broader encoding process will be clear based on the context of the particular description and, therefore, is believed to be well understood by one of ordinary skill in the art.

[0093] It should be noted that the syntax elements used herein are descriptive terms, and therefore do not preclude the use of other syntax element names.

[0094] Where a diagram is presented as a flow diagram, it should be understood that the diagram also provides a block diagram of the corresponding apparatus. Similarly, where a diagram is presented as a block diagram, it should be understood that the diagram also provides a flow diagram of the corresponding method / process.

[0095] Various embodiments may refer to parametric models or rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between rate and distortion is typically considered, often given a computational complexity constraint. This may be measured via a rate-distortion optimization (RDO) metric, or via least mean squares (LMS), mean of absolute errors (MAE), or other such measures. Rate-distortion optimization is typically formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are various approaches to solving the rate-distortion optimization problem. For example, some approaches may be based on extensive testing of all coding options (including all modes or coding parameter values ​​considered) with a thorough evaluation of their coding cost and associated distortion of the reconstructed signal after coding and decoding. Faster approaches may also be used to reduce coding complexity, particularly by calculating approximate distortion based on prediction or prediction residual signals (rather than the reconstructed signal). A hybrid of these two approaches may also be used, such as by using approximate distortion for only some of the possible encoding choices and the full distortion for others. Other approaches only evaluate a subset of the possible encoding choices. More generally, many approaches employ any of a variety of techniques to perform optimization, but the optimization is not necessarily a complete evaluation of both the coding cost and the associated distortion.

[0096] Implementations and aspects described herein may be embodied in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even if discussed in the context of only one type of implementation (e.g., discussed only as a method), implementation of the discussed features may also be embodied in other forms (e.g., in an apparatus or program). An apparatus may be realized in, for example, appropriate hardware, software, and firmware. The method may be implemented in, for example, a processor, which refers to processing devices in general (including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device). Processors also include, for example, communication devices such as computers, cellular phones, portable / personal digital assistants (PDAs), and other devices that facilitate the transfer of information between end users.

[0097] References to "one embodiment" or "embodiment" or "one implementation" or "implementation" and other variations mean that the particular features, structures, characteristics, etc. described in connection with an embodiment are included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" in various places throughout this application, and in any other variations, are not necessarily all referring to the same embodiment.

[0098] Additionally, the application may refer to "determining" various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from memory.

[0099] Additionally, the application may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, replicating information, calculating information, determining information, predicting information, or estimating information.

[0100] Additionally, the application may refer to "receiving" various information. Receiving is intended to be broad, similar to "accessing." Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from memory). Furthermore, "receiving" typically involves in some way an operation, such as, for example, storing information, processing information, transmitting information, moving information, duplicating information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0101] For example, in the cases of "A / B," "A and / or B," and "at least one of A and B," it should be understood that the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B). As another example, in the cases of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), or the selection of only the first and third listed alternatives (A and C), or the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A, B, and C). This can be extended to as many items as are listed, as will be apparent to those skilled in the art and pertinent arts.

[0102] Also, as used herein, the word "signaling" refers to, among other things, indicating something to a corresponding decoder. For example, in some embodiments, an encoder signals a particular one of multiple transforms, coding modes, or flags. Thus, in one embodiment, the same transform, parameter, or mode is used at both the encoder and decoder sides. Thus, for example, an encoder may transmit a particular parameter to a decoder so that the decoder can use the same parameter (explicit signaling). Conversely, if the decoder already has a particular parameter and other parameters, signaling may be used without transmission simply to enable the decoder to know and select the particular parameter (implicit signaling). By avoiding transmission of any actual function, bit savings are realized in various embodiments. It should be recognized that signaling can be achieved in various ways. For example, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder in various embodiments. Although the above relates to the verb form of the word "signal," the word "signal" may also be used herein as a noun.

[0103] As will be apparent to those skilled in the art, some implementations may generate a variety of signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for performing a method or data generated by one of the described embodiments. For example, the signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using a high frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information the signal carries may be, for example, analog or digital information. The signal may be transmitted over a variety of wired or wireless links, as is known. The signal may be stored on a processor-readable medium.

[0104] We have described many embodiments across various claim categories and types. Features of these embodiments may be provided alone or in any combination. Further, some embodiments may include one or more of the following features, devices, or aspects across various claim categories and types, alone or in any combination: A process or device for encoding or decoding video data by using a low frequency non-separable transform with multiple transform selection. A process or device for encoding or decoding video data by using low matrix based intra prediction with multiple transform selection. A process or device for encoding or decoding video data by using a low frequency non-separable transform with implicit multiple transform selection. A process or device for encoding or decoding video data by using low matrix based intra prediction with implicit multiple transform selection. - A bitstream or signal containing one or more of the described syntax elements or variations thereof. A bitstream or signal containing syntax conveying information generated according to any of the described embodiments. - Generating and / or transmitting and / or receiving and / or decoding according to any of the described embodiments. - A medium storing a method, process, apparatus, instructions, data or signal according to any of the described embodiments. Inserting signaling syntax elements that allow the decoder to determine the coding mode in a manner that corresponds to the one used by the encoder. - Generating and / or transmitting and / or receiving and / or decoding a bitstream or signal comprising one or more of the described syntax elements or variations thereof. A TV, set-top box, cellular phone, tablet or other electronic device that performs the conversion method according to any of the described embodiments. ● A TV, set-top box, cellular phone, tablet, or other electronic device that performs a conversion method determination according to any of the described embodiments and displays the resulting image (e.g., using a monitor, screen, or other type of display). ●A TV, set-top box, cellular phone, tablet or other electronic device that selects, band-limits or tunes (e.g., by using a tuner) a channel and performs a conversion method to receive a signal containing an encoded image according to any of the described embodiments. • A TV, set-top box, cellular phone, tablet or other electronic device that receives the signal containing the coded image wirelessly (for example, by using an antenna) and performs the conversion method.

Claims

1. decoding an index indicating whether a low frequency non-separable transform is used and decoding a flag indicating whether matrix-based intra prediction is used; selecting a vertical transform and a horizontal transform based on the index and the flag; decoding at least one video block using the selected vertical and horizontal transforms; and A method comprising:

2. 2. The method of claim 1 , further comprising: obtaining syntax information from a bitstream that indicates use of an implicit multiple transform selection mode; and selecting the vertical transform and the horizontal transform based on the index and the flag comprises implicitly selecting the vertical transform and the horizontal transform based on the index and the flag.

3. 3. The method of claim 1, wherein if the index indicates that a low-frequency separable transform is not used and the flag indicates that a matrix-based intra prediction mode is not used, selecting the vertical transform and the horizontal transform comprises selecting the vertical transform and the horizontal transform based on a size of a transform block.

4. 4. The method of claim 3, wherein selecting the vertical transform and the horizontal transform based on the size of the transform block comprises selecting a type 7 horizontal discrete sine transform if the width of the transform block is greater than 4 and less than 16, and selecting a type 2 horizontal discrete cosine transform otherwise.

5. 4. The method of claim 3, wherein selecting the vertical transform and the horizontal transform based on the size of the transform block comprises selecting a type 7 vertical discrete sine transform if the height of the transform block is greater than 4 and less than 16, and selecting a type 2 vertical discrete cosine transform otherwise.

6. 3. The method of claim 1, wherein if the index indicates that a low-frequency separable transform is used or the flag indicates that a matrix-based intra prediction mode is used, selecting the vertical transform and the horizontal transform comprises selecting a type 2 horizontal discrete cosine transform and a type 2 vertical discrete cosine transform.

7. determining an index indicating whether a low frequency non-separable transform is used and determining a flag indicating whether matrix-based intra prediction is used; selecting a vertical transform and a horizontal transform based on the index and the flag; encoding the index and the flag; encoding at least one video block using the selected vertical and horizontal transforms; and A method comprising:

8. 8. The method of claim 7, further comprising: encoding syntax information into a bitstream indicating use of an implicit multiple transform selection mode, wherein selecting the vertical transform and the horizontal transform based on the index and the flag comprises implicitly selecting the vertical transform and the horizontal transform based on the index and the flag.

9. 9. The method of claim 7, wherein if the index indicates that a low-frequency separable transform is not used and the flag indicates that a matrix-based intra prediction mode is not used, selecting the vertical transform and the horizontal transform comprises selecting the vertical transform and the horizontal transform based on a size of a transform block.

10. 10. The method of claim 9, wherein selecting the vertical transform and the horizontal transform based on the size of the transform block comprises selecting a type 7 horizontal discrete sine transform if the width of the transform block is greater than 4 and less than 16, and selecting a type 2 horizontal discrete cosine transform otherwise.

11. 10. The method of claim 9, wherein selecting the vertical transform and the horizontal transform based on the size of the transform block comprises selecting a type 7 vertical discrete sine transform if the height of the transform block is greater than 4 and less than 16, and selecting a type 2 vertical discrete cosine transform otherwise.

12. 9. The method of claim 7 or 8, wherein if the index indicates that a low-frequency separable transform is used or the flag indicates that a matrix-based intra prediction mode is used, selecting the vertical transform and the horizontal transform comprises selecting a type 2 horizontal discrete cosine transform and a type 2 vertical discrete cosine transform.

13. decoding an index indicating whether a low frequency non-separable transform is used and decoding a flag indicating whether matrix-based intra prediction is used; selecting a vertical transform and a horizontal transform based on the index and the flag; and decoding at least one video block using the selected vertical and horizontal transforms.

14. 14. The apparatus of claim 13, further comprising: obtaining syntax information from a bitstream that indicates use of an implicit multiple transform selection mode; and selecting the vertical transform and the horizontal transform based on the index and the flag comprises implicitly selecting the vertical transform and the horizontal transform based on the index and the flag.

15. 15. The apparatus of claim 13, wherein if the index indicates that a low-frequency non-separable transform is not used and the flag indicates that a matrix-based intra prediction mode is not used, selecting the vertical transform and the horizontal transform comprises selecting the vertical transform and the horizontal transform based on a size of a transform block.

16. 16. The apparatus of claim 15, wherein selecting the vertical transform and the horizontal transform based on the size of the transform block comprises selecting a type 7 horizontal discrete sine transform if the width of the transform block is greater than 4 and less than 16, and selecting a type 2 horizontal discrete cosine transform otherwise.

17. 16. The apparatus of claim 15, wherein selecting the vertical transform and the horizontal transform based on the size of the transform block comprises selecting a type 7 vertical discrete sine transform if the height of the transform block is greater than 4 and less than 16, and selecting a type 2 vertical discrete cosine transform otherwise.

18. 15. The apparatus of claim 13 or 14, wherein, if the index indicates that a low-frequency separable transform is used or the flag indicates that a matrix-based intra prediction mode is used, selecting the vertical transform and the horizontal transform comprises selecting a type 2 horizontal discrete cosine transform and a type 2 vertical discrete cosine transform.

19. determining an index indicating whether a low frequency non-separable transform is used and determining a flag indicating whether matrix-based intra prediction is used; selecting a vertical transform and a horizontal transform based on the index and the flag; encoding the index and the flag; and encoding at least one video block using the selected vertical and horizontal transforms.

20. 20. The apparatus of claim 19, further comprising: encoding syntax information into a bitstream indicating use of an implicit multiple transform selection mode; and wherein selecting the vertical transform and the horizontal transform based on the index and the flag comprises implicitly selecting the vertical transform and the horizontal transform based on the index and the flag.

21. 21. The apparatus of claim 19 or claim 20, wherein if the index indicates that a low-frequency non-separable transform is not used and the flag indicates that a matrix-based intra prediction mode is not used, selecting the vertical transform and the horizontal transform comprises selecting the vertical transform and the horizontal transform based on a size of a transform block.

22. 22. The apparatus of claim 21, wherein selecting the vertical transform and the horizontal transform based on a size of the transform block comprises selecting a type 7 horizontal discrete sine transform if the width of the transform block is greater than 4 and less than 16, and selecting a type 2 horizontal discrete cosine transform otherwise.

23. 22. The apparatus of claim 21, wherein selecting the vertical transform and the horizontal transform based on a size of the transform block comprises selecting a type 7 vertical discrete sine transform if a height of the transform block is greater than 4 and less than 16, and selecting a type 2 vertical discrete cosine transform otherwise.

24. 21. The apparatus of claim 19 or claim 20, wherein if the index indicates that a low-frequency separable transform is used or the flag indicates that a matrix-based intra prediction mode is used, selecting the vertical transform and the horizontal transform comprises selecting a type 2 horizontal discrete cosine transform and a type 2 vertical discrete cosine transform.

Images