Video encoding and decoding method, device and storage medium

Abstract

Aspects of the present disclosure provide a method, apparatus, and storage medium for video decoding. Processing circuitry can decode encoding information for a block from an encoded video stream. The encoding information can indicate an intra-frame prediction mode for the block, as well as one or a combination of transform partition information, block size, and block shape for the block. The processing circuitry can determine whether to disable secondary transform for the block based on one or a combination of the transform partition information, block size, and block shape. The processing circuitry can reconstruct the block based on whether secondary transform is disabled for the block.

Description

[0001] Incorporation by reference

[0002] This application claims priority to U.S. Patent Application No. 17 / 361,239, filed on June 28, 2021, for “Method and apparatus for video coding,” which claims priority to U.S. Provisional Application No. 63 / 112,533, filed on November 11, 2020, for “Methods for efficient application of secondary transforms.” The entire disclosure of each of the prior applications is hereby incorporated by reference in its entirety. Technical Field

[0003] This disclosure describes embodiments generally related to video coding. Background Art

[0004] The background description provided herein is intended to generally present the context of the present disclosure. The work of the presently named inventors is neither explicitly nor impliedly admitted to be prior art to the present disclosure to the extent described in this background section nor to any aspects of the description that may not constitute prior art at the time of filing.

[0005] Video encoding and decoding can be performed using inter-frame prediction with motion compensation. Uncompressed digital video can include a series of pictures, each picture having spatial dimensions of, for example, 1920×1080 luma samples and associated chroma samples. The series of pictures can have a fixed or variable picture rate (also informally referred to as a frame rate), such as 60 pictures per second or 60 Hz. Uncompressed video has specific bit rate requirements. For example, 1080p60 4:2:0 video (1920×1080 luma sample resolution at a 60 Hz frame rate) with 8 bits per sample requires close to 1.5 Gbit / s of bandwidth. One hour of such video requires over 600 GB of storage space.

[0006] One purpose of video encoding and decoding can be to reduce redundancy in the input video signal through compression. Compression can help reduce the bandwidth and / or storage space requirements mentioned above, in some cases by two orders of magnitude or more. Lossless compression and lossy compression, as well as combinations thereof, can be employed. Lossless compression refers to techniques that can reconstruct an exact replica of the original signal from the compressed original signal. When lossy compression is used, the reconstructed signal may not be the same as the original signal, but the distortion between the original and the reconstructed signal is small enough that the reconstructed signal can be used for the intended application. In the case of video, lossy compression is widely employed. The amount of distortion allowed depends on the application; for example, users of certain consumer streaming applications may tolerate higher distortion than users of television distribution applications. The achievable compression ratio can reflect that higher allowable / tolerable distortion can produce higher compression ratios.

[0007] Video encoders and decoders may utilize techniques from several broad categories including, for example, motion compensation, transforms, quantization, and entropy coding and decoding.

[0008] Video codec techniques may include techniques known as intra-frame coding. In intra-frame coding, sample values are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, pictures are spatially subdivided into blocks of samples. When all sample blocks are encoded in intra-frame mode, the picture may be an intra-frame picture. Intra-frame pictures and their derivatives (such as independent decoder refresh pictures) can be used to reset the decoder state and can therefore be used as the first picture in an encoded video stream and video session, or as a still image. The samples of the intra-frame block can be exposed to a transform, and the transform coefficients can be quantized before entropy coding. Intra-frame prediction can be a technique for minimizing sample values in the pre-transform domain. In some cases, the smaller the DC value after the transform and the smaller the AC coefficient, the fewer bits are required to represent the block after entropy coding at a given quantization step size.

[0009] Conventional intra-frame codecs, such as those known from, for example, the MPEG-2 generation of codecs, do not use intra-frame prediction. However, some newer video compression techniques include techniques for predicting a data block from, for example, surrounding sample data and / or metadata that was obtained during encoding and / or decoding of spatially adjacent data blocks and that precedes the data block in decoding order. Such techniques are hereinafter referred to as "intra-frame prediction" techniques. Note that, in at least some cases, intra-frame prediction uses only reference data from the current picture being reconstructed, and not reference data from reference pictures.

[0010] There are many different forms of intra-frame prediction. When more than one such technique can be used in a given video codec, the techniques used can be coded in an intra-frame prediction mode. In some cases, a mode can have sub-modes and / or parameters, and these sub-modes and / or parameters can be coded separately or included in a mode codeword. Such codewords for a given mode, sub-mode and / or parameter combination can affect the codec efficiency gain through intra-frame prediction, and therefore can affect the entropy coding and decoding technology used to convert the codeword into a bitstream.

[0011] A certain mode of intra prediction was introduced with H.264, improved in H.265, and further improved in newer codecs such as the Joint Exploration Model (JEM), Versatile Video Codec (VVC), and Benchmark Set (BMS). A predictor block can be formed using the values of neighboring samples belonging to already available samples. Sample values of neighboring samples are copied into the predictor block according to the direction. A reference to the direction used can be encoded in the bitstream or can be predicted itself.

[0012] refer to Figure 1A , depicted in the lower right corner are nine known predictor directions from the 33 possible predictor directions of H.265 (corresponding to the 33 angular modes of the 35 intra modes). The point where the arrows converge (101) represents the sample being predicted. The arrows represent the direction in which the sample is being predicted. For example, arrow (102) indicates that sample (101) is predicted from one or more samples to the upper right at an angle of 45 degrees to the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more samples to the lower left of sample (101) at an angle of 22.5 degrees to the horizontal.

[0013] Still refer to Figure 1A , a square block (104) of 4×4 samples is depicted at the top left (indicated by the dashed bold line). The square block (104) includes 16 samples, each sample labeled with "S", its position in the Y dimension (e.g., row index), and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from the top) and the first sample in the X dimension (from the left). Similarly, sample S44 is the fourth sample in the Y and X dimensions in the block (104). Since the size of the block is 4×4 samples, S44 is at the bottom right. Reference samples are further shown that follow a similar numbering scheme. The reference samples are labeled with R, their Y position (e.g., row index), and X position (column index) relative to the block (104). In H.264 and H.265, the prediction samples are adjacent to the block in the reconstruction; therefore, there is no need to use negative values.

[0014] Intra-picture prediction can work by copying reference sample values from neighboring samples as appropriate by the signaled prediction direction. For example, assume that the encoded video stream includes signaling that indicates, for this block, a prediction direction consistent with arrow (102)—that is, predicting samples from one or more prediction samples to the upper right at a 45-degree angle to the horizontal. In this case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Sample S44 is then predicted from reference sample R08.

[0015] In some cases, the values of multiple reference samples may be combined, for example by interpolation, in order to calculate the reference sample; in particular when the direction is not divisible by 45 degrees.

[0016] As video codec technology has developed, the number of possible directions has increased. In H.264 (2003), nine different directions could be represented. This was increased to 33 in H.265 (2013), and JEM / VVC / BMS can support up to 65 directions when they are disclosed. Experiments have been conducted to identify the most likely directions, and certain techniques in entropy codecs have been used to represent those possible directions with a small number of bits, accepting some penalty for less likely directions. Further, the direction itself can sometimes be predicted based on adjacent directions used in adjacent decoded blocks.

[0017] Figure 1B A schematic diagram (180) depicting 65 intra prediction directions according to the JEM is shown to illustrate that the number of prediction directions increases over time.

[0018] The mapping of intra-frame prediction direction bits in the coded video stream, which may represent directions that differ from those of the video codec, to the video codec; and can be, for example, simple direct mappings from prediction directions to intra-frame prediction modes, to codewords, to complex adaptive schemes involving most probable modes, and similar techniques. However, in all cases, there may be some directions in the video content that are statistically less likely to occur than some other directions. Since the goal of video compression is to reduce redundancy, in a well-functioning video codec, those unlikely directions will be represented by a larger number of bits than more likely directions.

[0019] Motion compensation can be a lossy compression technique and can involve a technique in which a block of sample data from a previously reconstructed picture or portion thereof (reference picture) is used to predict a newly reconstructed picture or picture portion after being spatially shifted in a direction indicated by a motion vector (hereinafter MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions, X and Y, or three dimensions, the third being an indication of the reference picture in use (the latter may indirectly be a temporal dimension).

[0020] In some video compression techniques, the MV applicable to a region of sample data can be predicted from other MVs, such as those associated with another region of sample data that is spatially adjacent to the region being reconstructed and precedes the MV in decoding order. This can substantially reduce the amount of data required to encode and decode the MV, thereby eliminating redundancy and increasing compression. MV prediction works effectively, for example, because when encoding and decoding an input video signal derived from a camera (referred to as natural video), there is a statistical likelihood that regions larger than the region to which a single MV applies will move in similar directions. Therefore, in some cases, similar motion vectors derived from MVs in neighboring regions can be used for prediction. This results in the MV found for a given region being similar or identical to the MV predicted from surrounding MVs, which, after entropy encoding and decoding, can be represented using fewer bits than would be used if the MV were encoded directly. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., an MV) derived from the original signal (i.e., a sample stream). In other cases, MV prediction itself can be lossy, for example due to rounding errors when computing the predicted value from several surrounding MVs.

[0021] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Recommendation H.265, "High Efficiency Video Coding," December 2016). Among the various MV prediction mechanisms provided by H.265, this application describes a technique referred to below as "spatial merging."

[0022] Referring to Figure 2, the current block (201) includes samples that have been discovered by the encoder during the motion search process, and these samples can be predicted based on a previous block of the same size that has been spatially offset. Alternatively, rather than encoding the MV directly, the MV can be derived from metadata associated with one or more reference pictures. For example, the MV associated with any of the five surrounding samples A0, A1 and B0, B1, B2 (202 to 206, respectively) can be used to derive the MV from metadata of the nearest reference picture (in decoding order). In H.265, MV prediction can use prediction values from the same reference picture that is also being used by neighboring blocks. Summary of the Invention

[0023] Various aspects of the present disclosure provide methods and apparatus for video encoding and / or decoding. In some examples, an apparatus for video decoding includes a processing circuit. The processing circuit can decode coding information for a block from an encoded video stream. The coding information can indicate an intra-frame prediction mode for the block, as well as one or a combination of transform partition information, block size, and block shape for the block. The processing circuit can determine whether to disable secondary transform for the block based on one or a combination of the transform partition information, block size, and block shape for the block. The processing circuit can reconstruct the block based on whether secondary transform is disabled for the block.

[0024] In an embodiment, one or a combination of the block's transform partition information, the block's size, and the block's shape includes the block's transform partition information signaled in the coded video bitstream. The block's transform partition information may indicate the block's partition depth. The processing circuitry may partition the block into a plurality of transform blocks. The processing circuitry may determine whether to disable secondary transform for the block based on the partition depth. In an example, the processing circuitry determines to disable secondary transform for the block in response to the partition depth being greater than a threshold and does not signal a secondary transform index, where the threshold is 0 or a positive integer. The secondary transform index may indicate a secondary transform kernel to be applied to the block. In an example, the threshold is 0.

[0025] In an embodiment, one or a combination of the transform partition information of the block, the size of the block, and the shape of the block includes the transform partition information of the block and the shape of the block, wherein the transform partition information of the block is signaled in the coded video bitstream. The transform partition information may indicate a partition depth of the block, and the shape of the block may be a non-square rectangle. The processing circuit may partition the block into a plurality of transform blocks. The processing circuit may determine whether to disable secondary transform for the block based on the partition depth. In an example, in response to the partition depth being greater than a threshold, the processing circuit determines to disable secondary transform for the block, where the threshold is 0 or a positive integer.

[0026] In an embodiment, one or a combination of the transform partition information of the block, the size of the block, and the shape of the block includes the shape of the block indicated by the aspect ratio of the block. The processing circuitry may determine whether to disable the secondary transform for the block based on the aspect ratio of the block. In an example, the aspect ratio of the block is a ratio of a first dimension of the block to a second dimension of the block, where the first dimension of the block is greater than or equal to the second dimension. The processing circuitry may determine to disable the secondary transform for the block in response to the aspect ratio of the block being greater than a threshold.

[0027] In an embodiment, one or a combination of transform partition information, block size, and block shape of the block includes transform partition information and block shape, wherein the transform partition information may indicate a partition depth, and the block shape is a square. The processing circuitry may partition the block into a plurality of transform blocks. The processing circuitry may determine whether to disable secondary transform for the block based on the partition depth. In an example, the processing circuitry determines to disable secondary transform for the block in response to the partition depth being greater than a threshold, wherein the threshold may be 0 or a positive integer.

[0028] In an embodiment, one or a combination of the transform partition information of the block, the size of the block, and the shape of the block includes the transform partition information of the block and the size of the block. The transform partition information may indicate a partition depth of the block, and the size of the block may indicate a width of the block and a height of the block, wherein the width and height are greater than a threshold size. The processing circuitry may partition the block into a plurality of transform blocks. The processing circuitry may determine whether to disable a secondary transform for the block based on the partition depth of the block. In an example, in response to the partition depth being greater than a threshold, the processing circuitry determines to disable a secondary transform for the block. The threshold may be zero or a positive integer.

[0029] In an embodiment, one of a width W' of the other block and a height H' of the other block is greater than a maximum transform size T. The processing circuitry may partition the other block into a plurality of sub-blocks including the block. The width W of the block may be the minimum of W' and T, and the height H of the block may be the minimum of H' and T. One or a combination of the transform partition information of the block, the size of the block, and the shape of the block may include the transform partition information of the block. The transform partition information may indicate a partition depth of the block. The processing circuitry may determine to disable secondary transform for the block in response to the partition depth of the block being greater than a threshold.

[0030] In an embodiment, one of a width W' of the other block and a height H' of the other block is greater than a predetermined constant K. The processing circuitry may divide the other block into a plurality of sub-blocks including the block. The width W of the block may be the minimum of W' and K, and the height H of the block may be the minimum of H' and K. One or a combination of the transform partition information of the block, the size of the block, and the shape of the block may include the size of the block, the size of the block being W and H. The processing circuitry may determine that a secondary transform is enabled for the block in response to the size of the block being W and H.

[0031] Aspects of the present disclosure also provide non-transitory computer-readable media storing instructions that, when executed by a computer for video decoding, cause the computer to perform methods for video decoding and / or encoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Additional features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0033] Figure 1A is a diagram illustrating an exemplary subset of intra prediction modes.

[0034] Figure 1B is a diagram of exemplary intra prediction directions.

[0035] FIG2 is a schematic diagram of a current block and its surrounding spatial merging candidates in an example.

[0036] Figure 3 is a schematic diagram of a simplified block diagram of a communication system (300) according to an embodiment.

[0037] Figure 4 is a schematic diagram of a simplified block diagram of a communication system (400) according to an embodiment.

[0038] Figure 5 is a schematic diagram of a simplified block diagram of a decoder according to an embodiment.

[0039] Figure 6 is a schematic diagram of a simplified block diagram of an encoder according to an embodiment.

[0040] Figure 7 A block diagram of an encoder according to another embodiment is shown.

[0041] Figure 8 A block diagram of a decoder according to another embodiment is shown.

[0042] Figure 9 An example of a nominal mode of a coding block according to an embodiment of the present disclosure is shown.

[0043] Figure 10 An example of non-directional smooth intra prediction according to aspects of the present disclosure is shown.

[0044] Figure 11 An example of an intra predictor based on recursive filtering according to an embodiment of the present disclosure is shown.

[0045] Figure 12 An example of multiple reference lines for encoding a block according to an embodiment of the present disclosure is shown.

[0046] Figure 13 An example of transform block partitioning on a block according to an embodiment of the present disclosure is shown.

[0047] Figure 14 An example of transform block partitioning on a block according to an embodiment of the present disclosure is shown.

[0048] Figure 15 An example of a main transform basis function according to an embodiment of the present disclosure is shown.

[0049] Figure 16AAn exemplary dependency of the availability of various transform kernels based on transform block size and prediction mode according to an embodiment of the present disclosure is shown.

[0050] Figure 16B An exemplary transform type selection based on intra prediction mode according to an embodiment of the present disclosure is shown.

[0051] Figure 16C An example of a generalized line graph transform (LGT) characterized by self-loop weights and edge weights according to an embodiment of the present disclosure is shown.

[0052] Figure 16D An exemplary generalized graph Laplacian (GGL) matrix according to an embodiment of the present disclosure is shown.

[0053] Figures 17 and 18 Examples of two transform encoding and decoding processes ( 1700 ) and ( 1800 ) using 16×64 transform and 16×48 transform, respectively, according to an embodiment of the present disclosure are shown.

[0054] Figure 19 A flow chart outlining a process (1900) according to an embodiment of the present disclosure is shown.

[0055] Figure 20 is a schematic diagram of a computer system according to an embodiment. DETAILED DESCRIPTION

[0056] Figure 3 3 is a simplified block diagram of a communication system (300) according to an embodiment disclosed herein. The communication system (300) includes a plurality of terminal devices that can communicate with each other via, for example, a network (350). For example, the communication system (300) includes a first terminal device (310) and a second terminal device (320) interconnected via the network (350). Figure 3 In an embodiment, the first terminal device (310) and the second terminal device (320) perform unidirectional data transmission. For example, the first terminal device (310) may encode video data (e.g., a video picture stream collected by the terminal device (310)) for transmission to the second terminal device (320) via the network (350). The encoded video data is transmitted in the form of one or more encoded video streams. The second terminal device (320) may receive the encoded video data from the network (350), decode the encoded video data to restore the video data, and display the video picture based on the restored video data. Unidirectional data transmission is more common in applications such as media services.

[0057] In another embodiment, a communication system (300) includes a third terminal device (330) and a fourth terminal device (340) that perform bidirectional transmission of encoded video data, which can occur, for example, during a video conference. For bidirectional data transmission, each of the third terminal device (330) and the fourth terminal device (340) can encode video data (e.g., a video picture stream collected by the terminal device) for transmission to the other of the third terminal device (330) and the fourth terminal device (340) via a network (350). Each of the third terminal device (330) and the fourth terminal device (340) can also receive the encoded video data transmitted by the other of the third terminal device (330) and the fourth terminal device (340), and can decode the encoded video data to restore the video data, and can display the video picture on an accessible display device based on the restored video data.

[0058] exist Figure 3 In an embodiment of the present invention, the terminal devices (310), (320), (330) and (340) may be servers, personal computers and smart phones, but the principles disclosed in this application may not be limited thereto. The embodiments disclosed in this application are applicable to laptop computers, tablet computers, media players and / or dedicated video conferencing equipment. Network (350) represents any number of networks that transmit encoded video data between terminal devices (310), (320), (330) and (340), including, for example, wired (wired) and / or wireless communication networks. The communication network (350) may exchange data in circuit switching and / or packet switching channels. The network may include a telecommunications network, a local area network, a wide area network and / or the Internet. For the purposes of this application, unless explained below, the architecture and topology of the network (350) may be irrelevant to the operation disclosed in this application.

[0059] As an example, Figure 4 The video encoder and video decoder are shown in a streaming environment. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0060] The streaming system may include an acquisition subsystem (413), which may include a video source (401) such as a digital camera, which creates an uncompressed video picture stream (402). In an embodiment, the video picture stream (402) includes samples captured by the digital camera. The video picture stream (402) is depicted as a thick line to emphasize the high data volume of the video picture stream compared to the encoded video data (404) (or encoded video code stream), and the video picture stream (402) can be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination of hardware and software to implement or implement various aspects of the disclosed subject matter as described in more detail below. Compared to the video picture stream (402), the encoded video data (404) (or the encoded video code stream (404)) is depicted as a thin line to emphasize the lower amount of data of the encoded video data (404) (or the encoded video code stream (404)), which can be stored on the streaming server (405) for future use. One or more streaming client subsystems, such as Figure 3 The client subsystem (406) and the client subsystem (408) in the streaming server (405) can access the streaming server (405) to retrieve the copy (407) and the copy (409) of the encoded video data (404). The client subsystem (406) can include, for example, a video decoder (410) in the electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and produces an output video picture stream (411) that can be presented on a display (412) (e.g., a display screen) or another presentation device (not depicted). In some streaming systems, the encoded video data (404), the video data (407), and the video data (409) (e.g., a video bitstream) can be encoded according to certain video encoding / compression standards. Examples of such standards include ITU-T H.265. In an embodiment, the video coding standard under development is informally referred to as Versatile Video Coding (VVC), and the present application can be used in the context of the VVC standard.

[0061] It should be noted that the electronic device (420) and the electronic device (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0062] Figure 51 is a block diagram of a video decoder (510) according to an embodiment disclosed in the present application. The video decoder (510) may be provided in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used to replace Figure 4 A video decoder (410) of an embodiment.

[0063] A receiver (531) may receive one or more encoded video sequences to be decoded by a video decoder (510); in the same or another embodiment, one encoded video sequence is received at a time, wherein each encoded video sequence is decoded independently of the other encoded video sequences. The encoded video sequence may be received from a channel (501), which may be a hardware / software link to a storage device storing the encoded video data. The receiver (531) may receive the encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams, which may be forwarded to their respective consuming entities (not shown). The receiver (531) may separate the encoded video sequence from the other data. To prevent network jitter, a buffer memory (515) may be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) is part of the video decoder (510). In other cases, the buffer memory (515) may be provided external to the video decoder (510) (not shown). In other cases, a buffer memory (not shown) is provided external to the video decoder (510) to, for example, mitigate network jitter, and another buffer memory (515) may be provided internally to the video decoder (510) to, for example, handle broadcast timing. Furthermore, when the receiver (531) receives data from a store / forward device with sufficient bandwidth and controllability or from an isochronous network, the buffer memory (515) may not be required, or the buffer memory may be smaller. Of course, for use on a traffic packet network such as the Internet, the buffer memory (515) may also be required. The buffer memory may be relatively large and have an adaptive size, and may be at least partially implemented in an operating system or similar component (not shown) external to the video decoder (510).

[0064] The video decoder (510) may include a parser (520) for reconstructing symbols (521) from the encoded video sequence. The types of symbols include information for managing the operation of the video decoder (510) and potential information for controlling a display device such as a display device (512) (e.g., a display screen) that is not part of the electronic device (530) but can be coupled to the electronic device (530), such as Figure 5As shown in . The control information for the display device may be a parameter set fragment (not shown) of Supplemental Enhancement Information (SEI message) or Video Usability Information (VUI). The parser (520) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may be performed according to a video coding technique or standard and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and the like. The parser (520) may extract a subgroup parameter set for at least one subgroup of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a Group of Pictures (GOP), a picture, a tile, a slice, a macroblock, a Coding Unit (CU), a block, a Transform Unit (TU), a Prediction Unit (PU), and the like. The parser (520) may also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[0065] The parser (520) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515), thereby creating symbols (521).

[0066] Depending on the type of coded video picture or portion of a coded video picture (e.g., inter-frame and intra-frame pictures, inter-frame blocks and intra-frame blocks) and other factors, the reconstruction of the symbol (521) may involve multiple different units. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the coded video sequence by the parser (520). For the sake of brevity, the flow of such subgroup control information between the parser (520) and the multiple units below is not described.

[0067] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually broken down into several functional units as described below. In practical embodiments operating under commercial constraints, many of these units interact closely with each other and may be integrated with each other. However, for the purposes of describing the disclosed subject matter, the conceptual breakdown into the following functional units is appropriate.

[0068] The first unit is a scaler / inverse transform unit (551). The scaler / inverse transform unit (551) receives quantized transform coefficients as symbols (521) from the parser (520) along with control information, including which transform method to use, block size, quantization factor, quantization scaling matrix, etc. The scaler / inverse transform unit (551) may output a block comprising sample values, which may be input to an aggregator (555).

[0069] In some cases, the output samples of the scaler / inverse transform unit (551) may belong to an intra-coded block; that is, a block that does not use predictive information from a previously reconstructed picture, but may use predictive information from a previously reconstructed portion of the current picture. Such predictive information may be provided by the intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) uses reconstructed information extracted from the current picture buffer (558) to generate surrounding blocks of the same size and shape as the block being reconstructed. For example, the current picture buffer (558) buffers partially reconstructed current pictures and / or fully reconstructed current pictures. In some cases, the aggregator (555) adds the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a per-sample basis.

[0070] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to inter-frame coded and potentially motion compensated blocks. In this case, the motion compensated prediction unit (553) may access the reference picture memory (557) to extract samples for prediction. After the extracted samples are motion compensated according to the symbols (521), these samples may be added to the output of the scaler / inverse transform unit (551) (in this case referred to as residual samples or residual signal) by the aggregator (555) to generate output sample information. The retrieval of the prediction samples by the motion compensated prediction unit (553) from the address in the reference picture memory (557) may be controlled by a motion vector, and the motion vector is provided to the motion compensated prediction unit (553) in the form of the symbols (521), which may include, for example, X, Y and reference picture components. Motion compensation may also include interpolation of sample values extracted from the reference picture memory (557) when using sub-sample accurate motion vectors, motion vector prediction mechanisms, etc.

[0071] The output samples of the aggregator (555) may be used by various loop filtering techniques in a loop filter unit (556). The video compression techniques may include in-loop filtering techniques that are controlled by parameters included in the coded video sequence (also referred to as the coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520). However, in other embodiments, the video compression techniques may also be responsive to meta-information obtained during decoding of a coded picture or a previous (in decoding order) portion of the coded video sequence, as well as to previously reconstructed and loop-filtered sample values.

[0072] The output of the loop filter unit (556) may be a sample stream that may be output to a display device (512) and stored in a reference picture memory (557) for subsequent inter-picture prediction.

[0073] Once fully reconstructed, certain coded pictures can be used as reference pictures for future prediction. For example, once the coded picture corresponding to the current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557) and a new current picture buffer can be reallocated before starting to reconstruct a subsequent coded picture.

[0074] The video decoder (510) may perform decoding operations according to a predetermined video compression technique, such as that in the ITU-T H.265 standard. The encoded video sequence may conform to the syntax specified by the video compression technology or standard used, in the sense that the encoded video sequence follows the syntax of the video compression technology or standard and the profile recorded in the video compression technology or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technology or standard as the only tools available for use under the profile. For compliance, the complexity of the encoded video sequence is also required to be within the range defined by the hierarchy of the video compression technology or standard. In some cases, the hierarchy limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (measured in, for example, megasamples per second), maximum reference picture size, etc. In some cases, the limits set by the hierarchy may be further defined by the Hypothetical Reference Decoder (HRD) specification and metadata about the HRD buffer management signaled in the encoded video sequence.

[0075] In an embodiment, a receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be part of the encoded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, and the like.

[0076] Figure 6 1 is a block diagram of a video encoder (603) according to an embodiment disclosed in the present application. The video encoder (603) is provided in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuit). The video encoder (603) can be used to replace Figure 4 A video encoder (403) in an embodiment.

[0077] The video encoder (603) can be used to generate a video from a video source (601) (not Figure 6 In another embodiment, the video source (601) is a part of the electronic device (620) to receive video samples, and the video source can capture video images to be encoded by the video encoder (603). In another embodiment, the video source (601) is a part of the electronic device (620).

[0078] The video source (601) may provide a source video sequence in the form of a stream of digital video samples to be encoded by the video encoder (603), wherein the stream of digital video samples may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 Y CrCB, RGB, etc.), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (601) may be a storage device storing previously prepared videos. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures that are given motion when viewed sequentially. The pictures themselves may be constructed as a spatial array of pixels, where each pixel may include one or more samples depending on the sampling structure, color space, etc. used. The relationship between pixels and samples can be readily understood by those skilled in the art. The following description focuses on samples.

[0079] According to an embodiment, the video encoder (603) may encode and compress pictures of a source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing an appropriate encoding speed is a function of the controller (650). In some embodiments, the controller (650) controls other functional units as described below and is functionally coupled to these units. For the sake of simplicity, the coupling is not shown in the figure. The parameters set by the controller (650) may include rate control related parameters (picture skipping, quantizer, lambda value of rate-distortion optimization technology, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) can be used to have other suitable functions that are related to the video encoder (603) optimized for a certain system design.

[0080] In some embodiments, the video encoder (603) operates in a coding loop. As a simplified description, in some embodiments, the coding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on the input picture to be encoded and the reference pictures) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data (because in the video compression techniques considered in this application, any compression between the symbols and the encoded video code stream is lossless). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream produces bit-accurate results regardless of the decoder location (local or remote), the contents of the reference picture memory (634) are also bit-accurate between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction portion of the encoder are exactly the same sample values that the decoder will "see" when using the prediction during decoding. This basic principle of reference picture synchronization (and the resulting drift when synchronization cannot be maintained, eg due to channel errors) is also used in some related techniques.

[0081] The operation of the "local" decoder (633) can be combined with the operation of Figure 5 The "remote" decoder described in detail for the video decoder (510) is identical. However, additional brief reference is made to Figure 5 , when symbols are available and the entropy encoder (645) and parser (520) are capable of losslessly encoding / decoding the symbols into an encoded video sequence, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and the parser (520), may not be fully implemented in the local decoder (633).

[0082] At this point, it can be observed that any decoder technology other than parsing / entropy decoding present in the decoder must also be present in a substantially identical functional form in the corresponding encoder. For this reason, this application focuses on the decoder operation. The description of the encoder technology can be simplified because the encoder technology is mutually inverse to the decoder technology described comprehensively. A more detailed description is only required in certain areas and is provided below.

[0083] During operation, in some embodiments, the source encoder (630) may perform motion-compensated predictive coding. Motion-compensated predictive coding predictively encodes an input picture with reference to one or more previously encoded pictures in a video sequence designated as "reference pictures." In this manner, the encoding engine (632) encodes the differences between pixel blocks of the input picture and pixel blocks of a reference picture that may be selected as a prediction reference for the input picture.

[0084] The local video decoder (633) may decode the coded video data of a picture that may be designated as a reference picture based on the symbols created by the source encoder (630). The operation of the encoding engine (632) may be a lossy process. When the coded video data is available at the video decoder ( Figure 6 When decoded at a remote location (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process that the video decoder may perform on the reference picture and may cause the reconstructed reference picture to be stored in the reference picture cache (634). In this way, the video encoder (603) may locally store a copy of the reconstructed reference picture that has common content (absent transmission errors) with the reconstructed reference picture that will be obtained by the remote video decoder.

[0085] The predictor (635) may perform a prediction search for the encoding engine (632). That is, for a new picture to be encoded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, etc., that may serve as suitable prediction references for the new picture. The predictor (635) may operate on a pixel-by-pixel-block basis based on sample blocks to find a suitable prediction reference. In some cases, based on the search results obtained by the predictor (635), it may be determined that the input picture may have prediction references taken from multiple reference pictures stored in the reference picture memory (634).

[0086] The controller (650) can manage encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding video data.

[0087] The outputs of all the above functional units may be entropy coded in an entropy encoder (645). The entropy encoder (645) losslessly compresses the symbols generated by the various functional units using techniques such as Huffman coding, variable length coding, arithmetic coding, etc., thereby converting the symbols into a coded video sequence.

[0088] The transmitter (640) can buffer the encoded video sequence created by the entropy encoder (645) in preparation for transmission over a communication channel (660), which can be a hardware / software link to a storage device where the encoded video data will be stored. The transmitter (640) can combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or an auxiliary data stream (source not shown).

[0089] The controller (650) can manage the operation of the video encoder (603). During encoding, the controller (650) can assign a certain coded picture type to each coded picture, but this may affect the coding techniques that can be applied to the corresponding picture. For example, a picture can generally be assigned to any of the following picture types:

[0090] An intra picture (I picture) can be a picture that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are aware of the variations of I pictures and their corresponding applications and features.

[0091] A predictive picture (P picture) may be a picture that can be encoded and decoded using intra prediction or inter prediction, which uses at most one motion vector and a reference index to predict sample values for each block.

[0092] Bidirectionally predictive pictures (B pictures) can be encoded and decoded using intra prediction or inter prediction, which uses up to two motion vectors and reference indices to predict sample values for each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata to reconstruct a single block.

[0093] A source picture is typically spatially subdivided into blocks of samples (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples) and coded block by block. These blocks may be predictively coded with reference to other (already coded) blocks, determined according to the coding allocation applied to the block's corresponding picture. For example, blocks of an I picture may be non-predictively coded, or they may be predictively coded (spatial or intra-predicted) with reference to already coded blocks of the same picture. Pixel blocks of a P picture may be predictively coded using spatial prediction with reference to one previously coded reference picture or using temporal prediction. Blocks of a B picture may be predictively coded using spatial prediction with reference to one or two previously coded reference pictures or using temporal prediction.

[0094] The video encoder (603) may perform encoding operations according to a predetermined video coding technique or standard, such as ITU-T Recommendation H.265. In operation, the video encoder (603) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to the syntax specified by the video coding technique or standard used.

[0095] In an embodiment, the transmitter (640) may transmit additional data along with the encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, redundant pictures and slices, and other forms of redundant data, SEI messages, VUI parameter set fragments, and the like.

[0096] The captured video may be presented as a temporal sequence of multiple source pictures (video pictures). Intra-picture prediction (often shortened to intra prediction) exploits spatial correlation within a given picture, while inter-picture prediction exploits correlation (temporal or otherwise) between pictures. In an embodiment, a particular picture being encoded / decoded is divided into blocks, and the particular picture being encoded / decoded is referred to as the current picture. When a block in the current picture is similar to a reference block in a previously encoded and buffered reference picture in the video, the block in the current picture can be encoded using a vector called a motion vector. The motion vector points to the reference block in a reference picture, and when multiple reference pictures are used, the motion vector may have a third dimension that identifies the reference picture.

[0097] In some embodiments, bidirectional prediction techniques can be used for inter-picture prediction. According to bidirectional prediction techniques, two reference pictures are used, for example, a first reference picture and a second reference picture, both preceding the current picture in the video in decoding order (but potentially in the past and future, respectively, in display order). A block in the current picture can be encoded using a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. Specifically, the block can be predicted using a combination of the first and second reference blocks.

[0098] In addition, merge mode technology can be used in inter-picture prediction to improve coding efficiency.

[0099] According to some embodiments disclosed in the present application, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, according to the HEVC standard, pictures in a video picture sequence are divided into coding tree units (CTUs) for compression, and the CTUs in the pictures have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally speaking, a CTU includes three coding tree blocks (CTBs), which are a luminance CTB and two chrominance CTBs. Furthermore, each CTU can be split into one or more coding units (CUs) using a quadtree. For example, a 64×64 pixel CTU can be split into a 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In an embodiment, each CU is analyzed to determine the prediction type used for the CU, such as an inter prediction type or an intra prediction type. In addition, depending on temporal and / or spatial predictability, the CU is split into one or more prediction units (PUs). Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In an embodiment, the prediction operation in encoding (encoding / decoding) is performed in units of prediction blocks. Taking the luma prediction block as an example, the prediction block includes a matrix of pixel values (e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, and the like.

[0100] Figure 7 is a diagram of a video encoder (703) according to another embodiment disclosed herein. The video encoder (703) is configured to receive a processed block (e.g., a prediction block) of sample values within a current video picture in a sequence of video pictures and to encode the processed block into an encoded picture that is part of an encoded video sequence. In this embodiment, the video encoder (703) is configured to replace Figure 4A video encoder (403) in an embodiment.

[0101] In an HEVC embodiment, a video encoder (703) receives a matrix of sample values for a processing block, such as a prediction block of 8×8 samples. The video encoder (703) uses, for example, rate-distortion (RD) optimization to determine whether to use intra mode, inter mode, or bi-prediction mode to encode the processing block. When encoding the processing block in intra mode, the video encoder (703) may use intra prediction techniques to encode the processing block into an encoded picture; and when encoding the processing block in inter mode or bi-prediction mode, the video encoder (703) may use inter prediction or bi-prediction techniques, respectively, to encode the processing block into an encoded picture. In some video coding techniques, merge mode may be an inter-picture prediction submode, in which motion vectors are derived from one or more motion vector predictors without resorting to coded motion vector components external to the predictor. In some other video coding techniques, there may be motion vector components applicable to the subject block. In an embodiment, the video encoder (703) includes other components, such as a mode decision module (not shown) for determining the processing block mode.

[0102] exist Figure 7 In an embodiment of the present invention, the video encoder (703) includes Figure 7 Shown are an inter-frame encoder (730), an intra-frame encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together.

[0103] The inter-frame encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the current block with one or more reference blocks in a reference picture (e.g., blocks in a previous picture and a subsequent picture), generate inter-frame prediction information (e.g., redundant information description according to an inter-frame coding technique, motion vectors, merge mode information), and calculate an inter-frame prediction result (e.g., a predicted block) based on the inter-frame prediction information using any suitable technique. In some embodiments, the reference picture is a decoded reference picture decoded based on the encoded video information.

[0104] The intra-frame encoder (722) is configured to receive samples of a current block (e.g., a processing block), compare the block with previously encoded blocks in the same picture in some cases, generate quantized coefficients after transformation, and in some cases also generate intra-frame prediction information (e.g., intra-frame prediction direction information based on one or more intra-frame coding techniques). In an embodiment, the intra-frame encoder (722) further calculates an intra-frame prediction result (e.g., a predicted block) based on the intra-frame prediction information and a reference block in the same picture.

[0105] The general controller (721) is used to determine general control data and control other components of the video encoder (703) based on the general control data. In an embodiment, the general controller (721) determines the mode of the block and provides a control signal to the switch (726) based on the mode. For example, when the mode is intra mode, the general controller (721) controls the switch (726) to select the intra mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select intra prediction information and add the intra prediction information to the bitstream; and when the mode is inter mode, the general controller (721) controls the switch (726) to select the inter prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select inter prediction information and add the inter prediction information to the bitstream.

[0106] The residual calculator (723) is used to calculate the difference (residual data) between the received block and the prediction result selected from the intra-frame encoder (722) or the inter-frame encoder (730). The residual encoder (724) is used to operate based on the residual data to encode the residual data to generate transform coefficients. In an embodiment, the residual encoder (724) is used to convert the residual data from the time domain to the frequency domain and generate transform coefficients. The transform coefficients are then quantized to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is used to perform an inverse transform and generate decoded residual data. The decoded residual data can be used appropriately by the intra-frame encoder (722) and the inter-frame encoder (730). For example, the inter-frame encoder (730) can generate a decoded block based on the decoded residual data and inter-frame prediction information, and the intra-frame encoder (722) can generate a decoded block based on the decoded residual data and intra-frame prediction information. The decoded blocks are appropriately processed to generate a decoded picture, and in some embodiments, the decoded picture may be buffered in memory circuitry (not shown) and used as a reference picture.

[0107] The entropy encoder (725) is used to format the codestream to produce encoded blocks. The entropy encoder (725) generates various information according to a suitable standard, such as the HEVC standard. In an embodiment, the entropy encoder (725) is used to obtain general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information from the codestream. It should be noted that according to the disclosed subject matter, when encoding a block in inter-frame mode or the merge sub-mode of bidirectional prediction mode, there is no residual information.

[0108] Figure 8FIG is a diagram of a video decoder (810) according to another embodiment disclosed herein. The video decoder (810) is configured to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed picture. In an embodiment, the video decoder (810) is configured to replace Figure 4 A video decoder (410) of an embodiment.

[0109] exist Figure 8 In one embodiment, the video decoder (810) includes Figure 8 , an entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-frame decoder (872) coupled together are shown.

[0110] The entropy decoder (871) can be used to reconstruct certain symbols from the encoded picture, which represent syntax elements that constitute the encoded picture. Such symbols may include, for example, the mode used to encode the block (e.g., intra mode, inter mode, bidirectional prediction mode, a combined submode of the latter two, or another submode), prediction information (e.g., intra prediction information or inter prediction information) that can identify certain samples or metadata for prediction by the intra decoder (872) or the inter decoder (880), respectively, residual information in the form of, for example, quantized transform coefficients, etc. In an embodiment, when the prediction mode is inter or bidirectional prediction mode, the inter prediction information is provided to the inter decoder (880); and when the prediction type is intra prediction type, the intra prediction information is provided to the intra decoder (872). The residual information may be inverse quantized and provided to the residual decoder (873).

[0111] The inter-frame decoder (880) is configured to receive inter-frame prediction information and generate an inter-frame prediction result based on the inter-frame prediction information.

[0112] The intra-frame decoder (872) is configured to receive intra-frame prediction information and generate a prediction result based on the intra-frame prediction information.

[0113] The residual decoder (873) is used to perform inverse quantization to extract dequantized transform coefficients and process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (873) may also require certain control information (to obtain the quantizer parameter QP), and this information can be provided by the entropy decoder (871) (the data path is not shown because this is only low-level control information).

[0114] The reconstruction module (874) is used to combine the residual output by the residual decoder (873) with the prediction result (which may be output by the inter-frame prediction module or the intra-frame prediction module) in the spatial domain to form a reconstructed block. The reconstructed block can be part of a reconstructed picture, which in turn can be part of a reconstructed video. It should be noted that other suitable operations such as deblocking can be performed to improve visual quality.

[0115] It should be noted that the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using any suitable technology. In an embodiment, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more processors that execute software instructions.

[0116] Video codec techniques related to efficient application of secondary transforms (such as efficient application of secondary transform sets) are disclosed. The efficient application of secondary transforms can be applicable to any suitable video codec format or standard. The video codec format may include an open video codec format designed for video transmission over the Internet, such as AOMedia Video 1 (AV1) or a next-generation AOMedia video format other than AV1. The video codec standard may include the High Efficiency Video Codec (HEVC) standard or a next-generation video codec other than HEVC (e.g., Versatile Video Codec (VVC), etc.

[0117] Various intra prediction modes may be used, for example, in intra prediction modes in AV1 and / or VVC. In an embodiment, such as in AV1, directional intra prediction is used. In an example, such as in the open video codec format VP9, eight directional modes correspond to eight angles from 45° to 207°. In order to exploit a wider variety of spatial redundancies in directional textures (e.g., in AV1), directional modes (also referred to as directional intra modes, directional intra prediction modes, angle modes) may be extended to a set of angles with a finer granularity, such as Figure 9 As shown in .

[0118] Figure 9An example of a nominal mode for a coding block (CB) (910) according to an embodiment of the present disclosure is shown. Certain angles (also referred to as nominal angles) may correspond to nominal modes. In the example, eight nominal angles (or nominal intra-frame angles) (901)-(908) correspond to eight nominal modes (e.g., V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED), respectively. The eight nominal angles (901)-(908) and the eight nominal modes may be referred to as V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED, respectively. Further, each nominal angle may correspond to a plurality of finer angles (e.g., seven finer angles), and thus 56 angles (or prediction angles) or 56 directional modes (or angle modes, directional intra prediction modes) may be used, for example, in AV1. Each prediction angle may be represented by a nominal angle and an angle offset (or angle increment). The angle offset may be obtained by multiplying an offset integer 1 (e.g., -3, -2, -1, 0, 1, 2, or 3) by a step size (e.g., 3°). In an example, the prediction angle is equal to the sum of the nominal angle and the angle offset. In an example, such as in AV1, nominal modes (e.g., eight nominal modes (901)-(908)) may be signaled together with certain non-angular smoothing modes (e.g., five non-angular smoothing modes as described below, such as DC mode, PAETH mode, SMOOTH mode, vertical SMOOTH mode, and horizontal SMOOTH mode). Subsequently, if the current prediction mode is a directional mode (or an angular mode), an index may be further signaled to indicate an angle offset corresponding to the nominal angle (e.g., an offset integer of 1). In an example, in order to implement directional prediction modes in a general manner, 56 directional modes such as those used in AV1 are implemented with a unified directional predictor that projects each pixel to a reference sub-pixel position and interpolates the reference pixel through a 2-tap bilinear filter.

[0119] A non-directional smooth intra predictor (also referred to as a non-directional smooth intra prediction mode, a non-directional smooth mode, a non-angular smooth mode) may be used for intra prediction of a block (such as a CB). In some examples (e.g., in AV1), the five non-directional smooth intra prediction modes include a DC mode or DC predictor (e.g., DC), a PAETH mode or PAETH predictor (e.g., PAETH), a SMOOTH mode or SMOOTH predictor (e.g., SMOOTH), a vertical SMOOTH mode (referred to as a SMOOTH_V mode, a SMOOTH_V predictor, SMOOTH_V), and a horizontal SMOOTH mode (referred to as a SMOOTH_H mode, a SMOOTH_H predictor, or SMOOTH_H).

[0120] Figure 10 An example of non-directional smooth intra prediction modes (e.g., DC mode, PAETH mode, SMOOTH mode, SMOOTH_V mode, and SMOOTH_H mode) according to aspects of the present disclosure is shown. To predict a sample (1001) in a CB (1000) based on a DC predictor, an average of a first value of a left neighboring sample (1012) and a second value of an above neighboring sample (or top neighboring sample) (1011) may be used as a predictor.

[0121] To predict sample (1001) based on the PAETH predictor, the first value of the left neighboring sample (1012), the second value of the top neighboring sample (1011), and the third value of the top-left neighboring sample (1013) can be obtained. Then, the reference value is obtained using Equation 1.

[0122] Reference value = first value + second value - third value (Equation 1)

[0123] One of the first value, the second value, and the third value closest to the reference value may be set as a predictor of the sample (1001).

[0124] SMOOTH_V mode, SMOOTH_H mode, and SMOOTH mode can predict CB (1000) using quadratic interpolation in the vertical direction, the horizontal direction, and the average direction of the vertical and horizontal directions, respectively. To predict sample (1001) based on the SMOOTH predictor, an average value (e.g., a weighted combination) of the first value, the second value, the value of the right sample (1014), and the value of the bottom sample (1016) can be used. In various examples, the right sample (1014) and the bottom sample (1016) are not reconstructed, and therefore, the value of the upper right adjacent sample (1015) and the value of the lower left adjacent sample (1017) can replace the value of the right sample (1014) and the bottom sample (1016), respectively. Accordingly, the average value (e.g., a weighted combination) of the first value, the second value, the value of the upper right adjacent sample (1015), and the value of the lower left adjacent sample (1017) can be used as the SMOOTH predictor. To predict the sample (1001) based on the SMOOTH_V predictor, the average value (e.g., weighted combination) of the second value of the top neighboring sample (1011) and the value of the bottom left neighboring sample (1017) may be used. To predict the sample (1001) based on the SMOOTH_H predictor, the average value (e.g., weighted combination) of the first value of the left neighboring sample (1012) and the value of the top right neighboring sample (1015) may be used.

[0125] Figure 11 An example of a recursive filtering-based intra predictor (also referred to as a filter intra mode or recursive filtering mode) according to an embodiment of the present disclosure is shown. In order to capture the attenuated spatial correlation with the reference on the edge, the filter intra mode can be used for a block such as CB (1100). In the example, CB (1100) is a luma block. The luma block (1100) can be divided into multiple patches (e.g., eight 4×2 patches B0-B7). Each of the patches B0-B7 can have multiple neighboring samples. For example, patch B0 has seven neighboring samples (or seven neighbors) R00-R06, including four top neighboring samples R01-R04, two left neighboring samples R05-R06, and one upper left neighboring sample R00. Similarly, patch B7 has seven neighboring samples R70-R76, including four top neighboring samples R71-R74, two left neighboring samples R75-R76, and one upper left neighboring sample R70.

[0126] In some examples, such as for AV1, multiple (e.g., five) filter intra modes (or multiple recursive filtering modes) are pre-designed. Each filter intra mode can be represented by a set of eight 7-tap filters that reflect the correlation between samples (or pixels) in the corresponding 4×2 patch (e.g., B0) and seven neighbors (e.g., R00-R06) adjacent to the 4×2 patch B0. The weighting factors of the 7-tap filters can be position-dependent. For each of the patches B0-B7, seven neighbors (e.g., R00-R06 for B0 and R70-R76 for B7) can be used to predict samples in the corresponding patch. In the example, neighbors R00-R06 are used to predict samples in patch B0. In the example, neighbors R70-R76 are used to predict samples in patch B7. For some patches in CB (1100) (such as patch B0), all seven neighbors (e.g., R00-R06) have been reconstructed. For other patches in CB (1100), at least one of the seven neighbors is not reconstructed, and therefore one or more predicted values of one or more direct neighbors (or one or more predicted samples of one or more direct neighbors) can be used as reference. For example, the seven neighbors R70-R76 of patch B7 are not reconstructed, so the predicted samples of the direct neighbors can be used.

[0127] Chroma samples can be predicted from luma samples. In an embodiment, chroma from luma mode (e.g., CfL mode, CfL predictor) is a chroma-only intra predictor that can model chroma samples (or pixels) as linear functions of reconstructed luma samples (or pixels). For example, CfL prediction can be expressed using Equation 2 below.

[0128] CfL(α)=αL A +D (Equation 2)

[0129] Among them, L A represents the AC contribution of the luma component, α represents the scaling parameter of the linear model, and D represents the DC contribution of the chroma component. In an example, the reconstructed luma pixels are subsampled based on the chroma resolution and the average value is subtracted to form the AC contribution (e.g., L A To approximate the chroma AC components based on their AC contributions, in some examples, such as AV1, without requiring the decoder to calculate the scaling parameter α, the CfL mode determines the scaling parameter α based on the original chroma pixels and signals the scaling parameter α in the codestream, thereby reducing decoder complexity and producing more accurate predictions. The DC contribution of the chroma components can be calculated using the intra-frame DC mode. The intra-frame DC mode is sufficient for most chroma content and has a mature and fast implementation.

[0130] Multi-line intra prediction can use more reference lines for intra prediction. A reference line can include multiple samples in a picture. In an example, a reference line includes samples in a row and samples in a column. In an example, the encoder can determine and signal a reference line for generating an intra predictor. An index indicating a reference line (also referred to as a reference line index) can be signaled before one or more intra prediction modes. In an example, only MPM is allowed when a non-zero reference line index is signaled. Figure 12 An example of four reference lines for CB (1210) is shown. Figure 12 , a reference line can include up to six fragments (e.g., fragments A to F) and the top left reference sample. For example, reference line 0 includes fragments B and E and the reference sample in the top left corner. For example, reference line 3 includes fragments A to F and the top left reference sample. Fragments A and F can be padded with the closest samples from fragments B and E, respectively. In some examples, such as in HEVC, only one reference line (e.g., reference line 0 adjacent to CB (1210)) is used for intra prediction. In some examples, such as in VVC, multiple reference lines (e.g., reference lines 0, 1, and 3) are used for intra prediction.

[0131] Typically, references such as the above can be used Figures 9 to 12 The block is predicted using one or a suitable combination of the various intra prediction modes described.

[0132] Transform block partitioning (also referred to as transform partitioning, transform unit partitioning) may be implemented to partition a block into multiple transform units. Figures 13 and 14 An exemplary transform block partition according to an embodiment of the present disclosure is shown. In some examples, such as in AV1, both intra-coded blocks and inter-coded blocks can be further partitioned into multiple transform units with a partition depth of up to multiple levels (e.g., 2 levels).

[0133] For intra-coded blocks, transform partitioning can be performed so that the transform blocks associated with an intra-coded block have the same size and can be encoded in raster scan order. Figure 13, transform block partitioning may be performed on a block (e.g., an intra-coded block) (1300). The block (1300) may be partitioned into transform units, such as four transform units (e.g., TBs) (1301)-(1304), and the partition depth is 1. The four transform units (e.g., TBs) (1301)-(1304) may have the same size and may be encoded in a raster scan order (1310) from the transform unit (1301) to the transform unit (1304). In an example, the four transform units (e.g., TBs) (1301)-(1304) are respectively transformed using, for example, different transform kernels. In some examples, each of the four transform units (e.g., TBs) (1301)-(1304) is further partitioned into four transform units. For example, the transform unit (1301) is partitioned into transform units (1321), (1322), (1325), and (1326), the transform unit (1302) is partitioned into transform units (1323), (1324), (1327), and (1328), the transform unit (1303) is partitioned into transform units (1329), (1330), (1333), and (1334), and the transform unit (1304) is partitioned into transform units (1331), (1332), (1335), and (1336). The partition depth is 2. The transform units (e.g., TBs) (1321)-(1336) may have the same size and may be encoded in a raster scan order (1320) from the transform unit (1321) to the transform unit (1336).

[0134] For inter-frame coded blocks, transform partitioning can be performed in a recursive manner, where the partitioning depth can be up to multiple levels (e.g., two levels). Transform partitioning can support any suitable transform unit size and shape. Transform unit shapes may include square shapes and non-square shapes (e.g., non-square rectangular shapes) with any suitable aspect ratio. The transform unit size can range from 4×4 to 64×64. The aspect ratio of the transform unit (e.g., the ratio of the width of the transform unit to the height of the transform unit) can be 1:1 (square), 1:2, 2:1, 1:4, or 4:1, etc. Transform partitioning can support 1:1 (square), 1:2, 2:1, 1:4, and / or 4:1 transform unit sizes ranging from 4×4 to 64×64. Reference Figure 14, transform block partitioning (1400) may be recursively performed on a block (e.g., an inter-coded block). For example, the block (1400) is partitioned into transform units (1401)-(1407). The transform units (e.g., TBs) (1401)-(1407) may have different sizes and may be encoded in a raster scan order (1410) from the transform unit (1401) to the transform unit (1407). In an example, the partition depth of the transform units (1401), (1406), and (1407) is 1, and the partition depth of the transform units (1402)-(1405) is 2.

[0135] In an example, if the coding block is smaller than or equal to 64x64, the transform partition may only be applied to the luma component.In an example, the coding block refers to a CTB.

[0136] If the coding block width W or the coding block height H is greater than 64, the coding block can be implicitly split into multiple TBs, where the coding block is a luma coding block. The width of one of the multiple TBs can be the minimum of W and 64, and the height of one of the multiple TBs can be the minimum of H and 64.

[0137] If the coding block width W or the coding block height H is greater than 64, the coding block can be implicitly split into multiple TBs, where the coding block is a chroma coding block. The width of one of the multiple TBs can be the minimum value of W and 32, and the height of one of the multiple TBs can be the minimum value of H and 32.

[0138] Embodiments of primary transforms such as those used in AOMedia Video 1 (AV1) are described below. To support extended coding block partitioning, such as described in the present disclosure, multiple transform sizes (e.g., ranging from 4 points to 64 points for each dimension) and transform shapes (e.g., square, rectangular shapes with width at a height ratio of 2:1, 1:2, 4:1, or 1:4) may be used, such as in AV1.

[0139] The 2D transform process can use a hybrid transform kernel that can include a different 1D transform for each dimension of the coded residual block. The primary 1D transform can include (a) 4-point, 8-point, 16-point, 32-point, 64-point DCT-2; (b) 4-point, 8-point, 16-point asymmetric DST (ADST) (e.g., DST-4, DST-7) and corresponding flipped versions (e.g., a flipped version of ADST or FlipADST can apply ADST in the reverse order); and / or (c) 4-point, 8-point, 16-point, 32-point identity transform (IDTX). Figure 15 An example of a main transform basis function according to an embodiment of the present disclosure is shown. Figure 15The main transform basis functions in the example include basis functions of DCT-2 with N-point input and asymmetric DST (DST-4 and DST-7). Figure 15 The main transform basis functions shown in can be used in AV1.

[0140] The availability of hybrid transform kernels may depend on the transform block size and the prediction mode. Figure 16A An exemplary dependency of the availability of various transform kernels (e.g., the transform types shown in the first column and described in the second column) based on transform block size (e.g., the size shown in the third column) and prediction mode (e.g., intra prediction and inter prediction shown in the third column) is shown. An exemplary hybrid transform kernel and availability based on prediction mode and transform block size can be used in AV1. Reference Figure 16A , the symbols “→” and “↓” represent the horizontal dimension (also referred to as the horizontal direction) and the vertical dimension (also referred to as the vertical direction), respectively. The symbols “√” and “x” represent the availability of the transform kernel for the corresponding block size and prediction mode. For example, the symbol “√” indicates that the transform kernel is available, and the symbol “x” indicates that the transform kernel is not available.

[0141] In the example, the transform type (1610) is represented by Figure 16A The ADST_DCT representation is shown in the first column of Figure 16A As shown in the second column of , the transform type (1610) includes ADST in the vertical direction and DCT in the horizontal direction. Figure 16A In the third column, when the block size is less than or equal to 16×16 (e.g., 16×16 samples, 16×16 luma samples), the transform type (1610) can be used for intra prediction and inter prediction.

[0142] In the example, the transform type (1620) is represented by Figure 16A The V_ADST shown in the first column indicates that Figure 16A As shown in the second column of , the transform type (1620) includes ADST in the vertical direction and IDTX (i.e., identity matrix) in the horizontal direction. Therefore, the transform type (1620) (e.g., V_ADST) is performed in the vertical direction but not in the horizontal direction. Figure 16A In the third column, regardless of the block size, the transform type (1620) is not available for intra prediction. When the block size is less than 16×16 (e.g., 16×16 samples, 16×16 luma samples), the transform type (1620) is available for inter prediction.

[0143] In the example, Figure 16A Applicable to luminance components. For chrominance components, the transform type (or transform kernel) selection can be performed implicitly. In the example, for intra prediction residuals, the transform type can be selected according to the intra prediction mode, such as Figure 16B In the example, Figure 16B The transform type selection shown in

[15] can be applied to chroma components. For inter-frame prediction residuals, the transform type can be selected based on the transform type selection of the co-located luma block. Therefore, in this example, the transform type of the chroma components is not signaled in the codestream.

[0144] For example, in AOMedia Video 2 (AV2), Line Graph Transform (LGT) can be used in transforms such as the main transform. 8-bit / 10-bit transform kernels can be used in AV2. In some examples, LGT includes various DCTs and discrete sine transforms (DSTs), as described below. LGT can include 32-point and 64-point one-dimensional (1D) DSTs.

[0145] A graph is a general mathematical structure consisting of a set of vertices and edges that can be used to model similarity relations between objects of interest. A weighted graph in which a set of weights is assigned to edges and optionally to vertices can provide a sparse representation for robust modeling of signals / data. LGT can improve codec efficiency by providing better adaptability to different block statistics. By learning a line graph from the data to model the potential row-by-row and column-by-column statistics of the residual signal of a block, a separable LGT can be designed and optimized, and the associated generalized graph Laplacian (GGL) matrix can be used to derive the LGT.

[0146] Figure 16C shows the self-loop weight (e.g., v c1 、v c2 ) and edge weight w c Example of a general LGT for characterization. Given a weighted graph G(W, V), the GGL matrix can be defined as follows.

[0147] L c =D-W+V (Equation 3)

[0148] Where W can be a non-negative edge weight w c The adjacency matrix, D can be a diagonal matrix, and V can be a self-loop weight v c1 and v c2 The diagonal matrix of . Figure 16D The matrix L is shown c .

[0149] LGT can be obtained by GGL matrix L as follows c The eigendecomposition of is derived.

[0150] L c =UΦU T (Equation 4)

[0151] Here, the columns of the orthogonal matrix U can be the basis vectors of LGT, and Φ can be the diagonal eigenvalue matrix.

[0152] In various examples, certain DCTs and DSTs (e.g., DCT-2, DCT-8, and DST-7) are subsets of the set of LGTs derived from certain forms of GGL. c1 Set to 0 (for example, v c1 =0) to derive DCT-2. c1 Set to w c (For example, c1 =w c ) to derive DST-7. c2 Set to w c (For example, v c2 =w c ) to derive DCT-8. c1 Set to 2w c (For example, v c1 =2w c ) to derive DST-4. This can be done by adding v c2 Set to 2w c (For example, v c2 =2w c ) to derive DCT-4.

[0153] In some examples, such as AV2, LGT can be implemented as a matrix multiplication. c Lieutenant General v c1 Set to 2w c The 4-point (4p) LGT kernel is derived, and thus the 4pLGT kernel is DST-4. c Lieutenant General v c1 Set to 1.5w c In an example, an LGT kernel such as a 16-point (16p) LGT kernel, a 32-point (32p) LGT kernel, or a 64-point (64p) LGT kernel can be derived by converting v c1 Set to w c And v c2 Set to 0 to export and the LGT kernel can become DST-7.

[0154] A transform such as a primary transform or a secondary transform may be applied to a block such as a CB. In an example, the transform includes a combination of a primary transform and a secondary transform. The transform may be an inseparable transform, a separable transform, or a combination of inseparable and separable transforms.

[0155] A secondary transform may be performed, such as in VVC. In some examples, such as in VVC, a low frequency non-separable transform (LFNST), also known as a reduced secondary transform (RST), may be applied between the forward primary transform and quantization at the encoder side and between dequantization and the inverse primary transform at the decoder side, such as Figures 17 and 18 to further decorrelate the main transform coefficients.

[0156] The application of a non-separable transform that can be used in LFNST can be described below using a 4×4 input block (or input matrix) X as an example (as shown in Equation 5). To apply a 4×4 non-separable transform (e.g., LFNST), the 4×4 input block X can be represented by the vector is represented as shown in Equation 5-6.

[0157]

[0158]

[0159] The non-separable transformation can be calculated as in The 16×1 coefficient vector is then transformed using the scan order of the 4×4 input block (e.g., horizontal scan order, vertical scan order, zigzag scan order, or diagonal scan order). Reorganize into 4×4 output blocks (or output matrices, coefficient blocks). Transform coefficients with smaller indices can be placed in the 4×4 coefficient blocks using smaller scan indices.

[0160] A non-separable secondary transform may be applied to a block (e.g., CB). In some examples, such as in VVC, Figures 17 and 18 As shown in , LFNST is applied between the forward main transform and quantization (eg, at the encoder side) and between dequantization and the inverse main transform (eg, at the decoder side).

[0161] Figures 17 and 18 Examples of two transform encoding and decoding processes (1700) and (1800) using a 16x64 transform (or a 64x16 transform, depending on whether the transform is a forward quadratic transform or an inverse quadratic transform) and a 16x48 transform (or a 48x16 transform, depending on whether the transform is a forward quadratic transform or an inverse quadratic transform) are shown, respectively. Figure 17, in the process (1700), on the encoder side, a forward primary transform (1710) can first be performed on the block (e.g., the residual block) to obtain a coefficient block (1713). Subsequently, a forward secondary transform (or forward LFNST) (1712) can be applied to the coefficient block (1713). In the forward secondary transform (1712), the 64 coefficients of the 4×4 sub-block AD at the upper left corner of the coefficient block (1713) can be represented by a 64-length vector, and the 64-length vector can be multiplied by a transform matrix of 64×16 (i.e., a width of 64 and a height of 16), resulting in a 16-length vector. The elements in the 16-length vector are filled back into the upper left 4×4 sub-block A of the coefficient block (1713). The coefficients in the sub-block BD can be zero. The coefficients obtained after the forward secondary transform (1712) are then quantized in a quantization step (1714) and entropy encoded to generate coded bits in the bitstream (1716).

[0162] The coded bits may be received at the decoder side and entropy decoded, followed by a dequantization step (1724) to generate a coefficient block (1723). An inverse secondary transform (or inverse LFNST) (1722) may be performed, such as an inverse RST 8×8, to obtain 64 coefficients, for example, from the 16 coefficients at the top left 4×4 sub-block E. The 64 coefficients may be padded back into the 4×4 sub-block EH. Further, the coefficients in the coefficient block (1723) after the inverse secondary transform (1722) may be processed with an inverse primary transform (1720) to obtain a recovered residual block.

[0163] Figure 18 The exemplary process (1800) is similar to the process (1700), except that fewer (i.e., 48) coefficients are processed during the forward secondary transform (1712). Specifically, the 48 coefficients in the sub-block AC are processed using a smaller transform matrix of size 48×16. Using a smaller transform matrix of size 48×16 can reduce the memory size used to store the transform matrix and the number of calculations (e.g., multiplications, additions, and / or subtractions, etc.), and thus can reduce computational complexity.

[0164] In an example, a 4×4 non-separable transform (e.g., 4×4 LFNST) or an 8×8 non-separable transform (e.g., 8×8 LFNST) is applied according to the block size of the block (e.g., CB). The block size of the block may include width, height, etc. For example, 4×4 LFNST is applied to a block whose minimum width and height are less than a threshold value, such as 8 (e.g., min(width, height)<8). For example, 8×8 LFNST is applied to a block whose minimum width and height are greater than a threshold value, such as 4 (e.g., min(width, height)>4).

[0165] A non-separable transform (e.g., LFNST) can be based on a direct matrix multiplication method and can thus be implemented in a single pass without iteration. To reduce the dimension of the non-separable transform matrix and minimize the computational complexity and memory space for storing transform coefficients, a reduced non-separable transform method (or RST) can be used in LFNST. Accordingly, in the reduced non-separable transform, an N-dimensional vector (e.g., for an 8×8 non-separable quadratic transform (NSST), N is 64) can be mapped to an R-dimensional vector in a different space, where N / R (R < N) is the reduction factor. Thus, instead of an N×N matrix, the RST matrix is an R×N matrix as described in Equation 7.

[0166]

[0167] In Equation 7, the R rows of the R×N transform matrix are the R bases of the N-dimensional space. The inverse transform matrix can be the transpose of the transform matrix used in the forward transform (e.g., T RxN ). For an 8×8 LFNST, a reduction factor of 4 can be applied, and the 64×64 direct matrix used in the 8×8 non-separable transform can be reduced to a 16×64 direct matrix, as Figure 17 shown. Alternatively, a reduction factor greater than 4 can be applied, and the 64×64 direct matrix used in the 8×8 non-separable transform can be reduced to a 16×48 direct matrix, as Figure 18 shown. Thus, a 48×16 inverse RST matrix can be used on the decoder side to generate the kernel (primary) transform coefficients in the 8×8 upper left region.

[0168] Referring to Figure 18 , when a 16×48 matrix is applied instead of a 16×64 matrix with the same transform set configuration, the input of the 16×48 matrix includes 48 input data from three 4×4 blocks A, B, and C in the upper left 8×8 block except for the lower right 4×4 block D. As the size decreases, the memory usage for storing the LFNST matrix can be reduced with minimal performance degradation, e.g., from 10 KB to 8 KB.

[0169] To reduce complexity, if the coefficients outside the first coefficient subgroup are non-important, the LFNST can be restricted to be applicable. In an example, the LFNST can be restricted to be applicable only when all the coefficients outside the first coefficient subgroup are non-effective. Referring to Figures 17 and 18 , the first coefficient subgroup corresponds to the upper left block E, and thus the coefficients outside block E are non-effective.

[0170] In an example, when LFNST is applied, only the main transform coefficients are non-significant (e.g., zero). In an example, when LFNST is applied, all main transform coefficients are zero. Main transform coefficients may refer to transform coefficients obtained from a main transform without a secondary transform. Accordingly, LFNST index signaling may be conditioned on the last significant position, thereby avoiding additional coefficient scanning in LFNST. In some examples, the additional coefficient scanning is used to check for significant transform coefficients at specific positions. In an example, the worst-case processing of LFNST (e.g., per-pixel multiplication) limits the non-separable transforms for 4×4 blocks and 8×8 blocks to 8×16 and 8×48 transforms, respectively. In these cases, when LFNST is applied, the last significant scan position may be less than 8. For other sizes, when LFNST is applied, the last significant scan position may be less than 16. For 4×N and N×4 CBs with N greater than 8, this restriction may mean that LFNST is applied to the top-left 4×4 region in the CB. In an example, this restriction means that LFNST is applied only once to the top-left 4×4 region in the CB. In this example, when LFNST is applied, all primary coefficients are insignificant (e.g., zero), reducing the number of operations required for the main transform. From the encoder's perspective, quantization of transform coefficients can be significantly simplified when testing LFNST transforms. Rate-distortion optimized quantization can be performed to the greatest extent possible for the first 16 coefficients, e.g., in scan order, and the remaining coefficients can be set to zero.

[0171] The LFNST transform (e.g., a transform kernel, a transform core, or a transform matrix) may be selected as described below. In an embodiment, multiple transform sets may be used, and one or more non-separable transform matrices (or kernels) may be included in each of the multiple transform sets in the LFNST. According to aspects of the present disclosure, a transform set may be selected from the multiple transform sets, and a non-separable transform matrix may be selected from one or more non-separable transform matrices in the transform set.

[0172] Table 1 shows an exemplary mapping from intra prediction modes to multiple transform sets according to an embodiment of the present disclosure. The mapping indicates the relationship between the intra prediction modes and the multiple transform sets. The relationship such as that indicated in Table 1 can be predefined and can be stored in the encoder and decoder.

[0173] Table 1: Transformation set selection table

[0174]

[0175]

[0176] Referring to Table 1, the plurality of transform sets includes four transform sets, for example, transform sets 0 to 3 represented by transform set indices (e.g., Tr.set index) from 0 to 3. The index (e.g., IntraPredMode) may indicate an intra prediction mode, and the transform set index may be obtained based on the index and Table 1. Accordingly, the transform set may be determined based on the intra prediction mode. In an example, if one of three cross-component linear model (CCLM) modes (e.g., INTRA_LT_CCLM, INTRA_T_CCLM, or INTRA_L_CCLM) is used for a CB (e.g., 81<=IntraPredMode<=83), transform set 0 is selected for the CB.

[0177] As described above, each transform set may include one or more non-separable transform matrices. One of the one or more non-separable transform matrices may be selected by an LFNST index, which is, for example, explicitly signaled. For example, the LFNST index may be signaled once in the bitstream of each intra-coded CU (e.g., CB) after the transform coefficients are signaled. In an embodiment, each transform set includes two non-separable transform matrices (kernels), and the selected non-separable secondary transform candidate may be one of the two non-separable transform matrices. In some examples, LFNST is not applied to the CB (e.g., the CB is encoded in transform skip mode or the number of non-zero coefficients of the CB is less than a threshold). In an example, when LFNST is not applied to the CB, the LFNST index of the CB is not signaled. The default value of the LFNST index may be zero and not signaled, indicating that LFNST is not applied to the CB.

[0178] In an embodiment, LFNST is restricted to being applied only when all coefficients outside the first coefficient subgroup are non-significant, and the encoding and decoding of the LFNST index may depend on the position of the last significant coefficient. The LFNST index may be context-coded. In an example, the context encoding and decoding of the LFNST index does not depend on the intra prediction mode, and only the first binary number is context-coded. LFNST may be applied to intra-coded CUs in intra or inter slices and for both luma and chroma components. If dual tree is enabled, the LFNST indexes for luma and chroma components may be signaled separately. For inter slices (e.g., dual tree is disabled), a single LFNST index may be signaled and used for both luma and chroma components.

[0179] The intra subpartitioning (ISP) codec mode can be used. In the ISP codec mode, the luma intra prediction block can be divided vertically or horizontally into 2 or 4 subpartitions, depending on the block size. In some examples, when RST is applied to each feasible subpartition, the performance improvement is not significant. Therefore, in some examples, when the ISP mode is selected, LFNST is disabled and the LFNST index (or RST index) is not signaled. Disabling RST or LFNST for the ISP predicted residue can reduce the codec complexity. In some examples, when the matrix-based intra prediction mode (MIP) is selected, LFNST is disabled and the LFNST index is not signaled.

[0180] In some examples, due to the maximum transform size limit (e.g., 64×64), CUs larger than 64×64 are implicitly partitioned (TU tiling), and LFNST index searches can quadruple the data buffer for a certain number of decoding pipeline stages. Therefore, the maximum size allowed for LFNST can be limited to 64×64. In an example, LFNST is enabled only with discrete cosine transform (DCT) type 2 (DCT-2) transforms.

[0181] In some examples, the separable transform scheme may not be effective for capturing directional texture patterns (e.g., edges along the 45° or 135° directions). For example, in the above scenario, the non-separable transform scheme can improve codec efficiency. In order to reduce computational complexity and memory usage, the non-separable transform scheme can be used as a secondary transform, which is applied to the low-frequency transform coefficients obtained from the primary transform. The secondary transform can be applied to a block, and information indicating the secondary transform can be signaled for the block based on prediction mode information, primary transform type, adjacent reconstructed samples, etc. Further, transform block partition information (also referred to as transform block partition information, transform partition information, or transform partition information), the size of the coding block, and the shape of the coding block can provide additional information for efficient application and / or signaling of the secondary transform.

[0182] According to various aspects of the present disclosure, encoding information of a block can be decoded from an encoded video stream, and the encoding information may indicate one or a combination of an intra prediction mode of the block, transform partition information of the block, block size, and block shape.

[0183] The transform partition information may indicate whether and / or how the block is further partitioned into multiple TBs or TUs. Figures 13 and 14 As described above, a block may be partitioned into multiple TUs or TBs based on transform partition information of the block. In an example, the transform partition information is signaled in the coded video stream. The transform partition information of the block may indicate the partition depth of the block.

[0184] In the present disclosure, the term "block" may refer to a prediction block (PB), a coding block (CB), a coded block, a coding unit (CU), a transform block (TB), a transform unit (TU), a luminance block (e.g., a luminance CB), a chrominance block (e.g., a chrominance CB), etc.

[0185] The size of a block may refer to block width, block height, block aspect ratio (e.g., ratio of block width to block height, ratio of block height to block width), block area size or block area (e.g., block width×block height), minimum value of block width and block height, maximum value of block width and block height, etc. The shape of a block may refer to any suitable shape of a block. The shape of a block may refer to, but is not limited to, a non-square shape (such as a rectangular shape), a square shape, etc. The shape of a block may refer to the block aspect ratio.

[0186] In an example, one or a combination of block transform partition information, block size, and block shape is signaled in the coded video stream. In an example, one or a combination of block transform partition information, block size, and block shape is determined based on other information in the coded video stream.

[0187] Whether to disable the secondary transform for a block may be determined based on one or a combination of the transform partition information of the block, the size of the block, and the shape of the block. In an example, whether to signal (e.g., notify in an encoded video bitstream) information associated with the secondary transform (e.g., a secondary transform index) may be determined based on one or a combination of the transform partition information of the block, the size of the block, and the shape of the block.

[0188] Furthermore, the block may be reconstructed based on whether the secondary transform is disabled for the block. If it is determined that the secondary transform is disabled for the block, the block may be reconstructed using only the primary transform (e.g., the inverse primary transform) without the secondary transform. In an example, it is determined that information associated with the secondary transform (e.g., the secondary transform index) is not signaled in the coded video bitstream. If it is determined that the secondary transform is not disabled for the block (e.g., it is determined that the secondary transform is enabled for the block), the block may be reconstructed using the primary transform (e.g., the inverse primary transform) and the secondary transform (e.g., the inverse secondary transform). For example, if it is determined that the secondary transform is not disabled for the block and it is further determined that the secondary transform is applied to the block, the block is reconstructed using the primary transform and the secondary transform.

[0189] The information associated with the secondary transform (e.g., a secondary transform index) may indicate a secondary transform (e.g., a secondary transform kernel, a secondary transform core, or a secondary transform matrix) to be applied to the block. In an example, the secondary transform is LFNST, RST, etc. As discussed above, in an embodiment, multiple transform sets may be used, and one or more secondary transform matrices (or kernels) may be included in each of the multiple transform sets. According to various aspects of the present disclosure, a transform set may be selected from the multiple transform sets using any suitable method including, but not limited to, that described with reference to Table 1, and a secondary transform (e.g., a secondary transform matrix) to be applied to the block may be selected from one or more secondary transform matrices in the transform set using information associated with the secondary transform (e.g., a secondary transform index).

[0190] For example, information (e.g., a secondary transform index) can be explicitly signaled in the coded video bitstream. In an example, the secondary transform index refers to the LFNST index described above. In some examples, a secondary transform is not applied to a block (e.g., a CB encoded in transform skip mode or the number of non-zero coefficients of the CB is less than a threshold). In an example, when a secondary transform is not applied to a block, a secondary transform index (e.g., an LFNST index) is not signaled for the block. The default value of the secondary transform index can be zero and not signaled, indicating that a secondary transform is not applied to the block.

[0191] In an embodiment, one or a combination of the block's transform partition information, the block's size, and the block's shape may include the block's transform partition information. The transform partition information may be signaled in the coded video bitstream. The block's transform partition information may indicate the block's partition depth. For example, as shown in FIG. Figure 13 As described, a block can be partitioned into multiple TUs or TBs based on the transform partition information of the block. Accordingly, it can be determined whether to disable the secondary transform for the block based on the partition depth. In an example, if the partition depth is greater than a threshold value n, it is determined that the secondary transform is disabled for the block, and it is determined not to signal the secondary transform index. The threshold value n can be any suitable integer. The threshold value n can be 0 or a positive integer. Exemplary values of the threshold value n include, but are not limited to, 0, 1, 2, etc. In an example, the threshold value n is 0. The secondary transform index (e.g., LFNST index) can indicate the secondary transform kernel to be applied to the block.

[0192] According to aspects of the present disclosure, one or a combination of transform partition information, the size of the block, and the shape of the block may be used to apply and / or signal a secondary transform for the block. In an example, one or a combination of the transform partition information, the size of the block, and the shape of the block may be used to apply and / or signal multiple secondary transforms for the block. Whether to disable or enable a secondary transform for the block may be determined based on one or a combination of the transform partition information, the size of the block, and the shape of the block. Whether to apply a secondary transform to the block may be determined based on one or a combination of the transform partition information, the size of the block, and the shape of the block. Whether to signal the application of a secondary transform to the block may be determined based on one or a combination of the transform partition information for the block, the size of the block, and the shape of the block.

[0193] In an embodiment, transform partition information for a block may be signaled, and the block may be partitioned into multiple TUs or TBs. Whether to disable the secondary transform for a block may depend on the transform partition information for the block. The transform partition information for the block may indicate the partition depth of the block. In an example, whether to disable the secondary transform for a block depends on the partition depth of the block. In some examples, whether to signal information associated with the secondary transform (e.g., a secondary transform index) depends on the transform partition information for the block (e.g., the partition depth of the block). In some examples, whether to signal information associated with the secondary transform (e.g., a secondary transform index) depends on the transform partition information for the block (e.g., the partition depth of the block). In an example, the secondary transform index is represented as stIdx. In an example, if the partition depth is based on a threshold, such as if the partition depth is greater than a threshold n, it is determined that the secondary transform is disabled and the secondary transform index is not signaled. The threshold n may be any suitable integer. In an example, the threshold n is 0. In an example, the threshold n is a positive integer. Exemplary values of the threshold n include, but are not limited to, 0, 1, 2, etc. In an example, if a block is partitioned into multiple TUs or TBs, whether to disable secondary transform for the block and / or whether to not signal the secondary transform index may depend on the partition depth and / or the threshold n.

[0194] In an embodiment, one or a combination of the transform partition information of the block, the size of the block, and the shape of the block may include the transform partition information of the block and the shape of the block. The transform partition information may be signaled in the coded video bitstream. The transform partition information may indicate the partition depth of the block. The shape of the block may be a non-square rectangle. The block may be partitioned into multiple TUs or TBs. Whether to disable secondary transform for the block may be determined based on the partition depth. In an example, if the partition depth is greater than a threshold, which may be 0 or a positive integer, it is determined that secondary transform is disabled for the block.

[0195] In an embodiment, transform partition information for a block may be signaled, the block may have a non-square rectangular shape (i.e., the shape of the block is a non-square rectangle), and the block may be further partitioned into multiple TUs or TBs. Whether to disable the secondary transform for the block may depend on the transform partition information for the block. The transform partition information for the block may indicate the partition depth of the block. In an example, whether to disable the secondary transform for the block depends on the partition depth of the block. In some examples, whether to signal information associated with the secondary transform (e.g., the secondary transform index stIdx) depends on the transform partition information for the block (e.g., the partition depth of the block). In an example, determining to disable the secondary transform and determining not to signal the secondary transform index is based on a threshold, such as if the partition depth is greater than a threshold n. As discussed above, the threshold n may be any suitable integer, such as 0 or a positive integer. Example values of the threshold n include, but are not limited to, 0, 1, 2, etc. In an example, if the block is partitioned into multiple TUs, whether to disable the secondary transform for the block and / or whether to signal the secondary transform index may depend on the partition depth and the threshold n.

[0196] In an embodiment, one or a combination of the transform partition information of the block, the size of the block, and the shape of the block may include the shape of the block indicated by the aspect ratio of the block. Accordingly, whether to disable secondary transform for a block may be determined based on the aspect ratio of the block.

[0197] In an embodiment, whether to disable the secondary transform for a block may depend on the shape of the block (e.g., the aspect ratio of the block). Whether to apply the secondary transform for a block may depend on the shape of the block (e.g., the aspect ratio of the block). In some examples, whether to signal information associated with the secondary transform (e.g., the secondary transform index stIdx) depends on the shape of the block (e.g., the aspect ratio of the block). The aspect ratio of the block may be a ratio of a first dimension of the block to a second dimension of the block, where the first dimension of the block is greater than or equal to the second dimension. If the aspect ratio of the block is greater than a threshold value L (e.g., 1, 2, 4, 8, etc.), it may be determined that the secondary transform is disabled for the block. In an example, the threshold value L is 2 m , where m is 0 or a positive integer.

[0198] In an example, when the aspect ratio of a block (eg, ratio of block width to block height) is greater than a threshold L (eg, 1, 2, 4, 8, etc.), a secondary transform index is not signaled and / or the secondary transform is not applied.

[0199] In an example, when the aspect ratio of the block (e.g., the ratio of block width to block height) is less than a threshold J (e.g., 1, 1 / 2, 1 / 4, 1 / 8, etc.), the secondary transform index is not signaled and / or the secondary transform is not applied. In an example, the threshold J is 2 -m , where m is 0 or a positive integer.

[0200] In an embodiment, one or a combination of the transform partition information, the size of the block, and the shape of the block may include the transform partition information and the shape of the block. The transform partition information may indicate a partition depth of the block. The shape of the block may be square. The block may be partitioned into multiple TUs or TBs. Whether to disable secondary transform for the block may be determined based on the partition depth. In an example, if the partition depth is greater than a threshold value that may be 0 or a positive integer, it is determined that secondary transform is disabled for the block.

[0201] In an embodiment, a block may be partitioned into multiple TUs or TBs. Further, the shape of the block may be square (e.g., the aspect ratio of the block is 1). Accordingly, whether to disable the secondary transform for a block may depend on the transform partition information of the block (e.g., the partition depth of the block). In an example, the transform partition information of the block is signaled. In some examples, whether to signal information associated with the secondary transform (e.g., the secondary transform index) depends on the transform partition information of the block (e.g., the partition depth of the block).

[0202] In an example, determining to disable the secondary transform and not signaling the secondary transform index is based on the partition depth, such as if the partition depth is greater than a threshold n. The threshold n can be any suitable integer, such as 0 or a positive integer (1, 2, etc.). In an example, if a block is partitioned into multiple TUs, whether to disable the secondary transform for the block and / or whether to not signal the secondary transform index can depend on the partition depth and the threshold n.

[0203] In an embodiment, one or a combination of the transform partition information of the block, the size of the block, and the shape of the block may include the transform partition information of the block and the size of the block. The transform partition information may indicate the partition depth of the block. The size of the block may indicate the width of the block (or block width) and the height of the block (or block height), wherein the width of the block and the height of the block are greater than a threshold size. For example, the block width and the block height are greater than the threshold size. The block may be partitioned into multiple TUs or TBs. Whether to disable secondary transform for the block may be determined based on the transform partition information of the block (e.g., partition depth). In an example, if the partition depth is greater than a threshold, the threshold may be 0 or a positive integer, and it is determined that secondary transform is disabled for the block.

[0204] In an embodiment, the size of the block (e.g., the minimum of the block width and the block height) may be greater than a threshold size. The threshold size may be any suitable size. In an example, the size of the block refers to the minimum of the block width and the block height, and the threshold size is 64, 128, 256, etc. The block may be partitioned into multiple TUs or TBs. In an example, transform partition information of the block is also signaled. Accordingly, whether to disable the secondary transform of the block may depend on the transform partition information of the block (e.g., the partition depth of the block). In some examples, whether to signal information associated with the secondary transform (e.g., the secondary transform index) depends on the transform partition information of the block (e.g., the partition depth of the block).

[0205] In an example, a determination is made based on the partition depth to disable the secondary transform and / or to not signal information associated with the secondary transform (e.g., a secondary transform index), such as if the partition depth is greater than a threshold n. The threshold n may be any suitable integer, such as 0 or a positive integer (1, 2, etc.). In an example, if a block is partitioned into multiple TUs or TBs, whether to disable the secondary transform for the block and / or whether to not signal the secondary transform index may depend on the partition depth and the threshold n.

[0206] In an example, exemplary values of the threshold size include, but are not limited to, 256×256, 256×128, 128×256, 128×128, 128×64, 64×128, 64×64, and the like.

[0207] In an embodiment, the width W′ of another block and / or the height H′ of another block may be greater than the maximum transform size T, and the another block may be implicitly partitioned into a plurality of sub-blocks including the block. The maximum transform size T may be, for example, a predetermined parameter available to a decoder and / or an encoder. In an example, the maximum transform size T is not signaled. The width W of a block (e.g., one of a plurality of sub-blocks) may be the minimum of W′ and T, and the height H of the block may be the minimum of H′ and T. If the partition depth of a block (e.g., one of a plurality of sub-blocks) is greater than a threshold, it is determined not to apply a secondary transform and / or it is determined not to signal information associated with the secondary transform (e.g., a secondary transform index). The partition depth may be signaled. Exemplary values of the threshold include, but are not limited to, 0, 1, 2, and 3. The plurality of sub-blocks may further include one or more other sub-blocks having a size of W×H.

[0208] In an embodiment, if one of the width W' of another block and the height H' of another block is greater than the maximum transform size T, the other block may be divided into a plurality of sub-blocks including the block. The width W of the block may be the minimum value between W' and T, and the height H of the block may be the minimum value between H' and T. One or a combination of the transform partition information of the block, the size of the block, and the shape of the block may include transform partition information of the block indicating the partition depth of the block. If the partition depth of the block is greater than a threshold, it may be determined that secondary transform is disabled for the block. Exemplary values of the threshold include, but are not limited to, 0, 1, 2, and 3.

[0209] In an embodiment, the width W' of the other block and / or the height H' of the other block is greater than a predetermined constant K. The other block may be implicitly divided into a plurality of sub-blocks. Exemplary values of K may include, but are not limited to, 16, 32, 64, 128, and 256. The secondary transform is applied only to one or more of the plurality of sub-blocks, and / or one or more secondary transform-associated information (e.g., one or more secondary transform indices) is signaled only for one or more of the plurality of sub-blocks having a minimum width W of W' and a minimum height H of H' and a minimum height H of K. One or more of the plurality of sub-blocks includes the block.

[0210] In an embodiment, one of the width W' of the other block and the height H' of the other block is greater than a predetermined constant K. The other block may be divided into a plurality of sub-blocks including the block. The width W of the block may be the minimum value of W' and K, and the height H of the block may be the minimum value of H' and K. One or a combination of the transform partition information of the block, the size of the block, and the shape of the block may include the size of the block having W and H. It may be determined that a secondary transform is enabled for blocks having block sizes of W and H. In an example, it is determined that a secondary transform is applied to blocks having block sizes of W and H.

[0211] Figure 19 A flowchart outlining a process (1900) according to an embodiment of the present disclosure is shown. The process (1900) can be used for reconstruction of blocks (such as CB, TB, luma CB, luma TB, chroma CB, chroma TB, etc.). In various embodiments, the process (1900) is performed by a processing circuit, such as a processing circuit in a terminal device (310), (320), (330) and (340), a processing circuit that performs the functions of a video encoder (403), a processing circuit that performs the functions of a video decoder (410), a processing circuit that performs the functions of a video decoder (510), a processing circuit that performs the functions of a video encoder (603), and the like. In some embodiments, the process (1900) is implemented using software instructions, so that when the processing circuit executes the software instructions, the processing circuit performs the process (1900). The process starts at (S1901) and proceeds to (S1910).

[0212] At (S1910), coding information of a block (e.g., CB, luma CB, chroma CB, intra coded CB, TB, etc.) may be decoded from the coded video stream. The coding information may indicate an intra prediction mode of the block and one or a combination of the following: transform partition information of the block, block size, and block shape. The transform partition information of the block may include a partition depth of the block.

[0213] At (S1920), whether to disable secondary transform for the block can be determined based on one or a combination of transform partition information of the block, the size of the block, and the shape of the block. In some examples, whether to signal information associated with the secondary transform (e.g., a secondary transform index) depends on one or a combination of the transform partition information of the block, the size of the block, and the shape of the block.

[0214] In an example, whether to disable the secondary transform for a block depends on the transform partition information of the block (e.g., the partition depth of the block). In an example, if the partition depth is greater than a threshold n (e.g., 0 or a positive integer), it is determined that the secondary transform is disabled and the secondary transform index is not signaled. In an example, the threshold n is 0.

[0215] In an example, the transform partition information of a block may be signaled, the block may have a non-square rectangular shape, and the block may be further partitioned into multiple TUs or TBs. Accordingly, whether to disable the secondary transform of the block may depend on the transform partition information of the block (e.g., the partition depth of the block).

[0216] Whether to apply a secondary transform to a block may depend on the shape of the block (e.g., the aspect ratio of the block). In some examples, whether to signal information associated with the secondary transform (e.g., the secondary transform index stIdx) depends on the shape of the block (e.g., the aspect ratio of the block).

[0217] In an embodiment, a block may be partitioned into multiple TUs or TBs. The shape of the block may be square. Accordingly, whether to disable the secondary transform of the block may depend on the transform partition information of the block (e.g., the partition depth of the block). In an example, the transform partition information of the block is signaled. In some examples, whether to signal information associated with the secondary transform (e.g., the secondary transform index) depends on the transform partition information of the block (e.g., the partition depth of the block).

[0218] In an embodiment, the size of the block (e.g., the minimum of the block width and the block height) may be higher than a threshold size (e.g., 64, 128, 256, etc.). The block may be partitioned into multiple TUs or TBs. In an example, transform partition information of the block is also signaled. Accordingly, whether to disable the secondary transform of the block may depend on the transform partition information of the block (e.g., the partition depth of the block). In some examples, whether to signal information associated with the secondary transform (e.g., the secondary transform index) depends on the transform partition information of the block (e.g., the partition depth of the block).

[0219] At (S1930), the block may be reconstructed based on whether the secondary transform is disabled for the block. In an example, at (S1920), it is determined that the secondary transform is disabled for the block, and thus the block may be reconstructed using only the primary transform without the secondary transform.

[0220] In the example, at (S1920), it is determined that a secondary transform is enabled for the block, and therefore at (S1930), if it is determined that a secondary transform is applied to the block, the block can be reconstructed using the primary transform and the secondary transform. If the partition depth of the block is greater than a threshold n (where n is 0 or a positive integer) and the block is partitioned into multiple TUs (or TBs), different secondary transforms can be applied to the multiple TUs (or TBs) respectively. The corresponding secondary transform index can be used to further indicate (e.g., signaled in the coded video code stream) which secondary transform (e.g., which secondary transform kernel) is applied to each TU (or TB). The process (1900) proceeds to (S1999) and ends.

[0221] The process (1900) may be modified as appropriate. One or more steps in the process (1900) may be modified and / or omitted. One or more additional steps may be added. Any suitable implementation order may be used. In an embodiment, the width W′ of another block and the height H′ of another block are greater than the maximum transform size T, and the other block may be implicitly split into a plurality of sub-blocks including the block. The width W of the block may be the minimum of W′ and T, and the height H of the block may be the minimum of H′ and T. One or a combination of the transform partition information of the block, the size of the block, and the shape of the block may include transform partition information of the block indicating the partition depth of the block. If the partition depth of the block is greater than a threshold, it may be determined that secondary transform is disabled for the block. The partition depth may be signaled. Exemplary values of the threshold include, but are not limited to, 0, 1, 2, and 3.

[0222] In an embodiment, a width W' of another block and a height H' of another block are greater than a predetermined constant K. The other block may be divided into a plurality of sub-blocks including the block. The width W of the block may be a minimum value between W' and K, and the height H of the block may be a minimum value between H' and K. One or a combination of the transform partition information of the block, the size of the block, and the shape of the block may include the size of the block having W and H. It may be determined that the secondary transform is applied only to blocks of sizes W and H.

[0223] When multiple transforms are applied to a block, the above description of determining whether to disable secondary transforms for the block and / or whether to signal information associated with the secondary transforms (e.g., secondary transform indices) may be appropriately modified. In an example, the block is partitioned into multiple TBs, and the multiple TBs may be transformed using multiple transforms, respectively. The multiple transforms may include multiple primary transforms. The multiple transforms may include multiple secondary transforms. The information associated with the multiple secondary transforms may include multiple secondary transform indices indicating the multiple secondary transforms, respectively. As described above, whether to disable multiple secondary transforms for the block and / or whether to signal information associated with the multiple secondary transforms (e.g., multiple secondary transform indices) may be determined based on one or a combination of the transform partition information of the block, the size of the block, and the shape of the block.

[0224] In an example, whether to disable multiple secondary transforms for a block and / or whether to signal multiple secondary transform indices associated with the multiple secondary transforms may be determined based on transform partition information of the block (e.g., partition depth). For example, if the partition depth is greater than a threshold value n (e.g., 0 or a positive integer), it is determined that multiple secondary transforms are disabled for the block, and the multiple secondary transform indices are not signaled. In an example, whether to disable multiple secondary transforms for the block and / or whether to signal multiple secondary transform indices associated with the multiple secondary transforms may be determined based on the shape of the block (e.g., aspect ratio).

[0225] The embodiments of the present disclosure may be used alone or in any combination. Furthermore, each of the method (or embodiment), encoder, and decoder may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-volatile computer-readable medium. The embodiments of the present disclosure may be applied to luminance blocks or chrominance blocks.

[0226] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. For example, Figure 20 A computer system (2000) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0227] Computer software may be encoded using any suitable machine code or computer language that may be assembled, compiled, linked, or similar mechanisms to create code comprising instructions that may be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or through interpretation, microcode execution, etc.

[0228] These instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, IoT devices, and the like.

[0229] Figure 20 The components shown for the computer system (2000) are exemplary in nature and are not intended to suggest any limitation on the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The configuration of components should not be interpreted as having any dependency or requirement on any one or combination of components illustrated in the exemplary embodiment of the computer system (2000).

[0230] The computer system (2000) may include certain human interface input devices. Such human interface input devices may be responsive to input from one or more human users through, for example, tactile input (such as keystrokes, swipes, data glove motions), audio input (such as voice, taps), visual input (such as gestures), and olfactory input (not depicted). Human interface devices may also be used to capture certain media that is not necessarily directly related to conscious human input, such as audio (such as voice, music, ambient sounds), images (such as scanned images, photographic images obtained from a still camera), and video (such as two-dimensional video, three-dimensional video including stereoscopic video).

[0231] The input human interface devices may include one or more of the following (only one of each is depicted): keyboard (2001), mouse (2002), touchpad (2003), touch screen (2010), data gloves (not shown), joystick (2005), microphone (2006), scanner (2007), camera (2008).

[0232] The computer system (2000) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses through, for example, tactile output, sound, light, and smell / taste. Such human interface output devices may include tactile output devices (e.g., tactile feedback through a touch screen (2010), a data glove (not shown), or a joystick (2005), although there may also be tactile feedback devices that do not function as input devices), audio output devices (such as speakers (2009), headphones (not depicted)), visual output devices (such as for screens (2010) including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities, some of which are capable of outputting two-dimensional visual output or more than three-dimensional output through means such as stereo output; virtual reality glasses (not depicted), holographic displays, and smoke canisters (not depicted), and printers (not depicted).

[0233] The computer system (2000) may also include human-accessible storage devices and their associated media, such as optical media including media (2021) such as CD / DVD ROM / RW (2020) with CD / DVD, thumb drives (2022), removable hard drives or solid-state drives (2023), traditional magnetic media such as tapes and floppy disks (not depicted), dedicated ROM / ASIC / PLD-based devices such as security dongles (not depicted), and the like.

[0234] Those skilled in the art will also understand that the term "computer-readable media" used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other volatile signals.

[0235] The computer system (2000) may also include an interface (2054) to one or more communication networks (2055). The network may be, for example, wireless, wired, or optical. The network may further be local, wide-area, metropolitan, vehicular, industrial, real-time, delay-tolerant, and the like. Examples of networks include local area networks (such as Ethernet), wireless LANs, cellular networks (including GSM, 3G, 4G, 5G, LTE, and the like), television wired or wireless wide-area digital networks (including cable, satellite, and terrestrial broadcast television), vehicle and industrial networks (including CANBus), and the like. Some networks typically require an external network interface adapter attached to some common data port or peripheral bus (2049) (such as a USB port of the computer system (2000)); other networks are typically integrated into the core of the computer system (2000) by attaching to a system bus as described below (e.g., an Ethernet interface integrated into a PC computer system or a cellular network interface integrated into a smartphone computer system). Using any of these networks, the computer system (2000) can communicate with other entities. Such communication can be one-way, receive-only (e.g., broadcast TV), one-way send-only (e.g., CANbus to certain CANbus devices), or two-way (e.g., to other computer systems using a local area digital network or a wide area digital network). Certain protocols and protocol stacks can be used on each of the networks and network interfaces as those described above.

[0236] The above-mentioned human interface device, human-accessible storage device, and network interface may be attached to the kernel (2040) of the computer system (2000).

[0237] The core (2040) may include one or more central processing units (CPUs) (2041), graphics processing units (GPUs) (2042), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (2043), hardware accelerators for certain tasks (2044), a graphics adapter (2050), and the like. These devices, along with read-only memory (ROM) (2045), random access memory (2046), and internal mass storage devices (2047) such as internal non-user accessible hard drives, solid-state drives, and the like, may be connected via a system bus (2048). In some computer systems, the system bus (2048) may be accessible in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, and the like. Peripheral devices may be attached directly to the core's system bus (2048) or to the system bus (2048) via a peripheral bus (2049). In an example, a screen (2010) may be connected to a graphics adapter (2050). Peripheral bus architectures include PCI, USB, and the like.

[0238] The CPU (2041), GPU (2042), FPGA (2043), and accelerator (2044) can execute certain instructions, the combination of which can constitute the aforementioned computer code. The computer code can be stored in ROM (2045) or RAM (2046). Transient data can also be stored in RAM (2046), while permanent data can be stored in, for example, an internal mass storage device (2047). Fast storage and retrieval of any memory device can be enabled by using a cache memory, which can be closely associated with one or more CPUs (2041), GPUs (2042), mass storage devices (2047), ROM (2045), RAM (2046), etc.

[0239] The computer readable medium may have computer code thereon for performing various computer-implemented operations. The media and computer code may be those specially designed and constructed for the purposes of this disclosure, or they may be of a type well known and available to those skilled in the art of computer software.

[0240] As an example and not by way of limitation, a computer system having the architecture (2000) and, in particular, the kernel (2040) can provide functionality as a result of one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage devices as described above, as well as certain storage devices of the kernel (2040) having non-volatile properties (such as kernel internal mass storage devices (2047) or ROM (2045)). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the kernel (2040). Depending on specific needs, the computer-readable medium can include one or more memory devices or chips. The software can cause the kernel (2040) and, in particular, the processors therein (including CPUs, GPUs, FPGAs, etc.) to perform specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM (2046) and modifying such data structures according to software-defined processes. In addition or as an alternative, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (2044)) that may operate in place of or in conjunction with software to perform specific processes or specific portions of specific processes described herein. Where appropriate, references to software may include logic and vice versa. Where appropriate, references to computer-readable media may include circuitry (such as an integrated circuit (IC)) storing software for execution, circuitry including logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software.

[0241] Appendix A: Acronyms

[0242] JEM: Joint Exploration Model

[0243] VVC: Universal Video Codec

[0244] BMS: Benchmark Set

[0245] MV: Motion Vector

[0246] HEVC: High-Efficiency Video Codec

[0247] SEI: Supplemental Enhancement Information

[0248] VUI: Video Availability Information

[0249] GOP: Group of Pictures

[0250] TU: Transform Unit

[0251] PU: prediction unit

[0252] CTU: Coding Tree Unit

[0253] CTB: Coding Tree Block

[0254] PB: prediction block

[0255] HRD: Hypothesized Reference Decoder

[0256] SNR: Signal-to-Noise Ratio

[0257] CPU: Central Processing Unit

[0258] GPU: Graphics Processing Unit

[0259] CRT: cathode ray tube

[0260] LCD: Liquid Crystal Display

[0261] OLED: Organic Light-Emitting Diode

[0262] CD: compact disc

[0263] DVD: Digital Video Disc

[0264] ROM: Read-Only Memory

[0265] RAM: Random Access Memory

[0266] ASIC: Application-Specific Integrated Circuit

[0267] PLD: Programmable Logic Device

[0268] LAN: Local Area Network

[0269] GSM: Global System for Mobile Communications

[0270] LTE: Long Term Evolution

[0271] CANBus: Controller Area Network Bus

[0272] USB: Universal Serial Bus

[0273] PCI: Peripheral Component Interconnect

[0274] FPGA: Field Programmable Gate Array

[0275] SSD: Solid State Drive

[0276] IC: integrated circuit

[0277] CU: Coding Unit

[0278] Although the present disclosure has described several exemplary embodiments, there are changes, permutations, and various substitute equivalents that fall within the scope of the present disclosure. Therefore, it should be understood that those skilled in the art will be able to devise many systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure.

Claims

1. A method for video decoding, characterized in that: include: Decoding encoding information of a block from an encoded video stream, where the encoding information indicates an intra-frame prediction mode of the block, and one or a combination of transform partition information of the block, a size of the block, and a shape of the block; determining whether to disable secondary transform for the block based on one or a combination of transform partition information of the block, a size of the block, and a shape of the block; wherein when secondary transform is disabled for the block, a secondary transform index is not signaled in the coded video stream; and The block is reconstructed based on whether the secondary transform is disabled for the block.

2. The method according to claim 1, characterized in that One or a combination of the transform partition information of the block, the size of the block, and the shape of the block comprises the transform partition information of the block, and the transform partition information of the block is signaled in the coded video stream. The transform partition information of the block indicates a partition depth of the block, The method further comprises partitioning the block into a plurality of transform blocks; as well as The determining whether to disable secondary transform for the block includes determining whether to disable secondary transform for the block based on the partition depth.

3. The method according to claim 2, characterized in that The determining whether to disable secondary transform for the block comprises: In response to the partition depth being greater than a threshold, it is determined to disable secondary transform for the block, where the threshold is 0 or a positive integer.

4. The method according to claim 3, characterized in that The threshold is 0.

5. The method according to claim 1, wherein One or a combination of the transform partition information of the block, the size of the block, and the shape of the block includes the transform partition information of the block and the shape of the block, the transform partition information is signaled in the coded video bitstream, the transform partition information indicates a partition depth of the block, and the shape of the block is a non-square rectangle, The method further comprises partitioning the block into a plurality of transform blocks; and The determining whether to disable secondary transform for the block includes determining whether to disable secondary transform for the block based on the partition depth.

6. The method according to claim 5, characterized in that The determining whether to disable the secondary transform for the block includes: in response to the partition depth being greater than a threshold, determining to disable the secondary transform for the block, where the threshold is 0 or a positive integer.

7. The method according to claim 1, characterized in that One or a combination of transform partition information of the block, a size of the block, and a shape of the block includes a shape of the block indicated by an aspect ratio of the block, and The determining whether to disable the secondary transform for the block includes determining whether to disable the secondary transform for the block based on an aspect ratio of the block.

8. The method according to claim 7, characterized in that The aspect ratio of the block is a ratio of a first dimension of the block to a second dimension of the block, the first dimension of the block being greater than or equal to the second dimension, and The determining whether to disable the secondary transform for the block includes determining to disable the secondary transform for the block in response to an aspect ratio of the block being greater than a threshold.

9. The method according to claim 1, characterized in that One or a combination of the transform partition information of the block, the size of the block, and the shape of the block includes the transform partition information and the shape of the block, the transform partition information indicates a partition depth, and the shape of the block is a square. The method further comprises: partitioning the block into a plurality of transform blocks; and The determining whether to disable secondary transform for the block includes determining whether to disable secondary transform for the block based on the partition depth.

10. The method according to claim 9, characterized in that The determining whether to disable the secondary transform for the block includes: in response to the partition depth being greater than a threshold, determining to disable the secondary transform for the block, where the threshold is 0 or a positive integer.

11. The method according to claim 1, wherein One or a combination of the transform partition information of the block, the size of the block, and the shape of the block includes the transform partition information of the block and the size of the block, the transform partition information indicating a partition depth of the block, the size of the block indicating a width of the block and a height of the block, the width and the height being greater than a threshold size; The method further comprises: partitioning the block into a plurality of transform blocks; and The determining whether to disable secondary transform for the block includes determining whether to disable secondary transform for the block based on the partition depth of the block.

12. The method according to claim 11, characterized in that The determining whether to disable the secondary transform for the block includes: in response to the partition depth being greater than a threshold, determining to disable the secondary transform for the block, where the threshold is zero or a positive integer.

13. The method according to any one of claims 1 to 12, characterized in that The block is a subblock of another block, one of the width W' and the height H' of the other block is larger than the maximum transform size T, The method further comprises: dividing the another block into a plurality of sub-blocks including the block, wherein a width W of the block is a minimum value between W' and T, and a height H of the block is a minimum value between H' and T, One or a combination of the transform partition information of the block, the size of the block, and the shape of the block includes transform partition information of the block, the transform partition information indicating a partition depth of the block, and The determining whether to disable the secondary transform for the block includes determining to disable the secondary transform for the block in response to a partition depth of the block being greater than a threshold.

14. The method according to any one of claims 1 to 12, characterized in that The block is a sub-block of another block, one of a width W' and a height H' of the other block is greater than a predetermined constant K, The method further comprises: dividing the another block into a plurality of sub-blocks including the block, wherein a width W of the block is a minimum value between W' and K, and a height H of the block is a minimum value between H' and K, One or a combination of the transform partition information of the block, the size of the block, and the shape of the block includes the size of the block, the size of the block being W and H, and The determining whether to disable the secondary transform for the block includes determining, in response to sizes of the block being W and H, to enable the secondary transform for the block.

15. A device for video decoding, characterized in that: include: a processing circuit, the processing circuit being configured to: Decoding encoding information of a block from an encoded video stream, where the encoding information indicates an intra-frame prediction mode of the block, and one or a combination of transform partition information of the block, a size of the block, and a shape of the block; determining whether to disable secondary transform for the block based on one or a combination of transform partition information of the block, a size of the block, and a shape of the block; wherein, when secondary transform is disabled for the block, a secondary transform index is not signaled in the coded video stream; and The block is reconstructed based on whether the secondary transform is disabled for the block.

16. A device for video decoding, characterized in that: The device comprises: at least one memory configured to store program code; and At least one processor is configured to access the at least one memory and operate according to the program code to implement the method according to any one of claims 1 to 14.

17. A non-volatile computer-readable storage medium storing instructions, characterized in that: The instructions cause at least one processor to execute the method according to any one of claims 1-14.

18. A video encoding method, characterized in that: The method comprises: determining an intra prediction mode for the block; determining one or a combination of transform partition information of the block, a size of the block, and a shape of the block; determining whether to disable secondary transform for the block based on one or a combination of transform partition information of the block, a size of the block, and a shape of the block; when it is determined that secondary transform is disabled for the block, not signaling a secondary transform index in the video code stream; and The block is encoded based on whether the secondary transform is disabled for the block.

19. A computer storage medium, characterized in that A code stream formed by a computer program is stored thereon, and when the computer program is executed by a computer, the computer is caused to execute the method according to any one of claims 1 to 14.

20. A computer storage medium, characterized in that Instructions are stored, and the instructions can be executed by at least one processor to perform the video encoding method according to claim 18, generate a code stream and store it.

Images