Video coding with training-based coding tools

Abstract

An example method of video coding includes receiving a video bitstream including a current block, and identifying a first prediction mode of the current block. The method also includes selecting a first set of transform kernels as transform kernels for the current block when the first prediction mode is a particular prediction mode, and selecting a second set of transform kernels as the transform kernels for the current block when the first prediction mode is not the particular prediction mode. The method further includes applying a transform to the current block using the transform kernels.

Description

[0001] Related applications

[0002] This application is a continuation-to-file of U.S. Patent Application No. 19 / 023,018, filed January 15, 2025, entitled "Video Coding with Training-based Coding Tool," which claims priority to U.S. Provisional Patent Application No. 63 / 623,197, filed January 19, 2024, entitled "Video Coding with Training-based Coding Tool," and U.S. Provisional Patent Application No. 63 / 633,563, filed April 12, 2024, entitled "Method and Apparatus for Selective Transform Set based on Decoder-Side Intra Mode Derivation," the entire contents of each of which are incorporated herein by reference. Technical Field

[0003] The disclosed embodiments generally relate to video encoding and decoding, including but not limited to systems and methods for selecting transform kernels. Background Technology

[0004] Digital video is supported by a variety of electronic devices such as digital televisions, laptops or desktop computers, tablets, digital cameras, digital recording devices, digital media players, video game consoles, smartphones, video conferencing equipment, and video streaming devices. Electronic devices transmit and receive digital video data via communication networks or otherwise transmit digital video data, and / or store digital video data on storage devices. Due to the limited bandwidth capacity of communication networks and the limited memory resources of storage devices, video encoding can be used to compress video data according to one or more video encoding standards before transmission or storage. Video encoding and decoding can be performed by hardware and / or software on servers or electronic / client devices providing cloud services.

[0005] Video coding typically uses prediction methods that leverage the inherent redundancy in video data (e.g., inter-frame prediction, intra-frame prediction, etc.). Video coding aims to compress video data into a form using a lower bitrate while avoiding or minimizing degradation in video quality. Several video codec standards have been developed. For example, High Efficiency Video Coding (HEVC / H.265) is a video compression standard designed as part of the MPEG-H project. The ITU-T and ISO / IEC published the HEVC / H.265 standard in 2013 (Revision 1), 2014 (Revision 2), 2015 (Revision 3), and 2016 (Revision 4). Universal Video Coding (VVC / H.266) is a video compression standard designed as a successor to HEVC. The ITU-T and ISO / IEC published the VVC / H.266 standard in 2020 (Revision 1) and 2022 (Revision 2). The Open Media Alliance Video 1 (AV1) is an open video coding format designed as an alternative to HEVC. On January 8, 2019, a confirmed version 1.0.0 with errata table 1 was released. Enhanced Compression Model (ECM) is a video coding standard under development. ECM aims to surpass existing standards such as HEVC / H.265 and VVC, significantly improving compression efficiency to achieve higher quality video at lower bitrates. ECM version 13 was released on July 7, 2024, at the 146th MPEG meeting. Summary of the Invention

[0006] This disclosure specifically describes a set of techniques for video (image) compression that relates to the selection of prediction modes and transform sets for corresponding residual blocks. Some embodiments include adaptively generating an intra predictor using decoder-side intra-mode derivation (DIMD). The system (e.g., the decoder) can then examine the intra predictor determined using DIMD to determine if it is within an intra-mode replacement set (e.g., this set can be hard-coded or accessed via a lookup table). If the intra predictor is within the replacement set, the final intra predictor can be derived using interpolation or matrix multiplication. Deriving the intra predictor in this manner can improve coding accuracy (resulting in a better intra predictor) compared to conventional intra predictor derivation (e.g., without utilizing a replacement set or performing a different final intra predictor derivation). Some embodiments include constructing a list of intra-modes for the current block (e.g., a list of most probable modes (MPMs)) using the intra-modes of adjacent blocks. For example, if adjacent intra-modes are position-dependent intra-prediction (PDPs), the PDP can be added to the intra-mode list (e.g., in place of the corresponding regular intra-mode or in addition to the corresponding regular intra-mode). Mapping (non-regular) intra-modes from adjacent blocks to construct the intra-mode list can improve coding accuracy (e.g., by increasing diversity in the intra-mode list). Some embodiments include using different sets of transform kernels (e.g., primary transform kernels or secondary transform kernels) based on whether DIMD is used on the current block. For example, the transform kernels may correspond to inseparable primary transforms or secondary transforms. Using different transform kernels based on whether DIMD is applied can improve coding accuracy (e.g., selecting the most appropriate kernel based on intra-mode derivation). Some embodiments include combining the intra-prediction step with the inseparable primary transform step (e.g., applying a single combination matrix instead of two separate matrices). Combining intra-prediction and inseparable primary transform steps can reduce computational complexity and coding time. Some embodiments include using the techniques described herein with other matrix-based encoding / decoding tools (e.g., improving the accuracy of these tools in a similar manner).

[0007] According to some embodiments, a video decoding method includes: (i) receiving a video bitstream (e.g., an encoded video sequence) including a current block; (ii) identifying a first prediction mode (e.g., using DIMD) for the current block; (iii) selecting a first set of transform kernels as the transform kernel for the current block when the first prediction mode is a specific prediction mode; (iv) selecting a second set of transform kernels as the transform kernel for the current block when the first prediction mode is not a specific prediction mode; and (v) applying a transform to the current block using the transform kernels.

[0008] According to some embodiments, a video decoding method includes: (i) receiving a video bitstream including a current block; (ii) identifying a first prediction mode of the current block using DIMD technology; (iii) determining whether the first prediction mode is in an intra-mode replacement set; (iv) when the first prediction mode is in the intra-mode replacement set, using a first technique to generate an intra-predictor; and (v) when the first prediction mode is not in the intra-mode replacement set, using a second technique to generate an intra-predictor.

[0009] According to some embodiments, a video decoding method includes: (i) receiving a video bitstream including a current block; (ii) identifying a first prediction mode for the current block; and (iii) when the first prediction mode is a specific prediction mode, generating a transformed residual block for the current block without performing prediction block computation.

[0010] According to some embodiments, a video coding method includes: (i) receiving video data (e.g., a source video sequence) comprising a plurality of blocks, the plurality of blocks including a current block; (ii) identifying a first prediction mode for the current block; (iii) selecting a first set of transform kernels as the transform kernel for the current block when the first prediction mode is a specific prediction mode; (iv) selecting a second set of transform kernels as the transform kernel for the current block when the first prediction mode is not a specific prediction mode; and (v) applying a transform to the current block using the transform kernels.

[0011] According to some embodiments, a computing system, such as a streaming system, server system, personal computer system, or other electronic device, is provided. The computing system includes control circuitry and a memory storing one or more sets of instructions. The one or more sets of instructions include instructions for performing any of the methods described herein. In some embodiments, the computing system includes encoder components and decoder components (e.g., a transcoder). According to some embodiments, a non-volatile computer-readable storage medium is provided. The non-volatile computer-readable storage medium stores one or more sets of instructions for execution by the computing system. The one or more sets of instructions include instructions for performing any of the methods described herein.

[0012] Therefore, apparatus and systems having methods for encoding and decoding video are disclosed. Such methods, apparatus, and systems may complement or replace conventional methods, devices, and systems for video encoding / decoding. The features and advantages described in the specification are not necessarily exhaustive, and in particular, some additional features and advantages will be apparent to those skilled in the art from the accompanying drawings, specification, and claims provided in this disclosure. Furthermore, it should be noted that the language used in the specification has been chosen primarily for readability and instruction purposes and is not necessarily intended to depict or limit the subject matter described herein. Attached Figure Description

[0013] To gain a more detailed understanding of this disclosure, reference can be made to the features of various embodiments, some of which are illustrated in the accompanying drawings. However, the drawings only illustrate relevant features of this disclosure and are therefore not necessarily to be considered limiting, as those skilled in the art will understand upon reading this disclosure that other valid features may be permitted.

[0014] Figure 1 A block diagram of an example communication system according to some embodiments is shown.

[0015] Figure 2A A block diagram of example elements of an encoder component according to some embodiments is shown.

[0016] Figure 2B A block diagram of example elements of a decoder component according to some embodiments is shown.

[0017] Figure 3 A block diagram of an example server system according to some embodiments is shown.

[0018] Figures 4A to 4C Example prediction blocks, residual blocks, and reconstruction blocks are shown according to some embodiments.

[0019] Figure 4D An example directional intra-frame prediction mode according to some embodiments is shown.

[0020] Figure 4E Example sample regions for matrix multiplication prediction are shown according to some embodiments.

[0021] Figure 5A An example low-frequency inseparable transformation process according to some embodiments is shown.

[0022] Figure 5B An example transformation process involving a quadratic transformation is shown according to some embodiments.

[0023] Figure 5C An example is shown of deriving an intra-frame prediction mode using reconstructed samples from a template, according to some embodiments.

[0024] Figure 6A An example video decoding method according to some embodiments is shown.

[0025] Figure 6B An example video encoding method according to some embodiments is shown.

[0026] By convention, the various features shown in the accompanying drawings are not necessarily drawn to scale, and the same reference numerals may be used to denote the same features throughout the specification and the accompanying drawings. Detailed Implementation

[0027] This disclosure describes video / image compression techniques that include selecting transform kernels based on intra-prediction modes (and / or intra-prediction mode derivation). Some embodiments include identifying a first prediction mode (and / or prediction mode derivation) for the current block, and selecting a first set of transform kernels as the transform kernel for the current block when the first prediction mode is a specific prediction mode, and selecting a second set of transform kernels as the transform kernel for the current block when the first prediction mode is not a specific prediction mode. Selecting transform kernels based on the prediction mode (and / or prediction mode derivation techniques) for the current block can improve coding accuracy (by using prediction information to select a more appropriate transform kernel).

[0028] Some embodiments include using decoder-side intra-mode derivation (DIMD) to identify a first prediction mode for the current block and to determine whether the first prediction mode is in the intra-mode replacement set. When the first prediction mode is in the intra-mode replacement set, a first technique is used to generate an intra-predictor; when the first prediction mode is not in the intra-mode replacement set, a second technique is used to generate an intra-predictor. Generating different intra-predictors based on whether the prediction mode is in the mode replacement set can improve coding accuracy (by selecting a more accurate intra-predictor).

[0029] Some embodiments include identifying a first prediction pattern for the current block, and when the first prediction pattern is a specific prediction pattern, generating a transformed residual block of the current block without performing prediction block computation. Generating the transformed residual block in a single step can reduce computational complexity and time (e.g., performing a single matrix multiplication operation instead of multiple sequential matrix multiplication operations).

[0030] Example systems and devices

[0031] Figure 1 This is a block diagram illustrating a communication system 100 according to some embodiments. The communication system 100 includes a source device 102 and a plurality of electronic devices 120 (e.g., electronic devices 120-1 to 120-m) that are communicatively coupled to each other via at least one network. In some embodiments, the communication system 100 is, for example, a streaming system used with video-enabled applications such as video conferencing applications, digital television applications, and media storage devices and / or distribution applications.

[0032] Source device 102 includes a video source 104 (e.g., a camera component or media storage device) and an encoder component 106. In some embodiments, the video source 104 is a digital camera (e.g., configured to create an uncompressed video sample stream). The encoder component 106 generates at least one encoded video stream from the video stream. The video stream from video source 104 can have a high data volume compared to the encoded video stream 108 generated by encoder component 106. Because the encoded video stream 108 has a lower data volume (less data) compared to the video stream from video source 104, it requires less bandwidth for transmission and less storage space for storage. In some embodiments, source device 102 does not include encoder component 106 (e.g., configured to transmit uncompressed video to at least one network 110).

[0033] At least one network 110 refers to any number of networks that transmit information between source device 102, server system 112, and / or electronic device 120, including, for example, wired (wired) and / or wireless communication networks. At least one network 110 can exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet.

[0034] At least one network 110 includes a server system 112 (e.g., a distributed / cloud computing system). In some embodiments, the server system 112 is or includes a streaming server (e.g., configured to store and / or distribute video content such as encoded video streams from source device 102). The server system 112 includes an encoder component 114 (e.g., configured to encode and / or decode video data). In some embodiments, the encoder component 114 includes encoder components and / or decoder components. In various embodiments, the encoder component 114 is instantiated as hardware, software, or a combination thereof. In some embodiments, the encoder component 114 is configured to decode the encoded video stream 108 and re-encode the video data using different encoding standards and / or methods to generate encoded video data 116. In some embodiments, the server system 112 is configured to generate multiple video formats and / or encodings from the encoded video stream 108. In some embodiments, the server system 112 serves as a media-aware network element (MANE). For example, the server system 112 may be configured to trim the encoded video stream 108 to cut potentially different streams for at least one of the electronic devices 120. In some embodiments, the MANE is provided separately from the server system 112.

[0035] Electronic device 120-1 includes a decoder component 122 and a display 124. In some embodiments, the decoder component 122 is configured to decode encoded video data 116 to generate an output video stream that can be rendered on a display or other type of rendering device. In some embodiments, at least one of the electronic devices 120 does not include a display component (e.g., communicatively coupled to an external display device and / or includes a media storage device). In some embodiments, electronic device 120 is a streaming client. In some embodiments, electronic device 120 is configured to access server system 112 to obtain encoded video data 116.

[0036] The source device and / or multiple electronic devices 120 are sometimes referred to as “terminal devices” or “user devices”. In some embodiments, at least one of the source device 102 and / or electronic devices 120 is an example of a server system, a personal computer, a portable device (e.g., a smartphone, tablet, or laptop), a wearable device, a video conferencing device, and / or other types of electronic devices.

[0037] In an example operation of communication system 100, source device 102 transmits an encoded video stream 108 to server system 112. For example, source device 102 may encode a stream of images captured by the source device. Server system 112 receives the encoded video stream 108 and may decode and / or encode the encoded video stream 108 using encoder component 114. For example, server system 112 may apply encoding to the video data that is optimized for network transmission and / or storage. Server system 112 may transmit encoded video data 116 (e.g., at least one encoded video stream) to at least one of electronic devices 120. Each electronic device 120 may decode the encoded video data 116 and optionally display video images.

[0038] Figure 2AThis is a block diagram illustrating example elements of an encoder component 106 according to some embodiments. The encoder component 106 receives video data (e.g., a source video sequence) from a video source 104. In some embodiments, the encoder component includes a receiver (e.g., transceiver) component configured to receive the source video sequence. In some embodiments, the encoder component 106 receives the video sequence from a remote video source (e.g., a video source that is a component of a device different from the encoder component 106). The video source 104 may provide the source video sequence as a digital video sample stream, which may have any suitable bit depth (e.g., 8-bit, 10-bit, or 12-bit), any color space (e.g., BT.601 YCrCB or RGB), and any suitable sampling structure (e.g., YCrCb 4:2:0 or YCrCb 4:4:4). In some embodiments, the video source 104 is a storage device storing previously captured / prepared video. In some embodiments, the video source 104 is a camera that captures local image information as a video sequence. The video data may be provided as multiple individual pictures that, when viewed sequentially, produce motion. An image itself can be organized as a spatial array of pixels, where each pixel includes at least one sample, depending on the sampling structure, color space, etc. Those skilled in the art can readily understand the relationship between pixels and samples.

[0039] Encoder component 106 is configured to encode and / or compress images of a source video sequence into an encoded video sequence 216 in real time or under other time constraints required by the application. In some embodiments, encoder component 106 is configured to perform a conversion between the source video sequence and the bitstream of visual media data (e.g., a video bitstream). Implementing an appropriate encoding rate is a function of controller 204. In some embodiments, controller 204 controls and is functionally coupled to other functional units as described below. Parameters set by controller 204 may include parameters related to rate control (e.g., image skipping, quantizer and / or rate distortion optimization techniques with λ values), image size, group of images (GOP) layout, maximum motion vector search range, etc. Other functions of controller 204 can be readily identified by those skilled in the art, as they may be related to encoder component 106 optimized for a particular system design.

[0040] In some embodiments, encoder component 106 is configured to operate within an encoding loop. In a simplified example, the encoding loop includes a source encoder 202 (e.g., responsible for creating symbols, such as a symbol stream, based on the input image to be encoded and a reference image) and a (local) decoder 210. Decoder 210 reconstructs the symbols in a manner similar to that of the (remote) decoder to create sampled data (when compression between the symbols and the encoded video bitstream is lossless). The reconstructed sample stream (sample data) is input to reference image memory 208. Because decoding of the symbol stream results in bit-accurate results independent of decoder location (local or remote), the contents of reference image memory 208 are also bit-accurate between the local encoder and the remote encoder. Thus, the prediction portion of the encoder interprets the same sample values ​​as the sample values ​​interpreted by the decoder during prediction as reference image samples.

[0041] The operation of decoder 210 can be the same as that of a remote decoder, such as decoder component 122, which will be discussed below. Figure 2B Detailed description. However, a brief reference is provided. Figure 2B Since symbols are available, and the encoding / decoding of symbols for the encoded video sequence by the entropy encoder 214 and the parser 254 can be lossless, the entropy decoding part of the decoder component 122, including the buffer memory 252 and the parser 254, may not be fully implemented in the local decoder 210.

[0042] Aside from the parsing / entropy decoding present in the decoder, the decoder techniques described herein can exist in the corresponding encoder with essentially the same functional form. Therefore, the subject matter disclosed focuses on decoder operations. Furthermore, the descriptions of encoder techniques can be abbreviated, as they are the inverse of the fully described decoder techniques.

[0043] As part of its operation, the source encoder 202 can perform motion-compensated predictive coding, which uses at least one previously encoded frame in the video sequence designated as a reference frame to predictively encode the input frame. In this way, the encoding engine 212 encodes the differences between pixel blocks of the input frame and pixel modules of the reference frame, which can be selected as a predictive reference for the input frame. The controller 204 can manage the encoding operations of the source encoder 202, including, for example, setting parameters and subgroup parameters for encoding video data.

[0044] Decoder 210 decodes encoded video data based on symbol pairs created by source encoder 202, which can be designated as reference frames. The operation of encoding engine 212 can advantageously be a lossy process. When in the video decoder ( Figure 2AWhen decoding the encoded video data at a location (not shown), the reconstructed video sequence may be a copy of the source video sequence with some errors. Decoder 210 replicates the decoding process that can be performed on the reference frame by a remote video decoder, and can store the reconstructed reference frame in reference picture memory 208. In this way, encoder component 106 locally stores a copy of the reconstructed reference frame with common content as a reconstructed reference frame (without transmission errors) to be obtained by the remote video decoder.

[0045] Predictor 206 can perform a prediction search on encoding engine 212. That is, for a new frame to be encoded, predictor 206 can search the reference image memory 208 for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., which can be used as appropriate prediction references for the new image. Predictor 206 can operate on sample blocks pixel by pixel to find appropriate prediction references. As determined by the search results obtained by predictor 206, the input image can have prediction references drawn from multiple reference images stored in reference image memory 208.

[0046] The outputs of all the above-described functional units can be entropy encoded in the entropy encoder 214. The entropy encoder 214 converts the symbols generated by the various functional units into an encoded video sequence by lossless compression of the symbols according to techniques known to those skilled in the art (e.g., Huffman coding, variable-length coding, and / or arithmetic coding).

[0047] In some embodiments, the output of entropy encoder 214 is coupled to a transmitter. The transmitter can be configured to buffer the encoded video sequences created by entropy encoder 214 in preparation for transmission via communication channel 218, which can be a hardware / software link to a storage device storing the encoded video data. The transmitter can be configured to combine encoded video data from source encoder 202 with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown). In some embodiments, the transmitter can send additional data along with the encoded video. Source encoder 202 can include such data as part of the encoded video sequence. Additional data can include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, supplementary enhancement information (SEI) messages, fragments of visual usability information (VUI) parameter sets, etc.

[0048] Controller 204 can manage the operation of encoder component 106. During encoding, controller 204 can assign a specific encoding picture type to each encoded picture, which may affect the encoding technique applied to the corresponding picture. For example, a picture can be designated as an intra-frame picture (I-picture), a prediction picture (P-picture), or a bidirectional prediction picture (B-picture). Intra-frame pictures can be encoded and decoded without using any other frames in the sequence as prediction sources. Some video codecs allow different types of intra-frame pictures, including, for example, Independent Decoder Refresh (IDR) pictures. These variations of I-pictures and their respective applications and characteristics are known to those skilled in the art, and therefore will not be repeated here. Predictive pictures can be encoded and decoded using intra-frame prediction or inter-frame prediction, using at most one motion vector and reference index to predict sample values ​​for each block. Bidirectional prediction pictures can be encoded and decoded using intra-frame prediction or inter-frame prediction, using at most two motion vectors and reference indices to predict sample values ​​for each block. Similarly, multiple predictive pictures can be used to reconstruct a single block using two or more reference pictures and associated metadata.

[0049] The source image can typically be spatially subdivided into multiple sample blocks (e.g., each sample block is 4×4, 8×8, 4×8, or 16×16) and encoded block by block. Blocks can be predicted by referencing other (already encoded) blocks determined by the encoding assignment applied to the corresponding image. For example, blocks of image I can be unpredictably encoded or predictively encoded by referencing already encoded blocks of the same image (spatial prediction or intra-frame prediction). Pixel blocks of images P can be unpredictably encoded by referencing a previously encoded reference image through spatial or temporal prediction. Blocks of image B can be unpredictably encoded by referencing one or two previously encoded reference images through spatial or temporal prediction.

[0050] Video can be captured as multiple source images (video frames) in a time series. Intra-frame prediction (often abbreviated as intra-frame prediction) utilizes spatial correlations within a given image, while inter-frame prediction utilizes (temporal or other) correlations between images. In one example, a specific image in the encoding / decoding process (called the current image) is divided into blocks. When a block in the current image is similar to a reference block in a previously encoded and still buffered reference image in the video, that block in the current image can be encoded using a vector called a motion vector. The motion vector points to the reference block in the reference image and, in the case of using multiple reference images, can have a third dimension identifying the reference images.

[0051] Encoder component 106 can perform encoding operations according to a predetermined video coding technique or standard (such as any described herein). In its operation, encoder component 106 can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

[0052] Figure 2B This is a block diagram illustrating example elements of a decoder component 122 according to some embodiments. Figure 2B The decoder component 122 is coupled to channel 218 and display 124. In some embodiments, the decoder component 122 includes a transmitter coupled to loop filter 256 and configured to transmit data to display 124 (e.g., via a wired or wireless connection).

[0053] In some embodiments, decoder component 122 includes a receiver coupled to channel 218 and configured to receive data from channel 218 (e.g., via a wired or wireless connection). The receiver may be configured to receive at least one encoded video sequence to be decoded by decoder component 122. In some embodiments, decoding of each encoded video sequence is independent of other encoded video sequences. Each encoded video sequence may be received from channel 218, which may be a hardware / software link to a storage device storing the encoded video data. The receiver may receive encoded video data with other data (e.g., encoded audio data and / or auxiliary data streams), which may be forwarded to their respective user entities (not depicted). The receiver may separate the encoded video sequences from other data. In some embodiments, the receiver receives additional (redundant) data accompanying the encoded video. The additional data may be included as part of at least one encoded video sequence. Decoder component 122 may use the additional data to decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0054] According to some embodiments, the decoder component 122 includes a buffer memory 252, a parser 254 (sometimes also called an entropy decoder), a scaler / inverse transform unit 258, an intra-frame prediction unit 262, a motion compensation prediction unit 260, an aggregator 268, a loop filter unit 256, a reference image memory 266, and a current image memory 264. In some embodiments, the decoder component 122 is implemented as an integrated circuit, a series of integrated circuits, and / or other electronic circuits. The decoder component 122 may be implemented at least partially in software.

[0055] Buffer memory 252 is coupled between channel 218 and parser 254 (e.g., to combat network jitter). In some embodiments, buffer memory 252 is separate from decoder component 122. In some embodiments, a separate buffer memory is provided between the output of channel 218 and decoder component 122. In some embodiments, a separate buffer memory is provided outside decoder component 122 (e.g., to combat network jitter) in addition to buffer memory 252 inside decoder component 122 (e.g., configured to handle playback timing). Buffer memory 252 may not be necessary when receiving data from a store / forward device with sufficient bandwidth and controllability or from an isochronous network, or buffer memory 252 may be very small. Buffer memory 252 may be necessary for use on best-effort packet networks such as the Internet; buffer memory 252 may be relatively large and / or have an adaptive size, and may be implemented at least partially outside decoder component 122 in an operating system or similar component.

[0056] Parser 254 is configured to reconstruct symbols 270 from the encoded video sequence. Symbols may include, for example, information for managing the operation of decoder component 122, and / or information for controlling rendering devices such as display 124. Control information for at least one rendering device may be in the form of, for example, Supplemental Enhancement Information (SEI) messages or Video Availability Information (VUI) parameter set fragments (not depicted). Parser 254 parses (entropy decodes) the encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. Parser 254 may extract a set of subgroup parameters for at least one subgroup of pixels in the video decoder from the encoded video sequence based on at least one parameter corresponding to that group. Subgroups may include picture groups (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. Parser 254 may also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc., from the encoded video sequence.

[0057] The reconstruction of symbol 270 can involve multiple different units, depending on the type of encoded video picture or its portions (such as inter- and intra-frame pictures, inter- and intra-frame blocks) and other factors. Which units are involved and how they are involved can be controlled by subgroup control information, which is parsed from the encoded video sequence by parser 254. For clarity, the flow of this subgroup control information between parser 254 and the multiple units is not depicted below.

[0058] The decoder component 122 can be conceptually subdivided into multiple functional units, many of which interact closely with each other and can be at least partially integrated with each other in some embodiments. However, for clarity, this document retains the conceptual subdivision of functional units.

[0059] The scaler / inverse transform unit 258 receives quantized transform coefficients and control information (such as the transform to be used, block size, quantization factor, and / or quantization scaling matrix) as at least one symbol 270 from the parser 254. The scaler / inverse transform unit 258 can output a block comprising sample values ​​that can be input into the aggregator 268.

[0060] In some cases, the output samples of the scaler / inverse transform unit 258 belong to intra-coded blocks; that is, blocks that do not use prediction information from previously reconstructed images but can use prediction information from previously reconstructed portions of the current image. This prediction information can be provided by the intra-prediction unit 262. The intra-image prediction unit 262 can use surrounding reconstructed information obtained from the current (partially reconstructed) image from the current image memory 264 to generate blocks with the same size and shape as the reconstructed blocks. The aggregator 268 can add the prediction information already generated by the intra-prediction unit 262 to the output sample information provided by the scaler / inverse transform unit 258 on a per-sample basis. In other cases, the output samples of the scaler / inverse transform unit 258 belong to inter-coded blocks that may be motion-compensated. In this case, the motion-compensated prediction unit 260 can access the reference image memory 266 to obtain samples for prediction. After motion compensation is performed on the acquired samples according to the symbol 270 belonging to the block, these samples can be added by the aggregator 268 to the output of the scaler / inverse transform unit 258 (referred to in this case as residual samples or residual signals) to generate output sample information. The address of the predicted sample obtained by the motion compensation prediction unit 260 from the reference image memory 266 can be controlled by the motion vector. The motion vector can be used by the motion compensation prediction unit 260 in the form of symbol 270, which can have, for example, X, Y, and reference image components. Motion compensation can also include interpolation of sample values ​​obtained from the reference image memory 266 when using subsample precise motion vectors, and a motion vector prediction mechanism.

[0061] The output samples of aggregator 268 can undergo various loop filtering techniques in loop filter unit 256. Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video bitstream and available to loop filter unit 256 as symbol 270 from parser 254, but may also be in response to metadata obtained during decoding of previous (in decoding order) portions of the encoded picture or encoded video sequence, and to sample values ​​obtained from previous reconstruction and loop filtering. The output of loop filter unit 256 can be a sample stream, which can be output to a rendering device such as display 124, and stored in reference picture memory 266 for use in future inter-frame prediction.

[0062] Once certain encoded images are reconstructed, they can be used as reference images for future predictions. Once an encoded image has been reconstructed and has been identified as a reference image (e.g., by parser 254), the current reference image can become part of the reference image memory 266, and a new current image memory can be reallocated before reconstructing subsequent encoded images begins.

[0063] Decoder component 122 can perform decoding operations according to a predetermined video compression technique that can be recorded in a standard (such as any of the standards described herein). The encoded video sequence can conform to the syntax specified by the video compression technique or standard used; in this sense, it adheres to the syntax of the video compression technique or standard, as specified in the video compression technique documentation or standard and, in particular, the brief document therein. Furthermore, the complexity of the encoded video sequence can be within the range defined by the level of the video compression technique or standard in order to conform to certain video compression techniques or standards. In some cases, the level limits the maximum image size, maximum frame rate, maximum reconstruction sampling rate (e.g., measured in megasamples per second), maximum reference image size, etc. In some cases, the limitations set by the level can be further restricted by the hypothetical reference decoder (HRD) specifications and metadata managed by the HRD buffer, which is signaled in the encoded video sequence.

[0064] Figure 3 This is a block diagram illustrating a server system 112 according to some embodiments. The server system 112 includes control circuitry 302, at least one network interface 304, memory 314, a user interface 306, and at least one communication bus 312 for interconnecting these components. In some embodiments, the control circuitry 302 includes at least one processor (e.g., CPU, GPU, and / or DPU). In some embodiments, the control circuitry includes at least one field-programmable gate array (FPGA), a hardware accelerator, and / or at least one integrated circuit (e.g., an application-specific integrated circuit).

[0065] At least one network interface 304 can be configured to interface with at least one communication network (e.g., wireless, wired, and / or optical network). The communication network can be local, wide area, metropolitan area, vehicular and industrial, real-time, latency-tolerant, etc. Examples of communication networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., cable or wireless wide area digital television networks including cable television, satellite television, and terrestrial broadcast television, vehicular and industrial networks including CANBus, etc. Such communication can be unidirectional (e.g., broadcast television), unidirectional (e.g., CANbus to certain CANbus devices), or bidirectional (e.g., to other computer systems using local area digital networks or wide area digital networks). Such communication can include communication to at least one cloud computing network.

[0066] User interface 306 includes at least one output device 308 and / or at least one input device 310. The at least one input device 310 may include at least one of the following: keyboard, mouse, touchpad, touchscreen, data glove, joystick, microphone, scanner, camera, etc. The at least one output device 308 may include at least one of the following: audio output device (e.g., speaker), visual output device (e.g., display or observer), etc.

[0067] Memory 314 may include high-speed random access memory (such as DRAM, SRAM, DDR RAM and / or other random access solid-state memory devices) and / or non-volatile memory (such as at least one disk storage device, optical disk storage device, flash memory device and / or other non-volatile solid-state storage devices). Memory 314 may optionally include at least one storage device remote from control circuitry 302. Alternatively, memory 314 or at least one non-volatile solid-state storage device within memory 314 may include a non-volatile computer-readable storage medium. In some embodiments, memory 314 or the non-volatile computer-readable storage medium of memory 314 stores programs, modules, instructions, and data structures, or subsets or supersets thereof:

[0068] ● Operating system 316, which includes programs for handling various basic system services and for performing hardware-related tasks;

[0069] ● Network communication module 318, which is used to connect server system 112 to other computing devices via at least one network interface 304 (e.g., via wired and / or wireless connection);

[0070] ● Encoding module 320, which performs various functions related to encoding and / or decoding data (such as video data). In some embodiments, encoding module 320 is an example of encoder component 114. Encoding module 320 includes, but is not limited to, at least one of the following:

[0071] Decoding module 322, which performs various functions related to decoding encoded data, such as those previously described with respect to decoder component 122; and

[0072] The encoding module 340 performs various functions related to encoding data, such as those previously described with respect to encoder component 106; and

[0073] ● Image memory 352 is used to store images and image data, for example, for use by encoding module 320. In some embodiments, image memory 352 includes at least one of the following: reference image memory 208, buffer memory 252, current image memory 264, and reference image memory 266.

[0074] In some embodiments, the decoding module 322 includes a parsing module 324 (e.g., configured to perform various functions previously described with respect to the parser 254), a transformation module 326 (e.g., configured to perform various functions previously described with respect to the scaler / inverse transformation unit 258), a prediction module 328 (e.g., configured to perform various functions previously described with respect to the motion compensation prediction unit 260 and / or the intra-frame prediction unit 262), and a filter module 330 (e.g., configured to perform various functions previously described with respect to the loop filter 256).

[0075] In some embodiments, the encoding module 340 includes an encoding module 342 (e.g., configured to perform the various functions previously described regarding the source encoder 202 and / or encoding engine 212) and a prediction module 344 (e.g., configured to perform the various functions previously described regarding the predictor 206). In some embodiments, the decoding module 322 and / or the encoding module 340 includes... Figure 3 A subset of the modules shown. For example, both decoding module 322 and encoding module 340 use a shared prediction module.

[0076] Each of the modules identified above and stored in memory 314 corresponds to a set of instructions for performing the functions described herein. The modules identified above (e.g., a set of instructions) do not need to be implemented as separate software programs, programs, or modules, and therefore various subsets of these modules can be combined or otherwise rearranged in various embodiments. For example, encoding module 320 may optionally not include separate decoding and encoding modules, but instead use a set of modules to perform both sets of functions. In some embodiments, memory 314 stores a subset of the modules and data structures identified above. In some embodiments, memory 314 stores additional modules and data structures not described above.

[0077] although Figure 3 The illustration depicts a server system 112 according to some embodiments, but... Figure 3 This is intended more as a functional description of various features that can exist in at least one server system, rather than a structural diagram of the embodiments described herein. In practice, items shown individually may be combined and some items may be separated. For example, Figure 3 Some items shown individually can be implemented on a single server, and a single item can be implemented by at least one server. The actual number of servers used to implement server system 112, and how features are allocated among them, will vary depending on the implementation method and, optionally, in part, on the amount of data traffic processed by the server system during peak usage periods and during average usage periods.

[0078] Example encoding techniques

[0079] The encoding processes and techniques described below can be performed on the devices and systems described above (e.g., source device 102, server system 112, and / or electronic device 120). According to some embodiments, a method for using a selective transformation set is described below.

[0080] As described in more detail below, if the intra-prediction mode is not a smooth mode, or if the intra-prediction mode is generating prediction samples according to a given prediction direction, then the intra-prediction mode may be referred to as an angle mode or a directional mode.

[0081] As discussed above, a block can refer to a coding tree block, a maximum coding block, a predefined fixed block size, a coding block, a prediction block, a residual block, or a transform block. An inter-mode coded block (or inter-block) refers to a block that uses an inter-prediction mode or a combination of intra-inter-prediction modes. An inter-mode can also refer to a block encoded using block vectors used to acquire prediction blocks within the same frame (e.g., using intra-block duplication). An intra-mode coded block (or intra-block) refers to a block that uses an intra-prediction mode or a combination of intra-inter-prediction modes. The intra-mode list can correspond to a list of the most likely intra-prediction modes for the current block. Additionally, the term "partition" can correspond to a block partition or a transform partition.

[0082] As an example, coding tree units (CTUs) can be partitioned into coding units (CUs) using a quadtree structure represented as a coding tree to accommodate various local characteristics. In some embodiments, a decision is made at the CU level regarding whether to use inter-picture (temporal) prediction or intra-picture (spatial) prediction to encode picture regions. Each CU can be further partitioned into one, two, or four prediction units (PUs) depending on the PU partitioning type. Within a PU, the same prediction process is applied, and relevant information can be sent to the decoder based on the PU. After obtaining residual blocks by applying a prediction process based on the PU partitioning type, the CUs can be partitioned into transform units (TUs) according to another quadtree structure, such as the coding tree of the CU.

[0083] The concept of multiple partition unit types can be replaced by a quadtree with a nested multi-type tree structure using binary and ternary partitioning. In the coding tree structure, CUs can be square or rectangular in shape. CTUs are first partitioned by the quadtree structure. Quadtree leaf nodes can be further partitioned using a multi-type tree structure. Example multi-type tree structures include four partition types. Multi-type leaf nodes are called CUs unless the CU is too large for the maximum transform length. This means that in a quadtree with a nested multi-type tree coding block structure, CUs, PUs, and TUs can have the same block size.

[0084] The coding tree scheme supports the ability for luma and chroma to have separate block tree structures, as in VTM7. In some cases, for P-slices and B-slices, the luma CTB and chroma CTB within a CTU share the same coding tree structure. However, for I-slices, luma and chroma can have separate block tree structures. When a separate block tree mode is applied, the luma CTB is partitioned into CUs via one coding tree structure, and the chroma CTB is partitioned into chroma CUs via another coding tree structure. This means that a CU in an I-slice can include a coding block of the luma component or coding blocks of the two chroma components, or consist of coding blocks of the luma component or coding blocks of the two chroma components, and a CU in a P- or B-slice can always include coding blocks of all three color components or consist of coding blocks of all three color components, unless the video is monochrome.

[0085] Turning now to transforms and transform blocks, transforms performed during the decoding of a video stream can be the inverse of transforms performed during the encoding of the video stream, and are sometimes referred to as "inverse transforms." It is worth noting that when the encoder component applies a transform, the decoder component performs the inverse transform. Therefore, in the following description, transforms described in the context of the decoder component can be the inverse transforms of transforms applied to the encoder side. For simplicity, the transforms described herein can be referred to as "transforms," ​​regardless of whether they are performed during encoding or decoding.

[0086] Multiple transform sizes (e.g., ranging from 4 to 64 points per dimension) and transform shapes (e.g., squares or rectangles with width / height ratios of 2:1 / 1:2 and 4:1 / 1:4) can be used. As described further below, transforms can correspond to primary or secondary transforms and can correspond to separable or non-separable transforms. A transform set is a grouping of at least one transform type. Therefore, a transform set indicates a group of multiple transform kernels / basis. Each entry in a transform set can be referred to as a transform candidate. For each block, the transform candidate selected from the transform set can be identified by signaling or implicitly.

[0087] Some embodiments include methods for signaling transform sets and / or for type selection of intra- and / or inter-frame coded blocks. Transform types may belong to the families of sinusoidal transform, KLT, or line graph transform (LGT). (Primary or secondary) transforms may belong to the family of sinusoidal transforms (DCT, DST, inverted versions of DCT and ADST). DCT can refer to any transform using a transform kernel derived from a discrete cosine transform, and DST / ADST can refer to any transform using a transform kernel derived from a discrete sine transform.

[0088] Example primary transforms can belong to a family of generalized line graph transforms (LGTs), or they can be based on trained kernels. Example secondary transform sets can be grouped from at least one inseparable secondary transform kernel transform type. A unique or common set of secondary transforms can be defined for each primary transform type and / or intra- or inter-frame mode type.

[0089] Additionally, an inseparable transform can refer to a primary transform applied directly to the residual, or a secondary transform applied to a block of transform coefficients generated by the primary transform. Transform kernels can be grouped into sets represented by set indices and kernel indices within the sets. An inseparable secondary transform can be a trained kernel applied to the primary transform coefficients at the encoder or to the dequantized coefficients at the decoder.

[0090] An inseparable secondary transformation kernel can be considered as a set of basis vectors in a vector space. If represented as an M × N (M rows and N columns) matrix, then N corresponds to the dimension of the vector space, and M corresponds to the cardinality. Therefore, the kernel size can be represented using M × N. Examples of kernel sizes include, but are not limited to, 64 × 64 samples, 32 × 64 samples, 16 × 64 samples, 8 × 84 samples, 4 × 64 samples, 16 × 16 samples, 8 × 16 samples, 4 × 16 samples, 8 × 8 samples, and 4 × 4 samples.

[0091] Scanning order refers to the coefficient recombination process of mapping a two-dimensional primary transform coefficient array to a one-dimensional primary transform coefficient array as the input of the forward secondary transform. It can also refer to the reverse coefficient recombination process of mapping a one-dimensional secondary transform coefficient array back to a two-dimensional primary transform coefficient array.

[0092] The End-of-Block (EOB) value corresponds to the position of the last valid coefficient in the encoded block following a given coefficient scan order. The EOB value can correspond to the position of the last valid (e.g., non-zero) coefficient in the encoded block following a given coefficient scan order. For a given coefficient scan order, all coefficients outside the EOB position are zero. In some embodiments, if an M×N inseparable secondary transform kernel is applied to the encoded block, the EOB value ≤ M.

[0093] The two-dimensional transformation process may involve the use of hybrid transformation kernels (e.g., consisting of different one-dimensional transformations for each dimension of the encoded residual block). The primary one-dimensional transformation may include at least one of the following: a) 4-point, 8-point, 16-point, 32-point, and 64-point discrete cosine transforms; b) 4-point, 8-point, and 16-point asymmetric discrete sine transforms and their inverted versions; or c) 4-point, 8-point, 16-point, and 32-point identity transforms.

[0094] For chroma components, transform type selection can be performed implicitly. For intra-frame prediction residuals, the transform type can be selected based on the intra-frame prediction mode. For inter-frame prediction residuals, the transform type can be selected based on the transform type selection of the co-occurring luma block. Therefore, for chroma components, transform type signaling in the bitstream may not be necessary.

[0095] Now let's turn to example encoding and decoding using predictive blocks and residual blocks. Figure 4A The illustration shows the calculation of a prediction block according to some embodiments. Figure 4A In the example, intra-frame prediction is performed on the current block 402 to generate a predicted block 404. In some embodiments, inter-frame prediction is performed to generate a predicted block. The current block 402 includes a set of samples (e.g., pixel blocks), and the predicted block 404 includes a set of predictions corresponding to the set of samples. Figure 4B The diagram illustrates the calculation of residual blocks according to some embodiments. For example... Figure 4B As shown, prediction block 404 is subtracted from current block 402 to generate residual block 406, which includes a set of residuals. For example, the corresponding difference between each sample and its corresponding prediction is calculated. Figure 4C The illustration shows the calculation of reconstructed blocks according to some embodiments. For example... Figure 4C As shown, residual block 406 undergoes at least one transform and quantization to generate a set of residual coefficients. The set of residual coefficients can be sent from the encoder component to the decoder component. The set of residual coefficients is inversely quantized and inversely transformed to generate reconstructed residual block 408. Reconstructed residual block 408 is combined with the predicted block 404 (e.g., adding the reconstruction residual of reconstructed residual block 408 to the prediction of prediction block 404) to generate reconstructed block 410 corresponding to the current block 402.

[0096] Intra-frame prediction explores the spatial redundancy between the current block and its neighboring samples. By convention, intra-frame prediction modes can be classified into directional and non-directional modes, indicating their directional or non-directional correlation between the current block and its neighboring reference blocks. Figure 4D The illustration depicts an example directional intra-prediction mode according to some embodiments. Figure 4D The pattern shown corresponds to the integer slope pattern, and additional fractional slope patterns can be scattered among the integer slope patterns. Figure 4D The shaded area 450 in the diagram corresponds to the wide-angle mode, and the number in parentheses corresponds to the mapping mode of the wide-angle mode. In the first example method, the intra-prediction mode is explicitly notified with a signal, and an interpolation filter applied to the reference sample is used to generate the corresponding prediction signal for the intra-prediction mode.

[0097] In the second example method, reference samples multiplied by weighting coefficients are used to generate the intra-frame prediction signal. These weighting coefficients can be trained offline and stored as matrices. Figure 4EThe illustration shows example sample regions for matrix multiplication prediction according to some embodiments. Figure 4E In this example, W and H represent the width and height of the current prediction block p. Consider reference samples at the top and left with double the width and double the height. The number of sample lines at the top and left are denoted as T1 and T2. In this example, the prediction signal is given by Equation 1 below.

[0098]

[0099] Equation 1 – Matrix Multiplication Prediction

[0100] Where F(x, y, k) are the training coefficients, and r(k) are the reference samples under consideration. In Equation 1, (x, y) represents the coordinates within the current prediction block, and k is the iterator that iterates through all reference samples. The final predictor is the weighted sum of all reference samples.

[0101] In the second example method, it's worth noting that no signaling overhead is introduced. This is because a matrix multiplication-based approach replaces the predefined subset of intra-frame modes used in the first example method. In other words, some regular intra-frame modes are established using a matrix multiplication-based method.

[0102] In the third example method, a hybrid approach (e.g., the PDP method) is applied, combining the first interpolation-based method and the second training-based matrix multiplication method. For a subset of the first intra-frame modes in the first method, predictions from the first method are generated using the second method. For example, the predictor corresponding to an even number of intra-frame modes in the first method is replaced by the predictor in the second example method.

[0103] In the fourth example method, intra-prediction is generated based on the first example method. However, its intra-prediction mode is not explicitly signaled. Instead, its intra-prediction mode is derived from the decoder side. Figure 5A An example of intra-frame mode derivation from the decoder side is shown.

[0104] Training-based methods are not limited to the prediction phase and can also be used in the transformation phase. In the fifth example method, such as... Figure 5B As shown, a secondary transform is applied between the forward primary transform and quantization (at the encoder) and between dequantization and the inverse primary transform (at the decoder).

[0105] Multiple sets of primary transform kernels and / or secondary transform kernels can be trained to adaptively compress different residual information. In the first example method, the set of secondary transform kernels can be determined based on the intra-frame prediction mode.

[0106] In the sixth example method, a single-stage non-separable primary transformation is applied to the residual block, instead of the conventional (e.g., DCT or DST) two-stage primary and secondary transformations.

[0107] Figure 5A The illustration depicts an example low-frequency non-separable transform (LFNST) process according to some embodiments. LFNST is also known as simplified secondary transform (RST). Figure 5A As shown, LFNST can be applied between the forward primary transform 502 and quantization 506 (e.g., at the encoding unit) and between dequantization 508 and inverse primary transform 512 (e.g., at the decoding unit).

[0108] Example LFNST 504 may include 16 input coefficients for a 4×4 forward LFNST or 64 input coefficients for an 8×8 forward LFNST. In the example LFNST, the 4×4 or 8×8 inseparable transform is applied depending on the block size. For example, a 4×4 LFNST may be applied to small blocks (e.g., with a width or height of less than 8 samples), and an 8×8 LFNST may be applied to larger blocks (e.g., with a width or height of more than 4 samples). Example LFNST 510 may include 8 input coefficients for a 4×4 inverse LFNST or 16 input coefficients for an 8×8 inverse LFNST.

[0109] As an example, each transform set used in LFNST can have 4 transform sets and 2 inseparable transform matrices (kernels). The mapping from intra-prediction modes to transform sets can be predefined as shown in Table 1 below. If one of the three CCLM modes (INTRA_LT_CCLM, INTRA_T_CCLM, or INTRA_L_CCLM) is used for the current block (81 <= IntraPredMode <= 83), then transform set 0 is selected (e.g., for the current chroma block). For each transform set, the selected inseparable secondary transform candidate can be further specified by an LFNST index that is explicitly signaled. The signaled index can be used once per intra-CU (e.g., after the transform coefficients) in the bitstream.

[0110]

[0111] Table 1 – Transformation Selection Table

[0112] The aforementioned LFNST can be adjusted as follows. First, more transformation sets can be used for finer-grained directions (e.g., 35 transformation sets instead of 4). Second, more transformation kernels can be included in each set (e.g., 3 transformation kernels instead of 2). Third, three different kernels (e.g., LFNST4, LFNST8, and LFNST16) can be defined to indicate the LFNST kernel sets, which can be applied to 4×N / N×4 (N... 4) 8×N / N×8 (N 8) and M×N(M, N) 16). Fourth, the set of LFNSTs (LFNST_set_idx) for a given intra-pred_mode can be derived according to different formulas, as shown in Table 2.

[0113] In another example, for Intra_pred_mode < 2, LFNST_set_idx equals 2; for Intra_pred_mode in the range [0,34], LFNST_set_idx = Intra_pred_mode; for Intra_pred_mode in the range [35,66], LFNST_set_idx = 68 – Intra_pred_mode.

[0114] For example, for block shapes of 4×4, 4×8 / 8×4, 4×16 / 16×4, 8×8, 8×16 / 16×8, 4×32 / 32×4, and 8×32 / 32×8, the Non-Separable Primary Transform (NSPT) can be used instead of the combination of separable DCT-II and LFNST transform. The NSPT set index for a given intra-frame mode can also be derived based on Table 2.

[0115]

[0116] Table 2 – Mapping between Intra-Frame Modes and LFNST Set Indexes

[0117] In some embodiments, LFNST is restricted to application only when all coefficients outside the first coefficient subgroup are invalid. In these embodiments, LFNST index encoding depends on the position of the last valid coefficient. LFNST indexes can be context-coded (e.g., independent of the intra-prediction mode). In some embodiments, only the first binary number is context-coded. In some embodiments, LFNST is applied to intra-CUs in both intra-strip and inter-strip configurations, and to both the luma and chroma components. If dual-tree is enabled, the LFNST indexes for the luma and chroma components can be signaled separately. For inter-strip configurations (with dual-tree disabled), a single LFNST index can be signaled and applied to both the luma and chroma components.

[0118] In some embodiments, when the Intra-Frame Sub-Partition (ISP) mode is selected for the current block, LFNST can be disabled and the RST index can be notified without signaling (e.g., because the performance improvement may be negligible even if RST is applied to every feasible partition block). Additionally, disabling RST on the residuals of ISP predictions can reduce coding complexity. LFNST can also be disabled (and the index can be notified without signaling) when the Matrix-Based Intra-Frame Prediction (MIP) mode is selected.

[0119] Due to the existing maximum transform size limit (64×64), large CUs (e.g., larger than 64×64) can be implicitly partitioned (TU tiling). For a given number of decoding pipeline stages, LFNST index search can quadruple the data buffer size. Therefore, the maximum size allowed by LFNST can be limited (e.g., limited to 64×64). In some embodiments, LFNST is enabled using only DCT2.

[0120] In some embodiments, a separable transform is applied to intra-frame and inter-frame residual samples. In some embodiments, an intra-frame secondary transform (IST) scheme is customized for the video codec library (e.g., for transforming intra-frame residual blocks). Compared to the non-separable primary transform, the IST scheme can efficiently capture directional patterns in intra-frame residual samples with lower complexity. In the IST scheme, the nominal intra-frame prediction angle can be used to classify the IST kernel.

[0121] In some embodiments, a secondary transform method (e.g., IST) is applied to the primary transform coefficient block before quantization is applied at the encoder (e.g., for the intra-prediction residual block of the luma component). Correspondingly, the inverse secondary transform can be applied to the dequantized transform coefficient block before applying the inverse primary transform at the decoder. In some embodiments, IST is not applied to the chroma color component. The use of IST in the encoding and decoding processes... Figure 5B The diagram is shown in the image. Figure 5BThe diagram illustrates a primary transform 552 applied to a residual block (e.g., corresponding to an intra-prediction block). A secondary transform 554 is applied to the output of the primary transform 552. Quantization 556 is applied to the output of the secondary transform 554, and the resulting quantization coefficients are entropy-encoded 558 and signaled via the video bitstream. The video bitstream is parsed 560 (e.g., at the decoder), and the quantized coefficients are dequantized 562. An inverse secondary transform 564 is applied to the dequantized data, and an inverse primary transform 566 is applied to the output of the secondary transform 564. In this manner, a reconstructed residual block is generated.

[0122] In some embodiments, 12 sets of secondary transforms (or IST sets) are defined, each containing 3 secondary transform kernels. In some embodiments, for each intra-coded transform block, a nominal intra-prediction mode and a primary transform type can be identified, and then an IST set is selected based on Table 3 below. In some embodiments, for Paeth prediction mode and recursive intra-prediction mode, ISTs are neither applied nor signaled.

[0123]

[0124] Table 3 – Selection of Secondary Transformation Sets

[0125] Given an IST set with 3 cores, there are four encoder options: 1) no secondary transform, 2) secondary transform using the first transform core from the given IST set, 3) secondary transform using the second transform core from the given IST set, and 4) secondary transform using the third transform core from the given IST set. The encoder can signal the selection using syntax elements (e.g., `ist_idx`). At the decoder, the values ​​of the syntax elements are parsed, and the secondary transform core is identified given the IST set and the value associated with `ist_idx`. The syntax element (`ist_idx`) can be signaled for each luma transform block after signaling the primary transform type. For example, signaling for ist_idx may be executed if at least one of the following is true: (i) the current block is an intra-coded luma transform block; (ii) the primary transform type is DCT in two dimensions or ADST in two dimensions; (iii) the intra-prediction mode is neither Paeth prediction mode nor recursive intra-prediction mode; (iv) the transform partition depth is 0; and (v) the EOB position falls within the low-frequency transform coefficient region where secondary transforms can be applied. In some embodiments, the entropy coding context of ist_idx is derived based on the transform block size. Conceptually, IST can be considered another name for LFSNT. As those skilled in the art will appreciate, IST and LFNST are interchangeable in this disclosure.

[0126] Secondary transforms (whether called LFNST, NSPT, or IST) are not limited to intra-prediction blocks. When the block is an inter-prediction block or uses intra-block copy mode prediction, decoder-side intra-mode derivation (DIMD) can be applied based on a template, such as... Figure 5C As shown. Figure 5C The illustration shows an example of deriving an intra-frame prediction mode using reconstructed samples from a template, according to some embodiments. Figure 5C The image shows the template region 578 used for the current block 580. During DIMD, a 3×3 window 582 is used to evaluate the reference sample set. The window 582 can slide within the template region to obtain different candidates. Figure 5C As shown, samples located at the top and left of the current coding block can be used as templates. Specifically, the template includes three sample rows and columns, and the texture orientation of adjacent samples can be estimated through the following steps. First, horizontal and vertical Sobel 3×3 filters can be applied to a 3×3 window, centered on the highlighted middle row and column samples, to obtain the horizontal gradient Gx and vertical gradient Gy. Next, the ratio of Gx and Gy can be calculated for each 3×3 window within the template (e.g., intra-mode information of adjacent templates). Then, the ratio of Gx and Gy can be matched against the closest intra-prediction mode in the regular pattern, and the count of the matched intra-modes is stored in the histogram. Next, the 3×3 window can be slid across the template, and the histogram is updated based on the matched intra-modes for each 3×3 window. Finally, at least one frequently matched intra-mode at the top associated with the texture orientation derived from the histogram can be used as an intra-prediction mode. For example, the set of the first five histograms is collected, and the first five corresponding intra-frame modes are obtained, labeled M0, M1, M2, M3, and M4 (e.g., where M0 has the highest histogram and M4 has the lowest histogram). The final predictor may be a fusion of a non-directional predictor (such as a planar mode predictor) and the five predictors predicted by M0 to M4 respectively.

[0127] For ease of description, the most frequently used mode M0 (e.g., derived by collecting the histogram of the template or the occurrence of adjacent blocks) is named the primary derived mode from the decoder side. However, the derived intra-frame modes may include more than one intra-frame mode.

[0128] In some embodiments, the intra-frame pattern is derived from the most frequently occurring intra-frame pattern in neighboring blocks (e.g., including near-neighboring blocks and non-near-neighboring blocks). For the purpose of deriving the intra-frame pattern, the positions of near-neighboring blocks and non-near-neighboring blocks can be predefined. The DIMD patterns described herein are examples, and in some embodiments, other DIMD methods are employed.

[0129] Figure 6A This is a flowchart illustrating a method 600 for decoding video according to some embodiments. Method 600 can be performed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having control circuitry and memory storing instructions for execution by the control circuitry. In some embodiments, method 600 is performed by executing instructions stored in memory (e.g., memory 314) of the computing system.

[0130] The system receives (602) the video bitstream including the current block. The system identifies (604) the first prediction mode of the current block. When the first prediction mode is a specific prediction mode, the system selects (606) a first set of transform kernels as the transform kernel for the current block. When the first prediction mode is not a specific prediction mode, the system selects (608) a second set of transform kernels as the transform kernel for the current block. The system applies (610) a transform to the current block using the transform kernels. In this way, when deriving intra-frame modes from the decoder side using DIMD, a separate set of inseparable transform kernels can be used. The transform kernels can be used for inseparable secondary transforms or inseparable primary transforms in a single stage.

[0131] In some embodiments, when using a secondary transform kernel or a separate set of non-separable primary transform kernels, fusion of a single predictor (in DIMD) is applied.

[0132] In some embodiments, an intra predictor is adaptively generated based on an intra mode derived from the decoder side using DIMD. In some embodiments, when an intra mode M is derived from the decoder side and M is in a subset of intra mode substitutions in the PDP, a first interpolation method is used instead of substitution matrix multiplication in the PDP to generate the intra predictor for that derived mode.

[0133] In some embodiments, when an intra-mode M is derived from the decoder side, and M is within a subset of intra-mode substitutions in the PDP, the intra-predictor for that derived mode M is generated using substitution matrix multiplication in the PDP. As an example, fusion in DIMD is not applied. Instead, the final predictor is the predictor corresponding to mode M0, which is generated based on substitution matrix multiplication using the trained coefficients. As another example, the fusion method in DIMD is retained. However, the individual predictors corresponding to M0-M4 are replaced using substitution matrix multiplication with the trained coefficients.

[0134] In some embodiments, when an intra-frame mode M is derived from the decoder side and M is in a subset of the replacement intra-frame modes, matrix multiplication is used instead of the replacement matrix multiplication in PDP to generate a predictor using a separate set of trained coefficients. As an example, fusion as in DIMD is not applied. Instead, the final predictor is a predictor corresponding to mode M0, which is generated using matrix multiplication based on a separate set of trained coefficients. As another example, the fusion method in DIMD is retained. However, the individual predictors corresponding to M0-M4 are replaced by separate sets of trained coefficients using matrix multiplication.

[0135] In some embodiments, when intra-modes are derived from adjacent blocks, the derived intra-modes are mapped to construct an intra-mode candidate list for the current block.

[0136] In some embodiments, when the exported intra-frame mode comes from a PDP, the PDP is added to the candidate list.

[0137] In some embodiments, when the exported intra-frame mode comes from a PDP, the PDP is considered an explicit intra-frame prediction mode and is added to the candidate list.

[0138] In some embodiments, when the exported intra-frame mode comes from a PDP, the PDP is considered the specific intra-frame mode and is added to the candidate list. As an example, mode M exported from a PDP is considered a planar mode. As another example, mode M exported from a PDP is considered a DC mode.

[0139] In some embodiments, a single trained method is used instead of the sequential application of a first method (e.g., matrix multiplication, DIMD, or PDP), followed by a transformation method (e.g., NSPT).

[0140] In some embodiments, intra-frame prediction is combined with an inseparable primary transform, and thus the transformed residual block is computed directly from the reference sample without requiring a separate prediction block computation step. For example, this method can be applied to all intra-frame prediction modes using matrix multiplication. In another example, the method is applied to at least a subset of all intra-frame prediction modes using PDP.

[0141] In some embodiments, in the second example method, a decoder-side intra-mode derivation method is used to determine a set of transforms for a predefined subset of intra-modes, wherein for these intra-modes, a matrix multiplication-based method is used to generate intra-predictive signals.

[0142] In some embodiments, the decoder-side intra-frame mode is derived from the predicted signal using a matrix multiplication-based method. In some embodiments, the determination of the transform set using the decoder-side intra-frame mode is conditionally applied based on other coding information. For example, it is applied only if the current block size is less than a predefined value N (e.g., N is 1024 samples). In another example, it is applied only if the ratio between the width and height of the current block is less than a predefined value R (e.g., R is 4). In yet another example, it is applied only if the width and height of the current block are less than predefined values ​​W and H (e.g., W=32, H=32). In yet another example, it is applied only if the template sizes (T1 and / or T2) satisfy a predefined size condition (e.g., size T1=T2=2).

[0143] In some embodiments, a predefined subset of intra-frame modes includes planar modes. In some embodiments, a predefined subset of intra-frame modes includes DC modes. In some embodiments, a predefined subset of intra-frame modes includes modes equal to (2 + 4 × k), where k = [0, 16]. In some embodiments, a predefined subset of intra-frame modes includes modes equal to (2 + 2 × k), where k = [0, 32].

[0144] In some embodiments, the determined transform set is a primary transform set. In some embodiments, the determined transform set is a secondary transform set. In some embodiments, the determination of the transform set is based on a predefined table between the derived intra-frame modes and the determined transform set.

[0145] In some embodiments, multiple intra-frame modes are derived using a decoder-side intra-frame mode derivation method. This decoder-side intra-frame mode derivation method uses matrix multiplication to analyze the most probable modes from the predicted signal. For example, the grammar can be signaled in the bitstream to indicate which intra-frame mode among the derived intra-frame modes was used to determine the transform set.

[0146] In some embodiments, in the second method, a separate set of transform kernels is trained for a predefined subset of intra-frame modes, wherein for these intra-frame modes, a matrix multiplication-based method is used to generate intra-frame prediction signals. In some embodiments, in the first example method, the predefined subset of intra-frame modes is separated from regular intra-frame modes and they are explicitly signaled in the bitstream.

[0147] The methods and techniques described above can also be used in other training-based encoding and decoding tools (e.g., other matrix-based tools).

[0148] Figure 6BThis is a flowchart illustrating a method 650 for encoding video according to some embodiments. Method 650 can be performed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having control circuitry and memory storing instructions for execution by the control circuitry. In some embodiments, method 650 is performed by executing instructions stored in memory (e.g., memory 314) of the computing system.

[0149] The system receives (652) video data comprising multiple blocks, including the current block. The system identifies (654) a first prediction mode for the current block. When the first prediction mode is a specific prediction mode, the system selects (656) a first set of transform kernels as the transform kernel for the current block. When the first prediction mode is not a specific prediction mode, the system selects (658) a second set of transform kernels as the transform kernel for the current block. The system applies (660) a transform to the current block using the transform kernels. As previously described, the encoding process can reflect the decoding process described herein (e.g., transform kernel selection). For the sake of brevity, these details will not be repeated here.

[0150] although Figure 6A and Figure 6B Multiple logical stages are illustrated in a specific order, but stages that are not in order can be reordered, and other stages can be combined or decomposed. Some reorderings or other groupings not specifically mentioned will be obvious to those skilled in the art, and therefore the orderings and groupings presented herein are not exhaustive. Furthermore, it should be recognized that these stages can be implemented using hardware, firmware, software, or any combination thereof.

[0151] Now let's turn to some example implementations.

[0152] In some embodiments, a multi-transform set selection method is applied to intra-blocks encoded using LFNST / NSPT. In this way, CUs encoded with decoder-side intra-mode derivation (DIMD), template-based intra-mode derivation (TIMD), MIP, spatial geometric partitioning mode (SGPM), enhanced intra-prediction (EIP), and IntraTMP modes can use alternative LFNST / NSPT transform sets. Transform set selection can be signaled in the bitstream.

[0153] In some embodiments, the encoding component is configured to select an LFNST / NSPT transform set from two or more candidate sets for a CU encoded using DIMD, TIMD, MIP, SGPM EIP, and IntraTMP. For example, if the current block is encoded using DIMD, TIMD, MIP, SGPM, IntraTMP, and LFNST / NSPT, additional binary numbers are used to indicate whether a first candidate transform set or a second candidate transform set is selected. In some embodiments, the first candidate transform set is the default transform set (e.g., the same as the current ECM design). In some embodiments, the second candidate is derived through a DIMD process with adjacent reconstructed pixels. For example, if fusion is applied to the TIMD encoded block, a second TIMD IPM can be used to derive the second candidate set. As another example, for an SGPM encoded block, the two IPMs used by SGPM can be used to derive the second candidate set. Simulation data on ECM-11 software with common test conditions has shown that selecting between two transform sets for LFNST / NSPT improves the encoding of the luma (Y) component by 0.08%. Simulation data on ECM-14 software with common test conditions have shown that choosing between the two transform sets of LFNST / NSPT improves the encoding of the luminance (Y) component by 0.14%.

[0154] In ECM-13.0, LFNST / NSPT can be applied to inter-frame prediction (inter-frame coding) blocks, where the transform kernels of intra-frame LFNST / NSPT can be reused in inter-frame blocks. Based on the prediction signal of the current block, a gradient histogram (HoG) can be constructed in a manner similar to DIMD, and the set of LFNST / NSPT kernels can be determined using the first DIMD intra-frame prediction mode (IPM) corresponding to the highest amplitude. However, as described here, having more than one option for selecting the kernel set can be beneficial. For example, for a coded block, a secondary IPM can be derived for inter-frame LFNST / NSPT. For example, in addition to the first DIMD IPM, a second DIMD IPM (corresponding to the second highest HOG amplitude) can also be used as an additional IPM candidate. Simulation data on ECM-13 software with common test conditions have shown that using DIMD to select between two transform sets for LFNST / NSPT improves the coding of the luma (Y) component and chroma component (e.g., V component) by 0.02%.

[0155] The NSPT and / or LFNST kernel set can be selected based on the transform block size and intra-frame mode. For example, NSPT can be used for block shapes of 4×4, 4×8, 4×16, 4×32, 8×8, 8×16, 8×32 and their corresponding transposed block shapes, while LFNST can be used for other block shapes. As mentioned above, NSPT and / or LFNST can be used with various intra-frame prediction tools, such as conventional intra-frame prediction (e.g., planar intra-frame prediction, DC intra-frame prediction, and directional intra-frame prediction), DIMD, TIMD, SGPM, MIP, EIP, and IntraTMP. NSPT and / or LFNST can also be used for inter-frame coded CUs. As previously mentioned, multiple kernel sets can be used for NSPT and LFNST. For example, the selection of the transform kernel can be based on whether the CU uses conventional intra-frame prediction, inter-frame prediction, or uses TIMD, DIMD, EIP, MIP, SGPM, or IntraTMP. Simulation data on ECM-14 software with common test conditions have shown that selecting among multiple transform kernels for LFNST / NSPT based on prediction modes (e.g., regular intra-frame, inter-frame coded CU, or unconventional intra-frame) improves the coding of the luma (Y) component by 0.17%, the coding of the chroma (U) component by 0.09%, and the coding of the chroma (V) component by 0.01%.

[0156] Although EMC has been mentioned in the preceding paragraphs, those skilled in the art will recognize that the methods described herein can be used in many existing codecs, such as those mentioned in the background section.

[0157] (A1) In one aspect, some embodiments include a method for video decoding (e.g., method 600). In some embodiments, the method is performed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having memory and at least one processor. In some embodiments, the method is performed at an encoding module (e.g., encoding module 320). In some embodiments, the method is performed by executing instructions stored in the memory (e.g., memory 314) of one or more computing systems. The method includes: (i) receiving a video bitstream (e.g., an encoded video sequence) including a current block; (ii) identifying a first prediction mode for the current block; (iii) selecting a first set of transform kernels as the transform kernel for the current block when the first prediction mode is a specific prediction mode; (iv) selecting a second set of transform kernels as the transform kernel for the current block when the first prediction mode is not a specific prediction mode; and (v) applying a transform to the current block using the transform kernels. For example, when deriving an intra-frame mode from the decoder side, a separate, inseparable set of primary (and / or secondary) transform kernels for that transform is used. In some embodiments, applying the transform includes applying an inverse transform of a transform performed at the encoder. In some embodiments, a first set of transform kernels is selected for the current block based on the determination that the first prediction mode is a specific prediction mode. In some embodiments, a second set of transform kernels is selected for the current block based on the determination that the first prediction mode is not a specific prediction mode. As an example, a decoder-side intra-mode derivation method can be used to determine the transform set for a predefined subset of intra-modes, wherein, for these intra-modes, a matrix multiplication-based method is used to generate the intra-prediction signal.

[0158] (A2) In some embodiments of A1, a specific prediction mode is identified using decoder-side intra-mode derivation (DIMD). For example, DIMD may include generating intra-prediction using interpolation filters applied to reference samples without explicitly signaling the intra-prediction mode. In some embodiments, a first set of transform kernels is selected based on the determination that a first prediction mode is identified using DIMD. In some embodiments, a first set of transform kernels is selected based on the determination that the first prediction mode is an unconventional intra-prediction mode (e.g., not a directional, DC, or planar intra-prediction mode). As an example, a decoder-side intra-mode derivation method may be used to derive multiple intra-prediction modes, which uses a matrix multiplication method to analyze the most probable mode from the prediction signal. In some embodiments, a signaling syntax is used in the bitstream to indicate which intra-prediction mode among the derived intra-prediction modes is used to determine the transform set.

[0159] (A3) In some embodiments of A2, applying the transformation includes the fusion of the individual predictors applying DIMD. For example, when using a first (separate) set of transform kernels, the fusion of the individual predictors in the DIMD method is applied.

[0160] (A4) In some embodiments of any of A1 through A3, the transformation is an irreparable primary transformation (NSPT). For example, a single-stage irreparable primary transformation is applied to the residual block instead of a conventional (e.g., DCT or DST) two-stage primary and secondary transformation. In some embodiments, the transformation is derived from a set of primary transformations.

[0161] (A5) In some embodiments of any of A1 to A3, the transform is a low-frequency non-separable transform (LFNST). In some embodiments, the transform is derived from a set of secondary transforms.

[0162] (A6) In some embodiments of any of A1 to A3, the transform is a secondary transform. For example, a secondary transform is applied between the forward primary transform and quantization (at the encoder) and between dequantization and the inverse primary transform (at the decoder).

[0163] (A7) In some embodiments of any of A1 to A6, a particular prediction pattern is identified using a matrix-based method.

[0164] (A8) In some embodiments of A7, applying a transform to the current block includes combining a matrix-based prediction with an inseparable primary transform. For example, a combination matrix is ​​generated based on the combination of intra-prediction and the inseparable primary transform. In some embodiments, a particular prediction mode and transform are applied in a single step (e.g., rather than sequentially). In some embodiments, the matrix corresponding to the combination of a particular prediction mode and transform is stored at a decoder unit and used when the particular prediction mode and transform are to be performed on the current block. As an example, intra-prediction is combined with the inseparable primary transform, and thus the transformed residual block is computed directly from the reference sample without requiring a separate prediction block computation step. In some embodiments, the intra-prediction mode is combined with the inseparable primary transform for all intra-prediction modes. In some embodiments, the intra-prediction mode is combined with the inseparable primary transform only for a subset of all intra-prediction modes (e.g., according to the PDP mode).

[0165] (A9) In some embodiments of any of A1 through A8, the first prediction mode is identified using a location-related prediction (PDP) method. For example, the PDP method may include a hybrid method that uses an interpolation filter applied to a reference sample and generates an intra-frame prediction signal by using the reference sample multiplied by weighting coefficients.

[0166] (A10) In some embodiments of any of A1 to A9, the method further includes: (i) deriving an intra-mode of the current block based on neighboring blocks; and (ii) populating an intra-mode candidate list with the derived intra-mode, wherein a first prediction mode is selected from the intra-mode candidate list. For example, when deriving an intra-mode from a neighboring block, the derived intra-mode is mapped to construct an intra-mode candidate list for the current block.

[0167] (A11) In some embodiments of A10, the intra-frame mode is derived using the PDP method. For example, when an intra-frame mode is derived using the PDP method, the derived intra-frame mode is added to a candidate list. As an example, when an intra-frame mode is derived using the PDP method, the derived intra-frame mode is considered a specific intra-frame mode. In some embodiments, the derived intra-frame mode is mapped to a regular intra-frame mode (e.g., a non-directional intra-frame mode). For example, the derived intra-frame mode is considered a planar mode or a DC mode. For example, the decoder-side intra-frame mode is derived based on the prediction signal using a matrix multiplication-based method.

[0168] (A12) In some embodiments of any of A1 to A1, the method further includes generating an intra-predictor using a first prediction mode, including: (i) identifying the first prediction mode using DIMD; (ii) determining whether the first prediction mode is in an intra-mode replacement set; (iii) generating an intra-predictor using a first technique when the first prediction mode is in the intra-mode replacement set; and (iv) generating an intra-predictor using a second technique when the first prediction mode is not in the intra-mode replacement set. For example, the intra-predictor is adaptively generated based on an intra-mode derived from the decoder side in DIMD. In some embodiments, the intra-mode replacement set is hard-coded at the decoder. In some embodiments, the intra-mode replacement set is stored as a lookup table.

[0169] (A13) In some embodiments of A12, the first technique is an interpolation technique. For example, when the intra-mode derived from the DIMD on the decoder side is in an intra-mode substitution subset (e.g., in the PDP method), the first interpolation method (e.g., regular interpolation) is used instead of the substitution matrix multiplication in the PDP method to generate the intra-predictor for that derived mode.

[0170] (A14) In some embodiments of A12, the first technique is a substitution matrix multiplication technique. For example, when an intra-mode derived from DIMD on the decoder side is in an intra-mode substitution subset, a substitution matrix multiplication of the derived mode is generated using a hybrid approach of interpolation and trained matrix multiplication (e.g., the PDP approach). In some embodiments, the fusion of intra-predictors in the DIMD approach is not applied. For example, the fusion in DIMD is not applied. Instead, the final predictor is a predictor corresponding to mode M0, which is generated based on substitution matrix multiplication using trained coefficients. In some embodiments, the fusion in DIMD is applied to the substitution predictor. For example, the respective predictors corresponding to M0 through M4 of DIMD are replaced using substitution matrix multiplication with trained coefficients.

[0171] (A15) In some embodiments of A14, the substitution matrix multiplication technique uses a set of trained coefficients corresponding to the combined DIMD and PDP methods. For example, when an intra-frame mode derived from the decoder-side DIMD method is in a subset of substitution intra-frame modes, matrix multiplication is used to generate a predictor using a separate set of trained coefficients instead of the substitution matrix multiplication of the PDP method. In some embodiments, the fusion of intra-frame predictors in the DIMD method is not applied. For example, the fusion in the DIMD method is not applied. Instead, the final predictor is a predictor corresponding to mode M0, which is generated based on a separate set of trained coefficients using matrix multiplication. In some embodiments, the fusion method in the DIMD method is applied. However, the individual predictors corresponding to M0 through M4 are replaced by separate sets of trained coefficients using matrix multiplication.

[0172] (A16) In some embodiments of any of A1 to A15, the second transform kernel set is selected from multiple transform kernel sets based on coding information. For example, the determination of the transform set using decoder-side intra-modes is based on the conditional application of other coding information. For example, the determination of the transform set is based on a predefined table between derived intra-modes and the determined transform set. In some embodiments, the second transform kernel set is trained using a predefined set of intra-modes. For example, a separate transform kernel set is trained for a predefined subset of intra-modes, wherein, for these intra-modes, a matrix multiplication-based method is used to generate the intra-predicted signal.

[0173] (A17) In some embodiments of A16, the encoding information includes at least one of the current block size, the aspect ratio of the current block, and the template size. In one example, a second transform kernel set is selected when the current block size is less than a predefined value N (e.g., N is 1024 samples). In another example, a second transform kernel set is selected when the ratio between the width and height of the current block is less than a predefined value R (e.g., R is 4). In another example, a second transform kernel set is selected when the width and height of the current block are less than predefined values ​​W and H (e.g., W=32, H=32). In yet another example, a second transform kernel set is selected when the template size (T1 and / or T2) satisfies a predefined size condition (e.g., size T1=T2=2).

[0174] (A18) In some embodiments of any of A1 through A17, a specific prediction mode is one of a subset of prediction modes. For example, a predefined subset of intra-modes includes one or more of the following: a planar mode, a DC mode, a mode equal to (2 + 4 × k), where k = [0, 16], and a mode equal to (2 + 2 × k), where k = [0, 32]. In some embodiments, the predefined subset of intra-modes is separated from the regular intra-modes and is explicitly signaled in the bitstream.

[0175] (B1) In another aspect, some embodiments include a video encoding method (e.g., method 650). In some embodiments, the method is performed at a computing system (e.g., server system 112) having memory and at least one processor. In some embodiments, the method is performed at an encoding module (e.g., encoding module 320). The method includes: (i) receiving video data (e.g., a source video sequence) comprising a plurality of blocks, the plurality of blocks including a current block; (ii) identifying a first prediction mode for the current block; (iii) selecting a first set of transform kernels as the transform kernel for the current block when the first prediction mode is a specific prediction mode; (iv) selecting a second set of transform kernels as the transform kernel for the current block when the first prediction mode is not a specific prediction mode; and (v) applying a transform to the current block using the transform kernels.

[0176] (B2) In some embodiments of B1, a particular prediction pattern is identified using decoder-side intra-frame pattern derivation (DIMD).

[0177] (B3) In some embodiments of B1 or B2, the transform is a non-separable primary transform (NSPT) or a low-frequency non-separable transform (LFNST).

[0178] (B4) In some embodiments of any of B1 to B3, a particular prediction pattern is identified using a matrix-based method.

[0179] (B5) In some embodiments of any of B1 to B4, the method further includes any of A1 to A18 described above.

[0180] (C1) In another aspect, some embodiments include a method for processing visual media data. In some embodiments, the method is performed at a computing system (e.g., server system 112) having memory and at least one processor. In some embodiments, the method is performed at an encoding module (e.g., encoding module 320). The method includes: (i) acquiring a source video sequence comprising a plurality of frames; and (ii) performing a conversion between the source video sequence and a video stream of visual media data according to format rules, wherein the video stream comprises a plurality of blocks, the plurality of blocks including a current block; and the format rules specify: (a) identifying a first prediction mode for the current block; (b) selecting a first set of transform kernels as the transform kernel for the current block when the first prediction mode is a specific prediction mode; (c) selecting a second set of transform kernels as the transform kernel for the current block when the first prediction mode is not a specific prediction mode; and (d) applying a transform to the current block using the transform kernels.

[0181] (D1) In another aspect, some embodiments include a method for video decoding. In some embodiments, the method is performed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having memory and at least one processor. In some embodiments, the method is performed at an encoding module (e.g., encoding module 320). In some embodiments, the method is performed by executing instructions stored in the memory (e.g., memory 314) of one or more computing systems. The method includes: (i) receiving a video bitstream including a current block; (ii) identifying a first prediction mode of the current block using decoder-side intra-mode derivation (DIMD) technology; (iii) determining whether the first prediction mode is in an intra-mode replacement set; (iv) generating an intra-predictor using a first technique when the first prediction mode is in the intra-mode replacement set; and (v) generating an intra-predictor using a second technique when the first prediction mode is not in the intra-mode replacement set. In some embodiments, the intra-predictor is generated using the first technique based on the determination that the first prediction mode is in the intra-mode replacement set. In some embodiments, based on the determination that the first prediction mode is not in the intra-mode replacement set, an intra-predictor is generated using a second technique different from the first technique.

[0182] (E1) In another aspect, some embodiments include a method for video decoding. In some embodiments, the method is performed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having memory and at least one processor. In some embodiments, the method is performed at an encoding module (e.g., encoding module 320). In some embodiments, the method is performed by executing instructions stored in the memory (e.g., memory 314) of one or more computing systems. The method includes: (i) receiving a video bitstream including a current block; (ii) identifying a first prediction mode for the current block; and (iii) when the first prediction mode is a specific prediction mode, generating a transformed residual block of the current block without performing prediction block computation. For example, a single matrix is ​​applied to samples to generate the transformed residual block. In some embodiments, the single matrix corresponds to a combination of a prediction block matrix and a transformation matrix.

[0183] In another aspect, some embodiments include a computing system (e.g., server system 112) including control circuitry (e.g., control circuitry 302) and a memory (e.g., memory 314) coupled to the control circuitry, the memory storing one or more sets of instructions configured to be executed by the control circuitry, the set of instructions including instructions for performing any of the methods described herein (e.g., A1 to A18, B1 to B5, C1, D1, and E1 above). In yet another aspect, some embodiments include a non-volatile computer-readable storage medium storing one or more sets of instructions for execution by the control circuitry of the computing system, the set of instructions including instructions for performing any of the methods described herein (e.g., A1 to A18, B1 to B5, C1, D1, and E1 above).

[0184] Unless otherwise stated, any syntax element described herein can be a high-level syntax (HLS). As used herein, HLS is signaled at a level higher than the block level. For example, HLS can correspond to the sequence level, frame level, stripe level, or tile level. As another example, HLS elements can be signaled in a video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), adaptation parameter set (APS), stripe header, picture header, tile header, and / or CTU header.

[0185] It will be understood that although the terms "first," "second," etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the claims. As used in the description of embodiments and appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or," as used herein, refers to and covers any and all possible combinations of one or more of the associated listed items. It will be further understood that, when used in this specification, the terms "comprising" and / or "including" specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0186] As used herein, the term "when" may be interpreted, depending on the context, as meaning "if," "when," "in response to determination," "according to determination," or "in response to detection," the stated precondition is true. Similarly, the phrases "if determination [the stated precondition is true]," "if [the stated precondition is true]," or "when [the stated precondition is true]" may be interpreted, depending on the context, as meaning "in response to determination," "according to determination," "in response to detection," or "in response to detection," the stated precondition is true. As used herein, N refers to the variable number. Unless explicitly stated otherwise, different instances of N may refer to the same number (e.g., the same integer value, such as the number 2) or different numbers.

[0187] For purposes of explanation, the foregoing description has been given with reference to specific embodiments. However, the illustrative discussion above is not intended to be exhaustive or to limit the claims to the precise forms disclosed. Many modifications and variations are possible in light of the foregoing teachings. The embodiments have been chosen and described to best explain the operating principles and practical applications, thereby enabling others skilled in the art to implement them.

Claims

1. A method for video decoding, executed in a computing system having a memory and at least one processor, characterized in that, The method includes: Receive the video stream including the current block; Identify the first prediction pattern of the current block; When the first prediction mode is a specific prediction mode, the first set of transformation kernels is selected as the transformation kernel of the current block; When the first prediction mode is not the specific prediction mode, a second set of transformation kernels is selected as the transformation kernel of the current block; and Apply the transformation to the current block using the transformation kernel.

2. The method according to claim 1, characterized in that, The specific prediction pattern is identified using decoder-side intra-frame pattern derivation (DIMD).

3. The method according to claim 2, characterized in that, Applying the transformation involves fusing the individual predictors of the DIMD.

4. The method according to claim 1, characterized in that, The transformation is an inseparable primary transformation (NSPT).

5. The method according to claim 1, characterized in that, The transformation is a low-frequency non-separable transformation (LFNST).

6. The method according to claim 1, characterized in that, The transformation is a secondary transformation.

7. The method according to claim 1, characterized in that, The specific prediction pattern was identified using a matrix-based prediction method.

8. The method according to claim 7, characterized in that, Applying the transformation to the current block includes combining the matrix-based prediction with an inseparable primary transformation.

9. The method according to claim 1, characterized in that, The first prediction pattern was identified using the location-related prediction (PDP) method.

10. The method according to claim 1, characterized in that, Further includes: The intra-frame mode of the current block is derived based on the adjacent blocks of the current block; as well as The intra-frame mode candidate list is populated with the exported intra-frame modes, wherein the first prediction mode is selected from the intra-frame mode candidate list.

11. The method according to claim 1, characterized in that, Further includes using the first prediction mode to generate an intra-frame predictor, including: Use DIMD to identify the first prediction pattern; Determine whether the first predicted mode is in the intra-frame mode replacement set; When the first prediction mode is in the intra-mode replacement set, the first technique is used to generate the intra-predictor; and When the first prediction mode is not in the intra-mode replacement set, the second technique is used to generate the intra-predictor.

12. The method according to claim 11, characterized in that, The first technique is an interpolation technique.

13. The method according to claim 11, characterized in that, The first technique is the substitution matrix multiplication technique.

14. The method according to claim 13, characterized in that, The substitution matrix multiplication technique uses a set of training coefficients corresponding to the combined DIMD and PDP methods.

15. The method according to claim 1, characterized in that, The second set of transform kernels is selected from multiple sets of transform kernels based on the encoding information.

16. A video encoding method, performed in a computing system having a memory and at least one processor, characterized in that, The method includes: Receive video data comprising multiple blocks, including the current block; Identify the first prediction pattern of the current block; When the first prediction mode is a specific prediction mode, the first set of transformation kernels is selected as the transformation kernel of the current block; When the first prediction mode is not the specific prediction mode, a second set of transformation kernels is selected as the transformation kernel of the current block; and Apply the transformation to the current block using the transformation kernel.

17. The method according to claim 16, characterized in that, The specific prediction pattern is identified using decoder-side intra-frame pattern derivation (DIMD).

18. The method according to claim 16, characterized in that, The transformation is either the Non-Separable Primary Transform (NSPT) or the Low-Frequency Non-Separable Transform (LFNST).

19. The method according to claim 16, characterized in that, The specific prediction pattern was identified using a matrix-based method.

20. A non-volatile computer-readable storage medium storing a video stream generated by a video coding method, characterized in that, The video encoding method includes: Receive video data comprising multiple blocks, including the current block; Identify the first prediction pattern of the current block; When the first prediction mode is a specific prediction mode, the first set of transformation kernels is selected as the transformation kernel of the current block; When the first prediction mode is not the specific prediction mode, a second set of transformation kernels is selected as the transformation kernel of the current block; and Apply the transformation to the current block using the transformation kernel.

Images