Selective transform set based on prediction mode

Abstract

An example method of video coding includes receiving a video bitstream including a current block and identifying a first prediction mode for the current block. The method also includes mapping the first prediction mode to a corresponding intra prediction mode and selecting a first transform set from a plurality of transform sets based on the corresponding intra prediction mode. The method also includes reconstructing the current block using the first transform set.

Description

[0001] Cross-references to related applications

[0002] This application is a continuation of U.S. Patent Application No. 19 / 023,013, filed January 15, 2025, entitled “Selective Transform Set Based on Prediction Mode,” which claims priority to U.S. Provisional Patent Application No. 63 / 622,058, filed January 17, 2024, entitled “LFNST / NSPT with Selective Transform Set based on Prediction Mode,” each of which is incorporated herein by reference in its entirety. Technical Field

[0003] The disclosed implementations generally relate to video encoding and decoding, including but not limited to systems and methods for selecting transform sets. Background Technology

[0004] Digital video is supported by various electronic devices such as digital televisions, laptop or desktop computers, tablet computers, digital camera devices, digital recording devices, digital media players, video game consoles, smartphones, video conferencing equipment, video streaming devices, etc. Electronic devices send and receive or otherwise transmit digital video data across communication networks, and / or store digital video data on storage devices. Due to the limited bandwidth capacity of communication networks and the limited memory resources of storage devices, video encoding and decoding can be used to compress video data according to one or more video codec standards before transmission or storage. Video encoding and decoding can be performed by hardware and / or software on electronic / client devices or servers providing cloud services.

[0005] Video codecs typically employ prediction methods that leverage the inherent redundancy in video data (e.g., inter-frame prediction, intra-frame prediction, etc.). Video codecs aim to compress video data into a form using a lower bitrate while avoiding or minimizing video quality degradation. Several video codec standards have been developed. For example, High-Efficiency Video Coding (HEVC / H.265) is a video compression standard designed as part of the MPEG-H project. ITU-T and ISO / IEC released the HEVC / H.265 standard in 2013 (Revision 1), 2014 (Revision 2), 2015 (Revision 3), and 2016 (Revision 4). Versatile Video Coding (VVC / H.266) is a video compression standard designed as a successor to HEVC. ITU-T and ISO / IEC released the VVC / H.266 standard in 2020 (Revision 1) and 2022 (Revision 2). AOMedia Video 1 (AV1) is an open video codec format designed as an alternative to HEVC. On January 8, 2019, a validation version 1.0.0 with errata table 1 was released. Enhanced Compression Model (ECM) is a video codec standard currently under development. ECM aims to significantly improve compression efficiency, surpassing existing standards such as HEVC / H.265 and VVC, essentially enabling higher quality video at lower bitrates. ECM version 13 was released in MPEG 146 on July 7, 2024. Summary of the Invention

[0006] Among other things, this disclosure describes a set of techniques for video (image) compression that involve prediction modes and transform set selection for corresponding residual blocks. Some implementations include identifying the transform set based on corresponding prediction information. As an example, the prediction information may indicate whether to predict blocks in intra-frame mode, inter-frame mode, or intra-block copy mode. The advantage of selecting the transform set based on prediction information is improved encoding and decoding accuracy (rather than using the same transform set regardless of the prediction mode).

[0007] According to some implementations, a video decoding method includes: (i) receiving a video bitstream (e.g., a encoded video sequence) including a current block; (ii) identifying a first prediction mode for the current block; (iii) mapping the first prediction mode (e.g., a non-intra-prediction mode) to a corresponding intra-prediction mode; (iv) selecting a first transform set from a plurality of transform sets based on the corresponding intra-prediction mode; and (v) reconstructing the current block using the first transform set.

[0008] According to some implementations, a video coding method includes: (i) receiving video data (e.g., a source video sequence) comprising a plurality of blocks, the plurality of blocks including a current block; (ii) identifying a first prediction mode for the current block; (iii) mapping the first prediction mode to a corresponding intra-frame prediction mode; (iv) selecting a first transform set from a plurality of transform sets based on the corresponding intra-frame prediction mode; and (v) encoding the current block using the first transform set.

[0009] According to some embodiments, a computing system, such as a streaming system, server system, personal computer system, or other electronic device, is provided. The computing system includes a control circuitry system and a memory storing one or more sets of instructions. The one or more sets of instructions include instructions for performing any of the methods described herein. In some embodiments, the computing system includes encoder components and decoder components (e.g., a transcoder). According to some embodiments, a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium stores one or more sets of instructions executable by the computing system. The one or more sets of instructions include instructions for performing any of the methods described herein.

[0010] Therefore, methods, apparatus, and systems for encoding and decoding video are disclosed. Such methods, apparatus, and systems may supplement or replace conventional methods, apparatus, and systems for encoding / decoding video. Not all features and advantages described in the specification are necessarily included, and in particular, some additional features and advantages will be apparent to those skilled in the art in light of the accompanying drawings, specification, and claims provided in this disclosure. Furthermore, it should be noted that the language used in the specification has been chosen primarily for readability and instruction purposes and is not necessarily intended to depict or limit the subject matter described herein. Attached Figure Description

[0011] To provide a more detailed understanding of this disclosure, a more specific description can be made by referring to the features of various embodiments, some of which are illustrated in the accompanying drawings. However, the drawings only show relevant features of this disclosure and are therefore not necessarily intended to be limiting, as those skilled in the art will understand upon reading this disclosure that other valid features may be permissible.

[0012] Figure 1 This is a block diagram illustrating an example communication system according to some implementations.

[0013] Figure 2A This is a block diagram illustrating example elements of an encoder component according to some embodiments.

[0014] Figure 2BThis is a block diagram illustrating example elements of a decoder component according to some embodiments.

[0015] Figure 3 This is a block diagram illustrating an example server system according to some implementation methods.

[0016] Figures 4A to 4C Example prediction blocks, residual blocks, and reconstruction blocks are shown according to some implementation methods.

[0017] Figure 5A An example of low-frequency non-separable transformation processing according to some implementations is shown.

[0018] Figure 5B An example transformation process involving a quadratic transformation is shown according to some implementations.

[0019] Figure 5C An example is shown of using reconstructed samples from a template to derive an intra-frame prediction mode according to some implementations.

[0020] Figure 5D An example of non-adjacent spatial adjacency candidates for the current codec unit according to some implementations is shown.

[0021] Figure 6A An example video decoding process according to some implementation methods is shown.

[0022] Figure 6B An example video encoding process according to some implementation methods is shown.

[0023] By convention, the various features shown in the accompanying drawings are not necessarily drawn to scale, and similar reference numerals may be used to indicate similar features throughout the specification and the drawings. Detailed Implementation

[0024] This disclosure describes video / image compression techniques that include selecting a transform set from multiple transform sets based on encoding / decoding information of the current block. Selecting an appropriate transform set from multiple transform sets can improve encoding / decoding accuracy compared to using a single transform set for all blocks. The encoding / decoding information may include the prediction mode of the current block. For example, a first (non-intra-frame) prediction mode can be identified for the current block. The first prediction mode can be mapped to a corresponding intra-frame prediction mode. The first prediction mode can be an inter-frame prediction mode or an intra-block copy (IBC) mode. A first transform set can be selected from multiple transform sets based on the corresponding intra-frame prediction mode. Finally, the first transform set can be used to reconstruct the current block.

[0025] Example systems and devices

[0026] Figure 1This is a block diagram illustrating a communication system 100 according to some embodiments. The communication system 100 includes a source device 102 and a plurality of electronic devices 120 (e.g., electronic devices 120-1 to 120-m) communicatively coupled to each other via one or more networks. In some embodiments, the communication system 100 is a streaming system, for example, for use with video-enabled applications such as video conferencing applications, digital TV applications, and media storage and / or distribution applications.

[0027] Source device 102 includes a video source 104 (e.g., a camera device component or media storage device) and an encoder component 106. In some embodiments, the video source 104 is a digital camera device (e.g., configured to create an uncompressed video sample stream). The encoder component 106 generates one or more encoded video bitstreams from the video stream. The video stream from video source 104 can have a higher data volume compared to the encoded video bitstream 108 generated by encoder component 106. Because the encoded video bitstream 108 has a lower data volume (less data) compared to the video stream from video source 104, it requires less bandwidth to transmit and less storage space to store compared to the video stream from video source 104. In some embodiments, source device 102 does not include encoder component 106 (e.g., configured to transmit uncompressed video to network 110).

[0028] One or more networks 110 represent any number of networks that transmit information between source device 102, server system 112, and / or electronic device 120, including, for example, wired (connected) and / or wireless communication networks. One or more networks 110 may exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and / or the Internet.

[0029] One or more networks 110 include server systems 112 (e.g., distributed / cloud computing systems). In some embodiments, server system 112 is or includes streaming servers (e.g., configured to store and / or distribute video content such as encoded video streams from source device 102). Server system 112 includes codec components 114 (e.g., configured to encode and / or decode video data). In some embodiments, codec components 114 include encoder components and / or decoder components. In various embodiments, codec components 114 are instantiated as hardware, software, or a combination thereof. In some embodiments, codec components 114 are configured to decode encoded video bitstream 108 and re-encode the video data using different encoding standards and / or methods to generate encoded video data 116. In some embodiments, server system 112 is configured to generate multiple video formats and / or encodings based on encoded video bitstream 108. In some embodiments, server system 112 serves as a Media-Aware Network Element (MANE). For example, server system 112 can be configured to trim the encoded video bitstream 108 to tailor potentially different bitstreams for one or more of the electronic devices 120. In some implementations, MANE is provided separately from server system 112.

[0030] Electronic device 120-1 includes a decoder component 122 and a display 124. In some embodiments, the decoder component 122 is configured to decode encoded video data 116 to generate an outgoing video stream that can be displayed on a display or other type of presentation device. In some embodiments, one or more of the electronic devices 120 do not include a display component (e.g., communicatively coupled to an external display device and / or includes media storage). In some embodiments, electronic device 120 is a streaming client. In some embodiments, electronic device 120 is configured to access server system 112 to obtain encoded video data 116.

[0031] The source device and / or multiple electronic devices 120 are sometimes referred to as “terminal devices” or “user devices”. In some implementations, one or more of the source device 102 and / or electronic devices 120 are instances of server systems, personal computers, portable devices (e.g., smartphones, tablets, or laptops), wearable devices, video conferencing equipment, and / or other types of electronic devices.

[0032] In an example operation of communication system 100, source device 102 transmits an encoded video bitstream 108 to server system 112. For example, source device 102 may encode and decode a stream of images captured by the source device. Server system 112 receives the encoded video bitstream 108 and may decode and / or encode the encoded video bitstream 108 using codec component 114. For example, server system 112 may apply encoding to video data that is better suited for network transmission and / or storage. Server system 112 may transmit encoded video data 116 (e.g., one or more encoded video bitstreams) to one or more electronic devices 120. Each electronic device 120 may decode the encoded video data 116 and optionally display video images.

[0033] Figure 2A This is a block diagram illustrating example elements of an encoder component 106 according to some embodiments. The encoder component 106 receives video data (e.g., a source video sequence) from a video source 104. In some embodiments, the encoder component includes a receiver (e.g., transceiver) component configured to receive the source video sequence. In some embodiments, the encoder component 106 receives the video sequence from a remote video source (e.g., a video source that is a component of a device different from the encoder component 106). The video source 104 may provide the source video sequence in the form of a digital video sample stream, which may have any suitable bit depth (e.g., 8-bit, 10-bit, or 12-bit), any color space (e.g., BT.601 YCrCb or RGB), and any suitable sampling structure (e.g., YCrCb 4:2:0 or YCrCb 4:4:4). In some embodiments, the video source 104 is a storage device storing previously captured / prepared video. In some embodiments, the video source 104 is a camera device that captures local image information as a video sequence. Video data can be provided as multiple individual images that are given motion when viewed sequentially. Each image can be organized as a spatial array of pixels, where, depending on the sampling structure, color space, etc., each pixel may include one or more samples. The relationship between pixels and samples will be readily understood by those skilled in the art.

[0034] Encoder component 106 is configured to encode, decode, and / or compress images of a source video sequence into a encoded video sequence 216 in real time or under other time constraints required by the application. In some embodiments, encoder component 106 is configured to perform a conversion between the source video sequence and a visual media data bitstream (e.g., a video bitstream). Implementing an appropriate encoding / decoding speed is a function of controller 204. In some embodiments, controller 204 controls and is functionally coupled to other functional units as described below. Parameters set by controller 204 may include rate control-related parameters (e.g., image skipping, quantizer, and / or the λ value of rate-distortion optimization techniques), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. Other functions of controller 204 can be readily identified by those skilled in the art, as these functions may belong to encoder component 106 optimized for a particular system design.

[0035] In some implementations, encoder component 106 is configured to operate within a codec loop. In a simplified example, the codec loop includes a source codec 202 (e.g., responsible for creating symbols, such as a symbol stream, based on the input image to be encoded and a reference image) and a (local) decoder 210. Decoder 210 reconstructs the symbols in a manner similar to that of the (remote) decoder to create sample data (when compression between the symbols and the encoded video bitstream is lossless). The reconstructed sample stream (sample data) is input to a reference image memory 208. Since decoding of the symbol stream produces bit-accurate results regardless of the decoder's location (local or remote), the contents of the reference image memory 208 are also bit-accurate between the local and remote encoders. In this way, the encoder's prediction portion interprets the same sample values ​​as the sample values ​​that the decoder interprets during prediction as reference image samples.

[0036] The operation of decoder 210 can be combined with a remote decoder, for example, as shown below. Figure 2B The operation of the decoder component 122 is the same as described in the detailed description. However, a brief reference is provided. Figure 2B Since symbols are available and encoding / decoding symbols into a codec video sequence by entropy codec 214 and parser 254 can be lossless, the entropy decoding portion of decoder component 122, including buffer memory 252 and parser 254, does not need to be fully implemented in local decoder 210.

[0037] Besides parsing / entropy decoding, the decoder techniques described in this paper can exist in the corresponding encoders in essentially the same form. For this reason, the subject matter focuses on decoder operations. Furthermore, the description of the encoder techniques can be simplified, as the encoder techniques are inverses of the decoder techniques.

[0038] As part of the operation of source codec 202, source codec 202 can perform motion-compensated predictive coding, which predictively codes the input frame with reference to one or more previously codecped frames from the video sequence designated as reference frames. In this manner, codec engine 212 encodes the differences between pixel blocks of the input frame and pixel blocks of the reference frame, which can be selected as the prediction reference for the input frame. Controller 204 can manage the coding and decoding operations of source codec 202, including, for example, setting parameters and subgroup parameters for encoding video data.

[0039] Decoder 210 decodes encoded video data based on symbols created by source codec 202, which can be designated as reference frames. The operation of codec engine 212 can advantageously support lossy processing. When encoded video data is processed by video decoder (… Figure 2A When decoded at (not shown), the reconstructed video sequence can be a copy of the source video sequence with some errors. Decoder 210 can replicate the decoding process performed on the reference frame by a remote video decoder, and the reconstructed reference frame can be stored in reference image memory 208. In this way, encoder component 106 locally stores a copy of the reconstructed reference frame that shares common content (no transmission errors) with the reconstructed reference frame to be obtained by the remote video decoder.

[0040] Predictor 206 can perform a prediction search against codec engine 212. That is, for a new frame to be encoded or decoded, predictor 206 can search the reference image memory 208 for sample data (as candidate reference pixel blocks) or specific metadata such as reference image motion vectors, block shapes, etc., that can be used as appropriate prediction references for the new image. Predictor 206 can operate pixel-by-pixel based on the sample blocks to find appropriate prediction references. As determined by the search results obtained by predictor 206, the input image can have prediction references obtained from multiple reference images stored in reference image memory 208.

[0041] The outputs of all the aforementioned functional units can undergo entropy encoding / decoding in entropy codec 214. Entropy codec 214 converts the symbols generated by the various functional units into a codec video sequence by lossless compression of the symbols according to techniques known to those skilled in the art (e.g., Huffman coding, variable-length coding, and / or arithmetic coding).

[0042] In some implementations, the output of entropy codec 214 is coupled to a transmitter. The transmitter can be configured to buffer the encoded video sequence created by entropy codec 214 in preparation for transmission via communication channel 218, which can be a hardware / software link to a storage device storing the encoded video data. The transmitter can be configured to combine encoded video data from source codec 202 with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown). In some implementations, the transmitter can transmit additional data along with the encoded video. Source codec 202 can include such data as part of the encoded video sequence. Additional data can include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant images and slices, Supplementary Enhancement Information (SEI) messages, Visual Usability Information (VUI) parameter set fragments, etc.

[0043] Controller 204 can manage the operation of encoder component 106. During encoding and decoding, controller 204 can assign a specific encoding / decoding picture type to each encoded picture, which may affect the encoding / decoding technique applied to the corresponding picture. For example, pictures can be assigned as intra-frame pictures (I-pictures), prediction pictures (P-pictures), or bidirectional prediction pictures (B-pictures). Intra-frame pictures can be encoded / decoded without using any other frames in the sequence as prediction sources. Some video codecs allow different types of intra-frame pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are familiar with those variations of I-pictures and their corresponding applications and characteristics, and therefore they will not be repeated here. Predictive pictures can be encoded / decoded using inter-frame prediction or intra-frame prediction, which uses at most one motion vector and reference index to predict sample values ​​for each block. Bidirectional prediction pictures can be encoded / decoded using inter-frame prediction or intra-frame prediction, which uses at most two motion vectors and reference indexes to predict sample values ​​for each block. Similarly, multiple prediction images can use more than two reference images and associated metadata to reconstruct a single block.

[0044] Source images can typically be spatially subdivided into multiple sample blocks (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples), and encoded / decoded on a block-by-block basis. Predictive encoding / decoding can be performed on these blocks with reference to other (already encoded / decoded) blocks, determined by the encoding / decoding assignments of the corresponding images applied to the blocks. For example, blocks of an I-image can be nonpredictively encoded / decoded, or blocks of an I-image can be predictively encoded / decoded (spatial prediction or intra-frame prediction) with reference to already encoded / decoded blocks of the same image. Pixel blocks of a P-image can be nonpredictively encoded / decoded with reference to a previously encoded / decoded reference image via spatial prediction or temporal prediction. Blocks of a B-image can be nonpredictively encoded / decoded with reference to one or two previously encoded / decoded reference images via spatial prediction or temporal prediction.

[0045] Video can be captured as multiple source images (video images) in a time-series manner. Intra-frame image prediction (often simply called intra-prediction) utilizes spatial correlations within a given image, while inter-frame image prediction utilizes (temporal or other) correlations between images. In the example, a specific image in the encoding / decoding process (referred to as the current image) is segmented into blocks. If a block in the current image is similar to a reference block in a previously encoded and decoded reference image in the video that is still buffered, the block in the current image can be encoded and decoded using a vector called a motion vector. The motion vector points to the reference block in the reference image, and when using multiple reference images, the motion vector can have a third dimension that identifies the reference images.

[0046] Encoder component 106 can perform encoding and decoding operations according to any predetermined video codec technique or standard described herein. In operation, encoder component 106 can perform various compression operations, including predictive codec operations utilizing temporal and spatial redundancy in the input video sequence. Therefore, the encoded and decoded video data can conform to the syntax specified by the video codec technique or standard used.

[0047] Figure 2B This is a block diagram illustrating example elements of a decoder component 122 according to some embodiments. Figure 2B The decoder component 122 is coupled to channel 218 and display 124. In some embodiments, the decoder component 122 includes a transmitter coupled to loop filter 256 and configured to transmit data to display 124 (e.g., via a wired or wireless connection).

[0048] In some embodiments, decoder component 122 includes a receiver coupled to channel 218 and configured to receive data from channel 218 (e.g., via a wired or wireless connection). The receiver may be configured to receive one or more codec video sequences decoded by decoder component 122. In some embodiments, the decoding of each codec video sequence is independent of other codec video sequences. Each codec video sequence may be received from channel 218, which may be a hardware / software link to a storage device storing the encoded video data. The receiver may receive the encoded video data along with other data, such as codec audio data and / or auxiliary data streams, which may be forwarded to their respective user entities (not depicted). The receiver may separate the codec video sequences from other data. In some embodiments, the receiver receives additional (redundant) data along with the encoded video. The additional data may be included as part of the codec video sequence. Decoder component 122 may use the additional data to decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0049] According to some embodiments, the decoder component 122 includes a buffer memory 252, a parser 254 (sometimes also referred to as an entropy decoder), a scaler / inverse transform unit 258, an intra-frame image prediction unit 262, a motion compensation prediction unit 260, an aggregator 268, a loop filter unit 256, a reference image memory 266, and a current image memory 264. In some embodiments, the decoder component 122 is implemented as an integrated circuit, a series of integrated circuits, and / or other electronic circuit systems. The decoder component 122 can be implemented at least partially in software.

[0050] Buffer memory 252 is coupled between channel 218 and parser 254 (e.g., to combat network jitter). In some embodiments, buffer memory 252 is separate from decoder component 122. In some embodiments, a separate buffer memory is provided between the output of channel 218 and decoder component 122. In some embodiments, a separate buffer memory is provided outside decoder component 122 (e.g., to combat network jitter) in addition to buffer memory 252 inside decoder component 122 (e.g., buffer memory 252 is configured to handle playback timing). Buffer memory 252 may not be necessary or may be small when receiving data from a store / forward device with sufficient bandwidth and controllability or from an isochronous synchronization network. Buffer memory 252 may be required to make the best use of packet networks such as the Internet; buffer memory 252 may be relatively large and / or have an adaptive size and may be implemented at least partially in an operating system or similar component outside decoder component 122.

[0051] Parser 254 is configured to reconstruct symbols 270 from the encoded video sequence. Symbols may include, for example, information for managing the operation of decoder component 122, and / or information for controlling a presentation device such as display 124. Control information for the presentation device may be in the form of, for example, Supplementary Enhancement Information (SEI) messages or fragments of Video Usability Information (VUI) parameter sets (not depicted). Parser 254 parses (entropy decodes) the encoded video sequence. The encoding and decoding of the encoded video sequence may be performed according to video codec techniques or standards, and may follow principles known to those skilled in the art, including variable-length codecs, Huffman codecs, arithmetic codecs with or without context sensitivity, etc. Parser 254 may extract a subgroup parameter set from the encoded video sequence for at least one subgroup of pixels in the video decoder, based on at least one parameter corresponding to a group. Subgroups can include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The parser 254 can also extract information such as transform coefficients, quantizer parameter values, and motion vectors from the encoded and decoded video sequence.

[0052] Depending on the type of the encoded video picture or its components (e.g., inter-frame and intra-frame pictures, inter-frame and intra-frame blocks) and other factors, the reconstruction of symbol 270 may involve multiple different units. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the encoded video sequence by parser 254. For clarity, such subgroup control information flow between parser 254 and the following multiple units is not depicted.

[0053] The decoder component 122 can be conceptually subdivided into multiple functional units, and in some implementations, these units interact closely with each other and can be at least partially integrated with each other. However, for clarity, this document retains the conceptual subdivision of the functional units.

[0054] The scaler / inverse transform unit 258 receives quantized transform coefficients and control information (such as which transform to use, block size, quantization factor, and / or quantization scaling matrix) as symbols 270 from the parser 254. The scaler / inverse transform unit 258 can output blocks comprising sample values, which can be input to the aggregator 268. In some cases, the output samples of the scaler / inverse transform unit 258 belong to intra-frame codec blocks; that is, blocks that do not use prediction information from previously reconstructed images, but can use prediction information from previously reconstructed portions of the current image. Such prediction information can be provided by the intra-frame image prediction unit 262. The intra-frame image prediction unit 262 can use surrounding reconstructed information obtained from the current (partially reconstructed) image from the current image memory 264 to generate blocks of the same size and shape as the blocks in the reconstruction. The aggregator 268 can add the prediction information already generated by the intra-frame image prediction unit 262 to the output sample information provided by the scaler / inverse transform unit 258 based on each sample.

[0055] In other cases, the output samples of the scaler / inverse transform unit 258 belong to the inter-frame encoding and decoding and potentially to the motion-compensated block. In such cases, the motion-compensated prediction unit 260 can access the reference image memory 266 to obtain samples for prediction. After motion compensation of the obtained samples according to the symbol 270 belonging to the block, these samples can be added by the aggregator 268 to the output of the scaler / inverse transform unit 258 (referred to in this case as residual samples or residual signals) to generate output sample information. The address in the reference image memory 266 from which the motion-compensated prediction unit 260 obtains the predicted samples can be controlled by motion vectors. Motion vectors can be used by the motion-compensated prediction unit 260 in the form of symbol 270, which can have, for example, an X component, a Y component, and a reference image component. Motion compensation can also include, for example, interpolation of sample values ​​obtained from the reference image memory 266 when using subsampled precise motion vectors, and motion vector prediction mechanisms.

[0056] The output samples of aggregator 268 can undergo various loop filtering techniques in loop filter unit 256. Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video bitstream and available to loop filter unit 256 as symbols 270 from parser 254. However, video compression techniques may also respond to metadata obtained during decoding of previous (in decoding order) portions of the encoded picture or encoded video sequence, and to sample values ​​obtained from previous reconstruction and loop filtering. The output of loop filter unit 256 may be a sample stream that can be output to a presentation device such as display 124 and stored in reference picture memory 266 for future inter-frame picture prediction.

[0057] Once a certain codec image is reconstructed, it can be used as a reference image for future predictions. Once a codec image is reconstructed and has been identified as a reference image (e.g., by parser 254), the current reference image can become part of the reference image memory 266, and a new current image memory can be reallocated before reconstructing subsequent codec images begins.

[0058] Decoder component 122 can perform decoding operations according to a predetermined video compression technique that may be recorded in a standard such as any standard described herein. As specified in a video compression technique document or standard, and particularly in a configuration file therein, the encoded / decoded video may conform to the syntax specified by the video compression technique or standard used, in the sense that the encoded / decoded video sequence follows the syntax of the video compression technique or standard. Furthermore, the complexity of the encoded / decoded video sequence can be within a range defined by a level of the video compression technique or standard in order to conform to some video compression techniques or standards. In some cases, the level may limit the maximum image size, maximum frame rate, maximum reconstruction sample rate (measured, for example, megasamples per second), maximum reference image size, etc. In some cases, the limitations set by the level can be further restricted by the Hypothetical Reference Decoder (HRD) specification and metadata managed by the HRD buffer, which is represented as a signal in the encoded / decoded video sequence.

[0059] Figure 3 This is a block diagram illustrating a server system 112 according to some embodiments. The server system 112 includes a control circuitry system 302, one or more network interfaces 304, a memory 314, a user interface 306, and one or more communication buses 312 for interconnecting these components. In some embodiments, the control circuitry system 302 includes one or more processors (e.g., CPU, GPU, and / or DPU). In some embodiments, the control circuitry system includes a field-programmable gate array, a hardware accelerator, and / or an integrated circuit (e.g., an application-specific integrated circuit).

[0060] Network interface 304 can be configured to interface with one or more communication networks (e.g., wireless networks, wired networks, and / or optical networks). These communication networks can be local, wide area, metropolitan area, vehicle-mounted, industrial, real-time, latency-tolerant, etc. Examples of communication networks include: local area networks such as Ethernet; wireless LANs; cellular networks including GSM, 3G, 4G, 5G, LTE, etc.; wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV; vehicle and industrial networks including CANbus, etc. Such communication can be one-way receiving (e.g., broadcast TV), one-way transmitting (e.g., to a CANbus device), or bidirectional (e.g., to other computer systems using local or wide area digital networks). Such communication can include communication to one or more cloud computing networks.

[0061] User interface 306 includes one or more output devices 308 and / or one or more input devices 310. Input devices 310 may include one or more of the following: keyboard, mouse, touchpad, touchscreen, data glove, joystick, microphone, scanner, camera, etc. Output devices 308 may include one or more of the following: audio output devices (e.g., speakers), visual output devices (e.g., displays or monitors), etc.

[0062] Memory 314 may include high-speed random access memory (e.g., DRAM, SRAM, DDR RAM, and / or other random access solid-state memory devices) and / or non-volatile memory (e.g., one or more disk storage devices, optical disk storage devices, flash memory devices, and / or other non-volatile solid-state memory devices). Memory 314 may optionally include one or more storage devices remote from the control circuitry system 302. Memory 314, or alternatively, the non-volatile solid-state memory devices within memory 314 include non-transitory computer-readable storage media. In some embodiments, memory 314 or the non-transitory computer-readable storage media of memory 314 stores programs, modules, instructions, and data structures, or subsets or supersets thereof: Operating system 316, which includes procedures for handling various basic system services and for performing hardware-related tasks; A network communication module 318 is used to connect the server system 112 to other computing devices via one or more network interfaces 304 (e.g., via wired and / or wireless connections); Codec module 320 is used to perform various functions related to encoding and / or decoding data such as video data. In some embodiments, codec module 320 is an instance of codec component 114. Codec module 320 includes, but is not limited to, one or more of the following: Decoding module 322, which performs various functions related to decoding encoded data, such as those previously described with respect to decoder component 122; and Encoding module 340, which performs various functions related to encoding data, such as those previously described with respect to encoder component 106; and Image memory 352 is used to store images and image data, for example, for use with encoding / decoding module 320. In some embodiments, image memory 352 includes one or more of the following: reference image memory 208, buffer memory 252, current image memory 264, and reference image memory 266.

[0063] In some implementations, the decoding module 322 includes a parsing module 324 (e.g., configured to perform various functions previously described with respect to the parser 254), a transformation module 326 (e.g., configured to perform various functions previously described with respect to the scaler / inverse transformation unit 258), a prediction module 328 (e.g., configured to perform various functions previously described with respect to the motion compensation prediction unit 260 and / or the intra-frame picture prediction unit 262), and a filter module 330 (e.g., configured to perform various functions previously described with respect to the loop filter 256).

[0064] In some embodiments, the encoding module 340 includes a code module 342 (e.g., configured to perform various functions previously described with respect to source codec 202 and / or codec engine 212) and a prediction module 344 (e.g., configured to perform various functions previously described with respect to predictor 206). In some embodiments, the decoding module 322 and / or the encoding module 340 includes... Figure 3 A subset of the modules shown. For example, a shared prediction module is used by both the decoding module 322 and the encoding module 340.

[0065] Each of the modules identified above and stored in memory 314 corresponds to a set of instructions for performing the functions described herein. The modules identified above (e.g., instruction sets) do not need to be implemented as separate software programs, processes, or modules, and therefore various subsets of these modules can be combined or otherwise rearranged in various implementations. For example, the codec module 320 may optionally not include separate decoding and encoding modules, but instead use the same set of modules to perform both sets of functions. In some implementations, memory 314 stores a subset of the modules and data structures identified above. In some implementations, memory 314 stores additional modules and data structures not described above.

[0066] although Figure 3 A server system 112 according to some embodiments is shown, but Figure 3 This is intended more as a functional description of various features that can exist in one or more server systems than as a structural diagram of the implementation described herein. In practice, items shown individually may be combined and some items may be separated. For example, Figure 3Some items shown individually can be implemented on a single server, and a single item can be implemented by one or more servers. The actual number of servers used to implement server system 112 and how features are distributed among them will vary depending on the implementation and, optionally, in part, depend on the amount of data traffic processed by the server system during peak usage periods and during average usage periods.

[0067] Example encoding and decoding techniques

[0068] The encoding and decoding processes and techniques described below can be performed on the aforementioned devices and systems (e.g., source device 102, server system 112, and / or electronic device 120). According to some embodiments, methods for using selective transform sets are described below.

[0069] As described in more detail below, if the intra-prediction mode is not a smooth mode, or if the intra-prediction mode is generating prediction samples based on a given prediction direction, then the intra-prediction mode may be referred to as an angle mode or a direction mode.

[0070] As discussed above, a block can refer to a codec tree block, a maximum codec block, a predefined fixed block size, a codec block, a prediction block, a residual block, or a transform block. An inter-mode codec block (or inter-block) refers to a block that uses inter-prediction modes or a combination of intra-inter-prediction modes. An inter-mode can also refer to a block encoded using block vectors, which are used, for example, to extract prediction blocks within the same frame using intra-block duplication. An intra-mode codec block (or intra-block) refers to a block that uses intra-prediction modes or a combination of intra-inter-prediction modes. An intra-mode list can correspond to a list of the most probable intra-prediction modes for the current block. Additionally, the term "segmentation" can correspond to block segmentation or transform segmentation.

[0071] As an example, a Coding Tree Unit (CTU) can be split into Coding Units (CUs) using a quadtree structure represented as a codec tree to suit various local characteristics. In some implementations, a decision is made at the CU level regarding whether to encode and decode a picture region using inter-frame (temporal) prediction or intra-frame (spatial) prediction. Each CU can be further split into one, two, or four prediction units (PUs) depending on the PU splitting type. Within a PU, the same prediction processing is applied, and relevant information can be transferred to the decoder based on the PU. After obtaining the residual block by applying prediction processing based on the PU splitting type, the CU can be divided into transform units (TUs) according to another quadtree structure, such as the codec tree used for the CU.

[0072] The concept of multiple partition unit types can be replaced by a quadtree with nested multi-type trees using binary and ternary splitting structures. In the codec tree structure, a CU can be square or rectangular. A CTU is first partitioned by a quadtree structure. Quadtree leaf nodes can be further partitioned using multi-type tree structures. Example multi-type tree structures include four splitting types. Multi-type leaf nodes are called CUs unless the CU is too large for the maximum transform length. This means that CUs, PUs, and TUs can have the same block size in a quadtree with a nested multi-type tree codec block structure.

[0073] For example, in VTM7, the codec tree scheme supports the ability for luma and chroma to have separate block tree structures. In some cases, for P-slices and B-slices, the luma CTB and chroma CTB within a CTU share the same codec tree structure. However, for I-slices, luma and chroma can have separate block tree structures. When a separate block tree mode is applied, the luma CTB is divided into CUs via one codec tree structure, and the chroma CTB is divided into chroma CUs via another codec tree structure. This means that a CU in an I-slice can include codec blocks for the luma component or codec blocks for both chroma components, or consist of codec blocks for the luma component or codec blocks for both chroma components, while a CU in a P-slice or B-slice can always include codec blocks for all three color components or consist of codec blocks for all three color components, unless the video is monochrome.

[0074] Turning now to transforms and transform blocks, the transforms performed during the decoding of the video bitstream can be the inverses of the transforms performed during the encoding of the video bitstream, and are sometimes referred to as "inverse transforms." It is worth noting that when the encoder component applies a transform, the decoder component performs the inverse transform. Therefore, in the following description, the transforms described in the context of the decoder component can be the inverses of the transforms applied to the encoder side. For simplicity, the transforms described herein can be referred to as "transforms," ​​regardless of whether they are performed during encoding or decoding.

[0075] Multiple transform sizes (e.g., ranging from 4 to 64 points per dimension) and transform shapes (e.g., squares or rectangles with width / height ratios of 2:1 / 1:2 and 4:1 / 1:4) can be utilized. As described further below, transforms can correspond to primary or secondary transforms, and to separable or non-separable transforms. A transform set is a grouping of one or more transform types. Therefore, a transform set indicates a grouping of multiple transform kernels / basis. Each entry in a transform set can be referred to as a transform candidate. For each block, the transform candidate selected from the transform set can be represented by a signal or implicitly identified.

[0076] Some implementations include methods for selecting transform sets and / or types to represent intra- and / or inter-frame codec blocks using signals. Transform types can belong to the families of sine transforms, KLTs, or line graph transforms (LGTs). (Main or quadratic) transforms can belong to the family of sine transforms (DCTs, DSTs, inverted versions of DCTs and ADSTs). A DCT can refer to any transform using a transform kernel derived from a discrete cosine transform basis, and a DST / ADST can refer to any transform using a transform kernel derived from a discrete sine transform basis.

[0077] Example primary transforms can belong to the generalized line graph transform (LGT) family, or they can be based on trained kernels. Example set of secondary transforms can be a grouping of one or more inseparable secondary transform kernel transform types. A unique or common set of secondary transforms can be defined for each primary transform type and / or intra-frame or inter-frame mode type.

[0078] Furthermore, an inseparable transform can refer to a master transform applied directly to the residual or a quadratic transform applied to a block of transform coefficients generated by the master transform. Transform kernels can be grouped into sets represented by set indices and kernel indices within those sets. An inseparable quadratic transform can be a trained kernel applied to the master transform coefficients at the encoder or to the dequantized coefficients at the decoder.

[0079] An inseparable quadratic transformation kernel can be considered as a set of basis vectors in a vector space. If represented as an M×N (M rows and N columns) matrix, then N corresponds to the dimension of the vector space, and M corresponds to the number of basis vectors. Therefore, M×N can be used to represent the kernel size. Examples of kernel sizes include, but are not limited to, 64×64, 32×64, 16×64, 8×84, 4×64, 16×16, 8×16, 4×16, 8×8, and 4×4 samples.

[0080] Scanning order refers to the coefficient recombination process of mapping a two-dimensional principal transform coefficient array to a one-dimensional principal transform coefficient array that serves as the input to the forward quadratic transform. It can also refer to the backward coefficient recombination process of mapping a one-dimensional quadratic transform coefficient array back to a two-dimensional principal transform coefficient array.

[0081] The End-of-Block (EOB) value corresponds to the position of the last valid coefficient in the code-decode block after a given coefficient scan order. The EOB value can correspond to the position of the last valid (e.g., non-zero) coefficient in the code-decode block after a given coefficient scan order. For a given coefficient scan order, all coefficients outside the EOB position are zero. In some implementations, if an M×N inseparable quadratic transform kernel is applied to the code-decode block, the EOB value ≤ M.

[0082] Two-dimensional transform processing may involve the use of hybrid transform kernels (e.g., composed of different one-dimensional transforms for each dimension of the encoded / decoded residual block). A single one-dimensional transform may include at least one of the following: a) 4-point, 8-point, 16-point, 32-point, or 64-point discrete cosine transforms; b) 4-point, 8-point, or 16-point asymmetric discrete sine transforms and their inverted versions; or c) 4-point, 8-point, 16-point, or 32-point identity transforms.

[0083] For chroma components, transform type selection can be performed implicitly. For intra-frame prediction residuals, the transform type can be selected based on the intra-frame prediction mode. For inter-frame prediction residuals, the transform type can be selected based on the transform type selection of the co-occurring luma block. Therefore, for chroma components, transform type signaling in the bitstream is not required.

[0084] Now let's move on to example encoding and decoding using prediction blocks and residual blocks. Figure 4A The calculation of a prediction block according to some implementations is shown. Figure 4A In the example, intra-frame prediction is performed on the current block 402 to generate prediction block 404. In some implementations, inter-frame prediction is performed to generate prediction blocks. The current block 402 includes a sample set (e.g., pixel blocks), and prediction block 404 includes a prediction set corresponding to that sample set. Figure 4B The calculation of residual blocks according to some implementation methods is shown. For example... Figure 4B As shown, prediction block 404 is subtracted from the current block 402 to generate residual block 406, which includes the residual set. For example, the corresponding difference between each sample and the corresponding prediction is calculated. Figure 4C The calculation of the reconstructed blocks according to some implementation methods is shown. For example... Figure 4C As shown, residual block 406 undergoes one or more transforms and quantizations to generate a set of residual coefficients. This set of residual coefficients can be transferred from the encoder component to the decoder component. The set of residual coefficients undergoes inverse quantization and inverse transform to generate a reconstructed residual block 408. The reconstructed residual block 408 is combined with the prediction block 404 (e.g., the reconstructed residuals of the reconstructed residual block 408 are added to the predictions of the prediction block 404) to generate a reconstructed block 410 corresponding to the current block 402.

[0085] Figure 5A An example of low-frequency non-separable transform (LFNST) processing according to some implementations is shown. LFNST is also known as reduced quadratic transform (RST). For example... Figure 5A As shown, LFNST can be applied between the forward master transform 502 and quantization 506 (e.g., at the encoding unit) and between dequantization 508 and inverse master transform 512 (e.g., at the decoding unit).

[0086] Example LFNST 504 may include 16 input coefficients for a 4×4 forward LFNST or 64 input coefficients for an 8×8 forward LFNST. In the example LFNST, either a 4×4 or 8×8 inseparable transform is applied depending on the block size. For example, a 4×4 LFNST can be applied to small blocks (e.g., with a width or height of less than 8 samples), and an 8×8 LFNST can be applied to larger blocks (e.g., with a width or height of more than 4 samples). Example LFNST 510 may include 8 input coefficients for a 4×4 inverse LFNST or 16 input coefficients for an 8×8 inverse LFNST.

[0087] As an example, there can be four transform sets for LFNST and two inseparable transform matrices (kernels) for each transform set. The mapping from intra-prediction modes to transform sets can be predefined as shown in Table 1 below. If one of the three CCLM modes (INTRA_LT_CCLM, INTRA_T_CCLM, or INTRA_L_CCLM) is used for the current block (81 <= intra-prediction mode <= 83), then transform set 0 is selected (e.g., for the current chroma block). For each transform set, the selected inseparable quadratic transform candidate can be further specified by an LFNST index explicitly represented as a signal. The index can be represented as a signal in the bitstream once per intra-CU (e.g., after the transform coefficients).

[0088]

[0089] Table 1 - Transformation Selection Table

[0090] The LFNST described above can be adjusted as follows: First, more transform sets can be used for finer directional granularity (e.g., 35 transform sets instead of 4). Second, more transform kernels can be included in each set (e.g., 3 transform kernels instead of 2). Third, three different kernels (e.g., LFNST4, LFNST8, and LFNST16) can be specified to indicate the LFNST kernel set, which can be applied to 4×N / N×4 (N ≥ 4), 8×N / N×8 (N ≥ 8), and M×N (M, N ≥ 16), respectively. Fourth, the LFNST set (LFNST_set_idx) for a given intra-pred_mode can be derived according to different formulas, as shown in Table 2.

[0091] In another example, for Intra_pred_mode < 2, LFNST_set_idx equals 2, for Intra_pred_mode in [0, 34], LFNST_set_idx = Intra_pred_mode, and for Intra_pred_mode in [35, 66], LFNST_set_idx = 68 - Intra_pred_mode.

[0092] The Non-Separable Master Transform (NSPT) can be used to replace the separable DCT-II plus LFNST transform combination, for example, for block shapes of 4×4, 4×8 / 8×4, 4×16 / 16×4, 8×8, 8×16 / 16×8, 4×32 / 32×4, and 8×32 / 32×8. The NSPT set index for a given intra-frame mode can also be derived based on Table 2.

[0093]

[0094] Table 2 - Mapping between Intra-Frame Modes and LFNST Set Indexes

[0095] In some implementations, LFNST is restricted to apply only when all coefficients outside the first coefficient subgroup are invalid. In these implementations, LFNST index encoding / decoding depends on the position of the last valid coefficient. The LFNST index can be context-coded (e.g., independent of the intra-prediction mode). In some implementations, only the first binary number is context-coded. In some implementations, LFNST is applied to intra-CUs in both intra-slices and inter-slices, and to both the luma and chroma components. If dual-tree is enabled, the LFNST indexes for the luma and chroma components can be represented as signals, respectively. For inter-slices (where dual-tree is disabled), a single LFNST index can be represented as a signal and used for both the luma and chroma components.

[0096] In some implementations, when the Intra-Frame Sub-Partitioning (ISP) mode is selected for the current block, LFNST can be disabled and the RST index can be omitted (e.g., because the performance improvement may be negligible even if RST is applied to every feasible partition block). Additionally, disabling RST for ISP prediction residuals can reduce coding complexity. When the Matrix-Based Intra-Frame Prediction (MIP) mode is selected, LFNST can also be disabled (and the signal representation index can be omitted).

[0097] Due to the existing maximum transform size limit (64×64), large CUs (e.g., larger than 64×64) can be implicitly partitioned (TU tiling). For a certain number of decoding pipeline stages, LFNST index search can increase the data buffer size by a factor of four. Therefore, the maximum size allowed by LFNST can be limited (e.g., limited to 64×64). In some implementations, LFNST is enabled only with DCT2.

[0098] In some implementations, a separable transform is applied to intra-frame and inter-frame residual samples. In other implementations, an intra-secondary transform (IST) scheme is customized for the video codec library (e.g., for transforming intra-frame residual blocks). Compared to the non-separable master transform, the IST scheme can efficiently capture directional patterns in intra-frame residual samples with lower complexity. In the IST scheme, the IST kernel can be classified using a nominal intra-frame prediction angle.

[0099] In some implementations, a quadratic transform method (e.g., IST) is applied to the main transform coefficient block before quantization is applied at the encoder (e.g., for the intra-prediction residual block of the luma component). Therefore, the inverse quadratic transform can be applied to the dequantized transform coefficient block before the inverse main transform is applied at the decoder. In some implementations, IST is not applied to the chroma color component. Figure 5B The diagram illustrates the use of IST in encoding and decoding processes. Figure 5B A master transform 552 is shown applied to the residual block (e.g., corresponding to an intra-prediction block). A second transform 554 is applied to the output of the master transform 552. Quantization 556 is applied to the output of the second transform 554, and the resulting quantization coefficients are entropy encoded 558 and represented as signals via the video bitstream. The video bitstream is parsed 560 (e.g., at the decoder), and the quantization coefficients are dequantized 562. An inverse second transform 564 is applied to the dequantized data, and an inverse master transform 566 is applied to the output of the second transform 564. In this way, the reconstructed residual block is generated.

[0100] In some implementations, 12 quadratic transform sets (or IST sets) are defined, each containing 3 quadratic transform kernels. In some implementations, for each intra-encoder transform block, the nominal intra-prediction mode and main transform type can be identified, and then the IST set is selected based on Table 3 below. In some implementations, for Paeth prediction mode and recursive intra-prediction mode, neither signal representation of the IST is applied nor used.

[0101]

[0102] Table 3 - Selection of the Quadratic Transform Set

[0103] Given an IST set with 3 cores, there are four encoder options: 1) no secondary transform, 2) secondary transform using the first transform core in the given IST set, 3) secondary transform using the second transform core in the given IST set, and 4) secondary transform using the third transform core in the given IST set. The encoder can use syntax elements (e.g., ist_idx) to signal-represent this selection. At the decoder, the values ​​of the syntax elements are parsed, and the secondary transform core is identified given the IST set and the values ​​associated with ist_idx. After signal-representing the main transform type, the syntax element (ist_idx) can be signal-represented for each luma transform block. For example, signal-representing ist_idx can be performed if at least one of the following is true: (i) the current block is an intra-encoded luma transform block, (ii) the main transform type is DCT in two dimensions or ADST in two dimensions, (iii) the intra-prediction mode is neither Paeth prediction mode nor recursive intra-prediction mode, (iv) the transform partition depth is 0, and (v) the EOB position falls within the low-frequency transform coefficient region to which the secondary transform applies. In some implementations, the entropy encoding / decoding context of ist_idx is derived based on the transform block size. Conceptually, IST can be considered another name for LFSNT. As those skilled in the art will understand, IST and LFNST are interchangeable in this disclosure.

[0104] The quadratic transform (whether called LFNST, NSPT, or IST) is not limited to application only to intra-prediction blocks. For example... Figure 5C As shown, when the block is an inter-frame prediction block or uses intra-frame block copy mode prediction, the decoder-side intra-frame mode derivation (DIMD) can be applied based on the template. Figure 5C An example is shown illustrating how to derive intra-frame prediction modes using reconstructed samples from a template, according to some implementation methods. Figure 5C The image shows the template region 578 used for the current block 580. During DIMD, a 3×3 window 582 is used to evaluate the reference sample set. The window 582 can slide within the template region to obtain different candidates. Figure 5CAs shown, samples located at the top and left of the current codec block can be used as templates. Specifically, the template includes three sample rows and columns, and the texture orientation of neighboring samples can be estimated through the following steps. First, with the highlighted middle row and column samples as the center, horizontal and vertical Sobel 3×3 filters can be applied to a 3×3 window to obtain the horizontal gradient Gx and vertical gradient Gy. Next, the ratio of Gx to Gy can be calculated for each 3×3 window within the template (e.g., intra-mode information of neighboring templates). Then, the ratio of Gx to Gy can be matched against the closest intra-prediction mode in the regular pattern, and the matched intra-modes are counted and stored in the histogram. Next, the 3×3 windows can be slid across the template, and the histogram is updated based on the matched intra-modes for each 3×3 window. Finally, one or more of the highest frequency matched intra-modes associated with the derived texture orientation in the histogram can be used as intra-prediction modes. For example, a set of the top five histograms is collected, and the five corresponding intra-frame modes at the top are obtained, denoted as M0, M1, M2, M3, and M4 (e.g., where M0 has the highest histogram and M4 has the lowest histogram). The final predictor can be a fusion of a non-directional predictor (e.g., a planar mode predictor) and the five predictors predicted by M0 through M4 respectively.

[0105] For ease of description, the most frequently used mode M0 (e.g., derived by collecting the histogram of the template or the occurrence of neighboring blocks) is named the main derivation mode from the decoder side. However, the derived intra-frame mode may include more than one intra-frame mode.

[0106] In some implementations, the intra-frame pattern is derived based on the most frequent intra-frame pattern occurring in neighboring blocks (e.g., including both adjacent and non-adjacent neighboring blocks). For the purpose of deriving the intra-frame pattern, the locations of adjacent and non-adjacent neighboring blocks can be predefined. Figure 5D An example of a non-adjacent spatial neighbor candidate for a current codec unit according to some implementations is shown. For example... Figure 5D As shown, non-adjacent blocks 14-31 (or some subset thereof) can be used to derive the intra-frame mode. In some implementations, other non-adjacent blocks (in...) can be used. Figure 5D (Unmarked). The DIMD patterns described herein are examples, and in some implementations, other DIMD methods are employed.

[0107] Figure 6AThis is a flowchart illustrating a method 600 for decoding video according to some embodiments. Method 600 can be executed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having a control circuitry and a memory storing instructions executed by the control circuitry. In some embodiments, method 600 is executed by executing instructions stored in the computing system's memory (e.g., memory 314).

[0108] The system receives (602) a video bitstream including the current block. The system identifies (604) a first prediction mode for the current block. The system maps (606) the first prediction mode to the corresponding intra-prediction mode. The system selects (608) a first transform set from multiple transform sets based on the corresponding intra-prediction mode. The system reconstructs (610) the current block using the first transform set. In this way, multiple transform sets are available when selecting a transform set for the residual block, and the index (tx_set_idx) of the selected transform set is derived based on the prediction information and the offset represented by the signal.

[0109] In some implementations, when the prediction block is an inter-prediction block, a decoder-side intra-mode derivation (DIMD) method is applied to derive its intra-prediction mode. For example, for a master derivation mode from the decoder side, denoted as M0, the final intra-prediction (N+M0) is derived based on the signal-represented offset N and the derived intra-prediction mode M, where M0 is a positive integer and N is a negative or positive integer. Based on the final intra-prediction mode, tx_set_idx can be derived using a predefined mapping table, such as Table 2. As an example, the number of allowed signal-represented offsets can be limited to R, where R is a positive integer. For example, when R equals 3, there are three allowed offsets, thus limiting the selection of transform sets to three transform sets.

[0110] In some implementations, the offset, represented by a signal, is selected from a predefined set. For example, when the set is {-2, 0, 2}, the offset value represented by the signal restricts the selection of the transform set to its neighboring transform sets (when the offset is equal to -2, +2), or does not use an offset (when the offset is equal to 0). The predefined set can be adaptively determined based on encoded / decoded information such as intra-prediction mode, transform segmentation mode / depth, transform coefficient values, last non-zero position (or EOB), the number of non-zero transform coefficients, and block width and height.

[0111] In some implementations, the prediction block is an inter-frame prediction block or predicted using intra-frame block replication, and its intra-frame prediction mode is derived using a decoder-side intra-frame mode derivation method. Assuming the derived intra-frame modes are M0 to M4, the offsets, represented by signals, indicate which modes from M0 to M4 are used. For example, three of the five derived modes (M0 to M4) are determined (corresponding to the number of offsets in a predefined set). In one example, offset 0 means using M0 to determine the transform set index. In one example, offset -2 means using M1 to determine the transform set index. In one example, offset 2 means using M2 to determine the transform set index. In another example, the number of offsets in the predefined set is extended to the same number of derived modes.

[0112] In some implementations, blocks are inter-frame predicted, and the inter-frame prediction mode is a combination of intra-frame and inter-frame prediction modes. In one example, the intra-frame mode is the mode used to generate intra-frame predictions. For example, the offset plus the intra-frame mode determines the final transform set index. In another example, the proposed selective transform set method is disabled for a combination of intra-frame and inter-frame prediction modes.

[0113] In some implementations, when a block is inter-predicted and the inter-prediction mode is GPM, an intra-prediction mode is derived for its partitioning direction. For example, an offset plus the intra-prediction mode determines the final transform set index. In one example, if a partition of a block predicted using the GPM mode is intra-predicted, the offset, represented by a signal, plus the intra-prediction mode determines the final transform set index. In one example, only one additional offset (e.g., -2) is added, indicating whether the transform set index is determined using the intra-prediction partition's intra-modulation mode or the intra-modulation mode corresponding to the partitioning direction. In another example, two additional offsets are added, indicating whether the transform set index is determined using the intra-prediction partition's intra-modulation mode, the intra-modulation mode corresponding to the partitioning direction, and / or the master derivation mode from the decoder side. In yet another example, the proposed selective transform set method is disabled for the GPM mode.

[0114] In some implementations, the number of available transform sets is expanded, and patterns between [-15, -1] and [67, 81] are mapped to additional transform sets.

[0115] In one example, the number of additional transform sets is 15, and each additional transform set corresponds to the mode range [-15, -1]. For modes in [67, 81], a transpose operation similar to that in the current design is applied. For example, the number of available transform sets may be less than 15, and multiple modes may be mapped to a single transform set.

[0116] In some implementations, the number of available transform sets is reduced to reduce memory storage. For example, the number of intra-frame modes used to select transform sets is reduced (e.g., the number of directional intra-frame modes is reduced by half).

[0117] In some implementations, the block prediction mode is the IBC mode, and the IBC mode is mapped to the corresponding intra-frame prediction mode in a similar manner to that described above for the inter-frame prediction mode.

[0118] Figure 6B This is a flowchart illustrating a method 650 for encoding video according to some embodiments. Method 650 can be executed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having a control circuitry and a memory storing instructions executed by the control circuitry. In some embodiments, method 650 is executed by executing instructions stored in the computing system's memory (e.g., memory 314).

[0119] The system receives (652) video data comprising multiple blocks, including the current block. The system identifies (654) a first prediction mode for the current block. The system maps (656) the first prediction mode to a corresponding intra-prediction mode. The system selects (658) a first transform set from multiple transform sets based on the corresponding intra-prediction mode. The system encodes the current block using the first transform set (660). As previously described, the encoding process may reflect the decoding process described herein (e.g., transform set selection). For brevity, these details are not repeated here.

[0120] although Figure 6A and Figure 6B Multiple logical stages are shown in a specific order, but stages that are not dependent on the order can be reordered, and other stages can be combined or split. Some reorderings or other groupings not specifically mentioned will be obvious to those skilled in the art, and therefore the orderings and groupings presented herein are not exhaustive. Furthermore, it should be recognized that the stages can be implemented in hardware, firmware, software, or any combination thereof.

[0121] Now let's turn to some example implementations.

[0122] In some implementations, a multiple transform set selection method is applied to intra-blocks encoded and decoded using LFNST / NSPT. In this way, CUs encoded using decoder-side intra-mode derivation (DIMD), template-based intra-mode derivation (TIMD), MIP, spatial geometric partitioning mode (SGPM), enhanced intra-prediction (EIP), and intra-TMP mode can use alternative LFNST / NSPT transform sets. The transform set selection can be represented as a signal in the bitstream.

[0123] In some implementations, for CUs encoded using DIMD, TIMD, MIP, SGPM, EIP, and IntraTMP, the codec unit is configured to select an LFNST / NSPT transform set from two or more candidate sets. For example, if the current block is encoded using DIMD, TIMD, MIP, SGPM, IntraTMP, and LFNST / NSPT, additional binary numbers are used to indicate whether a first or second candidate transform set is selected. In some implementations, the first candidate transform set is the default transform set (e.g., the same as the current ECM design). In some implementations, the second candidate is derived through DIMD processing with neighboring reconstructed pixels. For example, if fusion is applied to the TIMD codec block, a second TIMD IPM can be used to derive the second candidate set. As another example, for an SGPM codec block, the two IPMs used by the SGPM can be used to derive the second candidate set. Simulated data on ECM-11 software with common test conditions has shown that selecting between two transform sets for LFNST / NSPT improves the encoding and decoding of the luma (Y) component by 0.08%. Simulated data on ECM-14 software with common test conditions have shown that choosing between two transform sets for LFNST / NSPT improves the encoding and decoding of the luminance (Y) component by 0.14%.

[0124] In ECM-13.0, LFNST / NSPT can be applied to inter-frame prediction (inter-frame coding / decoding) blocks, where the transform kernels used for intra-frame LFNST / NSPT can be reused from inter-frame blocks. Based on the prediction signal of the current block, a gradient histogram (HoG) can be constructed in a manner similar to DIMD, and the LFNST / NSPT kernel set can be determined using the first DIMD intra-frame prediction mode (IPM) corresponding to the highest amplitude. However, as described herein, having more than one option for selecting the kernel set may be beneficial. For example, for a codec block, a second IPM can be derived for inter-frame LFNST / NSPT. For example, in addition to the first DIMD IPM, a second DIMD IPM (corresponding to the second highest HOG amplitude) can be used as an additional IPM candidate. Simulated data on ECM-13 software with common test conditions has shown that using DIMD to select between two transform sets for LFNST / NSPT improves the coding / decoding of the luma (Y) component and chroma component (e.g., V component) by 0.02%.

[0125] Although EMC has been mentioned in the preceding paragraphs, those skilled in the art will recognize that the methods described herein can be used with many existing codecs, such as those mentioned in the background section.

[0126] (A1) In one aspect, some implementations include a video decoding method (e.g., method 600). In some implementations, the method is performed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having memory and one or more processors. In some implementations, the method is performed at a codec module (e.g., codec module 320). In some implementations, the method is performed by executing instructions stored in memory (e.g., memory 314) of one or more computing systems. The method includes: (i) receiving a video bitstream (e.g., a codec video sequence) including a current block; (ii) identifying a first prediction mode for the current block; (iii) mapping the first prediction mode to a corresponding intra-prediction mode; (iv) selecting a first transform set from a plurality of transform sets based on the corresponding intra-prediction mode; and (v) reconstructing the current block using the first transform set. For example, in the case where the prediction block is an inter-prediction block, a decoder-side intra-prediction mode derivation method is applied to derive its intra-prediction mode. Assuming the master derivation mode from the decoder side is M0, the final intra-prediction (N+M0) is derived based on the offset N represented by a signal and the derived intra-prediction mode M0, where M0 is a positive integer and N is a negative or positive integer. Based on the final intra-prediction mode, a predefined mapping table, such as Table 3, can be used to derive the transform set index (e.g., tx_set_idx). In some implementations, when selecting a transform set for a residual block, multiple transform sets are available, and the index of the selected transform set is derived based on the prediction information and the offset or adjustment represented by a signal. In some implementations, the first prediction mode is not a conventional intra-prediction mode (e.g., not a directional, DC, or planar intra-prediction mode).

[0127] (A2) In some implementations of A1, the first prediction mode is an inter-frame prediction mode. In some implementations, the mapping is performed using a decoder-side intra-frame mode derivation method.

[0128] (A3) In some implementations of A1, the first prediction mode is intra-block copy (IBC) mode. In some implementations, a decoder-side intra-mode derivation method is used to perform the mapping. In some implementations, the same method is used when the first prediction mode is inter-frame prediction mode or IBC mode.

[0129] (A4) In some implementations of A1, the first prediction mode includes a template matching prediction mode. For example, the template matching prediction mode may be an intra-template matching prediction (ITMP) mode. In some implementations, a decoder-side intra-mode derivation method is used to perform the mapping. In some implementations, the same method is used when the first prediction mode is an inter-frame prediction mode, an IBC mode, or a template matching prediction mode.

[0130] (A5) In some implementations of A1, the first prediction mode includes a combination of inter-frame prediction mode and intra-frame prediction mode, and the corresponding intra-frame prediction mode is based on the intra-frame portion of the combination of inter-frame prediction mode and intra-frame prediction mode. For example, the first prediction mode may be a combination of inter-frame-intra-frame prediction (CIIP) mode. As an example, the intra-frame mode is a mode used to generate intra-frame prediction. In some implementations, intra-frame prediction is combined with an offset represented by a signal to determine the transform set index of multiple transform sets. In some implementations, transform set selection is disabled for the combination of inter-frame prediction mode and intra-frame prediction mode. For example, when the combination of inter-frame-intra-frame prediction mode is used in the current block, only a single transform set is allowed.

[0131] (A6) In some implementations of A1, the first prediction mode includes a geometric partitioning mode (GPM). For example, the GPM may be a spatial GPM. In some implementations, the GPM has a partitioning direction and at least one inter-frame prediction. In some implementations, the GPM has a partitioning direction and at least one intra-frame prediction. In some implementations, a selective transform set method is disabled for the GPM. For example, if the GPM is used in the current block, only a single transform set is allowed.

[0132] (A7) In some implementations of A6, the corresponding intra-prediction mode is based on the segmentation direction of the GPM. For example, the corresponding intra-mode is derived as its segmentation direction (e.g., using DMID). As an example, the offset plus the intra-mode is used to determine the transform set index.

[0133] (A8) In some implementations of A6 or A7, the corresponding intra-prediction mode is based on the intra-prediction portion of the GPM. For example, if a block partition is intra-predicted, the transform set index is determined using a signaled offset plus the intra-prediction mode. In some implementations, the signaled offset indicates whether the transform set index is determined using the intra-prediction partition's intra-mode or the intra-mode corresponding to the partition direction. In some implementations, the signaled offset indicates whether the transform set index is determined using the intra-prediction partition's intra-mode, the intra-mode corresponding to the partition direction, or the master derivation mode from the decoder side.

[0134] (A9) In some embodiments of any of A1 to A8, the method further includes parsing syntax elements from the video bitstream to determine whether to adjust the corresponding intra-prediction, wherein the first transform set is selected based on the corresponding intra-prediction mode and syntax elements.

[0135] (A10) In some implementations of A9, the syntax element indicates an intra-mode offset, and wherein a first transform set is selected using an index derived by adding the intra-mode offset to the corresponding intra-prediction mode. For example, a final intra-prediction mode (N+M) is derived based on an offset N represented by a signal and an intra-prediction mode M, where M and / or N can be negative or positive integers. Based on the final intra-prediction mode, a transform set index is derived (e.g., using a predefined mapping table such as Table 3). In some implementations, the syntax element indicates which intra-prediction mode from the list of intra-prediction modes to use. For example, the syntax element may indicate whether to use the first intra-prediction mode (e.g., the highest-ranked mode) in the ranked list of intra-prediction modes, or the second intra-prediction mode (e.g., the second highest-ranked mode) in the ranked list.

[0136] (A11) In some implementations of A9 or A10, the syntax element indicates the changes to be made to the corresponding intra-prediction mode, and the number of allowed changes is limited to a specific value. For example, the number of allowed offsets, represented by signals, is limited to R, where R is a positive integer. For example, when R equals 3, there are three allowed offsets, causing the selection of transform sets to be limited to three transform sets. In some implementations, the syntax element indicates which entry from the ranked list of intra-prediction modes to use for the current block. In some implementations, the specific value is 3, 2, or 1. In some implementations, an indicator value of 0 corresponds to the highest-ranked intra-prediction mode, and an indicator value of 2 (or 1) corresponds to the second-highest-ranked intra-prediction mode.

[0137] (A12) In some implementations of any of A9 through A11, the syntax element indicates a first change to the corresponding intra-prediction mode from a predefined change set. For example, an offset represented by a signal can be selected from the predefined set. For example, in the case that the set is {-2, 0, 2}, the offset value represented by the signal restricts the selection of the change set to its neighboring change sets (in the case of an offset equal to -2 or +2), or does not use an offset (in the case of an offset equal to 0). In some implementations, the predefined change set has a step size greater than 1 (e.g., a step size of 2, 3, or 4). Having a step size greater than 1 increases the diversity of intra-prediction mode options, thereby improving encoding and decoding accuracy. In some implementations, the predefined change set is based on encoding and decoding information. For example, the predefined set can be adaptively determined based on encoding and decoding information such as the intra-prediction mode, the transform segmentation mode / depth of the current block, transform coefficient values, the last non-zero position (or EOB), the number of non-zero transform coefficients, and / or the block size of the current block (e.g., block width and / or height)).

[0138] (A13) In some embodiments of any of A1 to A12, the method further includes deriving a transform set index for the current block based on the corresponding intra-prediction mode, wherein the transform set index is used to select a first transform set. In some embodiments, a table mapping transform set IDs to corresponding intra-prediction modes is stored at the decoder unit.

[0139] (A14) In some implementations of any of A1 through A13, a first prediction mode is mapped to a set of two or more corresponding intra-prediction modes, and the corresponding intra-prediction mode is selected based on syntax elements in the video bitstream. As an example, when the prediction block is an inter-prediction block or is predicted using an intra-block replication mode, a decoder-side intra-prediction mode derivation method is applied to derive its intra-prediction mode. Assuming the derived intra-prediction modes are M0 through M4 (e.g., M0 is the most frequent and M4 is the fifth most frequent in a ranked, sorted list of intra-prediction modes), syntax elements (e.g., offsets represented by signals) indicate which modes from M0 to M4 are used.

[0140] (A15) In some implementations of A14, the syntax element indicates the index of a set of two or more corresponding intra-prediction modes. For example, three of the five derived modes (M0 to M4) are determined (corresponding to the number of offsets in a predefined set). As an example, an offset of 0 means using M0 to determine the transform set index, an offset of -2 means using M1 to determine the transform set index, and an offset of 2 means using M2 to determine the transform set index. In some implementations, the number of offsets in the predefined set is the same as the number of derived modes.

[0141] (A16) In some embodiments of any of A1 through A15, the corresponding intra-prediction mode is in the range [-15, 81], including end values. For example, the intra-prediction mode is one of the modes shown in Table 3. As an example, the number of available transform sets is expanded, and intra-modes between [-15, -1] and [67, 81] are mapped to additional transform sets. In some embodiments, the number of additional transform sets is 15, and each additional transform set corresponds to an intra-mode in the range [-15, -1]. For modes in [67, 81], a transpose operation can be applied. In some embodiments, the number of available transform sets is less than 15, and multiple intra-modes are mapped to the same transform set.

[0142] (A17) In some embodiments of A16, each intra-prediction mode is mapped to a corresponding transform set in a plurality of transform sets, wherein at least one transform set in the plurality of transform sets is a transpose of another transform set in the plurality of transform sets. For example, the number of available transform sets is reduced to reduce memory storage (e.g., fewer modes but with transpose flags). In some embodiments, the number of intra-prediction modes used to select transform sets is reduced. For example, when mapping the first prediction mode, the number of directional intra-prediction modes is reduced by half.

[0143] (B1) In another aspect, some implementations include a method for video encoding (e.g., method 650). In some implementations, the method is performed at a computing system (e.g., server system 112) having memory and one or more processors. In some implementations, the method is performed at an encoding / decoding module (e.g., encoding / decoding module 320). The method includes: (i) receiving video data (e.g., a source video sequence) comprising a plurality of blocks, said plurality of blocks including a current block; (ii) identifying a first prediction mode for the current block; (iii) mapping the first prediction mode to a corresponding intra-frame prediction mode; (iv) selecting a first transform set from a plurality of transform sets based on said corresponding intra-frame prediction mode; and (v) encoding the current block using the first transform set. In some implementations, the method further includes signaling the encoded current block in a video bitstream.

[0144] (B2) In some implementations of B1, selecting the first transform set includes applying an adjustment to the corresponding intra-prediction mode, and the method further includes representing the adjustment with a signal in the video bitstream.

[0145] (B3) In some embodiments of B1 or B2, the method further includes any one of A1 to A17 above.

[0146] (C1) In another aspect, some implementations include a method for processing visual media data. In some implementations, the method is performed at a computing system (e.g., server system 112) having memory and one or more processors. In some implementations, the method is performed at a codec module (e.g., codec module 320). The method includes: (i) obtaining a source video sequence comprising multiple frames; and (ii) performing a conversion between the source video sequence and a video bitstream of visual media data according to format rules, wherein the video bitstream comprises multiple blocks, the multiple blocks including a current block; and the format rules specify: (a) to identify a first prediction mode for the current block; (b) to map the first prediction mode to a corresponding intra-frame prediction mode; (c) to select a first transform set from multiple transform sets based on the corresponding intra-frame prediction mode; and (d) to reconstruct the current block using the first transform set.

[0147] In another aspect, some embodiments include a computing system (e.g., server system 112) comprising a control circuitry system (e.g., control circuitry system 302) and a memory (e.g., memory 314) coupled to the control circuitry system, the memory storing one or more sets of instructions configured to be executed by the control circuitry system, the set of instructions including instructions for performing any of the methods described herein (e.g., A1 to A17, B1 to B3, and C1 above). In yet another aspect, some embodiments include a non-transitory computer-readable storage medium storing one or more sets of instructions for execution by the control circuitry system of the computing system, the set of instructions including instructions for performing any of the methods described herein (e.g., A1 to A17, B1 to B3, and C1 above).

[0148] Unless otherwise stated, any syntactic element described herein can be a High-Level Syntax (HLS). As used herein, HLS is represented by signals at a level higher than the block level. For example, HLS can correspond to the sequence level, frame level, slice level, or tile level. As another example, HLS elements can be represented by signals in Video Parameter Set (VPS), Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Adaptation Parameter Set (APS), slice headers, picture headers, tile headers, and / or CTU headers.

[0149] It should be understood that although the terms “first,” “second,” etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the claims. As used in the description of embodiments and appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or,” as used herein, refers to and covers any and all possible combinations of one or more associated listed items. It will also be understood that, when used in this specification, the terms “comprising” and / or “including” specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0150] As used herein, the term "when" can be interpreted, depending on the context, as "if the prerequisite is true," "after the prerequisite is true," "in response to determining that the prerequisite is true," "based on determining that the prerequisite is true," or "in response to detecting that the prerequisite is true." Similarly, the phrases "if [the prerequisite is true]," "if [the prerequisite is true]," or "when [the prerequisite is true]" can be interpreted, depending on the context, as meaning "after determining that the prerequisite is true," "in response to determining that the prerequisite is true," "based on determining that the prerequisite is true," "after detecting that the prerequisite is true," or "in response to detecting that the prerequisite is true." N This refers to a variable number. Unless otherwise specified, N Different instances of can refer to the same number (e.g., the same integer value, such as the number 2) or different numbers.

[0151] For illustrative purposes, the foregoing description has been described with reference to specific embodiments. However, the illustrative discussion above is not intended to be exhaustive or to limit the claims to the precise forms disclosed. In view of the above teachings, many modifications and variations are possible. These embodiments were chosen and described in order to best illustrate the operating principles and practical applications, thereby enabling others skilled in the art to implement them.

Claims

1. A method for video decoding, performed at a computing system having memory and one or more processors, the method comprising: Receive the video bitstream including the current block; Identify a first prediction pattern for the current block; Map the first prediction mode to the corresponding intra-frame prediction mode; A first transform set is selected from multiple transform sets based on the corresponding intra-frame prediction mode; as well as Reconstruct the current block using the first transform set.

2. The method according to claim 1, wherein, The first prediction mode is the inter-frame prediction mode.

3. The method according to claim 1, wherein, The first prediction mode is the intra-block copy (IBC) mode.

4. The method according to claim 1, wherein, The first prediction mode includes the template matching prediction mode.

5. The method according to claim 1, wherein, The first prediction mode includes a combination of inter-frame and intra-frame prediction modes, wherein the corresponding intra-frame prediction mode is based on the intra-frame portion of the combination of inter-frame and intra-frame prediction modes.

6. The method according to claim 1, wherein, The first prediction model includes the geometric partitioning model (GPM).

7. The method according to claim 6, wherein, The corresponding intra-frame prediction mode is based on the segmentation direction of the GPM.

8. The method according to claim 6, wherein, The corresponding intra-prediction mode is based on the intra-prediction portion of the GPM.

9. The method of claim 1, further comprising parsing syntax elements from the video bitstream to determine whether to adjust the corresponding intra-frame prediction, wherein, The first transform set is selected based on the corresponding intra-prediction mode and the syntax elements.

10. The method according to claim 9, wherein, The syntax element indicates an intra-frame mode offset, and the first transform set is selected using an index derived by adding the intra-frame mode offset to the corresponding intra-frame prediction mode.

11. The method according to claim 9, wherein, The syntax element indicates the changes to be made to the corresponding intra-prediction mode, and the number of allowed changes is limited to a specific value.

12. The method according to claim 9, wherein, The syntax element indicates a first change from a predefined set of changes for the corresponding intra-prediction mode.

13. The method of claim 1, further comprising deriving a transform set index for the current block based on the corresponding intra-frame prediction mode, wherein, The first transform set is selected using the transform set index.

14. The method according to claim 1, wherein, The first prediction mode is mapped to a set of two or more corresponding intra-prediction modes, wherein the corresponding intra-prediction mode is selected based on syntax elements in the video bitstream.

15. The method according to claim 14, wherein, The syntax element indicates an index for the set of the two or more corresponding intra-prediction modes.

16. The method according to claim 1, wherein, The corresponding intra-frame prediction mode is in the range of [-15, 81], including end values.

17. The method according to claim 16, wherein, Each intra-frame prediction mode is mapped to a corresponding transform set in the plurality of transform sets, wherein at least one transform set in the plurality of transform sets is a transpose of another transform set in the plurality of transform sets.

18. A video encoding method, performed at a computing system having memory and one or more processors, the method comprising: Receive video data comprising multiple blocks, including the current block; Identify a first prediction pattern for the current block; Map the first prediction mode to the corresponding intra-frame prediction mode; A first transform set is selected from multiple transform sets based on the corresponding intra-frame prediction mode; as well as The current block is encoded using the first transform set.

19. The method according to claim 18, wherein, Selecting the first transform set includes applying an adjustment to the corresponding intra-prediction mode, and wherein the method further includes representing the adjustment with a signal in the video bitstream.

20. A non-transitory computer-readable storage medium storing a video bitstream generated by a video coding method, the video coding method comprising: Receive video data comprising multiple blocks, including the current block; Identify a first prediction pattern for the current block; Map the first prediction mode to the corresponding intra-frame prediction mode; A first transform set is selected from multiple transform sets based on the corresponding intra-frame prediction mode; as well as The current block is encoded using the first transform set.

Images