Adaptive intra secondary transform set selection and signaling

Abstract

Various implementations described herein include methods and systems for coding a video. In one aspect, the method includes receiving a video bitstream including a plurality of blocks, the plurality of blocks including a current block. The method also includes determining an intra prediction mode of the current block, and selecting a set of secondary transforms from a plurality of sets of secondary transforms based on coded information, the coded information including the intra prediction mode of the current block. The method also includes and reconstructing the current block with a secondary transform from the selected set of secondary transforms.

Description

Related applications

[0001] This application is a continuation of U.S. Patent Application No. 19 / 205,943, filed May 12, 2025, which claims priority to U.S. Provisional Patent Application No. 63 / 666,619, filed July 1, 2024, entitled “Adaptive Intra Secondary Transform Set Selection and Signaling,” each of which is incorporated herein by reference in its entirety. Technical Field

[0002] The disclosed implementations generally relate to video encoding and decoding, including but not limited to systems and methods for selecting transform kernels. Background Technology

[0003] Digital video is supported by various electronic devices such as digital televisions, laptop or desktop computers, tablet computers, digital camera devices, digital recording devices, digital media players, video game consoles, smartphones, video conferencing equipment, video streaming devices, etc. Electronic devices send and receive or otherwise transmit digital video data across communication networks, and / or store digital video data on storage devices. Due to the limited bandwidth capacity of communication networks and the limited memory resources of storage devices, video encoding and decoding can be used to compress video data according to one or more video codec standards before transmission or storage. Video encoding and decoding can be performed by hardware and / or software on electronic / client devices or servers providing cloud services.

[0004] Video codecs typically employ prediction methods that leverage the inherent redundancy in video data (e.g., inter-frame prediction, intra-frame prediction, etc.). Video codecs aim to compress video data into a form using a lower bitrate while avoiding or minimizing video quality degradation. Several video codec standards have been developed. For example, High-Efficiency Video Coding (HEVC / H.265) is a video compression standard designed as part of the MPEG-H project. The ITU-T and ISO / IEC released the HEVC / H.265 standard in 2013 (Revision 1), 2014 (Revision 2), 2015 (Revision 3), and 2016 (Revision 4). Versatile Video Coding (VVC / H.266) is a video compression standard designed as a successor to HEVC. The ITU-T and ISO / IEC released the VVC / H.266 standard in 2020 (Revision 1) and 2022 (Revision 2). AOMedia Video 1 (AV1) is an open video codec format designed as an alternative to HEVC. On January 8, 2019, a validated version 1.0.0 of the specification with errata table 1 was released. Summary of the Invention

[0005] Among other things, this disclosure describes systems and methods for video (image) compression, and more specifically relates to the selection of a transform set (e.g., a quadratic transform set). Selecting a subset of the quadratic transform set from all available quadratic transform sets can be based on encoded / decoded information (e.g., information available for both encoding and decoding components), such as the intra-frame prediction mode of the current block. By adaptively selecting a subset of the transform set based on encoded / decoded information, more accurate and / or efficient transform kernels that improve encoding / decoding quality (e.g., more accurate encoding / decoding) can be selected even from a reduced set of the second transform set. In this way, signaling overhead for the transform set can be reduced. For example, a residual block to which the quadratic transform set is applied may have a directionality and / or distribution associated with the encoded / decoded information. Restricting the transform set to a subset of all available transform sets can reduce encoding time and / or signaling overhead by using fewer bits to signal the set index associated with the selected subset of the transform set.

[0006] According to some implementations, a video decoding method includes: (i) receiving a video bitstream comprising a plurality of blocks (e.g., an encoded video sequence), the plurality of blocks including a current block; (ii) determining an intra-frame prediction mode for the current block; (iii) selecting a set of quadratic transforms from a plurality of quadratic transform sets based on encoded information, the encoded information including the intra-frame prediction mode for the current block; and (iv) reconstructing the current block by applying quadratic transforms from the selected set of quadratic transforms.

[0007] According to some implementations, a video coding method includes: (i) receiving video data (e.g., a source video sequence) comprising a plurality of blocks, the plurality of blocks including a current block; (ii) determining an intra-prediction mode for the current block; (iii) selecting a set of quadratic transforms from a plurality of quadratic transform sets based on the intra-prediction mode for the current block; and (iv) encoding the current block by applying a quadratic transform from the selected set of quadratic transforms.

[0008] According to some implementations, a bitstream conversion method includes: (i) obtaining a source video sequence comprising multiple frames; and (ii) performing a conversion between the source video sequence and a video bitstream of visual media data according to format rules. The video bitstream comprises multiple encoded blocks, the multiple encoded blocks including a current block. The format rules specify: (a) determining an intra-frame prediction mode for the current block; selecting a set of quadratic transforms from multiple sets of quadratic transforms based on the intra-frame prediction mode of the current block; and (b) reconstructing the current block by applying quadratic transforms from the selected set of quadratic transforms.

[0009] According to some embodiments, a computing system, such as a streaming system, server system, personal computer system, or other electronic device, is provided. The computing system includes a control circuitry system and a memory storing one or more sets of instructions. The one or more sets of instructions include instructions for performing any of the methods described herein. In some embodiments, the computing system includes encoder components and decoder components (e.g., a transcoder). According to some embodiments, a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium stores one or more sets of instructions for execution by the computing system. The one or more sets of instructions include instructions for performing any of the methods described herein.

[0010] Therefore, methods, apparatus, and systems for encoding and decoding video are disclosed. Such methods, apparatus, and systems may supplement or replace conventional methods, apparatus, and systems for encoding / decoding video. Not all features and advantages described in the specification are necessarily included, and in particular, some additional features and advantages will be apparent to those skilled in the art in light of the accompanying drawings, specification, and claims provided in this disclosure. Furthermore, it should be noted that the language used in the specification has been chosen primarily for readability and instruction purposes and is not necessarily intended to depict or limit the subject matter described herein. Attached Figure Description

[0011] To provide a more detailed understanding of this disclosure, a more specific description can be made by referring to the features of various embodiments, some of which are illustrated in the accompanying drawings. However, the drawings only show relevant features of this disclosure and are therefore not necessarily intended to be limiting, as those skilled in the art will understand upon reading this disclosure that other valid features may be permissible.

[0012] Figure 1 This is a block diagram illustrating an example communication system according to some implementations.

[0013] Figure 2A This is a block diagram illustrating example elements of an encoder component according to some embodiments.

[0014] Figure 2B This is a block diagram illustrating example elements of a decoder component according to some embodiments.

[0015] Figure 3 This is a block diagram illustrating an example server system according to some implementation methods.

[0016] Figure 4 An example transformation process involving a quadratic transformation is shown according to some implementations.

[0017] Figure 5A An example video decoding process according to some implementation methods is shown.

[0018] Figure 5B An example video encoding process according to some implementation methods is shown.

[0019] By convention, the various features shown in the accompanying drawings are not necessarily drawn to scale, and similar reference numerals may be used to indicate similar features throughout the specification and the drawings. Detailed Implementation

[0020] This disclosure describes video / image compression techniques related to selecting a reduced set of transform sets (e.g., a quadratic transform set) based on encoded / decoded information including the intra-prediction mode of the current block. The residual blocks of the current block may have a directionality and / or distribution associated with the encoded / decoded information; therefore, a subset of the available transform sets can be selected to correspond to the directionality and / or distribution. Restricting the selectable transform set to a subset of all available transform sets can include reducing encoding / decoding time and / or signaling overhead. By adaptively selecting a subset of the transform set, for example, based on the corresponding intra-prediction mode, a more accurate and / or efficient transform kernel that improves encoding / decoding quality (e.g., more accurate encoding / decoding) can be selected from the reduced set of transform sets.

[0021] Example systems and devices

[0022] Figure 1 This is a block diagram illustrating a communication system 100 according to some embodiments. The communication system 100 includes a source device 102 and a plurality of electronic devices 120 (e.g., electronic devices 120-1 to 120-m) communicatively coupled to each other via one or more networks. In some embodiments, the communication system 100 is a streaming system, for example, for use with video-enabled applications such as video conferencing applications, digital TV applications, and media storage and / or distribution applications.

[0023] Source device 102 includes a video source 104 (e.g., a camera device component or media storage) and an encoder component 106. In some embodiments, the video source 104 is a digital camera device (e.g., configured to create an uncompressed video sample stream). The encoder component 106 generates one or more encoded video bitstreams from the video stream. The video stream from video source 104 can have a higher data volume compared to the encoded video bitstream 108 generated by encoder component 106. Because the encoded video bitstream 108 has a lower data volume (less data) compared to the video stream from video source 104, it requires less bandwidth to transmit and less storage space to store compared to the video stream from video source 104. In some embodiments, source device 102 does not include encoder component 106 (e.g., configured to transmit uncompressed video to network 110).

[0024] One or more networks 110 represent any number of networks that transmit information between source device 102, server system 112, and / or electronic device 120, including, for example, wired (connected) and / or wireless communication networks. One or more networks 110 may exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and / or the Internet.

[0025] One or more networks 110 include server systems 112 (e.g., distributed / cloud computing systems). In some embodiments, server system 112 is or includes a streaming server (e.g., configured to store and / or distribute video content, such as encoded video streams from source device 102). Server system 112 includes codec components 114 (e.g., configured to encode and / or decode video data). In some embodiments, codec components 114 include encoder components and / or decoder components. In various embodiments, codec components 114 are instantiated as hardware, software, or a combination thereof. In some embodiments, codec components 114 are configured to decode encoded video bitstream 108 and re-encode the video data using different encoding standards and / or methods to generate encoded video data 116. In some embodiments, server system 112 is configured to generate multiple video formats and / or encodings based on encoded video bitstream 108. In some embodiments, server system 112 is used as a Media-Aware Network Element (MANE). For example, server system 112 can be configured to trim the encoded video bitstream 108 to tailor potentially different bitstreams for one or more of the electronic devices 120. In some implementations, MANE is provided separately from server system 112.

[0026] Electronic device 120-1 includes a decoder component 122 and a display 124. In some embodiments, the decoder component 122 is configured to decode encoded video data 116 to generate an outgoing video stream that can be displayed on a display or other type of presentation device. In some embodiments, one or more of the electronic devices 120 do not include a display component (e.g., communicatively coupled to an external display device and / or includes media storage). In some embodiments, electronic device 120 is a streaming client. In some embodiments, electronic device 120 is configured to access server system 112 to obtain encoded video data 116.

[0027] The source device and / or multiple electronic devices 120 are sometimes referred to as “terminal devices” or “user devices”. In some implementations, one or more of the electronic devices 120 and / or the source device 102 are instances of server systems, personal computers, portable devices (e.g., smartphones, tablets, or laptops), wearable devices, video conferencing equipment, and / or other types of electronic devices.

[0028] In an example operation of communication system 100, source device 102 transmits an encoded video bitstream 108 to server system 112. For example, source device 102 may encode and decode a stream of images captured by the source device. Server system 112 receives the encoded video bitstream 108 and may decode and / or encode the encoded video bitstream 108 using codec component 114. For example, server system 112 may apply encoding to video data that is better suited for network transmission and / or storage. Server system 112 may transmit encoded video data 116 (e.g., one or more encoded / decoded video bitstreams) to one or more electronic devices 120. Each electronic device 120 may decode the encoded video data 116 and optionally display video images.

[0029] Figure 2A This is a block diagram illustrating example elements of an encoder component 106 according to some embodiments. The encoder component 106 receives video data (e.g., a source video sequence) from a video source 104. In some embodiments, the encoder component includes a receiver (e.g., transceiver) component configured to receive the source video sequence. In some embodiments, the encoder component 106 receives the video sequence from a remote video source (e.g., a video source that is a component of a device different from the encoder component 106). The video source 104 may provide the source video sequence in the form of a digital video sample stream, which may have any suitable bit depth (e.g., 8-bit, 10-bit, or 12-bit), any color space (e.g., BT.601 YCrCB or RGB), and any suitable sampling structure (e.g., YCrCb 4:2:0 or YCrCb 4:4:4). In some embodiments, the video source 104 is a storage device storing previously captured / prepared video. In some embodiments, the video source 104 is a camera device that captures local image information as a video sequence. Video data can be provided as multiple individual images that are given motion when viewed sequentially. Each image can be organized as a spatial array of pixels, where, depending on the sampling structure, color space, etc., each pixel may include one or more samples. The relationship between pixels and samples will be readily understood by those skilled in the art.

[0030] Encoder component 106 is configured to encode, decode, and / or compress images of a source video sequence into a encoded video sequence 216 in real time or under other time constraints required by the application. In some embodiments, encoder component 106 is configured to perform a conversion between the source video sequence and a visual media data bitstream (e.g., a video bitstream). Implementing an appropriate encoding / decoding speed is a function of controller 204. In some embodiments, controller 204 controls and is functionally coupled to other functional units as described below. Parameters set by controller 204 may include rate control-related parameters (e.g., image skipping, quantizer, and / or the λ value of rate-distortion optimization techniques), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. Other functions of controller 204 can be readily identified by those skilled in the art, as these functions may belong to encoder component 106 optimized for a particular system design.

[0031] In some implementations, encoder component 106 is configured to operate within a codec loop. In a simplified example, the codec loop includes a source codec 202 (e.g., responsible for creating symbols, such as a symbol stream, based on the input image to be encoded and a reference image) and a (local) decoder 210. Decoder 210 reconstructs the symbols in a manner similar to that of the (remote) decoder to create sample data (when compression between the symbols and the encoded video bitstream is lossless). The reconstructed sample stream (sample data) is input to a reference image memory 208. Since decoding of the symbol stream produces bit-accurate results regardless of the decoder's location (local or remote), the contents of the reference image memory 208 are also bit-accurate between the local and remote encoders. In this way, the encoder's prediction portion interprets the same sample values ​​as the sample values ​​that the decoder interprets during decoding using the prediction as reference image samples.

[0032] The operation of decoder 210 can be combined with a remote decoder, for example, as shown below. Figure 2B The operation of the decoder component 122 is the same as described in the detailed description. However, a brief reference is provided. Figure 2B Since symbols are available and the encoding / decoding of symbols into a encoded / decoded video sequence by entropy codec 214 and parser 254 can be lossless, the entropy decoding portion of decoder component 122, including buffer memory 252 and parser 254, does not need to be fully implemented in local decoder 210.

[0033] Besides parsing / entropy decoding, the decoder techniques described in this paper can exist in the corresponding encoders in essentially the same form. For this reason, the subject matter focuses on decoder operations. Furthermore, the description of encoder techniques can be simplified, as encoder techniques are inverses of decoder techniques.

[0034] As part of the operation of source codec 202, source codec 202 can perform motion-compensated predictive coding, which predictively codes the input frame with reference to one or more previously encoded frames from the video sequence designated as reference frames. In this manner, codec engine 212 codes the differences between pixel blocks of the input frame and pixel blocks of the reference frame, which can be selected as the prediction reference for the input frame. Controller 204 can manage the coding and decoding operations of source codec 202, including, for example, setting parameters and subgroup parameters for encoding video data.

[0035] Decoder 210 decodes encoded video data based on symbols created by source codec 202, frames of which can be designated as reference frames. The operation of codec engine 212 can advantageously support lossy processing. When encoded video data is processed by video decoder (… Figure 2A When decoded at (not shown), the reconstructed video sequence can be a copy of the source video sequence with some errors. Decoder 210 can replicate the decoding process performed on the reference frame by a remote video decoder, and the reconstructed reference frame can be stored in reference image memory 208. In this way, encoder component 106 locally stores a copy of the reconstructed reference frame that shares common content (no transmission errors) with the reconstructed reference frame to be obtained by the remote video decoder.

[0036] Predictor 206 can perform a prediction search against codec engine 212. That is, for a new frame to be encoded or decoded, predictor 206 can search the reference image memory 208 for sample data (as candidate reference pixel blocks) or specific metadata such as reference image motion vectors, block shapes, etc., that can be used as appropriate prediction references for the new image. Predictor 206 can operate pixel-by-pixel based on the sample blocks to find appropriate prediction references. As determined by the search results obtained by predictor 206, the input image can have prediction references obtained from multiple reference images stored in reference image memory 208.

[0037] The outputs of all the aforementioned functional units can undergo entropy encoding / decoding in entropy codec 214. Entropy codec 214 converts the symbols generated by the various functional units into encoded / decoded video sequences by lossless compression of the symbols according to techniques known to those skilled in the art (e.g., Huffman coding, variable-length coding, and / or arithmetic coding).

[0038] In some embodiments, the output of entropy codec 214 is coupled to a transmitter. The transmitter can be configured to buffer the encoded video sequence created by entropy codec 214 in preparation for transmission via communication channel 218, which can be a hardware / software link to a storage device storing the encoded video data. The transmitter can be configured to combine encoded video data from source codec 202 with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown). In some embodiments, the transmitter can transmit additional data along with the encoded video. Source codec 202 can include such data as part of the encoded video sequence. Additional data can include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant images and slices, Supplementary Enhancement Information (SEI) messages, Visual Usability Information (VUI) parameter set fragments, etc.

[0039] Controller 204 can manage the operation of encoder component 106. During encoding and decoding, controller 204 can assign a specific encoding / decoding picture type to each encoded picture, which may affect the encoding / decoding technique applied to the corresponding picture. For example, pictures can be assigned as intra-frame pictures (I-pictures), prediction pictures (P-pictures), or bidirectional prediction pictures (B-pictures). Intra-frame pictures can be encoded / decoded without using any other frames in the sequence as prediction sources. Some video codecs allow different types of intra-frame pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are familiar with those variations of I-pictures and their corresponding applications and characteristics, and therefore they will not be repeated here. Predictive pictures can be encoded / decoded using inter-frame prediction or intra-frame prediction, which uses at most one motion vector and reference index to predict sample values ​​for each block. Bidirectional prediction pictures can be encoded / decoded using inter-frame prediction or intra-frame prediction, which uses at most two motion vectors and reference indexes to predict sample values ​​for each block. Similarly, multiple prediction images can use more than two reference images and associated metadata to reconstruct a single block.

[0040] Source images can typically be spatially subdivided into multiple sample blocks (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples), and encoded / decoded block by block. Predictive encoding / decoding of these blocks can be performed with reference to other (already encoded / decoded) blocks, determined by the encoding / decoding assignments of the corresponding images applied to the blocks. For example, blocks of an I-image can be nonpredictively encoded / decoded, or blocks of an I-image can be predictively encoded / decoded (spatial prediction or intra-frame prediction) with reference to already encoded / decoded blocks of the same image. Pixel blocks of a P-image can be nonpredictively encoded / decoded with reference to a previously encoded / decoded reference image via spatial prediction or temporal prediction. Blocks of a B-image can be nonpredictively encoded / decoded with reference to one or two previously encoded / decoded reference images via spatial prediction or temporal prediction.

[0041] Video can be captured as multiple source images (video images) in a time-series manner. Intra-frame image prediction (often simply called intra-prediction) utilizes spatial correlations within a given image, while inter-frame image prediction utilizes (temporal or other) correlations between images. In the example, a specific image in the encoding / decoding process (referred to as the current image) is segmented into blocks. If a block in the current image is similar to a reference block in a previously encoded and decoded reference image in the video that is still buffered, the block in the current image can be encoded and decoded using a vector called a motion vector. The motion vector points to the reference block in the reference image, and when using multiple reference images, the motion vector can have a third dimension that identifies the reference images.

[0042] Encoder component 106 can perform encoding and decoding operations according to any predetermined video codec technique or standard described herein. In operation, encoder component 106 can perform various compression operations, including predictive codec operations utilizing temporal and spatial redundancy in the input video sequence. Therefore, the encoded and decoded video data can conform to the syntax specified by the video codec technique or standard used.

[0043] Figure 2B This is a block diagram illustrating example elements of a decoder component 122 according to some embodiments. Figure 2B The decoder component 122 is coupled to channel 218 and display 124. In some embodiments, the decoder component 122 includes a transmitter coupled to loop filter 256 and configured to transmit data to display 124 (e.g., via a wired or wireless connection).

[0044] In some embodiments, decoder component 122 includes a receiver coupled to channel 218 and configured to receive data from channel 218 (e.g., via a wired or wireless connection). The receiver may be configured to receive one or more encoded video sequences to be decoded by decoder component 122. In some embodiments, the decoding of each encoded video sequence is independent of other encoded video sequences. Each encoded video sequence can be received from channel 218, which may be a hardware / software link to a storage device storing the encoded video data. The receiver may receive encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams, which may be forwarded to their respective user entities (not depicted). The receiver may separate the encoded video sequences from other data. In some embodiments, the receiver receives additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. Decoder component 122 may use the additional data to decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or SNR enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0045] According to some embodiments, the decoder component 122 includes a buffer memory 252, a parser 254 (sometimes also referred to as an entropy decoder), a scaler / inverse transform unit 258, an intra-frame image prediction unit 262, a motion compensation prediction unit 260, an aggregator 268, a loop filter unit 256, a reference image memory 266, and a current image memory 264. In some embodiments, the decoder component 122 is implemented as an integrated circuit, a series of integrated circuits, and / or other electronic circuit systems. The decoder component 122 may be implemented at least partially in software.

[0046] Buffer memory 252 is coupled between channel 218 and parser 254 (e.g., to combat network jitter). In some embodiments, buffer memory 252 is separate from decoder component 122. In some embodiments, a separate buffer memory is provided between the output of channel 218 and decoder component 122. In some embodiments, a separate buffer memory is provided outside decoder component 122 (e.g., to combat network jitter) in addition to buffer memory 252 inside decoder component 122 (e.g., buffer memory 252 is configured to handle playback timing). Buffer memory 252 may not be necessary or may be small when receiving data from a store / forward device with sufficient bandwidth and controllability or from an isochronous synchronization network. Buffer memory 252 may be required to make the best use of packet networks such as the Internet; buffer memory 252 may be relatively large and / or have an adaptive size and may be implemented at least partially in an operating system or similar component outside decoder component 122.

[0047] Parser 254 is configured to reconstruct symbols 270 from the encoded video sequence. Symbols may include, for example, information for managing the operation of decoder component 122, and / or information for controlling a presentation device such as display 124. Control information for the presentation device may be in the form of, for example, Supplementary Enhancement Information (SEI) messages or fragments of Video Usability Information (VUI) parameter sets (not depicted). Parser 254 parses (entropy decodes) the encoded video sequence. The encoding and decoding of the encoded video sequence may be performed according to video codec techniques or standards, and may follow principles known to those skilled in the art, including variable-length codecs, Huffman codecs, arithmetic codecs with or without context sensitivity, etc. Parser 254 may extract a subgroup parameter set from the encoded video sequence for at least one subgroup of pixels in the video decoder based on at least one parameter corresponding to a group. Subgroups can include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The parser 254 can also extract information such as transform coefficients, quantizer parameter values, and motion vectors from the encoded / decoded video sequence.

[0048] Depending on the type of the encoded video picture or a portion of the encoded video picture (e.g., inter-frame and intra-frame pictures, inter-frame and intra-frame blocks) and other factors, the reconstruction of symbol 270 may involve multiple different units. Which units are involved and how these units are involved can be controlled by subgroup control information parsed from the encoded video sequence by parser 254. For simplicity, such subgroup control information flow between parser 254 and the following multiple units is not depicted.

[0049] The decoder component 122 can be conceptually subdivided into multiple functional units, and in some implementations, these units interact closely with each other and can be at least partially integrated with each other. However, for the sake of brevity, this paper retains the conceptual subdivision of the functional units.

[0050] The scaler / inverse transform unit 258 receives quantization transform coefficients and control information (e.g., which transform to use, block size, quantization factor, and / or quantization scaling matrix) from the parser 254 as symbol 270. The scaler / inverse transform unit 258 can output a block including sample values, which can be input to the aggregator 268.

[0051] In some cases, the output samples of the scaler / inverse transform unit 258 belong to intra-frame codec blocks; that is, blocks that do not use prediction information from previously reconstructed images, but can use prediction information from previously reconstructed portions of the current image. Such prediction information can be provided by the intra-frame image prediction unit 262. The intra-frame image prediction unit 262 can use surrounding already reconstructed information obtained from the current (partially reconstructed) image from the current image memory 264 to generate blocks of the same size and shape as the blocks in the reconstruction. The aggregator 268 can add the prediction information already generated by the intra-frame image prediction unit 262 to the output sample information provided by the scaler / inverse transform unit 258 based on each sample.

[0052] In other cases, the output samples of the scaler / inverse transform unit 258 belong to the inter-frame encoding and decoding and potentially to the motion-compensated block. In such cases, the motion-compensated prediction unit 260 can access the reference image memory 266 to obtain samples for prediction. After motion compensation of the obtained samples according to the symbol 270 belonging to the block, these samples can be added by the aggregator 268 to the output of the scaler / inverse transform unit 258 (referred to in this case as residual samples or residual signals) to generate output sample information. The address in the reference image memory 266 from which the motion-compensated prediction unit 260 obtains the predicted samples can be controlled by motion vectors. Motion vectors can be used by the motion-compensated prediction unit 260 in the form of symbol 270, which can have, for example, an X component, a Y component, and a reference image component. Motion compensation can also include, for example, interpolation of sample values ​​obtained from the reference image memory 266 when using subsampled precise motion vectors, and motion vector prediction mechanisms.

[0053] The output samples of aggregator 268 can undergo various loop filtering techniques in loop filter unit 256. Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video bitstream and available to loop filter unit 256 as symbols 270 from parser 254. However, video compression techniques may also respond to metadata obtained during decoding of previous (in decoding order) portions of the encoded picture or encoded video sequence, and to sample values ​​obtained from previous reconstruction and loop filtering. The output of loop filter unit 256 may be a sample stream that can be output to a presentation device such as display 124 and stored in reference picture memory 266 for future inter-frame picture prediction.

[0054] Once a certain codec image is reconstructed, it can be used as a reference image for future predictions. Once a codec image is reconstructed and has been identified as a reference image (e.g., by parser 254), the current reference image can become part of the reference image memory 266, and a new current image memory can be reallocated before reconstructing subsequent codec images begins.

[0055] Decoder component 122 can perform decoding operations according to a predetermined video compression technique that may be recorded in a standard such as any standard described herein. As specified in a video compression technique document or standard, and particularly in a configuration file therein, the encoded / decoded video sequence may conform to the syntax specified by the video compression technique or standard used, in the sense that the encoded / decoded video sequence follows the syntax of the video compression technique or standard. Furthermore, the complexity of the encoded / decoded video sequence may be within a range defined by a level of the video compression technique or standard in order to conform to some video compression techniques or standards. In some cases, the level limits the maximum image size, maximum frame rate, maximum reconstruction sample rate (measured, for example, megasamples per second), maximum reference image size, etc. In some cases, the limitations set by the level can be further restricted by the Hypothetical Reference Decoder (HRD) specification and metadata managed by the HRD buffer used for signaling in the encoded / decoded video sequence.

[0056] Figure 3 This is a block diagram illustrating a server system 112 according to some embodiments. The server system 112 includes a control circuitry system 302, one or more network interfaces 304, a memory 314, a user interface 306, and one or more communication buses 312 for interconnecting these components. In some embodiments, the control circuitry system 302 includes one or more processors (e.g., a CPU, GPU, and / or DPU). In some embodiments, the control circuitry system includes a field-programmable gate array, a hardware accelerator, and / or an integrated circuit (e.g., an application-specific integrated circuit).

[0057] Network interface 304 can be configured to interface with one or more communication networks (e.g., wireless networks, wired networks, and / or optical networks). Communication networks can be local, wide area, metropolitan area, vehicular and industrial, real-time, latency-tolerant, etc. Examples of communication networks include: local area networks such as Ethernet; wireless LANs (Local Area Networks); cellular networks including GSM (Global System for Mobile Communications), 3G (the Third Generation), 4G (the Fourth Generation), 5G (the Fifth Generation), LTE (Long Term Evolution), etc.; TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV; vehicle and industrial networks including CANBus (Controller Area Network Bus), etc. Such communication can be one-way receiving (e.g., broadcast TV), one-way transmitting (e.g., to a CANbus of certain CANbus devices), or bidirectional (e.g., to other computer systems using local area digital networks or wide area digital networks). Such communication may include communication to one or more cloud computing networks.

[0058] User interface 306 includes one or more output devices 308 and / or one or more input devices 310. Input devices 310 may include one or more of the following: keyboard, mouse, touchpad, touchscreen, data glove, joystick, microphone, scanner, camera, etc. Output devices 308 may include one or more of the following: audio output devices (e.g., speakers), visual output devices (e.g., displays or monitors), etc.

[0059] Memory 314 may include high-speed random access memory (e.g., DRAM, SRAM, DDR RAM, and / or other random access solid-state memory devices) and / or non-volatile memory (e.g., one or more disk storage devices, optical disk storage devices, flash memory devices, and / or other non-volatile solid-state memory devices). Memory 314 may optionally include one or more storage devices remote from the control circuitry system 302. Memory 314, or alternatively, the non-volatile solid-state memory devices within memory 314 include non-transitory computer-readable storage media. In some embodiments, memory 314 or the non-transitory computer-readable storage media of memory 314 stores programs, modules, instructions, and data structures, or subsets or supersets thereof:

[0060] • Operating system 316, which includes procedures for handling various basic system services and for performing hardware-related tasks;

[0061] • A network communication module 318 for connecting the server system 112 to other computing devices via one or more network interfaces 304 (e.g., via wired and / or wireless connections);

[0062] • Codec module 320, which performs various functions related to encoding and / or decoding data such as video data. In some embodiments, codec module 320 is an instance of codec component 114. Codec module 320 includes, but is not limited to, one or more of the following:

[0063] Decoding module 322, which performs various functions related to decoding encoded data, such as those previously described with respect to decoder component 122; and

[0064] Encoding module 340, which performs various functions related to encoding data, such as those previously described with respect to encoder component 106; and

[0065] • For example, an image memory 352 for storing images and image data, used in conjunction with the encoding / decoding module 320. In some embodiments, the image memory 352 includes one or more of the following: a reference image memory 208, a buffer memory 252, a current image memory 264, and a reference image memory 266.

[0066] In some implementations, the decoding module 322 includes a parsing module 324 (e.g., configured to perform various functions previously described with respect to the parser 254), a transformation module 326 (e.g., configured to perform various functions previously described with respect to the scaler / inverse transformation unit 258), a prediction module 328 (e.g., configured to perform various functions previously described with respect to the motion compensation prediction unit 260 and / or the intra-frame picture prediction unit 262), and a filter module 330 (e.g., configured to perform various functions previously described with respect to the loop filter 256).

[0067] In some embodiments, the encoding module 340 includes a code module 342 (e.g., configured to perform various functions previously described with respect to source codec 202 and / or codec engine 212) and a prediction module 344 (e.g., configured to perform various functions previously described with respect to predictor 206). In some embodiments, the decoding module 322 and / or the encoding module 340 includes... Figure 3 A subset of the modules shown. For example, a shared prediction module is used by both the decoding module 322 and the encoding module 340.

[0068] Each of the modules identified above and stored in memory 314 corresponds to a set of instructions for performing the functions described herein. The modules identified above (e.g., instruction sets) do not need to be implemented as separate software programs, processes, or modules, and therefore various subsets of these modules can be combined or otherwise rearranged in various implementations. For example, the codec module 320 may optionally not include separate decoding and encoding modules, but instead use the same set of modules to perform both sets of functions. In some implementations, memory 314 stores a subset of the modules and data structures identified above. In some implementations, memory 314 stores additional modules and data structures not described above.

[0069] although Figure 3 A server system 112 according to some embodiments is shown, but Figure 3 This is intended more as a functional description of various features that can exist in one or more server systems than as a structural diagram of the implementation described herein. In practice, items shown individually may be combined and some items may be separated. For example, Figure 3 Some items shown individually can be implemented on a single server, and a single item can be implemented by one or more servers. The actual number of servers used to implement server system 112 and how features are distributed among them will vary depending on the implementation and, optionally, in part, depend on the amount of data traffic processed by the server system during peak usage periods and during average usage periods.

[0070] Example encoding and decoding techniques

[0071] The encoding and decoding processes and techniques described below can be performed at the aforementioned devices and systems (e.g., source device 102, server system 112, and / or electronic device 120). As described above, intra-frame prediction and / or inter-frame prediction are performed on the current block to generate a prediction block. A residual block, including a residual set, is generated by subtracting the prediction block from the current block.

[0072] Transform coding can be applied to residual blocks to remove potential spatial correlations. The transform can refer to a primary transform (e.g., Multiple Transform Selection (MTS) or Non-Separable Primary Transform (NSPT)) or a secondary transform (e.g., Non-Separable Secondary Transform (NSST) or Low Frequency Non-Separable Transform (LFNST)). In some implementations, compressing video frames using intra-frame prediction involves applying a primary transform to the residual blocks. Subsequently, one or more secondary transform kernels from an IntraSecondary Transform (IST) set are applied to the coefficients obtained as the output of the primary transform to reduce redundancy. As an example, there can be seven sets of secondary transform kernels, and each set can contain three secondary transform kernels. In some implementations, after selecting an IST kernel, the IST set index and kernel index are entropy-encoded into the bitstream. Otherwise, if the kernel index is 0, no secondary transform is applied.

[0073] The principal transform can belong to a family of sinusoidal transforms (DCT, DST, inverted versions of DCT, and ADST). DCT can refer to any transform using a transform kernel derived from a discrete cosine transform basis (e.g., DCT type 2), and DST / ADST can refer to any transform using a transform kernel derived from a discrete sine transform basis (e.g., DST type 4 or 7). Example principal transforms can belong to a family of generalized line graph transforms (LGTs), or they can be based on trained kernels. Example sets of quadratic transforms can be groups of one or more quadratic transform kernels. A unique or common set of quadratic transforms can be defined for each principal transform type and / or intra- or inter-frame mode type. Example blocks can be transform blocks.

[0074] The End Of Block (EOB) value corresponds to the position of the last valid (e.g., non-zero) coefficient in the encoded / decoded block following a given coefficient scan order. For a given coefficient scan order, all coefficients after the EOB are zero. In some implementations, if an M×N indivisible quadratic transform kernel is applied to the encoded / decoded block, the EOB value ≤ M.

[0075] Figure 4The use of IST during encoding and decoding is illustrated. In some implementations, the current block comprises a set of samples (e.g., pixel blocks), while the prediction block comprises a set of predictions corresponding to the sample set. In some implementations, the prediction block is subtracted from the current block to generate a residual block comprising a set of residuals. For example, the difference between each sample and its corresponding prediction is calculated. Figure 4 The diagram illustrates the application of the main transform 402 to a residual block (e.g., corresponding to an intra-prediction block). A second transform 404 is applied to the output of the main transform 402. The second transform is an additional transform process following the main transform. For example, in NSST, an inseparable second transform is applied to lower-frequency coefficients to reduce the computational complexity of the inseparable transform. Quantization 406 is applied to the output of the second transform 404, and the resulting quantized coefficients are entropy-encoded 408 and signaled via the video bitstream. The video bitstream is parsed 410 (e.g., at the decoder), and the quantized coefficients are dequantized 412. An inverse second transform 414 is applied to the dequantized data, and an inverse main transform 416 is applied to the output of the second transform 414. In this way, the reconstructed residual block is generated.

[0076] It is worth noting that the transformations performed during the decoding of the video bitstream can be the inverse of the transformations performed during the encoding of the video bitstream, and are sometimes referred to as "inverse transformations". For simplicity, all transformations described in this article can be referred to as "transformations", regardless of whether they are performed during encoding or decoding.

[0077] In some implementations, the residual block undergoes one or more transforms and quantizations to generate a set of residual coefficients. This set of residual coefficients can be transferred from the encoder unit to the decoder unit. The set of residual coefficients undergoes inverse quantization and inverse transform to generate a reconstructed residual block. The reconstructed residual block is combined with a prediction block (e.g., adding the reconstructed residuals of the reconstructed residual block to the prediction of the prediction block) to generate a reconstructed block corresponding to the current block.

[0078] In some implementations, previously decoded information is used to select a subset of all available IST sets for the current block. In some implementations, the previously decoded information includes one or more of the following: intra-frame or inter-frame prediction mode, main transform type, EOB value, block size, block aspect ratio, etc.

[0079] Using previously decoded information allows for the adaptive selection of the quadratic transform set for the residual block of the current block. By using a subset of all available quadratic transform sets, fewer bits are needed to signal the selected quadratic transform set from the subset of quadratic transform sets.

[0080] In some implementations, the decoded set index is used as an offset to obtain the actual set index from a selected subset of all available IST sets, and is optionally implemented as a lookup table.

[0081] In some implementations, the size of the subset selected from all available IST sets is fixed. In some implementations, if N IST sets are available, only M (M<=N) IST sets can be selected and signaled for each intra-frame mode, where M is less than or equal to N. By using only a subset of all available IST sets, fewer bits are needed to signal the index associated with the selected IST set. As an example, if 7 IST sets are available, only 4 sets can be selected and signaled for each intra-frame mode. Compared to all 7 IST sets, only at most 2 bits are needed to represent the index when there are 4 possible IST sets in the subset. In some implementations, the index may correspond to a different IST set for each intra-frame mode.

[0082] In some implementations, the size of the IST set depends on the intra-prediction mode. In other implementations, the size of the IST set depends on whether the increment angle for intra-directional prediction is 0. For intra-directional prediction with an increment angle of 0, the intra-directional direction corresponds to the main direction. As an example, in some implementations, the size of the IST set can be smaller for intra-directional prediction modes with non-zero increment angles compared to intra-directional prediction modes with an increment angle of 0. For example, based on the content associated with the current block, an intra-prediction mode with a non-zero increment angle can provide higher encoding / decoding accuracy than an intra-prediction mode with an increment angle of 0, so increasing the size of the IST set may not result in a significant increase in encoding / decoding accuracy. As a trade-off between encoding / decoding accuracy and higher signaling overhead, the size of the IST set for intra-directional prediction modes with non-zero increment angles can be smaller. Different directional intra-prediction modes can have different transform kernel preferences. By providing the ability to select different transform kernels for different directional intra-prediction modes, the characteristics of the current block can be represented more accurately, which can improve encoding / decoding efficiency.

[0083] In some implementations, the size of the IST set depends on whether the intra-prediction mode is directional. For example, for non-directional intra-prediction modes such as DC, SMOOTH, or PLANA, a larger IST set is allowed. In some implementations, for non-directional intra-prediction modes, increasing the size of the IST set can lead to increased encoding / decoding accuracy by finding more suitable quadratic transforms within a larger IST set. For example, when using a non-directional intra-prediction mode, the distribution of residual blocks can be more aligned with the IST kernels in a larger IST set, which can provide more efficient basis functions for the residual blocks and be more aligned with the statistics of the residual blocks. Therefore, due to the diverse nature of image or video content, encoding / decoding efficiency can be improved by using more diverse transform kernels for different prediction residuals.

[0084] In some implementations, the size of the IST set depends on the type of the primary transform. In some implementations, the size of the IST set depends on whether a DCT is chosen as either direction of the primary transform. For example, the size of the IST set is larger when the primary transform type is DCT-DCT. For example, increasing the size of the IST set can result in increased encoding / decoding accuracy of the output from the DCT-DCT primary transform. As a trade-off between encoding / decoding accuracy and higher signaling overhead, the size of the IST set may be larger when the primary transform type is DCT-DCT.

[0085] In some implementations, the size of the IST set differs for different EOB values. In some implementations, the IST set index is signaled for a block only if EOB ≥ M. Otherwise, the IST is not applied. For example, possible values ​​for M include 1, 2, ..., 64, or other values. In some implementations, EOB values ​​may correspond to different frequency transform coefficient regions, where additional quadratic transforms may become applicable. In some implementations, EOB groups may be defined, and these groups are used as indicators for determining the size of the IST set. For example, EOB values ​​in the ranges of 1 to 8, 9 to 16, 17 to 32, and 33 to 64 may be mapped to EOB groups 0, 1, 2, and 3. In some implementations, the size of the IST set increases with the EOB value.

[0086] In some implementations, the size of the IST set can depend on the block size of the current block. In some implementations, the size of the IST set is smaller when the block width or block height is less than or equal to a threshold T1. In some implementations, T1 is 4.

[0087] In some implementations, the context used to signal the index of the IST set depends on the block size, transform type, or EOB value. In some implementations, the entropy encoding / decoding context for the IST set index is derived based on the EOB group.

[0088] Figure 5A This is a flowchart illustrating a method 500 for decoding video according to some embodiments. Method 500 can be executed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having a control circuitry and a memory storing instructions for execution by the control circuitry. In some embodiments, method 500 is executed by executing instructions stored in the memory of the computing system (e.g., memory 314).

[0089] The system receives (502) a video bitstream comprising multiple blocks, including the current block. The system determines (504) the intra-prediction mode of the current block. The system selects (506) a set of quadratic transforms from multiple sets of encode-decoded information, which includes the intra-prediction mode of the current block. The system reconstructs (508) the current block by applying quadratic transforms from the selected set of quadratic transforms (e.g., as inverses of the quadratic transforms applied during encoding). In this way, a subset of all available IST sets for the current block can be selected using previously decoded information. For example, when the current block is encode-decoded using a first intra-prediction mode, a subset of all available quadratic transform sets can be used for selection (e.g., as described above regarding...). Figure 4 (as described).

[0090] In some implementations, the previously decoded information may include intra-frame or inter-frame prediction mode, main transform type, EOB value, block size, block aspect ratio, etc.

[0091] In some implementations, the decoded set index is used as an offset to obtain the actual set index from a selected subset of all available IST sets, which can be implemented as a lookup table. In some implementations, the size of the subset of all available IST sets is fixed.

[0092] In some implementations, if N IST sets are available, only M (M <= N) IST sets are selected and signaled for each intra-frame mode. Therefore, fewer bits are required to signal the index. In some implementations, if 7 IST sets are available, only 4 sets can be selected and signaled for each intra-frame mode, requiring at most 2 bits to represent the index. For each intra-frame mode, the index can correspond to a different IST set.

[0093] In some implementations, the size of the IST set may depend on the intra-prediction mode. In some implementations, the size of the IST set depends on whether the increment angle for intra-directional prediction is 0. In some implementations, the size of the IST set is smaller for directional intra-prediction modes with non-zero increment angles.

[0094] In some implementations, the size of the IST set depends on whether the intra-prediction mode is a directional intra-prediction mode. In some implementations, a larger IST set is allowed if the intra-prediction mode is a non-directional mode such as DC, SMOOTH, or PLANA.

[0095] In some implementations, the size of the IST set may depend on the type of the primary transform. In some implementations, the size of the IST set depends on whether a DCT is chosen for either direction of the primary transform. In some implementations, the size of the IST set is larger when the primary transform type is DCT-DCT. In some implementations, the size of the IST set may differ for different end-of-block (EOB) values. In some implementations, the IST set index is signaled for a block only when EOB ≥ M. In some implementations, possible values ​​for M include 1, 2, ..., 64, or other values. In some implementations, EOB groups may be defined, and these groups are used as indicators for determining the size of the IST set. For example, EOB values ​​in the ranges of 1 to 8, 9 to 16, 17 to 32, and 33 to 64 may be mapped to EOB groups 0, 1, 2, and 3.

[0096] In some implementations, the size of the IST set can depend on the block size of the current block. In some implementations, the size of the IST set is smaller when the block width or block height is less than or equal to a threshold T1. For example, T1 can be set to 4.

[0097] In some implementations, the context used to signal the index of the IST set depends on the block size, transform type, or EOB value. In some implementations, EOB groups are used as the context for signaling the index of the IST set.

[0098] Figure 5B This is a flowchart illustrating a method 550 for encoding video according to some embodiments. Method 550 can be executed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having a control circuitry and a memory storing instructions for execution by the control circuitry. In some embodiments, method 550 is executed by executing instructions stored in the memory of the computing system (e.g., memory 314). In some embodiments, method 550 is executed by the same system as method 500 described above.

[0099] The system receives (552) video data comprising multiple blocks, including the current block. The system determines (554) the intra-prediction mode of the current block. The system selects (556) a set of quadratic transforms from multiple sets of quadratic transforms based on the intra-prediction mode of the current block. The system applies quadratic transforms from the selected set of quadratic transforms to encode the current block (558). As previously described, the encoding process can be mirrored with the decoding process described herein (e.g., the transform selection implementation described above). For the sake of brevity, these details will not be repeated here.

[0100] although Figure 5A and Figure 5B Multiple logical stages are shown in a specific order, but the stages, which are independent of the order, can be reordered, and other stages can be combined or split. Some reorderings or other groupings not specifically mentioned will be obvious to those skilled in the art, and therefore the orderings and groupings presented herein are not exhaustive. Furthermore, it should be recognized that the stages can be implemented in hardware, firmware, software, or any combination thereof.

[0101] Now let's turn to some example implementations.

[0102] Simulation data for AVM v8 anchors for both intra-frame and random access configurations have shown that simplifying the AVM IST by reducing the number of IST sets to four provides the following results for experiments excluding screen content sequence category B2: a 13% coding time saving and only 0.12% codec loss for intra-frame mode; a 4% coding time saving and only 0.03% codec loss for random access mode; and a 4% coding time saving and 0.06% codec loss for low-latency mode. For experiments including only category B2, the results show: a 5% coding time saving and 0.09% codec gain for intra-frame mode; a 3% coding time saving and only 0.20% codec gain for random access mode; and a 2% coding time saving and 0.03% codec loss for low-latency mode.

[0103] (A1) In one aspect, some implementations include a video decoding method (e.g., method 500). In some implementations, the method is performed at a computing system (e.g., server system 112) having memory and control circuitry. In some implementations, the method is performed at a source codec unit (e.g., source codec 202), a codec engine (e.g., codec engine 212), and / or an entropy codec (e.g., entropy codec 214). The method includes: (i) receiving a video bitstream (e.g., a codec video sequence) comprising a plurality of blocks, said plurality of blocks including a current block; (ii) determining an intra-prediction mode for the current block; (iii) selecting a set of secondary transforms from a plurality of secondary transform sets based on codec information, said codec information including the intra-prediction mode for the current block; and (iv) reconstructing the current block by applying secondary transforms from the selected set of secondary transforms. In some implementations, the set of secondary transforms is selected based on previously decoded information at the decoding unit. For example, previously decoded information may be used to select a subset of all available IST sets for the current block. In some implementations, the multiple quadratic transform sets are multiple intra-frame quadratic transform (IST) sets. In some implementations, the primary transform set is selected from multiple primary transform sets based on encoded / decoded information. For example, the primary transform set (or indivisible transform set) can be selected in the same manner as the quadratic transform sets described herein. As an example, the multiple quadratic transform sets may include seven sets.

[0104] (A2) In some implementations of A1, selecting a set of quadratic transforms from multiple sets of quadratic transforms includes: (i) determining a quadratic transform index based on encoded / decoded information; and (ii) using the quadratic transform index and a lookup table to determine the set of quadratic transforms from the multiple sets of quadratic transforms. For example, the decoded set index can be used as an offset to obtain the actual set index from a selected subset of all available IST sets, which can be implemented as a lookup table.

[0105] (A3) In some implementations of A1 or A2, each of the multiple quadratic transform sets has a fixed number of transforms. For example, the size of a subset of all available IST sets can be fixed. As an example, each transform set may include 3 cores.

[0106] (A4) In some implementations of A3, each of the multiple quadratic transform sets has the same number of transforms. For example, for each intra-frame mode, only M IST sets (M<=N) from N available IST sets can be selected and signaled.

[0107] (A5) In some implementations of any of A1 through A4, there are fewer than all available quadratic transform sets. For example, if seven IST sets are available, only four sets can be selected and signaled for each intra-frame mode, requiring at most two bits to represent the index. For each intra-frame mode, the index can correspond to a different IST set. For example, the same index value can correspond to different transform sets for blocks with different prediction modes.

[0108] (A6) In some embodiments of any of A1 to A5, the encoded / decoded information further includes at least one of the following: the inter-frame prediction mode of the current block, the main transform type of the current block, the block end value of the current block, the block size of the current block, and the aspect ratio of the current block. For example, previously decoded information may include intra-frame or inter-frame prediction mode, main transform type, EOB value, block size, and / or block aspect ratio.

[0109] (A7) In some implementations of any of A1 to A6, the number of secondary transform sets in the plurality of secondary transform sets is based on the intra-prediction mode of the current block. For example, the size of the IST set may depend on the intra-prediction mode.

[0110] (A8) In some implementations of A7, the number of quadratic transform sets is based on the increment angle of the intra-prediction mode. For example, the size of the IST set may depend on whether the increment angle for intra-directional prediction is 0. In some implementations, the size of the selected quadratic transform set is smaller for non-zero increment angles compared to zero increment angles. For example, the size of the IST set is smaller for directional intra-prediction modes with non-zero increment angles.

[0111] (A9) In some implementations of A7, the number of secondary transform sets is based on whether the intra-prediction mode is a directional mode. For example, the size of the IST set can depend on whether the intra-prediction mode is a directional intra-prediction mode. In the example, if the intra-prediction mode is a non-directional mode such as DC, SMOOTH, or PLANA, a larger IST set is allowed.

[0112] (A10) In some implementations of any of A1 to A9, the number of secondary transform sets in the plurality of secondary transform sets is based on the main transform type of the current block. For example, the size of the IST set may depend on the main transform type.

[0113] (A11) In some implementations of A10, the number of the secondary transform set is based on whether the principal transform type includes the discrete cosine transform (DCT). For example, the size of the IST set depends on whether a DCT is chosen as either direction of the principal transform. As an example, the IST set is larger when the principal transform type is DCT-DCT.

[0114] (A12) In some implementations of any of A1 to A11, the number of quadratic transform sets among the multiple quadratic transform sets is based on the end-of-block (EOB) value of the current block. For example, the size of the IST set may be different for different EOB values.

[0115] (A13) In some implementations of A12, the index of the selected quadratic transform set is signaled when the EOB value is greater than a threshold. For example, the IST set index may be signaled for a block only when EOB ≥ M. In some implementations, the index of the selected quadratic transform set is not signaled when the EOB value is less than a threshold. In some implementations, the index of the selected quadratic transform set is signaled when the EOB value is greater than a threshold.

[0116] (A14) In some implementations of A13, the threshold value is an integer in the range of 1 to 64. For example, possible values ​​of M include, but are not limited to, 1, 2, ..., 64.

[0117] (A15) In some implementations of A12, the number of quadratic transform sets among multiple quadratic transform sets is based on the EOB values ​​within a particular EOB group among multiple EOB groups. For example, EOB groups can be defined, and EOB groups can be used as indicators for determining the size of the IST set. For example, EOB values ​​in the ranges of 1 to 8, 9 to 16, 17 to 32, and 33 to 64 can be mapped to EOB groups 0, 1, 2, and 3.

[0118] (A16) In some implementations of any of A1 to A15, the number of quadratic transform sets in the plurality of quadratic transform sets is based on the block size of the current block. For example, the size of the IST set may depend on the block size of the current block.

[0119] (A17) In some implementations of A16, the number of the secondary transform set is based on whether the dimension of the current block is less than a threshold. For example, the size of the IST set can be smaller when the block width or block height is less than or equal to a threshold T1. T1 can be set to 3, 4, or 5.

[0120] (A18) In some embodiments of A1 through A17, the method includes: entropy decoding of the syntax elements of the video bitstream to determine an index of a quadratic transform set, wherein the context for entropy decoding is based on at least one of the following: the block size of the current block, the transform type of the current block, and the EOB value of the current block. For example, the context for signaling the index of the IST set may depend on the block size, transform type, or EOB value. As an example, an EOB group may be used as the context for signaling the index of the IST set.

[0121] (B1) In another aspect, some implementations include a video coding method (e.g., method 550). In some implementations, the method is performed at a computing system (e.g., server system 112) having memory and one or more processors. In some implementations, the method is performed at an encoding / decoding module (e.g., encoding / decoding module 320). The method includes: (i) receiving video data comprising a plurality of blocks, the plurality of blocks including a current block; (ii) determining an intra-prediction mode for the current block; (iii) selecting a set of quadratic transforms from a plurality of quadratic transform sets based on the intra-prediction mode of the current block; and (iv) encoding the current block by applying quadratic transforms from the selected set of quadratic transforms. In some implementations, the set of quadratic transforms is signaled in the video bitstream (e.g., using index values). In some implementations, the index of the set of quadratic transforms is entropy encoded and signaled in the video bitstream. In some implementations, the context for entropy encoding is based on encoded / decoded information of the current block.

[0122] (C1) In another aspect, some embodiments include a method for processing visual media data. In some embodiments, the method is performed at a computing system (e.g., server system 112) having memory and control circuitry. In some embodiments, the method is performed at an encoding / decoding module (e.g., encoding / decoding module 320). The method includes: (i) receiving video data comprising a plurality of blocks, the plurality of blocks including a current block; (ii) determining an intra-frame prediction mode for the current block; (iii) selecting a set of quadratic transforms from a plurality of quadratic transform sets based on the intra-frame prediction mode of the current block; and (iv) encoding the current block by applying quadratic transforms from the selected set of quadratic transforms. The video bitstream includes the encoded current block. In some embodiments, the video bitstream includes an indicator indicating an index value of the set of quadratic transforms.

[0123] (D1) In another aspect, some embodiments include a method for processing visual media data. In some embodiments, the method is performed at a computing system (e.g., server system 112) having memory and control circuitry. In some embodiments, the method is performed at a codec module (e.g., codec module 320). In some embodiments, the method is performed at a source codec component (e.g., source codec 202), a codec engine (e.g., codec engine 212), and / or an entropy codec (e.g., entropy codec 214). The method includes: (i) obtaining a source video sequence comprising a plurality of frames; and (ii) performing a conversion between the source video sequence and a video bitstream of visual media data according to format rules, wherein the video bitstream comprises a plurality of encoded blocks, the plurality of encoded blocks including a current block; and wherein the format rules specify: (a) determining an intra-frame prediction mode for the current block; (b) selecting a set of quadratic transforms from a plurality of quadratic transform sets based on the intra-frame prediction mode of the current block; and (c) reconstructing the current block by applying a quadratic transform from the selected set of quadratic transforms.

[0124] On the other hand, some implementations include a computing system (e.g., server system 112) that includes a control circuitry system (e.g., control circuitry system 302) and a memory (e.g., memory 314) coupled to the control circuitry system. The memory stores one or more sets of instructions configured to be executed by the control circuitry system, including instructions for performing any of the methods described herein (e.g., A1 to A18, B1, C1, and D1 above).

[0125] In another aspect, some implementations include a non-transitory computer-readable storage medium that stores one or more sets of instructions for execution by a control circuitry of a computing system, the set of instructions including instructions for performing any of the methods described herein (A1 to A18, B1, C1 and D1 above).

[0126] Unless otherwise stated, any syntax elements described herein (e.g., indicator) can be High-Level Syntax (HLS). As used herein, the HLS is signaled at a level higher than the block level. For example, the HLS can correspond to the sequence level, frame level, slice level, or tile level. As another example, HLS elements can be signaled in Video Parameter Set (VPS), Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Adaptation Parameter Set (APS), slice header, picture header, tile header, and / or CTU header.

[0127] It should be understood that although the terms “first,” “second,” etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the claims. As used in the description of embodiments and appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or,” as used herein, refers to and covers any and all possible combinations of one or more associated listed items. It will also be understood that, when used in this specification, the terms “comprising” and / or “including” specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0128] As used herein, the term "when" can be interpreted, depending on the context, as meaning "if the prerequisite is true," "after the prerequisite is true," "in response to determining that the prerequisite is true," "based on determining that the prerequisite is true," or "in response to detecting that the prerequisite is true." Similarly, the phrases "if [the prerequisite is true]," "if [the prerequisite is true]," or "when [the prerequisite is true]" can be interpreted, depending on the context, as meaning "after determining that the prerequisite is true," "in response to determining that the prerequisite is true," "based on determining that the prerequisite is true," "after detecting that the prerequisite is true," or "in response to detecting that the prerequisite is true." As used herein, N refers to the number of variables. Unless explicitly stated otherwise, different instances of N may refer to the same number (e.g., the same integer value, such as the number 2) or different numbers.

[0129] For illustrative purposes, the foregoing description has been described with reference to specific embodiments. However, the illustrative discussion above is not intended to be exhaustive or to limit the claims to the precise forms disclosed. In view of the above teachings, many modifications and variations are possible. These embodiments were chosen and described in order to best illustrate the operating principles and practical applications, thereby enabling others skilled in the art to implement them.

Claims

1. A video decoding method, the method being performed on a computing system having a memory and one or more processors, the method comprising: Receive a video bitstream comprising multiple blocks, including the current block; Determine the intra-prediction mode of the current block; A set of secondary transforms is selected from multiple sets of secondary transforms based on encoded and decoded information, wherein the encoded information includes the intra-prediction mode of the current block; as well as The current block is reconstructed by applying a quadratic transformation from the selected set of quadratic transformations.

2. The method according to claim 1, wherein, Selecting a set of quadratic transforms from multiple sets of quadratic transforms includes: Determine the secondary transform index based on the encoded and decoded information; and The quadratic transform set from the plurality of quadratic transform sets is determined using the quadratic transform index and lookup table.

3. The method according to claim 1, wherein, Each of the plurality of quadratic transformation sets has a fixed number of transformations.

4. The method according to claim 3, wherein, Each of the multiple quadratic transform sets has the same number of transformations.

5. The method according to claim 1, wherein, The number of quadratic transform sets is less than all available quadratic transform sets.

6. The method according to claim 1, wherein, The encoded and decoded information also includes at least one of the following: the inter-frame prediction mode of the current block, the main transform type of the current block, the block end value of the current block, the block size of the current block, and the aspect ratio of the current block.

7. The method according to claim 1, wherein, The number of the multiple quadratic transform sets is based on the intra-prediction mode of the current block.

8. The method according to claim 7, wherein, The number of the secondary transform set is based on the incremental angle of the intra-frame prediction mode.

9. The method according to claim 7, wherein, The number of the secondary transform sets is based on whether the intra-frame prediction mode is a directional mode.

10. The method according to claim 1, wherein, The number of quadratic transform sets in the plurality of quadratic transform sets is based on the main transform type of the current block.

11. The method according to claim 10, wherein, The number of the quadratic transform set is based on whether the main transform type includes the discrete cosine transform (DCT).

12. The method according to claim 1, wherein, The number of the multiple quadratic transform sets is based on the end-of-block (EOB) value of the current block.

13. The method according to claim 12, wherein, When the EOB value is greater than the threshold, the index of the selected quadratic transform set is notified by a signal.

14. The method according to claim 13, wherein, The threshold value is an integer in the range of 1 to 64.

15. The method according to claim 12, wherein, The number of the quadratic transform sets in the plurality of quadratic transform sets is based on the EOB value within a certain EOB group among the plurality of EOB groups.

16. The method according to claim 1, wherein, The number of quadratic transform sets in the plurality of quadratic transform sets is based on the block size of the current block.

17. The method according to claim 16, wherein, The number of the quadratic transformation set is based on whether the dimension of the current block is less than a threshold.

18. The method of claim 1, further comprising entropy decoding of the syntax elements of the video bitstream to determine the index of the quadratic transform set, wherein, The context used for the entropy decoding is based on at least one of the following: the block size of the current block, the transformation type of the current block, and the EOB value of the current block.

19. A video encoding method, the method being performed at a computing system having a memory and one or more processors, the method comprising: Receive video data comprising multiple blocks, including the current block; Determine the intra-prediction mode of the current block; A set of quadratic transforms is selected from multiple sets of quadratic transforms based on the intra-frame prediction mode of the current block. as well as The current block is encoded by applying a quadratic transform from the selected set of quadratic transforms.

20. A non-transitory computer-readable storage medium storing a video bitstream generated by a video coding method, the video coding method comprising: Receive video data comprising multiple blocks, including the current block; Determine the intra-prediction mode of the current block; A set of quadratic transforms is selected from multiple sets of quadratic transforms based on the intra-frame prediction mode of the current block. as well as The current block is encoded by applying a quadratic transform from the selected set of quadratic transforms. The video bitstream includes the encoded current block.

Images