Improvements in transform kernel derivation

Abstract

Exemplary methods of decoding and encoding using a classifier to determine transform information are described in this application. An exemplary method of video decoding includes receiving a video bitstream, the video bitstream including a plurality of blocks, the plurality of blocks including a current block. The method also includes applying a classifier to encoded information of the current block to determine a transform kernel for the current block. The method also includes reconstructing the current block by performing an inverse transform on the current block by applying the transform kernel.

Description

[0001] Related applications This application is a continuation to U.S. Patent Application No. 19 / 078,152, filed March 12, 2025, which claims priority to U.S. Provisional Patent Application No. 63 / 566,194, filed March 15, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The disclosed embodiments generally relate to video encoding and decoding, including but not limited to deriving transform kernels. Background Technology

[0003] Digital video is supported by a variety of electronic devices, such as digital televisions, laptops or desktop computers, tablets, digital cameras, digital recording devices, digital media players, video game consoles, smartphones, video conferencing equipment, and video streaming devices. These electronic devices send and receive, or otherwise transmit, digital video data via communication networks, and / or store digital video data on storage devices. Due to the limited bandwidth capacity of communication networks and the limited storage resources of storage devices, video encoding can be employed to compress video data according to at least one video encoding standard before transmission or storage. Video encoding can be performed by hardware and / or software on electronic devices / client devices or on servers providing cloud services.

[0004] Video coding typically employs predictive methods (e.g., inter-frame prediction, intra-frame prediction, etc.) that leverage the inherent redundancy in video data. Video coding aims to compress video data into a form using a lower bitrate while avoiding or minimizing video quality degradation. Several video codec standards have been developed. For example, High Efficiency Video Coding (HEVC / H.265) is a video compression standard designed as part of the MPEG-H project. The ITU-T and ISO / IEC released the HEVC / H.265 standard in 2013 (Revision 1), 2014 (Revision 2), 2015 (Revision 3), and 2016 (Revision 4). Multifunctional Video Coding (VVC / H.266) is a video compression standard designed to succeed HEVC. The ITU-T and ISO / IEC released the VVC / H.266 standard in 2020 (Revision 1) and 2022 (Revision 2). AOMedia Video 1 (AV1) is an open video coding format designed as an alternative to HEVC. On January 8, 2019, specification version 1.0.0, along with errata 1, was released. Enhanced Compression Model (ECM) is a video coding standard currently under development. ECM aims to significantly improve compression efficiency, surpassing existing standards such as HEVC / H.265 and VVC, thereby enabling substantially higher quality video at lower bitrates. Summary of the Invention

[0005] Among other things, this invention describes a set of methods for video (image) compression, including applying at least one classifier to information from encoded blocks to derive a primary transform kernel or a non-primary transform kernel. One advantage of using a classifier to derive the transform kernel is improved coding accuracy. In some embodiments, different classifiers can be trained based on different sets of information. For example, a first classifier can be used for intra-frame prediction blocks, and a second classifier can be used for inter-frame prediction blocks. Using multiple classifiers trained on different datasets (e.g., different types of predictions) can improve the accuracy of each classifier and increase the efficiency of using these classifiers.

[0006] According to some embodiments, a video decoding method includes: (i) receiving a video bitstream (e.g., an encoded video sequence), the video bitstream including a plurality of blocks, the plurality of blocks including a current block; (ii) applying a classifier to the encoded information of the current block to determine a transform kernel for the current block; and (iii) reconstructing the current block by performing an inverse transform on the current block by applying the transform kernel.

[0007] According to some embodiments, a video coding method includes: (i) receiving video data (e.g., a source video sequence), the video data including a plurality of blocks, the plurality of blocks including a current block; (ii) applying a classifier to encoded information of the current block to determine a transform kernel for the current block; and (iii) encoding the current block by performing a transform on the current block by applying the transform kernel.

[0008] According to some embodiments, a bitstream conversion method includes: (i) obtaining a source video sequence comprising multiple frames; and (ii) performing a conversion between the source video sequence and a video bitstream of visual media data according to format rules. The video bitstream comprises multiple encoded blocks, the multiple encoded blocks including a current block. The format rules specify: (a) applying a classifier to the encoded information of the current block to determine the transform kernel of the current block; and (b) reconstructing the current block by performing an inverse transform on the current block by applying the transform kernel.

[0009] According to some embodiments, a computing system, such as a streaming media system, a server system, a personal computer system, or other electronic devices, is provided. The computing system includes control circuitry and a memory storing at least one set of instructions. The at least one set of instructions includes instructions for performing any of the methods described in this application. In some embodiments, the computing system includes an encoder component and a decoder component (e.g., a transcoder).

[0010] According to some embodiments, a non-transitory computer-readable storage medium is provided. This non-transitory computer-readable storage medium stores at least one set of instructions executable by a computing system. The at least one set of instructions includes instructions for performing any of the methods described in this application.

[0011] Therefore, devices and systems utilizing video encoding and decoding methods are disclosed. Such methods, devices, and systems can complement or replace conventional methods, devices, and systems used for video encoding / decoding.

[0012] The features and advantages described in this specification are not necessarily all-encompassing. In particular, some additional features and advantages will become more apparent to those skilled in the art from the accompanying drawings, description, and claims provided with this invention. Furthermore, it should be noted that the wording used in this specification is chosen primarily for readability and guidance purposes and is not necessarily intended to define or limit the subject matter described herein. Attached Figure Description

[0013] To gain a more detailed understanding of the invention, reference can be made to the features of various embodiments, some of which are illustrated in the accompanying drawings. However, the drawings only illustrate relevant features of the invention and are therefore not necessarily limiting, as other effective features may be covered in this specification, as will be understood by those skilled in the art upon reading this invention.

[0014] Figure 1 This is a block diagram illustrating an exemplary communication system according to some embodiments.

[0015] Figure 2A This is a block diagram illustrating exemplary elements of an encoder assembly according to some embodiments.

[0016] Figure 2B This is a block diagram illustrating exemplary elements of a decoder component according to some embodiments.

[0017] Figure 3 This is a block diagram illustrating an exemplary server system according to some embodiments.

[0018] Figures 4A-4C Exemplary prediction blocks, residual blocks, and reconstruction blocks are shown according to some embodiments.

[0019] Figure 5A An exemplary low-frequency non-separable transform (LFNST) process according to some embodiments is shown.

[0020] Figure 5B An exemplary transformation process involving a quadratic transformation is shown according to some embodiments.

[0021] Figure 6A An exemplary video decoding process according to some embodiments is shown.

[0022] Figure 6B An exemplary video encoding process according to some embodiments is shown.

[0023] By convention, the various features shown in the accompanying drawings are not necessarily drawn to scale, and the same reference numerals may be used to denote the same features throughout the specification and the accompanying drawings. Detailed Implementation

[0024] Among other things, this invention describes a set of methods for video (image) compression, including applying a transform to a residual block and adaptively determining transform parameters. For example, an indivisible transform (e.g., a primary or secondary transform) can be selected from an available set of transforms. The transform parameters (e.g., kernel size and / or shape) of the selected indivisible transform can be determined using encoded information available to both the encoder and decoder. In this way, the transform parameters can be adaptively adjusted during encoding and decoding. Adaptive transform parameters allow for the use of smaller sizes where appropriate, resulting in hardware implementation benefits, such as increased throughput and / or reduced hardware components required. In some embodiments, a classifier is used to determine the transform kernel of the current block based on the encoded information of the current block. Using a classifier to determine the transform kernel can reduce signaling overhead (compared to indicating the transform kernel via signaling) and can improve encoding accuracy (compared to other methods of deriving the transform kernel).

[0025] Exemplary systems and devices Figure 1 This is a block diagram illustrating a communication system 100 according to some embodiments. The communication system 100 includes a source device 102 and a plurality of electronic devices 120 (e.g., electronic devices 120-1 to 120-m), which are communicatively coupled to each other via at least one network. In some embodiments, the communication system 100 is a streaming media system, for example, for use with video-enabled applications such as video conferencing applications, digital TV applications, and media storage and / or distribution applications.

[0026] Source device 102 includes a video source 104 (e.g., a camera assembly or media storage device) and an encoder assembly 106. In some embodiments, the video source 104 is a digital camera (e.g., configured to produce an uncompressed video sample stream). The encoder assembly 106 generates at least one encoded video stream from the video stream. The video stream from the video source 104 may have a larger data volume compared to the encoded video stream 108 generated by the encoder assembly 106. Because the encoded video stream 108 has a smaller data volume (less data) compared to the video stream from the video source, less bandwidth is required to transmit the encoded video stream 108, and less storage space is required to store the encoded video stream 108 compared to the video stream from the video source 104. In some embodiments, source device 102 does not include the encoder assembly 106 (e.g., configured to transmit uncompressed video to one or more networks 110).

[0027] At least one network 110 represents any number of networks that transmit information between source device 102, server system 112, and / or electronic device 120, including, for example, wired (connected) and / or wireless communication networks. At least one network 110 can exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and / or the Internet.

[0028] At least one network 110 includes a server system 112 (e.g., a distributed / cloud computing system). In some embodiments, the server system 112 is or includes a streaming media server (e.g., configured to store and / or distribute video content, such as an encoded video stream from source device 102). The server system 112 includes a codec component 114 (e.g., configured to encode and / or decode video data). In some embodiments, the codec component 114 includes an encoder component and / or a decoder component. In various embodiments, the codec component 114 is instantiated as hardware, software, or a combination thereof. In some embodiments, the codec component 114 is configured to decode the encoded video stream 108 and re-encode the video data using different encoding standards and / or methods to generate encoded video data 116. In some embodiments, the server system 112 is configured to generate multiple video formats and / or encodings based on the encoded video stream 108. In some embodiments, the server system 112 serves as a Media-Aware Network Element (MANE). For example, server system 112 can be configured to prune the encoded video stream 108 to customize a potentially different stream for at least one electronic device 120. In some embodiments, MANE is set up separately from server system 112.

[0029] Electronic device 120-1 includes a decoder component 122 and a display 124. In some embodiments, the decoder component 122 is configured to decode encoded video data 116 to generate an output video stream that can be displayed on a display or other type of display device. In some embodiments, at least one electronic device 120 does not include a display component (e.g., it is communicatively coupled to an external display device and / or includes a media storage device). In some embodiments, electronic device 120 is a streaming media client. In some embodiments, electronic device 120 is configured to access server system 112 to obtain encoded video data 116.

[0030] The source device and / or multiple electronic devices 120 are sometimes referred to as “terminal devices” or “user devices”. In some embodiments, the source device 102 and / or at least one electronic device 120 are examples of server systems, personal computers, portable devices (e.g., smartphones, tablets, or laptops), wearable devices, video conferencing equipment, and / or other types of electronic devices.

[0031] In an exemplary operation of the communication system 100, a source device 102 transmits an encoded video stream 108 to a server system 112. For example, the source device 102 may encode an image stream acquired by the source device. The server system 112 receives the encoded video stream 108 and may decode and / or encode the encoded video stream 108 using a codec component 114. For example, the server system 112 may apply encoding more suitable for network transmission and / or storage to the video data. The server system 112 may transmit encoded video data 116 (e.g., at least one encoded video stream) to at least one electronic device 120. Each electronic device 120 may decode the encoded video data 116 and optionally display the video image.

[0032] Figure 2A This is a block diagram illustrating exemplary elements of an encoder assembly 106 according to some embodiments. The encoder assembly 106 receives video data (e.g., a source video sequence) from a video source 104. In some embodiments, the encoder assembly includes a receiver (e.g., transceiver) assembly configured to receive the source video sequence. In some embodiments, the encoder assembly 106 receives the video sequence from a remote video source (e.g., a component of another device different from the encoder assembly 106). The video source 104 can provide a source video sequence in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8-bit, 10-bit, or 12-bit), any color space (e.g., BT.601 YCrCB or RGB), and any suitable sampling structure (e.g., YCrCb 4:2:0 or YCrCb 4:4:4). In some embodiments, the video source 104 is a storage device storing previously acquired / prepared video. In some embodiments, the video source 104 is a camera that acquires local image information as a video sequence. The video data can be provided as multiple independent images that, when viewed sequentially, present motion effects. An image itself can be organized as a spatial array of pixels, where each pixel may include at least one sample, depending on the sampling structure, color space, etc., used. Those skilled in the art can readily understand the relationship between pixels and samples.

[0033] Encoder component 106 is configured to encode and / or compress images of a source video sequence into an encoded video sequence 216 in real time or under other time constraints, as required by the application. In some embodiments, encoder component 106 is configured to perform a conversion between the source video sequence and a visual media data stream (e.g., a video stream). Enforcing an appropriate encoding rate is one of the functions of controller 204. In some embodiments, controller 204 controls and is functionally coupled to other functional units as described below. Parameters set by controller 204 may include rate control related parameters (e.g., image skipping, quantizer, and / or lambda value of rate-distortion optimization techniques), image size, group of pictures (GOP) layout, maximum motion vector (MV) search range, etc. Other functions of controller 204 will be readily recognized by those skilled in the art, as these other functions may involve encoder component 106 being optimized for a specific system design.

[0034] In some embodiments, encoder component 106 is configured to operate in an encoding loop. In a simplified example, the encoding loop includes a source encoder 202 (e.g., responsible for generating symbols, such as a symbol stream, based on the input image to be encoded and one or more reference images) and a (local) decoder 210. Decoder 210 reconstructs the symbols in a manner similar to that of the (remote) decoder to produce sample data (when the compression between the symbols and the encoded video stream is lossless). The reconstructed sample stream (sample data) is input to reference image memory 208. Since decoding the symbol stream produces a bit-by-bit accurate result regardless of the decoder's location (local or remote), the contents of reference image memory 208 are also bit-by-bit accurate between the local encoder and the remote encoder. In this way, the sample values ​​interpreted by the encoder's prediction portion as reference image samples are the same as the sample values ​​interpreted by the decoder when using the prediction during decoding.

[0035] The operation of decoder 210 can be the same as that of a remote decoder, such as decoder component 122, as described below. Figure 2B It will be described in detail. However, a brief reference is provided. Figure 2B When symbols are available and the entropy encoder 214 and the parser 254 are able to encode / decode the symbols into an encoded / decoded video sequence without loss, the entropy decoding portion of the decoder component 122, including the buffer memory 252 and the parser 254, may not be fully implemented in the local decoder 210.

[0036] Aside from parsing / entropy decoding, the decoder techniques described in this application can exist in the corresponding encoders with substantially the same functional form. Therefore, the disclosed subject matter focuses on decoder operation. Furthermore, the description of the encoder techniques can be brief, as these encoder techniques may be the inverse process of the decoder techniques.

[0037] As part of the operation of the source encoder 202, the source encoder 202 may perform motion-compensated predictive coding, referencing at least one previously encoded frame in the video sequence designated as a reference frame, to predictively encode the input frame. In this manner, the encoding engine 212 encodes the differences between pixel blocks of an input frame and pixel blocks of one or more reference frames, wherein the one or more reference frames may be selected as one or more predictive references for the input frame. The controller 204 may manage the encoding operations of the source encoder 202, including, for example, setting parameters and subgroup parameters for encoding video data.

[0038] Decoder 210 decodes encoded video data of frames that can be designated as reference frames based on symbols generated by source encoder 202. The operation of encoding engine 212 can advantageously be a lossy process. When in video decoder ( Figure 2A When decoding encoded video data at a location (not shown), the reconstructed video sequence can be a copy of the source video sequence with some errors. Decoder 210 replicates the decoding process, which can be performed on the reference frame by a remote video decoder, and the reconstructed reference frame can be stored in reference image memory 208. In this way, encoder component 106 locally stores a copy of the reconstructed reference frame that shares common content with the reconstructed reference frame obtained by the remote video decoder (in the absence of transmission errors).

[0039] Predictor 206 can perform a prediction search against encoding engine 212. That is, for a new frame to be encoded, predictor 206 can search the reference image memory 208 for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., which can be used as an appropriate prediction reference for the new image. Predictor 206 can operate pixel-by-pixel based on the sample blocks to find an appropriate prediction reference. Based on the search results obtained by predictor 206, the input image can have prediction references taken from multiple reference images stored in reference image memory 208.

[0040] The outputs of all the aforementioned functional units can be entropy encoded in the entropy encoder 214. The entropy encoder 214 converts these symbols into an encoded video sequence by lossless compression of the symbols generated by the various functional units according to techniques known to those skilled in the art (e.g., Huffman coding, variable-length coding, and / or arithmetic coding).

[0041] In some embodiments, the output of entropy encoder 214 is coupled to a transmitter. The transmitter can be configured to buffer one or more encoded video sequences generated by entropy encoder 214 in preparation for transmission via communication channel 218, which may be a hardware / software link to a storage device for storing the encoded video data. The transmitter can be configured to combine encoded video data from source encoder 202 with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (data sources not shown). In some embodiments, the transmitter may transmit additional data along with the encoded video. Source encoder 202 may include such data as part of the encoded video sequence. Additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant images and slices, Supplementary Enhancement Information (SEI) messages, Visual Usability Information (VUI) parameter set fragments, etc.

[0042] Controller 204 can manage the operation of encoder component 106. During encoding, controller 204 can assign a specific type of encoded image to each encoded image, which may affect the encoding technique applied to the corresponding image. For example, an image can be assigned as an intra-frame image (I-image), a predictive image (P-image), or a bidirectional predictive image (B-image). Intra-frame images can be encoded and decoded without using any other frames in the sequence as prediction sources. Some video codecs support different types of intra-frame images, including, for example, Independent Decoder Refresh (IDR) images. These variations of I-images and their corresponding applications and characteristics are well known to those skilled in the art and will not be repeated here. Predictive images can be encoded and decoded by predicting sample values ​​for each block via intra-frame prediction or via inter-frame prediction using at most one motion vector and reference index. Bidirectional predictive images can be encoded and decoded by predicting sample values ​​for each block via intra-frame prediction or via inter-frame prediction using at most two motion vectors and reference indices. Similarly, multiple predictive images can reconstruct a single block using more than two reference images and associated metadata.

[0043] The source image can typically be spatially subdivided into multiple sample blocks (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples each) and encoded block by block. Blocks can be predictively coded with reference to other (already coded) blocks, which are determined by the coding assignments of the corresponding images applied to the block. For example, blocks of an I-image can be unpredictably coded, or these blocks can be predictively coded (spatial prediction or intra-frame prediction) with reference to already coded blocks of the same image. Pixel blocks of a P-image can be unpredictably coded, encoded via spatial prediction, or encoded via temporal prediction with reference to a previously coded reference image. Blocks of a B-image can be unpredictably coded, encoded via spatial prediction, or encoded via temporal prediction with reference to one or two previously coded reference images.

[0044] Video can be captured as multiple source images (video images) in a time-series manner. Intra-frame image prediction (often simply called intra-prediction) utilizes spatial correlations within a given image, while inter-frame image prediction utilizes (temporal or other) correlations between images. In one example, a specific image in encoding / decoding (called the current image) is divided into multiple blocks. When a block in the current image is similar to a reference block in a previously encoded reference image in the video that is still being buffered, that block in the current image can be encoded using a vector called a motion vector. The motion vector points to the reference block in the reference image and, in the case of using multiple reference images, can have a third dimension that identifies that reference image.

[0045] Encoder component 106 can perform encoding operations according to a predetermined video coding technique or standard, which is any technique or standard such as those described in this application. In operation, encoder component 106 can perform various compression operations, including predictive coding operations utilizing temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

[0046] Figure 2B This is a block diagram illustrating exemplary elements of a decoder component 122 according to some embodiments. Figure 2B The decoder component 122 is coupled to the channel 218 and the display 124. In some embodiments, the decoder component 122 includes a transmitter coupled to the loop filter 256 and configured to transmit data to the display 124 (e.g., via a wired or wireless connection).

[0047] In some embodiments, decoder component 122 includes a receiver coupled to channel 218 and configured to receive data from channel 218 (e.g., via a wired or wireless connection). The receiver may be configured to receive at least one encoded video sequence to be decoded by decoder component 122. In some embodiments, decoding of each encoded video sequence is independent of other encoded video sequences. Each encoded video sequence may be received from channel 218, which may be a hardware / software link to a storage device storing the encoded video data. The receiver may receive the encoded video data along with other data, such as encoded audio data and / or auxiliary data streams, which may be forwarded to their respective user entities (not depicted). The receiver may separate the encoded video sequences from other data. In some embodiments, the receiver receives additional (redundant) data along with the encoded video. The additional data may be included as part of one or more encoded video sequences. The additional data may be used by decoder component 122 to decode the data and / or more accurately reconstruct the original video data. Additional data can take the form of, for example, temporal, spatial, or SNR enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0048] According to some embodiments, decoder component 122 includes buffer memory 252, parser 254 (sometimes also called entropy decoder), scaler / inverse transform unit 258, intra-frame image prediction unit 262, motion compensation prediction unit 260, aggregator 268, loop filter unit 256, reference image memory 266, and current image memory 264. In some embodiments, decoder component 122 is implemented as an integrated circuit, a series of integrated circuits, and / or other electronic circuits. Decoder component 122 may be implemented at least partially in software.

[0049] Buffer memory 252 is coupled between channel 218 and parser 254 (e.g., for combating network jitter). In some embodiments, buffer memory 252 is separate from decoder component 122. In some embodiments, a separate buffer memory is provided between the output of channel 218 and decoder component 122. In some embodiments, in addition to buffer memory 252 inside decoder component 122 (e.g., buffer memory configured to handle playback timing), a separate buffer memory is also provided outside decoder component 122 (e.g., for combating network jitter). Buffer memory 252 may not be necessary or may be small when receiving data from a store / forward device with sufficient bandwidth and controllability or from an isosynchronous network. Buffer memory 252 may be required for use on best-effort packet networks such as the Internet. Buffer memory 252 may be relatively large and / or have an adaptive size and may be implemented at least partially in the operating system or in a similar element outside decoder component 122.

[0050] Parser 254 is configured to reconstruct symbols 270 from the encoded video sequence. These symbols may include, for example, information for managing the operation of decoder component 122, and / or information for controlling display devices such as display 124. Control information for one or more display devices may be in the form of, for example, SEI messages or VUI parameter set fragments (not depicted). Parser 254 parses (entropy decodes) the encoded video sequence. Encoding of the encoded video sequence may be performed according to video coding techniques or standards and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. Parser 254 may extract a subgroup parameter set from the encoded video sequence for at least one pixel subgroup in the video decoder based on at least one parameter corresponding to a group. Subgroups may include image groups, images, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. Parser 254 may also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the encoded video sequence.

[0051] Depending on the type of the encoded video image or its portions (such as inter-frame and intra-frame images, inter-frame blocks and intra-frame blocks) and other factors, the reconstruction of symbol 270 may involve multiple different units. Which units are involved and how these units are involved can be controlled by subgroup control information parsed from the encoded video sequence by parser 254. For clarity, the flow of such subgroup control information between parser 254 and the multiple units described below is not depicted.

[0052] The decoder component 122 can conceptually be divided into multiple functional units, and in some embodiments, these units interact closely with each other and can be at least partially integrated with each other. However, for clarity, the conceptual division of functional units is maintained in this application.

[0053] The scaler / inverse transform unit 258 receives from the parser 254 quantized transform coefficients as one or more symbols 270, along with control information (such as which transform to use, block size, quantization factor, and / or quantization scaling matrix). The scaler / inverse transform unit 258 can output blocks comprising sample values ​​that can be input to the aggregator 268. In some cases, the output samples of the scaler / inverse transform unit 258 relate to intra-coded blocks; that is, the blocks do not use predictive information from previously reconstructed images, but instead can use predictive information from previously reconstructed portions of the current image. Such predictive information can be provided by the intra-image prediction unit 262. The intra-image prediction unit 262 can use surrounding reconstructed information obtained from the current (partially reconstructed) image in the current image memory 264 to generate blocks of the same size and shape as the blocks in the reconstruction. The aggregator 268 can add the predictive information generated by the intra-image prediction unit 262 to the output sample information provided by the scaler / inverse transform unit 258 on a per-sample basis.

[0054] In other cases, the output samples of the scaler / inverse transform unit 258 may relate to inter-frame coded and potentially motion-compensated blocks. In such cases, the motion compensation prediction unit 260 can access the reference image memory 266 to obtain samples for prediction. After motion compensation of the obtained samples according to the block-related symbols 270, these samples can be added by the aggregator 268 to the output of the scaler / inverse transform unit 258 (referred to in this case as residual samples or residual signals) to generate output sample information. The address within the reference image memory 266 for the motion compensation prediction unit 260 to obtain predicted samples can be controlled by motion vectors. Motion vectors can be used by the motion compensation prediction unit 260 in the form of symbols 270, which may have, for example, X, Y, and reference image components. Motion compensation may also include, for example, interpolation of sample values ​​obtained from the reference image memory 266 when using subsampled precise motion vectors, and motion vector prediction mechanisms.

[0055] The output samples of aggregator 268 can be employed by various loop filtering techniques in loop filter unit 256. Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video bitstream and provided to loop filter unit 256 from parser 254 in the form of symbol 270, but may also respond to metadata obtained during decoding of previous (in decoding order) portions of the encoded image or encoded video sequence, and to previously reconstructed and loop-filtered sample values. The output of loop filter unit 256 may be a sample stream, which can be output to a display device such as display 124 and stored in reference image memory 266 for future inter-frame image prediction.

[0056] Once a certain encoded image is reconstructed, it can be used as a reference image for future predictions. Once an encoded image is reconstructed and has been identified as a reference image (e.g., by the parser 254), the current reference image can become part of the reference image memory 266, and a new current image memory can be reallocated before reconstructing subsequent encoded images begins.

[0057] Decoder component 122 can perform decoding operations according to predetermined video compression techniques that may be described in standards such as any standards described in this application. If the encoded video sequence follows the syntax of a video compression technique or standard, such as the syntax specified in a video compression technique document or standard, particularly in its configuration file document, the encoded video sequence may conform to the syntax specified by the video compression technique or standard used. Furthermore, to conform to some video compression techniques or standards, the complexity of the encoded video sequence may be within the range defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy defines the maximum image size, maximum frame rate, maximum reconstruction sampling rate (measured, for example, in megasamples per second), maximum reference image size, etc. In some cases, the limitations set by the hierarchy can be further defined by the Hypothetical Reference Decoder (HRD) specification and the metadata managed by the HRD buffer indicated by signaling in the encoded video sequence.

[0058] Figure 3This is a block diagram illustrating a server system 112 according to some embodiments. The server system 112 includes control circuitry 302, at least one network interface 304, memory 314, a user interface 306, and at least one communication bus 312 for interconnecting these components. In some embodiments, the control circuitry 302 includes at least one processor (e.g., CPU, GPU, and / or DPU). In some embodiments, the control circuitry includes one or more field-programmable gate arrays, multiple hardware accelerators, and / or one or more integrated circuits (e.g., application-specific integrated circuits).

[0059] One or more network interfaces 304 can be configured to connect to at least one communication network (e.g., a wireless network, a wired network, and / or an optical network). The communication network can be a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a vehicular and industrial network, a real-time network, a latency-tolerant network, etc. Examples of communication networks include: LANs such as Ethernet; wireless LANs; cellular networks including GSM, 3G, 4G, 5G, LTE, etc.; wired or wireless wide area digital television networks, including cable TV, satellite TV, and terrestrial broadcast TV; vehicular and industrial networks, including CANBus, etc. Such communication can be one-way receiving (e.g., broadcast TV), one-way transmitting (e.g., CANBus to certain CANBus devices), or bidirectional (e.g., connecting to other computer systems using a LAN or WAN digital network). Such communication can include communication with at least one cloud computing network.

[0060] User interface 306 includes at least one output device 308 and / or at least one input device 310. One or more input devices 310 may include at least one of the following: keyboard, mouse, touchpad, touchscreen, data glove, joystick, microphone, scanner, camera, etc. One or more output devices 308 may include at least one of the following: audio output device (e.g., speaker), visual output device (e.g., display or monitor), etc.

[0061] Memory 314 may include high-speed random access memory (such as DRAM, SRAM, DDR RAM and / or other random access solid-state storage devices) and / or non-volatile memory (such as at least one disk storage device, optical disk storage device, flash memory device and / or other non-volatile solid-state storage devices). Memory 314 may optionally include at least one storage device located remotely from control circuitry 302. Memory 314, or alternatively, one or more non-volatile solid-state storage devices within memory 314, include non-transitory computer-readable storage media. As used herein, "non-transitory computer-readable storage media" refers to any tangible medium capable of storing data, including but not limited to random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical disk, magnetic disk, etc., but excluding transiently propagated signals themselves. In some embodiments, memory 314 or the non-transitory computer-readable storage medium of memory 314 stores the following programs, modules, instructions, and data structures, or subsets or supersets thereof: ● Operating system 316, which includes programs for handling various basic system services and for performing hardware-dependent tasks; ● A network communication module 318 for connecting the server system 112 to other computing devices via at least one network interface 304 (e.g., via wired and / or wireless connection); ● An encoding / decoding module 320 is used to perform various functions related to encoding and / or decoding data such as video data. In some embodiments, the encoding / decoding module 320 is an instance of the codec component 114. The encoding / decoding module 320 includes, but is not limited to, at least one of the following: ○ Decoding module 322, which performs various functions related to decoding encoded data, such as those previously described for decoder component 122; and ○ Encoding module 340, which performs various functions related to encoding data, such as those previously described for encoder component 106; and ● Image memory 352, which is used to store images and image data, for example, for use by the encoding / decoding module 320. In some embodiments, image memory 352 includes at least one of the following: reference image memory 208, buffer memory 252, current image memory 264, and reference image memory 266.

[0062] In some embodiments, the decoding module 322 includes a parsing module 324 (e.g., configured to perform various functions previously described for the parser 254), a transformation module 326 (e.g., configured to perform various functions previously described for the scaler / inverse transformation unit 258), a prediction module 328 (e.g., configured to perform various functions previously described for the motion compensation prediction unit 260 and / or the intra-frame image prediction unit 262), and a filter module 330 (e.g., configured to perform various functions previously described for the loop filter 256).

[0063] In some embodiments, the encoding module 340 includes a code module 342 (e.g., configured to perform various functions previously described for the source encoder 202 and / or encoding engine 212) and a prediction module 344 (e.g., configured to perform various functions previously described for the predictor 206). In some embodiments, the decoding module 322 and / or the encoding module 340 includes... Figure 3 A subset of the modules shown. For example, the shared prediction module is shared by both the decoding module 322 and the encoding module 340.

[0064] The modules identified above and stored in memory 314 correspond to instruction sets for performing the functions described herein. The modules identified above (e.g., instruction sets) do not need to be implemented as separate software programs, processes, or modules; therefore, in various embodiments, subsets of these modules may be combined or otherwise rearranged. For example, the codec module 320 may optionally not include separate decoding and encoding modules, but instead use the same set of modules to perform both sets of functions. In some embodiments, memory 314 stores a subset of the modules and data structures identified above. In some embodiments, memory 314 stores additional modules and data structures not described above.

[0065] although Figure 3 A server system 112 according to some embodiments is shown, but Figure 3 This is intended more as a functional description of various features that may exist in at least one server system, rather than a structural schematic diagram of the embodiments described in this application. In practice, items shown individually may be combined, and some items may be separated. For example, Figure 3 Some of the items shown separately can be implemented on a single server, while a single item can be implemented by at least one server. The actual number of servers used to implement server system 112 and how features are distributed among these servers will vary depending on the specific implementation and, optionally, in part, on the amount of data traffic processed by the server system during peak and average usage periods.

[0066] Exemplary codec techniques The encoding and decoding processes and techniques described below can be performed at the aforementioned devices and systems (e.g., source device 102, server system 112, and / or electronic device 120). According to some embodiments, methods for determining the transform kernel are described.

[0067] As discussed above, a block can refer to a coding tree block, a maximum coding block, a predefined fixed block size, a coding block, a prediction block, a residual block, or a transform block. An inter-mode coded block (or inter-block) refers to a block that uses an inter-prediction mode or an intra-inter joint prediction mode. An inter-mode can also refer to a block coded using a block vector (BV), which is used within the same frame, for example, by copying intra-blocks to obtain prediction blocks. An intra-mode coded block (or intra-block) refers to a block that uses an intra-prediction mode or an intra-inter joint prediction mode. An intra-mode list can correspond to a list of the most likely intra-prediction modes for the current block. Additionally, the term "partition" can correspond to block partitioning or transform partitioning.

[0068] For example, a coding tree unit (CTU) can be partitioned into coding units using a quadtree structure represented as a coding tree to accommodate various local characteristics. In some embodiments, the decision on whether to use inter-frame (temporal) prediction or intra-frame (spatial) prediction to encode an image region is made at the coding unit level. Each coding unit can be further partitioned into one, two, or four prediction units based on the prediction unit partitioning type. Within a prediction unit, the same prediction process is applied, and relevant information can be transferred to the decoder based on the prediction unit. After obtaining residual blocks by applying the prediction process based on the prediction unit partitioning type, the coding unit can be partitioned into transform units according to another quadtree structure similar to the coding tree of the coding unit.

[0069] A quadtree partitioning structure with nested multi-type trees employing binary and ternary partitioning can be used to replace the concept of multiple partitioning unit types. In the coding tree structure, coding units can have square or rectangular shapes. Coding tree units are first partitioned using a quadtree structure. Quadtree leaf nodes can be further partitioned using multi-type tree structures. An exemplary multi-type tree structure includes four partitioning types. Multi-type leaf nodes are called coding units unless the coding unit is excessively large for the maximum transform length. This means that in a quadtree with a nested multi-type tree coding block structure, coding units, prediction units, and transform units can have the same block size.

[0070] The coding tree scheme supports independent block tree structures for luma and chroma, as in VTM7. In some cases, for P-slices and B-slices, luma CTBs and chroma CTBs within a coding tree unit share the same coding tree structure. However, for I-slices, luma and chroma can have separate block tree structures. When the separate block tree mode is applied, luma CTBs are divided into coding units through one coding tree structure, while chroma CTBs are divided into chroma coding units through another coding tree structure. This means that a coding unit in an I-slice can include one coding block of the luma component or two coding blocks of the chroma components, or a combination thereof, and unless the video is monochrome, a coding unit in a P-slice or B-slice can always include coding blocks of all three color components, or a combination thereof.

[0071] Turning now to transforms and transform blocks, a transform performed during the decoding of a video stream can be the inverse transform of a transform performed during the encoding of the video stream, sometimes referred to as an "inverse transform." It is important to note that the encoder component applies the transform, while the decoder component performs the inverse transform. Therefore, in the following description, a transform described in the context of the decoder component can be the inverse transform of the transform applied at the encoder end. For simplicity, any transform described in this application, whether performed during encoding or decoding, can be referred to as a "transform."

[0072] Various transformation sizes (e.g., dimensions ranging from 4 to 64 points) and transformation shapes (e.g., squares or rectangles with aspect ratios of 2:1 / 1:2 and 4:1 / 1:4) can be utilized. As described in further detail below, transformations can correspond to primary or secondary transformations, and separable or non-separable transformations. A transformation set is a grouping of at least one transformation type. Thus, a transformation set indicates a set of multiple transformation kernels / basis, and a single transformation kernel / basis. Entries in a transformation set can be referred to as transformation candidates. For each block, the transformation candidates selected from the transformation set can be indicated by signaling or implicitly identified.

[0073] Some embodiments include methods for indicating the transform set and / or type of intra- and / or inter-coded blocks via signaling. The transform type may belong to the sinusoidal transform, KLTs, or line-graph transform (LGT) families. The (primary or secondary) transform may belong to the sinusoidal transform family (DCT, DST, the inverted version of DCT, and ADST). DCT may refer to any transform using a transform kernel derived from a discrete cosine transform basis, and DST / ADST may refer to any transform using a transform kernel derived from a discrete sine transform basis.

[0074] An exemplary primary transform can belong to the generalized LGT family, or it can be based on a trained kernel. An exemplary set of secondary transforms can be a grouping of at least one inseparable secondary transform kernel transform type. For each primary transform type and / or intra-frame or inter-frame mode type, a unique or common set of secondary transforms can be defined.

[0075] Furthermore, an inseparable transform can refer to the master transform applied directly to the residuals, or the quadratic transform applied to the transform coefficient block generated by the master transform. Transform kernels can be grouped into sets, represented by a set index and a kernel index within the set. Inseparable quadratic transforms can be trained kernels applied to the master transform coefficients at the encoder, or applied to the inverse quantization coefficients at the decoder.

[0076] An inseparable quadratic transformation kernel can be viewed as a set of basis vectors in a vector space. If represented as an M×N (M rows and N columns) matrix, then N corresponds to the dimension of the vector space, and M is the number of basis vectors. Therefore, M×N can be used to represent the kernel size. Examples of kernel sizes include, but are not limited to, 64×64, 32×64, 16×64, 8×84, 4×64, 16×16, 8×16, 4×16, 8×8, and 4×4 samples.

[0077] The two-dimensional transformation process may involve the use of a hybrid transformation kernel (e.g., composed of different one-dimensional transformations for each dimension of the encoded residual block). The main one-dimensional transformation may include at least one of the following: a) 4-point, 8-point, 16-point, 32-point, or 64-point discrete cosine transforms; b) 4-point, 8-point, or 16-point asymmetric discrete sine transforms and their inverted versions; or c) 4-point, 8-point, 16-point, or 32-point identity transforms.

[0078] For chroma components, transform type selection can be performed implicitly. For intra-frame prediction residuals, the transform type can be selected based on the intra-frame prediction mode. For inter-frame prediction residuals, the transform type can be selected based on the transform type selection of the co-position luma block. Therefore, for chroma components, transform type signaling may not be required in the bitstream.

[0079] The principal transform can belong to the family of sinusoidal transforms (DCT, DST, the inverted version of DCT, and ADST). DCT can refer to any transform that uses a transform kernel derived from a discrete cosine transform basis (e.g., DCT type 2), and DST / ADST can refer to any transform that uses a transform kernel derived from a discrete sine transform basis (e.g., DST type 4 or 7).

[0080] The primary transform can belong to the generalized LGT family, or it can be based on a trained kernel. The set of secondary transforms can be a grouping of at least one inseparable secondary transform kernel transform type. For each primary transform type and / or intra-frame or inter-frame mode type, a unique or common set of secondary transforms can be defined.

[0081] The end-of-block (EOB) value corresponds to the position of the last valid (e.g., non-zero) coefficient in the encoded block according to a given coefficient scan order. For a given coefficient scan order, all coefficients after the EOB position are zero. In some embodiments, if an M×N inseparable quadratic transform kernel is applied to the encoded block, the EOB value ≤ M.

[0082] Scanning order refers to the coefficient recombination process of mapping a two-dimensional principal transform coefficient array to a one-dimensional principal transform coefficient array as the input for the forward quadratic transform. It can also refer to the reverse coefficient recombination process of mapping a one-dimensional quadratic transform coefficient array back to a two-dimensional principal transform coefficient array.

[0083] Now let's turn to an example encoding and decoding using predictive blocks and residual blocks. Figure 4A The calculation of a prediction block is shown according to some embodiments. Figure 4A In the example, intra-frame prediction is performed on the current block 402 to generate prediction block 404. In some embodiments, inter-frame prediction is performed to generate prediction blocks. The current block 402 includes a set of samples (e.g., pixel blocks), and prediction block 404 includes a set of predictions corresponding to that set of samples. Figure 4B The calculation of residual blocks according to some embodiments is shown. Figure 4B As shown, prediction block 404 is subtracted from current block 402 to generate residual block 406, which includes a set of residual values. For example, the corresponding difference between each sample and its corresponding prediction is calculated. Figure 4C The calculation of the reconstructed block is shown according to some embodiments. For example... Figure 4C As shown, residual block 406 undergoes at least one transform and quantization to generate a set of residual coefficients. This set of residual coefficients can be transferred from the encoder component to the decoder component. This set of residual coefficients undergoes inverse quantization and inverse transform to generate a reconstructed residual block 408. The reconstructed residual block 408 is combined with the prediction block 404 (e.g., the reconstructed residual value of the reconstructed residual block 408 is added to the prediction of the prediction block 404) to generate a reconstructed block 410 corresponding to the current block 402.

[0084] Figure 5A An exemplary LFNST procedure according to some embodiments is shown. LFNST is also known as Reduced Secondary Transform (RST). Figure 5AAs shown, LFNST can be applied between the forward master transform 502 and quantization 506 (e.g., at the encoding component), and between the inverse quantization 508 and the inverse master transform 512 (e.g., at the decoding component).

[0085] Exemplary LFNST 504 may include 16 input coefficients for a 4×4 forward LFNST or 64 input coefficients for an 8×8 forward LFNST. In the exemplary LFNST, a 4×4 or 8×8 inseparable transform is applied depending on the block size. For example, a 4×4 LFNST may be applied to small blocks (e.g., blocks with a width or height less than 8 samples) and an 8×8 LFNST may be applied to larger blocks (e.g., blocks with a width or height greater than 4 samples). Exemplary LFNST 510 may include 8 input coefficients for a 4×4 inverse LFNST or 16 input coefficients for an 8×8 inverse LFNST.

[0086] For example, four transform sets can be used in LFNST, and each transform set contains two inseparable transform matrices (kernels). The mapping from intra-prediction modes to transform sets can be predefined, as shown in Table 1 below. If the current block uses one of the three CCLM modes (INTRA_LT_CCLM, INTRA_T_CCLM, or INTRA_L_CCLM) (81 <= IntraPredMode <= 83), then (e.g., for the current chroma block) transform set 0 is selected. For each transform set, the selected inseparable quadratic transform candidate can be further specified by an LFNST index explicitly indicated by signaling. This index can be indicated once in the bitstream for each intra-coding unit by signaling (e.g., after the transform coefficients).

[0087]

[0088] Table 1 - Transformation Selection Table The above LFNST can be adjusted as follows. First, more transform sets can be used to achieve finer directional granularity (e.g., using 35 transform sets instead of 4). Second, each set can include more transform kernels (e.g., including 3 transform kernels instead of 2). Third, three different kernels (e.g., LFNST4, LFNST8, and LFNST16) can be defined to indicate the LFNST kernel set, which can be applied to 4×N / N×4 (N≥4), 8×N / N×8 (N≥8), and M×N (M, N≥16), respectively. Fourth, for a given intra-pred_mode, the LFNST set (LFNST_set_idx) can be derived according to different formulas, as shown in Table 2.

[0089] In another example, when Intra_pred_mode < 2, LFNST_set_idx equals 2; when Intra_pred_mode is in the range [0, 34], LFNST_set_idx = Intra_pred_mode; when Intra_pred_mode is in the range [35, 66], LFNST_set_idx = 68 – Intra_pred_mode.

[0090] The non-separable primary transform (NSPT) can be used to replace the transform combination of separable DCT-II and LFNST, for example, for block shapes of 4×4, 4×8 / 8×4, 4×16 / 16×4, 8×8, 8×16 / 16×8, 4×32 / 32×4, and 8×32 / 32×8. For a given intra-frame mode, the NSPT set index can also be derived based on Table 2.

[0091]

[0092] Table 2 – Mapping between Intra-Frame Modes and LFNST Set Indexes In some embodiments, LFNST is restricted to application only when all coefficients outside the first coefficient subgroup are invalid. In these embodiments, LFNST index encoding depends on the position of the last valid coefficient. LFNST indexes can be context-coded (e.g., independent of intra-prediction mode). In some embodiments, only the first bit is context-coded. In some embodiments, LFNST is applied to intra-coding units in intra-slices and inter-slices, and to both the luma and chroma components. If dual-tree is enabled, the LFNST indexes for the luma and chroma components can be indicated separately via signaling. For inter-slices (dual-tree disabled), a single LFNST index can be indicated via signaling and used for both the luma and chroma components.

[0093] In some embodiments, when the intra subpartition (ISP) mode is selected for the current block, LFNST can be disabled and the RST index can be indicated without signaling (e.g., because the performance improvement might be negligible even if RST were applied to every feasible partition block). Additionally, disabling RST on the ISP prediction residual can reduce coding complexity. When the matrix-based intra prediction (MIP) mode is selected, LFNST can also be disabled (and the index can be indicated without signaling).

[0094] Due to the existing maximum transform size limit (64×64), large coding units (e.g., larger than 64×64) may be implicitly partitioned (transform unit chunking). LFNST index search may increase the number of data buffers in a certain number of decoding pipeline stages by up to four times. Therefore, the maximum size allowed by LFNST can be limited (e.g., limited to 64×64). In some embodiments, LFNST is enabled only for DCT2.

[0095] In some embodiments, a separable transform is applied to intra-frame residual samples and inter-frame residual samples. In some embodiments, an intra-secondary transform (IST) scheme is customized for the video coding library (e.g., for transforming intra-frame residual blocks). Compared to NSPT, the IST scheme can efficiently capture directional patterns in intra-frame residual samples with lower complexity. In the IST scheme, the IST kernel can be classified using a nominal intra-frame prediction angle.

[0096] In some embodiments, a quadratic transform method (e.g., IST) is applied to the main transform coefficient block (e.g., the intra-prediction residual block for the luma component) before quantization is applied at the encoder. Therefore, an inverse quadratic transform can be applied to the inverse quantization transform coefficient block before the inverse main transform is applied at the decoder. In some embodiments, IST is not applied to the chroma component. Figure 5B The use of IST in the encoding and decoding process is shown. Figure 5B The diagram illustrates the application of a primary transform 552 to the residual block (e.g., corresponding to an intra-prediction block). A secondary transform 554 is applied to the output of the primary transform 552. Quantization 556 is applied to the output of the secondary transform 554, and the resulting quantization coefficients are entropy-coded 558 and signaled via the video bitstream. The video bitstream is parsed 560 (e.g., at the decoder), and the quantization coefficients are dequantized 562. An inverse secondary transform 564 is applied to the dequantized data, and an inverse primary transform 566 is applied to the output of the secondary transform 564. In this manner, the reconstructed residual block is generated.

[0097] In some embodiments, 12 quadratic transform sets (or IST sets) are defined, each containing 3 quadratic transform kernels. In some embodiments, for each intra-coded transform block, the nominal intra-prediction mode and the main transform type can be identified, and then the IST set is selected according to Table 3 below. In some embodiments, for Paeth prediction mode and recursive intra-prediction mode, neither the IST is applied nor indicated by signaling.

[0098]

[0099] Table 3 – Selection of the Quadratic Transform Set Given an IST set with three cores, there are four encoder options: 1) no secondary transform, 2) secondary transform using the first transform core in the given IST set, 3) secondary transform using the second transform core in the given IST set, and 4) secondary transform using the third transform core in the given IST set. The encoder can indicate this selection via signaling using a syntax element (e.g., ist_idx). At the decoder, the value of the syntax element is parsed, and given the IST set and the value associated with ist_idx, the secondary transform core is identified. After indicating the primary transform type via signaling, the syntax element (ist_idx) can be indicated via signaling for each luma transform block. For example, ist_idx can be indicated via signaling if at least one of the following is true: (i) the current block is an intra-coded luma transform block, (ii) the primary transform type is DCT in both dimensions or ADST in both dimensions, (iii) the intra-prediction mode is neither Pace prediction mode nor recursive intra-prediction mode, (iv) the transform partition depth is 0, and (v) the EOB position is located within the low-frequency transform coefficient region where the secondary transform is applicable. In some embodiments, the entropy coding context of ist_idx is derived based on the transform block size. Conceptually, IST can be considered another name for LFSNT. Those skilled in the art will understand that IST and LFNST are interchangeable in this invention.

[0100] If the intra-prediction mode is not a smooth mode, or if the intra-prediction mode generates prediction samples based on a given prediction direction, then the intra-prediction mode can be called an angle mode or a direction mode.

[0101] In some embodiments, classifiers are used at both the encoder and decoder, and the selection of a specific transform kernel is not indicated by signaling. A classifier can be defined as classifying encoded information (e.g., prediction pattern information, residual block information, and / or neighboring block information) of the current block, where each category is mapped to a specific transform kernel. Taking a classifier that selects between two transform kernels as an example, reference samples can be divided into two groups, and samples from each group can be used as a training set to derive the corresponding transform kernel. The classification process can be based on at least one characteristic of the reference samples (e.g., the mean, weighted mean, or median of all reference samples) being less than or equal to a threshold (e.g., classified into the first group, associated with the first transform kernel) or greater than a threshold (e.g., classified into the second group, associated with a second transform kernel different from the first transform kernel). For example, the threshold could be the average of the characteristics of all reference samples from a specific neighboring region in the training dataset.

[0102] In some embodiments, a gradient-based classifier (such as an adaptive loop filter) is used to derive a classification index based on the encoded information input. In some embodiments, the classification index is determined using the orientation of a reference sample based on horizontal and vertical gradients. In some embodiments, activity is computed by summing gradients. In some embodiments, samples are classified using orientation and / or activity. In some embodiments, another lookup table is used to map gradients (e.g., gradients derived from pixel-level information of the current block) to a specific transform kernel.

[0103] In some embodiments, a matrix-based classifier (e.g., a neural network-based classifier) ​​is used to derive the classification index. In some embodiments, the matrix is ​​trained offline to select a transformation kernel that minimizes the loss or another performance metric.

[0104] Figure 6A This is a flowchart illustrating a video decoding method 600 according to some embodiments. Method 600 can be executed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having control circuitry and a memory storing instructions for execution by the control circuitry. In some embodiments, method 600 is executed by executing instructions stored in the computing system's memory (e.g., memory 314).

[0105] The system receives (602) a received video stream (e.g., an encoded video sequence), the video stream comprising multiple blocks (e.g., corresponding to a set of images), the multiple blocks including the current block. The system applies a classifier to (604) the encoded information of the current block to determine the transform kernel of the current block. The system reconstructs (606) the current block by applying the transform kernel to perform an inverse transform on the current block. In this way, at least one classifier can be applied to the information of the encoded block to derive the primary transform kernel or a non-primary transform kernel.

[0106] In some embodiments, the information of the encoded block includes a residual block. The residual block is derived as the block difference between the current block and the predicted block, where the predicted block can be an intra-predicted block, a motion-compensated block pointed to by a motion vector derived from or given in the reference image, or an intra-copy block pointed to by a block vector or displacement vector (DV) within the reconstructed region. In one example, coefficients within a predefined region of the residual block are used to select the primary transform kernel or a non-primary transform kernel.

[0107] In some embodiments, the information of the encoded block includes neighboring residual samples of the residual block. In some embodiments, the information of the encoded block includes neighboring samples of the reconstructed current block. The neighboring reconstructed samples (e.g., templates) may be samples from the upper, left, and / or upper left reconstructed regions. In one example, at least one classifier is constructed based on template costs of a set of predefined prediction patterns.

[0108] In some embodiments, the information of the encoded block includes samples of the predicted block from the reference image. In some embodiments, the information of the encoded block includes neighboring samples of the predicted block. For example, neighboring samples of the predicted block may come from the upper, left, right, and / or lower regions of the reference image.

[0109] In some embodiments, at least one classifier is derived based on a histogram of gradient values ​​of information from the encoded block. In some embodiments, N associated classifiers with the highest distribution in the histogram are selected to derive the corresponding transformation kernel, where N is a non-zero positive value. In some embodiments, N associated classifiers with the lowest distribution in the histogram are used. In some embodiments, M associated classifiers with the highest distribution in the histogram are selected, and a pruning method is applied to these M classifiers to determine N classifiers, and the corresponding transformation kernel is derived from these N classifiers.

[0110] In some embodiments, the M lowest distributions in a histogram are used to determine one or more classifiers. In some embodiments, at least one classifier is derived based on a combination of magnitude and gradient values ​​of information from the encoded block. In some embodiments, the N associated classifiers with the highest combination values ​​are selected to derive the corresponding transform kernel, where N is a non-zero positive value. In some embodiments, the N associated classifiers with the lowest combination values ​​are used. In some embodiments, the M associated classifiers with the highest combination values ​​are selected, and a pruning method is applied to these M classifiers to determine N classifiers, and the corresponding transform kernel is derived from these N classifiers. In some embodiments, the M lowest combination values ​​are used to determine one or more classifiers.

[0111] In some embodiments, each category in each classifier has one or more corresponding principal transform kernels or non-principal transform kernels. In some embodiments, at least two classifiers are used (e.g., the classifiers are selected from a set of at least two classifiers). In one example, two different classifiers are used for intra-frame prediction blocks and inter-frame prediction blocks, respectively.

[0112] As disclosed above, for intra-coded blocks, the index of NSPT or LFNST can be determined using intra-prediction modes. For inter-coded blocks, the intra-prediction modes can be derived first using the inter-prediction block and decoder-side intra-mode derivation (DIMD) process. For example, the gradients of pixels within the prediction block (e.g., horizontal and / or vertical gradients) can be calculated (e.g., using the Sobel operator), and the magnitudes of the gradients can be accumulated over the corresponding directions. Then, the intra-prediction mode corresponding to the highest accumulated value can be used as the mapping intra-prediction mode for that block. As previously mentioned, using a classifier to map prediction modes to transform kernels in this way can improve coding accuracy.

[0113] In some embodiments, all classifiers have the same number of categories. In some embodiments, all classifiers may have any number of categories.

[0114] In some embodiments, at least one classifier is indicated via signaling at different levels in the bitstream (such as image level, sub-image level, slice level, and / or block level). In some embodiments, at least one classifier is implicitly derived from other decoded syntax.

[0115] Figure 6B This is a flowchart illustrating a video encoding method 650 according to some embodiments. Method 650 can be executed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having control circuitry and a memory storing instructions for execution by the control circuitry. In some embodiments, method 650 is executed by executing instructions stored in the computing system's memory (e.g., memory 314). In some embodiments, method 650 is executed by the same system as method 600 described above.

[0116] The system receives (652) received video data (e.g., a source video sequence), the video data comprising multiple blocks, including the current block. The system applies a classifier (654) to the encoded information of the current block to determine the transform kernel of the current block. The system encodes the current block by applying the transform kernel to perform a transform on the current block (656). As previously mentioned, the encoding process can be symmetrical to the decoding process described in this application (e.g., applying a transform). For the sake of brevity, these details will not be repeated here.

[0117] although Figure 6A and 6BMultiple logical stages are shown in a specific order, but these stages are independent of the order and can be reordered, and other stages can be combined or split. Some reorderings or other groupings not specifically mentioned will be obvious to those skilled in the art, and therefore the orderings and groupings presented in this application are not exhaustive. Furthermore, it should be understood that the stages can be implemented in hardware, firmware, software, or any combination thereof.

[0118] We now turn to some exemplary embodiments.

[0119] (A1) In one aspect, some embodiments include a video decoding method (e.g., method 600). In some embodiments, the method is performed at a computing system (e.g., server system 112) having memory and control circuitry. In some embodiments, the method is performed at an encoding / decoding module (e.g., encoding / decoding module 320). In some embodiments, the method is performed at a source encoding component (e.g., source encoder 202), an encoding engine (e.g., encoding engine 212), and / or an entropy encoder (e.g., entropy encoder 214). The method includes: (i) receiving a video bitstream (e.g., an encoded video sequence) comprising a plurality of blocks (e.g., a set corresponding to at least one image), the plurality of blocks including a current block; (ii) applying a classifier to encoded information of the current block to determine a transform kernel for the current block; and (iii) reconstructing the current block by performing an inverse transform on the current block by applying the transform kernel. For example, at least one classifier may be applied to information of the encoded block to derive a primary transform kernel or a non-primary transform kernel. In some embodiments, the transform kernel is selected from a set of transform kernels. For example, the transform kernel may be selected from a set of 35 transform kernels.

[0120] (A2) In some embodiments of A1, the transform kernel is a primary transform kernel. In some embodiments, the transform kernel is a non-primary (e.g., secondary) transform kernel.

[0121] (A3) In some embodiments of A1 or A2, the encoded information includes information about the residual blocks of the current block. For example, the information about the encoded block may include residual blocks, wherein the residual blocks are derived as block differences between the current block and the prediction block. For example, the prediction block may be an intra-prediction block, a motion-compensated block derived from a reference image or pointed to by a given MV, or an intra-copy block pointed to by a BV or DV within the reconstruction region. In some embodiments, the information about the residual blocks includes information about one or more residual magnitudes and / or one or more residual distributions.

[0122] (A4) In some embodiments of A3, the classifier uses one or more coefficients within a predefined region of the residual block to determine the transform kernel. For example, coefficients within a predefined region of the residual block (e.g., top left, top right, etc., depending on the subblock transform (SBT) mode) are used to select the primary or non-primary transform kernel.

[0123] (A5) In some embodiments of any of A1 to A4, the encoded information includes at least one residual sample from neighboring blocks of the current block. For example, the information of the encoded block may include neighboring residual samples from residual blocks. For example, latency can be reduced by using neighboring block residuals instead of the current block residual. In some embodiments, the classifier uses at least one neighboring residual and at least one residual from the current block.

[0124] (A6) In some embodiments of any of A1 to A5, the encoded information includes at least one reconstructed sample from neighboring blocks of the current block. For example, the information of the encoded block may include reconstructed neighboring samples of the current block. The reconstructed neighboring samples are referred to as templates and may include samples from the upper, left, and / or upper left reconstructed regions. In some embodiments, when intra-frame prediction is performed on the current block, a first classifier is used, and the first classifier determines the kernel based on the reconstructed samples from neighboring blocks of the current block.

[0125] (A7) In some embodiments of A6, the classifier determines the transformation kernel of the current block based on the template cost associated with the reconstructed sample. For example, at least one classifier is constructed based on the template cost.

[0126] (A8) In some embodiments of any of A1 to A7, the encoded information includes at least one sample from a predicted block of a reference image. For example, the information of the encoded block may be a sample from a predicted block of a reference image. In some embodiments, when performing inter-frame prediction on the current block, a first classifier is used, and the first classifier determines the kernel based on the samples of the predicted block.

[0127] (A9) In some embodiments of any of A1 to A8, the encoded information includes at least one neighboring sample from the predicted block of the reference image. For example, the information of the encoded block may be neighboring samples of the predicted block (e.g., a template of the predicted block). The neighboring samples of the predicted block may come from the upper, left, right and / or lower regions of the reference image.

[0128] (A10) In some embodiments of any of A1 through A9, the method further includes selecting a classifier from a set of classifiers based on histogram of gradient (HoG) information of the current block. For example, at least one classifier is derived based on a histogram of gradient values ​​of information from the encoded block.

[0129] (A11) In some embodiments of A10, a classifier is identified from the classifier set when it has the highest distribution in the histogram. For example, N associated classifiers with the highest distribution in the histogram can be selected to derive the corresponding transformation kernel, where N is a non-zero positive value. In some embodiments, the classifier is selected based on the determination that it has the highest histogram distribution compared to other classifiers in the classifier set. In some embodiments, a classifier is selected when it has the lowest distribution in the histogram. For example, N associated classifiers with the lowest distribution in the histogram can be used.

[0130] (A12) In some embodiments of A11, a subset of classifiers is selected from the set of classifiers based on the highest distribution in the histogram, wherein pruning techniques are used to identify classifiers from the subset. For example, M associated classifiers with the highest distribution in the histogram are selected, and a pruning method (e.g., a mapping method) is applied to these M classifiers to determine N classifiers, wherein corresponding transformation kernels are derived from these N classifiers. For example, a first subset of M is mapped to a first classifier, a second subset of M is mapped to a second classifier, and so on. In some embodiments, a subset of classifiers is selected from the set of classifiers based on the lowest distribution in the histogram. For example, the M lowest distributions in the histogram may be used to determine one or more classifiers.

[0131] (A13) In some embodiments of any of A1 to A12, the method further includes selecting a classifier from a set of classifiers, and the set of classifiers includes two or more classifiers. For example, at least two classifiers may be used to select a transform kernel. In some embodiments, the set of classifiers includes three or more classifiers. For example, the set of classifiers may include a first classifier for inter-frame prediction blocks, a second classifier for regular intra-frame prediction blocks, and a third classifier for non-regular intra-frame prediction blocks.

[0132] (A14) In some embodiments of A13, the two or more classifiers include a first classifier configured for inter-frame prediction blocks and a second classifier configured for intra-frame prediction blocks. For example, two different classifiers may be used for intra-frame prediction blocks and inter-frame prediction blocks, respectively. In some embodiments, the transform kernel for intra-frame prediction blocks is selected from a set of intra-frame transform kernels (e.g., a set of 35 intra-frame transform kernels), while the transform kernel for inter-frame prediction blocks is selected from a set of inter-frame transform kernels (e.g., a different set of 35 inter-frame transform kernels).

[0133] (A15) In some embodiments of A13 or A14, each of the two or more classifiers has the same number of categories. For example, all classifiers may have the same number of categories (e.g., 35 categories). In some embodiments, the two or more classifiers have different numbers of categories. For example, all classifiers may have any number of categories.

[0134] (A16) In some embodiments of any of A1 to A15, a classifier is selected from a set of classifiers based on the HoG information of the current block and the magnitude of the current block. For example, at least one classifier (e.g., similar to the ALF classifier used in VVC) can be derived based on a combination of the magnitude (e.g., the absolute value of residual data or transform coefficients) and the gradient value of the information of the encoded block.

[0135] (A17) In some embodiments of any of A1 to A16, the classifier has multiple categories, and each category in the multiple categories corresponds to a corresponding transform kernel in the set of available transform kernels. For example, each category in each classifier has one or more corresponding primary transform kernels or non-primary transform kernels.

[0136] (A18) In some embodiments of any of A1 through A17, the method further includes selecting a classifier based on an indicator indicated by signaling in the video bitstream. In some embodiments, the method includes parsing the indicator to determine the classifier. For example, at least one classifier may be indicated by signaling at different levels in the bitstream, such as image level, sub-image level, slice level, and / or block level. In some embodiments, the indicator indicates that a default classifier will be used (e.g., a classifier identifier not indicated by signaling). In some embodiments, the classifier is selected based on encoded information available at the decoding component. For example, the classifier may be derived based on a prediction pattern (e.g., inter-frame or intra-frame).

[0137] (B1) In another aspect, some embodiments include a video encoding method (e.g., method 650). In some embodiments, the method is performed at a computing system (e.g., server system 112) having memory and control circuitry. In some embodiments, the method is performed at an encoding / decoding module (e.g., encoding / decoding module 320). The method includes: (i) receiving video data (e.g., a source video sequence), the video data comprising a plurality of blocks (e.g., a set corresponding to at least one image), the plurality of blocks including a current block; (ii) applying a classifier to encoded information of the current block to determine a transform kernel for the current block; and (iii) encoding the current block by performing a transform on the current block by applying the transform kernel. In some embodiments, the classifier is indicated via signaling through the video bitstream. In various embodiments, any of the techniques described herein may be used to identify / select the classifier.

[0138] (B2) In some embodiments of B1, the encoded information includes information about the prediction pattern of the current block, the residual blocks of the current block, and / or the neighboring blocks of the current block.

[0139] (C1) In another aspect, some embodiments include a visual media data processing method. In some embodiments, the method is performed at a computing system (e.g., server system 112) having memory and control circuitry. In some embodiments, the method is performed at an encoding / decoding module (e.g., encoding / decoding module 320). The method includes: (i) obtaining a source video sequence comprising multiple frames; and (ii) performing a conversion between the source video sequence and a video stream of visual media data according to format rules. The video stream comprises multiple encoded blocks, the multiple encoded blocks including a current block. The format rules specify: (a) applying a classifier to the encoded information of the current block to determine a transform kernel for the current block; and (b) reconstructing the current block by performing an inverse transform on the current block by applying the transform kernel.

[0140] On the other hand, some embodiments include a computing system (e.g., server system 112) that includes control circuitry (e.g., control circuitry 302) and a memory (e.g., memory 314) coupled to the control circuitry, the memory storing at least one set of instructions configured to be executed by the control circuitry, the at least one set of instructions including instructions for performing any of the methods described in this application (e.g., A1 to A18, B1 to B2 and C1 above).

[0141] In another aspect, some embodiments include a non-transitory computer-readable storage medium storing at least one set of instructions for execution by control circuitry of a computing system, the at least one set of instructions including instructions for performing any of the methods described in this application (e.g., A1 to A18, B1 to B2 and C1 above).

[0142] Unless otherwise stated, any syntax element described in this application can be a high-level syntax (HLS). As used in this application, HLS is indicated by signaling at a level higher than the block level. For example, HLS can correspond to the sequence level, frame level, slice level, or tile level. As another example, HLS elements can be indicated by signaling in a video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), adaptation parameter set (APS), slice header, picture header, tile header, and / or coding tree unit header.

[0143] It should be understood that although the terms “first,” “second,” etc., may be used to describe various elements in this application, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to limit the claims. As used in the description of the embodiments and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and covers any and all possible combinations of at least one associated listed item. It should also be further understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of at least one other feature, integer, step, operation, element, component, and / or combination thereof.

[0144] As used in this application, the term "when" may be interpreted, depending on the context, as meaning "if" the stated prerequisite is true, or "when" the stated prerequisite is true, or "in response to determination" the stated prerequisite is true, or "according to determination" the stated prerequisite is true, or "in response to detection" the stated prerequisite is true. Similarly, the phrases "if it is determined [the stated prerequisite is true]" or "if [the stated prerequisite is true]" or "when [the stated prerequisite is true]" may be interpreted, depending on the context, as meaning "when" the stated prerequisite is true, or "in response to determination" the stated prerequisite is true, or "according to determination" the stated prerequisite is true, or "when" the stated prerequisite is true, or "in response to detection" the stated prerequisite is true. As used in this application, N refers to a variable value. Unless explicitly stated otherwise, different instances of N can refer to the same value (e.g., the same integer value, such as the value 2) or different values.

[0145] For illustrative purposes, the foregoing description has been illustrated with reference to specific embodiments. However, the exemplary arguments presented above are not intended to be exhaustive, nor are they intended to limit the claims to the precise forms disclosed. Many modifications and variations are possible in light of the foregoing teachings. These embodiments were chosen and described to best explain the principles of operation and practical application, thereby enabling those skilled in the art to implement them.

Claims

1. A video decoding method executed at a computing system, said computing system having a memory and at least one processor, characterized in that, The method includes: Receive a video stream, the video stream comprising multiple blocks, the multiple blocks including the current block; The classifier is applied to the encoded information of the current block to determine the transform kernel of the current block; and The current block is reconstructed by applying the transformation kernel to perform an inverse transformation on the current block.

2. The method according to claim 1, characterized in that, The transform kernel is the master transform kernel.

3. The method according to claim 1, characterized in that, The encoded information includes information about the residual blocks of the current block.

4. The method according to claim 3, characterized in that, The classifier uses at least one coefficient within a predefined region of the residual block to determine the transform kernel.

5. The method according to claim 1, characterized in that, The encoded information includes at least one residual sample from the adjacent blocks of the current block.

6. The method according to claim 1, characterized in that, The encoded information includes at least one reconstructed sample from the neighboring blocks of the current block.

7. The method according to claim 6, characterized in that, The classifier determines the transformation kernel of the current block based on the template cost associated with the reconstructed sample.

8. The method according to claim 1, characterized in that, The encoded information includes at least one sample from a prediction block of a reference image.

9. The method according to claim 1, characterized in that, The encoded information includes at least one neighboring sample from the prediction block of the reference image.

10. The method according to claim 1, characterized in that, It also includes selecting the classifier from a set of classifiers.

11. The method according to claim 10, characterized in that, The classifier is identified from the set of classifiers when it has the highest distribution in the histogram.

12. The method according to claim 11, characterized in that, A subset of classifiers is selected from the set of classifiers based on the highest distribution in the histogram, and a pruning technique is used to identify the classifiers from the subset of classifiers.

13. The method according to claim 10, characterized in that, The classifier set includes two or more classifiers.

14. The method according to claim 13, characterized in that, The two or more classifiers include a first classifier configured for inter-frame prediction blocks and a second classifier configured for intra-frame prediction blocks.

15. The method according to claim 13, characterized in that, Each of the two or more classifiers has the same number of categories.

16. The method according to claim 10, characterized in that, The classifier is selected from the classifier set based on the HoG information of the current block and the magnitude of the current block.

17. The method according to claim 1, characterized in that, The classifier has multiple categories, and each of the multiple categories corresponds to a corresponding transform kernel in the set of available transform kernels.

18. The method according to claim 1, characterized in that, It also includes selecting the classifier based on an indicator indicated by signaling in the video bitstream.

19. A video encoding method executed at a computing system having a memory and at least one processor, characterized in that, The method includes: Receive video data, the video data comprising multiple blocks, the multiple blocks including the current block; The classifier is applied to the encoded information of the current block to determine the transform kernel of the current block; and The current block is encoded by applying the transformation kernel to perform a transformation on the current block.

20. A non-transitory computer-readable storage medium storing a video stream generated by a video coding method, characterized in that, The video encoding method includes: Receive video data, the video data comprising multiple blocks, the multiple blocks including the current block; The classifier is applied to the encoded information of the current block to determine the transform kernel of the current block; and The current block is encoded by applying the transformation kernel to perform a transformation on the current block.

Images