Method and system for signaling and performing secondary transforms

Abstract

Methods and systems for encoding and decoding video are described. In one aspect, a method of video decoding includes receiving video data from a video bitstream, the video data including a first block and a first syntax element. The method further includes determining a quadratic transform kernel type value for the first block based on the first syntax element. In accordance with a determination that the secondary transform kernel type has the first value, a secondary transform set identifier is determined based on a second syntax element from the video bitstream, and the first block is inversely transformed using the determined secondary transform kernel type and the determined secondary transform set identifier. In accordance with a determination that the sub-transform kernel type has the second value, the first block is not inversely quadratic transformed.

Description

[0001] Related applications This application claims priority to U.S. Provisional Patent Application No. 63 / 530,461, entitled “Signaling Method for Transform,” filed August 2, 2023, and is a continuation to U.S. Patent Application No. 18 / 520,259, entitled “Methods and Systems for Signaling and Performing Secondary Transforms,” filed November 27, 2023, and claims priority to the earlier application. Technical Field

[0002] The disclosed embodiments generally relate to image and video encoding and compression, including but not limited to systems and methods for signal transmission and performing secondary transformations. Background Technology

[0003] Digital video is supported by various electronic devices, such as digital televisions, laptops or desktop computers, tablets, digital cameras, digital video recording devices, digital media players, video game consoles, smartphones, video conferencing equipment, video streaming devices, and so on. These electronic devices transmit and receive, or otherwise communicate, digital video data via communication networks, and / or store digital video data on storage devices. Due to the limited bandwidth capacity of communication networks and the limited memory resources of storage devices, video data can be compressed using video coding according to one or more video coding standards before transmission or storage. Video coding can be performed by hardware and / or software on electronic / client devices or servers providing cloud services.

[0004] Video coding typically utilizes prediction methods (e.g., inter-frame prediction, intra-frame prediction, etc.) that take advantage of the inherent redundancy in video data. Video coding aims to compress video data into a form using a lower bit rate while avoiding or minimizing video quality loss. Several video codec standards have been developed. For example, High Efficiency Video Coding (HEVC / H.265) is a video compression standard designed as part of the MPEG-H project. The International Telecommunication Union Telecommunication Standardization Sector (ITU-T) and the International Organization for Standardization / International Electrotechnical Commission (ISO / IEC) released the HEVC / H.265 standard in 2013 (Revision 1), 2014 (Revision 2), 2015 (Revision 3), and 2016 (Revision 4), respectively. Next Generation Video Coding (VVC / H.266) is a video compression standard designed to succeed HEVC. The ITU-T and ISO / IEC released the VVC / H.266 standard in 2020 (Revision 1) and 2022 (Revision 2), respectively. The Alliance for Open Media Video 1 (AV1) is an open video coding format designed as an alternative to HEVC. On January 8, 2019, version 1.0.0 of the specification, with empirical verification and errata 1, was released. Summary of the Invention

[0005] Video codecs typically process pixel blocks. For example, each pixel block is processed using a prediction-transform coding scheme, where predictions are derived from intra-frame reference pixels, inter-frame motion compensation, or some combination of both. The residuals from the predictions can be subjected to a two-dimensional transform to remove spatial correlations, and then the transform coefficients can be quantized. The prediction syntax elements and the quantized transform coefficient indices can then be entropy-coded using arithmetic coding. Some video codecs have begun to use directional or inseparable primary and secondary image transforms to further improve compression efficiency (e.g., because correlations still exist between transform coefficients after the primary transform).

[0006] In some methods, the parameters of the secondary transform are implicitly derived (e.g., based on the parameters of the primary transform). However, signaling at least a subset of the secondary transform parameters may be beneficial because it allows the application of the most accurate and / or precise secondary transform, thereby improving coding efficiency. Signaling the secondary transform parameters may increase coding overhead. However, as described in this disclosure, the coding overhead can be reduced / minimized (e.g., by first signaling the kernel type, and if the kernel type signaling indicates that the secondary transform is not used, then the transform set ID is not signaled).

[0007] According to some embodiments, a video coding method is provided. The method includes (i) receiving video data comprising a plurality of blocks, the plurality of blocks including a first block; (ii) determining a quadratic transform kernel type value for the first block; (iii) transmitting a first syntax element in the video bitstream, the first syntax element indicating the quadratic transform kernel type value; (iv) based on the determination that the quadratic transform kernel type has a first value: (a) determining a quadratic transform set identifier; (b) transmitting a second syntax element in the video bitstream, the second syntax element indicating the quadratic transform set identifier; and (c) performing a quadratic transform on the first block using the determined quadratic transform kernel type and the determined quadratic transform set identifier; and (v) based on the determination that the quadratic transform kernel type has a second value: (a) not transmitting the second syntax element; and (b) not performing a quadratic transform on the first block.

[0008] According to some embodiments, a video decoding method is provided. The method includes (i) receiving video data from a video bitstream, the video data including a plurality of blocks and a first syntax element, the plurality of blocks including a first block; (ii) determining a quadratic transform kernel type value for the first block based on the first syntax element; (iii) determining a quadratic transform set identifier based on the determined quadratic transform kernel type having a first value: (a) determining a quadratic transform set identifier based on a second syntax element from the video bitstream; and (b) performing an inverse quadratic transform on the first block using the determined quadratic transform kernel type and the determined quadratic transform set identifier, the inverse quadratic transform corresponding to a quadratic transform performed during the encoding of the video data; and (iv) not performing an inverse quadratic transform on the first block based on the determined quadratic transform kernel type having a second value.

[0009] According to some embodiments, a computing system, such as a streaming system, server system, personal computer system, or other electronic device, is provided. The computing system includes control circuitry and memory storing one or more instruction sets. The one or more instruction sets include instructions for performing any of the methods described herein. In some embodiments, the computing system includes an encoder component and a decoder component (e.g., a transcoder).

[0010] According to some embodiments, a non-transitory computer-readable storage medium is provided. This non-transitory computer-readable storage medium stores one or more instruction sets executable by a computing system. The one or more instruction sets include instructions for performing any of the methods described herein.

[0011] Therefore, this application discloses methods, apparatus, and systems for encoding and decoding video. Such methods, apparatus, and systems can supplement or replace existing methods, apparatus, and systems for encoding / decoding video. The features and advantages described in the specification are not exhaustive; in particular, some additional features and advantages will be apparent to those skilled in the art from the accompanying drawings, specification, and claims provided in this disclosure. Furthermore, it should be noted that the wording used in the specification is primarily for readability and illustrative purposes, and the choice of terminology is not intended to define or limit the subject matter described herein. Attached Figure Description

[0012] To gain a more detailed understanding of this disclosure, reference can be made to the features of various embodiments, some of which are illustrated in the accompanying drawings. However, the drawings only illustrate relevant features of this disclosure and should not be considered limiting, as those skilled in the art will understand upon reading this disclosure that other effective features may also be included in the specification.

[0013] Figure 1 This is a block diagram illustrating an exemplary communication system according to some embodiments.

[0014] Figure 2A This is a block diagram illustrating exemplary elements of an encoder component according to some embodiments.

[0015] Figure 2B This is a block diagram illustrating exemplary elements of a decoder component according to some embodiments.

[0016] Figure 3 This is a block diagram illustrating an exemplary server system according to some embodiments.

[0017] Figure 4 This is a block diagram illustrating an exemplary encoding and decoding process according to some embodiments.

[0018] Figure 5A This is a flowchart illustrating an exemplary video encoding method according to some embodiments.

[0019] Figure 5B This is a flowchart illustrating an exemplary video decoding method according to some embodiments.

[0020] Figure 6 This is a flowchart illustrating an exemplary encoding process according to some embodiments.

[0021] By convention, the various features shown in the accompanying drawings are not necessarily drawn to scale, and the same reference numerals may be used to represent the same features in the description and the accompanying drawings. Detailed Implementation

[0022] This disclosure describes systems and methods for signaling primary and secondary transforms. In some methods, the parameters of the secondary transform are implicitly derived (e.g., based on the coding context, such as the parameters of the primary transform). As described in more detail below, in some embodiments, writing secondary transform information (e.g., in the video bitstream) can improve coding efficiency (e.g., allow the use of the most accurate and precise secondary transform). Furthermore, some embodiments described herein reduce (e.g., minimize) coding overhead by writing the secondary transform kernel type before writing the transform set identifier. For example, in response to a secondary transform kernel type indication that a secondary transform is not used, the system does not signal the transform set identifier.

[0023] Exemplary systems and devices Figure 1 This is a block diagram illustrating a communication system 100 according to some embodiments. The communication system 100 includes a source device 102 and a plurality of electronic devices 120 (e.g., electronic devices 120-1 to 120-m), which are communicatively interconnected via one or more networks. In some embodiments, the communication system 100 is a streaming system, for example for applications capable of supporting video, such as video conferencing applications, digital television applications, and media storage and / or distribution applications.

[0024] Source device 102 includes a video source 104 (e.g., a camera component or media storage) and an encoder component 106. In some embodiments, the video source 104 is a digital camera (e.g., configured to create an uncompressed video sample stream). The encoder component 106 generates one or more encoded video streams from the video stream. The video stream from video source 104 may have a higher data volume compared to the encoded video stream 108 generated by encoder component 106. The encoded video stream 108 has a lower data volume (less data) compared to the video stream from video source 104, and therefore requires less bandwidth for transmission and less storage space for storage compared to the video stream from video source 104. In some embodiments, source device 102 does not include encoder component 106 (e.g., configured to transmit uncompressed video to one (or more) networks 110).

[0025] One or more networks 110 represent any number of networks that transmit information between source device 102, server system 112, and / or electronic device 120, including, for example, wired (connected) and / or wireless communication networks. One or more networks 110 may exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and / or the Internet.

[0026] One or more networks 110 include a server system 112 (e.g., a distributed / cloud computing system). In some embodiments, the server system 112 is a streaming server or includes a streaming server (e.g., configured to store and / or distribute video content, such as an encoded video stream from source device 102). The server system 112 includes an encoder component 114 (e.g., configured to encode and / or decode video data). In some embodiments, the encoder component 114 includes an encoder component and / or a decoder component. In various embodiments, the encoder component 114 is implemented in hardware, software, or a combination of hardware and software. In some embodiments, the encoder component 114 is configured to decode an encoded video stream 108 and re-encode the video data using different encoding standards and / or methods to generate encoded video data 116. In some embodiments, the server system 112 is configured to generate multiple video formats and / or encodings based on the encoded video stream 108. In some embodiments, the server system 112 operates as a Media-Aware Network Element (MANE). For example, server system 112 can be configured to trim encoded video stream 108 to customize potentially different streams for one or more electronic devices 120. In some embodiments, the MANE and server system 112 are provided separately.

[0027] Electronic device 120-1 includes a decoder component 122 and a display 124. In some embodiments, the decoder component 122 is configured to decode encoded video data 116 to generate an output video stream that can be displayed on a display or other type of presentation device. In some embodiments, one or more electronic devices in electronic device 120 do not include a display component (e.g., communicatively coupled to and / or including media storage with an external display device). In some embodiments, electronic device 120 is a streaming client. In some embodiments, electronic device 120 is configured to access server system 112 to obtain encoded video data 116.

[0028] The source device and / or multiple electronic devices 120 are sometimes referred to as “terminal devices” or “user devices”. In some embodiments, the source device 102 and / or one or more electronic devices 120 are instances of server systems, personal computers, portable devices (e.g., smartphones, tablets, or laptops), wearable devices, video conferencing equipment, and / or other types of electronic devices.

[0029] In an example operation of communication system 100, source device 102 transmits an encoded video stream 108 to server system 112. For example, source device 102 may encode an image stream acquired by the source device. Server system 112 receives the encoded video stream 108 and may decode and / or encode the encoded video stream 108 using encoder component 114. For example, server system 112 may apply an encoding method more suitable for network transmission and / or storage to encode the video data. Server system 112 may transmit encoded video data 116 (e.g., one or more encoded video streams) to one or more electronic devices 120. Each electronic device 120 may decode the encoded video data 116 and optionally display video images.

[0030] Figure 2A This is a block diagram of exemplary elements of encoder component 106 shown according to some embodiments. Encoder component 106 receives a source video sequence from video source 104. In some embodiments, the encoder component includes a receiver (e.g., transceiver) component configured to receive the source video sequence. In some embodiments, encoder component 106 receives a video sequence from a remote video source (e.g., a video source component from a device different from encoder component 106). Video source 104 can provide a source video sequence in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8-bit, 10-bit, or 12-bit), any color space (e.g., BT.601 YCrCB or RGB), and any suitable sampling structure (e.g., YCrCb 4:2:0 or YCrCb 4:4:4). In some embodiments, video source 104 is a storage device storing previously acquired / prepared video. In some embodiments, video source 104 is a camera that acquires local image information as a video sequence. Video data can be provided as multiple individual pictures, which will be given motion when viewed sequentially. An image itself can be constructed as a spatial pixel array, where each pixel may include one or more samples, depending on the sampling structure, color space, etc. Those skilled in the art can easily understand the relationship between pixels and samples. The following focuses on describing samples.

[0031] Encoder component 106 is configured to encode and / or compress images of a source video sequence into an encoded video sequence 216 in real time or under other time constraints required by the application. Implementing an appropriate encoding rate is a function of controller 204. In some embodiments, controller 204 controls and is functionally coupled to other functional units described below. Parameters set by controller 204 may include rate control-related parameters (e.g., λ values ​​for image skipping, quantizers, and / or rate-distortion optimization techniques), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. Other functions of controller 204 will be readily identifiable to those skilled in the art, as these functions may relate to encoder component 106 optimized for a particular system design.

[0032] In some embodiments, encoder component 106 is configured to operate within an encoding loop. In a simplified example, the encoding loop includes a source encoder 202 (e.g., responsible for creating symbols, such as a symbol stream, based on the input image to be encoded and a reference image) and a (local) decoder 210. Decoder 210 reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data (when compression between the symbols and the encoded video stream is lossless). The reconstructed sample stream (sample data) is input to reference image memory 208. Since decoding of the symbol stream produces bit-accurate results regardless of the decoder's location (local or remote), the contents of reference image memory 208 are also bit-accurately corresponding between the local encoder and the remote encoder. In this way, the reference image samples interpreted by the encoder's prediction portion are exactly the same as the sample values ​​that the decoder will interpret when using the prediction during decoding. This basic principle of reference image synchronization (and the drift that occurs when synchronization cannot be maintained, for example, due to channel errors) is known to those skilled in the art.

[0033] The operation of decoder 210 can be combined with, for example, the following: Figure 2B The remote decoder is the same as that of decoder component 122, described in detail. However, a brief reference is provided. Figure 2B When symbols are available and the entropy encoder 214 and the parser 254 are able to encode / decode the symbols into an encoded video sequence without loss, the entropy decoding portion of the decoder component 122, including the buffer memory 252 and the parser 254, may not be fully implemented in the local decoder 210.

[0034] The decoder techniques described herein (apart from parsing / entropy decoding) can exist in essentially the same functional form in their corresponding encoders. Therefore, the focus of this disclosure is on decoder operations. The description of encoder techniques can be simplified because encoder techniques are inverses of decoder techniques.

[0035] During its operation, the source encoder 202 can perform motion-compensated predictive coding. Referring to one or more previously encoded frames in the video sequence designated as reference frames, this motion-compensated predictive coding predictively encodes the input frame. In this way, the encoding engine 212 encodes the differences between pixel blocks in the input frame and pixel blocks in the reference frame, which can be selected as the prediction reference for the input frame. The controller 204 can manage the encoding operations of the source encoder 202, including, for example, setting parameters and subgroup parameters for encoding video data.

[0036] Decoder 210 decodes encoded video data of frames that can be designated as reference frames, based on symbols created by source encoder 202. Encoding engine 212 can operate as a lossy process. When encoded video data is processed by video decoder (… Figure 2A When decoded at (not shown), the reconstructed video sequence can be a copy of the source video sequence with some errors. Decoder 210 replicates the decoding process, which can be performed on the reference frame by a remote video decoder, and allows the reconstructed reference frame to be stored in reference image memory 208. In this way, encoder component 106 locally stores a copy of the reconstructed reference frame that shares common content (no transmission errors) with the reconstructed reference frame that will be obtained by the remote video decoder.

[0037] Predictor 206 can perform a prediction search against encoding engine 212. That is, for a new frame to be encoded, predictor 206 can search the reference image memory 208 for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that can serve as appropriate prediction references for the new image. Predictor 206 can operate pixel-by-pixel based on the sample blocks to find suitable prediction references. Based on the search results obtained by predictor 206, it can be determined that the input image may have prediction references obtained from multiple reference images stored in the reference image memory 208.

[0038] The outputs of all the aforementioned functional units can be entropy encoded in the entropy encoder 214. The entropy encoder 214 applies lossless compression to the symbols generated by the various functional units according to techniques known to those skilled in the art (e.g., Huffman coding, variable-length coding, and / or arithmetic coding), thereby converting the symbols into an encoded video sequence.

[0039] In some embodiments, the output of entropy encoder 214 is coupled to a transmitter. This transmitter can be configured to buffer the encoded video sequence created by entropy encoder 214 in preparation for transmission via communication channel 218, which may be a hardware / software link to a storage device storing the encoded video data. The transmitter can be configured to combine encoded video data from source encoder 202 with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown). In some embodiments, the transmitter may transmit additional data while transmitting the encoded video. Source encoder 202 may include such data as part of the encoded video sequence. Additional data may include temporal / spatial / signal noise ratio (SNR) enhancement layers, other forms of redundant data such as redundant pictures and slices, supplemental enhancement information (SEI) messages, fragments of visual usability information (VUI) parameter sets, etc.

[0040] Controller 204 manages the operation of encoder component 106. During encoding, controller 204 can assign a specific type of encoded picture to each encoded picture, which may affect the encoding techniques applicable to the corresponding picture. For example, a picture may be designated as an intra-picture (I-picture), a predictive picture (P-picture), or a bidirectional predictive picture (B-picture). Intra-pictures can be encoded and decoded without using any other frames in the sequence as prediction sources. Some video codecs allow different types of intra-pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Variations of these I-pictures and their respective uses and characteristics are well known to those skilled in the art and will not be described further here. Predictive pictures can be encoded and decoded using intra-prediction or inter-prediction, which uses at most one motion vector and reference index to predict the sample values ​​for each block. Bidirectional predictive pictures can be encoded and decoded using intra-prediction or inter-prediction, which uses at most two motion vectors and reference indices to predict the sample values ​​for each block. Similarly, multiple predictive images can be used with two or more reference images and associated metadata to reconstruct a single block.

[0041] Source images are typically spatially subdivided into multiple sample blocks (e.g., 4×4, 8×8, 4×8, or 16×16 sample blocks), and encoded block by block. These blocks can be predictively coded with reference to other (already coded) blocks, determined by the coding assignment of the corresponding images applied to the block. For example, a block of an I image can be nonpredictively coded, or it can be predictively coded (spatial prediction or intra-frame prediction) with reference to already coded blocks of the same image. A pixel block of a P image can be nonpredictively coded with reference to a previously coded reference image via spatial prediction or temporal prediction. A block of a B image can be nonpredictively coded with reference to one or two previously coded reference images via spatial prediction or temporal prediction.

[0042] The captured video can be presented as multiple source images (video images) in a time-series format. Intra-frame image prediction (often simply called intra-prediction) utilizes spatial correlations within a given image, while inter-frame image prediction utilizes (temporal or other) correlations between images. In one example, a specific image being encoded / decoded is segmented into blocks; this specific image being encoded / decoded is called the current image. When a block in the current image resembles a reference block in a previously encoded and still buffered reference image in the video, the block in the current image can be encoded using a vector called a motion vector. This motion vector points to the reference block in the reference image, and when using multiple reference images, the motion vector can have a third dimension that identifies the reference image.

[0043] Encoder component 106 can perform encoding operations according to a predetermined video coding technique or standard (such as any technique or standard described herein). During operation, encoder component 106 can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

[0044] Figure 2B This is a block diagram of exemplary elements of decoder component 122 shown according to some embodiments. Figure 2B The decoder component 122 is coupled to the channel 218 and the display 124. In some embodiments, the decoder component 122 includes a transmitter coupled to the loop filter 256 and configured to transmit data to the display 124 (e.g., via a wired or wireless connection).

[0045] In some embodiments, decoder component 122 includes a receiver coupled to channel 218 and configured to receive data from channel 218 (e.g., via a wired or wireless connection). The receiver may be configured to receive one or more encoded video sequences to be decoded by decoder component 122. In some embodiments, the decoding of each encoded video sequence is independent of the decoding of other encoded video sequences. Each encoded video sequence may be received from channel 218, which may be a hardware / software link to a storage device storing the encoded video data. The receiver may receive the encoded video data along with other data, such as encoded audio data and / or auxiliary data streams that may be forwarded to their respective user entities (not indicated). The receiver may separate the encoded video sequences from other data. In some embodiments, the receiver receives additional (redundant) data along with the encoded video. This additional data may be part of the encoded video sequence. This additional data may be used by decoder component 122 to decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0046] According to some embodiments, decoder component 122 includes buffer memory 252, parser 254 (sometimes referred to as entropy decoder), scaler / inverse transform unit 258, intra-frame image prediction unit 262, motion compensation prediction unit 260, aggregator 268, loop filter unit 256, reference image memory 266, and current image memory 264. In some embodiments, decoder component 122 is implemented as one or more integrated circuits and / or other electronic circuits. In some embodiments, decoder component 122 is at least partially implemented as software.

[0047] Buffer memory 252 is coupled between channel 218 and parser 254 (e.g., to prevent network jitter). In some embodiments, buffer memory 252 is separate from decoder component 122. In some embodiments, a separate buffer memory is provided between the output of channel 218 and decoder component 122. In some embodiments, in addition to configuring buffer memory 252 internally to decoder component 122 (e.g., configured to handle playback timing), a separate buffer memory is also provided externally to decoder component 122 (e.g., to prevent network jitter). Buffer memory 252 may not be necessary or may be made smaller when receiving data from a store / forward device with sufficient bandwidth and controllability or from an isochronous synchronization network. Of course, buffer memory 252 may also be needed for use on packet networks such as the Internet; this buffer memory may be relatively large and adaptive in size, and may be at least partially implemented in the operating system or similar components (not shown) external to decoder component 122.

[0048] Parser 254 is configured to reconstruct symbols 270 from the encoded video sequence. These symbols may include, for example, information for managing the operation of decoder component 122, and / or information for controlling presentation devices such as display 124. Control information for the presentation device may exist in the form of parameter set fragments (not indicated), such as SEI messages or Video Usability Information (VUI). Parser 254 parses (entropy decodes) the encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. Parser 254 may extract a subgroup parameter set from the encoded video sequence for at least one subgroup of pixels in the subgroups used in the video decoder, based on at least one parameter corresponding to a group. Subgroups may include group of pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), prediction units (PU), etc. The parser 254 can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0049] Depending on the type of encoded video frames or a subset of encoded video frames (e.g., inter-frame and intra-frame frames, inter-frame and intra-frame blocks) and other factors, the reconstruction of symbol 270 may involve multiple different units. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the encoded video sequence by parser 254. For clarity, the flow of such subgroup control information between parser 254 and the various units described below is not described.

[0050] The decoder component 122 can be conceptually subdivided into several functional units. In some implementations, these units interact closely with each other and can be at least partially integrated with each other. However, for clarity, the conceptual subdivision of the functional units is retained here.

[0051] The scaler / inverse transform unit 258 receives from the parser 254 the quantization transform coefficients as symbols 270, as well as control information (e.g., which transform mode to use, block size, quantization factor, and / or quantization scaling matrix). The scaler / inverse transform unit 258 can output a block including sample values, which can be input into the aggregator 268.

[0052] In some cases, the output samples of the scaler / inverse transform unit 258 belong to intra-coded blocks. Intra-coded blocks are blocks that do not use predictive information from previously reconstructed images, but can use predictive information from previously reconstructed portions of the current image. This predictive information can be provided by the intra-picture prediction unit 262. The intra-picture prediction unit 262 uses surrounding reconstructed information extracted from the current (partially reconstructed) image in the current picture memory 264 to generate blocks of the same size and shape as the block being reconstructed. The aggregator 268, based on each sample, can add the predictive information generated by the intra-picture prediction unit 262 to the output sample information provided by the scaler / inverse transform unit 258.

[0053] In other cases, the output samples of the scaler / inverse transform unit 258 belong to inter-frame coding and latent motion compensation blocks. In this case, the motion compensation prediction unit 260 can access the reference image memory 266 to extract samples for prediction. After motion compensation is performed on the extracted samples according to the block-related symbols 270, these samples can be added by the aggregator 268 to the output of the scaler / inverse transform unit 258 (referred to in this case as residual samples or residual signals) to generate output sample information. The motion compensation prediction unit 260 can obtain predicted samples from addresses within the reference image memory 266 under motion vector control. This motion vector is used by the motion compensation prediction unit 260 in the form of symbols 270, which, for example, have X, Y, and reference image components. Motion compensation may also include interpolation of sample values ​​extracted from the reference image memory 266 when using subsample precise motion vectors, motion vector prediction mechanisms, etc.

[0054] The output samples of aggregator 268 can be employed by various loop filtering techniques in loop filter unit 256. Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video bitstream, which are available as symbols 270 from parser 254 to loop filter unit 256. Video compression techniques may also respond to metadata obtained during decoding of a previous (in decoding order) portion of an encoded picture or encoded video sequence, and to previously reconstructed and loop-filtered sample values. The output of loop filter unit 256 can be a sample stream that can be output to a presentation device such as display 124 and stored in reference picture memory 266 for subsequent inter-frame picture prediction.

[0055] Once reconstructed, certain encoded images can be used as reference images for future predictions. Once an encoded image is reconstructed and is identified as a reference image (e.g., by parser 254), the current reference image can become part of the reference image memory 266, and a new current image memory can be reallocated before reconstructing subsequent encoded images begins.

[0056] Decoder component 122 can perform decoding operations according to a predetermined video compression technique documented in a standard (such as any standard described herein). An encoded video sequence may conform to the syntax specified by the video compression technique or standard used, in the sense that the encoded video sequence follows the syntax of the video compression technique or standard specified in the video compression technique document or standard, particularly the configuration document therein. To conform to some video compression techniques or standards, the complexity of the encoded video sequence is also required to be within the range defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum image size, maximum frame rate, maximum reconstruction sampling rate (measured, for example, megasamples per second), maximum reference image size, etc. In some cases, the limitations set by the hierarchy can be further limited by the Hypothetical Reference Decoder (HRD) specification and metadata managed by the HRD buffers that are signaled in the encoded video sequence.

[0057] Figure 3 This is a block diagram of a server system 112 illustrated according to some embodiments. The server system 112 includes control circuitry 302, one or more network interfaces 304, memory 314, a user interface 306, and one or more communication buses 312 for connecting these components. In some embodiments, the control circuitry 302 includes one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a digital processing unit (DPU)). In some embodiments, the control circuitry includes one or more field-programmable gate arrays (FPGAs), hardware accelerators, and / or integrated circuits (e.g., application-specific integrated circuits).

[0058] One or more network interfaces 304 can be configured to interface with one or more communication networks (e.g., wireless, wired, and / or optical networks). Communication networks can be local area networks, wide area networks, metropolitan area networks, vehicle and industrial networks, real-time networks, latency-tolerant networks, and so on. Examples of communication networks include local area networks (such as Ethernet), wireless LANs, cellular networks (including GSM, 3G, 4G, 5G, LTE, etc.), cable or wireless wide area digital networks (including cable television, satellite television, and terrestrial broadcast television), vehicle and industrial networks (including CANbus), and so on. This communication can be one-way receiving (e.g., broadcast television), one-way transmitting (e.g., CANbus connected to certain CANbus devices), or bidirectional, such as connecting to other computer systems using a local area network or wide area network. This communication may include communication with one or more cloud computing networks.

[0059] User interface 306 includes one or more output devices 308 and / or one or more input devices 310. The one or more input devices 310 may include one or more of the following: keyboard, mouse, touchpad, touchscreen, data glove, joystick, microphone, scanner, camera, etc. The one or more output devices 308 may include one or more of the following: audio output devices (e.g., speakers), visual output devices (e.g., displays or monitors), etc.

[0060] Memory 314 may include high-speed random access memory (e.g., dynamic random access memory (DRAM), static random access memory (SRAM), double data rate random access memory (DDR RAM), and / or other random access solid-state storage devices) and / or non-volatile memory (e.g., one or more disk storage devices, optical disk storage devices, flash memory devices, and / or other non-volatile solid-state storage devices). Memory 314 may optionally include one or more storage devices remote from control circuitry 302. Memory 314, or alternatively, the non-volatile solid-state storage devices within memory 314, include non-transitory computer-readable storage media. In some embodiments, memory 314 or the non-transitory computer-readable storage media of memory 314 stores programs, modules, instructions, and data structures, or subsets or supersets thereof: ● Operating system 316, which includes programs that handle various basic system services and perform hardware-dependent tasks; ● Network communication module 318, for connecting server system 112 to other computing devices via one or more network interfaces 304 (e.g., via wired and / or wireless connections); ● Encoding module 320 is used to perform various functions related to encoding and / or decoding data (e.g., video data). In some embodiments, encoding module 320 is an instance of encoder component 114. Encoding module 320 includes, but is not limited to, one or more of the following: ○ Decoding module 322 is used to perform various functions related to decoding encoded data, such as those functions mentioned above regarding decoder component 122; ○ Encoding module 340, used to perform various functions related to encoding data, such as those described above regarding encoder component 106; and ● Image memory 352 is used to store images and image data, for example, for use with encoding module 320. In some embodiments, image memory 352 includes one or more of the following: reference image memory 208, buffer memory 252, current image memory 264, and reference image memory 266.

[0061] In some embodiments, the decoding module 322 includes a parsing module 324 (e.g., configured to perform the various functions described above regarding the parser 254), a transformation module 326 (e.g., configured to perform the various functions described above regarding the scaler / inverse transformation unit 258), a prediction module 328 (e.g., configured to perform the various functions described above regarding the motion compensation prediction unit 260 and / or the intra-frame image prediction unit 262), and a filter module 330 (e.g., configured to perform the various functions described above regarding the loop filter 256).

[0062] In some embodiments, the encoding module 340 includes an encoding module 342 (e.g., configured to perform the various functions described above regarding the source encoder 202 and / or encoding engine 212) and a prediction module 344 (e.g., configured to perform the various functions described above regarding the predictor 206). In some embodiments, the decoding module 322 and / or the encoding module 340 includes... Figure 3 A subset of the modules shown. For example, the shared prediction module is shared by the decoding module 322 and the encoding module 340.

[0063] Each of the identified modules stored in memory 314 corresponds to a set of instructions that perform the functions described herein. These identified modules (e.g., instruction sets) do not need to be implemented as separate software programs, processes, or modules; therefore, various subsets of these modules can be combined or otherwise rearranged in various embodiments. For example, encoding module 320 may optionally not include separate decoding and encoding modules, but instead use the same set of modules to perform both types of functions. In some embodiments, memory 314 stores a subset of the identified modules and data structures. In some embodiments, memory 314 stores additional modules and data structures not described above, such as audio processing modules.

[0064] Although Figure 3 A server system 112 according to some embodiments is shown, but... Figure 3 This document focuses more on the functional description of various features that may exist in one or more server systems, rather than serving as structural diagrams of the embodiments described herein. In practical applications, as those skilled in the art will recognize, items shown individually can be combined, and some items can be separated. For example, Figure 3 Some items shown individually can be implemented on a single server, while a single item can be implemented on one or more servers. The actual number of servers used to implement server system 112 and how features are distributed among them will vary depending on the different implementations and optionally in part on the scale of data traffic handled by the server system during peak and average usage periods.

[0065] Exemplary encoding process and techniques As described above, video codecs typically incorporate a variety of techniques: intra-frame prediction / inter-frame prediction, image transformation, quantization, coefficient coding, entropy coding, and / or intra-loop filtering. In particular, decorrelation of the prediction error (e.g., residual) signal is beneficial. This disclosure describes techniques for transmitting signals and applying a secondary transform. The primary transform and / or secondary transform signaling can be implicit or explicit (e.g., explicitly represented by signals in the bitstream). The techniques described in this disclosure can be used individually or in any combination in any order. Furthermore, each technique can be implemented using processing circuitry (e.g., one or more processors, or one or more integrated circuits).

[0066] The quadratic transform can be applied between the main transform component and the quantization component, such as... Figure 4 As shown. For example, the forward quadratic transform is applied between the main forward transform and the quantization component, while the inverse quadratic transform is applied between the dequantization and the main inverse transform components.

[0067] Figure 4 This is a block diagram illustrating an exemplary encoding and decoding process according to some embodiments. Figure 4 As shown, the prediction error 402 (e.g., residual) is transformed and then quantized to generate transform coefficients. A master forward transform 404 (sometimes also called the master transform) is applied to the prediction error 402. According to some embodiments, a second forward transform 406 (sometimes also called the second transform) is applied to the output of the master forward transform 404. Quantization 408 is applied to the output of the second forward transform 406 to generate transform coefficients. In some embodiments, the transform coefficients are written into the video bitstream.

[0068] Dequantization 410 is applied to the transform coefficients (e.g., at the decoder). An inverse quadratic transform 412 is applied to the output of dequantization 410 (e.g., to reverse the effect of the quadratic forward transform 406). An inverse principal transform 414 is applied to the output of the inverse quadratic transform 412 (e.g., to reverse the effect of the principal forward transform 404). This yields the decoding prediction error 416 from the transform coefficients.

[0069] In this paper, the primary transform set and / or secondary transform set (sometimes referred to as "transform set" or "set") refers to a group of transform kernels. For example, during the encoding process, the optimal transform set can be determined for each transform block. If the encoder (or transcoder) determines that a secondary transform should be applied, the secondary transform set can be identified during the encoding process.

[0070] In this paper, the primary transform kernel type and / or secondary transform kernel type (sometimes referred to as "kernel type") refer to the options for the transform kernel type in each transform set. For example, a primary transform set and / or secondary transform set may include one or more primary transform kernel types and / or secondary transform kernel types. For example, a set and kernel hierarchy may include 7 sets, where each set has 4 kernel types. In this example, one of the kernel types can indicate that there is no primary transform and / or secondary transform. For example, among the 4 kernel type indices (0, 1, 2, and 3), the system can define kernel type index 0 as indicating that there is no primary transform and / or secondary transform, and therefore the transform set will not be signaled. The encoder component can select the transform set identifier (ID) and the kernel type identifier (ID).

[0071] In this paper, the set context index (sometimes referred to as the “set context”) refers to a (nominal) set context index that can be derived from and predefined (e.g., as part of normalization) from the coding context. The “coding context” may include information about previously coded modes (e.g., intra-frame prediction modes or inter-frame prediction modes), the main transform type, the coding block size, and / or the transform block size.

[0072] In this paper, "context_to_probable_set" refers to a predefined mapping between set context indices and probable transform set IDs (e.g., most probable transform set IDs). For example, for a given coding context, one or more most probable sets can be identified, where the coding context may be defined by a video standard. For example, as shown in Code Snippet 1 below, 12 exemplary intra-frame prediction modes are mapped to set IDs ranging from [0, ..., 6] (e.g., a mapping from 12 symbols to 7 symbols).

[0073] The following code snippet 1 illustrates an exemplary context_to_probable_set[i] whose index is derived from an exemplary context grouping (e.g., composed of exemplary intra-prediction modes). In the example of code snippet 1, two intra-prediction modes (INTRA_PRED_1 and INTRA_PRED_2) are in the same group, which maps to the same set ID 1.

[0074]

[0075]

[0076] Code Snippet 1 - Exemplary Mapping Table In code snippet 1, the index of the mapping table can be an intra-prediction ID [0, ..., 11] as defined and shown in code snippet 2 below. The range of this index can vary, for example, depending on the encoding context used by the context_to_probable_set mapping table. For example, code snippet 2 shows an exemplary encoding context from an exemplary intra-prediction mode. This encoding context can form the index of a mapping function or table (e.g., context_to_probable_set[i]) to deduce a set of contexts, which can determine the most probable set of transformations for each unique encoding context.

[0077]

[0078] Code Snippet 2 — Example Intra-Frame Prediction ID In some embodiments, the intra-secondary transform (IST) coding tool implicitly derives the IST set based on the context of the intra-prediction direction (e.g., explicitly representing the kernel type with a signal within the set). Table 1 below shows an exemplary mapping from intra-prediction modes and master transform types to (IST) indices.

[0079] Table 1—Exemplary mapping from intra-frame modes and main transform to IST

[0080] In some embodiments, a set of quadratic transforms is explicitly signaled. Explicitly signaling a set of quadratic transforms allows for the selection of any set of quadratic transforms for a block, regardless of the applied intra-prediction mode. Code snippet 3 below illustrates an exemplary transform type syntax.

[0081]

[0082] Code Snippet 3—Example Transformation Type Syntax In the example of code snippet 3, the `stx_set` parameter is inserted after the `stx_type` parameter (e.g., because when `stx_type` is set to a specific value (e.g., 0), there is no secondary transform, so there is no need to encode, signal, or parse `stx_set`). In some embodiments, there are 14 sets of secondary transforms. In some embodiments, these sets of secondary transforms depend on the primary transform type (e.g., 2 primary transform types). Therefore, in some embodiments, 7 different sets need to be signaled for each primary transform type. In some embodiments, a probability context is selected for each set. In some embodiments, the probability context is derived from the intra-frame prediction mode.

[0083] The following code snippet 4 illustrates an exemplary definition of a quadratic transformation.

[0084]

[0085] Code snippet 4—Exemplary definition of a quadratic transformation The code snippet 5 below shows an example code for reading a quadratic transformation type.

[0086]

[0087] Code snippet 5—Example function for reading quadratic transformations In some embodiments, the stx_set_ctx parameter is derived based on the intra-prediction mode of the decoded transform block. Code snippet 6 below illustrates another exemplary quadratic transform mapping array (e.g., a mapping from intra-mode to IST kernel set).

[0088]

[0089] Code Snippet 6—Example Quadratic Transformation Mapping Array In some embodiments, the secondary transformation is enabled only for a subset of intra-modes (e.g., intra-modes smaller than PEATH_PRED). In this case, the array size of stx_transpose_mapping is set to INTRA_MODES - 1.

[0090] The cumulative distribution function (CDF) can be used to represent the probability that a random variable might take a value less than or equal to a certain threshold. For example, the CDF of a video codec can represent the probability that the sign value is less than or equal to a given level multiplied by 32768. Code snippet 7 below shows an exemplary CDF with a size of 7 letters.

[0091]

[0092] Code snippet 7—Example CDF In code snippet 7, IST_DIR_SIZE can be defined as 7, and the size of the CDF table is 112 bytes (e.g., 7 contexts * 8 (7 symbols + 1 counter) * 2 (bytes per number)). For example, as defined by #define IST_DIR_SIZE 7 in code snippet 4 above, the number of stx_set_ctx contexts can be 7. In some embodiments, the CDF probability model is set such that the most likely set of symbols has the highest possible range size. For example, for a CDF model with a maximum range of 2^15, the highest possible range size is 2^15 - 6 * 4, where 6 is the number of symbols in the lower likely set and 4 is the minimum CDF range. In some embodiments, the minimum range is defined as (#define EC_MIN_PROB 4). In this way, the probability represented by the CDF model is (2^15 - (6 * 4)) / 2^15, approximately 0.9994.

[0093] The code snippet 8 below illustrates an exemplary configuration for a quadratic transformation.

[0094]

[0095] Code snippet 8—Exemplary quadratic transformation configuration Figure 5A This is a flowchart illustrating a video encoding method 500 according to some embodiments. Method 500 can be executed on a computing system (e.g., server system 112, source device 102, or electronic device 120) having control circuitry and memory storing instructions executed by the control circuitry. In some embodiments, method 500 is executed by executing instructions stored in memory (e.g., memory 314) in the computing system.

[0096] The system (e.g., from video source 104) receives (502) video data comprising multiple blocks, including a first block. The system determines (504) a quadratic transform kernel type value for the first block. The system transmits (506) a first syntax element in the video stream, the first syntax element indicating the quadratic transform kernel type value. In some embodiments, the system performs a master transform (e.g., master forward transform 404) on the first block.

[0097] Based on the determination of a first value (508) for the secondary transform kernel type: the system determines (510) a secondary transform set identifier; transmits (512) a second syntax element in the video stream indicating the secondary transform set identifier; and performs a secondary transform (e.g., a secondary forward transform 406) on the first block using the determined secondary transform kernel type and the determined secondary transform set identifier. For example, the system performs a secondary transform on the output of the main transform applied to the first block.

[0098] According to the determination of the second value (516) of the second transformation kernel type: the system does not (518) transmit the second syntax element, and does not (520) perform a second transformation on the first block.

[0099] Figure 5B This is a flowchart illustrating a video decoding method 550 according to some embodiments. Method 550 can be executed on a computing system (e.g., server system 112, source device 102, or electronic device 120) having control circuitry and a memory storing instructions to be executed by the control circuitry. In some embodiments, method 550 is executed by executing instructions stored in a memory (e.g., memory 314) in the computing system.

[0100] The system receives (552) video data from a video stream (e.g., an encoded video sequence), comprising multiple blocks and a first syntax element (e.g., an indicator stx_flag), the multiple blocks including the first block. The system determines (554) the quadratic transform kernel type value of the first block based on the first syntax element.

[0101] Based on the determination of the second transform kernel type having a first value (556): the system determines (558) the second transform set identifier based on a second syntax element from the video bitstream (e.g., an instruction stx_set_flag), and performs (560) an inverse second transform (e.g., inverse second transform 412) on the first block using the determined second transform kernel type and the determined second transform set identifier, which corresponds to the second transform (e.g., second forward transform 406) performed during the encoding of the video data.

[0102] Based on the determination that the second value is obtained from the second transformation kernel type, the system does not (562) perform an inverse second transformation on the first block.

[0103] although Figure 5A and Figure 5BMultiple logical stages are shown in a specific order, but stages independent of the order can be reordered, and other stages can be combined or split. Some reorderings or other groupings not specifically mentioned will be obvious to those skilled in the art, and therefore the order and groupings presented herein are not exhaustive. Furthermore, it should be recognized that these stages can be implemented in hardware, firmware, software, or any combination thereof.

[0104] In some embodiments, the selected transform set and / or kernel type is signaled (e.g., within a set) by encoding the primary transform set ID and / or the secondary transform set ID and / or the secondary kernel type. For example, a decoding method includes determining a secondary transform kernel type value for a first block based on a first syntax element. In this example, in response to the secondary transform kernel type having a first value (e.g., non-zero), a secondary transform set identifier is determined based on a second syntax element in the video bitstream, and the first block is subjected to an inverse secondary transform using the determined secondary transform kernel type and the determined secondary transform set identifier.

[0105] In some embodiments, the kernel type is first represented by a signal, and then the transform set is represented by a signal. In some embodiments, the primary transform type includes at least one inseparable transform. In some embodiments, the secondary transform type includes at least one inseparable transform. In some embodiments, the transform set includes a mixture of primary transform-only and primary plus secondary transform combination types. In some embodiments, the kernel type of the primary transform and the kernel type of the secondary transform are represented together by a signal. In some embodiments, the kernel set of the primary transform and the kernel set of the secondary transform are represented together by a signal. In some embodiments, representing the kernel type by a signal also indicates whether the primary transform and / or secondary transform are used. In some embodiments, if the kernel type indicates that the primary transform and / or secondary transform are disabled, the transform set ID is not sent by a signal.

[0106] In some embodiments, a flag is first indicated by a signal to indicate whether a primary transform and / or secondary transform are applied. When the flag indicated by the signal has a value indicating the application of the primary transform and / or secondary transform, a kernel type is further indicated by a signal to indicate which transform kernel is used, and this kernel type does not include an option not to apply the primary transform and / or secondary transform. In some embodiments, the method of mapping a set context to a transform set ID is derived from any possible coding context. In some embodiments, the coding context includes any combination of previously encoded modes or combinations thereof, such as intra-frame prediction mode, primary transform type, primary transform set ID, secondary transform type, secondary transform set ID, code block size, or transform block size.

[0107] In some embodiments, predefined groups of encoded text are used to form indices in a mapping function or table `context_to_probable_set`. In some embodiments, the most probable set is derived from a predefined encoding context implemented in the mapping function or table. In some embodiments, specific primary transform sets and secondary transform kernel types are designed or selected from available transform kernels to achieve optimal results for a group of encoding contexts, such as optimal decorrelation or optimal video compression efficiency. In some embodiments, for each predefined group of encoding contexts, only one transform set or a subgroup of transform sets is applicable to the current block.

[0108] In some embodiments, the set ID is represented by a signal through arithmetic coding. In some embodiments, the default or initial probability model (e.g., CDF) for the arithmetic coding of the most likely quadratic transform set ID is set to a probability close to 100% or the highest permissible probability in the video coding system. In some embodiments, the default or initial probability model (e.g., CDF) for the arithmetic coding of the remaining less likely quadratic transform set IDs is set to a probability close to 0% or the lowest permissible probability in the video coding system. The near-0% probability is achieved by associating a minimum range value (e.g., 4) with a symbol representing one of these remaining less likely quadratic transform set IDs. In various embodiments, the default probability model (e.g., CDF) is adaptive or non-adaptive. In some embodiments, if the video coding system is able to select an adaptive or non-adaptive mode for the entropy coding of the set symbol, a non-adaptive mode is selected, for example, to encode the set symbol using the fewest possible number of bits.

[0109] In some embodiments, the signal representation of the set ID can be implemented using variable-length coding (also known as VLC) or Huffman coding. In some embodiments, the codeword length of the most likely transform set is the shortest possible length. In some embodiments, the codeword length of the less likely transform set is longer than that of the most likely set.

[0110] In some embodiments, a quadratic transformation is indicated by a signal starting with a binary flag (e.g., most_probable_flag) that tells the set whether it is the most probable set. For example, Figure 6 This demonstrates checking whether set (602) is the most likely set. (Example) Figure 6 As shown, if the flag is set to false (e.g., 0), the most probable symbol is not encoded (606). If the flag is set to true (e.g., 1), the most probable set is encoded (604). For example, the set is encoded using most_probable_flag when only one most probable transformation set is considered.

[0111] In some embodiments, when the most probable flag is true, the set of symbols that can be used for encoding is all the symbols in the set alphabet. In some embodiments, when the most probable flag is false, the set of symbols that can be used for encoding is all the symbols in the set alphabet minus the most probable set determined by the set context (e.g., a mapping function or the table context_to_probable_set).

[0112] In some embodiments, if the most likely flag is true and the set of symbols to be encoded is the most likely, then the set of symbols is encoded using a probability model that has the highest probability for the set of symbols. In some embodiments, if the most likely flag is false and the set of symbols to be encoded is less likely, then the set of symbols is encoded using a probability model that starts with a uniform default probability.

[0113] In some embodiments, for encoding a coded block or transform block, a set of one or more most likely transform sets is derived based on a predefined coding context (e.g., intra-prediction mode), and when the most likely flag is true, the most likely set index is further signaled to indicate which transform set to select. When the most likely flag is false, the remaining_set_index is further signaled to indicate which non-most likely transform set to select. In some embodiments, the most likely flag is context-coded using arithmetic coding.

[0114] In some embodiments, a list of quadratic transform sets or a list of principal transform sets is derived for each transform block. Previously parsed information can be used to sort the quadratic transform sets or principal transform sets in the list. In one example, the most likely quadratic transform set or principal transform set is located at the top of the list. The indices of the sets are represented by signals using bypass, Golomb, or arithmetic coding. Offsets can be further represented by signals using bypass, Golomb, or arithmetic coding to indicate the kernel type.

[0115] (A1) In one aspect, some embodiments include a method for video encoding (e.g., method 500). In some embodiments, the method is performed in a computing system (e.g., server system 112) having memory and control circuitry. In some embodiments, the method is performed in an encoding module (e.g., encoding module 320). In some embodiments, the method is performed in a source encoding component (e.g., source encoder 202), an encoding engine (e.g., encoding engine 212), and / or an entropy encoder (e.g., entropy encoder 214). The method includes: (i) receiving video data comprising a plurality of blocks, including a first block, from a video source 104; (ii) determining a quadratic transform kernel type value for the first block; (iii) transmitting a first syntax element in the video stream, the first syntax element indicating the quadratic transform kernel type value; (iv) based on determining that the quadratic transform kernel type has a first value (e.g., 1): (a) determining a quadratic transform set identifier; (b) transmitting a second syntax element in the video stream, the second syntax element indicating the quadratic transform set identifier; and (c) performing a quadratic transform on the first block using the determined quadratic transform kernel type and the determined quadratic transform set identifier; and (v) based on determining that the quadratic transform kernel type has a second value (e.g., 0): (a) not transmitting the second syntax element; and (b) not performing a quadratic transform on the first block (e.g., performing only the main transform).

[0116] (A2) In some embodiments of A1, the method further includes writing a first block into the video bitstream. For example, the transform coefficients of the quadratic transform are written into the video bitstream. As an example, the output of quantization 408 is represented by a signal.

[0117] (A3) In some embodiments of A1 or A2, the method further includes: (i) determining the main transform of the first block; and (ii) transmitting a third syntax element in the video stream that indicates the main transform (e.g., main forward transform 404).

[0118] (A4) In some embodiments of any of A1 to A3, a quadratic transformation is selected from the transformation set based on the determined quadratic transformation set identifier.

[0119] (A5) In some embodiments of any of A1 to A4, the method further includes representing the secondary transform kernel type and the main transform kernel type together with a signal.

[0120] (A6) In some embodiments of any of A1 to A5, the method further includes writing the kernel set of the secondary transform and the kernel set of the primary transform together into the video stream.

[0121] (A7) In some embodiments of any of A1 to A6, the method further includes writing a flag into the video bitstream that indicates whether a primary transform and / or secondary transform are applied.

[0122] (A8) In some embodiments of any of A1 to A7, the second syntax element is arithmetically encoded.

[0123] (B1) In another aspect, some embodiments include a method for video decoding (e.g., method 550). In some embodiments, the method is performed in a computing system (e.g., server system 112) having memory and control circuitry. In some embodiments, the method is performed in an encoding module (e.g., encoding module 320). In some embodiments, the method is performed in a parser (e.g., parser 254), a motion prediction component (e.g., motion compensation prediction unit 260), and / or an intra-frame prediction component (e.g., intra-frame picture prediction unit 262). The method includes: (i) receiving video data (e.g., an encoded video sequence) from a video stream (e.g., a video stream in A1) comprising multiple blocks and a first syntax element, the multiple blocks including a first block; (ii) determining a quadratic transform kernel type value for the first block based on the first syntax element; (iii) based on determining that the quadratic transform kernel type has a first value (e.g., 1): (a) determining a quadratic transform set identifier based on a second syntax element from the video stream; and (b) performing an inverse quadratic transform (e.g., inverse quadratic transform 412) on the first block using the determined quadratic transform kernel type and the determined quadratic transform set identifier; and (iv) not performing an inverse quadratic transform on the first block based on determining that the quadratic transform kernel type has a second value (e.g., 0). For example, the selection of transform sets and / or kernel types (in the set) is explicitly represented by signals by encoding the primary transform set ID and / or the secondary transform set ID and / or the secondary kernel type. In some embodiments, the kernel type is first represented by a signal, and then the transform set is represented by a signal. As an example, the kernel type can also be represented by a signal to indicate whether the primary transform and / or secondary transform are used. In some embodiments, the inverse quadratic transform corresponds to the quadratic transform performed during video data encoding.

[0124] (B2) In some embodiments of B1, a second syntax element is not included in the video bitstream, depending on the second value of the quadratic transform kernel type. For example, if the kernel type indicates that the primary transform and / or the quadratic transform is disabled, the corresponding transform set ID is not represented by a signal.

[0125] (B3) In some embodiments of B1 or B2, the method further includes determining the main transform (and / or the inverse main transform corresponding to the main transform) of the first block based on a third syntax element from the video bitstream.

[0126] (B4) In some embodiments of B3, the principal transform is a non-separable transform. For example, the principal transform type includes at least one non-separable transform.

[0127] (B5) In some embodiments of any of B1 to B4, the quadratic transformation is an inseparable transformation. For example, the type of quadratic transformation includes at least one inseparable transformation.

[0128] (B6) In some embodiments of any of B1 to B5, a quadratic transformation is selected from the transformation set based on the determined quadratic transformation set identifier.

[0129] (B7) In some embodiments of B6, the transform set includes one or more primary transform types and one or more primary-secondary transform types. For example, the transform set may include a mixture of primary transform types and primary-secondary transform combination types.

[0130] (B8) In some embodiments of any of B1 to B7, the secondary transform kernel type is written together with the primary transform kernel type in the video bitstream. For example, the primary transform kernel type and the secondary transform kernel type are represented together by a signal.

[0131] (B9) In some embodiments of any of B1 to B8, the method further includes determining the kernel set of the secondary transform based on information from the video bitstream.

[0132] (B10) In some embodiments of B9, the kernel set of the second-order transform is written together with the kernel set of the main transform in the video bitstream. For example, the kernel set of the main transform and the kernel set of the second-order transform are represented together by a signal.

[0133] (B11) In some embodiments of any of B1 to B10: (i) a flag is written in the video bitstream indicating whether a primary transform and / or secondary transform is applied; and (ii) a secondary transform kernel type is written via a first syntax element based on the written flag having a first value. For example, the flag is first represented by a signal to indicate whether a primary transform and / or secondary transform is applied. When the flag represented by a signal has a value indicating the application of a primary transform and / or secondary transform, the kernel type is further represented by a signal to indicate which transform kernel is used, and the kernel type does not include an option not to apply a primary transform and / or secondary transform.

[0134] (B12) In some embodiments of any of B1 to B11, a secondary transform set identifier is determined based on set context information. For example, the method of mapping set context to transform set ID can be derived from any possible encoding context.

[0135] (B13) In some embodiments of B12, the set context information includes one or more of the following: previously encoded modes, primary transform type, primary transform set identifier, secondary transform type, secondary transform set identifier, coding block size, and transform block size. For example, the coding context includes any combination of previously encoded modes or combinations thereof, such as intra-prediction mode, primary transform type, primary transform set ID, secondary transform type, secondary transform set ID, coding block size, or transform block size. In some embodiments, the most probable set is derived from the set context information. For example, the most probable set is derived from a predefined coding context that is time-limited in a mapping function or table.

[0136] (B14) In some embodiments of B12 or B13, groups of encoded text are predefined to form an index. For example, groups of encoded text are predefined to form an index for a mapping function or the table context_to_probable_set.

[0137] (B15) In some embodiments of any of B1 through B14, a set of transform kernel types for the secondary transform is determined based on the grouping of coding contexts. For example, a specific set of primary and secondary transform kernel types is designed or selected from already available transform kernels to achieve optimal results (e.g., optimal decorrelation or optimal video compression efficiency) for a given coding context group. In some embodiments, for each predefined coding context group, only one transform set or a subset of transform sets is applicable to the current block.

[0138] (B16) In some embodiments of any of B1 through B15, the second syntax element is arithmetic-coded. For example, the set ID represented by a signal is implemented using arithmetic coding. In some embodiments, the default or initial probability model (e.g., cumulative distribution function (CDF)) for the arithmetic coding of the most probable quadratic transform set ID is set to a probability close to 100% or the highest permissible probability in the video coding system. In some embodiments, the default or initial probability model (e.g., CDF) for the arithmetic coding of the remaining less probable quadratic transform set IDs is set to a probability close to 0% or the lowest permissible probability in the video coding system. For example, a probability close to 0% is achieved by representing a symbol that associates a minimum range value (e.g., 4) with one of these remaining less probable quadratic transform set IDs. In some embodiments, the default probability model (e.g., CDF) is adaptive or non-adaptive. In some embodiments, if the video coding system can select an adaptive or non-adaptive mode for entropy coding of the set symbol, the system is configured to select a non-adaptive mode to encode the set symbol using the fewest possible number of bits. In some embodiments, the second syntax element is Huffman coded. For example, the set ID can be represented by a signal using variable-length coding (VLC) or Huffman coding. In some embodiments, the codeword length of the most likely transform set is the shortest possible length. In some embodiments, the codeword length of the less likely transform set is longer than that of the most likely set.

[0139] (B17) In some embodiments of any of B1 through B16, the method further includes determining a most probable flag from the video bitstream, wherein the most probable flag indicates whether the set is the most probable. For example, a binary flag “most_probable_flag” is used to signal a quadratic transform, indicating whether the set is the most probable. For example, the set is encoded using most_probable_flag if only one set of most probable transforms is considered. In some embodiments, when the most probable flag is true, the set of symbols available for encoding is all the symbols in the set alphabet. In some embodiments, when the most probable flag is false, the set of symbols available for encoding is all the symbols in the set alphabet minus the most probable set determined by the set context (e.g., a mapping function or the table context_to_probable_set). In some embodiments, if the most probable flag is true and the set of symbols to be encoded is the most probable, the set of symbols is encoded using a probability model that has the highest probability for the set of symbols. In some embodiments, if the most probable flag is false and the set of symbols to be encoded is less probable, the set of symbols is encoded using a probability model that starts with a uniform default probability. In some embodiments, for encoding a coded block or transform block, a set of one or more most likely transform sets is derived based on a predefined coding context (e.g., intra-prediction mode), and when the most likely flag is true, the most likely set index is further signaled to indicate which transform set to select. When the most likely flag is false, the remaining_set_index is further signaled to indicate which non-most likely transform set to select. In some embodiments, the most likely flag is context-coded using arithmetic coding.

[0140] (B18) In some embodiments of any of B1 through B17, a list of quadratic transform sets is derived for each transform block. For example, a list of quadratic transform sets or a list of principal transform sets is derived for each transform block. Previously parsed information can be used to sort the quadratic transform sets or principal transform sets in the list. For example, the most likely quadratic transform set or principal transform set may be located at the top of the list. Using bypass, Golog, or arithmetic coding, the index of the set can be represented by a signal. Using bypass, Golog, or arithmetic coding, the offset can be further represented by a signal to indicate the kernel type.

[0141] On the other hand, some embodiments include a computing system (e.g., server system 112) including control circuitry (e.g., control circuitry 302) and a memory (e.g., memory 314) coupled to the control circuitry, the memory storing one or more instruction sets configured to be executed by the control circuitry, the one or more instruction sets including instructions for performing any of the methods described herein (e.g., A1 to A8 and B1 to B18 above). In yet another aspect, some embodiments include a non-transitory computer-readable storage medium storing one or more instruction sets executed by the control circuitry in the computing system, the one or more instruction sets including instructions for performing any of the methods described herein (e.g., A1 to A8 and B1 to B18 above).

[0142] It should be understood that although the terms “first,” “second,” etc., are used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the claims. In the description of embodiments and the appended claims, the singular forms “a,” “an,” and “described” are intended to include the plural forms unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “comprising” and / or “including” as used in this specification indicate the presence of the listed features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof.

[0143] In this document, the term "if" can be understood, depending on the context, as "when," "in the event of," "in response to determining," "according to determining," or "in response to detecting," indicating that the stated prerequisite is true. Similarly, the phrases "if it is determined [the stated prerequisite is true]," "if [the stated prerequisite is true]," or "when [the stated prerequisite is true]" can be understood, depending on the context, as "in the event of determining," "in response to determining," "according to determining," "in the event of detecting," or "in response to detecting," indicating that the stated prerequisite is true.

[0144] The above description has been provided for illustrative purposes with reference to specific embodiments. However, the foregoing illustrative discussion is not intended to be exhaustive, nor is it intended to limit the claims to the precise forms disclosed. Many modifications and variations are possible in light of the foregoing teachings. These embodiments were chosen and described in order to best explain the operating principles and practical applications, thereby enabling those skilled in the art to understand them.

Claims

1. A method for video decoding, the method being executed on a computing system having a memory and one or more processors, the method comprising: Receive video data from a video stream, the video data including multiple blocks and a first syntax element, the multiple blocks including the first block; Based on the first syntax element, determine the quadratic transformation kernel type of the first block; Based on the determination that the quadratic transform kernel type has a first value: Based on the second syntax element from the video bitstream, determine the quadratic transform set identifier; and Using the determined quadratic transform kernel type and the determined quadratic transform set identifier, an inverse quadratic transform is performed on the first block, the inverse quadratic transform corresponding to the quadratic transform performed during the encoding of the video data; and Based on the determination that the second value is obtained from the quadratic transformation kernel type, the inverse quadratic transformation is not performed on the first block.

2. The method according to claim 1, wherein, The video stream does not include the second syntax element if the second value is obtained based on the second transform kernel type.

3. The method according to claim 1, further comprising: The main transform of the first block is determined based on the third syntax element from the video bitstream.

4. The method according to claim 3, wherein, The main transform is an inseparable transform.

5. The method according to claim 1, wherein, The quadratic transformation is an inseparable transformation.

6. The method according to claim 1, wherein, The quadratic transformation is selected from the transformation set based on the determined quadratic transformation set identifier.

7. The method according to claim 6, wherein, The transform set includes one or more master-only transform types and one or more master-secondary transform types.

8. The method according to claim 1, wherein, The secondary transform kernel type is written together with the primary transform kernel type in the video bitstream.

9. The method according to claim 1, further comprising: Based on the information from the video stream, the kernel set of the secondary transform is determined.

10. The method according to claim 9, wherein, The kernel set of the secondary transform is written together with the kernel set of the primary transform into the video bitstream.

11. The method according to claim 1, wherein, A flag is written into the video stream indicating whether a primary transform and / or secondary transform are applied; and The secondary transformation kernel type is written through the first syntax element, based on the first value of the written flag.

12. The method according to claim 1, wherein, The identifier of the quadratic transformation set is determined based on the set context information.

13. The method according to claim 12, wherein, The set context information includes one or more of the following: previously encoded mode, primary transform type, primary transform set identifier, secondary transform type, secondary transform set identifier, code block size, and transform block size.

14. The method according to claim 12, wherein, Encoded text is grouped into predefined blocks to form an index.

15. The method according to claim 1, wherein, Based on the grouping of the encoding context, the set of transform kernel types of the second transformation is identified.

16. The method according to claim 1, wherein, The second syntax element is arithmetic encoded.

17. The method according to claim 1, further comprising: The most likely flag is determined from the video stream, wherein the most likely flag indicates whether the set is the most likely.

18. The method according to claim 1, wherein, Derive a list of quadratic transformation sets for each transform block.

19. A computing system, comprising: Control circuit; Memory; as well as One or more instruction sets, stored in the memory and configured to be executed by the control circuitry, the instruction sets comprising instructions for: Receive video data from a video stream, the video data including multiple blocks and a first syntax element, the multiple blocks including the first block; Based on the first syntax element, determine the quadratic transformation kernel type of the first block; Based on the determination that the type of the quadratic transform kernel is the first value: Based on the second syntax element from the video bitstream, determine the quadratic transform set identifier; and Using the determined quadratic transform kernel type and the determined quadratic transform set identifier, an inverse quadratic transform is performed on the first block, the inverse quadratic transform corresponding to the quadratic transform performed during the encoding of the video data; and Since the type of the quadratic transformation kernel is determined to be the second value, the inverse quadratic transformation is not performed on the first block.

20. A non-transitory computer-readable storage medium storing thereon one or more sets of instructions configured to be executed by a computing device having control circuitry and memory, the one or more sets of instructions comprising instructions for: Receive video data from a video stream, the video data including multiple blocks and a first syntax element, the multiple blocks including the first block; Based on the first syntax element, determine the quadratic transformation kernel type of the first block; Based on the determination that the type of the quadratic transform kernel is the first value: Based on the second syntax element from the video bitstream, determine the quadratic transform set identifier; and Using the determined quadratic transform kernel type and the determined quadratic transform set identifier, an inverse quadratic transform is performed on the first block, the inverse quadratic transform corresponding to the quadratic transform performed during the encoding of the video data; and Since the type of the quadratic transformation kernel is determined to be the second value, the inverse quadratic transformation is not performed on the first block.

Images