Systems and methods using adaptive transform kernel size

Abstract

An example method of video decoding includes receiving a video bitstream including a set of coded blocks and a corresponding set of transform coefficients. The method also includes selecting an unseparable quadratic transform from a set of one or more unseparable quadratic transforms based on an index value. The method also includes determining a kernel size and a kernel shape of the unseparable quadratic transform based on coding information, and decoding the set of coded blocks by applying the unseparable quadratic transform having the determined kernel size and kernel shape to the set of coded blocks.

Description

[0001] Related applications This application claims priority to U.S. Provisional Patent Application No. 63 / 570,220, filed March 26, 2024, entitled "Adaptive Kernel Size for Secondary Indivisible Transformations," and is a continuation of and claims priority to U.S. Patent Application No. 18 / 786,304, filed July 26, 2024, entitled "System and Method Using Adaptive Transformation Kernel Size." Technical Field

[0002] Embodiments of this disclosure generally relate to video coding, including but not limited to transform coding applied to prediction residuals. Background Technology

[0003] Various electronic devices support digital video, such as digital televisions, laptops or desktop computers, tablets, digital cameras, digital recording devices, digital media players, video game consoles, smartphones, video conferencing equipment, and video streaming devices. Electronic devices send and receive, or otherwise transmit, digital video data via communication networks, and / or store digital video data on storage devices. Because communication networks have limited bandwidth capacity and storage devices have limited memory resources, video encoding can be used to compress video data according to one or more video encoding standards before it is transmitted or stored. Video encoding can be performed by hardware and / or software on electronic / client devices, or by hardware and / or software on servers providing cloud services.

[0004] Video coding typically uses predictive methods (e.g., inter-frame prediction, intra-frame prediction, etc.) that leverage the inherent redundancy in video data. Video coding aims to compress video data into a form using a lower bitrate while avoiding or minimizing video quality degradation. Several video codec standards have been developed. For example, High Efficiency Video Coding (HEVC / H.265) is a video compression standard designed as part of the MPEG-H project. The ITU-T and ISO / IEC released the HEVC / H.265 standard in 2013 (Revision 1), 2014 (Revision 2), 2015 (Revision 3), and 2016 (Revision 4). Multi-Functional Video Coding (VVC / H.266) is a video compression standard designed as a successor to HEVC. The ITU-T and ISO / IEC released the VVC / H.266 standard in 2020 (Revision 1) and 2022 (Revision 2). AOMedia Video 1 (AV1) is an open video coding format designed as an alternative to HEVC. On January 8, 2019, empirical version 1.0.0 of the specification was released with Errata 1. Summary of the Invention

[0005] This disclosure describes, among other things, a set of methods for video (image) compression, including methods for adaptively determining transform parameters (e.g., kernel size, shape, and / or scan order) using encoded information. Encoded information refers to any information known to both the encoder and decoder when decoding the current block (e.g., information transmitted as a signal and information derived). Transform parameters can be used for primary and / or secondary transforms. Transform parameters can be used for separable or non-separable transforms. Adaptive transform parameters allow for smaller dimensions where appropriate, which provides benefits in hardware implementation, such as increased throughput and / or fewer hardware components (e.g., fewer logic gates).

[0006] According to some embodiments, a video decoding method includes: (i) receiving a video bitstream (e.g., an encoded video sequence), the video bitstream including a set of coded blocks and a corresponding set of transform coefficients; (ii) selecting a transform (e.g., a primary transform and / or a secondary transform) from a set of one or more transforms; (iii) determining the kernel size and kernel shape of the selected transform based on encoding information; and (iv) decoding the set of coded blocks by applying the selected transform having the determined kernel size and kernel shape to the set of coded blocks.

[0007] According to some embodiments, a video coding method includes: (i) receiving video data comprising a set of blocks (e.g., a source video sequence); (ii) selecting a transform (e.g., a primary transform and / or a secondary transform) from a set of one or more transforms; (iii) determining the kernel size and kernel shape of the selected transform based on coding information; and (iv) encoding the set of blocks by applying the transform having the determined kernel size and kernel shape to the set of blocks.

[0008] According to some embodiments, a method for bitstream conversion includes: (i) obtaining a source video sequence comprising multiple frames; and (ii) performing a conversion between the source video sequence and a video bitstream of visual media data according to format rules. The video bitstream includes: a set of coded blocks and a set of corresponding transform coefficients. The format rules specify: (a) selecting a transform (e.g., an indivisible transform) for the set of coded blocks from a set of one or more transforms, and (b) determining the kernel size and kernel shape for the selected transform based on encoding information.

[0009] According to some embodiments, a computing system, such as a streaming system, server system, personal computer system, or other electronic device, is provided. The computing system includes control circuitry and memory storing one or more instruction sets. The one or more instruction sets include instructions for performing any of the methods described herein. In some embodiments, the computing system includes an encoder component and a decoder component (e.g., a code converter).

[0010] According to some embodiments, a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium stores one or more sets of instructions executable by a computing system. The one or more sets of instructions include instructions for performing any of the methods described herein.

[0011] Therefore, this document discloses methods, apparatus, and systems for encoding and decoding video. Such methods, apparatus, and systems can supplement or replace conventional methods, apparatus, and systems for video encoding / decoding.

[0012] The features and advantages described in this specification are not necessarily exhaustive, and in particular, some additional features and advantages will be apparent to those skilled in the art when considered in conjunction with the accompanying drawings, description, and claims provided in this disclosure. Furthermore, it should be noted that the language used in this specification has been chosen primarily for readability and illustrative purposes, and not necessarily for depicting or defining the subject matter described herein. Attached Figure Description

[0013] To gain a more detailed understanding of this disclosure, a more specific description can be made by referring to the features of various embodiments, some of which are illustrated in the accompanying drawings. However, the drawings only illustrate relevant features of this disclosure and are therefore not necessarily to be considered limiting, as those skilled in the art will understand upon reading this disclosure that this description may contain other valid features.

[0014] Figure 1 This is a block diagram illustrating an example communication system according to some embodiments.

[0015] Figure 2A This is a block diagram illustrating example elements of an encoder assembly according to some embodiments.

[0016] Figure 2B This is a block diagram illustrating example elements of a decoder component according to some embodiments.

[0017] Figure 3 This is a block diagram illustrating an example server system according to some embodiments.

[0018] Figures 4A to 4D An exemplary encoding tree structure according to some embodiments is shown.

[0019] Figures 5A to 5C Exemplary prediction blocks, residual blocks, and reconstruction blocks according to some embodiments are shown.

[0020] Figure 6A An exemplary video decoding process according to some embodiments is shown.

[0021] Figure 6B An exemplary video encoding process according to some embodiments is shown.

[0022] As is customary, the various features shown in the accompanying drawings are not necessarily drawn to scale, and the same reference numerals may be used to denote the same features throughout the specification and the drawings. Detailed Implementation

[0023] This disclosure describes, among other things, a set of methods for video (image) compression, including methods for applying transforms to residual blocks and adaptively determining transform parameters. For example, an inseparable transform (e.g., a primary inseparable transform or a secondary inseparable transform) can be selected from a set of available transforms. The transform parameters (e.g., kernel size and / or shape) of the selected inseparable transform can be determined using encoding information available to both the encoder and decoder. In this way, the transform parameters can be adaptively adjusted during encoding and decoding. Adaptive transform parameters allow for the use of smaller dimensions where appropriate, which provides benefits in hardware implementation, such as increased throughput and / or fewer hardware components required.

[0024] Exemplary systems and devices Figure 1 This is a block diagram illustrating a communication system 100 according to some embodiments. The communication system 100 includes a source device 102 and a plurality of electronic devices 120 (e.g., electronic devices 120-1 to 120-m) communicatively coupled to each other via one or more networks. In some embodiments, the communication system 100 is a streaming system, for example, for use with video-enabled applications such as video conferencing applications, digital TV applications, and media storage and / or distribution applications.

[0025] Source device 102 includes a video source 104 (e.g., a camera component or media storage) and an encoder component 106. In some embodiments, the video source 104 is a digital camera (e.g., configured to create an uncompressed video sample stream). The encoder component 106 generates one or more encoded video streams from the video stream. The video stream from video source 104 can have a high data volume compared to the encoded video stream 108 generated by encoder component 106. Because the encoded video stream 108 has a lower data volume (less data) compared to the video stream from video source 104, it requires less transmission bandwidth and less storage space for storage compared to the video stream from video source 104. In some embodiments, source device 102 does not include encoder component 106 (e.g., configured to send uncompressed video to network 110).

[0026] One or more networks 110 represent any number of networks that transmit information between source device 102, server system 112, and / or electronic device 120, including, for example, wired (wired) and / or wireless communication networks. One or more networks 110 may exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and / or the Internet.

[0027] One or more networks 110 include a server system 112 (e.g., a distributed / cloud computing system). In some embodiments, the server system 112 is or includes a streaming server (e.g., configured to store and / or distribute video content, such as an encoded video stream from source device 102). The server system 112 includes an encoder component 114 (e.g., configured to encode and / or decode video data). In some embodiments, the encoder component 114 includes an encoder component and / or a decoder component. In various embodiments, the encoder component 114 is instantiated as hardware, software, or a combination thereof. In some embodiments, the encoder component 114 is configured to decode an encoded video stream 108 and re-encode the video data using different encoding standards and / or methods to produce encoded video data 116. In some embodiments, the server system 112 is configured to generate multiple video formats and / or encodings based on the encoded video stream 108. In some embodiments, the server system 112 serves as a Media Aware Network Element (MANE). For example, server system 112 can be configured to trim encoded video stream 108 to customize potentially different streams for one or more electronic devices in electronic device 120. In some embodiments, MANE is provided separately from server system 112.

[0028] Electronic device 120-1 includes a decoder component 122 and a display 124. In some embodiments, the decoder component 122 is configured to decode encoded video data 116 to produce an output video stream that can be displayed on a display or other type of presentation device. In some embodiments, one or more of the electronic devices 120 do not include a display component (e.g., communicatively coupled to an external display device and / or includes a media storage device). In some embodiments, the electronic device 120 is a streaming client. In some embodiments, the electronic device 120 is configured to access a server system 112 to obtain encoded video data 116.

[0029] The source device and / or multiple electronic devices 120 are sometimes referred to as “terminal devices” or “user equipment”. In some embodiments, one or more of the source device 102 and / or electronic devices 120 are instances of server systems, personal computers, portable devices (e.g., smartphones, tablets, or laptops), wearable devices, video conferencing equipment, and / or other types of electronic devices.

[0030] In an example operation of communication system 100, source device 102 sends an encoded video stream 108 to server system 112. For example, source device 102 may encode a stream of images captured by the source device. Server system 112 receives the encoded video stream 108 and may decode and / or encode the encoded video stream 108 using encoder component 114. For example, server system 112 may apply the optimal encoding for network transmission and / or storage to the video data. Server system 112 may transmit encoded video data 116 (e.g., one or more encoded video streams) to one or more electronic devices 120. Each electronic device 120 may decode the encoded video data 116 and optionally display video images.

[0031] Figure 2AThis is a block diagram illustrating example elements of an encoder assembly 106 according to some embodiments. The encoder assembly 106 receives video data (e.g., a source video sequence) from a video source 104. In some embodiments, the encoder assembly includes a receiver (e.g., transceiver) assembly configured to receive the source video sequence. In some embodiments, the encoder assembly 106 receives the video sequence from a remote video source (e.g., a video source that is a component of a device different from the encoder assembly 106). The video source 104 may provide the source video sequence as a digital video sample stream, which may have any suitable bit depth (e.g., 8-bit, 10-bit, or 12-bit), any color space (e.g., BT.601YCrCb or RGB), and any suitable sampling structure (e.g., YCrCb 4:2:0 or YCrCb 4:4:4). In some embodiments, the video source 104 is a storage device storing previously captured / prepared video. In some embodiments, the video source 104 is a camera that captures local image information as a video sequence. The video data may be provided as multiple individual pictures that impart motion when viewed sequentially. An image itself can be organized as a spatial array of pixels, where each pixel can include one or more samples, depending on the sampling structure, color space, etc., used. Those skilled in the art can readily understand the relationship between pixels and samples.

[0032] Encoder component 106 is configured to encode and / or compress images of a source video sequence into an encoded video sequence 216 in real time or under other time constraints required by the application. In some embodiments, encoder component 106 is configured to perform a conversion between a source video sequence and a bitstream of visual media data (e.g., a video bitstream). Implementing an appropriate encoding rate is a function of controller 204. In some embodiments, controller 204 controls and is functionally coupled to other functional units described below. Parameters set by controller 204 may include rate control-related parameters (e.g., image skipping, quantizer and / or rate distortion optimization technique λ value), image size, group of images (GOP) layout, maximum motion vector search range, etc. Other functions of controller 204 can be readily identified by those skilled in the art, as these other functions may be related to encoder component 106 optimized for a particular system design.

[0033] In some embodiments, encoder component 106 is configured to operate in an encoding loop. In a simplified example, the encoding loop includes: a source encoder 202 (e.g., responsible for creating symbols (such as a symbol stream) based on the input image to be encoded and reference images(s)) and a (local) decoder 210. Decoder 210 reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data (when compression between the symbols and the encoded video stream is lossless). The reconstructed sample stream (sample data) is input to reference image memory 208. Since decoding of the symbol stream produces bit-precise results independent of decoder location (local or remote), the contents of reference image memory 208 are also bit-precisely corresponding between the local encoder and the remote encoder. In this way, the reference image samples interpreted by the encoder's prediction portion are the same as the sample values ​​interpreted by the decoder during prediction.

[0034] The operation of decoder 210 can be combined with remote decoders (such as those described below). Figure 2B The operation of decoder component 122 is the same as described in the detailed description. However, a brief reference is provided. Figure 2B When symbols are available and the entropy encoder 214 and the parser 254 are able to encode / decode the symbols into an encoded video sequence without loss, the entropy decoding portion of the decoder component 122, including the buffer memory 252 and the parser 254, may not be fully implemented in the local decoder 210.

[0035] Aside from parsing / entropy decoding, the decoder techniques described in this paper can exist in the corresponding encoders with essentially the same functional form. Therefore, the subject matter focuses on decoder operations. Furthermore, the description of the encoder techniques can be simplified because the encoder techniques are inverses of the decoder techniques.

[0036] As part of the operation of source encoder 202, source encoder 202 can perform motion-compensated predictive coding, referencing one or more previously encoded frames from the video sequence designated as reference frames, to predictively encode the input frames. In this way, encoding engine 212 encodes the differences between pixel blocks of the input frame and pixel blocks of the reference frame, which can be selected as the prediction reference for the input frames. Controller 204 can manage the encoding operations of source encoder 202, including, for example, setting parameters and subgroup parameters for encoding video data.

[0037] Decoder 210 decodes encoded video data of frames that can be designated as reference frames, based on symbols created by source encoder 202. The operation of encoding engine 212 can advantageously be a lossy process. When encoded video data is processed by video decoder (… Figure 2AWhen decoded at a location (not shown), the reconstructed video sequence can be a copy of the source video sequence with some errors. Decoder 210 replicates the decoding process performed on the reference frame by a remote video decoder, and the reconstructed reference frame can be stored in reference image memory 208. In this way, encoder component 106 locally stores a copy of the reconstructed reference frame that shares common content (no transmission errors) with the reconstructed reference frame to be obtained by the remote video decoder.

[0038] Predictor 206 can perform a prediction search against encoding engine 212. That is, for a new frame to be encoded, predictor 206 can search the reference image memory 208 for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that can be used as appropriate prediction references for the new image. Predictor 206 can operate pixel-by-pixel based on sample blocks to find appropriate prediction references. As determined by the search results obtained by predictor 206, the input image can have prediction references extracted from multiple reference images stored in reference image memory 208.

[0039] The outputs of all the aforementioned functional units can be entropy encoded in entropy encoder 214. Entropy encoder 214 converts the symbols generated by the various functional units into an encoded video sequence by lossless compression of the symbols according to techniques known to those skilled in the art (e.g., Huffman coding, variable-length coding, and / or arithmetic coding).

[0040] In some embodiments, the output of entropy encoder 214 is coupled to a transmitter. The transmitter may be configured to buffer the encoded video sequence created by entropy encoder 214 in preparation for transmission via communication channel 218, which may be a hardware / software link to a storage device storing the encoded video data. The transmitter may be configured to combine encoded video data from source encoder 202 with other data to be transmitted (e.g., encoded audio data and / or auxiliary data streams (sources not shown)). In some embodiments, the transmitter may send additional data along with the encoded video. Source encoder 202 may include such data as part of the encoded video sequence. Additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, supplementary enhancement information (SEI) messages, fragments of visual usability information (VUI) parameter sets, etc.

[0041] Controller 204 can manage the operation of encoder component 106. During encoding, controller 204 can assign a specific encoding picture type to each encoded picture, but this may affect the encoding technique applied to the corresponding picture. For example, a picture can be assigned as an intra-picture (I-picture), a predictive picture (P-picture), or a bidirectional predictive picture (B-picture). Intra-pictures can be encoded and decoded without using any other frames in the sequence as prediction sources. Some video codecs allow different types of intra-pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are familiar with those variations of I-pictures and their corresponding applications and characteristics, and therefore will not be repeated here. Predictive pictures can be encoded and decoded using intra-picture or inter-picture prediction, which uses at most one motion vector and reference index to predict the sample values ​​for each block. Bidirectional predictive pictures can be encoded and decoded using intra-picture or inter-picture prediction, which uses at most two motion vectors and reference indices to predict the sample values ​​for each block. Similarly, multi-predictive images can use more than two reference images and associated metadata to reconstruct a single block.

[0042] Source images can typically be spatially subdivided into multiple sample blocks (e.g., each block containing 4×4, 8×8, 4×8, or 16×16 samples), and encoded block by block. These blocks can be predictively coded with reference to other (already coded) blocks, determined based on the coding assignment of the corresponding images applied to the block. For example, blocks of an I-image may be non-predictively coded, or the block may be predictively coded (spatial or intra-frame prediction) with reference to already coded blocks of the same image. Pixel blocks of a P-image may be predictively coded with reference to a previously coded reference image via spatial or temporal prediction. Blocks of a B-image may be predictively coded with reference to one or two previously coded reference images via spatial or temporal prediction.

[0043] The acquired video can serve as multiple source images (video images) in a time series. Intra-image prediction (often abbreviated as intra-frame prediction) utilizes spatial correlations within a given image, while inter-image prediction utilizes (temporal or other) correlations between images. In the example, a specific image being encoded / decoded (referred to as the current image) is segmented into blocks. When a block in the current image resembles a reference block in a previously encoded and still buffered reference image in the video, the block in the current image can be encoded using a vector called a motion vector. The motion vector points to the reference block in the reference image and, in the case of using multiple reference images, may have a third dimension identifying the reference images.

[0044] Encoder component 106 can perform encoding operations according to a predetermined video coding technique or standard (such as any described herein). In its operation, encoder component 106 can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

[0045] Figure 2B This is a block diagram illustrating example elements of a decoder component 122 according to some embodiments. Figure 2B The decoder component 122 is coupled to channel 218 and display 124. In some embodiments, the decoder component 122 includes a transmitter coupled to loop filter 256 and configured to (e.g., via a wired or wireless connection) transmit data to display 124.

[0046] In some embodiments, decoder component 122 includes a receiver coupled to channel 218 and configured to receive data from channel 218 (e.g., via a wired or wireless connection). The receiver may be configured to receive one or more encoded video sequences to be decoded by decoder component 122. In some embodiments, decoding of each encoded video sequence is independent of other encoded video sequences. Each encoded video sequence may be received from channel 218, which may be a hardware / software link to a storage device storing the encoded video data. The receiver may receive the encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams that may be forwarded to their respective user entities (not indicated). The receiver may separate the encoded video sequences from other data. In some embodiments, the receiver receives additional (redundant) data with the encoded video. The additional data may be included as part of the encoded video sequence. Decoder component 122 may use the additional data to decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0047] According to some embodiments, decoder component 122 includes buffer memory 252, parser 254 (sometimes also called entropy decoder), scaler / inverse transform unit 258, intra-frame image prediction unit 262, motion compensation prediction unit 260, aggregator 268, loop filter unit 256, reference image memory 266, and current image memory 264. In some embodiments, decoder component 122 is implemented as an integrated circuit, a series of integrated circuits, and / or other electronic circuits. Decoder component 122 may be implemented at least partially in software.

[0048] Buffer memory 252 is coupled between channel 218 and parser 254 (e.g., to combat network jitter). In some embodiments, buffer memory 252 is decoupled from decoder component 122. In some embodiments, a separate buffer memory is provided between the output of channel 218 and decoder component 122. In some embodiments, a separate buffer memory is provided outside decoder component 122 (e.g., to combat network jitter) in addition to buffer memory 252 inside decoder component 122 (e.g., configured to handle playback timing). Buffer memory 252 may be unnecessary or can be made small when receiving data from a store / forward device with sufficient bandwidth and controllability or from an isochronous synchronization network. Buffer memory 252 may also be required for use on packet-switched networks such as the Internet, and may be relatively large and / or have an adaptive size, and may be implemented at least partially in an operating system or similar element outside of decoder component 122.

[0049] Parser 254 is configured to reconstruct symbols 270 from the encoded video sequence. Symbols may include, for example, information for managing the operation of decoder component 122 and / or information for controlling a display device such as display 124. Control information for the display device may be in the form of, for example, Supplemental Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not indicated). Parser 254 parses (entropy decodes) the encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. Parser 254 may extract a set of subgroup parameters from the encoded video sequence for at least one subgroup of pixels in a subgroup for use in the video decoder, based on at least one parameter corresponding to a group. Subgroups may include picture groups (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The parser 254 can also extract information such as transform coefficients, quantizer parameter values, and motion vectors from the encoded video sequence.

[0050] Depending on the type of encoded video frames or a subset of encoded video frames (such as inter-frame and intra-frame frames, inter-frame and intra-frame blocks) and other factors, the reconstruction of symbol 270 may involve multiple different units. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the encoded video sequence by parser 254. For the sake of brevity, the flow of this subgroup control information between parser 254 and the multiple units described below is not depicted.

[0051] The decoder component 122 can be conceptually subdivided into multiple functional units, and in some implementations, these units interact closely with each other and can be at least partially integrated with each other. However, for clarity, the conceptual subdivision of functional units is retained herein.

[0052] The scaler / inverse transform unit 258 receives quantized transform coefficients and control information (such as which transform to use, block size, quantization factor, and / or quantization scaling matrix) as symbols 270 from the parser 254. The scaler / inverse transform unit 258 can output blocks that include sample values ​​that can be input to the aggregator 268. In some cases, the output samples of the scaler / inverse transform unit 258 belong to intra-coded blocks; that is, blocks that do not use prediction information from previously reconstructed images, but can use prediction information from previously reconstructed portions of the current image. Such prediction information can be provided by the intra-picture prediction unit 262. The intra-picture prediction unit 262 can use surrounding reconstructed information extracted from the current (partially reconstructed) image in the current image memory 264 to generate blocks of the same size and shape as the blocks in the reconstruction. The aggregator 268 can add the prediction information already generated by the intra-picture prediction unit 262 to the output sample information provided by the scaler / inverse transform unit 258 based on each sample.

[0053] In other cases, the output samples of the scaler / inverse transform unit 258 belong to inter-frame coded and potentially motion-compensated blocks. In this case, the motion compensation prediction unit 260 can access the reference image memory 266 to extract samples for prediction. After motion compensation is performed on the extracted samples according to the block-related symbols 270, these samples can be added by the aggregator 268 to the output of the scaler / inverse transform unit 258 (referred to as residual samples or residual signals in this case) to generate output sample information. The address of the predicted samples obtained by the motion compensation prediction unit 260 from the address in the reference image memory can be controlled by motion vectors. Motion vectors can be provided to the motion compensation prediction unit 260 in the form of symbols 270, which can have, for example, X, Y, and reference image components. Motion compensation may also include, for example, the difference in sample values ​​(e.g., sample values ​​extracted from the reference image memory 266) when using subsampled exact motion vectors, motion vector prediction mechanisms, etc.

[0054] The output samples of aggregator 268 can be employed by various loop filtering techniques in loop filter unit 256. Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video bitstream, which are available to loop filter unit 256 as symbols 270 from parser 254, but may also be responsive to metadata obtained during decoding of previous (in decoding order) portions of the encoded picture or encoded video sequence, and to previously reconstructed and loop-filtered sample values. The output of loop filter unit 256 can be a sample stream, which can be output to a display device such as display 124, and stored in reference picture memory 266 for future inter-picture prediction.

[0055] Once certain encoded images are reconstructed, they can be used as reference images for future predictions. Once an encoded image is reconstructed and an encoded image (e.g., by parser 254) has been identified as a reference image, the current reference image can become part of the reference image memory 266, and a new current image memory can be reallocated before the reconstruction of subsequent encoded images begins.

[0056] Decoder component 122 can perform decoding operations according to a predetermined video compression technique recorded in a standard (such as any standard described herein). An encoded video sequence may conform to the syntax specified by the video compression technique or standard being used, in the sense that the encoded video sequence follows the syntax of the video compression technique or standard (as specified in the video compression technique document or standard, and particularly in the configuration file therein). Furthermore, the complexity of the encoded video sequence may be within a range defined by the hierarchy of the video compression technique or standard in order to conform to some video compression techniques or standards. In some cases, the hierarchy limits the maximum image size, maximum frame rate, maximum reconstruction sampling rate (measured, for example, megasamples per second), maximum reference image size, etc. In some cases, the limitations set by the hierarchy can be further limited by the assumed reference decoder (HRD) specification and the metadata managed by the HRD buffer, represented as a signal in the encoded video sequence.

[0057] Figure 3 This is a block diagram illustrating a server system 112 according to some embodiments. The server system 112 includes control circuitry 302, one or more network interfaces 304, memory 314, a user interface 306, and one or more communication buses 312 for interconnecting these components. In some embodiments, the control circuitry 302 includes one or more processors (e.g., CPU, GPU, and / or DPU). In some embodiments, the control circuitry includes a field-programmable gate array, a hardware accelerator, and / or an integrated circuit (e.g., an application-specific integrated circuit).

[0058] Network interface 304 can be configured to interface with one or more communication networks (e.g., wireless, wired, and / or optical networks). Communication networks can be local area, wide area, metropolitan area, vehicle-mounted and industrial, real-time, latency-tolerant, etc. Examples of communication networks include: local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., cable or wireless wide area networks including cable television, satellite television, and terrestrial broadcast television, vehicle-mounted and industrial networks including CAN bus (CANBus), etc. Such communication can be unidirectional and receive-only (e.g., broadcasting TV), unidirectional and transmit-only (e.g., CANbus to certain CANbus devices), or bidirectional (e.g., communication with other computer systems using a local area network or wide area network). Such communication can include communication to one or more cloud computing networks.

[0059] User interface 306 includes one or more output devices 308 and / or one or more input devices 310. Input devices 310 may include one or more of the following: keyboard, mouse, touchpad, touchscreen, data glove, joystick, microphone, scanner, camera, etc. Output devices 308 may include one or more of the following: audio output devices (e.g., speakers), visual output devices (e.g., displays or monitors), etc.

[0060] Memory 314 may include high-speed random access memory (such as DRAM, SRAM, DDR RAM, and / or other random access solid-state memory devices) and / or non-volatile memory (such as one or more disk storage devices, optical disk storage devices, flash memory devices, and / or other non-volatile solid-state memory devices). Memory 314 may optionally include one or more storage devices located remotely from control circuitry 302. Memory 314 or the non-volatile solid-state memory device within memory 314 includes a non-transitory computer-readable storage medium. In some embodiments, memory 314 or the non-transitory computer-readable storage medium of memory 314 stores programs, modules, instructions, and data structures, or subsets or supersets thereof: Operating system 316, which includes procedures for handling various basic system services and for performing hardware-related tasks; The network communication module 318 is used to connect the server system 112 to other computing devices via one or more network interfaces 304 (e.g., via wired and / or wireless connections); Encoding module 320 is used to perform various functions related to encoding and / or decoding data (such as video data). In some embodiments, encoding module 320 is an instance of encoder component 114. Encoding module 320 includes, but is not limited to, one or more of the following: ○ Decoding module 322, for performing various functions related to decoding encoded data, such as those previously described with respect to decoder component 122; and ○ Encoding module 340, for performing various functions related to encoded data, such as those previously described with respect to encoder component 106; and Image memory 352 is used to store images and image data, for example, for use with encoding module 320. In some embodiments, image memory 352 includes one or more of the following: reference image memory 208, buffer memory 252, current image memory 264, and reference image memory 266.

[0061] In some embodiments, the decoding module 322 includes a parsing module 324 (e.g., configured to perform various functions previously described with respect to the parser 254), a transformation module 326 (e.g., configured to perform various functions previously described with respect to the scaler / inverse transformation unit 258), a prediction module 328 (e.g., configured to perform various functions previously described with respect to the motion compensation prediction unit 260 and / or the intra-frame image prediction unit 262), and a filter module 330 (e.g., configured to perform various functions previously described with respect to the loop filter 256).

[0062] In some embodiments, the encoding module 340 includes a code module 342 (e.g., configured to perform various functions previously described with respect to source encoder 202 and / or encoding engine 212) and a prediction module 344 (e.g., configured to perform various functions previously described with respect to predictor 206). In some embodiments, the decoding module 322 and / or the encoding module 340 includes... Figure 3 A subset of the modules shown. For example, the shared prediction module is used by both the decoding module 322 and the encoding module 340.

[0063] Each of the identified modules stored in memory 314 corresponds to a set of instructions that performs the functions described herein. The identified modules (e.g., instruction sets) do not need to be implemented as separate software programs, processes, or modules; therefore, various subsets of these modules can be combined or otherwise rearranged in various embodiments. For example, encoding module 320 may optionally not include separate decoding and encoding modules, but instead use the same set of modules to perform both sets of functions. In some embodiments, memory 314 stores a subset of the modules and data structures identified above. In some embodiments, memory 314 stores additional modules and data structures not described above.

[0064] although Figure 3 A server system 112 according to some embodiments is shown, but Figure 3 This is intended more as a functional description of various features that can exist in one or more server systems, rather than a structural diagram of the embodiments described herein. In practice, items shown individually may be combined, and some items may be separated. For example, Figure 3 Some items shown individually can be implemented on a single server, and a single item can be implemented by one or more servers. The actual number of servers used to implement server system 112 and how features are allocated among them will differ in different implementations and may optionally depend in part on the amount of data traffic processed by the server system during peak usage periods and during average usage periods.

[0065] Exemplary encoding techniques The encoding processes and techniques described below can be performed at the devices and systems described above (e.g., source device 102, server system 112, and / or electronic device 120). According to some embodiments, methods for applying an indivisible transform kernel to a residual block are described. As used herein, a block can refer to a coding tree block, a maximum coding block, a predefined fixed block size, a coding block, a prediction block, a residual block, or a transform block. An inter-frame mode coded block (or inter-frame block) refers to a block using an inter-frame prediction mode, and an intra-frame mode coded block (or intra-frame block) refers to a block using an intra-frame prediction mode.

[0066] The principal transform can belong to the family of sinusoidal transforms (DCT, DST, the inverted version of DCT, and the inverted version of ADST). DCT can refer to any transform using a transform kernel derived from the discrete cosine transform basis (e.g., DCT type 2), and DST / ADST can refer to any transform using a transform kernel derived from the discrete sine transform basis (e.g., DST type 4 or 7).

[0067] The primary transform can belong to the generalized line graph transform (LGT) family, or it can be based on a trained kernel. The set of secondary transforms can be a grouping of one or more indivisible secondary transform kernel transform types. A unique or common set of secondary transforms can be defined for each primary transform type and / or intra-frame or inter-frame mode type.

[0068] Furthermore, an inseparable transform can refer to the main transform applied directly to the residual, or a quadratic transform applied to the transform coefficient block generated by the main transform. Inseparable transform kernels can be grouped into sets represented by set indices and kernel indices within those sets. An inseparable quadratic transform can be a trained kernel applied to the main transform coefficients at the encoder or the inverse quantization coefficients at the decoder.

[0069] An indivisible quadratic transformation kernel can be considered as a set of basis vectors in a vector space. If represented as an M×N (M rows and N columns) matrix, then N corresponds to the dimension of the vector space, and M corresponds to the number of basis vectors. Therefore, M×N can be used to represent the kernel size. Examples of kernel sizes include, but are not limited to, 64×64, 32×64, 16×64, 8×84, 4×64, 16×16, 8×16, and 4×16 samples.

[0070] The End-of-Block (EOB) value corresponds to the position of the last valid (e.g., non-zero) coefficient in the coded block according to a given coefficient scan order. For a given coefficient scan order, all coefficients outside the EOB position are zero. In some embodiments, if an indivisible quadratic transform kernel of size M×N is applied to a coded block, then the EOB value ≤ M.

[0071] The scanning order refers to the coefficient recombination process of mapping the two-dimensional principal transform coefficient array to the one-dimensional principal transform coefficient array, which serves as the input of the forward quadratic transform. It can also refer to the backward coefficient recombination process of mapping the one-dimensional quadratic transform coefficient array back to the two-dimensional principal transform coefficient array.

[0072] First, we move on to block partitioning. Figures 4A to 4D An exemplary encoding tree structure according to some embodiments is shown. Figure 4A As shown in the first coding tree structure (400), some coding methods use a 4-way partitioning tree starting from the 64×64 level and going down to the 4×4 level, for example, with some additional constraints for 8×8 blocks. Figure 4A In this context, the partition designated "R" is recursive because the same partition tree is repeated at smaller scales until the lowest level is reached. For example... Figure 4BAs shown in the exemplary coding tree structure (402), some coding methods extend the partition tree to a 10-way structure and increase the maximum size (e.g., sometimes referred to as the superblock) to start from 128×128. The second coding tree structure includes 4:1 / 1:4 rectangular partitions that are not present in the first coding tree structure. Figure 4B A partition type with 3 sub-partitions in the second row is called a T-partition. In addition to the code block size, the code tree depth can also be defined to indicate the partition depth from the root node.

[0073] As an example, coding tree units (CTUs) can be partitioned into coding units (CUs) using a quadtree structure represented as a coding tree to accommodate various local characteristics. In some embodiments, a decision is made at the CU level regarding whether to use inter-image (temporal) or intra-image (spatial) predictions to encode image regions. Depending on the PU partitioning type, each CU can be further partitioned into one, two, or four prediction units (PUs). Within a PU, the same prediction process can be applied, and relevant information can be sent to the decoder based on the PU. After obtaining residual blocks by applying prediction processing based on the PU partitioning type, the CUs can be partitioned into transform units (TUs) according to another quadtree structure similar to the coding tree of the CU.

[0074] Quadtrees with nested multi-type tree partitioning structures (using binary and ternary partitioning) can be used to replace the concept of multiple partition unit types. In the coding tree structure, the CU can have a square or rectangular shape. The CTU is first partitioned by a quadtree structure. The leaf nodes of the quadtree can be further partitioned using multi-type tree structures. For example... Figure 4C As shown in the third coding tree structure (404), the multi-type tree structure includes four partition types. The leaf nodes of the multi-type tree are called CUs, and are only considered large if the CU is too large for the maximum transform length. This means that in a quadtree with a nested multi-type tree coding block structure, CUs, PUs, and TUs can have the same block size. Figure 4D An example of block partitioning for a CTU(406) is shown, which illustrates a quadtree example.

[0075] The coding tree scheme supports the ability for luma and chroma to have separate block tree structures, as in VTM7. In some cases, for P and B slices, the luma and chroma CTBs within a single CTU share the same coding tree structure. However, for I slices, luma and chroma can have separate block tree structures. When the separate block tree mode is applied, the luma CTB is partitioned into CUs via one coding tree structure, and the chroma CTB is partitioned into chroma CUs via another coding tree structure. This means that a CU in an I slice can include or consist of coding blocks for the luma component or coding blocks for both chroma components, and a CU in a P or B slice can always include or consist of coding blocks for all three color components, unless the video is monochrome.

[0076] Turning now to transforms and transform blocks, the transforms performed during the decoding of the video stream can be the inverse of the transforms performed during the encoding of the video stream, and are sometimes referred to as "inverse transforms". For simplicity, the transforms described in this article are referred to as "transforms" regardless of whether they are performed during encoding or decoding.

[0077] Multiple transform dimensions (e.g., ranging from 4 to 64 points per dimension) and transform shapes (e.g., squares or rectangles with width / height ratios of 2:1 / 1:2 and 4:1 / 1:4) can be utilized. It is worth noting that when the encoder component applies a transform, the decoder component performs the inverse transform. Therefore, in the following description, the transform described in the context of the decoder component can be the inverse of the transform applied on the encoder side.

[0078] The two-dimensional transformation process may involve the use of a hybrid transform kernel (e.g., composed of different one-dimensional transforms for each dimension of the encoded residual block). The main one-dimensional transform may include at least one of the following: a) 4-point, 8-point, 16-point, 32-point, and 64-point Discrete Cosine Transform (DCT-2); b) 4-point, 8-point, and 16-point Asymmetric Discrete Sine Transform (DST-4, DST-7) and their inverted versions; or c) 4-point, 8-point, 16-point, and 32-point Identity Transform. Table 1 lists the basis functions for DCT-2 and Asymmetric DST, such as those used in AV1.

[0079]

[0080] Table 1 - Exemplary Principal Transformation Basis Functions The availability of hybrid transform kernels can be based on the transform block size and prediction mode. Exemplary dependencies are listed in Table 2 below, where “→” and “↓” represent horizontal and vertical dimensions, and “√” and “×” indicate the availability of kernels for that block size and prediction mode. IDTX (or IDT) stands for identity transform.

[0081]

[0082] Table 2 - Availability of Hybrid Transform Cores For chroma components, transform type selection can be performed implicitly. For intra-frame prediction residuals, the transform type can be selected based on the intra-frame prediction mode, as specified in Table 3. For inter-frame prediction residuals, the transform type can be selected based on the transform type selection of the co-occurring luma block. Therefore, for chroma components, transform type signaling in the bitstream may not be necessary.

[0083]

[0084] Table 3 - Transformation Type Selection for Chroma Intra-Frame Prediction Residuals Now let's turn to an example encoding and decoding using predictive blocks and residual blocks. Figure 5A The calculation of a prediction block according to some embodiments is illustrated. Figure 5A In the example, intra-frame prediction is performed on the current block 502 to generate prediction block 504. In some embodiments, inter-frame prediction is performed to generate prediction blocks. The current block 502 includes a set of samples (e.g., pixel blocks), and prediction block 504 includes a set of predicted values ​​corresponding to that set of samples. Figure 5B The calculation of residual blocks according to some embodiments is illustrated. For example... Figure 5B As shown, the prediction block 504 is subtracted from the current block 502 to generate a residual block 506 that includes a set of residuals. For example, the corresponding difference between each sample and the corresponding predicted value is calculated. Figure 5C The calculation of the reconstructed block according to some embodiments is shown. For example... Figure 5C As shown, residual block 506 undergoes one or more transforms and quantizations to generate a set of residual coefficients. This set of residual coefficients can be sent from the encoder component to the decoder component. This set of residual coefficients undergoes inverse quantization and inverse transform to generate reconstructed residual block 508. Reconstructed residual block 508 is combined with prediction block 504 (e.g., the reconstructed residuals of reconstructed residual block 508 are added to the predicted values ​​of prediction block 504) to generate reconstructed block 510 corresponding to the current block 502.

[0085] In some embodiments, separable transforms, such as those shown in Table 1, are applied to intra- and inter-residual samples. In some embodiments, an intra-quadratic transform (IST) scheme is customized for the video coding library (e.g., for transforming intra-residual blocks). Compared to the non-separable master transform, the IST scheme can efficiently capture directional patterns in intra-residual samples with lower complexity. In the IST scheme, the nominal intra-prediction angle can be used to classify the IST kernel.

[0086] Intra-frame residual samples can represent any directional texture pattern that can be captured more effectively by inseparable transforms. However, their use for larger block sizes is limited by the complexity of their implementation. Inseparable quadratic transform schemes (which can capture most directions but have lower complexity because they are applied only to the low-frequency coefficients of the separable master transform) can be applied to larger block sizes with lower complexity.

[0087] In some embodiments, the IST scheme is incorporated on top of the intra-prediction scheme. The IST scheme may include 12 sets of quadratic transforms, each with 3 cores. Table 4 shows exemplary quadratic transform set selection and the corresponding indexes used for transform set selection. The left column represents the intra-prediction modes with available transform cores, and the right column indicates the set index. For example, at the encoder, for each mode, the best core is selected from the set based on RDO and sent as a signal (4 symbols, excluding IST). In this example, at the decoder, the bitstream is parsed to obtain the cores to be used.

[0088]

[0089] Table 4 - Selection of the set of quadratic transformations In some embodiments, a set of quadratic transforms is derived based on the intra-frame prediction direction, and the kernel type within the set is explicitly signaled. In some embodiments, IST is enabled when both DCT-2 and ADST are used as the horizontal and vertical master transforms. In some embodiments, IST is enabled only for lumen intra-frame blocks. For example, depending on the block size, a 4×4 or 8×8 inseparable transform can be selected. If min(tx_width, tx_height) < 8, a 4×4 IST can be selected. For larger blocks where both tx_width and tx_height are greater than or equal to 8, an 8×8 IST can be used. Here, tx_width and tx_height correspond to the transform block width and height, respectively. The input to IST can be low-frequency master transform coefficients arranged in a zigzag scan order, which can be the default scan order. This facilitates more efficient decorrelation of adjacent low-frequency coefficients.

[0090] In some embodiments, both intra-frame coded blocks and inter-frame coded blocks can be further divided into multiple transform units (e.g., with a partition depth of up to 2 levels). In some embodiments, the application of IST is limited to the root of the transform partition tree structure (depth 0). This limitation allows for a reduction in overall coding time complexity (~50%) while minimizing the impact on compression efficiency (approximately 0.25% loss). In some embodiments using the IST scheme, context information is derived using the forward transform block size, thereby deriving the context for entropy coding of the kernel index. For rectangular transform blocks, the next smallest square size can be used.

[0091] In some embodiments, the IST scheme defines 14 sets of quadratic transforms, each with 3 kernels. The selection of the IST set can depend on the intra-prediction mode used for residual generation. Table 5 below describes the mapping between intra-prediction modes, main transform types, and IST set indices. The first column represents the intra-prediction mode with available kernels, the second column represents the main transform type, and the third column represents the set index. Depending on the block size, a 16-point or 64-point IST can be selected. If min(tx_width, tx_height) < 8, a 16-point IST can be selected. For larger blocks where both tx_width and tx_height are greater than or equal to 8, a 64-point IST can be used. Here, tx_width and tx_height correspond to the transform block width and height, respectively. Transform coefficients outside the region of application (RoA) of the IST (main transform coefficients only) can be set to zero.

[0092] Table 5 below shows the 14 sets of quadratic transforms that depend on two main transform types: DCT_DCT and ADST_ADST. Therefore, for one main transform type, only 7 different sets need to be signaled.

[0093]

[0094] Table 5 - Selection of the set of quadratic transformations In some embodiments, a probabilistic context for selection of each set is derived based on the intra-prediction mode. In some embodiments, the encoder component implicitly selects the IST set based on a predefined mapping between the intra-prediction mode and the IST set. In some embodiments, the encoder performs an additional search on all available IST sets (e.g., instead of checking only one IST set based on the intra-prediction mode), allowing the encoder to make rate-distortion optimized (RDO) decisions regarding the selection of the IST set.

[0095] In some embodiments, IST is enabled for inter-frame coded blocks. For example, the IST core used for intra-frame coded blocks is also applied to inter-frame coded blocks. In some embodiments, the IST core used for intra-frame coded blocks is reused for inter-frame coded blocks without modification or addition. In some embodiments, the above description of IST for intra-frame coded blocks also applies to inter-frame coded blocks. For example, the encoder can select from multiple sets and multiple cores within a set (e.g., 7 sets and 3 cores within a set). Cores and set indices can be explicitly signaled.

[0096] In some embodiments, IST is enabled for inter-frame coded blocks that use DCT_DCT or ADST_ADST as the primary transform. For example, for inter-frame blocks, the kernel index is only notified by signaling, and the set used corresponds to DC_PRED or set index 0.

[0097] Figure 6A This is a flowchart illustrating a video decoding method 600 according to some embodiments. Method 600 may be executed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having control circuitry and a memory storing instructions for execution by the control circuitry. In some embodiments, method 600 is executed by executing instructions stored in the computing system's memory (e.g., memory 314).

[0098] The system receives (602) a video stream (e.g., an encoded video sequence), the video stream comprising a set of one or more coded blocks (e.g., corresponding to one or more images) and a corresponding set of transform coefficients. The system selects (604) an indivisible quadratic transform from a set of one or more indivisible quadratic transforms based on index values ​​(e.g., index values ​​indicated by signals or derived index values). In some embodiments, the system selects a quadratic transform from a set of one or more quadratic transforms (e.g., separable and / or indivisible). In some embodiments, the system selects a master transform from a set of one or more master transforms (e.g., separable and / or indivisible). The system determines (606) the kernel size and kernel shape of the indivisible quadratic transform based on encoding information. In some embodiments, the system determines only one of the kernel size and kernel shape based on encoding information. The system decodes the set of coded blocks by applying the indivisible quadratic transform with the determined kernel size and kernel shape to the set of coded blocks (608).

[0099] In some embodiments, the encoding information is used to adaptively determine the size, shape (e.g., representing a subset of the complete kernel), and / or the scan order within the quadratic transform set of indivisible quadratic (or separable quadratic or indivisible quadratic or indivisible quadratic) transform kernel. The encoding information refers to any information known to the encoder and decoder when encoding the current block, such as flags indicated by signals or reconstructed information.

[0100] In some embodiments, the size of the indivisible quadratic transform kernel is different in the transform set corresponding to different master transform types. For example, the encoding information includes the master transform type. As an example, if DCT is the master transform, the kernel size M×N used can be a first size (e.g., 32×64), and if DST / ADST is the master transform, the kernel size can be a second size (e.g., 16×64 or 8×64), where the first size can be greater than or less than the second block size.

[0101] In some embodiments, the encoding information includes the transform block size. For example, if the transform block size is 8×8, the kernel size may be 32×64 (e.g., a numerical matrix) or 16×64 (e.g., a subset of a 32×64 matrix).

[0102] In some embodiments, the indivisible quadratic transform kernel size is different in the transform set corresponding to different intra-frame or inter-frame prediction modes. For example, the coding information includes the prediction mode. As an example, if the DC (or SMOOTH or PLANAR) prediction is an intra-frame mode, the kernel size M×N used can be a first size (e.g., 32×64), and for one or more other modes, the kernel size can be a second size (e.g., 16×64 or 8×64), where the first size can be larger or smaller than the second block size.

[0103] In some embodiments, the size of the indivisible quadratic transformation kernel is different for each kernel in the set. For example, in a transformation set with 3 kernels, the kernel size can be 8×64, 16×64, or 32×64.

[0104] In some embodiments, the transform kernel index and / or transform set for a block are signaled only when the EOB is greater than M. In the examples, possible values ​​for M include, but are not limited to, 1, 2, ..., 64. In some embodiments, since the value of M can be different for different transform kernels in the set, the sign size used to signal the kernel index is different (e.g., to select the kernel that covers the EOB value). In some embodiments, EOB groups are defined, and EOB groups are used as the context for signaling the indivisible transform kernel index and / or transform set. For example, EOB values ​​in the range 1-8, 9-16, 17-32, 33-64 can be mapped to EOB groups 0, 1, 2, and 3. In some embodiments, the encoded information includes the EOB and / or EOB groups of the current block and / or neighboring blocks.

[0105] In some embodiments, the region of principal transform coefficients to which the indivisible quadratic transform kernel can be applied depends on the coding information. For example, for a first indivisible quadratic transform kernel, the upper left M×N rectangular principal coefficient region can be the input of the forward quadratic transform or the output of the inverse quadratic transform. As another example, for a second indivisible quadratic transform kernel, the low-frequency rectangular principal coefficients located in the upper left triangular region can be the input of the forward quadratic transform or the output of the inverse quadratic transform.

[0106] In some embodiments, the scan order applied to an indivisible quadratic transform kernel depends on the encoding information. For example, for a first indivisible quadratic transform kernel, the scan order is based on the scan order of the M×N coefficient group, and for a second indivisible quadratic transform kernel, the scan order is based on the scan order of the zigzag / diagonal encoded blocks.

[0107] In some embodiments, the principal transform coefficient regions where the quadratic transform is not applied are selectively zeroed based on the coding information. For example, for a first indivisible quadratic transform kernel, a first number of principal transform coefficients are zeroed, while for a second indivisible quadratic transform kernel, a second number of principal transform coefficients are zeroed. The first and second numbers may differ depending on the coding information.

[0108] Figure 6B This is a flowchart illustrating a method 650 for encoding video according to some embodiments. Method 650 may be executed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having control circuitry and memory storing instructions for execution by the control circuitry. In some embodiments, method 650 is executed by executing instructions stored in the memory of the computing system (e.g., memory 314). In some embodiments, method 650 is executed by the same system as method 600 described above.

[0109] The system receives (652) video data (e.g., a source video sequence), which comprises one or more blocks. The system selects (654) an indivisible quadratic transform from one or more indivisible quadratic transforms. In some embodiments, the system selects a quadratic transform from one or more quadratic transforms (e.g., separable and / or indivisible). In some embodiments, the system selects a master transform from one or more master transforms (e.g., separable and / or indivisible master transforms). The system determines (656) the kernel size and kernel shape of the indivisible quadratic transform based on encoding information. In some embodiments, the system determines only one of the kernel size and kernel shape based on encoding information. The system encodes the block by applying the indivisible quadratic transform with the determined kernel size and kernel shape to the block (658). As previously mentioned, the encoding process can mirror the decoding process described herein (e.g., applying transforms). For brevity, these details are not repeated here.

[0110] although Figure 6A and 6B Multiple logical levels are shown in a specific order, but levels independent of the order can be reordered, and other levels can be combined or separated. Some reorderings or other groupings not specifically mentioned will be obvious to those skilled in the art, and therefore the orderings and groupings presented herein are not exhaustive. Furthermore, it should be recognized that these levels can be implemented in hardware, firmware, software, or any combination thereof.

[0111] Now turn to some exemplary embodiments. In the embodiments described below, an indivisible quadratic transformation is depicted. However, in some embodiments, the transformation is a divisible quadratic transformation and / or a principal transformation.

[0112] (A1) In one aspect, some embodiments include a video decoding method (e.g., method 600). In some embodiments, the method is performed at a computing system (e.g., server system 112) having memory and control circuitry. In some embodiments, the method is performed at an encoding / decoding module (e.g., encoding / decoding module 320). In some embodiments, the method is performed at a source encoding component (e.g., source encoder 202), an encoding engine (e.g., encoding engine 212), and / or an entropy encoder (e.g., entropy encoder 214). The method includes: (i) receiving a video bitstream (e.g., an encoded video sequence), the video bitstream including a set of coded blocks (e.g., corresponding to one or more pictures) and a corresponding set of transform coefficients; (ii) selecting an indivisible quadratic transform from a set of one or more indivisible quadratic transforms based on an index value (e.g., an index value notified by a signal or derived); (iii) determining the kernel size and / or kernel shape of the indivisible quadratic transform based on encoding information; and (iv) decoding the set of coded blocks by applying the indivisible quadratic transform having the determined kernel size and kernel shape to the set of coded blocks. For example, encoding information can be used to adaptively determine the transform kernel size, shape, and / or scan order within a transform set. In some embodiments, the transform is an indivisible quadratic transform. Encoding information can refer to any information known to both the encoder and decoder when encoding the block, such as flags indicated by signals and reconstructed information. In some embodiments, the kernel size and shape are determined based on the encoding information for the indivisible quadratic transform. In some embodiments, the kernel size and shape are determined based on the encoding information for the master transform. In some embodiments, different kernel sizes are used for different block sizes. In some embodiments, the kernel size and shape are selected from a corresponding set of kernel sizes and shapes available to the decoder.

[0113] (A2) In some embodiments of A1, the encoding information includes the main transform type for the set of coded blocks. For example, the size of the indivisible quadratic transform kernel may be different in transform sets corresponding to different main transform types.

[0114] (A3) In some embodiments of A2: (i) when the primary transform type includes a Discrete Cosine Transform (DCT), a first kernel size is selected for the set of coding blocks; and (ii) when the primary transform type includes a Discrete Sine Transform (DST), a second kernel size is selected for the set of coding blocks, the second kernel size being different from the first kernel size. For example, if the DCT is the primary transform, the kernel size (M×N) used can be the first size (e.g., 32×64). As another example, if the DST / ADST is the primary transform, the kernel size can be the second size (e.g., 16×64 or 8×64). The first size can be larger or smaller than the second size.

[0115] (A4) In some embodiments of any of A1 to A3, the encoding information includes the transform block size. For example, if the transform block size is 8×8, the kernel size may be a 32×64 numerical matrix or a 16×64 numerical matrix (e.g., a subset of a 32×64 matrix).

[0116] (A5) In some embodiments of any of A1 to A4, the coding information includes prediction mode information for the set of coding blocks. For example, the size of the indivisible quadratic transform kernel may be different in the transform set corresponding to different intra-frame or inter-frame prediction modes.

[0117] (A6) In some embodiments of A5: (i) when the prediction mode information includes a non-directional intra-prediction mode, a first kernel size is selected for the set of coding blocks; and (ii) when the prediction mode information does not include a non-directional intra-prediction mode, a second first kernel size is selected for the set of coding blocks. For example, if DC (or SMOOTH or PLANA) prediction is an intra-prediction mode, the kernel size (M×N) used can be a first size (e.g., 32×64), and for one or more other modes, the kernel size can be a second size (e.g., 16×64 or 8×64). The first size can be larger or smaller than the second size.

[0118] (A7) In some embodiments of any of A1 to A6, the encoded information includes a selected kernel for the indivisible quadratic transform. For example, the kernel size for the indivisible quadratic transform may be different for each kernel in the set. As an example, in a transform set with 3 kernels, the kernel size may be 8×64, 16×64, or 32×64.

[0119] (A8) In some embodiments of any of A1 to A7, the method further includes: (i) determining whether the end-of-block (EOB) value of the set of coded blocks satisfies one or more criteria; (ii) when the EOB value satisfies the one or more criteria, obtaining the index value from the video bitstream; and (iii) when the EOB value does not satisfy one or more criteria, abandoning the acquisition of the index value from the video bitstream (e.g., deriving the index value). For example, the indivisible quadratic transform kernel index and / or transform set are signaled for the block only when the EOB is greater than (or equal to) M, where the value of M may include 1, 2, ... 64.

[0120] (A9) In some embodiments of A8, the index value is obtained from symbols in the video bitstream, where the size of the symbol corresponds to the kernel size. For example, because the value of M can be different for different transform kernels in the set, the size of the symbol used to signal the kernel index can be different (e.g., selecting a kernel that covers the EOB value).

[0121] (A10) In some implementations of A8 or A9, one or more criteria include criteria based on EOB groups of EOB values. For example, EOB groups can be defined, and EOB groups are used as context for signaling the kernel index and / or transform set of indivisible quadratic transforms. As an example, EOB values ​​in the ranges 1-8, 9-16, 17-32, and 33-64 can be mapped to EOB groups 0, 1, 2, and 3. In this way, EOB groups can be used as contextual information for the kernel index and / or transform set.

[0122] (A11) In some embodiments of any of A1 to A10, the method further includes: identifying a region of the main transform, wherein an indivisible quadratic transform is applied to the identified region. For example, the region to which the main transform coefficients to which the indivisible quadratic transform kernel can be applied may depend on any coding information.

[0123] (A12) In some embodiments of A11, the identified region includes the upper left rectangular region of the main transform. For example, for the first indivisible quadratic transform kernel, the upper left M×N rectangular main coefficient region may be the input of the forward quadratic transform or the output of the inverse quadratic transform.

[0124] (A13) In some embodiments of A11, the identified region includes the upper left triangular region of the principal transform. For example, for the second indivisible quadratic transform kernel, the low-frequency rectangular principal coefficients located in the upper left triangular region can be the input of the forward quadratic transform or the output of the inverse quadratic transform.

[0125] (A14) In some implementations of any of A11 to A13: (i) the main transform includes the identified region and one or more other regions; and (ii) the method further includes: selectively setting coefficients in one or more other regions to zero based on coding information. For example, whether to apply zeroing (for the main transform coefficient region where no quadratic transform is applied) can depend on any coding information. As an example, for a first indivisible quadratic transform kernel, a first number of main transform coefficients are set to zero, while for a second indivisible quadratic transform kernel, a second number of main transform coefficients are set to zero. The first number and the second number can differ based on any coding information.

[0126] (A15) In some embodiments of any of A1 to A14, the method further includes determining a scan order for the indivisible quadratic transform based on the encoding information, wherein the indivisible quadratic transform is applied according to the scan order. For example, the scan order applied in the indivisible quadratic transform kernel can depend on any encoding information.

[0127] (A16) In some embodiments of A15, the scanning order is one of the following: a scanning order based on coefficient groups, a diagonal scanning order, or a zigzag scanning order. For example, for a first indivisible quadratic transform kernel, the scanning order is based on the scanning order of M×N coefficient groups, and for a second indivisible quadratic transform kernel, the scanning order is based on the scanning order of zigzag or diagonal coded blocks.

[0128] (B1) In another aspect, some embodiments include a video encoding method (e.g., method 650). In some embodiments, the method is performed at a computing system (e.g., server system 112) having memory and control circuitry. In some embodiments, the method is performed at an encoding module (e.g., encoding / decoding module 320). The method includes: (i) receiving video data (e.g., a source video sequence), the video data comprising a set of blocks (e.g., corresponding to one or more pictures); (ii) selecting an indivisible quadratic transform from a set of one or more indivisible quadratic transforms; (iii) determining the kernel size and kernel shape of the indivisible quadratic transform based on encoding information; and (iv) encoding the set of blocks by applying the indivisible quadratic transform having the determined kernel size and kernel shape to the set of blocks.

[0129] (B2) In some embodiments of B1, the encoding information includes the main transform type for the set of blocks and / or the prediction mode information for the set of blocks.

[0130] (C1) In another aspect, some embodiments include a visual media data processing method. In some embodiments, the method is performed at a computing system (e.g., server system 112) having memory and control circuitry. In some embodiments, the method is performed at an encoding module (e.g., encoding module 320). The method includes: (i) acquiring a source video sequence comprising multiple frames; and (ii) performing a conversion between the source video sequence and a video stream of visual media data according to format rules. The video stream comprises a set of coded blocks and a corresponding set of transform coefficients. The format rules specify: (a) selecting an indivisible quadratic transform for the set of coded blocks from a set of one or more indivisible quadratic transforms, and (b) determining the kernel size and kernel shape for the selected indivisible quadratic transform based on encoding information.

[0131] On the other hand, some embodiments include a computing system (e.g., server system 112) that includes control circuitry (e.g., control circuitry 302) and a memory (e.g., memory 314) coupled to the control circuitry. The memory stores one or more sets of instructions configured to be executed by the control circuitry, the one or more sets of instructions including instructions for performing any of the methods described herein (e.g., A1 to A16, B1 to B2 and C1 above).

[0132] In another aspect, some embodiments include a non-transitory computer-readable storage medium storing one or more sets of instructions for execution by control circuitry of a computing system, the one or more sets of instructions including instructions for performing any of the methods described herein (e.g., A1 to A16, B1 to B2 and C1 above).

[0133] Unless otherwise stated, any syntax element described herein may be a High-Level Syntax (HLS). As used herein, the HLS is signaled at a level higher than the block level. For example, the HLS may correspond to the sequence level, frame level, slice level, or tile level. As another example, HLS elements may be signaled in the Video Parameter Set (VPS), Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Adaptive Parameter Set (APS), slice header, picture header, tile header, and / or CTU header.

[0134] It should be understood that although the terms “first,” “second,” etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. The terminology used herein is intended only to describe particular embodiments and is not intended to limit the claims. As used in the description of embodiments and the appended claims, the singular forms “a,” “an,” and “the” are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will be further understood that, when used in this specification, the terms “comprising” and / or “including” specify the presence of the described features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof.

[0135] As used herein, the term "when" can be interpreted, depending on the context, as meaning "if," "at," "in response to determination," "according to determination," or "in response to detection," the described prerequisite is true. Similarly, depending on the context, the phrases "if determination [the described prerequisite is true]," "if [the described prerequisite is true]," or "when [the described condition prerequisite is true]" can be interpreted as meaning "in the case of determination," "in response to determination," "according to determination," "in the case of detection," or "in response to detection," the stated prerequisite is true. As used herein, N refers to a variable. Unless explicitly stated otherwise, different instances of N may refer to the same number (e.g., the same integer value, such as the number 2) or different numbers.

[0136] The above description (for illustrative purposes only) is based on specific embodiments. However, the foregoing illustrative discussion is not intended to exhaustively limit the precise form covered by the claims. In view of the foregoing teachings, many variations and modifications are possible. The selected embodiments are provided to best explain the operating principles and practical applications, thereby enabling those skilled in the art to understand.

Claims

1. A method for video decoding, the method being performed at a computing system having a memory and one or more processors, the method comprising: Receive a video stream, the video stream comprising a set of coded blocks and a corresponding set of transform coefficients; Based on the index value, select an indivisible quadratic transformation from a set of one or more indivisible quadratic transformations; Based on the encoded information, the kernel size and kernel shape of the indivisible quadratic transformation are determined; as well as The set of coded blocks is decoded by applying the indivisible quadratic transform with a defined kernel size and shape to the set of coded blocks.

2. The method according to claim 1, wherein, The encoding information includes the main transform type used for the set of encoding blocks.

3. The method according to claim 2, wherein: When the main transform type includes Discrete Cosine Transform (DCT), a first kernel size is selected for the set of coding blocks; as well as When the main transform type includes Discrete Sine Transform (DST), a second kernel size is selected for the set of coded blocks, the second kernel size being different from the first kernel size.

4. The method according to claim 1, wherein, The encoded information includes the transform block size.

5. The method according to claim 1, wherein, The encoding information includes prediction mode information for the set of encoding blocks.

6. The method according to claim 5, wherein: When the prediction mode information includes a non-directional intra-frame prediction mode, a first kernel size is selected for the set of coding blocks; and When the prediction mode information does not include non-directional intra-frame prediction modes, a second first kernel size is selected for the set of coding blocks.

7. The method according to claim 1, wherein, The encoded information includes the selected kernel for the indivisible quadratic transformation.

8. The method according to claim 1, wherein, The method further includes: Determine whether the End-of-Block (EOB) value of the set of coded blocks meets one or more criteria; When the EOB value meets one or more of the criteria, the index value is obtained from the video stream; and If the EOB value does not meet one or more of the criteria, the acquisition of the index value from the video stream is abandoned.

9. The method according to claim 8, wherein, The index value is obtained from symbols in the video bitstream, and the size of the symbol corresponds to the kernel size.

10. The method according to claim 8, wherein, The one or more criteria include criteria based on the EOB group used for the EOB value.

11. The method according to claim 1, wherein, The method further includes: identifying the region of the main transformation, wherein the indivisible quadratic transformation is applied to the identified region.

12. The method according to claim 11, wherein, The identified region includes the upper left rectangular region of the main transformation.

13. The method according to claim 11, wherein, The identified region includes the upper left triangular region of the main transformation.

14. The method according to claim 11, wherein, The main transformation includes the identified region and one or more other regions; as well as The method further includes selectively setting coefficients in one or more other regions to zero based on the encoding information.

15. The method according to claim 1, wherein, The method further includes: determining the scanning order of the indivisible quadratic transform based on the encoded information, wherein the indivisible quadratic transform is applied according to the scanning order.

16. The method according to claim 15, wherein, The scanning order is one of the following: a scanning order based on coefficient groups, a diagonal scanning order, or a zigzag scanning order.

17. A computing system, comprising: Control circuit; Memory; as well as One or more instruction sets stored in the memory and configured to be executed by the control circuitry, the one or more instruction sets including instructions for the following operations: Receive video data consisting of a set of blocks; Choose an indivisible quadratic transformation from a set of one or more indivisible quadratic transformations; Based on the encoded information, the kernel size and kernel shape for the indivisible quadratic transformation are determined; as well as The set of blocks is encoded by applying the indivisible quadratic transformation having a defined kernel size and shape to the set of blocks.

18. The computing system according to claim 17, wherein, The encoding information includes the main transform type for the set of blocks.

19. The computing system according to claim 17, wherein, The encoded information includes prediction mode information for the set of blocks.

20. A non-transitory computer-readable storage medium storing one or more instruction sets configured to be executed by a computing device having control circuitry and memory, the one or more instruction sets including instructions for: Acquire a source video sequence comprising multiple frames; and The conversion between the source video sequence and the video bitstream of the visual media data is performed according to the format rules. in, The video stream includes a set of coded blocks and a corresponding set of transform coefficients; and The format rules specify: selecting an indivisible quadratic transform from a set of one or more indivisible quadratic transforms for the set of coded blocks, and determining the kernel size and kernel shape for the selected indivisible quadratic transform based on the encoding information.

Images