selectively signaling inter-frame transform type and / or transform set

Abstract

An example method of video decoding includes receiving a video bitstream including at least two blocks corresponding to at least two pictures, wherein the at least two blocks include a first block. When the first block is an inter-coded block, the method includes determining whether to signal transform information for the first block based on coding information including an end of block (EOB) value, wherein the transform information includes at least one of a transform type and a transform set. The method further includes parsing the transform information from at least one indicator of the video bitstream when the transform information for the first block is signaled, and deriving the transform information without parsing the at least one indicator when the transform information for the first block is not signaled. The method further includes decoding the first block by applying an inverse transform to the first block using the transform information.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 626,805, filed January 30, 2024, entitled "Selective Signaling of Inter Transform Types and / or Transform Sets," and is a successor to and claims priority to U.S. Patent Application No. 18 / 781,827, filed July 23, 2024, entitled "Selective Signaling of Inter Transform Types and / or Transform Sets." Technical Field

[0003] This application relates to video encoding and decoding technologies, including but not limited to transform encoding and decoding for signaling notifications applied to prediction residuals. Background Technology

[0004] Digital video is supported by a variety of electronic devices, such as digital televisions, laptops or desktop computers, tablets, digital cameras, digital recording devices, digital media players, video game consoles, smartphones, video conferencing equipment, and video streaming devices. Electronic devices transmit and receive digital video data via communication networks, or otherwise transmit digital video data, and / or store digital video data on storage devices. Due to the limited bandwidth capacity of communication networks and the limited memory resources of storage devices, video data can be compressed using video coding according to one or more video coding standards before it is transmitted or stored. Video coding can be performed by hardware and / or software on electronic / client devices or servers providing cloud services.

[0005] Video coding typically employs predictive methods (such as inter-frame prediction and intra-frame prediction), which utilize the inherent redundancy in video data. Video coding aims to compress video data into a form using a lower bitrate while avoiding or minimizing video quality degradation. Several video codec standards have been developed. For example, High-Efficiency Video Coding (HEVC / H.265) is a video compression standard designed as part of the MPEG-H project. The ITU-T and ISO / IEC released the HEVC / H.265 standard in 2013 (Revision 1), 2014 (Revision 2), 2015 (Revision 3), and 2016 (Revision 4). Versatile Video Coding (VVC / H.266) is intended as a successor to HEVC. The ITU-T and ISO / IEC released the VVC / H.266 standard in 2020 (Revision 1) and 2022 (Revision 2). AOMedia Video 1 (AV1) is an open video coding format designed as an alternative to HEVC. On January 8, 2019, a validated version of the specification (including Errata Table 1) was officially released. Summary of the Invention

[0006] Among other things, this application describes a set of methods for video (image) compression, including: a method for selectively signaling and parsing transform information (such as transform type and transform set) of blocks (e.g., inter-frame coded blocks) based on encoded information known to the encoder and decoder. The transform information may correspond to a quadratic transform (e.g., a separable quadratic transform or a non-separable quadratic transform). For example, the encoded information may include end-of-block (EOB) information, block size information, partition depth, and / or master transform information. The advantage of selectively signaling transform information compared to systems without quadratic transforms is improved decoding quality, while the signaling cost is increased compared to systems that always signal transform information.

[0007] According to some embodiments, a video decoding method includes (i) receiving a video bitstream (e.g., an encoded video sequence) comprising at least two blocks corresponding to at least two images, wherein the at least two blocks include a first block; (ii) when the first block is an inter-frame coded block: determining whether to signal transform information of the first block based on encoded information including block end-of-frame (EOB) values, wherein the transform information includes at least one transform type and transform set; (iii) when the transform information of the first block is signaled, parsing the transform information from at least one indicator of the video bitstream; (iv) when the transform information of the first block is not signaled, deriving the transform information without parsing the at least one indicator; and (v) decoding the first block by applying an inverse transform to the first block using the transform information.

[0008] According to some embodiments, a video coding method includes: (i) receiving video data comprising a set of blocks (e.g., a source video sequence), wherein the set of blocks includes a first block; (ii) when the first block is an inter-frame coded block: determining whether to signal transform information of the first block based on coding information including block end-of-frame (EOB) values, wherein the transform information includes at least one transform type and transform set; (iii) when signaling transform information of the first block is required, signaling the transform information via a video bitstream; (iv) when not signaling transform information of the first block is not required, abandoning the signaling of the transform information; and (v) encoding the first block by applying a transform to the first block using the transform information.

[0009] According to some embodiments, a bitstream conversion method includes: (i) obtaining a source video sequence comprising at least two frames; and (ii) converting between the source video sequence and a video bitstream of visual media data according to a format rule. The video bitstream includes: at least two blocks comprising a first block; and at least one identifier indicating the conversion information when conversion information of the first block is signaled. The format rule specifies that the conversion information of the first block is selectively signaled according to encoded information including an end-of-block (EOB) value corresponding to the first block.

[0010] According to some embodiments, a computing system, such as a streaming system, server system, personal computer system, or other electronic device, is provided. The computing system includes control circuitry and a memory storing at least one set of instructions. The at least one set of instructions includes instructions for performing any of the methods described herein. In some embodiments, the computing system includes an encoder component and a decoder component (e.g., a code converter).

[0011] According to some embodiments, a non-volatile computer-readable storage medium is provided. This non-volatile computer-readable storage medium stores at least one set of instructions executable by a computing device. The at least one set of instructions includes instructions for performing any of the methods described herein.

[0012] Therefore, at least two devices and systems employing methods for encoding and decoding video are disclosed. These methods, devices, and systems can complement or replace conventional methods, devices, and systems used for video encoding / decoding.

[0013] The features and advantages described in the specification are not necessarily all-encompassing. In particular, some additional features and advantages will be apparent to those skilled in the art in light of the accompanying drawings, specification, and claims provided in this disclosure. Furthermore, it should be noted that the language used in the specification has been chosen primarily for readability and instructional purposes, and is not necessarily selected to depict or limit the subject matter described herein. Attached Figure Description

[0014] To gain a more detailed understanding of this disclosure, reference can be made to the features of various embodiments, some of which are illustrated in the accompanying drawings. However, the drawings only illustrate relevant features of this disclosure and are therefore not necessarily limiting, as those skilled in the art will understand upon reading this disclosure that the specification may include other valid features.

[0015] Figure 1 A block diagram of an example communication system according to some embodiments is shown.

[0016] Figure 2A A block diagram of example elements of an encoder component according to some embodiments is shown.

[0017] Figure 2B A block diagram of example elements of a decoder component according to some embodiments is shown.

[0018] Figure 3 A block diagram of an example server system according to some embodiments is shown.

[0019] Figures 4A to 4D An example encoded tree structure according to some embodiments is shown.

[0020] Figures 5A to 5C Example prediction blocks, residual blocks, and reconstructed blocks are shown according to some embodiments.

[0021] Figure 6A An example video decoding process according to some embodiments is shown.

[0022] Figure 6B An example video encoding process according to some embodiments is shown.

[0023] By convention, the features shown in the accompanying drawings are not necessarily drawn to scale, and the same reference numerals may be used to denote the same features throughout the specification and the accompanying drawings. Detailed Implementation

[0024] This disclosure describes a set of methods for video (image) compression, including transform coding methods that selectively signal / parse transform information for transform coding (e.g., for intra-secondary transforms). In some embodiments, transform information is selectively signaled / parsed based on encoded information known to both the encoder and decoder components. The encoded information may correspond to previously decoded blocks and / or the current block. For the current block, the encoded information may include one or more of the following: EOB information, block size information, master transform information, and / or partitioning information. Applying intra-secondary transforms (ISTs) to inter blocks can improve coding efficiency compared to systems that restrict ISTs (described in detail below). Selectively signaling / parses transform information (e.g., for ISTs) to only signal / parse applicable blocks, compared to systems that always signal transform information, improves signaling cost (reduces bandwidth usage).

[0025] Example systems and devices

[0026] Figure 1 A block diagram of a communication system 100 according to some embodiments is shown. The communication system 100 includes a source device 102 and a plurality of electronic devices 120 (e.g., electronic devices 120-1 to 120-m) that are communicatively connected to each other via one or more networks. In some embodiments, the communication system 100 is a streaming system, for example, for video-enabled applications such as video conferencing applications, digital television applications, and media storage and / or distribution applications.

[0027] Source device 102 includes a video source 104 (e.g., a camera assembly or media storage) and an encoder assembly 106. In some embodiments, the video source 104 is a digital camera (e.g., configured to create an uncompressed video sample stream). The encoder assembly 106 generates one or more encoded video bitstreams from the video stream. The video stream from video source 104 may have a larger data volume compared to the encoded video bitstream 108 generated by encoder assembly 106. Because the encoded video bitstream 108 has less data volume than the video stream from the video source, it requires less transmission bandwidth and storage space compared to the video stream from video source 104. In some embodiments, source device 102 does not include encoder assembly 106 (e.g., configured to transmit uncompressed video to one or more networks 110).

[0028] One or more networks 110 represent any number of networks that transmit information between source device 102, server system 112, and / or electronic device 120, including, for example, wired (wired connections) and / or wireless communication networks. One or more networks 110 may exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and / or the Internet.

[0029] One or more networks 110 include a server system 112 (e.g., a distributed / cloud computing system). In some embodiments, the server system 112 is a streaming server (e.g., configured to store and / or distribute video content, such as an encoded video stream from source device 102), or includes a streaming server. The server system 112 includes a codec component 114 (e.g., configured to encode and / or decode video data). In some embodiments, the codec component 114 includes an encoder component and / or a decoder component. In various embodiments, the codec component 114 is instantiated as hardware, software, or a combination thereof. In some embodiments, the codec component 114 is configured to decode an encoded video bitstream 108 and re-encode the video data using different encoding standards and / or methods to generate encoded video data 116. In some embodiments, the server system 112 is configured to generate multiple video formats and / or encodings from the encoded video bitstream 108. In some embodiments, the server system 112 operates as a Media-Aware Network Element (MANE). For example, server system 112 can be configured to prune encoded video bitstream 108 to tailor potentially different bitstreams for one or more devices in electronic device 120. In some embodiments, MANE is provided separately from server system 112.

[0030] Electronic device 120-1 includes a decoder component 122 and a display 124. In some embodiments, the decoder component 122 is configured to decode encoded video data 116 to generate an output video stream that can be presented on a display or other type of rendering device. In some embodiments, one or more electronic devices 120 do not include a display component (e.g., a communication connection to an external display device and / or include media storage). In some embodiments, electronic device 120 is a streaming client. In some embodiments, electronic device 120 is configured to access server system 112 to obtain encoded video data 116.

[0031] The source device and / or multiple electronic devices 120 are sometimes referred to as “terminal devices” or “user devices”. In some embodiments, the source device 102 and / or one or more electronic devices 120 may be instances of server systems, personal computers, portable devices (e.g., smartphones, tablets, or laptops), wearable devices, video conferencing equipment, and / or other types of electronic devices.

[0032] In an example operation of communication system 100, source device 102 transmits an encoded video bitstream 108 to server system 112. For example, source device 102 may encode a stream of images it captures. Server system 112 receives the encoded video bitstream 108 and decodes and / or encodes it using codec component 114. For example, server system 112 may encode video data to be more suitable for network transmission and / or storage. Server system 112 may transmit encoded video data 116 (e.g., one or more encoded video bitstreams) to one or more electronic devices 120. Each electronic device 120 may decode the encoded video data 116 and selectively display video images.

[0033] Figure 2A A block diagram of example elements of an encoder component 106 according to some embodiments is shown. The encoder component 106 receives video data (e.g., a source video sequence) from a video source 104. In some embodiments, the encoder component includes a receiver (e.g., transceiver) component configured to receive the source video sequence. In some embodiments, the encoder component 106 receives the video sequence from a remote video source (e.g., a video source belonging to a different device than the encoder component 106). The video source 104 may provide the source video sequence as a digital video sample stream, which may have any suitable bit depth (e.g., 8-bit, 10-bit, or 12-bit), any color space (e.g., BT.601 YCrCB or RGB), and any suitable sampling structure (e.g., YCrCb 4:2:0 or YCrCb 4:4:4). In some embodiments, the video source 104 is a storage device storing previously captured / prepared video. In some embodiments, the video source 104 is a camera that captures local image information as a video sequence. Video data can be provided as multiple individual images that, when viewed sequentially, present a motion effect. The images themselves can be organized as a spatial array of pixels, where each pixel may contain one or more samples depending on the sampling structure, color space, etc. The relationship between pixels and samples will be readily understood by those skilled in the art.

[0034] Encoder component 106 is configured to encode and / or compress images of a source video sequence into an encoded video sequence 216, either in real-time or under other time constraints, as required by the application. In some embodiments, encoder component 106 is configured to perform a conversion between the source video sequence and a visual media data bitstream (e.g., a video bitstream). One function of controller 204 is to control an appropriate encoding rate. In some embodiments, controller 204 controls and is functionally coupled to other functional units as described below. Parameters set by controller 204 may include rate control-related parameters (e.g., image skipping, quantizers, and / or Lagrange multipliers for rate-distortion optimization techniques), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. Other functions of controller 204 can be readily identified by those skilled in the art as they may be related to encoder component 106 optimized for a particular system design.

[0035] In some embodiments, encoder component 106 is configured to operate in a coding loop. In a simplified example, the coding loop includes source encoder 202 (e.g., responsible for creating symbols, such as a symbol stream, based on the input image to be encoded and one or more reference images) and (local) decoder 210. Decoder 210 reconstructs the symbols to create sample data in a manner similar to that of the (remote) decoder (when compression between the symbols and the encoded video bitstream is lossless). The reconstructed sample stream (sample data) is input into reference image memory 208. Since the decoding result of the symbol stream is independent of the decoder location (local or remote), the contents of reference image memory 208 are also bit-precise between the local encoder and the remote encoder. Thus, the prediction portion of the encoder will interpret the same sample values ​​as the decoder interprets when using prediction during the decoding process as reference image samples.

[0036] The operation of decoder 210 can be the same as that of a remote decoder (such as decoder component 122), as will be discussed below. Figure 2B This will be described in detail. However, a brief reference is provided. Figure 2B Since symbols are available and the encoding / decoding of symbols to encoded video sequences by the entropy encoder 214 and the parser 254 can be lossless, the entropy decoding portion of the decoder component 122 (including the buffer memory 252 and the parser 254) may not be fully implemented in the local decoder 210.

[0037] Aside from parsing / entropy decoding, the decoder techniques described in this paper may exist in their corresponding encoders with essentially the same functionalities. Therefore, the focus of this paper is on decoder operations. Furthermore, since encoder techniques can be the inverse of decoder techniques, the description of encoder techniques can be simplified.

[0038] As part of its operation, the source encoder 202 can perform motion-compensated predictive coding, referencing one or more previously encoded frames in the video sequence (designated as reference frames), to predictively encode the input frame. In this way, the encoding engine 212 encodes the differences between pixel blocks of the input frame and pixel blocks of the reference frames that may be selected as prediction references for the input frame. The controller 204 can manage the encoding operations of the source encoder 202, including, for example, setting parameters and subgroup parameters for encoding the video data.

[0039] Decoder 210 decodes encoded video data of frames that may be designated as reference frames, based on symbols created by source encoder 202. The operation of encoding engine 212 can be a lossy process, which may have advantages. When encoded video data is processed by video decoder (… Figure 2A When decoded at (not shown), the reconstructed video sequence may be a copy of the source video sequence, but with some errors. Decoder 210 copies the decoding process that a remote video decoder may perform on the reference frame, and can store the reconstructed reference frame in reference image memory 208. In this way, encoder component 106 locally stores copies of the reconstructed reference frames that have the same content as the reconstructed reference frames that the remote video decoder would obtain (in the absence of transmission errors).

[0040] Predictor 206 can perform prediction search for encoding engine 212. That is, for a new frame to be encoded, predictor 206 can search the reference image memory 208 for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., which can serve as appropriate prediction references for the new image. Predictor 206 can operate on sample blocks pixel by pixel to find appropriate prediction references. Based on the search results obtained by predictor 206, the prediction references for the input image may come from multiple reference images stored in reference image memory 208.

[0041] The outputs of all the above-described functional units can be entropy encoded in the entropy encoder 214. The entropy encoder 214 performs lossless compression on the symbols according to techniques known to those skilled in the art (e.g., Huffman coding, variable-length coding, and / or arithmetic coding), converting the symbols generated by each functional unit into an encoded video sequence.

[0042] In some embodiments, the output of entropy encoder 214 is connected to a transmitter. The transmitter may be configured to cache the encoded video sequence created by entropy encoder 214 in preparation for transmission via communication channel 218. Communication channel 218 may be a hardware / software link connected to a storage device for storing encoded video data. The transmitter may be configured to combine encoded video data from source encoder 202 with other data to be transmitted (e.g., encoded audio data and / or auxiliary data streams (source not shown)). In some embodiments, the transmitter may transmit additional data along with the encoded video. Source encoder 202 may include such data as part of the encoded video sequence. Additional data may include temporal / spatial / signal-to-noise ratio (SNR) enhancement layers, other forms of redundant data (such as redundant pictures and slices), supplementary enhancement information (SEI) messages, visual usability information (VUI) parameter set fragments, etc.

[0043] Controller 204 can manage the operation of encoder component 106. During encoding, controller 204 can assign a specific encoding picture type to each encoded picture, which may affect the encoding technique applied to the corresponding picture. For example, a picture can be assigned as an intra-frame picture (I-picture), a predictive picture (P-picture), or a bidirectional predictive picture (B-picture). Intra-frame pictures can be encoded and decoded without using any other frames in the sequence as prediction sources. Some video codecs support different types of intra-frame pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Variations of I-pictures and their corresponding applications and characteristics are familiar to those skilled in the art, and therefore will not be repeated here. Predictive pictures can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most one motion vector and reference index to predict sample values ​​for each block. Bidirectional predictive pictures can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most two motion vectors and reference indexes to predict sample values ​​for each block. Similarly, at least two predictive pictures can use more than two reference pictures and associated metadata to reconstruct a single block.

[0044] The source image is typically spatially subdivided into at least two sample blocks (e.g., 4×4, 8×8, 4×8, or 16×16 sample blocks), and encoded block by block. These blocks can be predictively coded with reference to other (already coded) blocks, which are determined based on the coding assignment of the corresponding images applied to the blocks. For example, blocks of an I-image can be non-predictively coded, or the blocks can be predictively coded (spatial prediction or intra-frame prediction) with reference to already coded blocks of the same image. Pixel blocks of a P-image can be non-predictively coded with reference to a previously coded reference image via spatial prediction or temporal prediction. Blocks of a B-image can be non-predictively coded with reference to one or two previously coded reference images via spatial prediction or temporal prediction.

[0045] Video can be captured as at least two source images (video images) in a time series. Intra-frame image prediction (often simplified to intra-frame prediction) utilizes spatial correlations within a given image, while inter-frame image prediction utilizes (temporal or other) correlations between images. In an embodiment, a specific image being encoded / decoded is segmented into blocks, referred to as the current image. When a block in the current image resembles a reference block in a previously encoded and still buffered reference image in the video, the block in the current image can be encoded using a vector called a motion vector. The motion vector points to the reference block in the reference image, and when using at least two reference images, the motion vector may have a third dimension that identifies the reference image.

[0046] Encoder component 106 can perform encoding operations according to a predetermined video coding technique or standard (such as any of the techniques or standards described herein). In its operation, encoder component 106 can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

[0047] Figure 2B A block diagram of example elements of decoder component 122 according to some embodiments is shown. Figure 2B The decoder component 122 is coupled to the channel 218 and the display 124. In some embodiments, the decoder component 122 includes a transmitter coupled to the loop filter 256 and configured to transmit data to the display 124 (e.g., via a wired or wireless connection).

[0048] In some embodiments, decoder component 122 includes a receiver coupled to channel 218 and configured to receive data from channel 218 (e.g., via a wired or wireless connection). The receiver may be configured to receive at least one encoded video sequence to be decoded by decoder component 122. In some embodiments, decoding of each encoded video sequence is independent of other encoded video sequences. Each encoded video sequence may be received from channel 218, which may be a hardware / software link to a storage device storing the encoded video data. The receiver may receive encoded video data along with other data (e.g., encoded audio data and / or auxiliary data streams), which may be forwarded to their respective user entities (not depicted). The receiver may separate the encoded video sequences from other data. In some embodiments, the receiver receives additional (redundant) data accompanying the encoded video. The additional data may be included as part of at least one encoded video sequence. Decoder component 122 may use the additional data to decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0049] According to some embodiments, decoder component 122 includes buffer memory 252, parser 254 (sometimes also called entropy decoder), scaler / inverse transform unit 258, intra-frame prediction unit 262, motion compensation prediction unit 260, aggregator 268, loop filter unit 256, reference image memory 266, and current image memory 264. In some embodiments, decoder component 122 is implemented as an integrated circuit, a series of integrated circuits, and / or other electronic circuits. Decoder component 122 may be implemented at least partially in software.

[0050] Buffer memory 252 is coupled between channel 218 and parser 254 (e.g., to combat network jitter). In some embodiments, buffer memory 252 is decoupled from decoder component 122. In some embodiments, a separate buffer memory is provided between the output of channel 218 and decoder component 122. In some embodiments, a separate buffer memory is provided outside decoder component 122 (e.g., to combat network jitter) in addition to buffer memory 252 inside decoder component 122 (e.g., configured to handle playback timing). Buffer memory 252 may not be necessary or may be small when receiving data from a store / forward device with sufficient bandwidth and controllability or from an isochronous network. Buffer memory 252 may be necessary for use on best-effort packet networks such as the Internet. Buffer memory 252 may be relatively large and / or may have an adaptive size and may be implemented at least partially outside decoder component 122 in an operating system or similar component.

[0051] Parser 254 is configured to reconstruct symbols 270 from the encoded video sequence. Symbols may include, for example, information for managing the operation of decoder component 122, and / or information for controlling rendering devices such as display 124. Control information for at least one rendering device may be in the form of, for example, Supplemental Enhancement Information (SEI) messages or Video Availability Information (VUI) parameter set fragments (not depicted). Parser 254 parses (entropy decodes) the encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. Parser 254 may extract a set of subgroup parameters from the encoded video sequence for at least one subgroup of pixels in the video decoder based on at least one parameter corresponding to that set. Subgroups may include picture groups (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The parser 254 can also extract information such as transform coefficients, quantizer parameter values, and motion vectors from the encoded video sequence.

[0052] The reconstruction of symbol 270 may involve at least two distinct units, depending on the type of encoded video picture or its portion (e.g., inter-frame and intra-frame pictures, inter-frame and intra-frame blocks) and other factors. Which units are involved and how they are involved can be controlled by subgroup control information, which is parsed from the encoded video sequence by parser 254. For clarity, the flow of this subgroup control information between parser 254 and the following at least two units is not depicted.

[0053] The decoder component 122 can be conceptually subdivided into at least two functional units, which in some embodiments interact closely with each other and can be at least partially integrated with each other. However, for clarity, this document retains the conceptual subdivision of the functional units.

[0054] The scaler / inverse transform unit 258 receives quantized transform coefficients and control information (e.g., transform to be used, block size, quantization factor, and / or quantization scaling matrix) as at least one symbol 270 from the parser 254. The scaler / inverse transform unit 258 can output blocks comprising sample values, which can be input to the aggregator 268. In some cases, the output samples of the scaler / inverse transform unit 258 belong to intra-coded blocks; that is, blocks that do not use prediction information from previously reconstructed images but can use prediction information from previously reconstructed portions of the current image. Such prediction information can be provided by the intra-prediction unit 262. The intra-prediction unit 262 can use surrounding reconstructed information obtained from the current (partially reconstructed) image in the current image memory 264 to generate blocks with the same size and shape as the blocks in the reconstruction. The aggregator 268 can add the prediction information already generated by the intra-prediction unit 262 to the output sample information provided by the scaler / inverse transform unit 258 on a per-sample basis.

[0055] In other cases, the output samples of the scaler / inverse transform unit 258 belong to inter-frame encoded blocks that may have undergone motion compensation. In this case, the motion compensation prediction unit 260 can access the reference image memory 266 to obtain samples for prediction. After motion compensation of the obtained samples according to the symbols 270 belonging to the block, the aggregator 268 can add these samples to the output of the scaler / inverse transform unit 258 (referred to as residual samples or residual signals in this case) to generate output sample information. The address at which the motion compensation prediction unit 260 obtains the predicted samples from the reference image memory 266 can be controlled by motion vectors. Motion vectors can be provided to the motion compensation prediction unit 260 in the form of symbols 270, which may have, for example, X, Y, and reference image components. Motion compensation may also include interpolation of sample values ​​obtained from the reference image memory 266 when using subsampled precise motion vectors, motion vector prediction mechanisms, etc.

[0056] The output samples of aggregator 268 can undergo various loop filtering techniques in loop filter unit 256. Video compression techniques may include in-loop filtering controlled by parameters contained in the encoded video bitstream and provided to loop filter unit 256 as symbols 270 from parser 254, but may also be in response to metadata obtained during decoding of previous (in decoding order) portions of the encoded picture or encoded video sequence, and to sample values ​​obtained from previous reconstruction and loop filtering. The output of loop filter unit 256 may be a sample stream, which can be output to a rendering device such as display 124, or stored in reference picture memory 266 for use in future inter-frame prediction.

[0057] Once certain encoded images are reconstructed, they can be used as reference images for future predictions. Once an encoded image is reconstructed and has been identified as a reference image (e.g., by parser 254), the current reference image can become part of the reference image memory 266, and a new current image memory can be reallocated before starting the reconstruction of the next encoded image.

[0058] Decoder component 122 can perform decoding operations according to a predetermined video compression technique, which may be documented in a standard, such as any of the standards described herein. The encoded video sequence may conform to the syntax specified by the video compression technique or standard used; in this sense, it follows the syntax of the video compression technique or standard as specified in the video compression technique document or standard, particularly the syntax specified in the configuration file document. Furthermore, the complexity of the encoded video sequence can be within the range defined by the level of the video compression technique or standard to conform to certain video compression techniques or standards. In some cases, the level limits the maximum image size, maximum frame rate, maximum reconstruction sampling rate (e.g., measured in megasamples per second), maximum reference image size, etc. In some cases, the limitations set by the level can be further restricted by the HRD specifications and metadata used for the management of the hypothetical reference decoder (HRD) buffer, which are signaled in the encoded video sequence.

[0059] Figure 3 A block diagram of a server system 112 according to some embodiments is shown. The server system 112 includes control circuitry 302, at least one network interface 304, memory 314, a user interface 306, and at least one communication bus 312 for interconnecting these components. In some embodiments, the control circuitry 302 includes at least one processor (e.g., CPU, GPU, and / or DPU). In some embodiments, the control circuitry includes at least one field-programmable gate array (FPGA), a hardware accelerator, and / or at least one integrated circuit (e.g., an application-specific integrated circuit).

[0060] At least one network interface 304 can be configured to interface with at least one communication network (e.g., wireless, wired, and / or optical network). The communication network can be local, wide area, metropolitan area, vehicular and industrial, real-time, latency-tolerant, etc. Examples of communication networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., cable or wireless wide area digital television networks including cable television, satellite television, and terrestrial broadcast television, vehicular and industrial networks including CANbus, etc. Such communication can be unidirectional (e.g., broadcast television), unidirectional (e.g., to a CAN bus of some CAN bus device), or bidirectional (e.g., to other computer systems using a local area digital network or a wide area digital network). Such communication can include communication to at least one cloud computing network.

[0061] User interface 306 includes at least one output device 308 and / or at least one input device 310. Input device 310 may include at least one of the following: keyboard, mouse, touchpad, touch screen, data glove, joystick, microphone, scanner, camera, etc. At least one output device 308 may include at least one of the following: audio output device (e.g., speaker), visual output device (e.g., display or monitor), etc.

[0062] Memory 314 may include high-speed random access memory (such as DRAM, SRAM, DDR RAM, and / or other random access solid-state memory devices) and / or non-volatile memory (such as at least one disk storage device, optical disk storage device, flash memory device, and / or other non-volatile solid-state storage devices). Memory 314 may optionally include at least one storage device remote from control circuitry 302. Alternatively, memory 314 or at least one non-volatile solid-state storage device within memory 314 may include a non-volatile computer-readable storage medium. In some embodiments, memory 314 or the non-volatile computer-readable storage medium of memory 314 stores programs, modules, instructions, and data structures, or subsets or supersets thereof:

[0063] • Operating system 316, which includes programs for handling various basic system services and for performing hardware-related tasks;

[0064] • Network communication module 318, which is used to connect server system 112 to other computing devices via at least one network interface 304 (e.g., via wired and / or wireless connection);

[0065] • Encoding / decoding module 320, which performs various functions related to encoding and / or decoding data (e.g., video data). In some embodiments, encoding / decoding module 320 is an instance of codec component 114. Encoding / decoding module 320 includes, but is not limited to, at least one of the following:

[0066] Decoding module 322, which performs various functions related to decoding encoded data, such as those previously described concerning decoder component 122; and

[0067] The encoding module 340 performs various functions related to encoded data, such as those previously described concerning encoder component 106; and

[0068] • Image memory 352 is used to store images and image data, for example, for use by the encoding / decoding module 320. In some embodiments, image memory 352 includes at least one of the following: reference image memory 208, buffer memory 252, current image memory 264, and reference image memory 266.

[0069] In some embodiments, the decoding module 322 includes a parsing module 324 (e.g., configured to perform the various functions previously described regarding the parser 254), a transformation module 326 (e.g., configured to perform the various functions previously described regarding the scaler / inverse transform unit 258), a prediction module 328 (e.g., configured to perform the various functions previously described regarding the motion compensation prediction unit 260 and / or the intra-frame prediction unit 262), and a filter module 330 (e.g., configured to perform the various functions previously described regarding the loop filter 256).

[0070] In some embodiments, the encoding module 340 includes a code module 342 (e.g., configured to perform the various functions previously described regarding the source encoder 202 and / or encoding engine 212) and a prediction module 344 (e.g., configured to perform the various functions previously described regarding the predictor 206). In some embodiments, the decoding module 322 and / or the encoding module 340 includes... Figure 3 A subset of the modules shown. For example, both decoding module 322 and encoding module 340 use a shared prediction module.

[0071] Each of the modules described above, stored in memory 314, corresponds to a set of instructions for performing the functions described herein. These modules (e.g., instruction sets) do not need to be implemented as separate software programs, programs, or modules; therefore, various subsets of these modules can be combined or otherwise rearranged in various embodiments. For example, the codec module 320 may optionally not include separate decoding and encoding modules, but instead use the same set of modules to perform both sets of functions. In some embodiments, memory 314 stores a subset of the modules and data structures described above. In some embodiments, memory 314 stores additional modules and data structures not described above.

[0072] although Figure 3 A server system 112 according to some embodiments is shown, but Figure 3 This is intended more as a functional description of various features that can exist in at least one server system, rather than a structural diagram of the embodiments described herein. In practice, items shown individually may be combined, and some items may be separate. For example, Figure 3 Some items shown individually can be implemented on a single server, and a single item can be implemented by at least one server. The actual number of servers used to implement server system 112 and how features are allocated among them will vary depending on the implementation method and, optionally, in part, depend on the amount of data traffic handled by the server system during peak and average usage periods.

[0073] Example encoding techniques

[0074] The encoding processes and techniques described below can be performed at the aforementioned devices and systems (e.g., source device 102, server system 112, and / or electronic device 120). According to some embodiments, methods for selectively signaling and parsing transform information (e.g., methods for selectively signaling transform types and / or transform sets for inter-frame coded blocks) are described.

[0075] In this paper, "block" can refer to a coding tree block, a maximum coding block, a predefined fixed block size, a coding block, a prediction block, a residual block, or a transform block. An inter-mode coded block (or inter-block) refers to a block that uses an inter-prediction mode, or a combination of intra-inter-prediction modes. An inter-mode can also refer to a block coded using block vectors, which are used to acquire prediction blocks within the same frame, for example, using intra-block duplication. An intra-mode coded block (or intra-block) refers to a block that uses an intra-prediction mode, or a combination of intra-inter-prediction modes. An intra-mode list may correspond to a list of the most probable intra-prediction modes for the current block. Furthermore, the term "partition" may correspond to a block partition or a transform partition.

[0076] As described further below, a transform may correspond to a primary or secondary transform, and a separable or non-separable transform. The transform type may belong to the family of sinusoidal transforms (such as DCT, DST, the inverted version of DCT, and ADST), KLT, LGT, or trained kernels. A transform set is a combination of one or more transform types. Each entry in the transform set can be called a transform candidate. For each block, transform candidates selected from the transform set can be signaled or implicitly identified.

[0077] The End of Block (EOB) value corresponds to the position of the last valid coefficient after a given coefficient scan order in the encoded block.

[0078] Some embodiments include methods for signaling the selection of transform types for intra- and / or inter-coded blocks. Transforms may correspond to primary or secondary transforms, and separable or non-separable transforms. Transform types may belong to the sinusoidal transform family, KLT, or line-graph transforms (LGT). A transform set is a grouping of one or more transform types. The primary transform may belong to the sinusoidal transform family (DCT, DST, a flipped version of DCT, and a flipped version of ADST). DCT may refer to any transform using a transform kernel derived from a discrete cosine transform basis (e.g., DCT type 2), and DST / ADST may refer to any transform using a transform kernel derived from a discrete sine transform basis (e.g., DST type 4 or 7).

[0079] The primary transform may belong to the generalized line graph transform (LGT) family or be based on a trained kernel. The set of secondary transforms may be a grouping of one or more inseparable secondary transform kernel transform types. A unique or generic set of secondary transforms can be defined for each primary transform type and / or, intra-frame or inter-frame mode type.

[0080] Furthermore, an inseparable transform can refer to a primary transform applied directly to the residuals, or a secondary transform applied to the transform coefficient block generated by the primary transform. Transform kernels can be grouped into sets represented by set indices and kernel indices within those sets. An inseparable secondary transform can be a training kernel applied to the primary transform coefficients at the encoder, or applied to the dequantized coefficients at the decoder.

[0081] An inseparable quadratic transformation kernel can be considered as a set of basis vectors in a vector space. If represented as an M×N (M rows and N columns) matrix, then N corresponds to the dimension of the vector space, and M corresponds to the cardinality. Therefore, M×N can be used to represent the kernel size. Examples of kernel sizes include, but are not limited to, 64×64 samples, 32×64 samples, 16×64 samples, 8×84 samples, 4×64 samples, 16×16 samples, 8×16 samples, and 4×16 samples.

[0082] As described above, the EOB value corresponds to the position of the last valid (e.g., non-zero) coefficient after a given coefficient scan order in the coded block. For a given coefficient scan order, all coefficients outside the EOB position are zero. In some embodiments, if an inseparable quadratic transform kernel of size M×N is applied to the coded block, then the EOB value ≤ M.

[0083] Scanning order refers to the coefficient reorganization process of mapping a two-dimensional principal transform coefficient array to a one-dimensional principal transform coefficient array as input to a forward secondary transform. It can also refer to the backward coefficient reorganization process of mapping a one-dimensional secondary transform coefficient array back to a two-dimensional principal transform coefficient array.

[0084] Switch to block partitioning. Figures 4A to 4D An example encoded tree structure according to some embodiments is shown. Figure 4A As shown in the first coding tree structure (400), some coding schemes employ a quadtree partitioning structure, starting from a 64×64 level and dividing downwards to a 4×4 level. For example, there are some additional restrictions on 8×8 size blocks. Figure 4A In this context, partitions designated as "R" are recursively defined because the same partition tree is repeated at a lower percentage until the lowest level is reached. For example... Figure 4B As shown in the example coding tree structure (402), some coding methods extend the partition tree to a decatree structure and increase the maximum size (e.g., sometimes called the superblock) to start from 128×128. The second coding tree structure includes 4:1 / 1:4 rectangular partitions not present in the first coding tree structure. Figure 4B The partition type with 3 sub-partitions in the second row is called a T-partition. In addition to the code block size, the code tree depth can also be defined to indicate the splitting depth starting from the root node.

[0085] As an example, coding tree units (CTUs) can be split into coding units (CUs) using a quadtree structure represented as a coding tree to accommodate various local characteristics. In some embodiments, a decision is made at the CU level as to whether to encode a picture region using inter-frame (temporal) or intra-frame (spatial) prediction. Each CU can be further divided into one, two, or four PUs, depending on the prediction unit (PU) segmentation type. Within a PU, the same prediction process can be applied, and relevant information can be transferred to the decoder on a PU-by-PU basis. After obtaining residual blocks by applying a prediction process based on the PU segmentation type, the CUs can be divided into transform units (TUs) according to another quadtree structure (such as the coding tree of the CU).

[0086] Quadtrees, combined with nested multi-type trees and binary and ternary partitioning structures, can replace the concept of multiple partitioning unit types. In the coding tree structure, the CU (Coding Unit) can have a square or rectangular shape. The CTU (Coding Unit) is first partitioned using a quadtree structure. The leaf nodes of the quadtree can be further partitioned using multi-type tree structures. For example... Figure 4C As shown in the third coding tree structure (404), the multi-type tree structure includes four partition types. Multi-type leaf nodes are called CUs unless the CU is too large for the maximum transform length. This means that CUs, PUs, and TUs may have the same block size in a quadtree with a nested multi-type tree coding block structure. Figure 4D The example of a block partition of a CTU (406) is shown, which illustrates an example quadtree.

[0087] The coding tree scheme supports the ability for luma and chroma to have separate block tree structures, as in VTM7. In some cases, for P and B slices, the luma and chroma CTBs within a CTU share the same coding tree structure. However, for I slices, luma and chroma can have separate block tree structures. When a separate block tree mode is applied, the luma CTB is partitioned into CUs using one coding tree structure, and the chroma CTB is partitioned into chroma CUs using another coding tree structure. This means that a CU in an I slice can include, or consist of, coding blocks for the luma component, or coding blocks for the two chroma components, while a CU in a P or B slice can always include, or consist of, coding blocks for all three color components, unless the video is monochrome.

[0088] Transforms and transform blocks are now described. Transforms performed during video bitstream decoding are likely the inverse of transforms performed during video bitstream encoding, and are sometimes referred to as "inverse transforms". For simplicity, the transforms described herein are referred to as "transforms" regardless of whether they are performed during encoding or decoding.

[0089] Various transform sizes (e.g., from 4 to 64 points per dimension) and transform shapes (e.g., squares or rectangles with width / height ratios of 2:1 / 1:2 and 4:1 / 1:4) can be used. It is worth noting that when the encoder component applies a transform, the decoder component performs the inverse transform. Therefore, in the following description, the transforms described in the context of the decoder component can be the inverse transforms of the transforms applied to the encoder side.

[0090] Two-dimensional transformation processes may involve the use of hybrid transform kernels (e.g., composed of different one-dimensional transforms for each dimension of the encoded residual block). Primary one-dimensional transforms can include at least one of the following: a) 4-point, 8-point, 16-point, 32-point, and 64-point Discrete Cosine Transform (DCT-2); b) 4-point, 8-point, and 16-point Asymmetric Discrete Sine Transforms (DST-4, DST-7) and their inverted versions; or c) 4-point, 8-point, 16-point, and 32-point Identity Transforms. Table 1 lists the basis functions of DCT-2 and Asymmetric DST, as used in AV1.

[0091]

[0092] Table 1 - Example Principal Transformation Basis Functions

[0093] The availability of hybrid transform kernels may be based on transform block size and prediction mode. Table 2 below lists example dependencies, where "→" and "↓" represent horizontal and vertical dimensions, respectively. "and" "Indicates the block size and kernel availability in prediction mode. IDTX (or IDT) stands for identity transformation."

[0094]

[0095] Table 2 - Availability of Hybrid Transform Cores

[0096] For chroma components, transform type selection can be performed implicitly. For intra-frame prediction residuals, the transform type can be selected based on the intra-frame prediction mode, as shown in Table 3. For inter-frame prediction residuals, the transform type can be selected based on the transform type of the co-located luma block. Therefore, for chroma components, transform instructions in the bitstream may not be required.

[0097]

[0098] Table 3 - Transformation Type Selection for Chroma Intra-Frame Prediction Residuals

[0099] The following describes example encoding and decoding using prediction blocks and residual blocks. Figure 5A The calculation of a prediction block according to some embodiments is illustrated. Figure 5A In the example, intra-frame prediction is performed on the current block 502 to generate prediction block 504. In some embodiments, inter-frame prediction is performed to generate prediction blocks. The current block 502 includes a set of samples (e.g., pixel blocks), and prediction block 504 includes a set of predictions corresponding to that set of samples. Figure 5B The calculation of residual blocks according to some embodiments is illustrated. For example... Figure 5B As shown, prediction block 504 is subtracted from the current block 502 to generate residual block 506, which includes a set of residuals. For example, the corresponding difference between each sample and its corresponding prediction is calculated. Figure 5C The calculation of the reconstructed block according to some embodiments is shown. For example... Figure 5C As shown, residual block 506 undergoes one or more transforms and quantizations to generate a set of residual coefficients. This set of residual coefficients can be transferred from the encoder component to the decoder component. This set of residual coefficients undergoes inverse quantization and inverse transform to generate a reconstructed residual block 508. The reconstructed residual block 508 is combined with prediction block 504 (e.g., the reconstructed residuals of the reconstructed residual block 508 are added to the prediction of prediction block 504) to generate a reconstructed block 510 corresponding to the current block 502.

[0100] In some embodiments, the separable transform (as shown in Table 1) is applied to both intra-residual and inter-residual samples. In some embodiments, an intra-secondary transform (IST) scheme is customized for the video coding library (e.g., for transforming intra-residual blocks). Compared to the non-separable master transform, the IST scheme can efficiently capture directional patterns in the intra-residual samples with lower complexity. In the IST scheme, the nominal intra-prediction angle can be used to classify the IST kernel.

[0101] Intra-frame residual samples can exhibit arbitrary directed texture patterns, which can be captured more effectively by non-separable transforms. However, non-separable transforms are limited in applications with larger block sizes due to implementation complexity. A non-separable quadratic transform scheme can capture most of the directionality, but it is less complex because it is applied only to the low-frequency coefficients of the separable master transform, and can be applied to even larger block sizes with lower complexity.

[0102] In some embodiments, the IST scheme is combined with an intra-prediction scheme. The IST scheme may include 12 sets of quadratic transforms, each with 3 kernels. Table 4 shows an example selection of quadratic transform sets and the corresponding indices used for transform set selection. The left column represents the intra-prediction modes with available transform kernels, and the right column represents the set index. For example, at the encoder, for each mode, the best kernel is selected from the set according to RDO and signaled (4 symbols, including the case without IST). In this example, at the decoder, the bitstream is parsed to obtain the kernels used.

[0103]

[0104] Table 4 - Selection of the Quadratic Transform Set

[0105] In some embodiments, a quadratic transform set is derived based on the intra-frame prediction direction, and the kernel type in the set is explicitly signaled. In some embodiments, IST can be enabled when DCT-2 or ADST is used as the horizontal and vertical master transforms. In some embodiments, IST is enabled only for lumen intra-frame blocks. For example, depending on the block size, a 4×4 or 8×8 non-separable transform can be selected. If min(tx_width, tx_height) < 8, a 4×4 IST can be selected. For larger blocks where both tx_width and tx_height are greater than or equal to 8, an 8×8 IST can be used. Here, tx_width and tx_height correspond to the width and height of the transform block, respectively. The input to the IST can be low-frequency master transform coefficients in a zig-zagscan order, which can be the default scan order. This facilitates more efficient decorrelation of adjacent low-frequency coefficients.

[0106] In some embodiments, intra-frame and inter-frame coded blocks can be further divided into multiple transform units (e.g., partitioning depth up to 2 levels). In some embodiments, the application of IST is limited to the root (depth 0) of the transform partitioning tree structure. This limitation allows for a reduction in overall coding time complexity (~50%) with minimal impact on compression efficiency (~0.25% loss). In some embodiments using the IST scheme, the squared transform block size is used to derive context information, and the context used for entropy coding of the kernel index is also derived. For rectangular transform blocks, the next smallest square size can be used.

[0107] In some embodiments, the IST scheme defines 14 sets of quadratic transforms, each with 3 kernels. The selection of the IST set may depend on the intra-prediction mode used for residual generation. Table 5 below describes the mapping between intra-prediction modes, primary transform types, and IST set indices. The first column indicates the intra-prediction mode with available kernels, the second column indicates the primary transform type, and the third column indicates the set index. Depending on the block size, a 16-point or 64-point IST can be selected. A 16-point IST can be selected if min(tx_width, tx_height) < 8. A 64-point IST can be used for larger blocks where both tx_width and tx_height are greater than or equal to 8. Here, tx_width and tx_height correspond to the width and height of the transform block, respectively. Transform coefficients (primary transform coefficients only) outside the region of application (RoA) of the IST can be zeroed.

[0108] Table 5 below shows that the 14 quadratic transform sets depend on two master transform types (DCT_DCT and ADST_ADST). Therefore, for one master transform type, only 7 different sets need to be signaled.

[0109]

[0110] Table 5 - Selection of the Quadratic Transform Set

[0111] In some embodiments, the probabilistic context selected for each set is derived from the intra-prediction mode. In some embodiments, the encoder component implicitly selects the IST set based on a predefined mapping between the intra-prediction mode and the IST set. In some embodiments, the encoder performs an additional search on all available IST sets (e.g., instead of checking only one IST set based on the intra-prediction mode) so that the encoder can make rate-distortion optimized decisions regarding the selection of the IST set.

[0112] In some embodiments, IST is enabled for inter-frame coded blocks. For example, the IST kernel used for intra-frame coded blocks is also applied to inter-frame coded blocks. In some embodiments, the IST kernel used for intra-frame coded blocks is reused for inter-frame coded blocks without change or addition. In some embodiments, the above description of IST for intra-frame coded blocks also applies to inter-frame coded blocks. For example, the encoder can select from multiple sets and multiple kernels within a set (e.g., a set with 7 sets and 3 kernels). The kernel and set indices can be explicitly signaled.

[0113] In some embodiments, IST is enabled for inter-frame coded blocks. In some embodiments, DCT_DCT or ADST_ADST is used as the primary transform to enable IST for inter-frame coded blocks. For example, for inter-frame blocks, the kernel index is signaled, and the set used corresponds to DC_PRED or set index 0.

[0114] In some embodiments, only the kernel index of the inter-coded block is signaled (e.g., the set is predefined or implicitly derived). For example, the set used may correspond to a set index of zero. In some embodiments, IST is enabled (only) for inter-coded blocks that use DCT_DCT as the primary transform. In some embodiments, IST is enabled (only) for inter-transform blocks with a width greater than or equal to 16 and / or a height greater than or equal to 16. In some embodiments, IST is enabled (only) for blocks with the last valid coefficient position or EOB greater than 3.

[0115] In some embodiments, additional contexts (e.g., 5 additional contexts) are introduced depending on whether the block is inter-frame coded or intra-frame coded.

[0116] As described in this paper, extending IST to inter-frame coding blocks can improve coding efficiency. Table 6 below illustrates the improvements in encoding and decoding based on simulations performed using the current design (e.g., AVM design v6) and various video data (e.g., representing AOM general test conditions v6.0).

[0117]

[0118] Table 6 - Simulation Results

[0119] Figure 6A A flowchart of a video decoding method 600 according to some embodiments is shown. Method 600 can be executed at a computing system, wherein the computing system (e.g., server system 112, source device 102, or electronic device 120) has control circuitry and a memory storing instructions executed by the control circuitry. In some embodiments, method 600 is executed by executing instructions stored in the memory of the computing system (e.g., memory 314).

[0120] The system receives (602) a video bitstream (e.g., an encoded video sequence) comprising multiple blocks (e.g., encoded video blocks, transform blocks) corresponding to multiple images, the multiple blocks including a first block (e.g., the current block). When the first block is (604) an inter-frame coded block, based on encoded information including block end-of-frame (EOB) values, the system determines (606) whether to signal transform information for the first block, the transform information including at least one transform type and transform set. When transform information is signaled for the first block, the system parses (608) the transform information from at least one indicator in the video bitstream. When transform information is not signaled for the first block, the system derives (610) the transform information without parsing at least one indicator. The system decodes (612) the first block by applying an inverse transform to the first block using the transform information. In some embodiments, the transform type and / or transform set information of the inter-frame coded block is selectively signaled based on previously encoded information known to both the encoder and decoder (e.g., corresponding to previously decoded blocks and / or the first block).

[0121] In some embodiments, the previously encoded information includes one or more of the following: EOB value, inter-frame prediction mode, block size, primary transform type and / or set (e.g., when selectively signaling secondary transform type and / or set), partition type and / or partition depth, frame time ID or layering information, and associated color components. In some embodiments, the previously encoded information includes two or more of the above.

[0122] In some embodiments, a signal is sent to notify the transform type and / or transform set only when the EOB value is greater than (and optionally equal to) N. Possible values ​​for N include 1, 2, ..., or 32. In some embodiments, the value of N is different for different color components. In some embodiments, the value of N is different for signaling of the primary transform type (or set) or the secondary transform type (or set).

[0123] In some embodiments, for a subset (or combination) of primary transform types, secondary transform types and / or sets of secondary transforms are signaled. In one example, a subset of primary transform types includes bidirectional DCT or DST (or ADST, LGT, or KLT) or their inverted versions, or combinations of their inverted and non-inverted versions.

[0124] In some embodiments, different types and / or sets of secondary transformations are applicable to different combinations of the main transformation type.

[0125] In some embodiments, the transform type and / or transform set are signaled only for blocks in frames with a time ID or time layer <= K. In some embodiments, the transform type and / or transform set are signaled only for blocks in frames with a time ID or time layer > K. Possible values ​​for K include 1, 2, ... or 32.

[0126] In some embodiments, based on the block size or aspect ratio, only blocks that meet certain conditions are signaled to indicate the transform type and / or transform set. For example, all rectangular blocks are signaled to indicate the transform type and / or transform set.

[0127] In some embodiments, the transform type and / or transform set are signaled only for specific coded block partition types and / or partition depths. For example, only non-split blocks can signal the transform type and / or transform set. In some embodiments, the transform type and / or transform set are signaled only for specific transform partition types and / or partition depths. For example, only non-split transform blocks can signal the transform type and / or transform set. In some embodiments, the transform type and / or transform set are signaled only for blocks in unidirectional inter-prediction, composite, or bidirectional inter-prediction modes.

[0128] Figure 6B A flowchart of a video encoding method 650 according to some embodiments is shown. Method 650 can be executed at a computing system having control circuitry and memory (e.g., server system 112, source device 102, or electronic device 120) storing instructions executed by the control circuitry. In some embodiments, method 650 is executed by executing instructions stored in the computing system's memory (e.g., memory 314). In some embodiments, method 650 is executed by the same system as method 600 described above.

[0129] The system receives (652) video data (e.g., a source video sequence) comprising a set of blocks (e.g., corresponding to one or more images), the set of blocks including a first block. When the first block is (654) an inter-frame coded block, based on coded information including (e.g., the EOB value of the first block), the system determines (656) whether to signal transform information for the first block, the transform information including a transform type and at least one of a transform set. When signaling transform information for the first block is required, the system signals (658) the transform information via the video bitstream. When not signaling transform information for the first block is not required, the system abandons (660) signaling transform information. The system encodes (662) the first block by applying a transform to it using the transform information. As previously stated, the encoding process may reflect the decoding process described herein (e.g., applying a transform). For brevity, these details are not repeated here.

[0130] although Figure 6A and 6B Multiple logical stages are shown in a specific order, but stages that are not dependent on order can be reordered, and other stages can be combined or decomposed. Some reorderings or other groupings not specifically mentioned will be obvious to those skilled in the art, and therefore the orderings and groupings presented herein are not exhaustive. Furthermore, it should be recognized that these stages can be implemented in hardware, firmware, software, or any combination thereof.

[0131] Now let's turn to some example implementations.

[0132] (A1) In one aspect, some embodiments include a video decoding method (e.g., method 600). In some embodiments, the method is performed at a computing system (e.g., server system 112) having memory and control circuitry. In some embodiments, the method is performed at an encoding module (e.g., encoding module 320). In some embodiments, the method is performed at a source encoding component (e.g., source encoder 202), an encoding engine (e.g., encoding engine 212), and / or an entropy encoder (e.g., entropy encoder 214). The method includes: (i) receiving a video bitstream (e.g., an encoded video sequence) comprising multiple blocks corresponding to multiple images, the multiple blocks including a first block; (ii) when the first block is an inter-frame coded block: (a) determining, based on encoded information including an end-of-block value (EOB), whether to signal transform information for the first block, the transform information including at least one transform type and transform set; (iii) when signaling transform information for the first block, parsing the transform information from at least one indicator in the video bitstream; (iv) when not signaling transform information for the first block, deriving the transform information without parsing the at least one indicator; and (v) decoding the first block by applying an inverse transform to the first block using the transform information. For example, the transform type and / or transform set information of the inter-frame coded block can be selectively signaled based on previously encoded information known to both the encoder and decoder. Previously encoded information may include EOB values, inter-frame prediction modes, block size, primary transform type and / or set (e.g., for selectively signaling secondary transform types and / or sets), partition type and / or partition depth, frame time ID or layering information, and / or associated color components. In some embodiments, when it is determined that transform information for signaling the first block is to be transmitted, transform information is parsed from at least one indicator in the video bitstream. In some embodiments, when it is determined that the first block is an inter-frame coded block, signaling for the transform information is determined based on encoded information such as EOB values.

[0133] (A2) In some embodiments of A1, the encoding information also includes the block size corresponding to the first block. For example, for inter-frame coded transform blocks whose width and / or height meet predefined criteria (e.g., 8 samples, 16 samples, or 32 samples), only the transform information may be signaled.

[0134] (A3) In some embodiments of A1 or A2, the coding information also includes the main transform type of the first block. For example, for an inter-frame coded transform block with a Discrete Cosine Transform (DCT) type, such as DCT-DCT, only the transform set information may be signaled.

[0135] (A4) In some embodiments of any of A1-A3, the encoding information also includes the transform segmentation depth corresponding to the first block. For example, for a block with a transform segmentation depth of zero, only the transform set information may be signaled.

[0136] (A5) In some embodiments of any of A1-A4, the encoded information further includes a time identifier for the first block and one or more layers of the first block. For example, the transform type and / or transform set may be signaled only for blocks in frames with a time ID or layer <= K. As another example, the transform type and / or transform set may be signaled only for blocks in frames with a time ID or time layer > K. Example values ​​for K include 1, 2, ..., and 32.

[0137] (A6) In some embodiments of any of A1-A5, the encoding information further includes one or more of the block size and aspect ratio of the first block. For example, the transform type and / or transform set may be signaled only to blocks that meet the conditions of their size or aspect ratio. For example, the transform type and / or transform set information of all rectangular blocks may be signaled.

[0138] (A7) In some embodiments of any of A1-A6, the encoding information further includes one or more of the partition type of the first block and the partition depth corresponding to the first block. For example, the transform type and / or transform set may be signaled only for a specific coded block partition type and / or partition depth. For example, for non-split blocks, only the transform type and / or transform set information may be signaled.

[0139] (A8) In some embodiments of any of A1-A7, the encoding information further includes one or more of the transform partition type and the transform partition depth corresponding to the first block. For example, the transform type and / or transform set may be signaled only for a specific transform partition type and / or partition depth. For example, for non-split transform blocks, only the transform type and / or transform set information may be signaled.

[0140] (A9) In some embodiments of any of A1-A8, the encoding information also includes the prediction mode of the first block. For example, the transform type and / or transform set may be signaled only for blocks having a unidirectional inter-frame prediction, compound prediction, or bidirectional inter-frame prediction mode.

[0141] (A10) In some embodiments of any of A1-A9, (i) when the EOB value is greater than a predefined threshold, a signal is sent to notify the transformation information; and (ii) when the EOB value is less than or equal to the predefined threshold, no signal is sent to notify the transformation information. For example, the transformation type and / or transformation set can only be signaled when the EOB value is > N. N can be 1, 2, ... 32 (e.g., N=3).

[0142] (A11) In some embodiments of A10, the predefined threshold is different for different color components of the first block. For example, the value of N may be different for different color components.

[0143] (A12) In some embodiments of A10 or A11, the predefined thresholds are different for primary transform information and secondary transform information. For example, the value of N may be different for signaling of primary transform type (or set) and secondary transform type (or set).

[0144] (A13) In some embodiments of any of A1-A12, the transformation information corresponds to the quadratic transformation of the first block, and the transformation information is signaled only for a subset of the main transform type. For example, the quadratic transform type and / or quadratic transform set may be signaled only for a subset (or combination of transform types) of the main transform type. For example, the subset of the main transform type may include DCT (DCT-DCT), and / or Discrete Sine Transform (DST) (and / or ADST, or LGT, or KLT), such as bidirectional or inverted versions, and / or combinations of inverted and non-inverted versions.

[0145] (A14) In some embodiments of any of A1-A13, the transformation information corresponds to the quadratic transformation of the first block, and different numbers of transformation information are mapped to different principal transformations. For example, different quadratic transformation types and / or sets may be applicable to different combinations of principal transformation types. For example, different numbers of quadratic transformation sets / types may be mapped to different principal transformations.

[0146] (B1) In another aspect, some embodiments include a method for video encoding (e.g., method 650). In some embodiments, the method is performed at a computing system (e.g., server system 112) having memory and control circuitry. In some embodiments, the method is performed at an encoding module (e.g., encoding module 320). The method includes: (i) receiving video data comprising a set of blocks (e.g., a source video sequence), the set of blocks including a first block; (ii) when the first block is an inter-frame coded block: (a) determining, based on encoding information including an end-of-block (EOB) value, whether to signal transform information of the first block, the transform information including at least one transform type and transform set; (iii) when signaling transform information of the first block is to be signaled, signaling transform information via a video bitstream; (iv) when not signaling transform information of the first block is not to be signaled, abandoning signaling transform information; and (v) encoding the first block by applying a transform to the first block using the transform information. In some embodiments, the encoded first block is signaled via a video bitstream.

[0147] (B2) In some embodiments of B1, the encoding information also includes the block size corresponding to the first block.

[0148] (B3) In some embodiments of B1 or B2, the encoding information also includes the main transform type of the first block.

[0149] (C1) In another aspect, some embodiments include a method for processing visual media data. In some embodiments, the method is performed at a computing system (e.g., server system 112) having memory and control circuitry. In some embodiments, the method is performed at an encoding module (e.g., encoding module 320). The method includes: (i) obtaining a source video sequence comprising multiple frames; and (ii) performing a conversion between the source video sequence and a video bitstream of the visual media data according to format rules. The video bitstream includes (a) multiple blocks comprising a first block, and (b) at least one indicator indicating the conversion information when the first block is signaled. The format rules specify that the conversion information of the first block is selectively signaled according to encoded information including an end-of-block (EOB) value corresponding to the first block.

[0150] (C2) In some embodiments of C1, the encoding information also includes the block size corresponding to the first block.

[0151] (C3) In some embodiments of C1 or C2, the encoding information also includes the main transform type of the first block.

[0152] On the other hand, some embodiments include a computing system (e.g., server system 112) that includes control circuitry (e.g., control circuitry system 302) and a memory (e.g., memory 314) coupled to the control circuitry, the memory storing at least one set of instructions configured to be executed by the control circuitry, the at least one set of instructions including instructions for performing any of the methods described herein (e.g., A1-A14, B1-B3, and C1-C3 above).

[0153] In another aspect, some embodiments include a non-volatile computer-readable storage medium storing at least one set of instructions executable by control circuitry of a computing system, the at least one set of instructions including instructions for performing any of the methods described herein (e.g., A1-A14, B1-B3, and C1-C3 above). Unless otherwise stated, any syntax element described herein can be a High-Level Syntax (HLS). As used herein, an HLS signals at a level higher than the block level. For example, an HLS may correspond to a sequence level, frame level, slice level, or tile level. As another example, an HLS element may signal in a Video Parameter Set (VPS), Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Adaptive Parameter Set (APS), slice header, picture header, tile header, and / or CTU header.

[0154] It should be understood that although the terms “first,” “second,” etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the claims. As used in the description of the embodiments and the appended claims, the singular forms “a,” “an,” and “the” are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of at least one of the associated listed items. It should be further understood that when the terms “comprises” and / or “comprising” are used in this specification, it indicates the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of at least one other feature, integer, step, operation, element, component, and / or group thereof.

[0155] As used herein, the term “when” can be interpreted, depending on the context, as meaning “if” or “at the time of” or “in response to determination” or “according to determination” or “in response to detection”, if the stated prerequisite is true. Similarly, depending on the context, the phrases “if it is determined [the stated prerequisite is true]” or “if [the stated prerequisite is true]” or “when [the stated prerequisite is true]” can be interpreted as meaning “at the time of determination” or “in response to determination” or “according to determination” or “at the time of detection” or “in response to detection”, if the stated prerequisite is true. As used herein, N refers to a variable number. Unless explicitly stated otherwise, different instances of N may refer to the same number (e.g., the same integer value, such as the number 2) or different numbers.

[0156] For purposes of explanation, the above description has been described with reference to specific embodiments. However, the foregoing illustrative discussion is not intended to be exhaustive or to limit the claims to the precise forms disclosed. Many modifications and variations are possible in light of the foregoing teachings. The embodiments were chosen and described in order to best explain the principles of operation and practical application, thereby enabling others skilled in the art to understand.

Claims

1. A video decoding method, wherein, The method is performed on a computing system having memory and one or more processors, characterized in that the method comprises: Receive a video bitstream comprising at least two blocks corresponding to at least two images, wherein the at least two blocks include a first block; When the first block is an inter-frame coded block: Based on the encoded information including the block end EOB value, it is determined whether to signal the transformation information of the first block, wherein the transformation information includes at least one of the transformation type and transformation set; When a signal is sent to notify the first block of the transformation information, the transformation information is parsed from at least one indicator of the video bitstream; When no signal is sent to notify the first block of the transformation information, the transformation information is derived without parsing the at least one indicator; and, The first block is decoded by applying an inverse transform to the first block using the transformation information.

2. The method according to claim 1, characterized in that, The encoded information further includes the block size corresponding to the first block.

3. The method according to claim 1, characterized in that, The encoding information further includes the main transform type of the first block.

4. The method according to claim 1, characterized in that, The encoding information further includes the transform partition depth corresponding to the first block.

5. The method according to claim 1, characterized in that, The encoded information further includes at least one of the following: the time identifier of the first block, and the layering of the first block.

6. The method according to claim 1, characterized in that, The encoded information further includes at least one of the following: the block size of the first block, and the aspect ratio of the first block.

7. The method according to claim 1, characterized in that, The encoding information further includes at least one of the following: the partition type of the first block, and the partition depth corresponding to the first block.

8. The method according to claim 1, characterized in that, The encoding information further includes at least one of the following: the transform partition type corresponding to the first block, and the transform partition depth corresponding to the first block.

9. The method according to claim 1, characterized in that, The encoded information further includes the prediction pattern of the first block.

10. The method according to claim 1, characterized in that, When the EOB value is greater than a predetermined threshold, a signal is sent to notify the transformation information; and When the EOB value is less than or equal to the predetermined threshold, no signal is sent to notify the transformation information.

11. The method according to claim 10, characterized in that, The predetermined threshold is different for different color components of the first block.

12. The method according to claim 10, characterized in that, The predetermined threshold is different for the primary transformation information and the secondary transformation information.

13. The method according to claim 1, characterized in that, The transformation information corresponds to the second transformation of the first block, and only signals are sent to notify the transformation information of at least two subsets of the main transformation types.

14. The method according to claim 1, characterized in that, The transformation information corresponds to the second transformation of the first block, and different numbers of transformation information are mapped to different main transformations.

15. A computing system, characterized in that, include: Control circuit; Memory; as well as, At least one set of instructions stored in memory and configured to be executed by control circuitry, the at least one set of instructions including instructions for the following operations: Receive video data comprising a set of blocks, wherein the set of blocks includes a first block; When the first block is an inter-frame coded block: Based on the encoded information including the block end EOB value, it is determined whether to send a signal to notify the transformation information of the first block, wherein the transformation information includes at least one transformation type and transformation set; When it is necessary to send a signal to notify the first block of the transformation information, the transformation information is notified by sending a signal through the video bitstream; When no signal is sent to notify the first block of the transformation information, the signaling of the transformation information is abandoned; and, The first block is encoded by applying a transformation to it using the transformation information.

16. The computing system according to claim 15, characterized in that, The encoded information further includes the block size corresponding to the first block.

17. The computing system according to claim 15, characterized in that, The encoding information further includes the main transform type of the first block.

18. A non-volatile computer-readable storage medium, characterized in that, The non-volatile computer-readable storage medium stores at least one set of instructions configured to be executed by a computing device having control circuitry and memory, the at least one set of instructions including instructions for the following operations: Obtain a source video sequence that includes at least two frames; and According to format rules, conversion is performed between the source video sequence and the video bitstream of the visual media data, wherein... The video bitstream includes: Including at least two blocks, including the first block; and, When signaling the transformation information of the first block, at least one identifier indicating the transformation information is provided; and, The format rule specifies that, based on the encoded information including the end-of-block (EOB) value corresponding to the first block, the transformation information of the first block is selectively signaled.

19. The non-volatile computer-readable storage medium according to claim 18, characterized in that, The encoded information further includes the block size corresponding to the first block.

20. The non-volatile computer-readable storage medium according to claim 18, characterized in that, The encoding information further includes the main transform type of the first block.

Images