Methods, apparatus and storage media for processing video data

Abstract

A method, apparatus, and storage medium for processing video data. The method includes extracting data blocks from the video data; scanning a first number of data items in the data blocks following a first scan order to generate a first data sequence; performing an inseparable transformation on the first data sequence to obtain a second data sequence having a second number of data items; and replacing at least a portion of the first number of data items in the data blocks with a portion or all of the second data sequence following a second scan order.

Description

[0001] References merged

[0002] This application is based on and claims priority to U.S. Provisional Application No. 63 / 238,646, filed August 30, 2021, and U.S. Non-Provisional Application No. 17 / 587,164, filed January 28, 2022, both of which are incorporated herein by reference in their entirety. Technical Field

[0003] This disclosure describes a set of advanced video coding techniques. More specifically, the disclosed techniques relate to the implementation of inseparable transformations of data blocks in video coding and decoding. Background Technology

[0004] The background description provided herein is intended to present the overall context of this application. The extent of the work of the currently named inventors described in the background section and various aspects of this specification does not imply that it was prior art at the time of filing of this application, nor is it expressly or implied that it was acknowledged as prior art to this application.

[0005] Video encoding and decoding can be performed using inter-frame picture prediction with motion compensation. Uncompressed digital video can comprise a series of pictures, each with a spatial dimension of, for example, 1920×1080 luminance samples and associated full-sampled or subsampled chrominance samples. The series of pictures has a fixed or variable picture rate (or frame rate), for example, 60 pictures per second or 60 frames per second. Uncompressed video has specific bitrate requirements. For example, a video with a pixel resolution of 1920×1080, a frame rate of 60 frames per second, and chrominance subsampling of 4:2:0, with 8 bits per pixel per color channel, requires approximately 1.5 Gbit / s of bandwidth. One hour of such video would require over 600 GB of storage space.

[0006] One objective of video encoding and decoding is to reduce redundant information in the uncompressed input video signal through compression. Video compression can help reduce the bandwidth and / or storage requirements mentioned above, in some cases by two or more orders of magnitude. Lossless compression, lossy compression, and combinations of both can be employed. Lossless compression refers to the technique of reconstructing an exact copy of the original signal from the compressed original signal via a decoding process. Lossy compression refers to an encoding / decoding process in which the original video information is not fully preserved during encoding and cannot be fully recovered during decoding. When using lossy compression, the reconstructed signal may differ from the original signal, but the distortion between the original and reconstructed signals is small enough to make the reconstructed signal usable for the intended application, despite some information loss. In the case of video, lossy compression is widely used in many applications. The tolerable amount of distortion depends on the application. For example, users of some consumer video streaming applications may tolerate higher distortion than users of film or television broadcasting applications. The compression ratio achievable through a particular encoding algorithm can be selected or adjusted to reflect various distortion tolerances: higher tolerable distortion generally allows encoding algorithms that produce higher losses and higher compression ratios.

[0007] Video encoders and decoders can utilize techniques from a wide range of categories and steps, including, for example, motion compensation, Fourier transform, quantization, and entropy coding.

[0008] Video codec techniques can include known intra-frame coding techniques. In intra-frame coding, sample values ​​are represented without reference to samples or other data from a previously reconstructed reference picture. In some video codecs, a picture is spatially subdivided into sample blocks. When all sample blocks are encoded in intra-frame mode, the picture can be called an intra-frame picture. Intra-frame pictures and their derivatives (such as independent decoder refresh pictures) can be used to reset the decoder state and thus can be used as the first picture in the encoded video bitstream and video session, or as a still image. The samples of the intra-predicted block can then be transformed to the frequency domain, and the transform coefficients thus generated can be quantized before entropy coding. Intra-frame prediction represents a technique that minimizes the sample values ​​in the pre-transform domain. In some cases, the smaller the transformed DC value and the smaller the AC coefficients, the fewer bits are needed to represent the entropy-coded block at a given quantization step size.

[0009] As is known from technologies such as MPEG-2, traditional intra-frame coding does not use intra-frame prediction. However, some newer video compression techniques include attempts to encode / decode blocks based on, for example, surrounding sample data and / or metadata, which are obtained during spatially adjacent encoding and / or decoding and are decoded prior to the data blocks being intra-frame encoded or decoded. Such techniques are hereby referred to as "intra-frame prediction" techniques. Note that in at least some cases, intra-frame prediction uses only reference data from the current frame being reconstructed, and not reference data from other reference frames.

[0010] There can be many different forms of intra-prediction. When more than one such technique is available in a given video coding technique, the technique used can be called an intra-prediction mode. One or more intra-prediction modes can be provided in a particular codec. In some cases, a mode can have sub-modes and / or can be associated with various parameters, and the mode / sub-mode information and intra-coding parameters for the video block can be contained in the mode codeword, which can be encoded individually or together. For a given combination of modes, sub-modes, and / or parameters, which codeword is used can affect the coding efficiency gain through intra-prediction, and the same is true for the entropy coding techniques used to convert the codeword into a bitstream.

[0011] A certain mode of intra-frame prediction was introduced with H.264, revised in H.265, and further refined in newer coding techniques such as Joint Probe Mode (JEM), Universal Video Coding (VVC), and Base Set Reference (BMS). Typically, for intra-frame prediction, predictor blocks can be formed using neighboring sample values ​​that have become available. For example, available values ​​from a specific set of neighboring samples along a particular direction and / or line can be copied into the predictor block. The reference to the direction used can be encoded in the bitstream or can be predicted itself.

[0012] refer to Figure 1A The lower right corner depicts a subset of nine predictor directions specified among the 33 possible intra-frame predictor directions in H.265 (corresponding to the 33 angular modes of the 35 intra-frame modes specified in H.265). The point (101) where the arrows converge represents the sample being predicted. The arrows indicate the direction on which the sample at 101 is predicted using neighboring samples. For example, arrow (102) indicates that sample (101) is predicted based on one or more neighboring samples to the upper right at a 45-degree angle to the horizontal direction. Similarly, arrow (103) indicates that sample (101) is predicted based on one or more neighboring samples to the lower left of sample (101) at a 22.5-degree angle to the horizontal direction.

[0013] Still referencing Figure 1AA square block (104) comprising 4×4 samples is shown in the upper left (represented by a thick dashed line). The square block (104) consists of 16 samples, each labeled with "S" and its position in the Y dimension (e.g., row index) and X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from top to bottom) and the first sample in the X dimension (from left to right). Similarly, sample S44 is the fourth sample in block (104) in both the Y and X dimensions. Since the block is 4×4 in size, S44 is located in the lower right corner. Example reference samples following a similar numbering scheme are also shown. Reference samples are labeled with "R" and their Y position (e.g., row index) and X position (e.g., column index) relative to block (104). In H.264 and H.265, predicted samples that are adjacent to the block being reconstructed are used.

[0014] Intra-image prediction for block 104 can begin by copying reference sample values ​​from neighboring samples based on the prediction direction indicated by a signal. For example, assuming the encoded video stream includes signaling for block 104, this signaling indicates the prediction direction of arrow (102)—that is, predicting samples based on one or more prediction samples at a 45-degree angle to the upper right of the horizontal direction. In such a case, samples S41, S32, S23, and S14 are predicted based on the same reference sample R05. Sample S44 is then predicted based on reference sample R08.

[0015] In some cases, such as through interpolation, the values ​​of multiple reference samples can be combined to compute a reference sample, especially when the direction is not divisible by 45 degrees.

[0016] As video coding technologies continue to evolve, the number of possible directions increases. For example, in H.264 (2003), nine different directions were available for intra-frame prediction. This increased to 33 in H.265 (2013), and JEM / VVC / BMS at the time of this disclosure can support up to 65 directions. Experimental studies have been conducted to help identify the most suitable intra-frame prediction directions, and certain techniques from entropy coding can be used to encode those most suitable directions with a small number of bits, accepting a bit cost for the direction. Furthermore, the direction itself can sometimes be predicted based on neighboring directions used for intra-frame prediction of already decoded neighboring blocks.

[0017] Figure 1B A schematic diagram (180) depicting 65 intra-frame prediction directions according to JEM is shown to illustrate the increase in the number of prediction directions in various coding techniques over time.

[0018] The method used to map the bits representing the intra-prediction direction to the prediction direction in the encoded video bitstream can vary with different video coding techniques; and can range, for example, from prediction direction to intra-prediction mode, to codeword, to complex adaptive schemes involving the most probable mode, and similar simple direct mappings. However, in all cases, there may be certain directions in the video content that are statistically less likely to occur for intra-prediction than some other directions. Since the goal of video compression is to reduce redundancy, in well-designed video coding techniques, those less probable directions can be represented by a larger number of bits than the more probable directions.

[0019] Inter-image or inter-frame prediction can be based on motion compensation. In motion compensation, sample data from a previously reconstructed image or a portion thereof (the reference image), after being spatially shifted in a direction indicated by a motion vector (hereinafter MV), can be used for prediction of a newly reconstructed image or image portion (e.g., a patch). In some cases, the reference image can be the same as the image currently being reconstructed. The MV can have two dimensions (X and Y) or three dimensions, where the third dimension is an indication of the reference image in use (approximate temporal dimension).

[0020] In some video compression techniques, the current motion vector (MV) applicable to a region of sample data can be predicted from other MVs, such as those related to other regions of sample data in spatially adjacent reconstructed regions and preceding the current MV in decoding order. This increases compression efficiency by significantly reducing the total amount of data required to encode the MV by relying on removing redundancy from related MVs. MV prediction can be performed efficiently, for example, because when encoding an input video signal derived from a camera (called natural video), there is a statistical probability that a region larger than the area applicable to a single MV will move in a similar direction in the video sequence. Therefore, in some cases, similar motion vectors derived from MVs of neighboring regions can be used for prediction. This results in the actual MV for a given region being similar to or the same as the MV predicted from the surrounding MVs. After entropy coding, such MVs can be represented with fewer bits than would be used if the MV were directly encoded instead of predicted from one or more neighboring MVs. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, MV prediction itself may be lossy, for example, due to rounding errors when calculating predictions from several surrounding MVs.

[0021] H.265 / HEVC (ITU-T H.265 Recommendation, “High Efficiency Video Coding”, December 2016) describes various MV prediction mechanisms. Among the various MV prediction mechanisms specified in H.265, this application describes the technique hereinafter referred to as “spatial combining”.

[0022] Please refer to Figure 2 The current block (201) includes samples discovered by the encoder during the motion search process, which can be predicted based on previous blocks of the same size that have generated spatial offsets. Alternatively, the MV can be derived from metadata associated with one or more reference images, rather than being directly encoded. For example, using the MV associated with any of the five surrounding samples A0, A1 and B0, B1, B2 (corresponding to 202 to 206 respectively), the MV is derived from the metadata of the nearest reference image (in decoding order). In H.265, MV prediction can use predictions from the same reference images also used in adjacent blocks. Summary of the Invention

[0023] This disclosure provides methods, apparatus, and storage media for processing video data.

[0024] This disclosure also provides a non-transitory computer-readable medium storing instructions that, when executed by a computer to process video data, cause the computer to perform the method according to this application.

[0025] According to one aspect, embodiments of this disclosure provide a method for processing video data. The method extracts data blocks from the video data; scans a first number of data items in the data blocks following a first scan order to generate a first data sequence; performs an inseparable transformation on the first data sequence to obtain a second data sequence having a second number of data items; and replaces at least a portion of the first number of data items in the data blocks with a portion or all of the second data sequence following a second scan order.

[0026] According to another aspect, one embodiment of this disclosure provides a method for entropy encoding of transform coefficients associated with video data. The method includes, in response to the transform coefficients being inseparable, scanning the transform coefficients using a first scan order when performing entropy encoding of the transform coefficients, the first scan order being one of: a horizontal scan order; or a vertical scan order; and, in response to the transform coefficients being separable, scanning the transform coefficients using a second scan order different from the first scan order when performing entropy encoding of the transform coefficients.

[0027] According to another aspect, embodiments of this disclosure provide a method for processing video data. The method includes receiving the video data; determining whether to apply an inseparable transform as a secondary transform to the video data; and in response to applying the inseparable transform as the secondary transform to the video data: scanning a first number of primary transform coefficients, wherein the primary transform coefficients follow a first scan order; performing an inseparable transform using the first number of primary transform coefficients as input to obtain a second number of secondary transform coefficients as output, wherein the secondary transform coefficients follow a second scan order; replacing at least the second number of primary transform coefficients with the secondary transform coefficients following the second scan order; performing an inverse secondary transform corresponding to the inseparable transform using the second number of secondary transform coefficients as input to obtain the first number of primary transform coefficients as output; and replacing at least the first number of secondary transform coefficients with the primary transform coefficients following the first scan order.

[0028] According to another aspect, embodiments of this disclosure provide an apparatus. The apparatus includes a memory storing instructions and a processor communicating with the memory. When the processor executes the instructions, the processor is configured to cause the apparatus to perform the method according to this application.

[0029] According to another aspect, embodiments of this disclosure provide a non-volatile computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform the methods according to this application.

[0030] The above and other aspects and embodiments thereof are described in more detail in the accompanying drawings, description and claims. Attached Figure Description

[0031] Other features, properties, and various advantages of the subject matter will become more apparent from the following detailed description and accompanying drawings.

[0032] Figure 1A A schematic diagram of an exemplary subset of intra-frame prediction direction patterns is shown.

[0033] Figure 1B An illustration of an exemplary intra-frame prediction direction is shown.

[0034] Figure 2 A schematic diagram of the current block and its surrounding spatial merging candidates for motion vector prediction is shown in one example.

[0035] Figure 3 A simplified block diagram of a communication system according to an example embodiment is shown.

[0036] Figure 4A simplified block diagram of a communication system according to an example embodiment is shown.

[0037] Figure 5 A simplified block diagram of a video decoder according to an example embodiment is shown.

[0038] Figure 6 A schematic diagram of a simplified block diagram of a video encoder according to an example embodiment is shown.

[0039] Figure 7 A block diagram of a video encoder according to another example embodiment is shown.

[0040] Figure 8 A block diagram of a video decoder according to another example embodiment is shown.

[0041] Figure 9 A directional intra-frame prediction mode according to an example embodiment of this disclosure is shown.

[0042] Figure 10 A non-directional intra-frame prediction mode according to an example embodiment of the present disclosure is shown.

[0043] Figure 11 A recursive intra-frame prediction mode according to an example embodiment of this disclosure is shown.

[0044] Figure 12 Transform block partitioning and scanning of intra-prediction blocks according to an example embodiment of the present disclosure are illustrated.

[0045] Figure 13 Transform block partitioning and scanning of inter-frame prediction blocks are illustrated according to an example embodiment of the present disclosure.

[0046] Figure 14 A low-frequency inseparable transformation process according to an example embodiment of the present disclosure is illustrated.

[0047] Figure 15 A data stream for performing an inseparable transformation is shown according to an example embodiment of the present disclosure.

[0048] Figure 16 A flowchart is shown according to an example embodiment of the present disclosure.

[0049] Figure 17 A schematic diagram of a computer system according to an example embodiment of the present disclosure is shown. Detailed Implementation

[0050] Figure 3This is a simplified block diagram of a communication system (300) according to an embodiment disclosed in this application. The communication system (300) includes a plurality of terminal devices that can communicate with each other via, for example, a network (350). For example, the communication system (300) includes a first terminal device (310) and a second terminal device (320) interconnected via a network (350). Figure 3 In this embodiment, the first terminal device (310) and the second terminal device (320) perform unidirectional data transmission. For example, the first terminal device (310) may encode video data (e.g., a video image stream captured by the first terminal device (310)) for transmission over a network (350) to the second terminal device (320). The encoded video data is transmitted in the form of one or more encoded video streams. The second terminal device (320) may receive the encoded video data from the network (350), decode the encoded video data to recover the video data, and display video images based on the recovered video data. Unidirectional data transmission is common in applications such as media services.

[0051] In another embodiment, the communication system (300) includes a third terminal device (330) and a fourth terminal device (340) that perform bidirectional transmission of encoded video data, which may be implemented, for example, during a video conference. For bidirectional data transmission, each of the third terminal device (330) and the fourth terminal device (340) may encode video data (e.g., a stream of video images captured by the terminal device) for transmission over a network (350) to the other terminal device. Each of the third terminal device (330) and the fourth terminal device (340) may also receive encoded video data transmitted by the other terminal device and may decode the encoded video data to recover the video data, and may display the video images on an accessible display device based on the recovered video data.

[0052] exist Figure 3In the embodiments disclosed herein, the first terminal device (310), the second terminal device (320), the third terminal device (330), and the fourth terminal device (340) may be servers, personal computers, and smartphones, but the scope of application of the basic principles disclosed herein is not limited thereto. The embodiments disclosed herein are applicable to desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated video conferencing equipment, etc. Network (350) refers to any number or type of network that transmits encoded video data between the first terminal device (310), the second terminal device (320), the third terminal device (330), and the fourth terminal device (340), including, for example, wired (connected) and / or wireless communication networks. The communication network (350) may exchange data in circuit-switched, packet-switched, and / or other types of channels. The network may include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this application, unless explicitly explained herein, the architecture and topology of the network (350) may be irrelevant to the operation of this application.

[0053] As an example, Figure 4 This illustration shows the placement of a video encoder and a video decoder in a video streaming environment. The subject matter disclosed in this application is equally applicable to other video applications, including, for example, video conferencing, digital TV broadcasting, gaming, virtual reality, compressed video storage on digital media including CDs, DVDs, memory sticks, etc.

[0054] The video streaming system may include an acquisition subsystem (413) that may include a video source (401), such as a digital camera, to create, for example, uncompressed video pictures or image streams (402). In an embodiment, the video picture stream (402) includes samples recorded by the digital camera of the video source 401. The uncompressed video picture stream (402) is depicted as a thick line to emphasize the high data volume of the video picture stream compared to encoded video data (404) (or encoded video bitstream). The video picture stream (402) may be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination of hardware and software to implement or carry out aspects of the disclosed subject matter as described in more detail below. Compared to the uncompressed video image stream (402), the encoded video data (404) (or the encoded video bitstream (404)) is depicted as a thin line to emphasize the lower data volume of the encoded video data (404) (or the encoded video bitstream (404)), which may be stored on a streaming server (405) for future use or directly for use with downstream video devices (not shown). One or more streaming client subsystems, such as Figure 4Client subsystems (406) and (408) can access a streaming server (405) to retrieve copies (407) and (409) of encoded video data (404). Client subsystem (406) may include, for example, a video decoder (410) in an electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and produces an uncompressed output video picture stream (411) that can be displayed on a display (412) (e.g., a screen) or another presentation device (not depicted). The video decoder 410 may be configured to perform some or all of the functions described in this disclosure. In some streaming systems, the encoded video data (404), video data (407), and video data (409) (e.g., video streams) may be encoded according to certain video encoding / compression standards. Examples of such standards include ITU-T H.265. In this embodiment, the video coding standard under development is informally referred to as next-generation video coding (VVC), and this application can be used in the context of the VVC standard and other video coding standards.

[0055] It should be noted that electronic devices (420) and (430) may include other components (not shown). For example, electronic device (420) may include a video decoder (not shown), and electronic device (430) may also include a video encoder (not shown).

[0056] Figure 5 This is a block diagram of a video decoder (510) according to an embodiment disclosed below. The video decoder (510) may be disposed in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., receiving circuitry). The video decoder (510) may be used in place of... Figure 4 The video decoder (410) in the embodiment.

[0057] The receiver (531) may receive one or more encoded video sequences to be decoded by the video decoder (510); in the same embodiment or another embodiment, one encoded video sequence is decoded at a time, wherein the decoding of each encoded video sequence is independent of other encoded video sequences. Each video sequence may be associated with multiple video frames or images. Encoded video sequences may be received from a channel (501), which may be a hardware / software link to a storage device storing encoded video data or a streaming source transmitting encoded video data. The receiver (531) may receive encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams that may be forwarded to their respective processing circuits (not shown). The receiver (531) may separate the encoded video sequences from other data. To prevent network jitter, a buffer memory (515) may be configured between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) may be implemented as part of the video decoder (510). In other applications, the buffer memory (515) may be located outside the video decoder (510) and separate from it (not shown). In other applications, the buffer memory (not shown) may be located outside the video decoder (510) to prevent network jitter, for example, while another buffer memory (515) may exist inside the video decoder (510) for, for example, handling broadcast timing. When the receiver (531) receives data from a store / forward device with sufficient bandwidth and controllability or from an isochronous synchronization network, the buffer memory (515) may not be necessary, or it may be made smaller. Of course, for use on packet networks such as the Internet, a buffer memory (515) of sufficient size may be required, and the buffer memory may be relatively large. Such a buffer memory may have an adaptive size and may be at least partially implemented in the operating system or a similar component (not shown) outside the video decoder (510).

[0058] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. These symbols may or may not be part of the electronic device (530), but may be coupled to the electronic device (530), such as... Figure 5As shown in the diagram. The control information for the display device may be a parameter set fragment (not shown) of Supplemental Enhancement Information (SEI message) or Video Usability Information (VUI). The parser (520) may perform parsing / entropy decoding on the encoded video sequence received by the parser (520). The entropy coding of the encoded video sequence may be performed according to video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) may extract a subgroup parameter set of at least one subgroup of pixels in the subgroup from the encoded video sequence for use in the video decoder based on at least one parameter corresponding to the subgroup. The subgroup may include Group of Pictures (GOP), picture, tile, slice, macroblock, Coding Unit (CU), block, Transform Unit (TU), Prediction Unit (PU), etc. The parser (520) can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0059] The parser (520) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515) to create symbols (521).

[0060] Depending on the type of encoded video frames or portions thereof (e.g., inter-frame and intra-frame frames, inter-frame and intra-frame blocks) and other factors, the reconstruction of the symbol (521) may involve multiple different processing or functional units. Which units are involved and how they are involved can be controlled by the subgroup control information parsed from the encoded video sequence by the parser (520). For brevity, the flow of such subgroup control information between the parser (520) and the various processing or functional units described below is not described.

[0061] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into several functional units as described below. In practical embodiments operating under commercial constraints, many of these functional units interact closely with each other and can be integrated with one another. However, for the purpose of clearly describing the various functions of the disclosed subject matter, a conceptual subdivision of the functional units is adopted in the following disclosure.

[0062] The first unit may include a scaler / inverse transform unit (551). The scaler / inverse transform unit (551) receives quantization transform coefficients as symbols (521) from the parser (520) and control information, including information indicating which type of inverse transform to use, block size, quantization factor / parameter, quantization scaling matrix, etc. The scaler / inverse transform unit (551) may output a block including sample values, which may be input into the aggregator (555).

[0063] In some cases, the output samples of the scaler / inverse transform unit (551) may belong to intra-coded blocks; for example, blocks that do not use predictive information from previously reconstructed images but use predictive information from previously reconstructed portions of the current image. Such predictive information may be provided by the intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) uses the reconstructed surrounding block information stored in the current picture buffer (558) to generate surrounding blocks of the same size and shape as the block being reconstructed. For example, the current picture buffer (558) buffers partially reconstructed and / or fully reconstructed current images. In some implementations, the aggregator (555) adds the predictive information generated by the intra-picture prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) based on each sample.

[0064] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to inter-frame coding and latent motion compensation blocks. In this case, the motion compensation prediction unit (553) may access the reference image memory (557) to extract samples for inter-frame image prediction. After motion compensation is performed on the extracted samples according to the symbols (521), these samples may be added by the aggregator (555) to the output of the scaler / inverse transform unit (551) (the output of unit 551 is referred to as residual samples or residual signals), thereby generating output sample information. The motion compensation prediction unit (553) may obtain the predicted samples from the address in the reference image memory (557) under motion vector control, and the motion vector is available to the motion compensation prediction unit (553) in the form of the symbols (521), which may include, for example, X, Y components (displacement) and reference image components (time). Motion compensation may also include interpolation of sample values ​​extracted from the reference image memory (557) when using subsample precise motion vectors, and motion compensation may also be associated with a motion vector prediction mechanism, etc.

[0065] The output samples of the aggregator (555) can be employed by various loop filtering techniques in the loop filter unit (554). Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video sequence (also referred to as the encoded video stream), and these parameters can be used as symbols (521) from the parser (520) in the loop filter unit (556). However, in other embodiments, the video compression techniques may also respond to metadata obtained during decoding of a previous (in decoding order) portion of the encoded picture or encoded video sequence, and to previously reconstructed and loop-filtered sample values. Several types of loop filters may be included in various orders as part of the loop filter unit 556, as will be described in further detail below.

[0066] The output of the loop filter unit (556) can be a sample stream, which can be output to a display device (512) and stored in a reference image memory (557) for subsequent inter-frame image prediction.

[0067] Once fully reconstructed, some of the encoded images can be used as reference images for future predictions. For example, once the encoded images corresponding to the current image have been fully reconstructed and the encoded images (by, for example, the parser (520)) are identified as reference images, the current image buffer (558) can become part of the reference image memory (557), and a new current image buffer can be reallocated before the reconstruction of subsequent encoded images begins.

[0068] The video decoder (510) can perform decoding operations according to a predetermined video compression technique, such as that used in the ITU-T H.265 standard. The encoded video sequence may conform to the syntax specified by the video compression technique or standard in the sense that the encoded video sequence follows the syntax of the video compression technique or standard and the configuration file recorded in the video compression technique or standard. Specifically, the configuration file may select certain tools from all available tools in the video compression technique or standard as the only tools available under said configuration file. For compliance, the complexity of the encoded video sequence is within the limits defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum image size, maximum frame rate, maximum reconstruction sampling rate (measured in megasamples per second, for example), maximum reference image size, etc. In some cases, the limitations set by the hierarchy can be further limited by the Hypothetical Reference Decoder (HRD) specification and the metadata managed by the HRD buffer, which is represented by signals in the encoded video sequence.

[0069] In one embodiment, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be a portion of the encoded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0070] Figure 6 This is a block diagram of a video encoder (603) according to an embodiment disclosed in this application. The video encoder (603) is disposed in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuit). The video encoder (603) can be used to replace... Figure 4 The video encoder (403) in the embodiment.

[0071] The video encoder (603) can obtain data from the video source (601) (not) Figure 6 In one embodiment, the electronic device (620) receives video samples, the video source being capable of capturing video images to be encoded by a video encoder (603). In another embodiment, the video source (601) may be implemented as part of the electronic device (620).

[0072] A video source (601) can provide a sequence of source video samples encoded by a video encoder (603) in the form of a digital video sample stream, which may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601YCrCb, RGB, XYZ, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (601) may be a storage device capable of storing previously prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. Video data can be provided as multiple individual pictures or images, which are given motion when viewed sequentially. A picture itself can be constructed as a spatial pixel array, where each pixel may include one or more samples depending on the sampling structure, color space, etc., being used. Those skilled in the art can easily understand the relationship between pixels and samples. The following focuses on describing samples.

[0073] According to an embodiment, the video encoder (603) can encode and compress images of a source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing an appropriate encoding rate is a function of the controller (650). In some embodiments, the controller (650) controls and is functionally coupled to other functional units described below. For simplicity, coupling is not shown in the figures. Parameters set by the controller (650) may include rate control related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of pictures (GOP) layout, maximum allowed motion vector search range, etc. The controller (650) may be used with other suitable functions related to the video encoder (503) optimized for a particular system design.

[0074] In some embodiments, the video encoder (603) operates within an encoding loop. As a simplified description, in an embodiment, the encoding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on the input image to be encoded and a reference image) and a (local) decoder (633) embedded within the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data, even though the embedded decoder 633 processes the encoded video stream through the source encoder 630 without entropy encoding (because any compression between the symbols and the encoded video stream is lossless in the video compression techniques considered in this application). The reconstructed sample stream (sample data) is input to a reference image memory (634). Since decoding of the symbol stream produces bit-precise results independent of the decoder's location (local or remote), the contents of the reference image memory (634) are also bit-precisely corresponding between the local and remote encoders. In other words, the reference image samples "seen" by the encoder's prediction portion are exactly the same sample values ​​that the decoder will "see" when using the prediction during decoding. This basic principle of reference picture synchronization (and the drift that occurs when synchronization cannot be maintained, for example, due to channel errors) is used to improve coding quality.

[0075] The operation of the “local” decoder (633) can be combined with, for example, the above-mentioned Figure 5 The video decoder (510) is described in detail as the same as the "remote" decoder. However, a further brief reference is provided. Figure 5 When symbols are available and the entropy encoder (645) and parser (520) are able to encode / decode the symbols into an encoded video sequence without loss, the entropy decoding portion of the video decoder (510), including the buffer (515) and parser (520), may not be fully implemented in the encoder’s local decoder (633).

[0076] It can be observed that any decoder technique other than parsing / entropy decoding present in the decoder must also exist in the corresponding encoder in essentially the same functional form. For this reason, this application sometimes focuses on decoder operations, which are related to the decoding part of the encoder. The description of the encoder technique can be simplified because the encoder technique is the inverse of the fully described decoder technique. The encoder is described in more detail below only in certain areas or aspects.

[0077] During operation, in some embodiments, the source encoder (630) may perform motion-compensated predictive coding. The motion-compensated predictive coding predictively encodes the input image, referencing one or more previously encoded images designated as “reference images” from the video sequence. In this manner, the encoding engine (632) encodes the differences (residues) between pixel blocks in the color channels of the input image and pixel blocks in the reference image, which may be selected as a predictive reference for the input image. The term “residue” and its adjective form “residual” are used interchangeably.

[0078] The local video decoder (633) can decode encoded video data of a picture that can be designated as a reference picture, based on symbols created by the source encoder (630). The operation of the encoding engine (632) can be a lossy process. When the encoded video data can be decoded by the video decoder (633), Figure 6 When the source video sequence (not shown) is decoded, the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process, which can be performed by the video decoder on the reference image, and allows the reconstructed reference image to be stored in a reference image cache (634). In this way, the video encoder (603) can locally store a copy of the reconstructed reference image that shares the same content (no transmission errors) as the reconstructed reference image to be obtained by the remote video decoder.

[0079] The predictor (635) can perform a prediction search against the encoding engine (632). That is, for a new image to be encoded, the predictor (635) can search in the reference image memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that can serve as appropriate prediction references for the new image. The predictor (635) can operate pixel-by-pixel based on the sample blocks to find suitable prediction references. In some cases, based on the search results obtained by the predictor (635), it can be determined that the input image may have prediction references obtained from multiple reference images stored in the reference image memory (634).

[0080] The controller (650) can manage the encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding video data.

[0081] The outputs of all the above-mentioned functional units can be entropy encoded in the entropy encoder (645). The entropy encoder (645) performs lossless compression on the symbols generated by the various functional units according to techniques such as Huffman coding, variable length coding, and arithmetic coding, thereby converting the symbols into an encoded video sequence.

[0082] The transmitter (640) can buffer the encoded video sequence created by the entropy encoder (645) in preparation for transmission via a communication channel (660), which may be a hardware / software link to a storage device that will store the encoded video data. The transmitter (640) can combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0083] The controller (650) manages the operation of the video encoder (603). During encoding, the controller (650) can assign a specific encoded image type to each encoded image, but this may affect the encoding techniques applicable to the corresponding images. For example, images can typically be assigned to any of the following image types:

[0084] An intra-frame picture (I-picture) is a picture that can be encoded and decoded without using any other pictures in the sequence as a prediction source. Some video codecs allow different types of intra-frame pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art will understand variations of I-pictures and their corresponding applications and characteristics.

[0085] A predictive image (P-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0086] A bidirectional predictive image (B-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most two motion vectors and a reference index to predict sample values ​​for each block. Similarly, multiple predictive images can use more than two reference images and associated metadata to reconstruct a single block.

[0087] Source images are typically spatially subdivided into multiple sample blocks (e.g., 4×4, 8×8, 4×8, or 16×16 sample blocks), and encoded block by block. These blocks can be predictively coded with reference to other (already coded) blocks, determined based on the coding assignments of the corresponding images applied to the blocks. For example, blocks of an I-image can be non-predictively coded, or the blocks can be predictively coded (spatial or intra-frame prediction) with reference to already coded blocks of the same image. Pixel blocks of a P-image can be predictively coded with reference to a previously coded reference image via spatial or temporal prediction. Blocks of a B-image can be predictively coded with reference to one or two previously coded reference images via spatial or temporal prediction. For other purposes, source images or intermediate images can be subdivided into other types of blocks. The partitioning of coded blocks and other types of blocks may or may not follow the same manner, as described in further detail below.

[0088] The video encoder (603) can perform encoding operations according to a predetermined video coding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (603) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

[0089] In this embodiment, the transmitter (640) may transmit additional data while transmitting encoded video. The source encoder (630) may include such data as part of the encoded video sequence. Additional data may include temporal / spatial / SNR enhancement layers, redundant images and slices, other forms of redundant data, SEI messages, VUI parameter set fragments, etc.

[0090] The acquired video can serve as multiple source images (video images) presented in a time series. Intra-frame image prediction (often simplified to intra-frame prediction) utilizes spatial correlations within a given image, while inter-frame image prediction utilizes (temporal or other) correlations between images. In an embodiment, a specific image being encoded / decoded is segmented into blocks, referred to as the current image. When a block in the current image resembles a reference block in a previously encoded and still buffered reference image in the video, the block in the current image can be encoded using a vector called a motion vector. This motion vector points to the reference block in the reference image, and when multiple reference images are used, the motion vector may have a third dimension that identifies the reference image.

[0091] In some embodiments, bidirectional prediction techniques can be used for inter-frame image prediction. According to bidirectional prediction, two reference images are used, such as a first reference image and a second reference image, both preceding the current image in the video in decoding order (but possibly past or future in display order). A block in the current image can be encoded using a first motion vector pointing to a first reference block in the first reference image and a second motion vector pointing to a second reference block in the second reference image. Specifically, the block can be predicted jointly using a combination of the first and second reference blocks.

[0092] In addition, merging mode techniques can be used in inter-frame image prediction to improve coding efficiency.

[0093] According to some embodiments disclosed in this application, predictions such as inter-frame image prediction and intra-frame image prediction are performed on a block-by-block basis. For example, images in a video image sequence are segmented into coding tree units (CTUs) for compression, with each CTU having the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU comprises three parallel coding tree blocks (CTBs): one luma CTB and two chroma CTBs. Furthermore, each CTU can be further subdivided into one or more coding units (CUs) using a quadtree. For example, a 64×64 pixel CTU can be subdivided into one 64×64 pixel CU, or four 32×32 pixel CUs. Each of the one or more 32×32 blocks can be further subdivided into four 16×16 pixel CUs. In one embodiment, each CU can be analyzed during encoding to determine the prediction type used for the CU among various prediction types, such as inter-frame prediction or intra-frame prediction. Furthermore, depending on temporal and / or spatial predictability, the CU is divided into one or more prediction units (PUs). Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In embodiments, prediction operations in encoding (encoding / decoding) are performed on a per-prediction-block basis. The division of the CU into PUs (or PBs for different color channels) can be performed in various spatial modes. For example, a luma or chroma PB may comprise a matrix of sample values ​​(e.g., luma values), such as 8x8 pixels, 16x16 pixels, 8x16 pixels, and 16x8 samples, etc.

[0094] Figure 7This is a diagram of a video encoder (703) according to another illustrative embodiment disclosed in this application. The video encoder (703) is used to receive processing blocks (e.g., prediction blocks) of sample values ​​within a current video image in a video image sequence, and to encode the processing blocks into an encoded image that is part of an encoded video sequence. In this embodiment, the video encoder (703) is used instead of Figure 4 The video encoder (303) in the embodiment.

[0095] In one embodiment, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as an 8×8 sample prediction block. The video encoder (703) then uses, for example, rate-distortion optimization (RDO) to determine whether to encode the processing block using an intra-frame mode, an inter-frame mode, or a bidirectional prediction mode. When it is determined that the processing block is to be encoded in intra-frame mode, the video encoder (703) can use intra-frame prediction techniques to encode the processing block into an encoded picture; and when it is determined that the processing block is to be encoded in inter-frame mode or a bidirectional prediction mode, the video encoder (703) can use inter-frame prediction or bidirectional prediction techniques to encode the processing block into an encoded picture, respectively. In some illustrative embodiments, the merging mode may be a sub-mode of inter-frame picture prediction, wherein motion vectors are derived from one or more motion vector prediction values ​​without relying on encoded motion vector components outside the prediction values. In some other illustrative embodiments, motion vector components applicable to the subject block may exist. Therefore, the video encoder (703) includes methods for encoding the processing block in the inter-frame mode. Figure 7 Components explicitly shown include, for example, a pattern decision module (not shown) used to determine the prediction pattern of a processing block.

[0096] exist Figure 7 In one embodiment, the video encoder (703) includes, as shown below: Figure 7 The exemplary arrangement shows an inter-frame encoder (730), an intra-frame encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together.

[0097] An inter-frame encoder (730) is configured to receive samples of the current block (e.g., the processing block), compare the block with one or more reference blocks in a reference image (e.g., blocks in previous and subsequent images in display order), generate inter-frame prediction information (e.g., descriptions based on redundancy information of the inter-frame coding technique, motion vectors, merging mode information), and calculate inter-frame prediction results (e.g., predicted blocks) based on the inter-frame prediction information using any suitable technique. In some embodiments, the reference image is used using embedded... Figure 6 Decoding unit 633 in example encoder 620 (e.g. Figure 7 As shown in the residual decoder 728 (see below), a decoded reference image is decoded based on the encoded video information.

[0098] The intra encoder (722) is used to receive samples of the current block (e.g., the processing block), in some cases compare the block with previously encoded blocks in the same image, generate quantization coefficients after transformation, and in some cases also (e.g., based on intra prediction direction information from one or more intra coding techniques) generate intra prediction information. The intra encoder (722) calculates intra prediction results (e.g., prediction blocks) based on the intra prediction information and reference blocks in the same image.

[0099] A general controller (721) is used to determine general control data and, based on the general control data, control other components of the video encoder (703). In an embodiment, the general controller (721) determines the prediction mode of a block and, based on the prediction mode, provides control signals to a switch (726). For example, when the prediction mode is an intra-frame mode, the general controller (721) controls the switch (726) to select an intra-frame mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select intra-frame prediction information and add the intra-frame prediction information to the bitstream; and when the prediction mode for a block is an inter-frame mode, the general controller (721) controls the switch (726) to select an inter-frame prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select inter-frame prediction information and add the inter-frame prediction information to the bitstream.

[0100] A residual calculator (723) is used to calculate the difference (residual data) between the received block and the prediction result of a block selected from the intra encoder (722) or the inter encoder (730). A residual encoder (724) is used to encode the residual data to generate transform coefficients. In an embodiment, the residual encoder (724) is used to transform the residual data from the time domain to the frequency domain to generate transform coefficients. The transform coefficients are then quantized to obtain quantized transform coefficients. In various illustrative embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is used to perform an inverse transform and generate decoded residual data. The decoded residual data can be used appropriately by the intra encoder (722) and the inter encoder (730). For example, the inter encoder (730) can generate a decoded block based on the decoded residual data and inter-frame prediction information, and the intra encoder (722) can generate a decoded block based on the decoded residual data and intra-frame prediction information. The decoded blocks are processed appropriately to generate a decoded image, which can be buffered in a memory circuit (not shown) and used as a reference image.

[0101] An entropy encoder (725) is used to format the bitstream to produce encoded blocks and perform entropy encoding. The entropy encoder (725) is configured to include various information in the bitstream. In an embodiment, the entropy encoder (725) is used to obtain general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the bitstream. It should be noted that residual information is not present when blocks are encoded in a merged sub-mode of inter-frame mode or bidirectional prediction mode.

[0102] Figure 8 This is a diagram of a video decoder (810) according to another embodiment disclosed in this application. The video decoder (810) is used to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed picture. In an embodiment, the video decoder (810) is used instead of Figure 4 The video decoder (410) in the embodiment.

[0103] exist Figure 8 In the embodiment, the video decoder (810) includes, as follows: Figure 8 The schematic arrangement shows an entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-frame decoder (872) coupled together.

[0104] An entropy decoder (871) can be used to reconstruct certain symbols from an encoded image, these symbols representing the syntax elements constituting the encoded image. Such symbols may include, for example, a mode used to encode the block (e.g., intra-frame mode, inter-frame mode, bidirectional prediction mode, merged sub-mode, or another sub-mode), prediction information (e.g., intra-frame prediction information or inter-frame prediction information) that can identify certain samples or metadata used by the intra-frame decoder (872) or inter-frame decoder (880) for prediction, residual information in the form of, for example, quantized transform coefficients, and so on. In an embodiment, when the prediction mode is inter-frame or bidirectional prediction mode, inter-frame prediction information is provided to the inter-frame decoder (880); and when the prediction type is intra-frame prediction type, intra-frame prediction information is provided to the intra-frame decoder (872). Residual information may be provided to the residual decoder (873) via inverse quantization.

[0105] The inter-frame decoder (880) is used to receive inter-frame prediction information and generate inter-frame prediction results based on the inter-frame prediction information.

[0106] The intra-frame decoder (872) is used to receive intra-frame prediction information and generate prediction results based on the intra-frame prediction information.

[0107] The residual decoder (873) performs inverse quantization to extract the dequantized transform coefficients and processes the dequantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (873) may also use some control information (to obtain the quantizer parameter QP), which may be provided by the entropy decoder (871) (the data path is not indicated because this is only low-data-volume control information).

[0108] The reconstruction module (874) combines the residual output by the residual decoder (873) with the prediction result (which may be output by the inter-frame prediction module or the intra-frame prediction module) in the spatial domain to form a reconstructed block, which forms part of a reconstructed image, which may be used as part of a reconstructed video. It should be noted that other suitable operations, such as deblocking, may be performed to improve visual quality.

[0109] It should be noted that any suitable technology can be used to implement the video encoder (403), video encoder (603), and video encoder (703), as well as the video decoder (410), video decoder (510), and video decoder (810). In one embodiment, one or more integrated circuits can be used to implement the video encoder (403), video encoder (603), and video encoder (703), as well as the video decoder (410), video decoder (510), and video decoder (810). In another embodiment, one or more processors executing software instructions can be used to implement the video encoder (403), video encoder (603), and video encoder (703), as well as the video decoder (410), video decoder (510), and video decoder (810).

[0110] Returning to the intra-prediction process, samples in a block (e.g., a luma or chroma prediction block, or a coded block that is not further segmented into prediction blocks) are predicted from samples in adjacent, next adjacent, or other one or more rows, or a combination thereof, to generate a prediction block. The residual between the correctly encoded ground block and the prediction block can then be processed by transform, followed by quantization. Various intra-prediction modes can be made available, and parameters and other parameters related to intra-prediction mode selection can be signaled in the bitstream. For example, various intra-prediction modes may involve one or more row positions for predicting samples, the direction along which predicting samples are selected from one or more prediction rows, and other special intra-prediction modes.

[0111] For example, the set of intra-prediction modes (which may be interchangeably referred to as "intra-modes") may include a predetermined number of directional intra-prediction modes. As described above with respect to the example embodiment of FIG1, these intra-prediction modes may correspond to a predetermined number of directions along which out-of-block samples are selected for prediction of samples being predicted in a particular block. In another particular example embodiment, eight (8) primary directional modes corresponding to angles of 45 to 207 degrees with respect to the horizontal axis may be supported and predetermined.

[0112] In some other implementations of intra-frame prediction, to further utilize more types of spatial redundancy in directional textures, the directional intra-frame mode can be further extended to an angle set with finer granularity. For example, the above eight angle implementations can be configured to provide eight nominal angles, such as... Figure 9 The diagram illustrates these as V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED, and for each nominal angle, a predetermined number (e.g., 7) of finer angles can be added. With such expansion, corresponding to the same number of predetermined directional intra-frame modes, a larger total number of directional angles (e.g., 56 in this example) are available for intra-frame prediction. The predicted angle can be represented by the nominal intra-frame angle plus an angle increment. For the specific example above with 7 finer angular directions for each nominal angle, the angle increment can be -3 to 3 times a 3-degree step.

[0113] In some implementations, as an alternative to or supplement to the aforementioned directional intra-frame modes, a predetermined number of non-directional intra-frame prediction modes can also be predefined and provided. For example, five non-directional intra-frame modes, referred to as smooth intra-frame prediction modes, can be specified. These non-directional intra-frame prediction modes can be specifically referred to as DC, PAETH, SMOOTH, SMOOTH_V, and SMOOTH_H intra-frame modes. Figure 10 The image illustrates the predictions for samples of a specific block under these example non-directional modes. As an example, Figure 10The diagram shows a 4×4 block 1002 being predicted by samples from the top adjacent row and / or the left adjacent row. A specific sample 1010 in block 1002 can correspond to sample 1004 directly above sample 1010 in the top adjacent row of block 1002, sample 1006 to the left of sample 1010 which is the intersection of the top and left adjacent rows, and sample 1008 directly to the left of sample 1010 in the left adjacent row of block 1002. For the example DC intra-prediction mode, the average of the left and top adjacent samples 1008 and 1004 can be used as the predictor for sample 1010. For the example PAETH intra-prediction mode, top, left, and top-left reference samples 1004, 1008, and 1006 can be extracted, and the value closest among these three reference samples (top sample plus left sample minus top-left sample (top+left–topleft)) can be set as the predictor for sample 1010. For the example SMOOTH_V intra-prediction mode, sample 1010 can be predicted by quadratic interpolation in the vertical direction between the upper-left adjacent sample 1006 and the left adjacent sample 1008. For the example SMOOTH_H intra-prediction mode, sample 1010 can be predicted by quadratic interpolation in the horizontal direction between the upper-left adjacent sample 1006 and the upper adjacent sample 1004. For the example SMOOTH intra-prediction mode, sample 1010 can be predicted by the average of the quadratic interpolations in the vertical and horizontal directions. The above non-directional intra-prediction mode implementations are only illustrated as non-limiting examples. Other non-directional choices of adjacent rows and samples, as well as combinations of predicted samples used to predict specific samples in a prediction block, are also considered.

[0114] The encoder can be signaled within the bitstream to select a specific intra-prediction mode from the above directional or non-directional modes at various coding levels (pictures, slices, blocks, units, etc.). In some example implementations, eight exemplary nominal directional modes and five non-angular smoothing modes (a total of 13 options) can be signaled first. Then, if the signaled mode is one of the eight nominal angular intra-prediction modes, the signaled index is further used to indicate the selected angle increment as the corresponding nominal angle signaled. In some other example implementations, all intra-prediction modes can be indexed together (e.g., 56 directional modes plus 5 non-directional modes to produce 61 intra-prediction modes) for signaling.

[0115] In some example implementations, example 56 or other numbers of directional intra-prediction modes can be implemented using a unified directional predictor that projects each sample of a block to a reference subsample location and interpolates the reference sample using a 2-tap bilinear filter.

[0116] In some implementations, additional filter patterns, referred to as filter intra-modes, can be designed to capture the attenuation space associated with references on the edges. For these modes, intra-block predicted samples, in addition to out-of-block samples, can be used as intra-prediction reference samples for some smaller blocks within the block. For example, these modes can be predefined and made available for intra-prediction of at least luma blocks (or luma blocks only). A predetermined number (e.g., 5) of filter intra-modes can be pre-designed, each represented by a set of n-tap filters (e.g., 7-tap filters) reflecting the correlation between samples in, for example, a 4×2 block and its n neighboring samples. In other words, the weighting factors for the n-tap filters can be position-dependent. For example, with 8×8 blocks, 4×2 blocks, and 7-tap filtering, as... Figure 11 As shown, the 8×8 block 1102 can be divided into eight 4×2 smaller blocks. In Figure 11 In the diagram, these small blocks are represented by B0, B1, B2, B3, B4, B5, B6, and B7. For each small block, its 7 adjacent small blocks (in...) Figure 11 The values ​​(denoted by R0 to R7) can be used to predict samples within the current block. For block B0, all neighboring blocks may have already been reconstructed. However, for other blocks, some of the neighboring blocks are in the current block and therefore may not have been reconstructed; the nearest neighbor predictions are then used as a reference. For example, Figure 11 None of the adjacent blocks of the small block B7 shown in the diagram were reconstructed, so the predicted samples of the adjacent blocks were used instead.

[0117] In some implementations of intra-frame prediction, a color component can be predicted using one or more other color components. The color component can be any component from the YCrCb, RGB, XYZ color spaces, etc. For example, prediction of a chromaticity component (e.g., a chromaticity patch) (referred to as chromaticity from luminance, or CfL) from a luminance component (e.g., a luminance reference sample) can be implemented. In some example implementations, cross-color prediction may only allow from luminance to chromaticity. For example, chromaticity samples in a chromaticity patch can be modeled as a linear function conforming to the reconstructed luminance samples. CfL prediction can be implemented as follows:

[0118] CfL(α)=α×L AC +DC (1)

[0119] Where L ACLet α represent the AC contribution of the luminance component, α represent the parameters of the linear model, and DC represent the DC contribution of the chrominance component. For example, an AC component is obtained for each sample of the block, while a DC component is obtained for the entire block. Specifically, the reconstructed luminance samples can be subsampled to the chrominance resolution, and then the average luminance value (DC of luminance) can be subtracted from each luminance value to form the AC contribution in luminance. The AC contribution of luminance is then used in the linear mode of equation (1) to predict the AC value of the chrominance component. To approximate or predict the chrominance AC component from the luminance AC contribution, instead of requiring the decoder to calculate the scaling parameters, the example CfL implementation can determine the parameters α based on the original chrominance samples and signal them in the bitstream. This reduces the decoder complexity and produces more accurate predictions. As for the DC contribution of the chrominance component, in some example implementations, this DC contribution can be calculated by using the intra-frame DC mode within the chrominance component.

[0120] The residuals of intra-frame or inter-frame prediction blocks can then be transformed, followed by quantization of the transform coefficients. For the purpose of performing the transform, both intra-frame and inter-frame coded blocks can be further partitioned into multiple transform blocks (sometimes used interchangeably as "transform units," even though the term "unit" is generally used to refer to a set of three-color channels; for example, a "coding unit" would include luma-coded blocks and chroma-coded blocks). In some implementations, the maximum partition depth of the coded block (or prediction block) can be specified (the term "coded block" can be used interchangeably with "coded block"). For example, such partitioning can not exceed two levels. The partitioning of prediction blocks into transform blocks can be handled differently between intra-frame and inter-frame prediction blocks. However, in some implementations, such partitioning between intra-frame and inter-frame prediction blocks can be similar.

[0121] In some example implementations, for intra-frame coded blocks, transform partitioning can be done in such a way that all transform blocks have the same size, and transform blocks are encoded in raster scan order. Figure 12 An example of such transform block partitioning of an intra-frame coded block is shown. Specifically, Figure 12 The illustration shows that coded block 1202 is partitioned into 16 transform blocks of the same size via intermediate-level quadtree partitioning 1204, as shown in 1206. The example raster scan order used for encoding is... Figure 12 The diagram shows ordered arrows.

[0122] In some example implementations, and for inter-frame coded blocks, transform unit partitioning can be performed recursively, where the partitioning depth reaches a predefined number of levels (e.g., 2 levels). Figure 13 As shown, for any subpartition and at any level, the split can stop or continue recursively. In particular, Figure 13An example is given where block 1302 is divided into four quadtree sub-blocks 1304, and one of these sub-blocks is further divided into four second-level transform blocks, while the division of the other sub-blocks stops after the first level, resulting in a total of seven transform blocks of two different sizes. The example raster scan order used for encoding is further... Figure 13 The ordered arrow diagram in the image. Although Figure 13 An example implementation of quadtree partitioning for up to two levels of square transform blocks is shown, but in some generated implementations, transform partitioning can support 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1 transform block shapes and sizes ranging from 4×4 to 64×64. In some example implementations, if the coded block is less than or equal to 64×64, the transform block partitioning can be applied only to the luma component (in other words, the chroma transform block will be the same as the coded block under this condition). Otherwise, if the coded block width or height is greater than 64, both the luma and chroma coded blocks can be implicitly partitioned into multiples of min(W, 64)×min(H, 64) and min(W, 32)×min(H, 32) transform blocks, respectively.

[0123] Then, the primary transform can be performed on each of the transform blocks above. The primary transform essentially moves the residuals in the transform block from the spatial domain to the frequency domain. In some implementations of the actual primary transform, to support the extended coding block partitioning in the example above, multiple transform sizes (ranging from 4 to 64 points for each of the two dimensions) and multiple transform shapes (squares; rectangles with width / height ratios of 2:1 / 1:2 and 4:1 / 1:4) can be allowed.

[0124] Turning to practical primary transforms, in some example implementations, the 2-D transform process can involve the use of hybrid transform kernels (e.g., which can consist of different 1-D transforms for each dimension of the encoded residual transform block). Example 1-D transform kernels can include, but are not limited to: a) 4-point, 8-point, 16-point, 32-point, and 64-point DCT-2; b) 4-point, 8-point, and 16-point asymmetric DSTs (DST-4, DST-7) and their inverted versions; c) 4-point, 8-point, 16-point, and 32-point identity transforms. The choice of transform kernel for each dimension can be based on rate-distortion (RD) criteria. For example, Table 1 lists the basis functions for implementable DCT-2 and asymmetric DSTs.

[0125] Table 1: Example primary transform basis functions (DCT-2, DST-4, and DST-7 for N-point inputs)

[0126]

[0127] In some example implementations, the availability of the hybrid transform kernel can be based on the transform block size and prediction mode for a specific primary transform implementation. Example dependencies are listed in Table 2. For chroma components, transform type selection can be performed implicitly. For example, for intra-frame prediction residuals, the transform type can be selected based on the intra-frame prediction mode, as shown in Table 3. For inter-frame prediction residuals, the transform type of the chroma block can be selected based on the transform type selection of the luma blocks at the same location. Therefore, for chroma components, there is no transform type signaling in the bitstream.

[0128] Table 2: AV1 Hybrid Transform Kernels and their Availability Based on Prediction Mode and Block Size. Here, → and ↓ represent horizontal and vertical dimensions; √ and × represent the availability of kernels for that block size and prediction mode.

[0129]

[0130]

[0131] Table 3: Transformation Type Selection for Intra-Frame Prediction Residuals of Chroma Components

[0132]

[0133]

[0134] In some implementations, secondary transformations can be performed on the primary transform coefficients. For example, such as... Figure 14 As shown, LFNST (low-frequency non-separable transform) (which is referred to as a reduced secondary transform) can be applied between the forward primary transform and quantization (on the encoder side) and between dequantization and the inverse primary transform (on the decoder side) to further decorrelate the primary transform coefficients. Essentially, LFNST can continue the secondary transform on a portion of the primary transform coefficients, such as the low-frequency portion (thus "reduced" from the complete set of primary transform coefficients in the transform block). In the example LFNST, a 4×4 non-separable transform or an 8×8 non-separable transform can be applied depending on the transform block size. For example, a 4×4 LFNST can be applied to a small transform block (e.g., min(width, height) < 8), while an 8×8 LFNST can be applied to a larger transform block (e.g., min(width, height) > 8). For example, if a 4×4 LFNST is performed on an 8×8 transform block, only the low-frequency 4×4 portion of the 8×8 primary transform coefficients undergoes further secondary transform.

[0135] Specifically, such as Figure 14As shown, the transform block can be 8×8 (or 16×16). Therefore, the forward primary transform 1402 of the transform block produces an 8×8 (or 16×16) primary transform coefficient matrix 1404, where each square cell represents a 2×2 (or 4×4) portion. The input to the forward LFNST may not be all 8×8 (or 16×16) primary transform coefficients. For example, a 4×4 (or 8×8) LFNST can be used for the secondary transform. Thus, as indicated by the shaded portion (top left) 1406, only the 4×4 (or 8×8) low-frequency primary transform coefficients of the primary transform coefficient matrix 1404 can be used as the input to the LFNST. The remaining portion of the primary transform coefficient matrix may not undergo the secondary transform. Thus, after the secondary transform, the LFNST-treated portions of the primary transform coefficients become secondary transform coefficients, while the untreated portions (e.g., the unshaded portion of matrix 1404) retain the corresponding primary transform coefficients. In some example implementations, the remaining portion that does not undergo secondary transformation can be set to zero coefficients.

[0136] The following describes an example of an application of the non-separable transform used in LFNST. To apply the example 4×4 LFNST, a 4×4 input block X (representing, for example, the 4×4 low-frequency portion of the primary transform coefficient block, such as...) Figure 14 The shaded portion (1406) of the primary transformation matrix 1404 can be represented as:

[0137]

[0138] The 2D input matrix can first be linearized or scanned into a vector, as shown in the example.

[0139]

[0140] Then, the inseparable transformation of 4×4LFNST can be calculated as follows: in, This indicates the output transform coefficient vector, and T is a 16×16 transform matrix. The resulting 16×1 coefficient vector is then processed using the scan order of this block (e.g., horizontal, vertical, or diagonal). The reverse scan is a 4×4 block. Coefficients with smaller indices can be placed in a 4×4 coefficient block along with smaller scan indices. In this way, redundancy in the primary transform coefficients X can be further utilized via the second transform T, thereby providing additional compression enhancement.

[0141] The above example LFNST is based on a direct matrix multiplication method that applies a non-separable transform, enabling the LFNST to be implemented in a single pass without multiple iterations. Further, in some example embodiments, the dimension of the non-separable transform matrix (T) of, for example, a 4×4 LFNST can be further reduced to minimize the computational complexity and the memory space requirements for storing the transform coefficients. Such an approach can be referred to as a reduced non-separable transform (RST). More specifically, the main idea of RST is to map an N-dimensional vector (where N is 4×4 = 16 in the above example, but can be equal to 64 for an 8×8 block) to an R-dimensional vector in a different space, where N / R (R < N) represents the dimension reduction factor. Thus, instead of an N×N transform matrix, the RST matrix becomes an R×N matrix as follows:

[0142]

[0143] where the R rows of the transform matrix are the reduced R basis of the N-dimensional space. Thus, this transform converts the input vector or N-dimension into an output vector of reduced R-dimension. In this way, and as Figure 14 shown, the secondary transform coefficients 1408 transformed from the primary coefficients 1406 are reduced in dimension by a factor or N / R. Figure 14 The three squares around 1408 in

[0144] can be zero-padded. The inverse transform matrix of RTS can be the transpose of its forward transform. For an example 8×8 LFNST (contrasted with the above 4×4 LFNST for more different descriptions here), an example reduction factor of 4 can be applied, and thus the 64×64 direct non-separable transform matrix is correspondingly reduced to a 16×64 direct matrix. Further, in some implementations, only a part rather than all of the input primary coefficients can be linearized into the input vector of the LFNST. For example, only a part of the example 8×8 input primary transform coefficients can be linearized into the above X vector. For a specific example, in the four 4×4 quadrants of the 8×8 primary transform coefficient matrix, the lower right (high-frequency coefficients) can be omitted, and only the other three quadrants are linearized into a 48×1 vector rather than a 64×1 vector using a predefined scan order. In such an implementation, the non-separable transform matrix can be further reduced from 16×64 to 16×48.

[0145] Therefore, a reduced 48×16 inverse RST matrix can be used on the decoder side to generate the top-left, top-right, and bottom-left 4×4 quadrants of the 8×8 core (primary) transform coefficients. Specifically, when a further reduced 16×48 RST matrix is ​​applied instead of a 16×64 RST with the same transform set configuration, the inseparable secondary transform takes 48 vectorized matrix elements as input from the three 4×4 quadrant blocks of the 8×8 primary coefficient block, excluding the bottom-right 4×4 block. In such an implementation, the omitted bottom-right 4×4 primary transform coefficients are ignored in the secondary transform. This further reduced transform converts the 48×1 vector into a 16×1 output vector, which is then scanned inversely into a 4×4 matrix to fill the gaps. Figure 14 The three squares surrounding the secondary transformation coefficients of 1408 can be filled with zeros.

[0146] This dimension reduction in RST reduces memory usage for storing all LFNST matrices. In the example above, memory usage can be reduced from 10KB to 8KB, for example, compared to an implementation without dimension reduction, with a relatively small performance degradation.

[0147] In some implementations, to reduce complexity, LFNST can be further restricted to only the portion of the primary transform coefficients that will undergo LFNST (e.g., in...). Figure 14 The LFNST is only applicable when all coefficients (excluding part 1406 of 1404) are invalid. Therefore, when applying the LFNST, all coefficients of only primary transformations (e.g., Figure 4 The unshaded portion of the primary coefficient matrix 1404 can be close to zero. This constraint allows for adjustment of the LFNST index signaling at the end of the valid position and thus avoids some extra coefficient scans that might be needed to check valid coefficients at specific positions when this constraint is not applied. In some implementations, the worst-case handling of LFNST (in terms of per-pixel multiplication) can constrain the inseparable transformations of 4×4 blocks and 8×8 blocks to 8×16 and 8×48 transformations, respectively. In these cases, the end of the valid scan position must be less than 8 for other sizes smaller than 16 when LFNST is applied. For blocks of shapes 4×N and N×4 where N>8, the above constraint means that LFNST is now applied only once to the top-left 4×4 region. Since all coefficients of the primary transformation only are zero when LFNST is applied, the number of operations required for the primary transformation is reduced in this case. From the encoder's perspective, coefficient quantization can be simplified when testing the LFNST transformation. For the first 16 coefficients (in scan order), rate-distortion optimized quantization (RDO) must be performed to the maximum extent possible, which can force the remaining coefficients to zero.

[0148] In some example implementations, the available RST kernels can be specified as multiple transform sets, where each transform set includes multiple inseparable transform matrices. For example, each transform set used in LFNST can have a total of 4 transform sets and 2 inseparable transform matrices (kernels). These kernels can be pre-trained offline and are therefore data-driven. Offline-trained transform kernels can be stored in memory or hard-coded in the encoding or decoding device for use during the encoding / decoding process. The selection of transform sets during the encoding or decoding process can be determined by the intra-prediction mode. A mapping from intra-prediction mode to transform sets can be predefined. Table 4 shows examples of such predefined mappings. For example, as shown in Table 4, transform set 0 can be selected for the current chroma block if one of the three cross-component linear model (CCLM) modes (INTRA_LT_CCLM, INTRA_T_CCLM, or INTRA_L_CCLM) is used for the current block (i.e., 81 ≤ predModeIntra ≤ 83). For each transform set, the selected inseparable secondary transform candidate can be further specified by an LFNST index explicitly signaled by the signal. For example, after the transform coefficients, the CU can signal the index once in the form of a bitstream within each frame.

[0149] Table 4: Transformation Selection Table

[0150]

[0151] Because LFNST is restricted to applying only if all coefficients outside the first coefficient subgroup or portion are invalid in the example implementation above, LFNST index encoding depends on the position of the last valid coefficient. Additionally, LFNST indices can be context-coded, but not dependent on the intra-prediction mode, and only the first binary number can be context-coded. Furthermore, LFNST can be applied to intra-CUs in both intra-slices and inter-slices, and to both luma and chroma. If dual-tree is enabled, the LFNST indices for luma and chroma can be signaled separately. For inter-slices (where dual-tree is disabled), a single LFNST index can be signaled and used for both luma and chroma.

[0152] In some example implementations, when the Intra-Frame Sub-Partition (ISP) mode is selected, LFNST can be disabled and the RST index can be ignored without signaling, because the performance improvement may be small even if RST is applied to every feasible partition block. Furthermore, disabling RST on the residuals of ISP predictions can reduce coding complexity. Further, in some implementations, when the Multilinear Regression Intra-Frame Prediction (MIP) mode is selected, LFNST can also be disabled, and the RST index can be ignored without signaling.

[0153] Considering that large CUs larger than 64×64 (or any other predefined size representing the maximum transform block size) are implicitly partitioned (e.g., TU tiling) due to existing maximum transform size limitations (e.g., 64×64), LFNST index search can quadruple the data buffer for a specific number of decoding pipeline stages. Therefore, in some implementations, the maximum size allowed by LFNST can be limited to, for example, 64×64. In some implementations, LFNST can be enabled using only DCT2 as the primary transform.

[0154] In some other implementations, intra-frame secondary transforms (ISTs) are provided for the luma component by defining, for example, 12 sets of secondary transforms (each set having, for example, 3 kernels). Intra-frame mode-related indices can be used for transform set selection. Kernel selection within a set can be based on syntax elements notified by a signal. ISTs can be enabled when DCT2 or ADST is used as the horizontal and vertical primary transforms. In some implementations, 4×4 or 8×8 non-separable transforms can be selected depending on the block size. A 4×4 IST can be selected if min(tx_width, tx_height) < 8. An 8×8 IST can be used for larger blocks. Here, tx_width and tx_height correspond to the transform block width and height, respectively. The input to the IST can be low-frequency primary transform coefficients in a zigzag scan order.

[0155] When using only separable transform schemes, various transforms during video encoding or decoding (e.g., primary transforms of samples in residual blocks or secondary transforms of blocks of primary transform coefficients) may not all be effective at capturing directional texture patterns (such as edges in a 45-degree direction (e.g., directions generally away from the horizontal or vertical)). As described above, in some example implementations, one or more non-separable transform designs can be used for secondary transforms of primary transform coefficients. As further described below, such non-separable transform schemes can also be used for primary transforms of residual sample blocks to generate primary transform coefficients. In some example implementations, primary transform coefficients generated via separable or non-separable transforms can be directly quantized and subsequently entropy encoded, or alternatively, separable or non-separable secondary transforms can be performed before quantization and entropy encoding. In implementations where primary transform coefficients obtained via non-separable transforms are further subjected to non-separable secondary transforms, the two non-separable transforms are used in a concatenated manner.

[0156] This disclosure further describes, below, some example implementations of inseparable transform schemes applied to primary and secondary transforms. These inseparable transform designs described below aim to improve coding efficiency, particularly for directional image modes. Specifically, intra-frame mode-dependent inseparable primary and / or secondary transform schemes are disclosed. These example embodiments focus on the data scan order within blocks being encoded / decoded during a forward or inverse transform process. The block being processed can generally be referred to as a data block; for a forward transform, this data block may include samples of the residual transform / encoded / predicted block or primary transform coefficients, and for an inverse transform, this data block may include secondary transform coefficients to be inversely transformed into primary transform coefficients, or primary transform coefficients to be inversely transformed into residual samples.

[0157] To optimize energy compression achieved by non-separable primary transforms or non-separable secondary transforms and to effectively quantize primary or secondary transform coefficients (to discard high-frequency coefficients), an input and output scanning process for non-separable primary transforms or non-separable secondary transforms can be performed, taking into account the intra-frame prediction mode, block size, and / or primary transform type, as described in further detail below.

[0158] In some example implementations, a transform set can refer to a group of one or more transform kernels that can serve as candidates or options for the encoder to select for data blocks during the encoding process.

[0159] In some implementations, primary transformations can be performed using inseparable transformations or by performing a series of 1-D transformations. For example, in a DCT_DCT combination, DCTs are applied horizontally and vertically on the block. As another example, in an ADST_ADST combination, 1-D ADSTs are applied horizontally and vertically on the block. In some implementations, different transformation types can be used horizontally and vertically. Such transformations can be referred to as hybrid primary transformations.

[0160] Figure 15 Example data stream 1500 for forward and inverse inseparable transformations is shown. Various data scan processes are identified as S1 and S2, and the forward and inverse inseparable transformations are... Figure 15 The data streams 1502 and 1504 are respectively labeled as such. The operations of example data stream 1500 apply to both primary and secondary inseparable transformation processes, and are explained in further detail below.

[0161] Data scan in inseparable secondary transformation

[0162] In this example embodiment, the forward secondary transformation and the inverse secondary transformation can be inseparable transformations, such as... Figure 15As shown in 1502 and 1504. From the encoding end, the input to the forward secondary transform 1502 can be N primary transform coefficients 1506 of the primary coefficient data block 1508. The primary coefficient data block 1508 is scanned into the data sequence 1510 in the scanning process 1512 following the first scanning order S1. The output of the forward secondary transform 1502 can be a secondary transform coefficient sequence 1514 of K data items, which replaces the K input primary transform coefficients 1516 of the primary transform coefficients to generate a modified primary transform coefficient data block 1507 after the data scan 1518 using the second scanning order S2. N and K are positive integers.

[0163] From the decoding end, the input of the inverse secondary transform 1504 is the secondary transform coefficient sequence 1520 of K data items obtained after inverse scanning of the K input coefficients 1517 of the modified primary coefficient data block 1509 in the inverse S2 scan order during the scan process 1522. The output of the inverse secondary transform 1504 can be the primary transform coefficient sequence 1524 of N coefficients, which replaces the N primary transform coefficients 1526 of 1509 to generate the primary coefficient data block 1528 using the inverse scan process 1530 following the inverse S1 scan order.

[0164] In some example implementations, N can be greater than K. In other words, the inseparable secondary transform 1502 can be a reduced inseparable secondary transform. In some example embodiments, the N primary transform coefficients 1506 can be the first N coefficients of the primary coefficient data block 1506, for example, the top left corner (or low-frequency portion) of the primary transform coefficient data block 1506.

[0165] In one example implementation, the scan order S1 may depend on at least one of the following: the intra-prediction mode associated with the input data block 1508, the primary transform type associated with the data block, or the block size of the data block.

[0166] In one example implementation, the scan order S2 may depend on at least one of the following: the intra-prediction mode associated with the input data block 1508, the primary transform type associated with the data block, or the block size of the data block.

[0167] In one example implementation, the S1 and S2 scan order may include at least one of the following: zigzag scan order, diagonal scan order, or row and column scan order.

[0168] In one implementation, when the same set of secondary transforms (with one or more secondary transform kernels) is used as a secondary transform candidate for multiple intra-prediction modes, the S1 and S2 scan order can depend on the intra-prediction mode associated with the data block.

[0169] In some example implementations, the scan order of S1 and S2 can be different.

[0170] In some example implementations, N and K include, but are not limited to, any integer between 0 and 127, including both 0 and 127.

[0171] In some example implementations, when the data block is a square block, the S1 and / or S2 scan order may include a zigzag order. When the block is not a square block, the S1 and / or S2 scan order may include a diagonal scan order.

[0172] Figure 15 The quantization and entropy encoding 1540, as well as the corresponding entropy decoding and dequantization 1550, are further illustrated.

[0173] Data scanning in an inseparable primary transformation as a primary transformation

[0174] In this example embodiment, the inseparable transform processes 1502 and 1504 can be performed for the primary transform. Thus, the input to data stream 1500 (1508) will be, for example, a block of residual sample data instead of the primary transform coefficients, and the output 1507 will be a modified block of residual sample data containing K primary transform coefficients 1516 from the inseparable transform process 1502. Specifically, from the encoder side, the input to the forward primary transform can be the first N residual samples following the first scan order S1, and the output of the forward primary transform is K inseparable transform coefficients that replace the K input residual samples in the second scan order S2. N and K are positive integers. From the decoder side, the input to the inverse inseparable transform is the K inseparable transform coefficients following the scan order S2, and the output of the inverse inseparable transform is N residual samples that replace the N inseparable transform coefficients following the scan order S1, where N is an integer.

[0175] In one example implementation, the scan order S1 may depend on at least one of the following: the intra-prediction mode associated with the data block, the primary transform type associated with the data block, or the block size of the data block.

[0176] In one example implementation, the scan order S2 may depend on at least one of the following: the intra-prediction mode associated with the data block, the primary transform type associated with the data block, or the block size of the data block.

[0177] In one example implementation, the S1 and S2 scan order may include at least one of the following: zigzag scan order, diagonal scan order, or row and column scan order.

[0178] In one example implementation, when the same set of transforms (with one or more secondary transform kernels) is used as a secondary transform candidate for multiple intra-prediction modes, the S1 and S2 scan order can depend on the intra-prediction mode associated with the data block.

[0179] In some example implementations, the scan order of S1 and S2 can be different.

[0180] In one example implementation, N and K include, but are not limited to, any integer between 0 and 127, including 0 and 127.

[0181] In one example implementation, when the data block is a square block, the S1 and / or S2 scan order can include a zigzag scan order. When the block is not a square block, the S1 and / or S2 scan order can include a diagonal scan order. In other words, a diagonal scan order can be implemented for both square and non-square data blocks.

[0182] In one example implementation, the S1 scan order can include either a horizontal (row) scan order or a vertical (column) scan order.

[0183] For these implementations, the quantization / entropy process 1540 and the entropy decoding / dequantization process 1550 can be performed directly on 1507 or 1509 without additional secondary transformations. In some other alternative implementations, additional secondary transformations can be performed before the encoding process 1540 and after the decoding process 1550. These additional secondary transformations can be separable or non-separable, for example, following the implementations described above for non-separable secondary transformations.

[0184] Data scanning in an inseparable primary transformation as a primary transformation

[0185] In some embodiments, when performing transform coefficient encoding (e.g., entropy encoding of transform coefficients), the type of transform (i.e., separable or non-transferable) can influence the scan order. In one implementation, when applying non-separable transforms (to primary and / or secondary transforms), the horizontal (row) or vertical (column) scan order can be used for transform coefficient encoding.

[0186] Figure 16An exemplary method 1600 for processing video data is shown. Method 1600 may include some or all of the following steps: step 1610, extracting data blocks from video data; step 1620, scanning a first number of data items in the data blocks following a first scan order to generate a first data sequence; step 1630, performing an inseparable transformation on the first data sequence to obtain a second data sequence having a second number of data items; and step 1640, replacing at least a portion of the first number of data items in the data blocks with a portion or all of the second data sequence following a second scan order.

[0187] In embodiments of this disclosure, any steps and / or operations may be combined or arranged in any number or order as needed. Two or more steps and / or operations may be performed in parallel.

[0188] The embodiments in this disclosure can be used individually or in any combination in any order. Furthermore, each of the method (or embodiment), encoder, and decoder can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-volatile computer-readable medium. The embodiments in this disclosure can be applied to luma blocks or chroma blocks.

[0189] The techniques described above can be implemented as computer software that uses computer-readable instructions and is physically stored on one or more computer-readable media. For example, Figure 17 A computer system (1700) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0190] Computer software can be coded using any suitable machine code or computer language. Machine code or computer language can be created through assembly, compilation, linking or similar mechanisms to produce code that includes instructions that can be executed directly or through interpretation, microcode execution or other means by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0191] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0192] Figure 17 The components shown for the computer system (1700) are exemplary in nature and are not intended to impose any limitation on the scope or functionality of computer software implementing embodiments of this disclosure. The configuration of the components should also not be construed as having any dependency or requirement on any component or combination thereof illustrated in the exemplary embodiments of the computer system (1700).

[0193] The computer system (1700) may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from one or more human users through, for example, tactile input (such as keystrokes, swipes, or movement with a data glove), audio input (such as speech or tapping), visual input (such as gestures), or olfactory input (not shown). The human-machine interface device may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (such as speech, music, or ambient sound), images (such as scanned images or photographic images obtained from a still image camera), and video (such as two-dimensional video or three-dimensional video including stereoscopic video).

[0194] The input human-machine interface device may include one or more of the following (only one of each is depicted): keyboard (1701), mouse (1702), touchpad (1703), touch screen (1710), data glove (not shown), joystick (1705), microphone (1706), scanner (1707), camera (1708).

[0195] The computer system (1700) may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback from touchscreens (1710), data gloves (not shown), or joysticks (1705), but may also include tactile feedback devices that are not used as input devices), audio output devices (such as speakers (1709), headphones (not depicted)), visual output devices (such as screens (1710), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touchscreen input capability, each with or without tactile feedback capability—some of which are capable of outputting two-dimensional or more than three-dimensional visual output in a manner such as stereoscopic output; virtual reality glasses (not depicted), holographic displays, and smoke canisters (not depicted)), and printers (not depicted).

[0196] The computer system (1700) may also include human-accessible storage devices and their associated media, such as optical media including media (1721) such as CD / DVD ROM / RW (1720) with CD / DVD, thumb drives (1722), removable hard disk drives or solid-state drives (1723), conventional magnetic media such as magnetic tapes and floppy disks (not depicted), devices based on dedicated ROM / ASIC / PLD such as security dongles (not depicted), etc.

[0197] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other volatile signals.

[0198] The computer system (1700) may also include an interface (1754) to one or more communication networks (1755). The network may be, for example, wireless, wired, or optical. The network may also be local, wide area, metropolitan area, vehicular and industrial, real-time, latency-tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicular and industrial networks including CAN buses, etc. Some networks typically require external network interface adapters (such as, for example, a USB port of the computer system (1700)) attached to certain general-purpose data ports or peripheral buses (1749); other networks are typically integrated into the core of the computer system (1700) by attaching to system buses as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1700) can communicate with other entities. Such communication can be unidirectional (receive-only, e.g., broadcasting TV), unidirectional (transmit-only, e.g., CANbus to certain CANbus devices), or bidirectional, e.g., to other computer systems using local area digital networks or wide area digital networks. Certain protocols and protocol stacks can be used on each of those networks and network interfaces described above.

[0199] The aforementioned human-machine interface devices, human-accessible storage devices, and network interfaces can be attached to the core (1740) of the computer system (1700).

[0200] The core (1740) may include one or more central processing units (CPU) (1741), graphics processing units (GPUs) (1742), dedicated programmable processing units in the form of field-programmable gate arrays (FPGAs) (1743), hardware accelerators (1744) for certain tasks, graphics adapters (1750), etc. These devices, along with read-only memory (ROM) (1745), random access memory (1746), and internal mass storage (1747) such as internal non-user-accessible hard disk drives, SSDs, etc., can be connected via a system bus (1748). In some computer systems, the system bus (1748) may be accessed as one or more physical plugs to allow for expansion by adding CPUs, GPUs, etc. Peripheral devices may be attached directly or via a peripheral bus (1749) to the core's system bus (1748). In one example, a screen (1710) may be connected to a graphics adapter (1750). Peripheral bus architectures include PCI, USB, etc.

[0201] The CPU (1741), GPU (1742), FPGA (1743), and accelerator (1744) can execute certain instructions, and combinations of these instructions can constitute the aforementioned computer code. This computer code can be stored in ROM (1745) or RAM (1746). Transient data can also be stored in RAM (1746), while permanent data can be stored, for example, in internal mass storage (1747). Fast storage and retrieval of any memory device can be enabled by using cache memory, which can be closely associated with one or more CPUs (1741), GPUs (1742), mass storage (1747), ROM (1745), RAM (1746), etc.

[0202] Computer-readable media may have computer code thereon for performing various computer-implemented operations. The media and computer code may be those specifically designed and constructed for the purposes of this disclosure, or they may be of types known and available to those skilled in the art of computer software.

[0203] As a non-limiting example, a computer system (1700) having an architecture, and particularly a core (1740), can provide functionality as a result of one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as described above, as well as some memory of the core (1740) having non-volatile properties, such as internal mass storage (1747) or ROM (1745). Software implementing various embodiments of this disclosure can be stored in such devices and executed by the core (1740). Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software can cause the core (1740), and particularly the processors therein (including CPUs, GPUs, FPGAs, etc.), to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM (1746) and modifying such data structures according to software-defined processes. Alternatively or as an alternative, a computer system may provide functionality, as a result of hard-wired logic or otherwise embodied in circuitry (e.g., an accelerator (1744)), which may replace or operate with software to perform the specific process or a specific portion of the specific process described herein. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may include circuitry storing software for execution (such as integrated circuits (ICs)), circuitry embodying logic for execution, or both. This disclosure includes any suitable combination of hardware and software.

[0204] While this application has described several exemplary embodiments, various modifications, arrangements, and equivalent substitutions of the embodiments are all within the scope of this application. Therefore, it should be understood that those skilled in the art can design various systems and methods that, although not explicitly shown or described herein, embody the principles of this application and are thus within its spirit and scope.

[0205] Appendix A: Acronyms

[0206] JEM: Joint Exploration Model

[0207] VVC: Versatile Video Coding

[0208] BMS: Benchmark set

[0209] MV: Motion Vector

[0210] HEVC: High Efficiency Video Coding

[0211] SEI: Supplementary Enhancement Information

[0212] VUI: Video Usability Information

[0213] GOPs: Groups of Pictures

[0214] TUs: Transform Units

[0215] PUs: Prediction Units

[0216] CTUs: Coding Tree Units

[0217] CTBs: Coding Tree Blocks

[0218] PBs: Prediction Blocks

[0219] HRD: Hypothetical Reference Decoder

[0220] SNR: Signal-to-Noise Ratio

[0221] CPUs: Central Processing Units

[0222] GPUs: Graphics Processing Units

[0223] CRT: Cathode Ray Tube

[0224] LCD: Liquid Crystal Display

[0225] OLED: Organic Light-Emitting Diode

[0226] CD: Compact Disc

[0227] DVD: Digital Video Disc

[0228] ROM: Read-Only Memory

[0229] RAM: Random Access Memory

[0230] ASIC: Application-Specific Integrated Circuit

[0231] PLD: Programmable Logic Device

[0232] LAN: Local Area Network

[0233] GSM: Global System for Mobile Communications

[0234] LTE: Long-Term Evolution

[0235] CANBus: Controller Area Network Bus

[0236] USB: Universal Serial Bus

[0237] PCI: Peripheral Component Interconnect

[0238] FPGA: Field Programmable Gate Array

[0239] SSD: Solid-state drive

[0240] IC: Integrated Circuit

[0241] CU: Coding Unit

[0242] PU: Prediction Unit

[0243] TU: Transform Unit

[0244] CTU: Coding Tree Unit

[0245] PDPC: Position Dependent Prediction Combination

[0246] ISP: Intra Sub-Partitions

[0247] SPS: Sequence Parameter Setting

[0248] PPS: Picture Parameter Set

[0249] APS: Adaptive Parameter Set

[0250] VPS: Video Parameter Set

[0251] DPS: Decoding Parameter Set

[0252] ALF: Adaptive Loop Filter

[0253] SAO: Sample Adaptive Offset

[0254] CC-ALF: Cross-Component Adaptive Loop Filter

[0255] CDEF: Constrained Directional Enhancement Filter

[0256] CCSO: Cross-Component Sample Offset

[0257] LSO: Local Sample Offset

[0258] LR: Loop Restoration Filter

[0259] AV1: AOMedia Video 1

[0260] AV2: AOMedia Video 2

Claims

1. A method of video decoding, the method comprising: The method comprises: receiving a video bitstream, the video bitstream comprising a plurality of video blocks, the plurality of video blocks comprising a first block; scanning a first number of transform coefficients in the first block in a first scan order to generate a first data sequence, the plurality of transform coefficients being secondary transform coefficients or primary transform coefficients; obtaining a signaled first index, the first index being used to indicate a transform matrix of an inverse non-separable transform, the first index depending on a position of a last significant coefficient of the plurality of transform coefficients of the first block, and context encoding, wherein the inverse non-separable transform is not a low frequency non-separable transform when the plurality of video blocks apply an intra sub-partition mode or a multi-linear regression intra prediction mode; performing the inverse non-separable transform on the first data sequence according to the first index to obtain a second data sequence, wherein a number of transform coefficients of the second data sequence is different from the first data sequence, wherein the inverse non-separable transform is an inverse secondary transform and the second data sequence is primary transform coefficients when the plurality of transform coefficients are secondary transform coefficients, and the inverse non-separable transform is an inverse primary transform and the second data sequence is residual samples when the plurality of transform coefficients are primary transform coefficients; and replacing at least a portion of the first data sequence with a portion or all of the second data sequence following a second scan order; wherein the second scan order is different from the first scan order; the first scan order or the second scan order is determined based on an intra prediction mode associated with the first block when a plurality of intra prediction modes with respect to the first block share a same set of transform matrices.

2. The method of claim 1, wherein: the inverse non-separable transform is a non-separable secondary transform.

3. The method according to any one of claims 1-2, characterized in that, the first scan order or the second scan order comprises one of: a zigzag scan order; a diagonal scan order; or a row and column scan order.

4. The method of any one of claims 1-2, wherein: the first scan order and the second scan order are respectively determined based on at least one of: an intra prediction mode associated with the first block; a size of the first block; a type of the inverse non-separable transform.

5. The method according to any of claims 1-2, characterized by, a number of transform coefficients of the first data sequence and the second data sequence is an integer between 0 and 127, inclusive.

6. The method of any one of claims 1-2, wherein, the first scan order and the second scan order are: a zigzag scan order when the first block is a square block; or a diagonal scan order when the first block is a non-square block.

7. The method of claim 2, wherein, the first scan order is a horizontal or vertical scan order.

8. A method of video coding, the method comprising: The method comprises: receiving video data, the video data comprising a plurality of video blocks, the plurality of video blocks comprising a first block; obtaining a second data sequence corresponding to data of the first block according to a second scan order, the data of the first block being residual samples or primary transform coefficients; determining a first index indicating a transform matrix of a non-separable transform, the first index depending on a position of a last significant coefficient of a plurality of transform coefficients of the first block, and context encoding, wherein the non-separable transform is not a low-frequency non-separable transform when the plurality of video blocks apply an intra sub-partition mode or a multi-linear regression intra prediction mode; obtaining a first data sequence of the first block by applying a forward non-separable transform on the second data sequence according to the first index, the second data sequence having a size different from the first data sequence, wherein the forward non-separable transform is a primary transform and the first data sequence is primary transform coefficients when the data of the first block is residual samples, and the forward non-separable transform is a secondary transform and the first data sequence is secondary transform coefficients when the data of the first block is primary transform coefficients; replacing at least a portion of the second data sequence with a portion or all of the first data sequence following a first scan order, wherein the first scan order is different from the second scan order, and the first scan order or the second scan order is determined based on an intra prediction mode associated with the first block when a plurality of intra prediction modes with respect to the first block share a same set of transform matrices.

9. An electronic device, comprising: comprising a memory configured to store instructions; a processor configured to execute the instructions to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, a computer program product having computer code stored thereon, the computer code, when executed by one or more processors, causes the one or more processors to implement the method according to any one of claims 1 to 8.

11. A method of transmitting or storing a video bitstream, characterized in that, The method of claim 8 for video coding generates the video bitstream, and transmitting or storing the video bitstream.

Images