Method, device and storage medium for processing video information

Abstract

The present disclosure relates to transform core sharing in video encoding and decoding. For example, a method for such transform core sharing is disclosed. The method may include: identifying a plurality of transform cores, wherein each of the plurality of transform cores includes a set of basis vectors from low frequency to high frequency; N high-frequency basis vectors of two or more of the plurality of transform cores are shared, N being a positive integer; and low-frequency basis vectors of two or more of the plurality of transform cores that are different from the N high-frequency basis vectors are personalized. The method may further include extracting a data block from a video bitstream; selecting a transform core from the plurality of transform cores based on information associated with the data block; and applying the transform core to at least a portion of the data block to generate a transform block.

Description

[0001] This application is based on and claims priority to U.S. non-provisional patent application No. 17 / 568,871 filed on January 5, 2022, which claims priority to U.S. provisional application No. 63 / 172,060, entitled “ORTHOGONAL TRANSFORM GENERATION WITH SUBSPACE CONSTRAINT”, filed on April 7, 2021. The entire contents of these two applications are incorporated herein by reference. Technical Field

[0002] The present disclosure describes a set of advanced video coding techniques. More specifically, the disclosed techniques relate to transform kernel sharing methods in video encoding and decoding. Background Art

[0003] The background description provided herein is intended to present the background of the present application as a whole. The extent to which the work of the presently named inventors described in the background section and various aspects of this specification is performed does not indicate that it is prior art at the time of filing this application, and it is never explicitly or implicitly admitted that it is prior art for this application.

[0004] Video encoding and decoding can be performed by inter-frame picture prediction technology with motion compensation. Uncompressed digital video may include a series of pictures, each picture having a spatial dimension of, for example, 1920×1080 luminance samples and associated full or sub-sampled chrominance samples. The series of pictures has a fixed or variable picture rate (or frame rate), such as 60 pictures per second or 60 frames per second. Uncompressed video has specific bit rate requirements for streaming or data processing. For example, a video with a pixel resolution of 1920×1080, a frame rate of 60 frames / second, and a chrominance subsampling of 4:2:0 (8 bits / pixel / color channel) requires a bandwidth of nearly 1.5 Gbit / s. One hour of such video requires more than 600 GB of storage space.

[0005] One of the goals of video encoding and decoding is to reduce the redundant information of the input video signal through compression. Video compression can help reduce the requirements for the above-mentioned bandwidth and / or storage space, in some cases by two or more orders of magnitude. Lossless and lossy compression, as well as a combination of the two, can be used. Lossless compression refers to the technology of reconstructing an exact copy of the original signal from the compressed original signal through the decoding process. Lossy compression refers to the encoding / decoding process in which the original video information is not completely preserved during encoding and cannot be completely restored during decoding. When lossy compression is used, the reconstructed signal may be different from the original signal, but the distortion between the original signal and the reconstructed signal is small enough to make the reconstructed signal usable for the intended application, despite some information loss. In the case of video, lossy compression is widely used in many applications. The amount of tolerable distortion depends on the application. For example, users of certain consumer video streaming applications can tolerate higher distortion than users of movie or television broadcast applications. The compression ratio achievable by a particular encoding algorithm can be selected or adjusted to reflect various distortion tolerances: higher tolerable distortion generally allows encoding algorithms that produce higher losses and higher compression ratios.

[0006] Video encoders and decoders may utilize techniques from several broad categories and steps, including, for example, motion compensation, Fourier transforms, quantization, and entropy coding and decoding.

[0007] Video codec techniques may include techniques known as intra-frame coding. In intra-frame coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, pictures are spatially subdivided into blocks of samples. When all sample blocks are encoded in intra-frame mode, the picture may be referred to as an intra-frame picture. Intra-frame pictures and their derivatives (such as independent decoder refresh pictures) can be used to reset the decoder state, and can therefore be used as the first picture in an encoded video stream and video session, or as a still image. Then, the samples of the block after intra-frame prediction can be transformed to the frequency domain, and the transform coefficients so generated can be quantized before entropy coding. Intra-frame prediction represents a technique for minimizing sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after transformation and the smaller the AC coefficient, the fewer bits are required to represent the block after entropy coding at a given quantization step size.

[0008] Conventional intra-frame codecs, such as those known from, for example, the MPEG-2 generation of codecs, do not use intra-frame prediction. However, some newer video compression techniques include techniques that attempt to use intra-frame prediction from, for example, surrounding sample data and / or metadata that is obtained during encoding / decoding of spatially adjacent data blocks and precedes the data blocks in decoding order. Such techniques are hereinafter referred to as "intra-frame prediction" techniques. Note that, in at least some cases, intra-frame prediction uses only reference data from the current picture being reconstructed, and not reference data from reference pictures.

[0009] There can be many different forms of intra-frame prediction. When more than one such technique is available in a given video coding technique, the techniques used may be referred to as intra-frame prediction modes. One or more intra-frame prediction modes may be provided in a particular codec. In some cases, a mode may have sub-modes and / or may be associated with various parameters, and the mode / sub-mode information and intra-frame coding parameters for a video block may be encoded separately or included together in a mode codeword. The codeword to be used for a given mode / sub-mode / parameter combination may have an impact on the coding efficiency gain through intra-frame prediction, and so does the entropy coding and decoding technique that converts the codeword into a bitstream.

[0010] Some mode of intra prediction was introduced with H.264, improved in H.265, and further improved in newer codecs such as joint exploration model (JEM), versatile video coding (VVC), and benchmark set (BMS). Typically, for intra prediction, the predictor block can be formed using the values ​​of neighboring samples that have become available. For example, the available values ​​of a particular set of neighboring samples along a particular direction and / or line can be copied into the predictor block. A reference to the direction of use can be encoded in the bitstream or can be predicted itself.

[0011] Reference Figure 1A , the lower right corner depicts a subset of nine known prediction directions from the 33 possible intra-frame prediction directions specified by H.265 (corresponding to the 33 angular modes of the 35 intra-frame modes). The point (101) where the arrows converge represents the sample being predicted. The arrows represent the direction in which the sample at 101 is predicted using neighboring samples. For example, arrow (102) indicates that sample (101) is predicted based on one or more neighboring samples at a 45 degree angle to the horizontal direction at the upper right. Similarly, arrow (103) indicates that sample (101) is predicted based on one or more samples at a 22.5 degree angle to the horizontal direction at the lower left.

[0012] Still reference Figure 1A, a square block (104) including 4×4 samples is shown in the upper left (indicated by the thick dashed line). The square block (104) includes 16 samples, each of which is labeled with "S" and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (starting from the top) and the first sample in the X dimension (starting from the left). Similarly, sample S44 is the fourth sample in the block (104) in both the X dimension and the Y dimension. Since the block is a 4×4 size sample, S44 is located in the lower right corner. An exemplary reference sample following a similar numbering scheme is also shown. The reference sample is labeled with "R" and its Y position (e.g., row index) and X position (e.g., column index) relative to the block (104). In H.264 and H.265, the prediction sample is adjacent to the block being reconstructed.

[0013] The intra picture prediction of block 104 may start by copying reference sample values ​​from neighboring samples according to a signaled prediction direction. For example, assuming that the encoded video bitstream includes signaling indicating, for this block 104, the arrow (102) prediction direction, i.e., predicting the sample according to one or more prediction samples to the upper right at a 45 degree angle to the horizontal direction. In this case, samples S41, S32, S23, and S14 are predicted according to the same reference sample R05. Sample S44 is predicted according to reference sample R08.

[0014] In some cases, for example by interpolation, the values ​​of multiple reference samples may be combined in order to calculate the reference sample, particularly when the direction is not divisible by 45 degrees.

[0015] As video coding techniques continue to advance, the number of possible directions is also increasing. For example, in H.264 (2003), nine different directions were available for intra prediction. This was increased to 33 in H.265 (2013), and JEM / VVC / BMS can support up to 65 directions at the time of this application. Experimental studies have been conducted to help identify the most suitable intra prediction directions, and certain techniques in entropy coding can be used to encode those most suitable directions with a small number of bits, accepting some bit cost for the direction. Further, the direction itself can sometimes be predicted from neighboring directions used in intra prediction of already decoded neighboring blocks.

[0016] Figure 1B A schematic diagram (180) depicting 65 intra prediction directions according to the JEM is shown to illustrate the increasing number of prediction directions in various encoding techniques as they evolve over time.

[0017] The manner in which the bits representing the intra-frame prediction direction are mapped to the prediction direction in the encoded video bitstream may vary from video coding technique to video coding technique; and may range, for example, from a simple direct mapping of the prediction direction to the intra-frame prediction mode to a codeword, to complex adaptive schemes involving the most probable mode, and similar techniques. However, in all cases, there may be certain directions for intra-frame prediction that are statistically less likely to appear in the video content than certain other directions. Since the goal of video compression is to reduce redundancy, in a well-designed video coding technique, those unlikely directions may be represented by a larger number of bits than more likely directions.

[0018] Inter prediction or inter prediction can be based on motion compensation. In motion compensation, a block of sample data from a previously reconstructed picture or part of a reconstructed picture (reference picture) is spatially shifted in the direction indicated by a motion vector (hereinafter referred to as MV) and used for prediction of a newly reconstructed picture or part of a picture (e.g., a block). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions, X and Y, or three dimensions, where the third dimension represents the reference picture in use (similar to the time dimension).

[0019] In some video compression techniques, the current MV applicable to a region of sample data can be predicted from other MVs, such as those related to other regions of sample data that are spatially adjacent to the region being reconstructed and precede the current MV in decoding order. This can significantly reduce the total amount of data required to encode the MV by relying on removing redundancy in related MVs, thereby improving compression efficiency. MV prediction can work effectively, for example, because when encoding and decoding an input video signal derived from a camera (called natural video), there is a statistical probability that regions with an area larger than the area applicable to a single MV will move in a similar direction in the video sequence, and therefore, in some cases, can be predicted using similar motion vectors derived from MVs of neighboring regions. This results in the actual MV of a given region being similar or identical to the MV predicted from the surrounding MVs. After entropy coding, such an MV can again be represented with fewer bits than if the MV were directly encoded rather than predicted from one or more neighboring MVs. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from the original signal (i.e., sample stream). In other cases, the MV prediction itself may be lossy, for example due to round-off errors when computing the predicted value from several surrounding MVs.

[0020] H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016) describes various MV prediction mechanisms. Among the various MV prediction mechanisms provided by H.265, this article describes a technique hereinafter referred to as "spatial merging".

[0021] refer to Figure 2 , the current block (201) includes samples found by the encoder during motion search, which can be predicted based on the previous block that has been spatially shifted by the same amount. The MV is not encoded directly, but is derived from metadata associated with one or more reference pictures, such as the most recent (in decoding order) reference picture, by using the MV associated with any of the five surrounding samples. The five surrounding samples are represented by A0, A1 and B0, B1, B2 (from 202 to 206), respectively. In H.265, MV prediction can use the prediction value of the same reference picture used by neighboring blocks. Summary of the invention

[0022] Aspects of the present disclosure provide methods and apparatus for transform core sharing in video encoding and decoding.

[0023] In some implementations, a method for such transform kernel sharing is disclosed. The method may include: identifying a plurality of transform kernels, wherein each transform kernel in the plurality of transform kernels includes a set of basis vectors from low frequency to high frequency; N high frequency basis vectors of two or more transform kernels in the plurality of transform kernels are shared, N is a positive integer; and low frequency basis vectors different from the N high frequency basis vectors in two or more transform kernels in the plurality of transform kernels are personalized. The method may further include extracting a data block from a video code stream; selecting a transform kernel from the plurality of transform kernels based on information associated with the data block; and applying the transform kernel to at least a portion of the data block to generate a transform block.

[0024] In the above implementation, multiple transform kernels may be pre-trained offline. In some further implementations, multiple transform kernels may be jointly trained offline to determine the shared N high-frequency basis vectors.

[0025] In any of the above implementations, the N high frequency basis vectors may include basis vectors having frequencies above a predetermined threshold frequency. In some further implementations, a plurality of transform kernels and a predetermined threshold frequency may be pre-trained offline.

[0026] In any of the above implementations, the plurality of transform kernels may include a secondary transform kernel adapted to transform primary transform coefficients, and the data block includes an array of primary transform coefficients.

[0027] In any of the above implementations, two or more transform cores among the multiple transform cores that share N high-frequency basis vectors correspond to the same one of the multiple intra-picture prediction modes; and all transform cores among the multiple transform cores share N high-frequency basis vectors and other personalized low-frequency basis vectors, and the multiple transform cores are assigned to the same one of the multiple intra-picture prediction modes.

[0028] In any of the above implementations, two or more transform cores among the plurality of transform cores sharing N high frequency basis vectors may be assigned to two or more different intra picture prediction modes.

[0029] In any of the above implementations, the plurality of transform cores are secondary transform cores; and two or more of the plurality of transform cores sharing N high-frequency basis vectors are configured to transform primary transform coefficients generated from a primary transform having the same transform type.

[0030] In any of the above implementations, N is an integer power of 2, and / or N is less than a predefined upper limit, and / or N depends on a transform size.

[0031] In any of the above implementations, the plurality of transform cores may be configured for intra secondary transform (IST). In some implementations, the plurality of transform cores may be secondary low frequency non-separable transform (LFNST) cores. In some implementations, the plurality of transform cores are configured for linear graph transform (LGT).

[0032] In some other implementations, a device for processing video information is disclosed. The device may include circuitry configured to perform any of the above method implementations.

[0033] Aspects of the present disclosure also provide non-transitory computer-readable media storing instructions that, when executed by a computer, cause the computer to perform methods for video decoding and / or encoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Further features, properties and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0035] Figure 1A A schematic diagram showing an exemplary subset of intra-prediction directional modes.

[0036] Figure 1B A diagram of exemplary intra prediction directions is shown.

[0037] Figure 2 A schematic diagram showing spatial merging candidates of a current block and its surroundings for motion vector prediction in one example is shown.

[0038] Figure 3 A schematic diagram showing a simplified block diagram of a communication system (300) according to an example embodiment is shown.

[0039] Figure 4 A schematic diagram showing a simplified block diagram of a communication system (400) according to an example embodiment is shown.

[0040] Figure 5 A schematic diagram showing a simplified block diagram of a video decoder according to an example embodiment is shown.

[0041] Figure 6 A schematic diagram showing a simplified block diagram of a video encoder according to an example embodiment is shown.

[0042] Figure 7 A block diagram of a video encoder according to another example embodiment is shown.

[0043] Figure 8 A block diagram of a video decoder according to another example embodiment is shown.

[0044] Fig. 9 A directional intra prediction mode according to an example embodiment of the present disclosure is shown.

[0045] Fig.10 A non-directional intra prediction mode according to an example embodiment of the present disclosure is shown.

[0046] Fig.11 A recursive intra prediction mode according to an example embodiment of the present disclosure is shown.

[0047] Fig.12 Transform block partitioning and scanning of an intra prediction block according to an example embodiment of the present disclosure is shown.

[0048] Fig.13 Transform block partitioning and scanning of an inter-prediction block according to an example embodiment of the present disclosure is shown.

[0049] Fig.14 A low frequency inseparable transformation process according to an example embodiment of the present disclosure is shown.

[0050] Fig.15 The sharing of high frequency basis vectors between different transform kernels according to an example embodiment of the present disclosure is shown.

[0051] Fig.16 A flow chart according to an example embodiment of the present disclosure is shown.

[0052] Fig.17 A schematic diagram of a computer system according to an example embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0053] Figure 3 3 is a simplified block diagram of a communication system (300) according to an embodiment disclosed in the present application. The communication system (300) includes a plurality of terminal devices, which can communicate with each other through, for example, a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and a terminal device (320) interconnected through the network (350). Figure 3 In an embodiment, the terminal device (310) and the terminal device (320) perform unidirectional data transmission. For example, the terminal device (310) may encode video data (e.g., a video picture stream collected by the terminal device (310)) for transmission to another terminal device (320) via the network (350). The encoded video data is transmitted in the form of one or more encoded video code streams. The terminal device (320) may receive the encoded video data from the network (350), decode the encoded video data to restore the video data, and display the video picture based on the restored video data. The unidirectional data transmission can be implemented in applications such as media services.

[0054] In another embodiment, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of encoded video data, which can be implemented, for example, during a video conference. For bidirectional data transmission, each of the terminal devices (330) and the terminal device (340) can encode video data (e.g., a video picture stream collected by the terminal device) for transmission to the other terminal device (330) and the terminal device (340) through the network (350). Each of the terminal devices (330) and the terminal device (340) can also receive the encoded video data transmitted by the other terminal device (330) and the terminal device (340), and can decode the encoded video data to restore the video data, and can display the video picture on an accessible display device based on the restored video data.

[0055] exist Figure 3In the embodiment of the present invention, the terminal device (310), the terminal device (320), the terminal device (330) and the terminal device (340) can be implemented as a server, a personal computer and a smart phone, but the applicability of the basic principles disclosed in the present application may not be limited to this. The embodiments disclosed in the present application can be implemented on a laptop computer, a tablet computer, a media player and / or a dedicated video conferencing device. The network (350) represents any number or type of network that transmits encoded video data between the terminal device (310), the terminal device (320), the terminal device (330) and the terminal device (340), including, for example, a wired (wired) and / or wireless communication network. The communication network (350) can exchange data in a circuit switching, a packet switching channel and / or other types of channels. The network may include a telecommunications network, a local area network, a wide area network and / or the Internet. For the purpose of this discussion, unless explicitly explained below, the architecture and topology of the network (350) may be irrelevant to the operation disclosed in the present application.

[0056] As an example, Figure 4 The video encoder and the video decoder are shown in a video streaming environment. The subject matter disclosed in the present application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV broadcasting, gaming, virtual reality, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0057] The video streaming system may include a video acquisition subsystem (413), which may include a video source (401) such as a digital camera, which creates an uncompressed video picture or image stream (402). In an embodiment, the video picture stream (402) includes samples recorded by the digital camera of the video source. Compared to the encoded video data (404) (or the encoded video bitstream), the video picture stream (402) is depicted as a thick line to emphasize the high data volume of the video picture stream, and the video picture stream (402) can be processed by an electronic device (420), which includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination of hardware and software to implement or implement various aspects of the disclosed subject matter as described in more detail below. Compared to the uncompressed video picture stream (402), the encoded video data (404) (or the encoded video bitstream (404)) is depicted as a thin line to emphasize the lower amount of data of the encoded video data (404) (or the encoded video bitstream (404)), which can be stored on the streaming server (405) for future use or directly connected downstream to a video device (not shown). One or more streaming client subsystems, such as Figure 4The client subsystem (406) and the client subsystem (408) in the video transmission system can access the streaming server (405) to retrieve the copy (407) and the copy (409) of the encoded video data (404). The client subsystem (406) may include, for example, a video decoder (410) in the electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and generates an uncompressed output video picture stream (411) that can be presented on a display (412) (e.g., a display screen) or another presentation device (not depicted). The video decoder 410 can be configured to perform some or all of the various functions described in the present disclosure. In some streaming transmission systems, the encoded video data (404), the video data (407), and the video data (409) (e.g., a video bitstream) can be encoded according to certain video encoding / compression standards. Examples of these standards include ITU-T H.265. In an embodiment, the video coding standard under development is informally referred to as Versatile Video Coding (VVC), and the present application may be used in the context of the VVC standard or other video coding standards.

[0058] It should be noted that the electronic device (420) and the electronic device (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0059] Figure 5 is a block diagram of a video decoder (510) according to an embodiment disclosed in the present application. The video decoder (510) may be provided in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used to replace Figure 4 A video decoder (410) of an embodiment.

[0060] The receiver (531) can receive one or more encoded video sequences to be decoded by the video decoder (510). In the same or another embodiment, one encoded video sequence can be decoded at a time, wherein the decoding of each encoded video sequence is independent of other encoded video sequences. Each video sequence can be associated with multiple video frames or images. The encoded video sequence can be received from a channel (501), which can be a hardware / software link to a storage device storing the encoded video data or a streaming source transmitting the encoded video data. The receiver (531) can receive the encoded video data and other data (such as encoded audio data and / or auxiliary data streams), which can be forwarded to their respective processing circuits (not depicted). The receiver (531) can separate the encoded video sequence from the other data. To combat network jitter, a buffer memory (515) can be provided between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) can be implemented as part of the video decoder (510). In other applications, the buffer memory may be external to and separate from the video decoder (510) (not depicted). In still other applications, there may also be a buffer memory (not depicted) external to the video decoder (510), for purposes such as combating network jitter, and there may be another additional buffer memory (515) internal to the video decoder (510), for example, to handle playback timing. When the receiver (531) receives data from a store / forward device with sufficient bandwidth and controllability, or from an isochronous network, the buffer memory (515) may not be required, or the buffer memory (515) may be very small. For use on an optimal packet network such as the Internet, a buffer memory (515) of sufficient size may be required, and its size may be relatively large. Such a buffer memory may be implemented with an adaptive size, and may be implemented at least in part in an operating system or similar element (not depicted) external to the video decoder (510).

[0061] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. The types of symbols include information used to manage the operation of the video decoder (510) and potential information used to control a display device such as a display (512) (e.g., a display screen) that is not part of the electronic device (530) but can be coupled to the electronic device (530), such as Figure 5As shown in . The control information for the display device may be a parameter set fragment (not indicated) of Supplemental Enhancement Information (SEI message) or Video Usability Information (VUI). The parser (520) may parse / entropy decode the received coded video sequence. Entropy coding of the coded video sequence may be performed according to a video coding technique or standard, and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and the like. The parser (520) may extract a subgroup parameter set of at least one subgroup of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the subgroup. The subgroup may include a Group of Pictures (GOP), a picture, a tile, a slice, a macroblock, a Coding Unit (CU), a block, a Transform Unit (TU), a Prediction Unit (PU), and the like. The parser (520) may also extract information from the encoded video sequence, such as transform coefficients (Fourier transform coefficients), quantizer parameter values, motion vectors, and so on.

[0062] The parser (520) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515), thereby creating symbols (521).

[0063] Depending on the type of coded video picture or part of coded video picture (e.g., inter-picture and intra-picture, inter-block and intra-block) and other factors, the reconstruction of symbol (521) may involve multiple different processing or functional units. The units involved and the way they are involved may be controlled by subgroup control information parsed by parser (520) from the coded video sequence. For the sake of brevity, such subgroup control information flow between parser (520) and the multiple processing or functional units described below is not described.

[0064] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into a plurality of functional units as described below. In a practical embodiment operating under commercial constraints, many of these functional units interact closely with each other and may be at least partially integrated with each other. However, in order to clearly describe the various functions of the disclosed subject matter, the conceptual subdivision into functional units is adopted in the following disclosure.

[0065] The first unit may include a scaler / inverse transform unit (551). The scaler / inverse transform unit (551) may receive quantized transform coefficients as symbols (521) and control information from the parser (520), the control information including information indicating which type of inverse transform to use, block size, quantization factors / parameters, quantization scaling matrices, etc. The scaler / inverse transform unit (551) may output a block including sample values, which may be input into an aggregator (555).

[0066] In some cases, the output samples of the scaler / inverse transform unit (551) may belong to an intra-coded block; that is, a block that does not use predictive information from a previously reconstructed picture, but may use predictive information from a previously reconstructed portion of the current picture. Such predictive information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) may use surrounding block information that has been reconstructed and stored in a current picture buffer (558) to generate a block of the same size and shape as the block being reconstructed. For example, the current picture buffer (558) buffers a partially reconstructed current picture and / or a fully reconstructed current picture. In some cases, the aggregator (555) adds the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a per-sample basis.

[0067] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to inter-frame coded and potentially motion compensated blocks. In this case, the motion compensated prediction unit (553) may access the reference picture memory (557) to extract samples for inter-frame picture prediction. After the extracted samples are motion compensated according to the symbols (521), these samples may be added to the output of the scaler / inverse transform unit (551) by the aggregator (555) (the output of the unit 551 may be referred to as residual samples or residual signals), thereby generating output sample information. The acquisition of prediction samples by the motion compensated prediction unit (553) from the address in the reference picture memory (557) may be controlled by a motion vector, and the motion vector is provided to the motion compensated prediction unit (553) in the form of the symbols (521), for example, including X, Y components (shift) and reference picture components (time). Motion compensation may also include interpolation of sample values ​​extracted from the reference picture memory (557) when using sub-sampled accurate motion vectors, and may also be associated with motion vector prediction mechanisms, etc.

[0068] The output samples of the aggregator (555) may be employed by various loop filtering techniques in a loop filter unit (556). The video compression techniques may include in-loop filter techniques that are controlled by parameters included in an encoded video sequence (also referred to as an encoded video bitstream) and that are available to the loop filter unit (556) as symbols (521) from the parser (520). However, in other embodiments, the video compression techniques may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of an encoded picture or encoded video sequence, as well as to previously reconstructed and loop filtered sample values. As will be described in further detail below, several types of loop filters may be included in various orders as part of the loop filter unit 556.

[0069] The output of the loop filter unit (556) may be a sample stream that may be output to a display device (512) and stored in a reference picture memory (557) for subsequent inter-picture prediction.

[0070] Once fully reconstructed, certain coded pictures may be used as reference pictures for future inter-picture prediction. For example, once the coded picture corresponding to the current picture is fully reconstructed, and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) may become part of the reference picture memory (557), and a new current picture buffer may be reallocated before starting reconstruction of a subsequent coded picture.

[0071] The video decoder (510) may perform decoding operations according to a predetermined video compression technique employed, for example, in the ITU-T H.265 standard. The encoded video sequence may conform to the syntax specified by the video compression technique or standard used in the sense that the encoded video sequence follows the syntax of the video compression technique or standard and the profile recorded in the video compression technique or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technique or standard as the only tools available for use under the profile. In order to conform to the standard, the complexity of the encoded video sequence is also required to be within the range defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum picture size, the maximum frame rate, the maximum reconstruction sampling rate (measured in, for example, mega samples per second), the maximum reference picture size, etc. In some cases, the limits set by the hierarchy may be further limited by the Hypothetical Reference Decoder (HRD) specification and metadata of the HRD buffer management signaled in the encoded video sequence.

[0072] In some embodiments, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be part of the encoded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0073] Figure 6 6 is a block diagram of a video encoder (603) according to an exemplary embodiment disclosed in the present application. The video encoder (603) is provided in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuit). The video encoder (603) can be used to replace Figure 4 A video encoder (403) in an embodiment.

[0074] The video encoder (603) can be used to obtain the video source (601) (not Figure 6 In another embodiment, the video source (601) may be implemented as a part of the electronic device (620) to receive video samples, and the video source may capture video images to be encoded by the video encoder (603). In another embodiment, the video source (601) may be implemented as a part of the electronic device (620).

[0075] The video source (601) may provide a source video sequence in the form of a digital video sample stream to be encoded by the video encoder (603), wherein the digital video sample stream may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, XYZ, ...), and any suitable sampling structure (e.g., YCrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the video source (601) may be a storage device storing previously prepared videos. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of separate pictures or images that are given motion when viewed sequentially. The picture itself may be constructed as a spatial pixel array, wherein each pixel may include one or more samples depending on the sampling structure, color space, etc. used. The relationship between pixels and samples may be easily understood by those skilled in the art. The following description focuses on samples.

[0076] According to some embodiments, the video encoder (603) may encode and compress pictures of a source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing an appropriate encoding speed may constitute a function of the controller (650). In some embodiments, as described below, the controller (650) is functionally coupled to and controls other functional units. For the sake of simplicity, the coupling is not shown in the figure. The parameters set by the controller (650) may include rate control related parameters (picture skipping, quantizer, lambda value of rate distortion optimization technology, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) may be used to have other suitable functions that are related to the video encoder (603) optimized for a certain system design.

[0077] In some embodiments, the video encoder (603) operates in a coding loop. As a simple description, in embodiments, the coding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be encoded and a reference picture) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data, even if the embedded decoder 633 processes the encoded video stream through the source encoder 630 without entropy coding (because in the video compression technology considered in this application, any compression between the symbols in the entropy coding and the encoded video code stream is lossless). The reconstructed sample stream (sample data) is input to the reference picture memory (634). Since the decoding of the symbol stream produces a bit-accurate result that is independent of the decoder location (local or remote), the contents of the reference picture memory (634) are also bit-accurate between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction part of the encoder are exactly the same as the sample values ​​that the decoder will "see" when using the prediction during decoding. This basic principle of reference picture synchronization (and the drift that occurs when synchronization cannot be maintained, eg due to channel errors) is used to improve the coding quality.

[0078] The operation of the "local" decoder (633) can be combined with the above Figure 4 The "remote" decoder described in detail for the video decoder (510) is identical. However, additional brief reference is made to Figure 5 , when symbols are available and the entropy encoder (645) and parser (520) are capable of losslessly encoding / decoding the symbols into an encoded video sequence, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and the parser (520), may not be fully implemented in the local decoder (633) in the encoder.

[0079] At this point, it can be observed that any decoder technology other than the parsing / entropy decoding present in the decoder must also be present in the corresponding encoder in substantially the same functional form. For this reason, the present application focuses on the decoder operation related to the decoding part of the encoder. This can simplify the description of the encoder technology because the encoder technology is mutually inverse to the decoder technology described comprehensively. Only a more detailed description of the encoder is needed in some areas or aspects is provided below.

[0080] During operation, in some embodiments, the source encoder (630) may perform motion compensated predictive coding. The motion compensated predictive coding predictively encodes an input picture with reference to one or more previously encoded pictures from a video sequence designated as "reference pictures". In this manner, the encoding engine (632) encodes the differences (or residuals) in the color channels between pixel blocks of the input picture and pixel blocks of a reference picture that may be selected as a prediction reference for the input picture. The term "residual" and its adjective form "residual" may be used interchangeably.

[0081] The local video decoder (633) may decode the encoded video data of the picture that may be designated as the reference picture based on the symbol created by the source encoder (630). The operation of the encoding engine (632) may be a lossy process. When the encoded video data is available at the video decoder ( Figure 6 When the video encoder (603) is decoded at a local (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process that may be performed by the video decoder on the reference picture and may cause the reconstructed reference picture to be stored in the reference picture cache (634). In this way, the video encoder (603) may store a copy of the reconstructed reference picture locally that has common content (absent transmission errors) with the reconstructed reference picture to be obtained by the far-end (remote) video decoder.

[0082] The predictor (635) may perform a prediction search for the encoding engine (632). That is, for a new picture to be encoded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, etc., that may serve as appropriate prediction references for the new picture. The predictor (635) may operate pixel-by-pixel based on sample blocks to find a suitable prediction reference. In some cases, based on the search results obtained by the predictor (635), it may be determined that the input picture may have prediction references taken from a plurality of reference pictures stored in the reference picture memory (634).

[0083] The controller (650) may manage encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding and / or decoding video data.

[0084] The outputs of all the above functional units may be entropy encoded in an entropy encoder (645). The entropy encoder (645) performs lossless compression on the symbols generated by the various functional units according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc., thereby converting the symbols into a coded video sequence.

[0085] The transmitter (640) may buffer the encoded video sequence created by the entropy encoder (645) in preparation for transmission over a communication channel (660), which may be a hardware / software link to a storage device where the encoded video data will be stored. The transmitter (640) may combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or ancillary data streams (source not shown).

[0086] The controller (650) may manage the operation of the video encoder (603). During encoding, the controller (650) may assign a certain coded picture type to each coded picture, but this may affect the coding techniques that can be applied to the corresponding picture. For example, a picture may generally be assigned to any of the following picture types:

[0087] An intra picture (I picture) may be a picture that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of the variations of I pictures and their corresponding applications and features.

[0088] A predictive picture (P picture) may be a picture that can be encoded and decoded using intra prediction or inter prediction, which uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0089] Bidirectional predictive pictures (B pictures), which can be pictures that can be encoded and decoded using intra prediction or inter prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata for reconstruction of a single block.

[0090] The source picture may typically be spatially subdivided into blocks of samples (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples) and encoded block by block. These blocks may be predictively encoded with reference to other (already encoded) blocks, which are determined according to the coding allocation applied to the corresponding picture of the block. For example, blocks of an I picture may be non-predictively encoded, or the blocks may be predictively encoded (spatial prediction or intra-frame prediction) with reference to already encoded blocks of the same picture. Blocks of pixels of a P picture may be predictively encoded by spatial prediction with reference to one previously encoded reference picture or by temporal prediction. Blocks of a B picture may be predictively encoded by spatial prediction with reference to one or two previously encoded reference pictures or by temporal prediction. For other purposes, source pictures or intermediately processed pictures may be subdivided into other types of blocks. The division of coding blocks and other types of blocks may or may not follow the same approach, as described in further detail below.

[0091] The video encoder (603) may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (603) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in an input video sequence. Thus, the encoded video data may accordingly conform to the syntax specified by the video encoding technique or standard used.

[0092] In some embodiments, the transmitter (640) may transmit additional data when transmitting the encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include other forms of redundant data such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0093] The captured video may be taken as a plurality of source pictures (video pictures) in a temporal sequence. Intra-picture prediction (often simplified to intra-prediction) exploits spatial correlations in a given picture, while inter-picture prediction exploits (temporal or other) correlations between pictures. In an embodiment, a particular picture being encoded / decoded is divided into blocks, and the particular picture being encoded / decoded is referred to as the current picture. When a block in the current picture is similar to a reference block in a previously encoded and still buffered reference picture in the video, the block in the current picture may be encoded by a vector called a motion vector. The motion vector points to a reference block in a reference picture, and in the case where multiple reference pictures are used, the motion vector may have a third dimension that identifies the reference picture.

[0094] In some example embodiments, bidirectional prediction techniques may be used for inter-frame prediction. According to such bidirectional prediction techniques, two reference pictures are used, such as a first reference picture and a second reference picture, both of which precede the current picture in the video in decoding order (but may be in the past or future in display order, respectively). A block in the current picture may be encoded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. Specifically, the block may be predicted by a combination of the first reference block and the second reference block.

[0095] Furthermore, merge mode techniques may be used in inter-frame prediction to improve coding efficiency.

[0096] According to some example embodiments of the present disclosure, predictions such as inter-frame prediction and intra-frame prediction are performed in units of blocks. For example, a picture in a video picture sequence is divided into coding tree units (CTUs) for compression, and the CTUs in the picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU includes three parallel coding tree blocks (CTBs), each of which is a luminance CTB and two chrominance CTBs. Furthermore, each CTU can be split into one or more coding units (CUs) in a quadtree. For example, a CTU of 64×64 pixels can be divided into a CU of 64×64 pixels, or 4 CUs of 32×32 pixels. Each of one or more of the 32×32 blocks can be further divided into 4 CUs of 16×16 pixels. In some example embodiments, each CU can be analyzed during encoding to determine the prediction type of the CU among various prediction types (such as an inter-frame prediction type or an intra-frame prediction type). Depending on temporal and / or spatial predictability, a CU can be partitioned into one or more prediction units (PUs). Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In an embodiment, the prediction operation in the codec (encoding / decoding) is performed in units of prediction blocks. The partitioning of CUs into PUs (or PBs of different color channels) can be performed in various spatial modes. For example, a luma or chroma PB may include a value matrix (e.g., luma values) of samples such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 samples, and the like.

[0097] Figure 7A diagram of a video encoder (703) according to another example embodiment of the present disclosure is shown. The video encoder (703) is configured to receive a processed block (e.g., a prediction block) of sample values ​​within a current video picture in a sequence of video pictures, and encode the processed block into an encoded picture as part of an encoded video sequence. The example video encoder (703) may be used instead of Figure 4 The video encoder (403) in the example. Figure 7 is a diagram of a video encoder (703) according to another exemplary embodiment disclosed herein. The video encoder (703) is configured to receive a processed block (e.g., a prediction block) of sample values ​​within a current video picture in a video picture sequence and to encode the processed block into an encoded picture that is part of an encoded video sequence. In this embodiment, the video encoder (703) is configured to replace Figure 4 A video encoder (403) in an embodiment.

[0098] For example, the video encoder (703) receives a matrix of sample values ​​for a processing block, the processing block being, for example, a prediction block of 8×8 samples, etc. The video encoder (703) uses, for example, rate-distortion (RD) optimization to determine whether to use intra mode, inter mode, or bidirectional prediction mode to encode the processing block. When it is determined that the processing block is encoded in intra mode, the video encoder (703) may use intra prediction techniques to encode the processing block into an encoded picture; and when it is determined that the processing block is encoded in inter mode or bidirectional prediction mode, the video encoder (703) may encode the processing block into an encoded picture using inter prediction or bidirectional prediction techniques, respectively. In some example embodiments, merge mode may be used as a sub-mode of inter prediction, in which a motion vector is derived from one or more motion vector prediction values ​​without the benefit of encoded motion vector components other than the prediction values. In some other example embodiments, there may be motion vector components that can be applied to the subject block. Accordingly, the video encoder (703) may include Figure 7 Components not explicitly shown in the example (such as a mode decision module) are used to determine the prediction mode of the processing block.

[0099] exist Figure 7 In an embodiment of the present invention, the video encoder (703) includes: Figure 7 The exemplary arrangement shown in FIG. 1 includes an inter-frame encoder (730), an intra-frame encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together.

[0100] The inter-frame encoder (730) is used to receive samples of a current block (e.g., a processing block), compare the block with one or more reference blocks in a reference picture (e.g., blocks in a previous picture and a subsequent picture), generate inter-frame prediction information (e.g., redundant information description according to an inter-frame coding technique, motion vectors, merge mode information), and calculate an inter-frame prediction result (e.g., a predicted block) based on the inter-frame prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture, and the decoded reference picture is obtained using a decoded reference picture embedded in Figure 6 The exemplary encoder 620 (shown as Figure 7 The residual decoder 728, as described in further detail below, in which the decoding unit 633 performs decoding based on the encoded video information.

[0101] The intra-frame encoder (722) is used to receive samples of a current block (e.g., a processing block), compare the block with an encoded block in the same picture, generate quantization coefficients after transformation, and in some cases also generate intra-frame prediction information (e.g., intra-frame prediction direction information according to one or more intra-frame coding techniques). The intra-frame encoder (722) also calculates an intra-frame prediction result (e.g., a predicted block) based on the intra-frame prediction information and a reference block in the same picture.

[0102] The general controller (721) is used to determine general control data and control other components of the video encoder (703) based on the general control data. In an embodiment, the general controller (721) determines the prediction mode of the block and provides a control signal to the switch (726) based on the prediction mode. For example, when the prediction mode is intra mode, the general controller (721) controls the switch (726) to select the intra mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select intra prediction information and add the intra prediction information to the bitstream; and when the prediction mode is inter mode for the block, the general controller (721) controls the switch (726) to select the inter prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select inter prediction information and add the inter prediction information to the bitstream.

[0103] The residual calculator (723) is used to calculate the difference (residual data) between the received block and the prediction result of the block selected from the intra-frame encoder (722) or the inter-frame encoder (730). The residual encoder (724) is used to encode the residual data to generate a transform coefficient. In an embodiment, the residual encoder (724) is used to convert the residual data from the time domain to the frequency domain and generate a transform coefficient. The transform coefficient is then processed by quantization to obtain a quantized transform coefficient. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is used to perform an inverse transform and generate decoded residual data. The decoded residual data can be used appropriately by the intra-frame encoder (722) and the inter-frame encoder (730). For example, the inter-frame encoder (730) can generate a decoded block based on the decoded residual data and the inter-frame prediction information, and the intra-frame encoder (722) can generate a decoded block based on the decoded residual data and the intra-frame prediction information. The decoded blocks are appropriately processed to generate a decoded picture, and in some embodiments, the decoded picture may be buffered in a memory circuit (not shown) and used as a reference picture.

[0104] The entropy encoder (725) may be configured to format a code stream to include coded blocks and perform entropy coding. The entropy encoder (725) may be configured to include various information in the code stream. For example, the entropy encoder (725) may be configured to include general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the code stream. When a block is encoded and decoded in an inter-frame mode or a merge sub-mode of a bidirectional prediction mode, residual information may not be present.

[0105] Figure 8 FIG. 8 is a diagram of an exemplary video decoder (810) according to another embodiment disclosed herein. The video decoder (810) is configured to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed picture. In an embodiment, the video decoder (810) is configured to replace Figure 4 A video decoder (410) of an embodiment.

[0106] exist Figure 8 In an exemplary embodiment of the arrangement of the video decoder (810), the video decoder (810) includes Figure 8 An entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874) and an intra-frame decoder (872) coupled together are shown in FIG.

[0107] The entropy decoder (871) may be used to reconstruct certain symbols from the encoded picture, which represent syntax elements that constitute the encoded picture. Such symbols may include, for example, a mode for encoding the block (e.g., intra mode, inter mode, bidirectional prediction mode, merge submode, or another submode), prediction information (e.g., intra prediction information or inter prediction information) that may identify certain samples or metadata for prediction by the intra decoder (872) or the inter decoder (880), residual information in the form of, for example, quantized transform coefficients, and the like. In an embodiment, when the prediction mode is inter or bidirectional prediction mode, the inter prediction information is provided to the inter decoder (880); and when the prediction type is an intra prediction type, the intra prediction information is provided to the intra decoder (872). The residual information may be inverse quantized and provided to the residual decoder (873).

[0108] The inter-frame decoder (880) may be configured to receive the inter-frame prediction information and generate an inter-frame prediction result based on the inter-frame prediction information.

[0109] The intra decoder (872) may be configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0110] The residual decoder (873) is used to perform inverse quantization to extract dequantized transform coefficients, and process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (873) may also use some control information (to obtain the quantizer parameter QP), and the information may be provided by the entropy decoder (871) (the data path is not indicated because this is only low-volume data control information).

[0111] The reconstruction module (874) can be configured to combine the residual output by the residual decoder (873) and the prediction result (output by the inter-frame prediction module or the intra-frame prediction module as appropriate) in the spatial domain to form a reconstructed block, which forms a part of the reconstructed picture, which is a part of the reconstructed video. It should be noted that other suitable operations such as deblocking operations can be performed to improve visual quality.

[0112] It should be noted that the video encoder (403), video encoder (603) and video encoder (703) and video decoder (410), video decoder (510) and video decoder (810) may be implemented using any suitable technology. In some embodiments, the video encoder (403), video encoder (603) and video encoder (703) and video decoder (410), video decoder (510) and video decoder (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoder (403), video encoder (603) and video encoder (703) and video decoder (410), video decoder (510) and video decoder (810) may be implemented using one or more processors that execute software instructions.

[0113] Returning to the intra prediction process, samples in a block (e.g., a luma or chroma prediction block, or a coding block if it is not further partitioned into prediction blocks) are predicted by samples of an adjacent, next adjacent, or other row or rows, or a combination thereof, to generate a prediction block. The residual between the actual block being encoded and the prediction block may be processed by a transform, and then quantized. Various intra prediction modes may be made available, and parameters related to intra mode selection and other parameters may be signaled in the bitstream. For example, various intra prediction modes may involve one or more row positions for prediction samples, the direction along which prediction samples are selected from a prediction row or rows, and other special intra prediction modes.

[0114] For example, a set of intra-prediction modes (interchangeably referred to as "intra-modes") may include a predefined number of directional intra-prediction modes. As described above with respect to the exemplary implementation of FIG. 1, these intra-prediction modes may correspond to a predefined number of directions along which out-of-block samples are selected as prediction results for samples being predicted in a particular block. In another specific example implementation, eight (8) primary directional modes may be supported and predefined, corresponding to angles of 45 to 207 degrees from a horizontal axis.

[0115] In some other implementations of intra prediction, in order to further exploit more types of spatial redundancy in directional textures, the directional intra mode can be further extended to a set of angles with finer granularity. For example, the above 8-angle implementation can be configured to provide eight nominal angles, referred to as V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED (e.g. Fig. 9), and for each nominal angle, a predefined number (e.g., 7) of finer angles can be added. With such an extension, a larger total number (e.g., 56 in this example) of directional angles can be used for intra prediction, corresponding to the same number of predefined directional intra modes. The predicted angle can be represented by the nominal intra angle plus the angle increment. For the specific example above, each nominal angle has 7 finer angular directions, and the angle increment can be -3 to 3 times the step size of 3 degrees.

[0116] In some implementations, instead of the above directional intra-frame mode or as a supplement thereof, a predetermined number of non-directional intra-frame prediction modes can also be predefined and made available. For example, 5 non-directional intra-frame modes referred to as smooth intra-frame prediction modes can be specified. These non-directional intra-frame mode prediction modes can be specifically referred to as DC, PAETH, SMOOTH, SMOOTH_V and SMOOTH_H intra-frame modes. Fig.10 The prediction of samples for a particular block under these exemplary non-directional modes is illustrated in FIG. As an example, Fig.10 A 4×4 block 1002 is shown that is predicted by samples from a top neighboring row and / or a left neighboring row. A particular sample 1010 in the block 1002 may correspond to a sample 1004 directly above the sample 1010 in the top neighboring row of the block 1002, a sample 1006 above and to the left of the sample 1010 as the intersection of the top and left neighboring rows, and a sample 1008 directly to the left of the sample 1010 in the left neighboring row of the block 1002. For the exemplary DC intra prediction mode, an average of the left neighboring sample 1008 and the above neighboring sample 1004 may be used as a prediction value for the sample 1010. For the exemplary PAETH intra prediction mode, the top reference sample 1004, the left reference sample 1008, and the top left reference sample 1006 may be extracted, and then any value among the three reference samples that is closest to (top+left-topleft) may be set as the prediction value for the sample 1010. For the exemplary SMOOTH_V intra prediction mode, sample 1010 may be predicted by quadratic interpolation in the vertical direction of the upper left neighboring sample 1006 and the left neighboring sample 1008. For the exemplary SMOOTH_H intra prediction mode, sample 1010 may be predicted by quadratic interpolation in the horizontal direction of the upper left neighboring sample 1006 and the top neighboring sample 1004. For the exemplary SMOOTH intra prediction mode, sample 1010 may be predicted by the average of the quadratic interpolation in the vertical and horizontal directions. The above non-directional intra mode implementations are illustrated only as non-limiting examples. Other neighboring rows and other non-directional selections of samples, as well as ways of combining prediction samples for predicting a particular sample in a prediction block, also need to be considered.

[0117] The selection of a particular intra-frame prediction mode from the above directional or non-directional mode can be signaled in the code stream to the encoder at various coding levels (picture, strip, block, unit, etc.). In some exemplary implementations, exemplary 8 nominal directional modes and 5 non-angular smooth modes (a total of 13 options) can be first signaled. Then, if the signaled mode is one of the 8 nominal angle intra-frame modes, an index is further signaled to indicate the selected angle increment as the corresponding nominal angle signaled. In some other exemplary implementations, all intra-frame prediction modes can be indexed together (e.g., 56 directional modes plus 5 non-directional modes to produce 61 intra-frame prediction modes) for signaling.

[0118] In some example implementations, Example 56 or other number of directional intra prediction modes may be implemented with a unified directional predictor that projects each sample of the block to a reference subsample location and interpolates the reference samples through a 2-tap bilinear filter.

[0119] In some implementations, in order to capture the attenuated spatial correlation with the reference on the edge, additional filter modes called filter intra-frame modes can be designed. For these modes, the predicted samples within the block in addition to the samples outside the block can be used as intra-frame prediction reference samples for some patches within the block. For example, these modes can be predefined and made available for intra-frame prediction of at least luminance blocks (or only luminance blocks). A predefined number (for example, five) of filter intra-frame modes can be pre-designed, each filter intra-frame mode is represented by a set of n-tap filters (for example, 7-tap filters), and the set of n-tap filters reflects the correlation between the samples in, for example, a 4×2 patch and its n adjacent neighbors. In other words, the weighting factors of the n-tap filters can be position-dependent. Taking 8×8 blocks, 4×2 patches and 7-tap filtering as an example, as Fig.11 As shown in , the 8×8 block 1102 can be partitioned into eight 4×2 patches. These patches are Fig.11 In the example, the patches are denoted by B0, B1, B2, B3, B4, B5, B6, and B7. For each patch, its 7 neighbors (in Fig.11 The samples in the current patch (indicated by R0~R7) can be used to predict the samples in the current patch. For patch B0, all neighbors may have been reconstructed. But for other patches, some neighbors are in the current block and therefore may not be reconstructed, and then the predicted values ​​of the direct neighbors are used as reference. For example, Fig.11 All neighbors of patch B7 indicated in are not reconstructed, so the predicted samples of the neighbors are used instead.

[0120] In some implementations of intra prediction, one color component may be predicted using one or more other color components. The color component may be any of the components in a color space such as YCrCb, RGB, XYZ, etc. For example, prediction of a chrominance component (e.g., a chrominance block) (referred to as chrominance from luminance, CfL) from a luminance component (e.g., a luminance reference sample) may be implemented. In some exemplary implementations, only cross-color prediction is allowed from luminance to chrominance. For example, chrominance samples in a chrominance block may be modeled as a linear function of consistently reconstructed luminance samples. CfL prediction may be implemented as follows:

[0121] CfL(α)=α×L AC +DC (1)

[0122] Among them, L AC represents the AC contribution of the luma component, α represents the parameters of the linear model, and DC represents the DC contribution of the chroma component. For example, an AC component is obtained for each sample of a block, and a DC component is obtained for the entire block. Specifically, the reconstructed luma samples can be subsampled to the chroma resolution, and then the average luma value (DC of luma) can be subtracted from each luma value to form the AC contribution in luma. The AC contribution of luma is then used in the linear model of equation (1) to predict the AC values ​​of the chroma components. In order to approximate or predict the chroma AC components based on the luma AC contribution, instead of requiring the decoder to calculate the scaling parameters, an exemplary CfL implementation can determine the parameter α based on the original chroma samples and signal them in the codestream. This reduces decoder complexity and produces more accurate predictions. As for the DC contribution of the chroma component, in some exemplary implementations, it can be calculated using the intra-frame DC model within the chroma component.

[0123] The transformation of the residual of the intra-frame prediction block or the inter-frame prediction block can then be implemented, followed by quantization of the transform coefficients. For the purpose of performing the transformation, both the intra-frame coding block and the inter-frame coding block can be further partitioned into multiple transform blocks (sometimes used interchangeably as "transform units" even though the term "unit" is generally used to represent a set of three color channels, for example, a "coding unit" will include a luminance coding block and a chrominance coding block) before the transformation. In some implementations, the maximum partition depth of the coded block (or prediction block) can be specified (the term "coded block" can be used interchangeably with "coding block"). For example, such partitioning may not exceed 2 levels. The division from the prediction block to the transform block can be handled differently between the intra-frame prediction block and the inter-frame prediction block. However, in some implementations, such division between the intra-frame prediction block and the inter-frame prediction block can be similar.

[0124] In some example implementations, and for intra-coded blocks, transform partitioning may be done in such a way that all transform blocks have the same size, and the transform blocks are encoded in raster scan order. Fig.12 An example of such transform block partitioning of an intra-coded block is shown in FIG. Fig.12 The coded block 1202 is shown partitioned into 16 transform blocks of the same block size by an intermediate level quadtree partition 1204, as shown at 1206. An exemplary raster scan order for encoding is represented by Fig.12 The ordered arrows in are shown.

[0125] In some exemplary implementations, for inter-coded blocks, transform unit partitioning may be performed in a recursive manner, where the partitioning depth reaches a predefined number of levels (e.g., 2 levels). Fig.13 As shown in , the splitting can be stopped or continued for any subpartition and at any level recursively. In particular, Fig.13 An example is shown in which a block 1302 is partitioned into four quadtree subblocks 1304, and one of the subblocks is further partitioned into four second-level transform blocks, while the partitioning of the other subblocks stops after the first level, resulting in a total of 7 transform blocks of two different sizes. An exemplary raster scan order for encoding is further represented by Fig.13 The ordered arrows in the figure are shown. Fig.13 An exemplary implementation of quadtree partitioning of up to two levels of square transform blocks is shown, but in some generation implementations, transform partitioning can support transform block shapes of 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1, and sizes from 4×4 to 64×64. In some exemplary implementations, if the coding block is less than or equal to 64×64, the transform block partitioning can be applied only to the luma component (in other words, the chroma transform block will be the same as the coding block under this condition). Otherwise, if the coding block width or height is greater than 64, both the luma coding block and the chroma coding block can be implicitly partitioned into multiples of min(W, 64)×min(H, 64) and min(W, 32)×min(H, 32) transform blocks, respectively.

[0126] Each of the above transform blocks may then undergo a primary transform. The primary transform essentially moves the residuals in the transform block from the spatial domain to the frequency domain. In some implementations of the actual primary transform, multiple transform sizes (ranging from 4 to 64 points for each of the two dimensions) and transform shapes (square; rectangular with width / height ratios of 2:1 / 1:2 and 4:1 / 1:4) may be allowed in order to support the above exemplary extended coding block partitions.

[0127] Turning to the actual main transform, in some exemplary implementations, the 2-D transform process may involve the use of a hybrid transform kernel (e.g., which may be composed of a different 1-D transform for each dimension of the encoded residual transform block). Example 1-D transform kernels may include, but are not limited to: a) 4-point, 8-point, 16-point, 32-point, 64-point DCT-2; b) 4-point, 8-point, 16-point asymmetric DST (DST-4, DST-7) and their flipped versions; c) 4-point, 8-point, 16-point, 32-point identity transform. The selection of the transform kernel for each dimension may be based on a rate-distortion (RD) criterion. For example, the basis functions of DCT-2 and asymmetric DST that can be implemented are listed in Table 1.

[0128] Table 1: Exemplary primary transform basis functions (DCT-2, DST-4, and DST-7 for N-point input).

[0129]

[0130] In some exemplary implementations, the availability of hybrid transform kernels for a particular primary transform implementation may be based on the transform block size and the prediction mode. Exemplary dependencies are listed in Table 2. For chroma components, transform type selection may be performed in an implicit manner. For example, for intra-prediction residuals, a transform type as specified in Table 3 may be selected based on the intra-prediction mode. For inter-prediction residuals, the transform type for the chroma block may be selected based on the transform type selection for the co-located luma block. Therefore, for chroma components, there is no transform type signaling in the bitstream.

[0131] Table 2: AV1 hybrid transform kernels and their availability based on prediction mode and block size. Here, → and ↓ represent the horizontal and vertical dimensions; √ and × represent the availability of the kernel for that block size and prediction mode.

[0132]

[0133]

[0134] Table 3: Transform type selection for chroma component intra prediction residual.

[0135]

[0136] In some implementations, a secondary transform may be performed on the primary transform coefficients. Fig.14As shown in , LFNST (Low Frequency Non-separable Transform), which is called a reduced secondary transform, can be applied between the forward main transform and quantization (at the encoder) and between dequantization and the inverse main transform (at the decoder end) to further decorrelate the main transform coefficients. In essence, LFNST can take a portion of the main transform coefficients, such as the low-frequency portion (thus "reduced" from the complete set of main transform coefficients of the transform block) to proceed to the secondary transform. In the example LFNST, a 4×4 non-separable transform or an 8×8 non-separable transform can be applied depending on the transform block size. For example, 4×4 LFNST can be applied to small transform blocks (e.g., min (width, height) <8), while 8×8 LFNST can be applied to larger transform blocks (e.g., min (width, height)>8). For example, if an 8×8 transform block undergoes 4×4 LFNST, only the low-frequency 4×4 portion of the 8×8 main transform coefficients is further subjected to a secondary transform.

[0137] like Fig.14 As specifically shown in , the transform block can be 8×8 (or 16×16). Therefore, the forward main transform 1402 of the transform block produces an 8×8 (or 16×16) main transform coefficient matrix 1404, where each square unit represents a 2×2 (or 4×4) part. The input of the forward LFNST may not be the entire 8×8 (or 16×16) main transform coefficients, for example. For example, a 4×4 (or 8×8) LFNST may be used for the secondary transform. In this way, as indicated by the shaded portion (upper left) 1406, only the 4×4 (or 8×8) low-frequency main transform coefficients of the main transform coefficient matrix 1404 can be used as input to the LFNST. The remaining portion of the main transform coefficient matrix may not be subjected to the secondary transform. In this way, after the secondary transform, the portion of the main transform coefficient that is subjected to LFNST becomes the secondary transform coefficient, while the remaining portion that is not subjected to LFNST (e.g., the unshaded portion of the matrix 1404) retains the corresponding main transform coefficient. In some example implementations, the remaining portions that are not subjected to secondary transformation may all be set to zero coefficients.

[0138] The following describes an application example using a non-separable transform in LFNST. For the application example 4×4 LFNST, a 4×4 input block X (denoted e.g. Fig.14 The 4×4 low frequency portion of the main transform coefficient block 1406 of the main transform matrix 1404 can be expressed as:

[0139]

[0140] The 2-D input matrix can first be linearized or scanned into a vector in the example order.

[0141]

[0142] The non-separable transform of the 4×4 LFNST can then be calculated as where indicates the output transform coefficient vector, and T is a 16×16 transform matrix. Subsequently, the resulting 16×1 coefficient vector is scanned in reverse order as 4×4 blocks using the scan order of the block (e.g., horizontal, vertical, or diagonal). Coefficients with smaller indices can be placed in the 4×4 coefficient block together with smaller scan indices. In this way, the redundancy in the primary transform coefficients X can be further exploited via the second transform T, thereby providing additional compression enhancement.

[0143] The above example LFNST applies the non-separable transform based on the direct matrix multiplication method, such that it is implemented in a single pass without multiple iterations. In some further exemplary implementations, the dimension of the non-separable transform matrix (T) of the example 4×4 LFNST can be further reduced to minimize the computational complexity and memory space requirements for storing the transform coefficients. Such an implementation can be referred to as a reduced non-separable transform (RST). More specifically, the main idea of RST is to map an N-dimensional vector (where N is 4×4 = 16 in the above example, but can be equal to 64 for an 8×8 block) to an R-dimensional vector in a different space, where N / R (R < N) represents the dimension reduction factor. Thus, instead of an N×N transform matrix, the RST matrix becomes an R×N matrix as follows:

[0144]

[0145] where the R rows of the transform matrix are the reduced R basis of the N-dimensional space. Thus, this transform converts the input vector or N dimensions into an output vector of reduced R dimensions. In this way, and as Fig.14 shown, the secondary transform coefficients 1408 transformed from the primary coefficients 1406 are reduced in dimension by a factor or N / R. Fig.14 The three squares around 1408 in

[0146] The inverse transform matrix of the RTS can be the transpose of its forward transform. For the example 8×8 LFNST (in contrast to the 4×4 LFNST above, which is described more diversely here), an example reduction factor of 4 can be applied, and thus the 64×64 direct non-separable transform matrix is ​​correspondingly reduced to a 16×64 direct matrix. Further, in some implementations, a portion of the input primary coefficients, rather than all, can be linearized into an input vector for the LFNST. For example, only a portion of the example 8×8 input main transform coefficients can be linearized into the above X vector. For a specific example, in the four 4×4 quadrants of the 8×8 main transform coefficient matrix, the lower right (high frequency coefficient) can be omitted, and only the other three quadrants can be linearized into 48×1 vectors instead of 64×1 vectors using a predefined scan order. In such an implementation, the non-separable transform matrix can be further reduced from 16×64 to 16×48.

[0147] Therefore, the example reduced 48×16 inverse RST matrix can be used at the decoder end to generate the upper left, upper right, and lower left 4×4 quadrants of the 8×8 core (primary) transform coefficients. Specifically, when a further reduced 16×48 RST matrix is ​​applied instead of a 16×64 RST with the same transform set configuration, the non-separable secondary transform takes as input the vectorized 48 matrix elements taken from the three 4×4 quadrant blocks of the 8×8 primary coefficient block except for the lower right 4×4 block. In such an implementation, the omitted lower right 4×4 primary transform coefficients will be ignored in the secondary transform. This further reduced transform will convert the 48×1 vector into a 16×1 output vector, which is scanned back into a 4×4 matrix to fill the Fig.14 1408 in . The three squares surrounding the secondary transform coefficients of 1408 may be filled with zeros.

[0148] With this dimensionality reduction in RST, the memory usage for storing all LFNST matrices is reduced. In the above example, the memory usage can be reduced from 10KB to 8KB, for example, with insignificant performance degradation compared to an implementation without dimensionality reduction.

[0149] In some implementations, to reduce complexity, LFNST can be further restricted to only outside the main transform coefficient portion to be subjected to LFNST (e.g., in Fig.14 Therefore, when LFNST is applied, all the main transform coefficients (e.g., Figure 4The unshaded portion of the primary coefficient matrix 1404 of ) can be close to zero. Such a restriction allows the LFNST index signaling to be adjusted at the last-significant position and thus avoids some additional coefficient scans that may be required to check the significant coefficients at specific positions when such a restriction is not applied. In some implementations, the worst-case processing of LFNST (in terms of per-pixel multiplications) can limit the inseparable transforms of 4×4 blocks and 8×8 blocks to 8×16 transforms and 8×48 transforms, respectively. In these cases, when LFNST is applied, the last-significant scan position must be less than 8 for other sizes less than 16. For blocks of shape 4×N and N×4 with N>8, the above restriction means that LFNST is now only applied once to the upper left 4×4 region. Since all primary-only coefficients are zero when LFNST is applied, the number of operations required for the main transform is reduced in this case. From the encoder's perspective, the quantization of coefficients can be simplified when testing the LFNST transform. For the first 16 coefficients (in scan order), rate-distortion optimized quantization (RDO) must be performed to the greatest extent possible, and the remaining coefficients can be forced to zero.

[0150] In some example implementations, the available RST kernels may be specified as multiple transform sets, each of which includes multiple inseparable transform matrices. For example, each transform set used in LFNST may have a total of 4 transform sets and 2 inseparable transform matrices (kernels). These kernels may be pre-trained offline, and therefore they are data-driven. The transform kernels trained offline may be stored in a memory, or hard-coded in an encoding device or a decoding device for use during the encoding / decoding process. The selection of a transform set during the encoding or decoding process may be determined by an intra-frame prediction mode. The mapping from an intra-frame prediction mode to a transform set may be predefined. An example of such a predefined mapping is shown in Table 4. For example, as shown in Table 4, if one of the three cross-component linear models (CCLM, Cross-Component LinearModel) modes (INTRA_LT_CCLM, INTRA_T_CCLM, or INTRA_L_CCLM) is used for the current block (ie, 81≤predModeIntra≤83), then transform set 0 may be selected for the current chroma block. For each transform set, the selected inseparable secondary transform candidate may be further specified by an explicitly signaled LFNST index. For example, for each intra CU, the index may be signaled once in the codestream after the transform coefficients.

[0151] Table 4: Transformation selection table

[0152] IntraPredMode Transformation Collection Index IntraPredMode<0 1 0<=IntraPredMode<=1 0 2<=IntraPredMode<=12 1 13<=IntraPredMode<=23 2 24<=IntraPredMode<=44 3 45<=IntraPredMode<=55 2 56<=IntraPredMode<=80 1 81<=IntraPredMode<=83 0

[0153] Because LFNST in the above exemplary implementation is limited to being applicable only to all coefficients outside the first coefficient subgroup or part when they are non-significant, the LFNST index encoding depends on the position of the last significant coefficient. In addition, the LFNST index can be context-coded, but does not depend on the intra prediction mode, and only the first binary number can be context-coded. In addition, LFNST can be applied to intra CUs in both intra and inter slices, as well as to both luma and chroma. If dual tree is enabled, the LFNST indexes for luma and chroma can be signaled separately. For inter slices (dual tree disabled), a single LFNST index can be signaled and used for both luma and chroma.

[0154] In some example implementations, when the intra sub-partitioning (ISP) mode is selected, LFNST may be disabled and the RST index may not be signaled, because even if RST is applied to each feasible partition block, the performance improvement may be negligible. In addition, disabling RST for the residual of ISP prediction can reduce coding complexity. In some further implementations, when the multiple linear regression intra prediction (MIP) mode is selected, LFNST may also be disabled and the RST index may not be signaled.

[0155] Considering that large CUs larger than 64×64 (or any other predefined size representing the maximum transform block size) are implicitly partitioned (e.g., TU tiling) due to the existing maximum transform size limit (e.g., 64×64), LFNST index search may increase the data buffer by a factor of four for a certain number of decoding pipeline stages. Therefore, in some implementations, the maximum size allowed for LFNST may be limited to, for example, 64×64. In some implementations, LFNST may be enabled only with DCT2 as the primary transform.

[0156] In some other implementations, an intra secondary transform (IST) is provided for the luminance component by defining, for example, 12 sets of secondary transforms, where, for example, each set has 3 kernels. Intra-frame mode-related indexes can be used for transform set selection. Kernel selection within a set can be based on syntax elements signaled. IST can be enabled when DCT2 or ADST is used as horizontal main transform and vertical main transform. In some implementations, depending on the block size, a 4×4 inseparable transform or an 8×8 inseparable transform can be selected. If min(tx_width, tx_height)<8, a 4×4 IST can be selected. For larger blocks, an 8×8 IST can be used. Here, tx_width and tx_height correspond to the transform block width and height, respectively. The input to the IST can be low-frequency main transform coefficients in a zigzag scan order.

[0157] Using data-driven or offline-trained transform kernels for primary or secondary transforms (especially inseparable transform kernels) can promote enhanced coding performance compared to using only fixed transforms such as DCT / DST. However, applying such data-driven transform kernels can increase the complexity of the video codec. For example, the data-driven transform kernel may require a large amount of memory to store all possible kernels, especially for inseparable transforms. Specifically, for one of the above example implementations, there may be a set of 12 kernels. Each set may include, for example, 3 kernels for different LFNSTs (e.g., 4×4 and 8×8). Further, the LFNST kernel may be different for different primary transform types (e.g., DCT or ADST primary types of LFNST may be allowed). In other words, if memory is used to store offline trained LFNST kernels, a large amount of memory will be required. The various implementations described below for using data-driven transforms can be designed to reduce the memory footprint of the stored kernels and the complexity of the codec, as described in further detail below.

[0158] These various implementations or embodiments are merely examples. They can be used alone or can be combined in any order or manner. Further, each of the implementations can be implemented as a method, and an encoder device or decoder including a processing circuit (e.g., one or more processors or one or more integrated circuits) to implement the method. In one example, the processing circuit may include one or more processors that execute a program stored in a non-volatile computer-readable medium. In another example, the processing circuit may be hard-coded to implement the method. In yet another example, the processing circuit may be a mixture of a processor component and a hard-coded circuit for executing computer-readable instructions.

[0159] In the following example implementations, the term "block" may be interpreted as a prediction block or a coding block, etc. The term "block" may also be used herein to refer to a transform block, which may be a part of a coding block as described above. The term "block size" is used to refer to any of the width, height, block aspect ratio, block area size, or minimum / maximum value between width and height of a block.

[0160] In various example implementations below, one or more indexes may be signaled in a video bitstream, the one or more indexes being used to identify one or more separable or non-separable transforms (or transform kernels) from a set of separable or non-separable secondary transforms for decoding a current transform block. Each of such indexes may be denoted as stIdx. These implementations may be applied to a primary transform, a secondary transform, or an additional transform applied after a secondary transform. Further, the basic principles may be applied to a separable transform or a non-separable transform.

[0161] In the following example implementations, the term "basis vector" is used to refer to the spatial frequency components of a transform kernel used for image transformation from the spatial domain to the frequency domain. The term "high-frequency basis vector" and the like are used to refer to basis vectors that can be used to generate transform coefficients that are scanned from low-frequency components to high-frequency components after N coefficients, and the term "high-frequency basis vector" can also refer to basis vectors that are located after N rows (or columns) of an inseparable transform kernel (and therefore above the frequency corresponding to N). Example values ​​of N include, but are not limited to, any integer value between 1 and 128 (including the end value). The definition of high and low frequencies (the number N) can be determined by RDO.

[0162] In some general example implementations, when multiple transform kernels can be applied during the encoding and decoding process and provided as selectable options, a portion of the basis vectors of the multiple transform kernels can be shared. The multiple transform kernels are secondary transform kernels; the two or more transform kernels of the multiple transform kernels that share the N high-frequency basis vectors are configured to transform the main transform coefficients, which are generated from the main transform with the same transform type. For example, high-frequency basis vectors between some or all of the multiple transform kernels can be shared, while low-frequency basis vectors can be personalized between the multiple transform kernels. In this case, such offline trained and data-driven transform kernels may require less memory in total. In particular, the shared basis vectors do not need to be copied, and a copy may need to be saved in memory between the shared transform kernels. The manner in which the shared and personalized basis vectors are stored in memory can be predefined.

[0163] This is achieved in Fig.15, which shows an example of how basis vectors are shared between two LFNST transformation matrices or kernels for a forward secondary transform. Two blocks 1502 and 1504 indicate two transformation matrices, i.e., transformation matrices A (left) and B (right), each row of the transformation matrices represents a basis vector, and the bottom row (shaded) refers to the basis vectors shared between A and B. The bottom basis vectors may have a higher frequency. The higher frequency basis vectors may be shared in memory between these example kernels. For the inverse transform, as a result of the transpose of the inverse transform, the shared basis vectors of multiple transformation matrices may be the right N columns of the transformation matrix.

[0164] Fig.15 An example transform kernel of is shown for a secondary transform kernel that is applied to a linearized vector of primary transform coefficients. Fig.15 The frequency axis of such a transform kernel shown in is one-dimensional (from top to bottom). This will be the case for all 1-D kernels. For a 2-D transform kernel (e.g., when a data-driven 2-D kernel is used for the main transform), the lower frequency components will be at the upper left portion of the 2-D transform kernel, while the higher frequency components will be at the lower right portion of the 2-D transform kernel. In this case, the lower right portion can be shared among different 2-D kernels.

[0165] In some example implementations, for a set of multiple transform kernels applied to a particular intra-frame prediction mode, the high-frequency basis vectors can be constructed to be the same (shared), while the low-frequency basis vectors can be different (personalized). In other words, the set of kernels corresponding to a particular intra-frame prediction mode (e.g., the various directional, non-directional, and other intra-frame prediction modes described above) can have similar dimensions, and thus can share (for example) high-frequency basis vectors with personalized low-frequency basis vectors. Predefined thresholds can be used to define low and high frequencies. In some implementations, the definition of high and low frequencies can be data-driven and therefore pre-trained. In this way, the frequency definition thresholds can be different for different intra-frame prediction modes.

[0166] In some example implementations, when multiple secondary transform kernels are applied with different primary transforms, some or all of the multiple secondary transform kernels may be shared between different primary transform types. In other words, different types of primary transforms may correspond to different kernels for secondary transforms. These different secondary transform kernels may contain personalized low-frequency basis vectors and still be constructed with shared high-frequency basis vectors. That is, if applicable to multiple primary transform types, the secondary transform kernel selection may be independent of the primary transform type (e.g., independent of the DCT or ADST primary transform type).

[0167] In some example implementations, when multiple secondary transform kernels are applied with different primary transforms, the high frequency basis vectors in the multiple secondary transform kernels can be shared between different primary transform types, while the low frequency basis vectors can be different or personalized. Predefined thresholds can be used to define low and high frequencies. In addition, the definition of high and low frequencies (or the number N) can be data driven and thus pre-trained. In this way, the frequency definition threshold can be different for different intra prediction modes.

[0168] In some example implementations, for multiple secondary transform kernels applied with a particular type of primary transform, the high frequency basis vectors may be the same, while the low frequency basis vectors may be different. In some further implementations, the high frequency basis vectors and low frequency basis vectors of kernels corresponding to different primary transform types may be independent.

[0169] In some example implementations, the number N of high-frequency basis vectors shared between various transform kernels may be an integer power of 2, such as 1, 2, 4, 8, 16, 32, 64, 128. In particular, for the reduced secondary transform matrix, the number N of high-frequency rows may be a value that is a power of 2, such as 1, 2, 4, 8, 16, 32, 64, 128, where N is less than a predefined upper limit.

[0170] In some example implementations, the value of N may depend on the transform size. For example, a ratio between N and the total number of basis vectors of the transform kernel may be specified.

[0171] In some examples, each and any combination of the above implementations may be applied to a LFNST secondary transform. Thus, a set of kernels that share high frequency basis vectors but contain personalized low frequency basis vectors may comprise a secondary LFNST transform kernel.

[0172] In some examples, each and any combination of the above implementations may be applied to an Intra Secondary Transform (IST).

[0173] In some examples, each and any combination of the above implementations can be applied to a Line Graph Transform (LGT).

[0174] The various kernels above may be predetermined. They may be pre-trained offline. For kernels that share high frequency basis vectors, they may be trained jointly. In some embodiments, the plurality of transform kernels are jointly trained offline to determine the N shared high frequency basis vectors. The N high frequency basis vectors include frequency basis vectors having a frequency higher than a predetermined threshold frequency. The threshold frequency defining the shared basis vectors and the personalized basis vectors may be trained offline. The plurality of transform kernels and the predetermined threshold frequency are pre-trained offline.

[0175] During the encoding or decoding process, the above various cores can reside in the memory space for use by the encoder or decoder, and only one copy of the shared base vectors needs to be stored in the memory space. When the various cores are used during the encoding or decoding process, the shared base vectors can be accessed using pointers to the memory locations of the shared base vectors.

[0176] Fig.16 A flowchart 1600 of an example method following the principles on which the above implementation is based is shown. The example method flow starts at 1601. In S1610, a plurality of transform kernels are identified, wherein each transform kernel of the plurality of transform kernels includes a set of basis vectors from low frequency to high frequency; N high frequency basis vectors of two or more transform kernels of the plurality of transform kernels are shared, N is a positive integer; and low frequency basis vectors of two or more transform kernels of the plurality of transform kernels that are different from the N high frequency basis vectors are personalized. In S1620, a data block is extracted from a video bitstream. The plurality of transform kernels include secondary transform kernels suitable for transforming primary transform coefficients; the data block includes an array of primary transform coefficients. In S1630, a transform kernel is selected from the plurality of transform kernels based on information associated with the data block. In S1640, the transform kernel is applied to at least a portion of the data block to generate a transform block. The two or more transform cores among the multiple transform cores that share the N high-frequency basis vectors correspond to the same one intra-frame picture prediction mode among multiple intra-frame picture prediction modes; all transform cores among the multiple transform cores share the N high-frequency basis vectors and other personalized low-frequency basis vectors, and the multiple transform cores are assigned to the same one intra-frame picture prediction mode among the multiple intra-frame picture prediction modes. In some embodiments, the two or more transform cores among the multiple transform cores that share the N high-frequency basis vectors are assigned to two or more different intra-frame picture prediction modes. The example method flow ends at S1699.

[0177] The embodiments of the present disclosure may be used alone or in any order. In addition, each of the method (or embodiment), encoder and decoder may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In some embodiments, a device for processing video information includes a circuit configured to perform the method of each embodiment of the present application. In one example, one or more processors execute a program stored in a non-volatile computer readable medium. The embodiments of the present disclosure may be applied to luminance blocks or chrominance blocks.

[0178] The above techniques may be implemented as computer software via computer-readable instructions and physically stored in one or more computer-readable media. Fig.17 A computer system (1700) is shown that is suitable for implementing certain embodiments of the disclosed subject matter.

[0179] The computer software may be encoded in any suitable machine code or computer language, and may be assembled, compiled, linked, or the like to create a code comprising instructions, which may be directly executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or executed by decoding, microcode, or the like.

[0180] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablets, servers, smartphones, gaming devices, IoT devices, etc.

[0181] Fig.17 The components shown for the computer system (1700) are exemplary in nature and are not intended to limit the scope of use or functionality of the computer software implementing the embodiments of the present application. The configuration of the components should not be interpreted as having any dependency or requirement on any component or combination of components shown in the exemplary embodiment of the computer system (1700).

[0182] The computer system (1700) may include certain human-computer interface input devices. Such human-computer interface input devices may respond to input from one or more human users through tactile input (e.g., keyboard input, sliding, data glove movement), audio input (e.g., sound, applause), visual input (e.g., gestures), and olfactory input (not shown). The human-computer interface devices may also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (e.g., speech, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still image camera), and videos (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0183] The human-computer interface input device may include one or more of the following (only one of which is drawn): keyboard (1701), mouse (1702), touchpad (1703), touch screen (1710), data gloves (not shown), joystick (1705), microphone (1706), scanner (1707), camera (1708).

[0184] The computer system (1700) may also include certain human-computer interface output devices. Such human-computer interface output devices may stimulate one or more human user senses through, for example, tactile output, sound, light, and smell / taste. Such human-computer interface output devices may include tactile output devices (e.g., tactile feedback through a touch screen (1710), a data glove (not shown), or a joystick (1705), but there may also be tactile feedback devices that are not used as input devices), audio output devices (e.g., speakers (1709), headphones (not shown)), visual output devices (e.g., screens (1710) including cathode ray tube screens, liquid crystal screens, plasma screens, organic light emitting diode screens, each of which has or does not have a touch screen input function, each of which has or does not have a tactile feedback function - some of which can output two-dimensional visual output or output of more than three dimensions through means such as stereoscopic picture output; virtual reality glasses (not shown), holographic displays, and smoke boxes (not shown)) and printers (not shown).

[0185] The computer system (1700) may also include human-accessible storage devices and their associated media, such as optical media including high-density read-only / rewritable compact disks (CD / DVD ROM / RW) (1720) with CD / DVD or similar media (1721), thumb drives (1722), removable hard disk drives or solid state drives (1723), traditional magnetic media such as tapes and floppy disks (not shown), special-purpose ROM / ASIC / PLD-based devices such as security software protectors (not shown), and the like.

[0186] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0187] The computer system (1700) may also include an interface (1754) to one or more communication networks (1755). The network may be wireless, wired, or optical. The network may also be a local area network, a wide area network, a metropolitan area network, an in-vehicle network, an industrial network, a real-time network, a delay-tolerant network, and the like. Examples of networks may include local area networks such as Ethernet, wireless local area networks, cellular networks (GSM, 3G, 4G, 5G, LTE, etc.), television wired or wireless wide area digital networks (including cable television, satellite television, and terrestrial broadcast television), in-vehicle and industrial networks (including CANBus), and the like. Some networks typically require an external network interface adapter for connecting to some universal data port or peripheral bus (1749) (e.g., a USB port of the computer system (1700)); other systems are typically integrated into the core of the computer system (1700) by connecting to a system bus as described below (e.g., an Ethernet interface integrated into a PC computer system or a cellular network interface integrated into a smart phone computer system). By using any of these networks, the computer system (1700) can communicate with other entities. The communication can be one-way, for receiving only (e.g., wireless television), one-way for sending only (e.g., CAN bus to certain CAN bus devices), or two-way, such as to other computer systems via a local or wide area digital network. Each of the above networks and network interfaces can use certain protocols and protocol stacks.

[0188] The above-mentioned human-machine interface devices, human-accessible storage devices, and network interfaces may be connected to the core (1740) of the computer system (1700).

[0189] The core (1740) may include one or more central processing units (CPUs) (1741), graphics processing units (GPUs) (1742), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (1743), hardware accelerators for specific tasks (1744), etc. These devices, as well as read-only memory (ROM) (1745), random access memory (1746), internal mass storage (e.g., internal non-user accessible hard disk drives, solid-state drives, etc.) (1747), etc., may be connected via a system bus (1748). In some computer systems, the system bus (1748) may be accessed in the form of one or more physical plugs so that it may be expanded by additional central processing units, graphics processing units, etc. Peripheral devices may be directly attached to the core's system bus (1748) or connected via a peripheral bus (1749). The architecture of the peripheral bus includes PCI, USB, etc. In one example, the screen (1710) may be connected to a graphics adapter (1750). The architecture of the peripheral bus includes PCI, USB, etc.

[0190] The CPU (1741), GPU (1742), FPGA (1743) and accelerator (1744) can execute certain instructions, which can be combined to form the above-mentioned computer code. The computer code can be stored in ROM (1745) or RAM (1746). Transition data can also be stored in RAM (1746), while permanent data can be stored in, for example, internal mass storage (1747). Fast storage and retrieval of any memory device can be achieved by using a cache memory, which can be closely associated with one or more CPUs (1741), GPUs (1742), mass storage (1747), ROM (1745), RAM (1746), etc.

[0191] The computer readable medium may have computer code thereon for performing various computer-implemented operations. The medium and computer code may be specially designed and constructed for the purpose of this application, or may be medium and code well known and available to those skilled in the art of computer software.

[0192] As a non-limiting example, a computer system having an architecture (1700), in particular a core (1740), can provide the function of executing software contained in one or more tangible computer-readable media as a processor (including a CPU, GPU, FPGA, accelerator, etc.). Such a computer-readable medium can be a medium associated with the above-mentioned user-accessible mass storage, as well as a specific memory of the core (1740) having non-volatility, such as a core internal mass storage (1747) or ROM (1745). Software implementing various embodiments of the present application can be stored in such a device and executed by the core (1740). Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software can enable the core (1740), in particular the processor therein (including a CPU, GPU, FPGA, etc.) to perform a specific process or a specific part of a specific process described herein, including defining a data structure stored in RAM (1746) and modifying such a data structure according to a software-defined process. Additionally or alternatively, the computer system may provide functionality hardwired in logic or otherwise contained in circuitry (e.g., accelerator (1744)) that may operate in place of or in conjunction with software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may include logic and vice versa. Where appropriate, references to computer-readable media may include circuitry (e.g., an integrated circuit (IC)) storing the executing software, circuitry containing the executing logic, or both. The present application includes any suitable combination of hardware and software.

[0193] Although the present disclosure has described several exemplary embodiments, there are changes, permutations and various alternative equivalents that fall within the scope of the present disclosure. In the above implementations and embodiments, any operation of the process can be combined or arranged in any quantity or order as needed. Further, two or more operations of the process described above can be performed in parallel. Therefore, it should be understood that those skilled in the art will be able to design many systems and methods, which, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope.

[0194] Appendix A: Acronyms

[0195] JEM: Joint Exploration Model

[0196] VVC: Next Generation Video Coding

[0197] BMS: Benchmark Set

[0198] MV: Motion Vector

[0199] HEVC: High Efficiency Video Coding

[0200] SEI: Supplemental Enhancement Information

[0201] VUI: Video Availability Information

[0202] GOP: Group of Pictures

[0203] TU: Transform Unit

[0204] PU: prediction unit

[0205] CTU: Coding Tree Unit

[0206] CTB: Coding Tree Block

[0207] PB: prediction block

[0208] HRD: Hypothesized Reference Decoder

[0209] SNR: Signal to Noise Ratio

[0210] CPU: Central Processing Unit

[0211] GPU: Graphics Processing Unit

[0212] CRT: cathode ray tube

[0213] LCD: Liquid Crystal Display

[0214] OLED: Organic Light Emitting Diode

[0215] CD: compact disc

[0216] DVD: Digital Video Disc

[0217] ROM: Read Only Memory

[0218] RAM: Random Access Memory

[0219] ASIC: Application-Specific Integrated Circuit

[0220] PLD: Programmable Logic Device

[0221] LAN: Local Area Network

[0222] GSM: Global System for Mobile Communications

[0223] LTE: Long Term Evolution

[0224] CANBus: Controller Area Network Bus

[0225] USB: Universal Serial Bus

[0226] PCI: Peripheral Component Interconnect

[0227] FPGA: Field Programmable Gate Array

[0228] SSD: Solid State Drive

[0229] IC: Integrated Circuit

[0230] HDR: High Dynamic Range

[0231] SDR: Standard Dynamic Range

[0232] JVET: Joint Video Exploration Team

[0233] MPM: Most Probable Mode

[0234] WAIP: Wide Angle Intra Prediction

[0235] CU: Coding Unit

[0236] PU: prediction unit

[0237] TU: Transform Unit

[0238] CTU: Coding Tree Unit

[0239] PDPC: Position Dependent Prediction Combination

[0240] ISP: Intra-frame sub-partitioning

[0241] SPS: Sequence Parameter Set

[0242] PPS: Picture Parameter Set

[0243] APS: Adaptive Parameter Set

[0244] VPS: Video Parameter Set

[0245] DPS: Decoding Parameter Set

[0246] ALF: Adaptive Loop Filter

[0247] SAO: Sample Adaptive Offset

[0248] CC-ALF: Cross-Component Adaptive Loop Filter

[0249] CDEF: Constrained Directional Enhancement Filter

[0250] CCSO: Cross Component Sample Offset

[0251] LSO: Local Sample Offset

[0252] LR: Loop Restoration Filter

[0253] AV1: AOMedia Video 1

[0254] AV2: AOMedia Video 2

Claims

1. A method for processing video information, characterized in that: include: Identify multiple transform kernels, where: Each of the plurality of transform kernels includes a set of basis vectors from low frequency to high frequency; N high-frequency basis vectors of two or more transform kernels among the plurality of transform kernels are shared, where N is a positive integer; and The low-frequency basis vectors of the two or more transform kernels that are different from the N high-frequency basis vectors are different; extracting a data block from a video bitstream; selecting a transform core from the plurality of transform cores based on information associated with the data block; and The transform kernel is applied to at least a portion of the data block to generate a transformed block.

2. The method according to claim 1, characterized in that: The plurality of transformation kernels are pre-trained offline.

3. The method according to claim 2, characterized in that The plurality of transform kernels are jointly trained offline to determine the shared N high-frequency basis vectors.

4. The method according to claim 1, characterized in that: The N high frequency basis vectors include frequency basis vectors having frequencies higher than a predetermined threshold frequency.

5. The method according to claim 4, characterized in that The plurality of transform kernels and the predetermined threshold frequency are pre-trained offline.

6. The method according to claim 1, characterized in that: The plurality of transform kernels include secondary transform kernels adapted to transform primary transform coefficients; and The data block comprises an array of main transform coefficients.

7. The method according to any one of claims 1 to 6, characterized in that: The two or more transform kernels sharing the N high-frequency basis vectors correspond to the same intra picture prediction mode among a plurality of intra picture prediction modes; and All transform kernels assigned to the same intra picture prediction mode share the N high-frequency basis vectors, and other low-frequency basis vectors are different.

8. The method according to any one of claims 1 to 6, characterized in that The two or more transform kernels sharing the N high frequency basis vectors are assigned to two or more different intra picture prediction modes.

9. The method according to any one of claims 1 to 6, characterized in that: The plurality of transform cores are secondary transform cores; and The two or more transform kernels sharing the N high frequency basis vectors are configured to transform primary transform coefficients generated from a primary transform having the same transform type.

10. The method according to any one of claims 1 to 6, characterized in that N is an integer power of 2.

11. The method according to any one of claims 1 to 6, characterized in that N depends on the transform size.

12. The method according to any one of claims 1 to 6, characterized in that The plurality of transform kernels are secondary low frequency non-separable transform LFNST kernels.

13. The method according to any one of claims 1 to 6, characterized in that The plurality of transform cores are configured for an intra secondary transform IST.

14. The method according to any one of claims 1 to 6, characterized in that The plurality of transform kernels are configured for linear graph transform LGT.

15. A device for processing video information, characterized in that comprising a circuit configured to: Identify multiple transform kernels, where: Each of the plurality of transform kernels includes a set of basis vectors from low frequency to high frequency; N high-frequency basis vectors of two or more transform kernels among the plurality of transform kernels are shared, where N is a positive integer; and The low-frequency basis vectors of the two or more transform kernels that are different from the N high-frequency basis vectors are different; extracting a data block from a video bitstream; selecting a transform core from the plurality of transform cores based on information associated with the data block; and The transform kernel is applied to at least a portion of the data block to generate a transformed block.

16. A non-volatile computer readable medium for storing computer instructions, characterized in that: The computer instructions, when executed by a processor, cause the processor to perform any one of claims 1 to 14.

Images