Video coding and decoding method and device

Abstract

The embodiment of the invention provides a video encoding and decoding method and device. In particular, aspects of the present disclosure provide an apparatus including processing circuitry and a method of video decoding. The processing circuitry decodes encoding information for a block to be reconstructed from an encoded video bitstream, the encoding information indicating intra prediction information for the block. For the block encoded using a directional mode, the processing circuitry determines the directional mode based on a nominal mode and an angular offset, the encoding information indicating the nominal mode and the angular offset. The processing circuitry determines an inseparable transform for the block based on the nominal mode and reconstructs the block based on the directional mode and the inseparable transform. In an example, the processing circuitry determines a transform set pattern associated with the nominal pattern, the transform set pattern indicating a set of one or more inseparable transforms including the inseparable transforms.

Description

[0001] Incorporated herein by reference

[0002] This application claims priority to U.S. Patent Application No. 17 / 031,272, filed on September 24, 2020, entitled “Video Coding Method and Apparatus,” which in turn claims priority to U.S. Provisional Application No. 62 / 941,359, filed on November 27, 2019, entitled “Secondary Transform for Intra-Frame Coding.” The entire disclosure of the prior application is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure describes embodiments generally related to video encoding and decoding. Background Art

[0004] The background description provided herein is intended to generally present the context of the present disclosure. To the extent that the work of the presently named inventors is performed in the background section and in various aspects of this specification, it is not intended that it be prior art at the time of filing this disclosure, and it is neither expressly nor implied that it is admitted to be prior art with respect to the present disclosure.

[0005] Video encoding and decoding can be performed using inter-picture prediction with motion compensation. An uncompressed digital video can include a series of pictures, each picture having spatial dimensions of, for example, 1920×1080 luma samples and associated chroma samples. The series of pictures can have a fixed or variable picture rate (also informally called a frame rate), for example, 60 pictures per second or 60 Hz. Uncompressed video has specific bitrate requirements. For example, 1080p60 4:2:0 video (1920x1080 luma sample resolution at 60 Hz frame rate) with 8 bits per sample requires close to 1.5 Gbit / s of bandwidth. One hour of such video would require over 600 GB of storage space.

[0006] One goal of video encoding and decoding is to reduce redundancy in the input video signal through compression. Compression can help reduce the bandwidth and / or storage space requirements mentioned above, in some cases by two or more orders of magnitude. Both lossless and lossy compression, as well as combinations of the two, can be used. Lossless compression refers to techniques that reconstruct an exact replica of the original signal from a compressed original signal. When lossy compression is used, the reconstructed signal may not be exactly the same as the original signal, but the distortion between the original and the reconstructed signal is small enough that the reconstructed signal can be used for the intended application. Lossy compression is widely used in video. The amount of distortion allowed depends on the application. For example, users of some consumer streaming applications may be able to tolerate higher distortion than users of television applications. The achievable compression ratio reflects the fact that higher allowed / tolerable distortion results in higher compression ratios.

[0007] Video encoders and decoders may utilize several broad categories of techniques including, for example: motion compensation, transforms, quantization, and entropy coding.

[0008] Video codec techniques may include known intra-frame coding techniques. In intra-frame coding, sample values ​​are represented without reference to samples of previously reconstructed reference pictures or other data. In some video codecs, pictures are spatially subdivided into blocks of samples. When all sample blocks are coded or decoded in intra-frame mode, the picture may be an intra-frame picture. Intra-frame pictures and their derivatives (such as independent decoder refresh pictures) can be used to reset the decoder state and can therefore be used as the first picture in a coded video bitstream and video session, or as a still image. The samples of the intra-frame block can be used for transformation, and the transform coefficients can be quantized before entropy coding. Intra-frame prediction can be a technique for minimizing sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after transformation and the smaller the AC coefficient, the fewer bits are required to represent the entropy coded block at a given quantization step size.

[0009] Traditional intra-frame codecs, as known from codecs such as MPEG-2, do not use intra-frame prediction. However, some newer video compression technologies include techniques that attempt to derive data for a block from, for example, surrounding sample data and / or metadata, obtained during the encoding / decoding of spatially neighboring blocks, prior to their decoding order. Such techniques have subsequently become known as "intra-frame prediction" techniques. It is important to note that, at least in some cases, intra-frame prediction uses only reference data from the current picture being reconstructed, not reference data from reference pictures.

[0010] There are many different forms of intra-frame prediction. When more than one such technique is available for use in a given video codec, the techniques used can be coded per intra-frame prediction mode. In some cases, a mode can have sub-modes and / or parameters, and these modes can be coded individually or contained in a mode codeword. Which codeword is used for a given mode / sub-mode / parameter combination affects the codec efficiency gain achieved through intra-frame prediction, and therefore the entropy coding technique used to convert the codewords into the bitstream.

[0011] H.264 introduced an intra prediction mode, which was improved in H.265 and further improved in newer codecs such as the Joint Exploitation Model (JEM), Versatile Video Codec (VVC), and Baseline Video Codec (BMS). A prediction block can be formed using the values ​​of neighboring samples belonging to already available samples. Sample values ​​from neighboring samples are copied into the prediction block in a certain direction. A reference to the direction used can be encoded in the bitstream or can be predicted itself.

[0012] ReferenceFigure 1 , a subset of nine known prediction directions from the 33 possible prediction directions of H.265 (corresponding to the 33 angular modes of the 35 intra modes) is depicted in the lower right. The point where the arrows converge (101) represents the sample being predicted. The arrows represent the direction in which the sample is being predicted. For example, arrow (102) represents the prediction of sample (101) based on one or more samples at the upper right that are at a 45 degree angle to the horizontal. Similarly, arrow (103) represents the prediction of sample (101) based on one or more samples at the lower left that are at a 22.5 degree angle to the horizontal.

[0013] Still refer to Figure 1 , a square block (104) consisting of 4×4 samples is shown in the upper left (indicated by the thick dashed line). The square block (104) consists of 16 samples, each of which is labeled with "S" and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (starting from the top) and the first sample in the X dimension (starting from the left). Similarly, sample S44 is the fourth sample in the block (104) in both the Y and X dimensions. Since the block is 4×4 sized samples, S44 is located in the lower right corner. Reference samples are also shown that follow a similar numbering scheme. Reference samples are labeled with "R" and their Y position (e.g., row index) and X position (e.g., column index) relative to the block (104). In H.264 and H.265, the prediction samples are adjacent to the block being reconstructed, so there is no need to use negative values.

[0014] Intra-picture prediction is performed by copying reference sample values ​​from neighboring samples, where the neighboring samples are determined by a signaled prediction direction. For example, assume that the coded video bitstream includes signaling that, for this block, the prediction direction is consistent with arrow (102), i.e., the sample is predicted based on one or more prediction samples located to the upper right at a 45-degree angle to the horizontal. In this case, samples S41, S32, S23, and S14 are predicted based on the same reference sample R05. Then, sample S44 is predicted based on sample R08.

[0015] In some cases, the values ​​of multiple reference samples may be combined, such as by interpolation, in order to calculate the reference sample, particularly when the direction is not divisible by 45 degrees.

[0016] As video codec technology has advanced, the number of possible directions has increased. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013) and JEM / VVC / BMS, and at the time of this disclosure, up to 65 directions can be supported. Experiments have been conducted to identify the most likely directions, and certain techniques in entropy coding are used to represent those likely directions using a small number of bits, accepting some penalty for less likely directions. Additionally, the direction itself can sometimes be predicted based on neighboring directions used in adjacent, already decoded blocks.

[0017] Figure 2 A schematic diagram (201) is shown, which describes 65 intra prediction directions according to JEM to illustrate the increase in the number of prediction directions over time.

[0018] The mapping of intra-prediction directions to bits representing the directions in the coded video bitstream can vary between different video codecs, ranging from simple direct mapping of prediction directions to codewords for intra-prediction modes to complex adaptive schemes including most probable modes and similar techniques. However, in all cases, there may be certain directions in the video content that are statistically less likely than other directions. Since the goal of video compression is to reduce redundancy, in well-behaved video codecs, those less likely directions will be represented using a larger number of bits than more likely directions. Summary of the Invention

[0019] Various aspects of the present disclosure provide methods and apparatus for video encoding / decoding. In some examples, the apparatus for video encoding and / or decoding includes a processing circuit. The processing circuit encodes / decodes encoding information of a block to be reconstructed from an encoded video bitstream, the encoding information indicating intra-frame prediction information for the block. For the block encoded using a directional mode, the processing circuit determines the directional mode based on a nominal mode and an angular offset, the encoding information indicating the nominal mode and the angular offset. The processing circuit determines an inseparable transform for the block based on the nominal mode, and reconstructs the block based on the directional mode and the inseparable transform.

[0020] In an embodiment, the processing circuit determines a transform set mode associated with the nominal mode, the transform set mode indicating a set of one or more inseparable transforms including the inseparable transform.

[0021] In an embodiment, the encoding information further indicates an inseparable transform index. The processing circuit determines the inseparable transform from the set of one or more inseparable transforms based on the inseparable transform index; and reconstructs the block based on the directional mode and the inseparable transform.

[0022] In an embodiment, the non-separable transform is a non-separable secondary transform. In an example, the non-separable secondary transform is not applied to at least one of a PAETH mode and a recursive filtering mode. In an example, the processing circuit applies the non-separable secondary transform only to the first N transform coefficients along a scan order used to entropy encode a first transform coefficient of the block. In an example, the processing circuit applies the non-separable secondary transform only to the first transform coefficient in the block, each of the first transform coefficients having coordinates (x, y) and a sum of the corresponding x-coordinate and y-coordinate being less than a threshold. In an example, the horizontal transform and the vertical transform in the primary transform of the block are included in a subset of the line map transform set.

[0023] In an example, the non-separable transform is a non-separable secondary transform. The block includes first transform coefficients obtained using the non-separable secondary transform and second transform coefficients obtained without using the non-separable secondary transform. The processing circuitry performs entropy decoding on the first transform coefficients and the second transform coefficients, respectively.

[0024] In an embodiment, the non-directional modes include a DC mode, a PAETH mode, a SMOOTH mode, a SMOOTH_V mode, a SMOOTH_H mode, a recursive filtering mode, and a chrominance prediction from luminance (CfL) mode. The DC mode, the PAETH mode, the SMOOTH mode, the SMOOTH_V mode, and the SMOOTH_H mode are based on an average of adjacent samples of the block. For the block encoded using one of the non-directional modes, the processing circuit determines a set of one or more inseparable transforms associated with one of the non-directional modes. One of the following is associated with the set of one or more inseparable transforms: (a) at least another of the non-directional modes, and (b) a nominal mode. The processing circuit determines an inseparable transform in the set of one or more inseparable transforms based on an inseparable transform index indicated by the encoding information. The processing circuit reconstructs the block based on the non-directional mode and the inseparable transform.

[0025] In an embodiment, one of the non-directional modes, and one of the following: at least another of the non-directional modes and the nominal mode, include one of the following: (a) the recursive filtering mode, and one of the DC mode and the SMOOTH mode; (b) the SMOOTH mode, the SMOOTH_H mode and the SMOOTH_V mode; (c) the SMOOTH mode, the SMOOTH_H mode, the SMOOTH_V mode and the PAETH mode; (d) the recursive filtering mode, the SMOOTH mode and the PAETH mode; (e) a vertical mode for the nominal mode, and the SMOOTH_V mode; (f) a horizontal mode for the nominal mode, and the SMOOTH_H mode; and (v) the CfL mode, and one of the DC mode, the SMOOTH mode and the PAETH mode.

[0026] Aspects of the present disclosure further provide a non-transitory computer-readable medium storing instructions, which, when executed by a computer for video encoding and / or decoding, causes the computer to perform the video encoding and / or decoding method. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Other features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which

[0028] Figure 1 is a diagram illustrating an exemplary subset of intra prediction modes.

[0029] Figure 2 is a schematic diagram of exemplary intra prediction directions.

[0030] Figure 3 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment.

[0031] Figure 4 is a schematic diagram of a simplified block diagram of a communication system according to another embodiment.

[0032] Figure 5 is a schematic diagram of a simplified block diagram of a decoder according to an embodiment.

[0033] Figure 6 is a schematic diagram of a simplified block diagram of an encoder according to an embodiment.

[0034] Figure 7 A block diagram of an encoder according to another embodiment is shown.

[0035] Figure 8 A block diagram of a decoder according to another embodiment is shown.

[0036] Figure 9 An example of a nominal mode of a coding block according to an embodiment of the present disclosure is shown.

[0037] Figure 10 An example of a non-directional smooth intra prediction mode according to aspects of the present disclosure is shown.

[0038] Figure 11 An example of an intra predictor based on recursive filtering according to an embodiment of the present disclosure is shown.

[0039] Figure 12 An example of multi-row intra prediction of a coding block according to an embodiment of the present disclosure is shown.

[0040] Figure 13-14 Two exemplary transform coding processes according to embodiments of the present disclosure are shown.

[0041] Figure 15A An example of line graph transformation according to an embodiment of the present disclosure is shown.

[0042] Figure 15B An exemplary generalized graph Laplacian matrix according to an embodiment of the present disclosure is shown.

[0043] Figure 16 A flow chart illustrating an overview process according to an embodiment of the present disclosure is shown.

[0044] Figure 17 is a schematic diagram of a computer system according to an embodiment. DETAILED DESCRIPTION

[0045] Figure 3 A simplified block diagram of a communication system (300) according to an embodiment of the present disclosure is shown. The communication system (300) includes a plurality of terminal devices that can communicate with each other via, for example, a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). Figure 3 In an embodiment, a first pair of terminal devices (310) and (320) perform unidirectional data transmission. For example, the terminal device (310) may encode video data (e.g., a video picture stream collected by the terminal device (310)) for transmission to another terminal device (320) via the network (350). The encoded video data is transmitted in the form of one or more encoded video streams. The terminal device (320) may receive the encoded video data from the network (350), decode the encoded video data to recover the video data, and display the video picture based on the recovered video data. Unidirectional data transmission is more common in applications such as media services.

[0046] In another embodiment, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of encoded video data, which can occur, for example, during a video conference. For bidirectional data transmission, in one example, each of the terminal devices (330) and (340) can encode video data (e.g., a video picture stream collected by the terminal device) for transmission to the other of the terminal devices (330) and (340) via the network (350). Each of the terminal devices (330) and (340) can also receive the encoded video data transmitted by the other of the terminal devices (330) and (340), and can decode the encoded video data to restore the video data, and can display the video pictures on an accessible display device based on the restored video data.

[0047] exist Figure 3 In an embodiment of the present disclosure, the terminal devices (310), (320), (330) and (340) may be servers, personal computers and smart phones, but the principles of the present disclosure may not be limited thereto. The embodiments of the present disclosure are applicable to laptop computers, tablet computers, media players and / or dedicated video conferencing equipment. The network (350) represents any number of networks that transmit encoded video data between the terminal devices (310), (320), (330) and (340), including, for example, wired (wired) and / or wireless communication networks. The communication network (350) may exchange data in circuit switching and / or packet switching channels. The network may include a telecommunications network, a local area network, a wide area network and / or the Internet. For the purposes of the present disclosure, unless explained below, the architecture and topology of the network (350) may be immaterial to the operation of the present disclosure.

[0048] As an embodiment of the disclosed subject matter, Figure 4 The placement of the video encoder and video decoder in a streaming environment is shown. The disclosed subject matter is equally applicable to other video-enabled applications including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0049] The streaming system may include an acquisition subsystem (413), which may include a video source (401) such as a digital camera, which creates an uncompressed video picture stream (402). In an embodiment, the video picture stream (402) includes samples captured by the digital camera. The video picture stream (402) is depicted as a thick line to emphasize the high data volume of the video picture stream compared to the encoded video data (404) (or encoded video code stream), and the video picture stream (402) can be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination of hardware and software to implement or implement various aspects of the disclosed subject matter as described in more detail below. Compared to the video picture stream (402), the encoded video data (404) (or the encoded video code stream (404)) is depicted as a thin line to emphasize the lower amount of data of the encoded video data (404) (or the encoded video code stream (404)), which can be stored on the streaming server (405) for future use. One or more streaming client subsystems, such as Figure 4 A client subsystem (406) and a client subsystem (408) in a streaming server (405) may access a streaming server (405) to retrieve a copy (407) and a copy (409) of encoded video data (404). The client subsystem (406) may include, for example, a video decoder (410) in an electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and produces an output video picture stream (411) that can be presented on a display (412) (e.g., a display screen) or another presentation device (not depicted). In some streaming systems, the encoded video data (404), the video data (407), and the video data (409) (e.g., a video bitstream) may be encoded according to certain video encoding / compression standards. Examples of such standards include ITU-T H.265. In an embodiment, the video codec standard under development is informally referred to as Versatile Video Coding (VVC), and the disclosed subject matter may be used in the context of the VVC standard.

[0050] It should be noted that the electronic device (420) and the electronic device (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0051] Figure 5A block diagram of a video decoder (510) according to an embodiment of the present disclosure is shown. The video decoder (510) may be provided in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used to replace Figure 4 A video decoder (410) of an embodiment.

[0052] A receiver (531) may receive one or more encoded video sequences to be decoded by a video decoder (510); in the same or another embodiment, one encoded video sequence is received at a time, wherein each encoded video sequence is decoded independently of the other encoded video sequences. The encoded video sequence may be received from a channel (501), which may be a hardware / software link to a storage device storing the encoded video data. The receiver (531) may receive the encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams, which may be forwarded to their respective consuming entities (not shown). The receiver (531) may separate the encoded video sequence from the other data. To prevent network jitter, a buffer memory (515) may be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) is part of the video decoder (510). In other cases, the buffer memory (515) may be provided external to the video decoder (510) (not shown). In other cases, a buffer memory (not shown) is provided external to the video decoder (510) to, for example, mitigate network jitter, and another buffer memory (515) may be provided internally to the video decoder (510) to, for example, handle broadcast timing. Furthermore, when the receiver (531) receives data from a store / forward device with sufficient bandwidth and controllability or from an isochronous network, the buffer memory (515) may not be required, or the buffer memory may be smaller. Of course, for use on a traffic packet network such as the Internet, the buffer memory (515) may also be required. The buffer memory may be relatively large and have an adaptive size, and may be at least partially implemented in an operating system or similar component (not shown) external to the video decoder (510).

[0053] The video decoder (510) may include a parser (520) for reconstructing symbols (521) from the encoded video sequence. The types of symbols include information for managing the operation of the video decoder (510) and potential information for controlling a display device such as a display device (512) (e.g., a display screen) that is not part of the electronic device (530) but can be coupled to the electronic device (530), such as Figure 5As shown in . The control information for the display device may be a parameter set fragment (not shown) of Supplemental Enhancement Information (SEI message) or Video Usability Information (VUI). The parser (520) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may be performed according to a video coding technique or standard and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and the like. The parser (520) may extract a subgroup parameter set for at least one subgroup of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a Group of Pictures (GOP), a picture, a tile, a slice, a macroblock, a Coding Unit (CU), a block, a Transform Unit (TU), a Prediction Unit (PU), and the like. The parser (520) may also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[0054] The parser (520) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515), thereby creating symbols (521).

[0055] Depending on the type of coded video picture or portion of a coded video picture (e.g., inter-frame and intra-frame pictures, inter-frame blocks and intra-frame blocks) and other factors, the reconstruction of the symbol (521) may involve multiple different units. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the coded video sequence by the parser (520). For the sake of brevity, the flow of such subgroup control information between the parser (520) and the multiple units below is not described.

[0056] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually broken down into several functional units as described below. In practical embodiments operating under commercial constraints, many of these units interact closely with each other and may be integrated with each other. However, for the purposes of describing the disclosed subject matter, the conceptual breakdown into the following functional units is appropriate.

[0057] The first unit is a scaler / inverse transform unit (551). The scaler / inverse transform unit (551) receives quantized transform coefficients as symbols (521) from the parser (520) along with control information, including which transform method to use, block size, quantization factor, quantization scaling matrix, etc. The scaler / inverse transform unit (551) may output a block comprising sample values, which may be input to an aggregator (555).

[0058] In some cases, the output samples of the scaler / inverse transform unit (551) may belong to an intra-coded block; that is, a block that does not use predictive information from a previously reconstructed picture, but may use predictive information from a previously reconstructed portion of the current picture. Such predictive information may be provided by the intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) generates a block of the same size and shape as the block being reconstructed using surrounding reconstructed information extracted from the current picture buffer (558). For example, the current picture buffer (558) buffers partially reconstructed current pictures and / or fully reconstructed current pictures. In some cases, the aggregator (555) adds the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a per-sample basis.

[0059] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to inter-frame coded and potentially motion compensated blocks. In this case, the motion compensated prediction unit (553) may access the reference picture memory (557) to extract samples for prediction. After the extracted samples are motion compensated according to the symbols (521), these samples may be added to the output of the scaler / inverse transform unit (551) (in this case referred to as residual samples or residual signal) by the aggregator (555) to generate output sample information. The retrieval of the prediction samples by the motion compensated prediction unit (553) from the address in the reference picture memory (557) may be controlled by a motion vector, and the motion vector is provided to the motion compensated prediction unit (553) in the form of the symbols (521), which may include, for example, X, Y and reference picture components. Motion compensation may also include interpolation of sample values ​​extracted from the reference picture memory (557) when using sub-sample accurate motion vectors, motion vector prediction mechanisms, etc.

[0060] The output samples of the aggregator (555) may be used by various loop filtering techniques in a loop filter unit (556). The video compression techniques may include in-loop filtering techniques that are controlled by parameters included in the coded video sequence (also referred to as the coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520). However, in other embodiments, the video compression techniques may also be responsive to meta-information obtained during decoding of a coded picture or a previous (in decoding order) portion of the coded video sequence, as well as to previously reconstructed and loop-filtered sample values.

[0061] The output of the loop filter unit (556) may be a sample stream that may be output to a display device (512) and stored in a reference picture memory (557) for subsequent inter-picture prediction.

[0062] Once fully reconstructed, certain coded pictures can be used as reference pictures for future prediction. For example, once the coded picture corresponding to the current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557) and a new current picture buffer can be reallocated before starting to reconstruct a subsequent coded picture.

[0063] The video decoder (510) may perform decoding operations according to a predetermined video compression technique, such as that in the ITU-T H.265 standard. The encoded video sequence may conform to the syntax specified by the video compression technology or standard used, in the sense that the encoded video sequence follows the syntax of the video compression technology or standard and the profile recorded in the video compression technology or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technology or standard as the only tools available for use under the profile. For compliance, the complexity of the encoded video sequence is also required to be within the range defined by the hierarchy of the video compression technology or standard. In some cases, the hierarchy limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (measured in, for example, megasamples per second), maximum reference picture size, etc. In some cases, the limits set by the hierarchy may be further defined by the Hypothetical Reference Decoder (HRD) specification and metadata about the HRD buffer management signaled in the encoded video sequence.

[0064] In an embodiment, a receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be part of the encoded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, and the like.

[0065] Figure 6 A block diagram of a video encoder (603) according to an embodiment of the present disclosure is shown. The video encoder (603) is provided in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuit). The video encoder (603) can be used to replace Figure 4 A video encoder (403) in an embodiment.

[0066] The video encoder (603) can be used to generate a video from a video source (601) (not Figure 6 In another embodiment, the video source (601) is a part of the electronic device (620) to receive video samples, and the video source can capture video images to be encoded by the video encoder (603). In another embodiment, the video source (601) is a part of the electronic device (620).

[0067] The video source (601) may provide a source video sequence in the form of a stream of digital video samples to be encoded by the video encoder (603), wherein the stream of digital video samples may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (601) may be a storage device storing previously prepared videos. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures that are given motion when viewed sequentially. The pictures themselves may be constructed as a spatial array of pixels, where each pixel may include one or more samples depending on the sampling structure, color space, etc. used. The relationship between pixels and samples can be readily understood by those skilled in the art. The following description focuses on samples.

[0068] According to an embodiment, the video encoder (603) may encode and compress pictures of a source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing an appropriate encoding speed is a function of the controller (650). In some embodiments, the controller (650) controls other functional units as described below and is functionally coupled to these units. For the sake of simplicity, the coupling is not shown in the figure. The parameters set by the controller (650) may include rate control related parameters (picture skipping, quantizer, lambda value of rate-distortion optimization technology, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) can be used to have other suitable functions that are related to the video encoder (603) optimized for a certain system design.

[0069] In some embodiments, the video encoder (603) operates in a coding loop. As a simplified description, in embodiments, the coding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be encoded and a reference picture) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data (because in the video compression techniques considered in the disclosed subject matter, any compression between the symbols and the encoded video stream is lossless). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Because the decoding of the symbol stream produces bit-accurate results regardless of the decoder's location (local or remote), the contents of the reference picture memory (634) also correspond bit-accurately between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction portion of the encoder are exactly the same sample values ​​that the decoder will "see" when using prediction during decoding. This basic principle of reference picture synchronization (and the resulting drift when synchronization cannot be maintained, eg due to channel errors) is also used in some related techniques.

[0070] The operation of the "local" decoder (633) can be combined with the operation of Figure 5 The "remote" decoder described in detail for the video decoder (510) is identical. However, additional brief reference is made to Figure 5 , when symbols are available and the entropy encoder (645) and parser (520) are capable of losslessly encoding / decoding the symbols into an encoded video sequence, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and the parser (520), may not be fully implemented in the local decoder (633).

[0071] At this point, it can be observed that any decoder technology other than parsing / entropy decoding present in a decoder must also be present in essentially the same functional form in the corresponding encoder. For this reason, the disclosed subject matter focuses on decoder operation. The description of encoder technology can be simplified because encoder technology is the inverse of the decoder technology described in detail. A more detailed description is required only in certain areas and is provided below.

[0072] During operation, in some embodiments, the source encoder (630) may perform motion-compensated predictive coding. Motion-compensated predictive coding predictively encodes an input picture with reference to one or more previously encoded pictures in a video sequence designated as "reference pictures." In this manner, the encoding engine (632) encodes the differences between pixel blocks of the input picture and pixel blocks of a reference picture that may be selected as a prediction reference for the input picture.

[0073] The local video decoder (633) can decode the coded video data that can be designated as a reference picture based on the symbols created by the source encoder (630). The operation of the encoding engine (632) can be a lossy process. When the coded video data is available at the video decoder ( Figure 6 When decoded at a remote location (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process that the video decoder may perform on the reference picture and may cause the reconstructed reference picture to be stored in the reference picture cache (634). In this way, the video encoder (603) may locally store a copy of the reconstructed reference picture that has common content (absent transmission errors) with the reconstructed reference picture that will be obtained by the remote video decoder.

[0074] The predictor (635) may perform a prediction search for the encoding engine (632). That is, for a new picture to be encoded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, etc., that may serve as suitable prediction references for the new picture. The predictor (635) may operate on a pixel-by-pixel-block basis based on sample blocks to find a suitable prediction reference. In some cases, based on the search results obtained by the predictor (635), it may be determined that the input picture may have prediction references taken from multiple reference pictures stored in the reference picture memory (634).

[0075] The controller (650) can manage encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding video data.

[0076] The outputs of all the above functional units may be entropy coded in an entropy encoder (645). The entropy encoder (645) losslessly compresses the symbols generated by the various functional units according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc., thereby converting the symbols into a coded video sequence.

[0077] The transmitter (640) can buffer the encoded video sequence created by the entropy encoder (645) in preparation for transmission over a communication channel (660), which can be a hardware / software link to a storage device where the encoded video data will be stored. The transmitter (640) can combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or an auxiliary data stream (source not shown).

[0078] The controller (650) can manage the operation of the video encoder (603). During encoding, the controller (650) can assign a certain coded picture type to each coded picture, but this may affect the coding techniques that can be applied to the corresponding picture. For example, a picture can generally be assigned to any of the following picture types:

[0079] An intra picture (I picture) can be a picture that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are aware of the variations of I pictures and their corresponding applications and features.

[0080] A predictive picture (P picture) may be a picture that can be encoded and decoded using intra prediction or inter prediction, which uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0081] Bidirectionally predictive pictures (B pictures) can be encoded and decoded using intra prediction or inter prediction, which uses up to two motion vectors and reference indices to predict sample values ​​for each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata to reconstruct a single block.

[0082] A source picture is typically spatially subdivided into blocks of samples (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples) and coded block by block. These blocks may be predictively coded with reference to other (already coded) blocks, determined according to the coding allocation applied to the block's corresponding picture. For example, blocks of an I picture may be non-predictively coded, or they may be predictively coded (spatial or intra-predicted) with reference to already coded blocks of the same picture. Pixel blocks of a P picture may be predictively coded using spatial prediction with reference to one previously coded reference picture or using temporal prediction. Blocks of a B picture may be predictively coded using spatial prediction with reference to one or two previously coded reference pictures or using temporal prediction.

[0083] The video encoder (603) may perform encoding operations according to a predetermined video coding technique or standard, such as ITU-T Recommendation H.265. In operation, the video encoder (603) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to the syntax specified by the video coding technique or standard used.

[0084] In an embodiment, the transmitter (640) may transmit additional data along with the encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, redundant pictures and slices, and other forms of redundant data, such as supplemental enhancement information (SEI) messages, and visual usability information (VUI) parameter set segments.

[0085] The captured video may be presented as a temporal sequence of multiple source pictures (video pictures). Intra-picture prediction (often shortened to intra prediction) exploits spatial correlation within a given picture, while inter-picture prediction exploits correlation (temporal or otherwise) between pictures. In an embodiment, a particular picture being encoded / decoded is divided into blocks, and the particular picture being encoded / decoded is referred to as the current picture. When a block in the current picture is similar to a reference block in a previously encoded and buffered reference picture in the video, the block in the current picture can be encoded using a vector called a motion vector. The motion vector points to the reference block in a reference picture, and when multiple reference pictures are used, the motion vector may have a third dimension that identifies the reference picture.

[0086] In some embodiments, bidirectional prediction techniques can be used for inter-picture prediction. According to bidirectional prediction techniques, two reference pictures are used, for example, a first reference picture and a second reference picture, both preceding the current picture in the video in decoding order (but potentially in the past and future, respectively, in display order). A block in the current picture can be encoded using a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. Specifically, the block can be predicted using a combination of the first and second reference blocks.

[0087] In addition, merge mode technology can be used in inter-picture prediction to improve coding efficiency.

[0088] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, according to the HEVC standard, pictures in a video picture sequence are divided into coding tree units (CTUs) for compression, and the CTUs in the pictures have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally speaking, a CTU includes three coding tree blocks (CTBs), which are a luminance CTB and two chrominance CTBs. Furthermore, each CTU can be split into one or more coding units (CUs) using a quadtree. For example, a 64×64 pixel CTU can be split into a 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In an embodiment, each CU is analyzed to determine the prediction type used for the CU, such as an inter prediction type or an intra prediction type. In addition, depending on temporal and / or spatial predictability, the CU is split into one or more prediction units (PUs). Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In an embodiment, the prediction operation in encoding (encoding / decoding) is performed in units of prediction blocks. Taking the luma prediction block as an example, the prediction block includes a matrix of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, and the like.

[0089] Figure 7 A schematic diagram of a video encoder (703) according to another embodiment of the present disclosure is shown. The video encoder (703) is configured to receive a processed block (e.g., a prediction block) of sample values ​​within a current video picture in a video picture sequence and to encode the processed block into an encoded picture that is part of an encoded video sequence. In this embodiment, the video encoder (703) is configured to replace Figure 4A video encoder (403) in an embodiment.

[0090] In an HEVC embodiment, a video encoder (703) receives a matrix of sample values ​​for a processing block, such as a prediction block of 8×8 samples. The video encoder (703) uses, for example, rate-distortion optimization to determine whether to use intra mode, inter mode, or bi-prediction mode to encode the processing block. When encoding the processing block in intra mode, the video encoder (703) may use intra prediction techniques to encode the processing block into an encoded picture; and when encoding the processing block in inter mode or bi-prediction mode, the video encoder (703) may use inter prediction or bi-prediction techniques, respectively, to encode the processing block into an encoded picture. In some video coding techniques, merge mode may be an inter-picture prediction submode, in which motion vectors are derived from one or more motion vector predictors without resorting to coded motion vector components external to the predictor. In some other video coding techniques, there may be motion vector components applicable to the subject block. In an embodiment, the video encoder (703) includes other components, such as a mode decision module (not shown) for determining the processing block mode.

[0091] exist Figure 7 In an embodiment of the present invention, the video encoder (703) includes Figure 7 Shown are an inter-frame encoder (730), an intra-frame encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together.

[0092] The inter-frame encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the current block with one or more reference blocks in a reference picture (e.g., blocks in a previous picture and a subsequent picture), generate inter-frame prediction information (e.g., redundant information description according to an inter-frame coding technique, motion vectors, merge mode information), and calculate an inter-frame prediction result (e.g., a predicted block) based on the inter-frame prediction information using any suitable technique. In some embodiments, the reference picture is a decoded reference picture decoded based on the encoded video information.

[0093] The intra-frame encoder (722) is configured to receive samples of a current block (e.g., a processing block), compare the block with previously encoded blocks in the same picture in some cases, generate quantized coefficients after transformation, and in some cases also generate intra-frame prediction information (e.g., intra-frame prediction direction information based on one or more intra-frame coding techniques). In an embodiment, the intra-frame encoder (722) further calculates an intra-frame prediction result (e.g., a predicted block) based on the intra-frame prediction information and a reference block in the same picture.

[0094] The general controller (721) is used to determine general control data and control other components of the video encoder (703) based on the general control data. In an embodiment, the general controller (721) determines the mode of the block and provides a control signal to the switch (726) based on the mode. For example, when the mode is intra mode, the general controller (721) controls the switch (726) to select the intra mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select intra prediction information and add the intra prediction information to the bitstream; and when the mode is inter mode, the general controller (721) controls the switch (726) to select the inter prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select inter prediction information and add the inter prediction information to the bitstream.

[0095] The residual calculator (723) is used to calculate the difference (residual data) between the received block and the prediction result selected from the intra-frame encoder (722) or the inter-frame encoder (730). The residual encoder (724) is used to operate based on the residual data to encode the residual data to generate transform coefficients. In an embodiment, the residual encoder (724) is used to convert the residual data from the spatial domain to the frequency domain and generate transform coefficients. The transform coefficients are then quantized to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is used to perform an inverse transform and generate decoded residual data. The decoded residual data can be used appropriately by the intra-frame encoder (722) and the inter-frame encoder (730). For example, the inter-frame encoder (730) can generate a decoded block based on the decoded residual data and inter-frame prediction information, and the intra-frame encoder (722) can generate a decoded block based on the decoded residual data and intra-frame prediction information. The decoded blocks are appropriately processed to generate a decoded picture, and in some embodiments, the decoded picture may be buffered in a memory circuit (not shown) and used as a reference picture.

[0096] The entropy encoder (725) is used to format the codestream to produce encoded blocks. The entropy encoder (725) generates various information according to a suitable standard, such as the HEVC standard. In an embodiment, the entropy encoder (725) is used to obtain general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information from the codestream. It should be noted that according to the disclosed subject matter, when encoding a block in inter-frame mode or the merge sub-mode of bidirectional prediction mode, there is no residual information.

[0097] Figure 8FIG2 shows a schematic diagram of a video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed picture. In an embodiment, the video decoder (810) is configured to replace Figure 4 A video decoder (410) of an embodiment.

[0098] exist Figure 8 In one embodiment, the video decoder (810) includes Figure 8 , an entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-frame decoder (872) coupled together are shown.

[0099] The entropy decoder (871) can be used to reconstruct certain symbols from the encoded picture, which represent syntax elements that constitute the encoded picture. Such symbols may include, for example, the mode used to encode the block (e.g., intra mode, inter mode, bidirectional prediction mode, a combined submode of the latter two, or another submode), prediction information (e.g., intra prediction information or inter prediction information) that can identify certain samples or metadata for prediction by the intra decoder (872) or the inter decoder (880), respectively, residual information in the form of, for example, quantized transform coefficients, etc. In an embodiment, when the prediction mode is inter or bidirectional prediction mode, the inter prediction information is provided to the inter decoder (880); and when the prediction type is intra prediction type, the intra prediction information is provided to the intra decoder (872). The residual information may be inverse quantized and provided to the residual decoder (873).

[0100] The inter-frame decoder (880) is configured to receive inter-frame prediction information and generate an inter-frame prediction result based on the inter-frame prediction information.

[0101] The intra-frame decoder (872) is configured to receive intra-frame prediction information and generate a prediction result based on the intra-frame prediction information.

[0102] The residual decoder (873) is used to perform inverse quantization to extract dequantized transform coefficients and process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (873) may also require certain control information (to obtain the quantizer parameter QP), and this information can be provided by the entropy decoder (871) (the data path is not shown because this is only low-level control information).

[0103] The reconstruction module (874) is used to combine the residual output by the residual decoder (873) with the prediction result (which may be output by the inter-frame prediction module or the intra-frame prediction module) in the spatial domain to form a reconstructed block. The reconstructed block can be part of a reconstructed picture, which in turn can be part of a reconstructed video. It should be noted that other suitable operations such as deblocking can be performed to improve visual quality.

[0104] It should be noted that the video encoder (403), video encoder (603), and video encoder (703), as well as the video decoder (410), video decoder (510), and video decoder (810) may be implemented using any suitable technology. In one embodiment, the video encoder (403), video encoder (603), and video encoder (703), as well as the video decoder (410), video decoder (510), and video decoder (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoder (403), video encoder (603), and video encoder (703), as well as the video decoder (410), video decoder (510), and video decoder (810) may be implemented using one or more processors executing software instructions.

[0105] Video codec techniques related to inseparable transforms, such as for primary transforms and / or secondary transforms, for intra-frame coding are disclosed. The inseparable transforms can be applied to any suitable video codec format or standard. The video codec format may include an open video codec format designed for video transmission over the Internet, such as AOMedia Video 1 (AV1) or a next-generation AOMedia video format that exceeds AV1. Video codec standards may include the High Efficiency Video Codec (HEVC) standard, a next-generation video codec that exceeds HEVC (e.g., Versatile Video Codec (VVC)), etc.

[0106] In intra prediction, for example in AV1, VVC, etc., various intra prediction modes may be used. In an embodiment, for example in AV1, directional intra prediction is used. In an example, for example in the open video codec format VP9, ​​eight directional modes correspond to eight angles from 45° to 207°. In order to exploit a wider variety of spatial redundancies in directional textures, for example in AV1, directional modes (also called directional intra modes, directional intra prediction modes, angle modes) may be extended to a set of angles with a finer granularity, such as Figure 9 shown.

[0107] Figure 9An example of a nominal mode of a coding block (CB) (910) according to an embodiment of the present disclosure is shown. Certain angles (referred to as nominal angles) may correspond to nominal modes. In the example, eight nominal angles (or nominal intra-frame angles) (901)-(908) correspond to eight nominal modes (e.g., V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED). The eight nominal angles (901)-(908) and the eight nominal modes may be referred to as V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED, respectively. In addition, each nominal angle may correspond to a plurality of finer angles (e.g., seven finer angles), and thus, for example, in AV1, 56 angles (or prediction angles) or 56 directional modes (or angle modes, directional intra prediction modes) may be used. Each prediction angle may be represented by a nominal angle and an angle offset (or angle increment). The angle offset may be obtained by multiplying an offset integer 1 (e.g., -3, -2, -1, 0, 1, 2, or 3) by a step size (e.g., 3°). In an example, the prediction angle is equal to the sum of the nominal angle and the angle offset. In an example, for example, in AV1, nominal modes (e.g., eight nominal modes (901)-(908)) and certain non-angle smoothing modes (e.g., DC mode, PAETH mode, SMOOTH mode, vertical SMOOTH mode, and horizontal SMOOTH mode as described below) may be signaled. Subsequently, if the current prediction mode is a directional mode (or angle mode), an index may be further signaled to indicate the angle offset corresponding to the nominal angle (e.g., offset integer 1). In the example, in order to implement directional prediction modes in a common way, the 56 directional modes such as those used in AV1 are implemented using a unified directional predictor that projects each pixel to a reference sub-pixel position and interpolates the reference pixel through a 2-tap bilinear filter.

[0108] A non-directional smooth intra predictor (also referred to as a non-directional smooth intra prediction mode, a non-directional smooth mode, a non-angular smooth mode) may be used for intra prediction of a CB. In some examples (e.g., in AV1), the five non-directional smooth intra prediction modes include a DC mode or a DC predictor (e.g., DC), a PAETH mode or a PAETH predictor (e.g., PAETH), a SMOOTH mode or a SMOOTH predictor (e.g., SMOOTH), a vertical SMOOTH mode (referred to as a SMOOTH_V mode, a SMOOTH_V predictor, SMOOTH_V), and a horizontal SMOOTH mode (referred to as a SMOOTH_H mode, a SMOOTH_H predictor, or SMOOTH_H).

[0109] Figure 10 An example of non-directional smooth intra prediction modes (e.g., DC mode, PAETH mode, SMOOTH mode, SMOOTH_V mode, and SMOOTH_H mode) according to aspects of the present disclosure is shown. To predict a sample (1001) in a CB (1000) based on a DC predictor, an average of a first value of a left neighboring sample (1012) and a second value of an above neighboring sample (or top neighboring sample) (1011) may be used as a prediction value.

[0110] To predict sample (1001) based on the PAETH predictor, the first value of the left neighboring sample (1012), the second value of the top neighboring sample (1011), and the third value of the top-left neighboring sample (1013) can be obtained. Then, the reference value is obtained using Equation 1.

[0111] Reference value = first value + second value - third value (Equation 1)

[0112] One of the first value, the second value, and the third value closest to the reference value may be set as the predicted value of the sample (1001).

[0113] SMOOTH_V mode, SMOOTH_H mode, and SMOOTH mode can predict CB (1000) using quadratic interpolation in the vertical direction, the horizontal direction, and the average of the vertical and horizontal directions, respectively. To predict sample (1001) based on the SMOOTH predictor, an average value (e.g., a weighted combination) of the first value, the second value, the value of the right sample (1014), and the value of the bottom sample (1016) can be used. In various examples, the right sample (1014) and the bottom sample (1016) are not reconstructed, and therefore, the value of the upper right adjacent sample (1015) and the value of the lower left adjacent sample (1017) can replace the values ​​of the right sample (1014) and the bottom sample (1016), respectively. Therefore, the average value (e.g., a weighted combination) of the first value, the second value, the value of the upper right adjacent sample (1015), and the value of the lower left adjacent sample (1017) can be used as the SMOOTH predictor. To predict the sample (1001) based on the SMOOTH_V predictor, the average of the second value of the top neighboring sample (1011) and the value of the bottom left neighboring sample (1017) may be used (e.g., a weighted combination). To predict the sample (1001) based on the SMOOTH_H predictor, the average of the first value of the left neighboring sample (1012) and the value of the top right neighboring sample (1015) may be used (e.g., a weighted combination).

[0114] Figure 11 An example of a recursive filtering-based intra predictor (also referred to as a filter intra mode or recursive filtering mode) according to an embodiment of the present disclosure is shown. In order to capture the attenuated spatial correlation with the reference on the edge, the filter intra mode can be used for CB (1100). In the example, CB (1100) is a luma block. The luma block (1100) can be divided into multiple patches (e.g., eight 4×2 patches B0-B7). Each of the patches B0-B7 can have multiple adjacent samples. For example, patch B0 has seven adjacent samples (or seven neighbors) R00-R06, including four top adjacent samples R01-R04, two left adjacent samples R05-R06, and one upper left adjacent sample R00. Similarly, patch B7 has seven adjacent samples R70-R76, including four top adjacent samples R71-R74, two left adjacent samples R75-R76, and one upper left adjacent sample R70.

[0115] In some examples, multiple (e.g., five) filter intra modes (or multiple recursive filtering modes) are pre-designed, for example, for AV1. Each filter intra mode can be represented by a set of eight 7-tap filters that reflect the correlation between samples (or pixels) in the corresponding 4×2 patch (e.g., B0) and seven neighbors (e.g., R00-R06) adjacent to the 4×2 patch B0. The weighting factors of the 7-tap filters can be position-dependent. For each of the patches B0-B7, seven neighbors (e.g., R00-R06 for B0, R70-R76 for B7) can be used to predict samples in the corresponding patch. In the example, neighbors R00-R06 are used to predict samples in patch B0. In the example, neighbors R70-R76 are used to predict samples in patch B7. For some patches in CB (1100) (e.g., patch B0), all seven neighbors (e.g., R00-R06) have been reconstructed. For other patches in CB (1100), at least one of the seven neighbors is not reconstructed, so the predicted values ​​of one or more direct neighbors (or the predicted samples of one or more direct neighbors) can be used as references. For example, the seven neighbors R70-R76 of patch B7 are not reconstructed, so the predicted samples of the direct neighbors can be used.

[0116] Chroma samples can be predicted from luma samples. In an embodiment, a chroma from luma mode (e.g., CfL mode, CfL predictor) is a chroma-only intra-frame predictor that can model chroma samples (or pixels) as linear functions of simultaneously reconstructed luma samples (or pixels). For example, CfL prediction can be expressed using the following equation 2.

[0117] CfL(α)=αL A +D (Equation 2)

[0118] Among them, L A represents the AC contribution of the luma component, α represents the scaling parameter of the linear model, and D represents the DC contribution of the chroma component. In the example, the reconstructed luma pixels are subsampled based on the chroma resolution and the average value is subtracted to form the AC contribution (e.g., L A ). To approximate the chroma AC components based on their AC contributions, in some examples, such as AV1, the decoder is not required to calculate the scaling parameter α. Instead, the CfL mode determines the scaling parameter α based on the original chroma pixels and signals the scaling parameter α in the bitstream, thereby reducing decoder complexity and producing more accurate predictions. The intra-frame DC mode can be used to calculate the DC contribution of the chroma components. The intra-frame DC mode can meet the needs of most chroma content and has a mature and fast implementation.

[0119] Multi-line intra prediction can use more reference lines for intra prediction. A reference line can include multiple samples in a picture. In an example, a reference line includes samples in a row and samples in a column. In an example, the encoder can determine and signal a reference line for generating an intra predictor. An index indicating a reference line (also called a reference line index) can be signaled before the intra prediction mode. In an example, only MPM is allowed when a non-zero reference line index is signaled. Figure 12 An example of four reference lines of CB (1210) is shown. Figure 12 , a reference line can include up to six segments (e.g., segments A to F) and an upper-left reference sample. For example, reference line 0 includes segments B and E and an upper-left reference sample. For example, reference line 3 includes segments A to F and an upper-left reference sample. Segments A and F can be padded with the closest samples from segments B and E, respectively. In some examples, such as in HEVC, only one reference line (e.g., reference line 0 adjacent to CB (1210)) is used for intra prediction. In some examples, such as in VVC, multiple reference lines (e.g., reference lines 0, 1, and 3) are used for intra prediction.

[0120] Transforms such as primary transforms and secondary transforms may be applied to CBs. Transforms may be non-separable or separable. According to aspects of the present disclosure, a transform (e.g., primary transform, secondary transform) may be a non-separable transform. The application of a non-separable transform may be described as follows using a 4×4 input block (or input matrix) X as an example (as shown in Equation 3). To apply a 4×4 non-separable transform, the 4×4 input block X may be represented by the vector Expressed as shown in Equation 3-4.

[0121]

[0122] The non-separable transformation can be calculated as in, represents the transform coefficient vector, and T is the 16×16 transform matrix. The 4×4 input block can be reorganized into a 4×4 output block (or output matrix, coefficient block) using the scanning order (e.g., horizontal scanning order, vertical scanning order, zigzag scanning order, or diagonal scanning order) of the input block. Transform coefficients with smaller indices can be placed in the 4×4 coefficient block with smaller scanning indices.

[0123] According to aspects of the present disclosure, a reduced non-separable transform may be used in a primary transform. A reduced non-separable transform may be used in a secondary transform, such as a low-frequency non-separable transform (LFNST).

[0124] A non-separable secondary transform can be applied to CB. In some examples, such as in VVC, Figure 13-14 As shown, LFNST is applied between the forward primary transform and quantization (eg, at the encoder side) and between dequantization and the inverse primary transform (eg, at the decoder side). In the example, LFNST is called a reduced secondary transform (RST).

[0125] Figure 13-14 Examples of two transform coding processes (1300) and (1400) using a 16x64 transform (or a 64x16 transform, depending on whether the transform is a forward quadratic transform or an inverse quadratic transform) and a 16x48 transform (or a 48x16 transform, depending on whether the transform is a forward quadratic transform or an inverse quadratic transform) are shown, respectively. Figure 13 , in process (1300), on the encoder side, a forward primary transform (1310) may first be performed on a block (e.g., a residual block) to obtain a coefficient block (1313). Subsequently, a forward secondary transform (or forward LFNST) (1312) may be applied to the coefficient block (1313). In the forward secondary transform (1312), the 64 coefficients of the 4×4 sub-block AD at the upper left corner of the coefficient block (1313) may be represented by a 64-length vector, which may be multiplied by a transform matrix of 64×16 (i.e., width 64 and height 16) to obtain a 16-length vector. The elements in the 16-length vector are filled back into the upper left 4×4 sub-block A of the coefficient block (1313). The coefficients in the sub-block BD may be zero. The coefficients obtained after the forward secondary transform (1312) are then quantized in step (1314) and entropy coded to generate coded bits in the bitstream (1316).

[0126] The coded bits may be received at the decoder side and entropy decoded and then dequantized (1324) to generate a coefficient block (1323). An inverse secondary transform (or inverse LFNST) (1322), such as inverse RST 8×8, may be performed to obtain 64 coefficients, for example, from the 16 coefficients at the top left 4×4 sub-block E. The 64 coefficients may be filled back into the 4×4 sub-block EH. Furthermore, the coefficients in the coefficient block (1323) after the inverse secondary transform (1322) may be processed with an inverse primary transform (1320) to obtain a recovered residual block.

[0127] Figure 14 The exemplary process (1400) is similar to the process (1300), except that fewer coefficients (i.e., 48) are processed during the forward second transformation (1412). Specifically, 48 coefficients in sub-blocks A-C are processed using a smaller transformation matrix of size 48×16. Using a smaller transformation matrix of 48×16 can reduce the memory size for storing the transformation matrix and reduce a large amount of calculations (e.g., multiplication, addition, subtraction, etc.), thus reducing the computational complexity.

[0128] In the example, a 4×4 non-separable transform (e.g., 4×4 LFNST) or an 8×8 non-separable transform (e.g., 8×8 LFNST) is applied according to the block size of CB. The block size of CB can include width, height, etc. For example, 4×4 LFNST is applicable to CBs where the minimum of the width and height is less than a threshold, e.g., the threshold is 8 (e.g., min(width, height) < 8). For example, 8×8 LFNST is applicable to CBs where the minimum of the width and height is greater than a threshold, e.g., the threshold is 4 (e.g., min(width, height) > 4).

[0129] The non-separable transform (e.g., LFNST) can be based on the direct matrix multiplication method, and thus can be implemented in a single pass without iteration. To reduce the dimension of the non-separable transform matrix and minimize the computational complexity and the storage space for storing the transform coefficients, a reduced non-separable transform method (or RST) can be used in LFNST. Thus, in the reduced non-separable transform, an N-dimensional vector (e.g., for an 8×8 non-separable second transform (NSST), N is 64) can be mapped to an R-dimensional vector in a different space, where N / R (R < N) is the reduction factor. Therefore, the RST matrix is not an N×N matrix, but an R×N matrix as described in Equation 5.

[0130]

[0131] In Equation 5, the R rows of the R×N transform matrix are the R bases of the N-dimensional space. The inverse transform matrix can be the transpose of the transform matrix used in the forward transform (e.g., T RxN )). For 8×8 LFNST, a reduction factor equal to 4 can be applied, and the 64×64 direct matrix used in the 8×8 non-separable transform can be reduced to a 16×64 direct matrix, as Figure 13 shown. Alternatively, a reduction factor greater than 4 can be applied, and the 64×64 direct matrix used in the 8×8 non-separable transform can be reduced to a 16×48 direct matrix, as Figure 14Therefore, the 48×16 inverse RST matrix can be used at the decoder side to generate the core (main) transform coefficients in the 8×8 upper left region.

[0132] refer to Figure 14 , when applying a 16×48 matrix instead of a 16×64 matrix with the same transform set configuration, the input of the 16×48 matrix includes 48 input data from the three 4×4 blocks A, B, and C in the upper left 8×8 block (excluding the lower right 4×4 block D). As the dimensionality is reduced, the memory usage for storing the LFNST matrix can be reduced, for example, from 10KB to 8KB, with minimal performance degradation.

[0133] To reduce complexity, LFNST can be restricted to be applicable if coefficients outside the first coefficient subgroup are non-significant. In an example, LFNST can be restricted to be applicable only when all coefficients outside the first coefficient subgroup are non-significant. Figure 13-14 , the first coefficient subgroup corresponds to the upper left block E, so the coefficients outside block E are invalid.

[0134] In an example, when LFNST is applied, only the primary transform coefficients are invalid (e.g., zero). The primary transform coefficients alone may refer to transform coefficients obtained from a primary transform without a secondary transform. Therefore, LFNST index signaling may be conditioned on the last significant position, thereby avoiding additional coefficient scanning in LFNST. In some examples, additional coefficient scanning is used to check for significant transform coefficients at specific positions. In an example, for example, the worst-case processing of LFNST, in terms of per-pixel multiplication, limits the inseparable transforms for 4×4 blocks and 8×8 blocks to 8×16 and 8×48 transforms, respectively. In these cases, when LFNST is applied, the last significant scan position may be less than 8. For other sizes, when LFNST is applied, the last significant scan position may be less than 16. For 4×N and N×4 CBs, where N is greater than 8, this restriction may mean that LFNST is applied to the top-left 4×4 region in the CB. In an example, this restriction means that LFNST is applied only once to the top-left 4×4 region in the CB. In this example, when LFNST is applied, all main coefficients are invalid (e.g., zero), reducing the number of main transform operations. From the encoder's perspective, when testing the LFNST transform, the quantization of the transform coefficients can be significantly simplified. For the first 16 coefficients, for example, rate-distortion optimized quantization can be performed to the maximum extent possible in scan order, and the remaining coefficients can be set to zero.

[0135] An LFNST transform (e.g., a transform kernel or transform matrix) may be selected as described below. In an embodiment, multiple transform sets may be used, and each of the multiple transform sets in the LFNST may include one or more non-separable transform matrices (or kernels). According to aspects of the present disclosure, a transform set may be selected from the multiple transform sets, and a non-separable transform matrix may be selected from the one or more non-separable transform matrices in the transform set.

[0136] Table 1 shows an exemplary mapping from intra prediction modes to multiple transform sets according to an embodiment of the present disclosure. The mapping indicates the relationship between the intra prediction modes and the multiple transform sets. The relationship shown in Table 1 can be predefined and can be stored in the encoder and decoder.

[0137] Referring to Table 1, the plurality of transform sets includes four transform sets, for example, transform sets 0 to 3 represented by transform set indices (e.g., Tr.set index) 0 to 3. The index (e.g., IntraPredMode) may indicate an intra prediction mode, and the transform set index may be obtained based on the index and Table 1. Thus, the transform set may be determined based on the intra prediction mode. In an example, if one of three cross component linear model (CCLM) modes (e.g., INTRA_LT_CCLM, INTRA_T_CCLM, or INTRA_L_CCLM) is used for a CB (e.g., 81 <= IntraPredMode <= 83), transform set 0 is selected for the CB.

[0138] As described above, each transform set may include one or more inseparable transform matrices. One of the one or more inseparable transform matrices can be selected by an explicitly signaled LFNST index. For example, after signaling the transform coefficients, the LFNST index may be signaled once in the bitstream for each intra-coded CU (e.g., CB). In an embodiment, each transform set includes two inseparable transform matrices (kernels), and the selected inseparable secondary transform candidate may be one of the two inseparable transform matrices. In some examples, LFNST is not applied to the CB (e.g., a CB encoded using a transform skip mode or a plurality of non-zero coefficients of the CB is less than a threshold). In the example, when LFNST is not applied to the CB, the LFNST index is not signaled for the CB. The default value of the LFNST index may be zero and not signaled, indicating that LFNST is not applied to the CB.

[0139] Table 1: Transformation set selection table

[0140] IntraPredMode Transform Set Index IntraPredMode<0 1 0<=IntraPredMode<=1 0 2<=IntraPredMode<=12 1 13<=IntraPredMode<=23 2 24<=IntraPredMode<=44 3 45<=IntraPredMode<=55 2 56<=IntraPredMode<=80 1 81<=IntraPredMode<=83 0

[0141] In an embodiment, LFNST is restricted to being applicable only when all coefficients outside the first coefficient subgroup are invalid, and the encoding of the LFNST index may depend on the position of the last significant coefficient. The LFNST index may be context-coded. In an example, the context encoding of the LFNST index is independent of the intra prediction mode, and only the first bin is context-coded. LFNST may be applied to intra-coded CUs in intra or inter slices, and to both luma and chroma components. If a dual tree is enabled, the LFNST indexes for luma and chroma components may be signaled separately. For inter slices (e.g., if the binary tree is disabled), a single LFNST index may be signaled and applied to both luma and chroma components.

[0142] Intra sub-partition (ISP) coding mode can be used. In ISP coding mode, the luma intra prediction block can be divided vertically or horizontally into 2 or 4 sub-partitions depending on the block size. In some examples, when RST is applied to each feasible sub-partition, the performance improvement is negligible. Therefore, in some examples, when ISP mode is selected, LFNST is disabled and the LFNST index (or RST index) is not signaled. Disabling RST or LFNST for the residual of ISP prediction can reduce the complexity of encoding and decoding. In some examples, when matrix-based intra prediction mode (MIP) is selected, LFNST is disabled and the LFNST index is not signaled.

[0143] In some examples, due to the maximum transform size limit (e.g., 64×64), CUs larger than 64×64 are implicitly split (TU tiling), and for a certain number of decoding pipeline stages, LFNST index searches can increase the data buffer by a factor of four. Therefore, the maximum size allowed for LFNST can be limited to 64×64. In the example, LFNST is enabled only with discrete cosine transform (DCT) type 2 (DCT-2) transform.

[0144] For example, a line graph transform (LGT) may be used for transforms such as the main transform. In an example, LGT includes various DCTs, discrete sine transforms (DSTs), as described below. LGTs may include 32-point and 64-point one-dimensional (1D) DSTs.

[0145] A graph is a general mathematical structure consisting of a set of vertices and edges that can be used to model close relationships between objects of interest. A weighted graph that assigns a set of weights to edges and optionally to vertices can provide a sparse representation for robust modeling of signals / data. LGT can improve encoding and decoding efficiency by better adapting to different block statistics. Separable LGT can be designed and optimized by learning line graphs from data to model the underlying row and column-wise statistics of the residual signal of the block, and the related generalized graph Laplacian (GGL) matrix can be used to derive the LGT.

[0146] Figure 15A The self-loop weights (e.g., v c1 、v c2 ) and edge weight w c Example of a general LGT for characterization. Given a weighted graph G(W,V), the GGL matrix can be defined as follows.

[0147] L c =D-W+V (Equation 6)

[0148] Where W can be a non-negative edge weight w c The adjacency matrix, D can be a diagonal matrix, V can be a self-loop weight v c1 and v c2 The diagonal matrix of . Figure 15B The matrix L is shown c .

[0149] LGT can be obtained by GGL matrix L c The Eigen-decomposition of is derived as follows.

[0150] L c =UΦU T (Equation 7)

[0151] Here, the columns of the orthogonal matrix U can be the basis vectors of LGT, and Φ can be the diagonal eigenvalue matrix.

[0152] In various examples, some DCTs and DSTs (e.g., DCT-2, DCT-8, and DST-7) are subsets of the set of LGTs derived from some forms of GGL. DCT-2 can be obtained by converting v c1 Set to 0 (for example, v c1 =0). DST-7 can be derived by c1 Set to w c (For example,c1 =w c ) can be derived. DCT-8 can be derived by converting v c2 Set to w c (For example, c2 =w c ) can be derived. DST-4 can be derived by adding v c1 Set to 2w c (For example, c1 =2w c ) can be derived. DCT-4 can be derived by converting v c2 Set to 2w c (For example, c2 =2w c ) to export.

[0153] In some examples, LGT can be implemented as a matrix multiplication. The 4p LGT core can be implemented by converting v c1 Set to 2w c To derive, the 4p LGT core is DST-4. The 8p LGT core can be obtained by replacing v in Lc c1 Set to 1.5w c In this example, the LGT core can be exported by c1 Set to w c And v c2 Set to 0 to export, and the LGT core can become DST-7.

[0154] In some examples, such as AV1, the transform scheme is only separable and may not be efficient for capturing certain directional texture patterns (e.g., certain edges along a 45° direction). To develop more efficient video and image codecs (e.g., advanced video and image codecs beyond AV1 intra-frame coding), non-separable transforms can be used to improve codec efficiency for certain directional image patterns, for example.

[0155] In some examples, a non-separable transform is used in VVC. The secondary transform scheme in VVC (e.g., a non-separable secondary transform scheme) may rely on the intra prediction scheme. However, the intra prediction schemes of AV1 and VVC may be different. Therefore, the secondary transform scheme in VVC may not be fully compatible with the intra coding scheme of AV1. Therefore, in order to apply the AV1-based secondary transform in, for example, the next-generation AOMedia video format that goes beyond AV1, further changes are applied, as described in this disclosure.

[0156] Various embodiments and examples may be used alone or in combination in any order. In the present disclosure, the intra prediction mode for generating prediction samples according to the prediction direction may be referred to as an angle mode (or angle intra prediction mode) or a direction mode (or directional intra prediction mode), such as the one described above with reference to FIG. Figure 9 The intra prediction mode for generating prediction samples not based on the prediction direction may be referred to as a non-directional intra prediction mode (or non-directional mode), such as that described above with reference to Figure 10-11 Non-directional intra prediction modes (or non-directional modes) may include, for example Figure 10 The non-directional smoothing modes described in (e.g., DC mode, PAETH mode, SMOOTH mode, SMOOTH_V mode, and SMOOTH_H mode), such as Figure 11 Filter intra mode (or recursive filtering mode), CfL mode, etc. described in

[0045] Certain non-directional intra prediction modes (e.g., non-directional smoothing modes including DC mode, PAETH mode, SMOOTH mode, SMOOTH_V mode, and SMOOTH_H mode, and filter intra mode) may be referred to as smoothing modes.

[0157] The coding information of the block to be reconstructed (e.g., CB) can be decoded from the coded video bitstream. The coding information can indicate the intra-frame prediction information of the block. The intra-frame prediction information can indicate the intra-frame prediction mode used to predict the block. The residual of the block can be encoded using a non-separable transform. The non-separable transform can be a primary transform or a secondary transform.

[0158] In an example, the intra prediction mode is a directional mode, and the block is encoded using the directional mode. The encoding information may indicate nominal mode information and angular offset information for the block. The nominal mode information may include a first index (e.g., a nominal mode index) signaled in the coded video bitstream that indicates the nominal mode of the block (e.g., D45_PRED). The angular offset information may include a second index (e.g., an angular offset index) signaled in the coded video bitstream that indicates an angular offset from a nominal angle (e.g., 45°) corresponding to the nominal mode.

[0159] According to aspects of the present disclosure, a directional mode may be determined based on a nominal mode and an angular offset indicated by encoded information.

[0160] The nominal angle (e.g., 45°) can be obtained based on the nominal mode (e.g., D45_PRED), as Figure 9As shown. For example, the second index may indicate an offset integer of 1. Therefore, the angle offset may be obtained based on the offset integer of 1 and the step size (e.g., 3°), for example, the angle offset is equal to the offset integer of 1 multiplied by the step size. Therefore, the predicted angle may be obtained based on the nominal angle and the angle offset. In the example, the predicted angle is equal to the sum of the nominal angle and the angle offset, as described above with reference to Figure 9 Therefore, as described above, Figure 9 As described, the direction mode (or intra prediction mode) may be determined according to the prediction angle.

[0161] The inseparable transform of the block may be determined based on the nominal mode. In an embodiment, a transform set mode (e.g., stMode) associated with the nominal mode may be determined, where the transform set mode may indicate a set of one or more inseparable transforms including the inseparable transform. The set of one or more inseparable transforms may be referred to as a transform set.

[0162] In an example, a directional mode is mapped to a transform set mode based on a mapping relationship between a plurality of nominal modes and a transform set mode (e.g., stMode). The mapping relationship can be stored in the encoder and the decoder. The mapping relationship can be a lookup table. According to aspects of the present disclosure, the directional mode is mapped to the transform set mode based on the nominal mode associated with the directional mode (e.g., indicated by the nominal mode index) and regardless of the angular offset associated with the directional mode. The value of the transform set mode (e.g., stMode) can then be used to identify the transform set.

[0163] As described above, a transform set may include one or more inseparable transforms. A signaled inseparable transform index (e.g., stIdx) may be used to further identify the inseparable transforms in the transform set. The coding information may further include an inseparable transform index (e.g., stIdx) signaled in the coded video bitstream. The inseparable transforms in the transform set may be determined based on the inseparable transform index. A block may be decoded based on the inseparable transform.

[0164] The block can be reconstructed based on the directional mode and the inseparable transform.

[0165] In an example, if a block is encoded using transform skip mode, then the non-separable transform is not applied to the block. For example, the non-separable transform index (eg, stIdx) is not signaled, thus indicating that the non-separable transform is not applied to the block.

[0166] In an embodiment, the non-separable transform is a non-separable secondary transform. In some examples, the non-separable secondary transform is not applied in the PAETH mode or the recursive filtering mode. In some examples, the non-separable secondary transform is applied only to the first N transform coefficients along the scan order used to entropy encode the first transform coefficient of the block. For example, on the encoder side, the input of a forward secondary transform (e.g., a forward non-separable secondary transform) is the first N transform coefficients along the forward scan order, and the output of the forward secondary transform is N modified transform coefficients that replace the first N transform coefficients. On the decoder side, the input of an inverse secondary transform (e.g., an inverse non-separable secondary transform) is the first N transform coefficients along the forward scan order (e.g., the N modified transform coefficients from the output of the forward secondary transform), and the output of the inverse secondary transform is N transform coefficients that replace the first N transform coefficients. The scan order, e.g., the forward scan order, can be any suitable scan order, such as a zigzag scan order, a diagonal scan order, etc. In examples, such as in VVC, a diagonal scan order is used for coefficient encoding. N can be any non-negative integer, such as in the range of 0 to 127. In an example, no secondary transform is applied to the block when N is 0. In an example, N is a positive integer.

[0167] In an embodiment, the inseparable transform is an inseparable secondary transform. The inseparable secondary transform is applied only to the first transform coefficients in the block. Each of the first transform coefficients may have coordinates (x, y), and the sum of the corresponding x-coordinate and y-coordinate may be less than a threshold value T. For example, if the coordinates (x, y) associated with the corresponding transform coefficient in the block satisfy the condition that the sum of the corresponding x-coordinate and y-coordinate is greater than or equal to the threshold value T, the transform coefficient may be set to 0. The transform coefficient is not included in the first transform coefficients. The threshold value T may be any suitable number, such as a non-negative integer. In the example, T is in the range of 0 to 32.

[0168] In an embodiment, the inseparable transform is applied as an inseparable secondary transform. In an example, the concurrent signaling inseparable secondary transform is applied only when the primary transform type of the block meets the condition. According to aspects of the present disclosure, the horizontal transform and the vertical transform in the primary transform of the block can be included in a subset of the LGT set. The horizontal transform and the vertical transform can use the same LGT or different LGTs. The subset of the LGT set can be represented by the edge weights w c and self-loop weights (e.g., v c1 and / or v c2 ) are characterized by the relationship between them. For example, a subset of the LGT set is characterized by a certain self-loop ratio. The self-loop ratio can be based on the edge weight w c and self-loop weights (e.g., v c1 or v c2 ) ratio.

[0169] In the example, the non-separable secondary transform is applied only when the horizontal transform and the vertical transform of the main transform are DCT-2. DCT-2 may correspond to a value with v c1 = 0 (for example, the self-circulation ratio is 0).

[0170] In the example, the non-separable secondary transform is applied only when the horizontal transform and the vertical transform of the main transform are DST-4 (or DST-7). DST-7 may correspond to a V c1 =w c (For example, LGT with a self-circulation ratio of 1).

[0171] In this example, the non-separable secondary transform is applied only when the following conditions are met: The horizontal transform of the main transform is DCT-2, DST-4, or DST-7, and the vertical transform of the main transform is DCT-2, DST-4, or DST-7. The horizontal transform and the vertical transform can have the same type of transform matrix or different types of transform matrices.

[0172] In an embodiment, the non-separable transform is applied as a non-separable secondary transform. According to aspects of the present disclosure, a block may include a first transform coefficient obtained using the non-separable secondary transform and a second transform coefficient obtained without the non-separable secondary transform. The first transform coefficient and the second transform coefficient may be entropy encoded (e.g., decoded) separately. In an example, the first transform coefficient is entropy encoded (e.g., decoded) before the second transform coefficient is entropy encoded (e.g., decoded). In an example, the second transform coefficient is entropy encoded (e.g., decoded) before the first transform coefficient is entropy encoded (e.g., decoded).

[0173] As described above, the non-directional intra prediction mode (or non-directional mode) may include a DC mode, a PAETH mode, a SMOOTH mode, a SMOOTH_V mode, a SMOOTH_H mode, a recursive filtering mode, and a CfL mode. The non-directional smoothing mode may include a DC mode, a PAETH mode, a SMOOTH mode, a SMOOTH_V mode, and a SMOOTH_H mode, as described above with reference to Figure 10 described.

[0174] In an embodiment, a block is encoded using one of the non-directional modes. Thus, a set of one or more inseparable transforms associated with one of the non-directional modes may be determined, wherein one of the following may be associated with the set of one or more inseparable transforms: (a) at least one other of the non-directional modes, and (b) a nominal mode. The nominal mode may be one of eight nominal modes (901)-(908) V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED, such as Figure 9 Similar to the above, an inseparable transform in a set of one or more inseparable transforms may be determined based on an inseparable transform index indicated by the encoding information. Subsequently, a block may be reconstructed based on the non-directional mode and the inseparable transform.

[0175] One of the non-directional modes, and one of the following: at least another of the non-directional modes and the nominal mode, which may include one of the following: (a) a recursive filtering mode, and one of a DC mode and a SMOOTH mode; (b) a SMOOTH mode, a SMOOTH_H mode, and a SMOOTH_V mode; (c) a SMOOTH mode, a SMOOTH_H mode, a SMOOTH_V mode, and a PAETH mode; (d) a recursive filtering mode, a SMOOTH mode, and a PAETH mode; (e) a V_PRED (or a vertical mode for the nominal mode) and a SMOOTH_V mode; (f) a H_PRED (or a horizontal mode for the nominal mode) and a SMOOTH_H mode; and (v) a CfL mode, and one of a DC mode, a SMOOTH mode, and a PAETH mode.

[0176] Typically, the same set of one or more inseparable transformations can be applied to two or more modes in a mode set, which includes multiple nominal modes (e.g., nominal modes (901)-(908)) and multiple non-directional modes (e.g., the non-directional modes described above).

[0177] The same set of one or more non-separable transforms (or the same set of one or more non-separable transform kernels) may be applied to the recursive filtering mode (or recursive filtering mode) and the DC mode.

[0178] The same set of one or more non-separable transforms (or the same set of one or more non-separable transform kernels) may be applied to the recursive filtering mode (or recursive filtering mode) and the SMOOTH mode.

[0179] The same set of one or more inseparable transforms (or the same set of one or more inseparable transform kernels) may be applied to the SMOOTH mode, the SMOOTH_H mode, and the SMOOTH_V mode.

[0180] The same set of one or more inseparable transforms (or the same set of one or more inseparable transform kernels) can be applied to SMOOTH mode, SMOOTH_H mode, SMOOTH_V mode and PAETH mode.

[0181] The same set of one or more non-separable transforms (or the same set of one or more non-separable transform kernels) may be applied to the recursive filtering mode, the SMOOTH mode, and the PAETH mode.

[0182] The same set of one or more non-separable transforms (or the same set of one or more non-separable transform kernels) may be applied to both the vertical mode (V_PRED) and the SMOOTH_V mode.

[0183] The same set of one or more non-separable transforms (or the same set of one or more non-separable transform kernels) may be applied to both the horizontal mode (H_PRED) and the SMOOTH_H prediction mode.

[0184] The same set of one or more inseparable transforms (or the same set of one or more inseparable transform kernels) can be applied to both CfL mode and DC mode.

[0185] The same set of one or more inseparable transforms (or the same set of one or more inseparable transform kernels) may be applied to both CfL mode and SMOOTH mode.

[0186] The same set of one or more inseparable transforms (or the same set of one or more inseparable transform kernels) can be applied to both the CfL mode and the PAETH mode.

[0187] Figure 16A flow chart of an overview process (1600) according to an embodiment of the present disclosure is shown. The process (1600) can be used for reconstruction of a block. In various embodiments, the process (1600) is performed by a processing circuit such as a processing circuit in a terminal device (310), (320), (330) and (340), a processing circuit that performs the functions of a video encoder (403), a processing circuit that performs the functions of a video decoder (410), a processing circuit that performs the functions of a video decoder (510), a processing circuit that performs the functions of a video encoder (603), etc. In some embodiments, the process (1600) is implemented as software instructions, so that when the processing circuit executes the software instructions, the processing circuit performs the process (1600). The process starts at (S1601) and proceeds to (S1610).

[0188] At (S1610), encoding information of a block to be reconstructed may be decoded from an encoded video bitstream. The encoding information may indicate intra-frame prediction information of the block.

[0189] At (S1620), for the block encoded using a directional mode, the directional mode may be determined based on a nominal mode and an angular offset, as described above. The encoding information may indicate the nominal mode (e.g., using a nominal mode index signaled in the encoded video bitstream) and the angular offset (e.g., an angular offset index signaled in the encoded video bitstream).

[0190] At (S1630), for the block encoded using the directional mode, an inseparable transform for the block may be determined based on the nominal mode, as described above. A set of one or more inseparable transforms including the inseparable transform may be determined based on the nominal mode. For example, a transform set mode associated with the nominal mode may be determined, the transform set mode indicating the set of one or more inseparable transforms. In addition, the encoding information may indicate an inseparable transform index. The inseparable transform in the set of one or more inseparable transforms may be determined based on the inseparable transform index.

[0191] At (S1640), for the block encoded using a directional mode, the block may be reconstructed based on the directional mode and the inseparable transform. In an example, the inseparable transform is a primary transform, and the inseparable primary transform is applied to the block. In an example, the inseparable transform is a secondary transform, and the inseparable secondary transform is applied to the block. The process (1600) proceeds to (S1699) and ends.

[0192] The process (1600) can be appropriately adapted to various scenarios, and the steps in the process (1600) can be adjusted accordingly. One or more steps in the process (1600) can be modified, omitted, repeated, and / or combined. The process (1600) can be implemented in any suitable order.

[0193] Each of the multiple methods (or embodiments), the encoder and the decoder can be implemented by a processing circuit (eg, one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium.

[0194] The above techniques can be implemented as computer software through computer-readable instructions and physically stored in one or more computer-readable media. For example, Figure 17 A computer system (1700) is shown that is suitable for implementing certain embodiments of the disclosed subject matter.

[0195] The computer software may be encoded in any suitable machine code or computer language, and may be assembled, compiled, linked, or other mechanisms to create code comprising instructions, which may be directly executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), or the like, or executed through decoding, microcode, or the like.

[0196] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablets, servers, smartphones, gaming devices, IoT devices, and the like.

[0197] Figure 17 The components shown for the computer system (1700) are exemplary in nature and are not intended to limit the scope of use or functionality of computer software implementing embodiments of the present disclosure. Nor should the configuration of components be interpreted as having any dependency or requirement on any one or combination of components shown in the exemplary embodiment of the computer system (1700).

[0198] The computer system (1700) may include certain human-computer interface input devices. Such human-computer interface input devices may respond to input from one or more human users through tactile input (e.g., keyboard input, swiping, data glove movement), audio input (e.g., sound, applause), visual input (e.g., gestures), and olfactory input (not shown). The human-computer interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0199] The human-computer interface input device may include one or more of the following (only one of which is shown): keyboard (1701), mouse (1702), touchpad (1703), touch screen (1710), data gloves (not shown), joystick (1705), microphone (1706), scanner (1707), camera (1708).

[0200] The computer system (1700) may also include certain human-computer interface output devices. Such human-computer interface output devices may stimulate one or more senses of a human user through, for example, tactile output, sound, light, and smell / taste. Such human-computer interface output devices may include tactile output devices (e.g., tactile feedback through a touch screen (1710), a data glove (not shown), or a joystick (1705), but there may also be tactile feedback devices that are not used as input devices), audio output devices (e.g., speakers (1709), headphones (not shown)), visual output devices (e.g., screens (1710) including cathode ray tube screens, liquid crystal screens, plasma screens, organic light emitting diode screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities - some of which may output two-dimensional visual output or output in more than three dimensions through means such as stereoscopic image output; virtual reality glasses (not shown), holographic displays, and cigarette boxes (not shown)), and printers (not shown).

[0201] The computer system (1700) may also include human-accessible storage devices and their associated media, such as optical media including high-density read-only / rewritable optical disks (CD / DVD ROM / RW) (1720) with CD / DVD or similar media (1721), a thumb drive (1722), a removable hard drive or solid state drive (1723), traditional magnetic media such as tapes and floppy disks (not shown), special-purpose ROM / ASIC / PLD-based devices such as security software dongles (not shown), and the like.

[0202] Those skilled in the art will also understand that the term "computer-readable media" used in connection with the disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0203] The computer system (1700) may also include an interface (1754) to one or more communication networks (1755). For example, the network may be wireless, wired, or optical. The network may also be a local area network, a wide area network, a metropolitan area network, an in-vehicle network, an industrial network, a real-time network, a delay-tolerant network, and the like. Networks also include local area networks such as Ethernet, wireless local area networks, cellular networks (GSM, 3G, 4G, 5G, LTE, etc.), television wired or wireless wide area digital networks (including cable television, satellite television, and terrestrial broadcast television), in-vehicle and industrial networks (including CANBus), and the like. Some networks typically require an external network interface adapter for connecting to some common data port or peripheral bus (1749) (for example, a USB port of the computer system (1700)); other systems are typically integrated into the core of the computer system (1700) by connecting to a system bus as described below (for example, an Ethernet interface integrated into a PC computer system or a cellular network interface integrated into a smartphone computer system). By using any of these networks, the computer system (1700) can communicate with other entities. The communication can be one-way, for receiving only (e.g., wireless television), one-way, for sending only (e.g., a CAN bus to certain CAN bus devices), or two-way, such as to other computer systems via a local or wide area digital network. Each of the above networks and network interfaces can use certain protocols and protocol stacks.

[0204] The aforementioned human-machine interface devices, human-accessible storage devices, and network interfaces may be connected to the core (1740) of the computer system (1700).

[0205] The core (1740) may include one or more central processing units (CPUs) (1741), graphics processing units (GPUs) (1742), specialized programmable processing units in the form of field programmable gate arrays (FPGAs) (1743), hardware accelerators for specific tasks (1744), a graphics adapter (1750), and the like. These devices, as well as read-only memory (ROM) (1745), random access memory (1746), internal mass storage (e.g., an internal non-user accessible hard drive, solid-state drive, etc.) (1747), and the like, may be connected via a system bus (1748). In some computer systems, the system bus (1748) may be accessed in the form of one or more physical plugs to allow expansion with additional central processing units, graphics processing units, and the like. Peripheral devices may be attached directly to the core's system bus (1748) or connected via a peripheral bus (1749). In an example, a touch screen (1710) may be connected to a graphics adapter (1750). Peripheral bus architectures include peripheral component interconnect (PCI), universal serial bus (USB), and the like.

[0206] The CPU (1741), GPU (1742), FPGA (1743), and accelerator (1744) can execute certain instructions, which, when combined, can constitute the aforementioned computer code. The computer code can be stored in ROM (1745) or RAM (1746). Transient data can also be stored in RAM (1746), while permanent data can be stored, for example, in internal mass storage (1747). Fast storage and retrieval of any memory device can be achieved by using a cache memory, which can be closely associated with one or more CPUs (1741), GPUs (1742), mass storage (1747), ROM (1745), RAM (1746), etc.

[0207] The computer readable medium may have computer code thereon for performing various computer-implemented operations. The medium and computer code may be specially designed and constructed for the purposes of the present disclosure, or may be medium and code well known and available to those skilled in the art of computer software.

[0208] As an example and not a limitation, a computer system having the architecture (1700), in particular the core (1740), can provide functionality as a processor (including a CPU, GPU, FPGA, accelerator, etc.) to execute software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with the aforementioned user-accessible mass storage, as well as specific memory of the core (1740) having non-volatile properties, such as the core's internal mass storage (1747) or ROM (1745). Software implementing various embodiments of the present disclosure can be stored in such a device and executed by the core (1740). Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software can enable the core (1740), in particular the processor therein (including a CPU, GPU, FPGA, etc.) to perform a specific process or a specific part of a specific process described herein, including defining a data structure stored in RAM (1746) and modifying such a data structure according to a software-defined process. Additionally or alternatively, the computer system may provide functionality hardwired in logic or otherwise contained in circuitry (e.g., accelerator (1744)) that may operate in place of or in conjunction with software to perform specific processes or specific portions of specific processes described herein. Where appropriate, references to software may include logic and vice versa. Where appropriate, references to computer-readable media may include circuitry (e.g., an integrated circuit (IC)) storing the executing software, circuitry containing the executing logic, or both. The present disclosure includes any suitable combination of hardware and software.

[0209] Appendix A: Acronyms

[0210] JEM: Joint Exploration Model

[0211] VVC: universal video coding

[0212] BMS: Benchmark set

[0213] MV: Motion Vector

[0214] HEVC: High Efficiency Video Coding

[0215] SEI: Supplementary Enhancement Information

[0216] VUI: Video Usability Information

[0217] GOPs: Groups of Pictures

[0218] TUs: Transform Units

[0219] PUs: Prediction Units

[0220] CTUs: Coding Tree Units

[0221] CTBs: Coding Tree Blocks

[0222] PBs: Prediction Blocks

[0223] HRD: Hypothetical Reference Decoder

[0224] SNR: Signal-to-Noise Ratio

[0225] CPUs: Central Processing Units

[0226] GPUs: Graphics Processing Units

[0227] CRT: Cathode Ray Tube

[0228] LCD: Liquid-Crystal Display

[0229] OLED: Organic Light-Emitting Diode

[0230] CD: Compact Disc

[0231] DVD: Digital Video Disc

[0232] ROM: Read-Only Memory

[0233] RAM: Random Access Memory

[0234] ASIC: Application-Specific Integrated Circuit

[0235] PLD: Programmable Logic Device

[0236] LAN: Local Area Network

[0237] GSM: Global System for Mobile communications

[0238] LTE: Long-Term Evolution

[0239] CANBus: Controller Area Network Bus

[0240] USB: Universal Serial Bus

[0241] PCI: Peripheral Component Interconnect

[0242] FPGA: Field Programmable Gate Areas

[0243] SSD: Solid-state Drive

[0244] IC:Integrated Circuit

[0245] CU: Coding Unit

[0246] Although this disclosure has described a number of exemplary embodiments, various modifications, permutations, and equivalent substitutions of the embodiments are within the scope of this disclosure. Therefore, it should be understood that those skilled in the art will be able to devise a variety of systems and methods that, although not explicitly shown or described herein, embody the principles of this disclosure and are therefore within the spirit and scope of this disclosure.

Claims

1. A video decoding method, characterized in that: include: Decoding encoding information of a block to be reconstructed from an encoded video bitstream, wherein the encoding information indicates intra-frame prediction information of the block; as well as When the block is encoded using directional mode, determining the directional mode based on a nominal mode and an angular offset, the encoded information indicating the nominal mode and the angular offset, determining a non-separable transform for the block based on the nominal mode, the non-separable transform being a non-separable quadratic transform, and reconstructing the block based on the directional mode and the inseparable transform; The reconstructing the block based on the directional mode and the inseparable secondary transform further comprises: The non-separable secondary transform is applied only when the main transform type of the block satisfies one of the following: (a) the horizontal transform and the vertical transform of the main transform are DCT-2, (b) the horizontal transform and the vertical transform of the main transform are DST-4 or the horizontal transform and the vertical transform of the main transform are DST-7, (c) the horizontal transform of the main transform is DCT-2, DST-4 or DST-7, and the vertical transform of the main transform is DCT-2, DST-4 or DST-7, wherein the horizontal transform and the vertical transform have the same type of transform matrix or different types of transform matrices.

2. The method according to claim 1, characterized in that The determining of the inseparable transform of the block further comprises: determining a transform set mode associated with the nominal mode, The transform set pattern indicates one or more inseparable transform sets including the inseparable transform.

3. The method according to claim 2, characterized in that The encoding information further indicates a non-separable transform index; and The reconstructing the block further comprises: determining the inseparable transform in the one or more inseparable transform sets based on the inseparable transform index; and The block is reconstructed based on the directional mode and the inseparable transform.

4. The method according to claim 1, wherein The non-separable secondary transform is not applied to at least one of the PAETH mode and the recursive filtering mode.

5. The method according to claim 1, wherein The reconstructing the block based on the directional mode and the inseparable secondary transform further comprises: The non-separable secondary transform is applied only to the first N transform coefficients along a scanning order for entropy encoding a first transform coefficient of the block.

6. The method according to claim 1, characterized in that The reconstructing the block based on the directional mode and the inseparable secondary transform further comprises: The non-separable secondary transform is applied only to first transform coefficients in the block, each of the first transform coefficients having coordinates (x, y) and a sum of corresponding x-coordinates and y-coordinates being less than a threshold.

7. The method according to claim 1, characterized in that The horizontal transform and the vertical transform in the main transform of the block are included in a subset of the line transform set.

8. The method according to claim 1, characterized in that The block includes a first transform coefficient obtained using the non-separable secondary transform and a second transform coefficient obtained without using the non-separable secondary transform; and The reconstructing the block further includes entropy decoding the first transform coefficient and the second transform coefficient, respectively.

9. The method according to any one of claims 1 to 8, characterized in that The non-directional modes include DC mode, PAETH mode, SMOOTH mode, SMOOTH_V mode, SMOOTH_H mode, recursive filtering mode, and chrominance prediction from luminance CfL mode, wherein the DC mode, the PAETH mode, the SMOOTH mode, the SMOOTH_V mode, and the SMOOTH_H mode are determined based on the average of adjacent samples of the block, and When the block is encoded using one of the non-directional modes, Determining one or more inseparable transform sets associated with one of the non-directional modes, one of the following being associated with the one or more inseparable transform sets: (a) at least one other of said non-directional modes, and (b) a non-directional nominal mode, determining an inseparable transform in the one or more inseparable transform sets based on an inseparable transform index indicated by the encoding information, and The block is reconstructed based on the non-directional mode and the non-separable transform.

10. The method according to claim 9, characterized in that one of the non-directional modes, and one of the following: at least another of the non-directional modes and the non-directional nominal mode, including one of the following: (a) the recursive filtering mode, and one of the DC mode and the SMOOTH mode; (b) the SMOOTH mode, the SMOOTH_H mode and the SMOOTH_V mode; (c) the SMOOTH mode, the SMOOTH_H mode, the SMOOTH_V mode and the PAETH mode; (d) the recursive filtering mode, the SMOOTH mode and the PAETH mode; (e) the vertical mode for the nominal mode, and the SMOOTH_V mode; (f) the horizontal mode for the nominal mode, and the SMOOTH_H mode; and (v) the CfL mode, and one of the DC mode, the SMOOTH mode and the PAETH mode.

11. A video encoding method, characterized in that: include: generating coding information of a block to be decoded and included in an encoded video bitstream, the coding information indicating intra-frame prediction information of the block; as well as When the block is encoded using directional mode, determining the directional mode based on a nominal mode and an angular offset, the encoded information indicating the nominal mode and the angular offset, determining a non-separable transform for the block based on the nominal mode, the non-separable transform being a non-separable quadratic transform, and The block is encoded based on the directional mode and the inseparable transform, wherein the encoding includes: applying the inseparable secondary transform only when the main transform type of the block satisfies one of the following: (a) the horizontal transform and the vertical transform of the main transform are DCT-2, (b) the horizontal transform and the vertical transform of the main transform are DST-4 or the horizontal transform and the vertical transform of the main transform are DST-7, (c) the horizontal transform of the main transform is DCT-2, DST-4 or DST-7, and the vertical transform of the main transform is DCT-2, DST-4 or DST-7, wherein the horizontal transform and the vertical transform have the same type of transform matrix or different types of transform matrices.

12. A method for storing a bit stream, characterized in that: Execute the video encoding method according to claim 11 to generate a bit stream, or execute the video decoding method according to any one of claims 1 to 10 to decode the bit stream; and store the bit stream.

13. A method for transmitting a bit stream, characterized in that: Execute the video encoding method according to claim 11 to generate a bit stream, or execute the video decoding method according to any one of claims 1 to 10 to decode the bit stream; and transmit the bit stream.

14. A video decoding device, characterized in that: The device comprises a processing circuit configured to execute the video decoding method according to any one of claims 1 to 10.

15. A video encoding device, characterized in that: The apparatus comprises a processing circuit configured to perform the video encoding method of claim 11 .

16. A non-volatile computer readable medium having stored thereon a computer program / instructions and a bitstream, characterized in that: When the computer program / instructions are executed by a processor, the computer program / instructions implement the steps of the video encoding method according to claim 11 to generate the bit stream, or implement the steps of the video decoding method according to any one of claims 1 to 10 to decode the bit stream.

Images